RU216092U1 - AUTOMATIC ADDITION OF THE ENDOSCOPIC VIDEO IMAGE - Google Patents

Abstract

A device for automatically completing an endoscopic video image for a task based on state data that describes the state of the surgical robot used to at least partially perform this task. The adder device contains an event detector and an adder. The event detector is configured to receive state data; and identifying one or more patterns in the state data indicating the occurrence of an event during task execution. The supplement means is configured to receive an endoscopic video image captured during the execution of a task, the endoscopic video image being synchronized in time with the status data; and, in response to the event detector identifying a pattern in the state data indicating the occurrence of an event, automatically supplementing the metadata forming part of or associated with the endoscopic video image with information identifying the event and the time of occurrence of the event relative to the endoscopic video image. 17 w.p. f-ly, 13 figs.

Description

BACKGROUND TO CREATING A UTILITY MODEL

[0001] The use of robots for assisting and performing surgical operations is known. In FIG. 1 depicts a conventional surgical robot 100 consisting of a base 108, an arm 102, and an instrument 105. The base provides support for the robot and is itself rigidly attached to, for example, the operating room floor, operating room ceiling, or cart. The hand passes between the base and the instrument. The arm is articulated through a plurality of flexible joints 103 along its length, which are used to position the surgical instrument in a predetermined position relative to the patient. The surgical instrument is attached to the distal end 104 of the robot's arm. The surgical instrument enters the body of the patient 101 at the trocar insertion site 107 in order to access the area of the surgical intervention. The instrument contains at its distal end an operating member 106 for participating in a medical procedure.

[0002] The surgical robot 100 is remotely controlled by an operator (eg, a surgeon) using the operator console 200 shown in FIG. 2. The operator console 200 may be located in the same room (eg, operating room) as the surgical robot 100 or away from it. The operator console 200 includes input devices 202, 204 for controlling the state of the hand 102 and/or instrument 105 attached to it. The input devices 202, 204 may be handles or manual controls mounted on pantograph mechanisms. The control means converts the movement of the manual controls into control signals in order to drive the joints of the arms and/or the working body of the instrument of the surgical robot. The operator console 200 also includes a display 206. The display 206 is positioned so that it is visible to the user operating the input devices 202, 204. The display is used to display the video stream from the surgical site (eg, endoscopic video).

[0003] Some surgical procedures may require multiple surgical robots, each of which carries an instrument or other device, that are used simultaneously with each other in the surgical area. In FIG. 3 depicts a surgical robotic tool 300 with a plurality of robots 302, 304, 306 operating in a common workspace with respect to a patient 308. For example, surgical robots are often used in endoscopic surgery (eg, laparoscopic surgery), which may also be referred to as minimally invasive surgery. As is known to those skilled in the art, during an endoscopic procedure, the surgeon inserts the endoscope through a small incision or natural body opening such as, but not limited to, the mouth or nostrils. An endoscope is a rigid or flexible tube with a miniature camera attached to it that transmits to a video monitor (eg, display 206) real-time images that the surgeon uses to help guide his instruments through the same incision/hole or through another incision/hole. The endoscope allows the surgeon to see a significant area of the body in detail without the need to open and expose the corresponding area. This solution allows the surgeon to see inside the patient's body and operate through a much smaller incision than would be required for traditional open surgery. Accordingly, in conventional robotic endoscopic surgery, there is an endoscope attached to one arm of the surgical robot and one or more surgical instruments such as forceps and/or a scalpel attached to one or more arms of other surgical robots.

[0004] In FIG. 4 shows an exemplary endoscope 400 configured to be attached to the end of a robotic arm for use in minimally invasive surgery. The endoscope 400 includes a distal end 402 for insertion into the patient's surgical site and a proximal end 404. The distal end 402 is connected to the proximal end 404 by an elongated shaft 406. The proximal end 404 includes an interface 408 for engaging with the end of the robot arm.

[0005] The endoscope 400 includes a power source and a light source for illuminating a surgical area. The endoscope 400 also includes a data link for retrieving image data from the surgical site. They can all be attached to the proximal end 404 of the endoscope 400 independently and externally to the robot arm, as shown in FIG. 4. In FIG. 4, power is supplied through shaft 412, image data is retrieved through shaft 412, and light is delivered through light shaft 410. In an alternative embodiment, any one or more of light input, power input, and data output may be supplied to/retrieved from the endoscope through the arm. robot.

[0006] The endoscope 400 is mounted on the end of the robot's arm. The endoscope interface 408 comes into contact with the reciprocal interface of the robot arm. The endoscope 400 is configured to be attached to and detached from the robot arm by interfaces between the robot arm and the endoscope. In some cases, the endoscope 400 is configured to operate independently of the robot's arm when in a detached state. In other words, in these cases, the endoscope 400, if detached from the robot arm, can be operated manually by a member of the operating room staff.

[0007] In addition to images captured by the endoscope (which may be collectively referred to herein as endoscopic video) and used during surgery, images captured by the endoscope can be recorded and subsequently used for a variety of purposes such as, but not limited to, studying surgical procedures and/or training in surgical procedures and evaluation and/or critical review of the surgeon's performance (by a third party or by the surgeon himself).

[0008] One of the advantages of robotic surgery over traditional manual surgery is that it allows easier collection of data on the performance of surgical procedures. For example, the movement and/or position of the robot(s) during a surgical operation may be monitored and recorded. This may be advantageous in allowing the endoscopic video image to be linked or otherwise connected to data generated/collected by or about the surgical robot(s) in order to increase the usefulness of the endoscopic video image in training, performance evaluation/criticism, etc.

[0009] The embodiments described below are provided by way of example only and are not intended to be limiting to implementations that overcome any or all of the shortcomings of known surgical robotics.

BRIEF DESCRIPTION OF THE ESSENCE OF THE UTILITY MODEL

[0010] This brief description is presented to introduce a selection of concepts, which are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0011] Described herein are methods and means for automatically augmenting an endoscopic video image for a task at least partially performed by a surgical robot based on state data describing the state of the surgical robot during the task. The tool contains an event detector and an add-on device. The event detector is configured to receive state data; and identifying one or more patterns in the state data indicating the occurrence of an event during task execution. The add-on device is configured to: receive an endoscopic video image captured during a task, the endoscopic video image being synchronized in time with the status data; and, in response to the event detector's identification in the state data of a pattern indicating the occurrence of an event, supplementing the metadata forming part of or associated with the endoscopic video image with information identifying the event and the time of occurrence of the event relative to the endoscopic video image.

[0012] In a first aspect, an add-on device is provided for generating an augmented endoscopic video image, the add-on device comprising: an event detector configured to receive state data describing the state of the surgical robot during a task at least partially performed by the surgical robot wherein the surgical robotic tool comprises at least one surgical robot comprising a base and an arm extending from the base to the instrument attachment site, the arm comprising articulations with which the arm configuration can be changed; and identifying patterns in the state data indicating the occurrence of an event during task execution; and supplementary means, configured to: receive an endoscopic video image captured during a task, the endoscopic video image being synchronized in time with the status data; and, in response to the event detector identifying a pattern in the state data indicating the occurrence of an event, automatically supplementing the metadata forming part of or associated with the endoscopic video image with information identifying the event and the time of the event relative to the endoscopic video image.

[0013] Information identifying the event and the time of the event may be visible to the user when the user subsequently views the augmented endoscopic video image.

[0014] Information identifying the event and the time of the event may be selectively visible to the user when the user subsequently views the augmented endoscopic video image.

[0015] The information identifying the event and the time of the event may or may not be automatically visible to the user when the user subsequently views the augmented endoscopic video based on the privileges associated with the user.

[0016] Complementing the metadata with information identifying the event and the time of the event relative to the endoscopic video image may include adding a bookmark to the endoscopic video image at the time of the event that identifies the event.

[0017] One or more patterns may include a plurality of patterns, with each pattern corresponding to a different event, and the event detector may be configured to selectively identify only some subset of the plurality of patterns.

[0018] One or more patterns may include a plurality of patterns, with each pattern corresponding to a different event, and the event detector may be configured to identify a subset of the plurality of patterns based on user privileges associated with the user who ran the add-on.

[0019] The state data may at least partially comprise data generated by the surgical robotic tool.

[0020] The state data may at least partially comprise data generated by one or more sources external to the surgical robotic tool.

[0021] The patterns may be configured to identify one or more instrument events, wherein the one or more instrument events may include one or more of the following: changes to an instrument attached to a surgical robotic arm; if at least one instrument attached to the surgical robot arm of the surgical robot is an energized instrument, changing the energized state of the instrument; if at least one instrument attached to the arm of the surgical robot of the surgical robot is an endoscope, cleaning the endoscope; if at least one instrument attached to the surgical robot arm of the surgical robot is an endoscope, performing white balance adjustment on the endoscope imaging means; if at least one instrument attached to the arm of the surgical robot of the surgical robot is an endoscope, the magnitude or frequency of movement of the endoscope exceeds a certain range; and changing the instrument actively controlled by the surgical robot.

[0022] The patterns may be configured to identify one or more collision events, wherein the one or more collision events may include one or more of the following: collisions between at least two surgical robotic arms of the surgical robotic tool; collisions between at least two instruments attached to different arms of the surgical robot of the surgical robot; and collision between the surgical robot arm of the surgical robot and an instrument attached to the other arm of the surgical robot of the surgical robot.

[0023] A task may be a surgical procedure performed on a patient, and one or more patterns may be configured to identify one or more patient events, wherein one or more patient events may include one or more of the following: exit of one or more than the basic indicators of the state of the body or one or more other indicators of the patient's health outside a certain range; and changes in one or more vital signs and/or one or more other signs of a patient's health.

[0024] The surgical robotic tool may be under the control of an operator and the patterns may be configured to identify one or more operator events, wherein the one or more operator events may include one or more of the following: output of one or more vital signs, and /or one or more other indicators of the operator's health outside a certain range; changes in one or more vital signs and/or one or more other health indicators of the operator; changing the grip force applied by the operator to one or more input devices used to control the surgical robotic tool; and changing the pushing force applied by the operator to one or more of the input devices.

[0025] The surgical robotic tool may be under the control of an operator, and the patterns may be configured to identify one or more operator performance events, wherein the one or more operator performance events may include one or more of the following: operator performance drops while performing the task below acceptable level, and changes in the efficiency of the operator when performing the task.

[0026] The performance of an operator in performing a task may be based on one or more performance criteria, wherein the one or more performance criteria may include one or more of the following: the trajectory along which the surgical robot arm moves in performing a task step; smoothness of movement of the hand of the surgical robot; and time to complete a task step.

[0027] One or more patterns may be configured to identify abnormal events.

[0028] A task may include multiple steps, and one or more patterns may be configured to automatically identify one or more task steps based on state data.

[0029] The state data may include instrument information describing instruments attached to at least one surgical robot during a task, and surgical robot position information describing the movement and/or position of at least one surgical robot during execution. tasks; and the event detector may be configured to identify the suturing step during the execution of the task by identifying patterns in the instrument information and position information of the surgical robot.

[0030] An event detector may be configured to identify a suturing step during a task if the event detector, based on the instrument information, identifies at least two needle holders being attached to the surgical robots, and the event detector, based on the position information of the surgical robot, identifies, that surgical robots make repetitive circular motions.

[0031] The means may be further configured to associate the augmented endoscopic video image and all or part of the status data such that the event identification information identifies the time of occurrence of the event relative to the endoscopic video image and the status data.

[0032] The part of the endoscopic video image and the part of the status data related to the identified event can be made with the possibility of extracting from the information identifying the event.

[0033] The task may be a surgical procedure performed on a person.

[0034] The augmentation device may comprise a surgical robotic tool.

[0035] In a second aspect, a computer-implemented method for generating an augmented endoscopic video image is provided, the method comprising, on one or more devices on a computational basis: receiving an endoscopic video image captured during a task at least partially performed by a surgical robotic tool, when this surgical robotic tool contains at least one surgical robot containing a base and an arm extending from the base to the attachment point of the tool, while the arm contains a plurality of joints, with which you can change the configuration of the hand; receiving status data describing the state of the surgical robot during the execution of a task, the status data being synchronized in time with the endoscopic video image; identifying one or more patterns in the state data indicating the occurrence of an event during the execution of a task; and, in response to identifying in the state data a pattern indicating the occurrence of an event during the execution of a task, supplementing the metadata forming part of or associated with the endoscopic video image with information identifying the event and the time of the event relative to the endoscopic video image.

[0036] If necessary, the features described above may be combined, as will be apparent to those skilled in the art, and may be combined with any of the aspects of the examples described herein.

BRIEF DESCRIPTION OF GRAPHICS

[0037] Examples will now be described in detail with reference to the accompanying drawings, in which:

[0038] in FIG. 1 is a schematic diagram of an exemplary surgical robot performing an exemplary surgical procedure;

[0039] in FIG. 2 is a schematic diagram of an exemplary operator console;

[0040] in FIG. 3 is a schematic diagram of an exemplary surgical robot with a plurality of surgical robots;

[0041] in FIG. 4 is a schematic diagram of an exemplary endoscope capable of being attached to the arm of a surgical robot;

[0042] in FIG. 5 is a block diagram of an exemplary add-on device;

[0043] in FIG. 6 is a schematic diagram of an exemplary surgical robotic tool;

[0044] in FIG. 7 is a schematic diagram of an exemplary surgical robot;

[0045] in FIG. 8 is a flowchart of an exemplary method for augmenting an endoscopic video image;

[0046] in FIG. 9 is a schematic diagram of an exemplary operator console display;

[0047] in FIG. 10 is a schematic diagram of an exemplary augmented endoscopic video image;

[0048] in FIG. 11 is a schematic diagram of an exemplary augmented endoscopic video viewed in a media player;

[0049] in FIG. 12 is a schematic diagram of an exemplary augmented endoscopic video image associated with related status data; and

[0050] in FIG. 13 is a block diagram of an exemplary computational device.

[0051] The accompanying graphics illustrate various examples. Those skilled in the art will appreciate that depicted borders of elements (eg, blocks, groups of blocks, or other shapes) in graphics represent one example of borders. In some examples, there may be a case in which one element may be designated as a plurality of elements, or a plurality of elements may be designated as one element. Where appropriate, common reference numbers are used throughout the figures to indicate like elements.

DETAILED DESCRIPTION

[0052] The following description is provided as an example to enable those skilled in the art to make and use the present invention. The present invention is not limited to the embodiments described herein, and various modifications of the described embodiments will be apparent to those skilled in the art. Embodiments are described by way of example only.

[0053] Described herein are methods and apparatus for automatically augmenting an endoscopic video image captured during a task (e.g., surgery) performed at least partially by a surgical robot based on state data describing the state of the surgical robot during execution. tasks. The surgical robotic tool comprises at least one surgical robot comprising a base and an arm extending from the base to the instrument attachment site, the arm having articulations with which the arm configuration can be changed. The method includes receiving an endoscopic video image; receiving state data describing the state of the surgical robot during the execution of the task, which is synchronized in time with the endoscopic video image; identifying one or more patterns in the state data indicating the occurrence of an event during the execution of a task; and, in response to identifying a pattern in the state data indicating the occurrence of an event, supplementing the endoscopic video image with information identifying the event and the time of the event relative to the endoscopic video image.

[0054] Referring first to FIG. 5, which depicts an exemplary augmentation device 500 for automatically augmenting an endoscopic video image captured during a task (e.g., surgery) performed at least partially by a surgical robotic tool 502. The surgical robotic tool 502 includes at least one surgical robot comprising a base and an arm extending from the base to the point of attachment of the surgical instrument, the arm having a plurality of articulations with which the configuration of the arm can be changed. An exemplary surgical robotic tool 502 is described below with reference to FIG. 6.

[0055] The padding device 500 includes an event detector 504 and an padding tool 506 . The event detector 504 and/or adder 506 may be implemented by one or more computational devices, such as, but not limited to, the computational device described below with reference to FIG. 12.

[0056] The event detector 504 is configured to receive status data indicating or describing the state of the surgical robot 502 during a task (eg, surgery) performed by the robot. The status data is time synchronized with the endoscopic video captured during the task. The state data may be time synchronized with the endoscopic video image in any suitable manner that allows events detected based on the state data to be correlated with a particular point or time in the endoscopic video image. For example, both endoscopic video and status data may include or may be associated with common timestamps or a common timeline.

[0057] As described in more detail below, the state data may include any data or information about the surgical robot 502 and/or any information about the environment in which the surgical robot 502 operated during a task. All or part of the state data may be generated by the surgical robot 502 itself (e.g., data related to the position and/or movement of the surgical robot's arm(s), its attached instrument, and any hand controls that control the movement. robot arm(s) In some instances, at least a portion of the state data may be generated by one or more external sources.

[0058] For example, as shown in FIG. 5, a portion of the state data may be generated by audio and/or video recording equipment such as, but not limited to, a camera 508 that captures audio and/or video in a room (eg, operating room) where a task is being performed; and/or health monitoring equipment, such as, for example, but not limited to, a 510 heart rate monitor that monitors one or more health indicators of a patient (if the task is performed on a patient) and/or an operator (e.g., a surgeon) operating a surgical robotic tool 502. Those skilled in the art will appreciate that these equipments are only examples of external health data sources, and other external health data sources may be available. For example, other external sources of state data may include a hospital bed, wherein the hospital bed is a "smart" bed configured to generate, for example, position data; and equipment for insufflation. As is known to those skilled in the art, during laparoscopic surgery, gases such as, but not limited to, carbon dioxide are often blown into the patient's body using insufflation equipment to inflate the cavity to obtain a larger volume.

[0059] The event detector 504 is configured to receive status data generated by the surgical robot 502 and/or one or more external sources 508, 510 by any suitable means. For example, in some cases, the surgical robot 502 and/or one or more external sources 508, 510 are configured to deliver status data to the event detector 504 using a wireless or wired communication connection 512, such as, but not limited to, an Ethernet connection, a Wi-Fi® connection, a Bluetooth® connection, a Near Field Communication (NFC) connection, and the like. In these examples, all or part of the state data may be delivered to the event detector 504 in real time (or substantially real time) simultaneously with performing a task (for example, a surgical operation). In other cases, the state data may be captured by another device (eg, a computing-based device) during the execution of a task and stored in a data storage device 514 (eg, a storage device) and then delivered to the event detector 504 via the communication connection 512 or upon by any other suitable means. In some cases, all or part of the state data may be stored at the task execution location (e.g., in the operating room) on a portable storage medium such as, but not limited to, a USB (Universal Serial Bus) memory stick, and physically transferred to the location of the event detector 504 where it is connected to the event detector 504 so that the event detector 504 can read the status data from the portable storage medium.

[0060] The event detector 504 is configured to analyze the state data to identify events that occurred during the execution of a task (eg, a surgical operation). Events occurring during the execution of a task (eg, a surgical operation) may include, but are not limited to: a change in an instrument attached to a robotic arm of a surgical robotic tool; the occurrence of a collision between a plurality of robot arms or between a plurality of tools attached to the robot arms; the output of one or more basic indicators of the state of the body and/or one or more other indicators of the patient's health outside a certain range; the output of one or more key indicators of the operator's body condition (for example, a surgeon) and/or one or more other indicators of the operator's health outside a certain range; using the surgical robotic tool 502 to perform a specific action (eg, suturing); and a departure from the expected sequence of events (eg, while the suturing step was expected, the cutting step was being performed). A more extensive list of exemplary events that can be identified by the event detector 504 is described below. As described in more detail below, the event detector 504 can be configured to identify events occurring during the execution of a task by identifying patterns in the state data. For example, the event detector 504 may be configured to determine whether the surgical robot 502 has been used to perform a suture if the path followed by the robot arm or the needle tool attached to it indicates that the suture has been performed. Patterns indicating the occurrence of an event may be predetermined or may be learned from previously completed tasks.

[0061] In some cases, in response to detecting an event occurring during a task (e.g., a surgical operation) based on the state data, the event detector 504 may be configured to generate output indicating that an event has been detected, event type, time event, and optionally the duration of the event, and its delivery to the add-on engine 506.

[0062] The adder 506 is configured to receive an endoscopic video image captured by the endoscope 516 during a task. In some instances, endoscopic video may be captured by an endoscope 516 attached to and controlled by one of the robotic arms of the surgical robot 502. However, in other instances, endoscopic video may be captured by a manually operated endoscope. For example, a user may be able to remove an endoscope attached to one of the robot's arms and manually control the movement and actions of the endoscope. In some instances, if the user removes the endoscope from the robot arm and operates the endoscope manually, the surgical robot may no longer be able to accurately track the position and/or movement of the endoscope. As a consequence, the surgical robot may be configured to limit the response to control inputs when such a situation occurs.

[0063] The augmentation means 506 is configured to receive the endoscopic video image by any suitable means. For example, in some cases, the endoscope 516 itself may be configured to deliver the endoscopic video image to the add-on 506 using a wireless or wired communication connection 512, such as, but not limited to, an Ethernet connection, a Wi-Fi® connection, a Bluetooth® connection, a non-contact communication (NFC); and the like. In these examples, all or part of the endoscopic video image may be delivered to the adder 506 in real time (or substantially real time) during a task (eg, surgery). In other cases, the endoscopic video image may be stored in a data storage device 514 (eg, a storage device) and then delivered to an adder 506 via a communication connection 512 or by any other suitable means. For example, the endoscopic video image may be stored at a task location (eg, in an operating room) on a portable storage medium, such as, but not limited to, a USB memory stick, that is physically transferred to the location of the add-on 506, where it is connected to the add-on 506 so that that the adder 506 can read the endoscopic video image from the portable storage medium.

[0064] The adder 506 is configured to, in response to the detection by the event detector 504 based on the event occurrence state data during the execution of the task, to add information (e.g., a label) to the endoscopic video image identifying the occurrence of the event and the time of occurrence of the event, and optionally identifying the duration of the event, in order to generate an augmented endoscopic video image. In some cases, the information or label may not be visible to the user when the augmented endoscopic video is subsequently played back or viewed in a video player. For example, in some cases, the information or label may be made detectable by a dedicated tool or software. Special tool or software may be configured to determine the identified events based on the information or labels and present a list of identified events to the user. If the user selects one of the identified events, the means is configured to present or output a portion of the endoscopic video image related to the identified event.

[0065] In other cases, the information or label may be visible to the user when the augmented endoscopic video is subsequently played or viewed in a video player. In some cases, the label may take the form of a bookmark, allowing the user to easily jump to a specific point in the endoscopic video when the additional endoscopic video is played back by the media player. For example, when a user loads an augmented endoscopic video image into a media player, the user may be presented with a list of bookmarks in the video image, and the user may navigate directly to one of the points or segments identified by the bookmark by selecting the bookmark. In some cases, adder 506 may be configured to automatically group labels and/or bookmarks based on, for example, event type and/or some other criteria. In some cases, adder 506 may also be configured to automatically identify additional events based on the events identified by the status data and add information/label to the endoscopic video image identifying each additional event. In some cases, information (eg, a label) related to a particular event may be included in the metadata that forms part of or is associated with the endoscopic video image.

[0066] Add-on device 500 may be remote from surgical robot 502. For example, surgical robot 502 may be located in an operating room, and add-on device 500 may be located elsewhere in a hospital or treatment center.

[0067] Reference is made below to FIG. 6, which depicts a surgical robot 502. In this example, the surgical robot 502 includes two surgical robots 602 and 604 controlled by a control unit 606. The control unit 606 receives input 608 from the operator's console 610 (such as, but not limited to, the operator's console 200 shown in FIG. 2), including input from the first and second manual controls 612, 614. The control unit 606 is configured to receive other input from the operator console 610, such as, for example, input from the foot pedal(s), voice recognition input, gesture recognition input, eye recognition input, etc. Block 606 The control also receives input 616 from surgical robots 602, 604. This input includes sensor data from position sensors and torque sensors located at the joints of the robot's arms. The control unit 606 is configured to receive other input 616 from each robot, such as haptic feedback, data from surgical instruments, or about surgical instruments. In response to input received from robots 602, 604 and operator console 610, control unit 606 controls robots 602, 604. Control unit 606 includes one or more processors 618 and memory 620. Memory 620 permanently stores program code, which may be executed by one or more processors 618 to control actuators.

[0068] Although the surgical robot 502 shown in FIG. 6 contains two surgical robots, it will be apparent to those skilled in the art that the methods and techniques described herein are equally applicable to surgical robots containing only one surgical robot and to surgical robots containing more than two surgical robots. robots.

[0069] Reference is made below to FIG. 7, which depicts an exemplary implementation of a surgical robot 700. One or more of the surgical robots 602, 604 shown in FIG. 6 may be implemented as the surgical robot 700 shown in FIG. 7. The surgical robot 700 includes an arm 702 extending from a base 704 secured in place while performing a task (eg, a surgical procedure). In some cases, the base 704 may be mounted on a chassis. The chassis may be a trolley, such as a bedside trolley, for mounting the robot at bed height. Alternatively, the chassis may be a ceiling-mounted device or a bed-mounted device.

[0070] The arm 702 extends from the base 704 of the robot to the attachment point 706 of the surgical instrument 708. The arm is flexible. It is articulated through multiple flexible joints 710 along its length. Between the joints are rigid elements 712 of the hand. The hand shown in Fig. 7 contains seven joints. Articulations include one or more pivot joints (having an axis of rotation along the longitudinal direction of the arm elements on either side of the hinge), one or more horizontal axis joints (having an axis of rotation transverse to the longitudinal direction of the previous arm element), one or more vertical axis joints (also having an axis of rotation transverse to the longitudinal direction of the previous arm element, as well as transverse to the axis of rotation of the combined joint by a horizontal axis). However, the arm may be articulated differently. For example, a hand may contain more or less articulations. The arm may comprise articulations allowing movement other than rotation between respective articulation parts, such as a telescoping articulation. The robot includes a set of actuators 714, with each actuator 714 driving one or more of the joints 710.

[0071] The attachment site 706 allows the surgical instrument 708 to be releasably attached to the distal end of the arm. The surgical instrument 708 includes a linear rigid axis and a working tip at the distal end of the axis. The working tip contains a working body for participation in a medical procedure. The surgical instrument may be configured to extend linearly parallel to the axis of rotation of the end articulation of the arm. For example, the surgical instrument may pass along an axis coinciding with the axis of rotation of the end articulation of the arm. The surgical instrument 708 may be, for example, a cutting, grasping, cauterizing, or imaging device (eg, an endoscope).

[0072] The robot arm includes a number of sensors 716, 718. These sensors include, for each joint, a position sensor 716 for detecting the position of the joint, and a torque sensor 718 for detecting the torque applied to the axis of rotation of the joint. One or both of the position and torque sensors for a joint may be built into the motor of that joint. The output of the sensors is passed to the control unit 606 where it forms the input to one or more processors 618.

[0073] Reference is made below to FIG. 8, which depicts an exemplary method 800 for automatically augmenting endoscopic video captured during a task (e.g., surgery) that is at least partially performed by the surgical robot 502 based on state data describing the state of the surgical robot during execution. tasks. In particular, in the method 800, events occurring during the execution of a task are detected based on the state data, and the endoscopic video image is supplemented with information (e.g., one or more labels) identifying the detected events and the time at which these events occurred relative to the endoscopic video image.

[0074] Method 800 begins at block 802, in which augmentation device 500 (eg, augmentation agent 506) receives endoscopic video captured by the endoscope during a task (eg, surgical operation) at least partially performed by surgical robotic instrument 502. As described above, in some cases, endoscopic video may be captured by an endoscope attached to and controlled by one of the robot arms of the surgical robot 502. In other cases, endoscopic video may be captured by a manually operated endoscope. As described above, the endoscopic video image may be received by the adder 500 (eg, adder 506) by any suitable means. For example, the endoscopic video image can be directly delivered to the endoscope add-on device 500 (e.g., add-on device 506) using a wired or wireless communication connection (e.g., Ethernet connection, Wi-Fi® connection, Bluetooth® connection, NFC connection, etc. .). In other cases, the endoscopic video image may be stored in a storage device or other device and delivered to the augmentation device 500 via another device.

[0075] At block 804, augmentation device 500 (eg, event detector 504) receives state data describing the state of surgical robot 502 during a task (eg, surgery). The state data may include any information describing the surgical robot 502 during a task (e.g., a surgical operation) or describing the environment in which the surgical robot 502 operated. As described above, the state data is time synchronized with the endoscopic video image. so that events detected based on the status data can be correlated to a point or time interval in the endoscopic video image.

[0076] All or part of the state data may be generated by the surgical robot 502 itself (for example, the surgical robot is configured to generate data relating to the position and/or movement of the surgical robot's arm(s) and/or instruments attached thereto) . In some cases, some of the state data may be generated by one or more external sources. The state data may comprise one or more data streams, each of which is separately time-synchronized with the endoscopic video image. The term "data stream" is used herein to mean a set of data that has been generated by a single source and describes a single element of the surgical robot 502. For example, as described above, each joint of the surgical robot 502 may include a position sensor 716 for detecting the position of the joint and a torque sensor 718 for detecting a torque about an axis of rotation of the joint. The data stream can be considered the data generated by each position sensor 716 . Similarly, data generated by each torque sensor 718 can be considered a data stream.

[0077] The state data may comprise information or data describing the current state of the surgical robotic arm(s), such as, but not limited to, position and/or torque information indicating the position and/or movement of the arm(s). ) of the robot or its joints. In particular, the status data may comprise position and/or torque data generated by the position and/or torque sensors 716, 718. For example, each position sensor 716 may report the mating angle of the joint it senses and/or each torque sensor may report the torque acting on the joint it senses. Those skilled in the art will appreciate that these sensors are only exemplary, and that the surgical robot(s) may contain other sensors that provide information regarding the condition of the robot's arm(s).

[0078] The state data may also or alternatively comprise data or information describing the state of the tools attached to the robot arm(s). For example, the state data may include information identifying whether a tool is attached to the robot's arm(s) and, if the instrument is attached, the type of instrument attached to the robot's arm(s). In particular, the surgical robotic tool 502 may include means for detecting the attachment of an instrument to the hand (for example, the surgical robotic tool may include means for entering into contact with the instrument, such as those described in Applicant's GB 2552855 A, which is incorporated herein). document by reference), and means for automatically detecting the type of instrument (for example, each instrument may contain an RFID or other component configured to automatically provide the surgical robot with information about its identifier when attached to the robot arm), and this information can be delivered to the detector 504 events as state data. The state data may also or alternatively comprise information such as, but not limited to, position and/or torque information indicative of the position and/or movement of the tool(s) attached to the robot arm(s). Those skilled in the art will appreciate that this information is merely an example, and that the surgical robot or the instruments themselves may contain other sensors that provide information about the status of the instruments.

[0079] If at least one of the instruments attached to the robot arm(s) is an energized instrument, such as, but not limited to, an electrocautery instrument, an electrosurgical instrument energized by an electric current to perform a surgical function (e.g., cauterized, cut, etc.), the condition data may contain information about the state of live tools, such as, but not limited to, whether the tool is currently live, and if it is , about the shape of the electric current curve used to apply voltage to the tool.

[0080] If at least one of the instruments attached to the robot arm(s) is an endoscope, the state data may include information indicative of the position, movement, state, and/or field of view of the endoscope.

[0081] The state data may also or alternatively comprise data or information describing the state of operator input devices (eg, manual controls 612, 614) used to control the operation of the robot's arms and tools. As described above, the operator input devices may, for example, be manually actuated mechanical input devices such as manual controls or control levers. In these cases, the status data may include data such as, but not limited to, position and/or torque data indicating the position and/or movement of input devices (eg, manual controls 612, 614). For example, one or more input devices may include a position sensor configured to detect the position of the input device and/or a torque sensor configured to detect torque or force applied to the input device. In some cases, the manual control can be mounted on pantographic mechanisms using gimbals. In these cases, these mechanisms and/or gimbals may contain position sensors and/or torque sensors that measure the positions and torque of the joint and/or gimbals. The results of these measurements may then be delivered to the augmentation device 500 (eg, event detector 504) as status data.

[0082] In some cases where the input devices are hand controls, each hand control may include one or more hand contact sensors, etc., that are configured to detect whether the user's hand is in contact with the hand control or gripping the manual control, which indicates whether the manual control is being used. In these cases, the hand contact detection information generated by the hand contact sensor(s) may be delivered to the augmentation device 500 (eg, event detector 504) as status data. In some cases where the input devices are manual controls, each manual control may also or alternatively comprise one or more grip sensors configured to detect the grip force applied to the manual control (i.e., how hard the operator grabs the hand control). In these cases, the grip force information generated by the grip force sensor(s) may be delivered to the augmentation device 500 (eg, event detector 504) as status data. In some cases, the hand contact sensor(s) and the grip sensors may be a single sensor or otherwise combined. In some cases, the operator may be equipped with motion sensors on his/her hands that detect the movement of the operator's hands. In these cases, the hand motion data generated by the hand motion sensor may be delivered to the augmentation device 500 (eg, event detector 504) as state data.

[0083] The state data may also or alternatively contain information about the mode of operation of the surgical robot 502. For example, in some instances, the surgical robot 502 may be configured to operate in one of a variety of modes, such as, but not limited to, active mode. in which an operator (eg, a surgeon) controls one or more arms of the robot to perform a task (eg, a surgical operation); a selection mode in which an operator (eg, a surgeon) selects or turns on the robot arm(s) to be controlled from the operator console 610; and a tool change mode in which at least one of the tools attached to the robot arm is removed, attached and/or changed. There may also be several types of active mode. For example, in some cases, there may be an endoscope active mode in which an operator controls an endoscope attached to a robot arm and an instrument active mode in which an operator controls one or more instruments attached to one or more robot arms. In these cases, the state data may include information about the current mode of operation of the surgical robot 502. Those skilled in the art will appreciate that these modes are only exemplary modes of operation of the surgical robot, and that there may be additional or other modes of operation of the surgical robot. 502.

[0084] If the task being performed is a surgical procedure performed on a patient, the condition data may also or alternatively contain data related to the patient. For example, the state data may contain information about the patient's health or data describing the state of the patient's health during a task. For example, health data may contain information that describes a patient's vital signs (eg, body temperature, pulse rate, respiration rate) or other health indicators (eg, blood pressure, oxygen saturation, blood glucose/sugar) during a task ( such as surgery). Typically, during surgery, vital signs and other health indicators of the patient are measured by special medical measuring equipment such as a heart rate monitor, pulse oximeter, continuous glucose meter (CGM), etc. In these cases, the medical measuring equipment can be configured to sending vital sign information or other patient health information to the augmentation device 500 (eg, event detector 504), for example, via a communication network (eg, a wired or wireless communication network).

[0085] However, in other cases, information about the basic indicators of the state of the body and/or other indicators of the health of the patient can be obtained in another way. For example, in some cases, the operating room in which a surgical operation is performed may be equipped with a video and/or audio recording device, and vital signs and/or other medical data of the patient may be obtained from the video and/or audio recording. For example, if the medical measuring equipment generates an audible representation of the measurement (eg, a set of audible tones or intermittent audio signals), an evaluation of the measurement can be obtained by analyzing the audio recorded by the video and/or audio recorder. In another example, if the medical measuring equipment visually displays the current or previous measurement result, the measurement result may be obtained by analyzing the video image recorded by the video recording device (for example, the measurement result may be read from the medical measuring equipment). In other instances, one or more vital signs of the patient's body condition may be obtained from the endoscopic video image. For example, a patient's pulse rate or a patient's blood oxygen saturation level can be identified from an endoscopic video image using video image analysis techniques. In particular, the oxygen saturation level of a patient's blood can be determined based on the color of the patient's blood. Illumination of the corresponding part of the patient's body can help determine the vital signs of the state of the body or other indicators of the health of the patient from the endoscopic video image.

[0086] The condition data may also or alternatively contain other information related to the patient, such as, but not limited to, demographic information such as age and weight, and other medical information such as previous images (e.g., x-rays, MRI), previous findings, medical history, diagnosis history, and/or course plan.

[0087] In some cases, the state data may also contain information about the health of the surgeon or data describing the health of the operator (eg, surgeon) operating the surgical robotic tool. For example, the state data may comprise information or data describing one or more of the body's basic health signals (eg, body temperature, pulse rate, respiration rate) and/or other health indicators (eg, blood pressure) of an operator (eg, surgeon) in the progress of a task (for example, a surgical operation). For example, in some cases where the input devices are manual controls, these manual controls may include sensors that measure the heart rate of the operator (e.g., the surgeon), the trembling of the operator (e.g., the surgeon), and/or the sweat rate of the operator (e.g., the operator). , surgeon) and providing information or data related to the measured heart rate, trembling and/or sweat rate to the event detector 504 as status data. In other instances, there may be external medical devices and the like capable of measuring vital signs or other health indicators of an operator (eg, surgeon) in the course of performing a task. As described above with respect to vital signs and other indicators of patient health, medical devices that measure vital signs or other indicators of the health of an operator (e.g., a surgeon) can be configured to deliver measured data to an event detector 504, e.g. , via a communication network, or an assessment of the measured values can be obtained by other means (for example, by analyzing the video image and / or sound of the operating room).

[0088] As described above, in some cases, the room (eg, operating room) in which the task is performed may be equipped with sound and/or video recording equipment to record the sound and/or video of the room during the task. In these cases, the state data may also or alternatively comprise video and audio data captured by the audio and/or video recording equipment.

[0089] In some cases, the state data may also or alternatively contain information about the task being performed. For example, the state data may contain information about a task (e.g., a surgical procedure) at least partially performed by the surgical robotic tool 502. have been completed and the timing of their completion. For example, an operator (e.g., a surgeon) of the surgical robotic tool 502 may be able to indicate, during the execution of a task, the step or phase of the task he/she is currently performing, and this information may be delivered to the augmentation device 500 (e.g., event detector 504). ) as state data. For example, as shown in FIG. 9, an operator console 610 may comprise a display or be associated with a display 902 that displays an endoscopic video image 904 as well as providing the operator with a list of steps in a task 906. The listed steps may be selected automatically based on the task, or manually entered by the operator (e.g., surgeon). For example, an operator (eg, a surgeon) may be able to manually enter steps or phases of a task that the operator expects to complete. An operator (e.g., surgeon) may be able to select or otherwise indicate which of the listed steps or phases are currently being performed using input devices (e.g., manual controls 612, 614), such as by pressing a button on a manual control. or by other means (for example, verbally or through a display if the display is a touch screen). Any selection made by the operator (eg, surgeon) may then be delivered to the augmentation device 500 (eg, event detector 504) as status data.

[0090] In some cases, the status data may also or alternatively contain information about users of the surgical robotic tool 502 (e.g., the operator (e.g., surgeon), operating the surgical robot(s), and/or other members of the work group (e.g., surgical team). )). For example, in some cases, before starting a task, the surgical robot 502 and/or the event detector 504 may be provided with information indicating which users will perform the task. In other cases, users (e.g., a surgical team) performing a task may be provided with a device (e.g., RFID device, mobile phone, tablet, etc.) capable of being automatically transferred to the surgical robot 502 and/or the detector 504 events of information identifying users when they are near the surgical robotic tool. An example of such a tool is described in the same British Patent Applicant's co-pending application entitled "DEVICE INTEROPERATION", filed concurrently with the current application and incorporated herein by reference.

[0091] In some cases, the operator (eg, surgeon) operating the surgical robot or another member of the work team (eg, surgical team) may be able to manually indicate that an event has occurred. In particular, in some cases, the operator (eg, surgeon) may be presented with a list of predefined events that may occur during the execution of a task, and the operator (eg, surgeon) may be able to indicate that one of the predefined events has occurred by selecting one from predefined events. Any selection made by the operator (eg, surgeon) may then be delivered to the augmentation device 500 (eg, event detector 504) as part of the state data. For example, as shown in FIG. 9, an operator console 610 may comprise a display or may be associated with a display 902 on which the endoscopic video image 904 is presented to an operator (eg, a surgeon). The display 902 is also configured to display one or more predefined events 908 that may occur during the course of a task, and the operator (eg, surgeon) may indicate that one of the predefined events has occurred by selecting one of the listed events. Predefined events can be selected automatically based on event type, specific operator, and/or other criteria; or an operator or other user may be able to manually enter a set of events that may occur during the execution of a task. In some cases, augmentation device 500 or other computationally based device may be configured to automatically generate based on state data for previously performed tasks, such as high probability or important events, and these high probability or important events are presented to the operator. (eg surgeon) as events that can be manually entered by the operator (eg surgeon) or another member of the workgroup.

[0092] An operator (e.g., a surgeon) may be able to select one of predefined events using input devices (e.g., manual controls), for example, by pressing a button on one of the manual controls or by other means (e.g., verbally or by touching the display 902 if the display 902 is a touch screen). In some cases, another member of the work group (for example, a surgical team) may have a device, such as, but not limited to, a computer or tablet, on which he is presented with a list of predefined events, and he may indicate that one of the predefined events has occurred, by selecting one of the predefined events. Any result of the selection may then be delivered to adder 500 (eg, event detector 504) as state data. An event that may be manually identified by the operator or another member of the work team may still be automatically detected by the augmentation device based on other status data.

[0093] With reference to FIG. 8, at block 806, augmentation device 500 (eg, event detector 504) analyzes the state data to identify events that occurred during the execution of a task (eg, a surgical operation). The term "event" is used in this document to mean the occurrence of an identifiable occurrence during the execution of a task. Events may include, but are not limited to, tool events; collision events; task events; patient events; operator events (for example, a surgeon); efficiency events and/or failure events. Examples of each of these types of events are described below.

[0094] Augmenter 500 (eg, event detector 504) can be configured to identify an event based on the state data by identifying a pattern in the state data. One or more events can be identified by identifying a pattern in one state data stream or by identifying patterns in multiple state data streams. The padding device 500 is typically pre-configured containing a pattern(s) identifying the occurrence of an event and what specific event that pattern identifies. For example, the pattern(s) may be stored in the storage device Addition Device 500. Events and/or specific pattern(s) indicating the occurrence of an event may be manually determined or may be learned based on state data from previously executed tasks. For example, in some cases, augmentation device 500 or other computationally based device may be configured to parse state data from previously executed tasks that have been manually augmented with event information in order to identify common patterns in the state data prior to the identified events.

[0095] In some embodiments, augmentation device 500 (eg, event detector 504) may also or alternatively be configured to be discoverable based on tool event state data. Instrument events are those events that are associated with instruments used in (eg, attached to) the surgical robot 502 during the execution of a task. The task of a surgical robot (eg, surgery) often requires many different surgical instruments. Multiple tools may be used simultaneously (for example, there may be multiple robot arms to which different instruments are attached and controlled), and/or multiple instruments may be used sequentially (for example, a first tool may be attached to a robot arm to perform a particular function, the first tool can then be removed and a second tool attached to that robot arm to perform another function). An exemplary set of tools that may be used in the course of performing a task by a surgical robot may include, but are not limited to, scissors, clamps of various shapes and geometries, single- and double-pole electrosurgical instruments, other power-driven instruments (e.g., laser or ultrasonic instruments), stapler and ligator. Tool events may include, but are not limited to, tool change; if at least one of the instruments attached to the robot arm during the task (eg, surgery) is an energized instrument, changing the state of the energized instrument; if one of the instruments is an endoscope, cleaning the endoscope; if one of the instruments is an endoscope, performing a white balance adjustment on the imaging means; if one of the instruments is an endoscope, the amount or frequency of movement is outside a certain range; and changing the active tool.

[0096] As described above, the state data may comprise information describing the state of the tools attached to the robot arm(s). For example, the state data may include information identifying the attachment of a tool to each arm(s), the type of tool attached to each robot arm, and, if the tool is a live tool, the state of the live tool (e.g., whether power is on the tool, and optionally the waveform of the electrical current applied to the energized tool). In these cases, augmentation device 500 (eg, event detector 504) may be configured to detect a tool change event in the event of detecting a tool being detached from the robot's hand or a tool being attached to the hand. In these cases, augmentation device 500 (e.g., event detector 504) may be configured to change the energized state of the tool upon detection, based on the state data, of a change in the energized state of the tool from energized to no voltage or vice versa, or upon detection of a shape change. curve of electric current applied to an energized tool.

[0097] As described above, the status data may include information about the status of the endoscope, such as whether the endoscope is in a cleaning state, or whether an operator has performed a white balance adjustment on an imaging device. In these cases, add-on device 500 (eg, event detector 504) may be configured to determine the occurrence of an instrument event if add-on device 500 (eg, event detector 504) detects, based on the endoscope state information, that the endoscope is in a cleaning state, or that the imaging tool is being adjusted for white balance. If the status data does not contain information about the state of the endoscope, the augmentation device 500 (e.g., the event detector 504) can be configured to automatically detect based on the status data that the endoscope is in a cleaning state or performing white balance adjustment with respect to the imaging engine. For example, add-on device 500 (eg, event detector 504) may be configured to detect that the endoscope is in a cleaning state or perform white balance adjustment on the imaging means if add-on device 500 (eg, event detector 504): identifies a specific force pattern in in relation to the endoscope; identifies the position of the endoscope tip outside the patient's body; identifies, based on the video image of the operating room, that the endoscope has been cleaned or white balance adjustment has been performed; and/or identifies a combination of the above.

[0098] As described above, the status data may contain information about the position and/or movement of the endoscope. If the endoscope is attached to the robot arm and is controlled by an operator at the operator console, information about the position and/or movement of the endoscope can be generated, for example, using position and/or torque sensors on the robot arm. In these cases, augmentation device 500 (e.g., event detector 504) may be configured to detect the occurrence of an instrument event upon detection, based on position and/or motion information, of an amount (i.e., amplitude) or frequency of movement of the endoscope outside a certain range. .

[0099] As described above, in some cases, the surgical robotic tool 502 may include a plurality of robot arms, each of which may have an instrument attached to it, and an operator (e.g., a surgeon) may be able to selectively control one or more robot arms and attached to them with instruments from the operator's console (for example, a surgeon). With an operator console, the operator may be able to select which robot arms, and thus which tools, will be under the operator's control by linking input devices (eg, hand controls) to specific tools. Instruments that are currently under control from the operator's console may be referred to as "active" instruments. In these cases, the state data may contain information indicating which tools were active at which points in time during the execution of a task (this information may be referred to as "active tool information"). In these cases, adder 500 (eg, event detector 504) may be configured to detect the occurrence of a tool event if adder 500 (eg, event detector 503) detects a change in one or more active tools based on the active tool information. By detecting changes in active instruments and supplementing the endoscopic video image with this information, a recording of active instruments during a task can be included in the endoscopic video image.

[00100] In some cases, augmentation device 500 (eg, event detector 504) may additionally or alternatively be configured to detect collision events. Collision events may include, but are not limited to, collision between the arms of the surgical robot 502, collision between instruments attached to different arms of the surgical robot 502, and collision between the arm of the surgical robot 502 and an instrument attached to the other arm of the surgical robot. robotic tool 502.

[00101] As described above, in some cases, the state data may comprise data or information indicative of the position and/or movement of the robot arm(s) and/or tools attached to it. In these cases, augmentation device 500 (e.g., event detector 504), for example, can be configured to detect the occurrence of an collision event (i.e., collision of a robot arm or tool with another robot arm or tool) based on data or information indicative of the position and/or movement of the robot's arms and/or tools. For example, augmentation device 500 (e.g., event detector 504) may be configured to determine the occurrence of a collision between hands, between tools, or between a hand and a tool, by, for example, (i) determining or evaluating the position of the hands and/or tools based on position and/or movement data and determining that the positions of one or more instruments and/or one or more hands are the same or substantially the same; or (ii) determining, based on position and/or motion information (e.g. based on torque data), that equal and opposite forces are detected on two of the arms and/or tools, as described in Applicant's UK Patent Application No. 1810754.0 which is incorporated herein by reference.

[00102] In some cases, augmentation device 500 (eg, event detector 504) may additionally or alternatively be configured to detect task events. Task events may include, for example, a change in the stage of a task in progress. For example, a task (eg, a surgical operation) may be associated with a predetermined sequence of steps, all or part of which may be performed during the execution of the task, and add-on device 500 (eg, event detector 504) may be configured to detect the occurrence of an event if it detects, based on the state data, a change in the stage of a task in progress. The automatic identification of task steps and the addition of this information to the endoscopic video may allow automatic segmentation of the endoscopic video based on task steps. This may allow the user to easily identify portions of the endoscopic video image related to specific task steps.

[00103] As described above, in some cases, the state data may contain information explicitly identifying which task steps were performed and at what time (which may be referred to herein as "step identifying information"). For example, as described above, an operator (eg, a surgeon) or another user (eg, another member of the operating team) may be able to manually indicate during the course of a task which steps of the task are performed. In these cases, adder 500 (eg, event detector 504) can be configured to identify a task step change based on information identifying the steps. If information explicitly identifying task steps is not received as part of the state data, adder 500 (eg, event detector 504) can be configured to dynamically detect task steps based on other state data. For example, augmentation device 500 (eg, event detector 504) may be configured to detect task steps based on patterns found in state data that match patterns in state data for previously executed tasks. In particular, augmentation device 500 (e.g., event detector 504) or other computationally based device may be configured to parse state data associated with previously executed tasks of the same type as the current task (e.g., state data associated with with the same surgical procedure or the same type of surgical procedure); identifying a common set of steps in a given task based on the analysis; and generating a map linking the identified stages. In some cases, the analysis may include at least an analysis of the endoscopic video image.

[00104] In other cases, augmentation device 500 (e.g., event detector 504) or other computationally based device may be provided with status data for one or more previously completed tasks that have been manually divided into segments associated with task steps, and device 500 an add-on (eg, event detector 504) or other computational device is configured to identify features or features of the state data associated with each step. For example, each step may differ in, for example, one or more of the following: tool type(s), robot arm movements, and/or manual control movements. For example, augmentation device 500 (eg, event detector 504) or other computationally based device may be configured to receive state data for a previously performed task in which the state data segment corresponding to the suturing step is identified as such. Then, an augmentation device 500 (e.g., an event detector 504) or other computationally based device is configured to parse a portion of the state data identified as corresponding to the suturing step and identify features such as attaching a needle holder to two of the robot's arms; performance by the operator of repetitive curvilinear movements with a small amplitude using the manual controls (and, thus, the working bodies); and high forces applied to the working body and clamping devices of the working body. Features and/or features of a step can then be used to identify that step based on status data in subsequent tasks. In some cases, augmentation device 500 (e.g., event detector 504) or other computationally based device may be configured to generate, from a single set of state data (i.e., state data relating to a single previously completed task), features or features that sufficiently describe the task step so that it can be identified in another set of state data (i.e., in state data related to another task). In other cases, augmentation device 500 (e.g., event detector 504) or other computationally based device may need multiple sets of data to generate features or features that sufficiently describe a task step so that it can be identified in another set of state data. about the state.

[00105] An exemplary set of robotic surgical procedure steps that may be automatically detected by augmentation device 500 (e.g., event detector 504) is shown in Table 1. A robotic surgical procedure may include all or only a portion (i.e., a subset) of these steps.

Table 1

Task steps Trocar insertion Tool introduction Abduction of organs/tissue to the sides tissue dissection Arrival at the target body Isolation of the blood supply Cross section of an organ Anastomosis/organ closure Sample extraction Hemostasis Closing/zooming in target area Tool removal Removal of the trocar/trocar insertion site wound closure

[00106] In many cases, each task includes a predetermined set of steps that can be performed during task execution, and there are only a small number of combinations of these steps that can occur. In these cases, after augmentation device 500 (eg, event detector 504) detects a particular step, there is typically a very limited number of steps that can occur after that step. This can ease the load on the adder 500 (eg, the event detector 504) when task steps are detected. It may also allow augmentation device 500 (eg, event detector 504) to detect deviations from the expected set of steps (eg, when a step that is not on the list of expected steps is executed, or when a step that is on the list of expected steps is executed out of order). As described below, if augmentation device 500 (eg, event detector 504) detects that there has been a deviation from the expected set of steps, then augmentation device 500 (eg, event detector 504) may be configured to detect an abnormal event.

[00107] In some cases, augmentation device 500 (eg, event detector 504) may additionally or alternatively be configured to detect patient events. Patient events may include one or more vital signs or one or more other indicators of the patient's health outside a certain range; and/or a change in one or more of the basic indicators of the state of the body and/or other indicators of the health of the patient (for example, by more than a certain amount).

[00108] As described above, in some instances, the health data may comprise information about the patient's health or data indicative of the status of one or more vital signs and/or one or more other health indicators of the patient during a task. In these cases, augmentation device 500 (e.g., event detector 504) can be configured to detect a patient event if the patient's health information or data indicates that one or more of the patient's health indicators (e.g., vital signs) are outside a certain range. This range may vary for different health indicators (for example, there may be different ranges for respiratory rate and heart rate).

[00109] In some cases, the ranges may be predefined. In other cases, ranges can be learned from status data for previously completed tasks. For example, add-on device 500 (e.g., event detector 504) or other computationally-based device is configured to identify, based on state data, for previously completed tasks of the level of various health indicators immediately prior to the occurrence of a negative result and generate ranges based on the identified levels (e.g., a range can be chosen covering the lowest and highest levels identified). These ranges may be defined for all tasks, or different ranges may be defined for different tasks (ie, different surgical procedures). For example, based on status data for previously performed tasks, it can be determined that a heart rate in the range X to Y is normal for Type A surgery, but not normal for Type B surgery.

[00110] In some cases, augmentation device 500 (eg, event detector 504) may additionally or alternatively be configured to detect operator (eg, surgeon) events. Operator (eg, surgeon) events may include, but are not limited to, one or more of: vital signs and/or other health indicators of the operator (eg, surgeon) falling outside a certain range; changes (eg, by more than a certain amount) in one or more vital signs and/or other health indicators of the operator (eg, surgeon); changing (eg, by more than a certain amount) the grip force applied by the operator (eg, surgeon) to one or more of the input devices (eg, manual controls); and/or changing (eg, by more than a certain amount) the pushing force applied by the operator (eg, surgeon) to one or more of the input devices (eg, manual controls). For example, augmentation device 500 (eg, an event detector) may be configured to detect the occurrence of an operator event if the operator's blood pressure is outside a certain range; or if the operator's heart rate is outside a certain range and the operator's applied grip force exceeds a certain threshold value/value.

[00111] As described above, in some cases, the health data may include information about the health of the operator (e.g., surgeon) or data indicating the status of one or more key health indicators of the operator (e.g., surgeon) and/or one or more other indicators. the health of the operator (e.g. surgeon) during a task. In these cases, augmentation device 500 (e.g., event detector 504) may be configured to detect an operator event if the operator health information or data indicates that one or more of the operator's health indicators (e.g., vital signs) are outside a certain range, or change by more than a certain amount. This range and/or the defined value may vary for different health indicators (eg, there may be different ranges for respiratory rate and heart rate), operators and/or tasks.

[00112] In some cases, ranges and/or certain values may be learned based on status data for previously completed tasks. For example, augmentation device 500 (e.g., event detector 504) or other computational device is configured to identify, based on state data, previously performed task levels of various operator health indicators just before the occurrence of an adverse event (e.g., an adverse event associated with an operator or by the patient) and generating ranges based on the levels identified (eg, a range can be selected spanning the lowest and highest levels identified). These ranges can be defined for all operators, or different ranges can be defined for different operators. For example, based on status data for previously completed tasks, it can be determined that a heart rate ranging from X to Y is normal for one operator but not normal for another operator.

[00113] As described above, in some instances, the status data may include hand control grip information that describes the grip force applied by an operator to one or more hand controls. In these cases, augmentation device 500 (eg, event detector 504) may be configured to detect an operator event if augmentation device 500 (eg, event detector 504) detects a change in grip force (or a change by more than a certain amount) based on the grip information. ).

[00114] As described above, in some cases, the manual control may be mounted on pantograph mechanisms using gimbals, and the status data may contain position and/or torque data indicating the position and/or torque of the joints of the connection and/ or gimbals. In these cases, augmentation device 500 (e.g., event detector 504) may be configured to determine the pushing force applied by an operator to one or more input devices (e.g., manual controls) based on connection position and/or torque information, and /or gimbals and detecting the occurrence of an operator event when the addition device 500 detects a change in pushing force (or a change by more than a certain amount).

[00115] In some cases, augmentation device 500 (eg, event detector 504) may additionally or alternatively be configured to detect performance events. Performance events are events related to an operator's (e.g., surgeon's) performance in performing a task (or part of a task) and may include, but are not limited to, an operator's (e.g., surgeon's) performance falling below an acceptable level, and/or changing (or changing more than a certain amount) of the efficiency of the operator (e.g. surgeon).

[00116] In some cases, augmentation device 500 (e.g., event detector 504) can be configured to detect a drop in operator (e.g., surgeon) performance below a threshold by comparing status data for a given task with status data for previously completed tasks ( for example, tasks of the same type). For example, add-on device 500 may have access to a data store that stores data for previously performed tasks and in which the performance of an operator (e.g., surgeon) is assessed or identified (e.g., performance can be rated on some scoring scale, without limitation, from 1 to 10); and the state data can be compared with the state data stored in the store to determine the level of efficiency of an operator (eg, a surgeon) for a task.

[00117] In some cases, augmentation device 500 (e.g., event detector 504) can be configured to determine an operator (e.g., surgeon) performance level for a task by comparing one or more operator (e.g., surgeon) performance metrics with operators performance metrics ( e.g. surgeons) for previously performed tasks. For example, add-on device 500 (eg, event detector 504) can be configured to determine an operator's performance level (eg, determine a level in points from 1 to 10) at multiple points in time during a task based on performance metrics, and add-on device 500 ( eg, event detector 504) is configured to detect a performance event if the performance level changes or if the performance level is out of range. In some cases, performance measures may include one or more of the following: the path that a robot arm or tool travels in performing a task or task step; the smoothness of the movement of the robot's arm and/or the tool when performing a task or a stage of a task; the time required to complete a task or task step; and/or the amplitude or frequency of movement of the robot's arm.

[00118] As described above, in some cases, the state data may contain information describing the position and/or movement of the robot arm(s) and/or instrument(s) of the surgical robotic tool 502. In these cases, the add-on device 500 (e.g., detector 504 events) may be configured to determine the trajectory of the robot hand and/or tool based on information describing the position and/or movement of the hand(s) and/or tool(s). Once the motion trajectory is identified, augmentation device 500 (eg, an event detector) can be configured to evaluate user performance by comparing the motion trajectory to the "best" or "optimal" trajectory for that task or task step. The "best" trajectory can be determined based on the status data for previously completed tasks. For example, augmentation device 500 or other computationally based device may be configured to determine the "best" trajectory based on state data for previously completed tasks in which the trajectory has been estimated. In addition to all the sources of status data described above, status data for previously completed tasks can also contain result data. As known to those skilled in the art, outcome data is information about the results (eg, success/failure) of a surgical operation. In some cases, the closer the trajectory is to the "best" or "optimal" trajectory, the better the efficiency, and the farther the trajectory is from the "best" or "optimal" trajectory, the worse the efficiency.

[00119] The padding device 500 (eg, event detector 504) can also be configured to measure the smoothness of the trajectory based on whether the trajectory contains smooth curves or jagged edges, and the padding device 500 (eg, event detector 504) can be configured to the ability to evaluate the effectiveness of the operator, for example, based on the number of smooth curves and/or the number of jagged edges.

[00120] As described above, in some cases, the state data may contain information explicitly identifying a task and/or a task step(s) being performed. In other cases, adder 500 may be configured to dynamically identify a task and/or task step based on state data. In either case, augmentation device 500 (eg, event detector 504) can be configured to determine the time required to complete a task or task step by measuring time intervals between tasks or between task steps. Augmentation device 500 (e.g., event detector 504) can be configured to evaluate operator performance by comparing the time required to complete a task or task step with the time (generated based on status data) taken to complete the same task or the same task. task stage in the course of previously completed tasks.

[00121] As described above, in some cases, the state data may contain information describing the position and/or movement of input devices (eg, manual controls). In these cases, the adder 500 may be configured to determine, based on the position and/or movement of the hand controls, the amplitude (eg, absolute value) and frequency of movement of the hand controls. The augmentation device 500 may then be configured to evaluate the operator's performance based on the measured amplitude and frequency of movement. In particular, large and/or multiple motion changes may indicate unsatisfactory performance, while small, low frequency motion may indicate suitable performance.

[00122] For example, augmentation device 500 (e.g., event detector 504) may be configured to convert hand control position and/or movement data from the time domain to the frequency domain (e.g., using a Fourier transform) and identify patterns indicative of the occurrence of an event, in the frequency domain. For example, if augmentation device 500 detects many low frequency components in the frequency domain, this may indicate fatigue or intoxication; while many high frequency components in the frequency domain may indicate a high level of trembling or panic. In some cases, there may be valid ranges of low and high frequency components, and if any of these fall outside the respective allowable range, augmentation device 500 (eg, event detector 504) can be configured to determine the occurrence of a performance event.

[00123] In some cases, the add-on device may be configured to determine a score, or descriptor, of the performance of a single operator based on a plurality of performance criteria. In other cases, augmentation device 500 (eg, event detector 504) may be configured to determine multiple operator performance scores, each based on a different performance criterion. Augmenter 500 (eg, event detector 504) may be configured to determine the occurrence of a performance event when any of the performance criteria falls below an acceptable level or when any of the performance criteria changes (eg, by at least a certain amount).

[00124] In some cases, augmentation device 500 (eg, event detector 504) may additionally or alternatively be configured to detect abnormal events. Emergency events are events that indicate the occurrence of an emergency or a high probability of its occurrence during the execution of a task. Add-on device 500 (eg, event detector 504) can be configured to detect an emergency event if add-on device 500 detects that: one or more of the people present at the task (eg, one or more members of the surgical team) is speaking raised tones; one or more of the people present during the task (eg, one or more members of the surgical team) utter a warning word or phrase; there was a sudden movement of the arm of the surgical robot; and/or there was a deviation or deviation from the predetermined steps of the task.

[00125] As described above, in some cases, the status data may include video and/or audio recording from the room (eg, operating room) in which the task is performed. In these cases, add-on device 500 (eg, event detector 504) may be configured to detect an emergency event if add-on device 500 (eg, event detector 504) detects, based on video and/or audio recording, that a person in the room is speaking in a raised voice. tones. In particular, the fact that a person speaks in a raised voice may indicate the presence of friction between people in the room / operating room and / or the presence of a contentious issue / problem in the room / operating room. Augmenter 500 (eg, event detector 504) may be configured to detect elevated tones using any known speech analysis technique.

[00126] If the status data contains video and/or audio recording, add-on device 500 (eg, event detector 504) may also or alternatively be configured to detect an emergency event if add-on device 500 (eg, event detector 504) is based on video and/or audio recordings detect that a person in the room is speaking a warning word or phrase (i.e., a word or phrase that indicates a potential problem). Adder 500 (eg, event detector 504) may be configured to detect warning words and/or phrases in video and/or audio recording using any known speech recognition or analysis technique. A list of warning words and/or phrases may be pre-loaded into add-on device 500 (eg, event detector 504) before running the analysis. In some cases, a list of warning words and/or phrases may be learned based on status data for previously completed tasks. For example, add-on device 500 (e.g., event detector 504) or other computational device is configured to identify, based on state data, for each of the plurality of previously performed tasks, any words and phrases spoken in a certain amount of time before a negative result, and generate a list of identified words and phrases that are common to a number of previously completed tasks.

[00127] It will be apparent to those skilled in the art that this set is an exemplary set of events that adder 500 is capable of detecting (e.g., event detector 504), and that in other examples, adder (e.g., event detector 504) may be configured to detect additional events, only some subset of events, or other events. For example, other events that augmentation device 500 is capable of detecting (eg, event detector 504) include, but are not limited to, pressure drops across insufflation equipment.

[00128] In some cases, after identifying a set of events based on state data, augmentation device 500 (eg, event detector 504) may be configured to automatically identify additional events based on event patterns. For example, some event patterns may indicate the occurrence of another event. Event patterns indicating the occurrence of a certain event can be learned (for example, based on status data for previously completed tasks, where previously completed tasks may be the same tasks as the current task, a set of tasks similar to the current task, or a set of tasks , different from the current task).

[00129] In some cases, augmentation device 500 (eg, event detector 504) may be configured to detect a predefined set of events based on state data. The predetermined set of events that adder 500 can detect based on the state data (eg, event detector 504) can be based on the types of state data received and/or one or more other criteria. In some cases, adder 500 may be configured to determine only a subset of a predefined set of events that can be detected. For example, the user may be able to configure the augmentation device 500 (eg, an event detector) to detect only a subset of a predefined set of events. This can be useful, for example, when the user is looking for something specific in the endoscopic video image. For example, there may be a user interface (eg, a graphical user interface (GUI)) associated with add-on device 500 (eg, event detector 504) that allows the add-on device 500 to be configured by the user. The user interface, for example, presents the user with a predefined set of events supported by add-on device 500, and the user can select which of the supported events the add-on device (eg, event detector 504) should detect.

[00130] Alternatively or additionally, the user has the ability to manually select the events that augmentation device 500 should detect (e.g., event detector 504), wherein augmentation device 500 (e.g., event detector 504) can be configured to automatically select events that should be discovered based on the user's registration on the add-on device 500. For example, in some cases, in order to generate an augmented endoscopic video image based on state data, a user may need to register or authenticate themselves with the augmentation device 500, and the events that may be detected by the augmentation device 500 (e.g., event detector 504) are based, for example, possible. For example, an operator performing a task may be able to access more events than another operator (eg, a surgeon). For example, an operator performing a task may be able to see performance events, while another operator (eg, a surgeon) may not be able to see performance events for that particular task. As described in more detail below, instead of configuring adder 500 (e.g., event detector 504) to detect only some subset of a predefined set of events based on state data, adder 500 can be configured to detect the entire predefined set of events, and events The , available to the user when viewing the augmented video, may be configurable and/or based on user privileges.

[00131] In some cases, for each event detected based on the status data, the padding device 500 (eg, event detector 504) is configured to generate event information used to pad the endoscopic video image. The event information may include, but is not limited to, an event identifier identifying the detected event, an event start time identifying when the event started, and optionally an event duration or event end time indicating when the event ended (in cases where the event has a duration) . The event ID can be a name or a number identifying the detected event. In some cases, there may be a predefined set of events that can be detected and which are listed, for example, in a lookup table with a number and a textual description. The start time and/or end time can be an absolute time (eg, 15:45) or a relative time (eg, 35 minutes after the start of the task).

[00132] After augmentation device 500 (eg, event detector 504) analyzes the state data to identify events that occurred during the execution of a task, method 800 continues at block 808 where the endoscopic video is augmented. It should be noted that while the method 800 shown in FIG. 8, the event detector analyzes the status data to identify all events that occurred during the execution of the task, and then the method 800 proceeds to the endoscopic video augmentation block 808, in other exemplary methods, the endoscopic video may be augmented based on individual events. In other words, in other cases, the endoscopic video image may be augmented immediately after an event is detected instead of waiting for all events to be detected.

[00133] At block 808, adder 500 (e.g., adder 506) generates an augmented endoscopic video image by padding the endoscopic video image received at block 802 with information for each event detected at block 806 that identifies the detected event and the time at which it the event occurred relative to the endoscopic video image. As described above, in some cases, the information added to the endoscopic video image may take the form of a label placed at the appropriate time in the endoscopic video image (eg, at the start of an event or during the duration of an event). The information or label may be visible to the user when the augmented endoscopic video is subsequently played back or viewed, for example with a media player. In some cases, the label may take the form of a bookmark, allowing the user to easily jump to a specific point in the endoscopic video when playing the augmented endoscopic video, for example, using a media player. For example, when a user loads an augmented endoscopic video into the media player, the user may be presented with a list of bookmarks in the video representing detected events, and the user may navigate directly to one of the points identified by the bookmark by selecting the bookmark. In some cases, information (eg, a label) related to a particular event may be included in the metadata that forms part of or is associated with the endoscopic video image.

[00134] For example, Table 2 provides a list of events that may be detected by padding device 500 (e.g., event detector 504) at block 806. Those of skill in the art will appreciate that these events represent exemplary events, start times, end times, and duration, and that other events with different start times, end times, and durations can be detected.

Table 2

Event type Event ID Event start End of event Task event Seam step Minute 25 Minute 45 Tool event Tool change - attaching the needle holder to the arm 0 Minute 25 Minute 30 Instrument event Instrument Change - Attaching the Needle Holder to Arm 1 Minute 30 Minute 35 Efficiency event Efficiency Change - Level 7 Efficiency Minute 65 ---

[00135] FIG. 10 shows an example of augmented endoscopic video 1000 that has been augmented based on the events shown in Table 2 and detected by augmentation device 500 (eg, an event detector) at block 806. FIG. 10, in this example, in order to generate the augmented endoscopic video image 1000, the augmentation device 500 (e.g., augmentation means 506) adds one label 1002, 1004, 1006, 1008 for each of the identified events to the original endoscopic video image 1010. In particular, the apparatus 500 add-ons (eg, add-on 506) adds a first mark 1002 indicating the onset of the suturing step between minute 25 and minute 45; a second mark 1004 indicating the attachment of the needle holder to the robot arm 0 between minute 25 and minute 30; a third mark 1006 indicating the attachment of the needle holder to the arm of the robot 1 between minute 30 and minute 35; and a fourth label 1008 indicating the change in operator efficiency to level 7 at minute 65. Those skilled in the art will appreciate that such padding is only an example of how endoscopic video can be padded, and that in other examples, endoscopic video can be padded. otherwise.

[00136] Once the augmented endoscopic video image is generated, it may be stored in the storage device of the augmentation device 500 or in the storage device of another device for later use. For example, the stored augmented endoscopic video may be subsequently viewed or played back using, for example, a media player to critically analyze operator performance, to educate operators on a procedure, etc. The information or labels added to the augmented endoscopic video may be configured to view by the user when viewing or playing the augmented endoscopic video image. In particular, in some cases, information or labels may be presented as tabs. For example, as shown in FIG. 11 as the user views the exemplary endoscopic video image 1000 shown in FIG. 10, for example, using the media player 1102, a list of bookmarks 1104 (or a menu of bookmarks) may be presented to the user along with the endoscopic video image 1106. For example, in FIG. 11 is a tab for each event found in Table 2: a tab for the suturing step, a tab for each tool change, and a tab for changing the efficiency level. The user can then automatically navigate to a specific part of the endoscopic video image 1106 by selecting one of the tabs 1104. In some cases, as shown in FIG. 11, bookmarks 1104 may be presented to the user in groups based on type or time, for example. However, those skilled in the art will appreciate that such a presentation is merely an example, and that bookmarks may be presented to the user in other ways.

[00137] In some cases, in addition to the information or label added to the endoscopic video for each detected event being presented as a bookmark to the user, information added to the endoscopic video for detected events may be presented to the user as the endoscopic video is played back. For example, the augmented endoscopic video may be augmented such that when the video is played back, event information is displayed at the top, bottom, or side of the screen with a banner, i.e., it displays information about detected events at appropriate or significant times in the endoscopic video.

[00138] In some cases, information or labels made viewable when viewing or playing augmented endoscopic video using, for example, a media player, can be customized. For example, a user of the augmented endoscopic video may be presented with a list of events that have been noted in the augmented endoscopic video and the user may be able to select which events to view. The user may be able to select individual events or types of events. For example, the user may be able to select all tool events and/or all task events; or some tool events and/or some task events. In some cases, the information or labels made viewable when viewing or playing augmented endoscopic video may be based on the user viewing the augmented endoscopic video. For example, in some cases, viewing augmented endoscopic video may require a user to register or otherwise authenticate with an endoscopic video viewer. In these cases, the information or marks rendered viewable when viewing or playing the augmented endoscopic video image may be determined based on the privileges associated with the user playing the video image. For example, an operator performing a task may be able to view all of the information or labels, while other users may be able to view only a subset of the information or labels. For example, other users may not be able to view any information or labels associated with performance events.

[00139] In some cases, in addition to being able to automatically augment the endoscopic video image with information or labels identifying events that occurred during the execution of a task, the augmentation device 500 may also be configured to generate a three-dimensional (3D) reconstruction based on the state data in virtual reality (VR) of instruments and their positions and/or movements. The augmentation device 500 is then configured to synchronize and link the augmented endoscopic video to the 3D VR reconstruction of the instruments so that when the endoscopic video is viewed or played back, the 3D VR reconstruction can be viewed along with the augmented endoscopic video. For example, as described above, in some cases, the state data may include information describing the position and/or movements of the robot arm(s) and/or tools attached to it during the execution of a task, and information describing the tools attached to each robot arm. during the execution of the task. In these cases, the adder 500 may be configured to determine the position and movement of the tools based on the position/motion information of the robot/tool arm. Once the positions and movements of the instruments have been identified, these movements can be displayed in the 3D model of the respective instruments based on the type of instrument attached to each arm (identified by the instrument information in the status data). Once movements are displayed in a 3D model, these 3D models can be merged to form 3D VR reconstruction tools as the task progresses. The 3D VR reconstruction can then be time synchronized and linked to the augmented endoscopic video. In some cases, the perspective or view of the tools in the 3D VR reconstruction can be customized (i.e. the user may be able to set or adjust the viewpoint). This may allow the user to view the instrument from a different vantage point than the endoscope. In some cases, the viewpoint may be dynamically reconfigured during playback. In particular, the 3D VR rendering may allow the user to adjust the viewpoint during playback or viewing.

[00140] In some cases, in addition to being able to automatically supplement the endoscopic video image with information or labels identifying events that occurred during the execution of the task, and the moment or time interval of their occurrence relative to the endoscopic video image, the addition device 500 can also be configured to link augmented endoscopic video image with all or part of the status data in a manner that links the identified events not only to the corresponding part of the endoscopic video image, but also to the corresponding parts of all or part of the status data. For example, the augmented endoscopic video may be associated with all or part of a subset of the data streams forming the state data. For example, if the status data comprises a patient heart rate data stream and a robot arm position information data stream, the endoscopic video image may be associated with either one or both of the data streams. The associated augmented endoscopic video and status data may be referred to as augmented combined data.

[00141] As described above, the status data is synchronized in time with the endoscopic video image, for example, using common timestamps or a common timeline. Accordingly, augmented composite data can be generated by associating the augmented endoscopic video image with all or part of the state data such that the respective portions of the endoscopic video image and the state data are synchronized or linked. For example, in FIG. 12 depicts an exemplary augmented endoscopic video image 1000 shown in FIG. 10 associated with the respective status data 1202 such that the respective portions of the endoscopic video image 1000 and the status data 1202 are synchronized or linked. In this example, the status data 1202 contains two data streams, a patient heart rate data stream 1206 and a surgical robot position information data stream, and both of these data streams are associated with the augmented endoscopic video image 1000. This means that, for example, the data heart rate and position information of the surgical robot at 30 minutes (from the start of the task) are linked to the endoscopic video image at 30 minutes (from the start of the task). However, those skilled in the art will appreciate that this binding is only an example, and that state data 1202 may comprise any combination of data streams described above. Associated augmented endoscopic video 1000 and status data 1202 form augmented combined data 1208.

[00142] After associating the endoscopic video image with the status data to form augmented combined data, the information and/or marks in the augmented endoscopic video may be used to identify portions of the endoscopic video and status data related to each of the identified events. Thus, the information or marks in the augmented endoscopic video image can be considered to identify time intervals of the combined data related to the identified events. For example, the label 1002 identifying the suturing step identifies portions of the endoscopic video image, the patient heart rate data stream, and the surgical robot position information data stream that correspond to the suturing step. The tags can then be used to extract data from the endoscopic video image and/or from one or more other status data streams related to the event. For example, there may be a means or software capable of receiving padded combined data, configured to present a list of identified events to a user based on information or labels in the padded combined data. Then, upon selecting one of the events, the user may be presented with portions of the endoscopic video image and status data related to that event. Portions of the endoscopic video and status data related to the event may be presented to the user in any suitable form. For example, when an event is selected from the list, a portion of the endoscopic video related to that event may be played along with displaying related status data (for example, the patient's heart rate during the corresponding portion may be displayed above, below, or to the side of the endoscopic video when it is played back) .

[00143] Although it has been described above that augmented composite data is generated by first generating augmented endoscopic video and then associating augmented endoscopic video, in other examples, the endoscopic video may be associated with all or part of the state data to generate combined data before padding so that to generate padded combined data, padding engine 506 pads the combined data with information or labels identifying the identified events and when they occurred.

[00144] Reference is made below to FIG. 13, which depicts various components of an exemplary computing device 1300 that may be implemented as some form of computing and/or electronic device, and in which embodiments of the augmentation methods and devices described herein may be implemented.

[00145] Compute-based device 1300 includes one or more processors 1302, which may be microprocessors, controllers, or any other type of processor, for processing computer-executable instructions and controlling operation of the device to verify the hardware design for the data link. In some examples, for example, if a device on a crystal architecture is used, the processors 1302 may contain one or more fixed functional blocks (also called accelerators) that implement part of the endoscopic video augmentation method, in the form of hardware (rather than software or firmware). ). The computing-based device may be provided with platform software comprising an operating system 1304, or any other suitable platform software that enables application software, such as software 1305, to execute the method shown in FIG. 8 on the device.

[00146] Computer-executable instructions may be provided using any computer-readable storage medium that can be accessed by device 1300 on a computational basis. The computer-readable storage medium may include, for example, a computer storage medium, such as a storage device 1306, and communication media. A computer storage medium (i.e., a persistent computer-readable storage medium), such as storage device 1306, includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules. or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other storage technologies, CD-ROMs, digital versatile disks (DVDs) or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other non-transferable medium capable of being used to store information for access by the device on a computational basis. In comparison, communication media may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism. As defined herein, a computer-readable storage medium does not contain communication media. Although a computer-readable storage medium (i.e., a non-volatile computer-readable storage medium, such as storage device 1306) is present in device 1300 on a computational basis, it should be understood that the storage device may be distributed or located remotely and may be accessible over a network or other communication channel. (for example, using the communication interface 1308).

[00147] The compute-based device 1300 also includes an I/O controller 1210 configured to output displayed information to a display device 1312, which may be separate from or combined with the compute-based device 1300. The displayed information may include a graphical user interface. The I/O controller 1310 is also configured to receive and process input from one or more devices, such as a user input device 1314 (eg, a mouse or keyboard). User input can be used to trigger verification. In one embodiment, display device 1312 may also act as user input device 1314 if it is a touch screen display device. The I/O controller 1310 is also configured to output data to devices other than a display device, such as a locally connected printing device (not shown in FIG. 13).

[00148] Applicant herein discloses individually each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations can be implemented based on the present description as a whole in light of the general knowledge those skilled in the art, whether or not these features or combinations of features solve any of the problems described herein. In view of the above description, it will be apparent to those skilled in the art that various modifications may be made within the scope of the present invention.

Claims (31)

1. An add-on device for generating an augmented endoscopic video image, the add-on device comprising: an event detector configured to: receiving status data describing the state of the surgical robot during a task at least partially performed by the surgical robot, the surgical robot comprising at least one surgical robot comprising a base and an arm extending from the base to the instrument attachment site, while the hand contains joints, with which you can change the configuration of the hand; wherein the state data comprises instrument information describing at least one instrument attached to at least one surgical robot during the execution of a task, and position information of the surgical robots describing the movement and position of at least one surgical robot during the execution tasks; identifying patterns in the state data indicating the occurrence of an event during the execution of a task, wherein said patterns in the state data comprise patterns in instrument information and patterns in position information of surgical robots; and identifying a suturing step during a task by identifying patterns in the instrument information and patterns in the position information of the surgical robots; and add-on tool configured to: receiving an endoscopic video image captured during a task, the endoscopic video image being synchronized in time with the status data; and in response to the identification by the event detector in the state data of a pattern indicating the occurrence of an event, automatically supplementing the metadata associated with the endoscopic video image with information identifying the event and the time of the event relative to the endoscopic video image. 2. The augmentation device of claim 1, wherein information identifying the event and the time of the event is visible to the user when the user views the augmented endoscopic video image after the task has been completed. 3. The add-on device of claim 1, wherein information identifying the event and the time of the event is selectively visible to the user when the user views the augmented endoscopic video image after completing the task. 4. The add-on device according to claim 1, wherein the information identifying the event and the time of the event is automatically visible to the user when the user views the augmented endoscopic video image after performing a task based on privileges associated with the user. 5. The augmentation device of claim 1, wherein augmenting the metadata with information identifying an event and an event time relative to the endoscopic video image includes adding a bookmark to the endoscopic video image at the time of the event that identifies the event. 6. Complement device according to claim 1, wherein each pattern corresponds to a different event and the event detector is configured to selectively identify only a portion of the patterns. 7. The add-on device according to claim 1, wherein each pattern corresponds to a different event, and the event detector is configured to identify a portion of the patterns based on user privileges associated with the user running the add-on. 8. The augmentation device of claim 1, wherein the state data at least partially comprises data generated by the surgical robotic tool. 9. The augmentation device of claim 8, wherein the state data at least partially comprises data generated by sources external to the surgical robotic tool. 10. The augmentation device of claim 1, wherein the patterns in the state data are configured to identify one or more tool events, wherein the one or more tool events comprise one or more of the following: changing at least one instrument attached to the arm of the surgical robot; - a change in the state of the energized instrument attached to the arm of the surgical robot; - cleaning the endoscope attached to the arm of the surgical robot; - performing a white balance adjustment with respect to the endoscope imaging means attached to the arm of the surgical robotic means; - the output of the magnitude or frequency of movement of the endoscope attached to the arm of the surgical robotic tool, out of range; and changing at least one instrument attached to the arm of the surgical robot actively controlled by the surgical robot. 11. The augmentation device of claim 1, wherein the patterns in the state data are configured to identify one or more collision events, wherein the one or more collision events include one or more of the following: collisions between at least two arms of a surgical robot of a surgical robotic means; collisions between at least two instruments attached to different arms of the surgical robot of the surgical robot; and collision between the surgical robot arm of the surgical robot and an instrument attached to the other arm of the surgical robot of the surgical robot. 12. The add-on device of claim 1, wherein the task is a surgical procedure performed on a patient and the patterns in the condition data are configured to identify one or more patient events, wherein the one or more patient events include one or more of the following: the output of one or more indicators of the state of the patient's body outside the range; and a change in one or more indicators of the state of the patient's body. 13. The augmentation device of claim 1, wherein the surgical robot is under the control of an operator and the patterns in the state data are configured to identify one or more operator events, wherein the one or more operator events include one or more of the following: output one or more indicators of the state of the operator's body outside the range; change in one or more indicators of the state of the operator's body; changing the grip force applied by the operator to one or more input devices used to control the surgical robotic tool; and changing the pushing force applied by the operator to one or more of the input devices. 14. The add-on device of claim 1, wherein the surgical robot is under the control of an operator and the patterns in the state data are configured to identify one or more operator performance events, wherein the one or more operator performance events include one or more of the following : drop in the efficiency of the operator when performing a task below an acceptable level; and changing the efficiency of the operator in performing the task. 15. The add-on device of claim 1, wherein the task includes steps and the patterns in the state data are configured to automatically identify task steps from the state data. 16. The add-on device according to claim 1, wherein the event detector is configured to identify the suturing step during the execution of the task, when the event detector identifies, based on the instrument information, that at least two needle holders are attached to the surgical robots, and the event detector identifies based on information about the positions of the surgical robots, that the surgical robots make repetitive circular motions. 17. The add-on device according to claim 1, wherein said add-on device is configured to link the augmented endoscopic video image and all or part of the state data such that the event identifying information identifies the time of occurrence of the event relative to the endoscopic video image and the state data. 18. The add-on device according to claim 17, wherein a portion of the endoscopic video image and a portion of the status data relating to the identified event are capable of being extracted from the information identifying the event.

Images