EP4713938A1

EP4713938A1 - Dynamic correlated surgical theater data playback and event discovery

Info

Publication number: EP4713938A1
Application number: EP24735393.1A
Authority: EP
Inventors: Xi Liu; Anthony M. JARC; Omid MOHARERI
Original assignee: Intuitive Surgical Operations Inc
Current assignee: Intuitive Surgical Operations Inc
Priority date: 2023-05-14
Filing date: 2024-05-10
Publication date: 2026-03-25
Also published as: CN121263846A; WO2024238351A1

Abstract

Various of the disclosed embodiments provide systems and methods for correlating and presenting surgical theater playback data so as to facilitate more insightful analysis and feedback. Specifically, various embodiments present kinematics and system event data collected during the surgery in parallel with wider theater context data as may be captured by, e.g., visual imaging systems or depth imaging systems within the surgical theater. In this manner, reviewers may more readily cross-reference the types of data during playback, distinguishing inefficiencies consistently affecting surgical outcomes from one-time anomalous events. Recognizing such distinctions may facilitate the reviewer's providing more actionable feedback to members of the surgical team. Metrics for assessing surgical performance using cross-referenced data are also disclosed.

Description

DYNAMIC CORRELATED SURGICAL THEATER DATA PLAYBACK AND EVENT DISCOVERY

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of, and priority to, United States Provisional Application No. 63/466,307, filed upon May 14, 2023, entitled “DYNAMIC CORRELATED SURGICAL THEATER DATA PLAYBACK AND EVENT DISCOVERY”, and which is incorporated by reference herein in its entirety for all purposes.

TECHNICAL FIELD

[0002] Various of the disclosed embodiments relate to systems and methods for reviewing data acquired during a surgical procedure.

BACKGROUND

[0003] Successful surgical teams seek to operate efficiently, achieving desired results within desired periods of time. Unfortunately, while the consequences of inefficient behavior may be readily manifest in undesired outcomes or prolonged operations, it can be difficult to recognize the nature of the inefficiency precipitating these consequences, as well as to identify the inefficiency’s cause or causes, such as poor habits, lack of adequate skill, poor team communication, inadequate preparation, failure to acclimate to new tools or techniques, etc. Being preoccupied with providing proper care during the procedure, surgical team members are themselves rarely in a position to recognize such inefficiencies in real-time, let alone to recognize their origins or appropriate remediations. While sensor data, such as video and instrument kinematics data, may instead be acquired in the theater and subsequently reviewed to identify such inefficiencies, the very copiousness of such data and the heterogeneous character of multi-modal sensors can itself make inefficiencies and their causes difficult to recognize. Further complicating the matter, some surgeries will include isolated adverse anomalies unrelated to any genuine inefficiency of the team, as when the team encounters one-time atypical variations in patient anatomy or unusual patient responses to an intervention. Naively construing such isolated deviations from historical patterns as corresponding to genuine inefficiencies would result in false positive inefficiency identification, which may itself precipitate improper feedback and potentially further inefficient behavior.

[0004] Naive detection of inefficiencies only from historical pattern outliers may also prove inadequate as some inefficiencies may manifest only, or primarily, via subtle correlations between dissimilar types of theater data. For example, an assisting team member’s choice of port placement for a surgical camera may agree poorly with a surgeon’s subsequent preferred method for operating the camera once inserted. Recognizing such an inefficiency may require more than comparing only a single data type to historical patterns. Rather, the reviewer may need to instead infer the inefficiency by considering both the location of the port (e.g., from video of the theater) and the surgeon’s subsequent behavior (e.g., from instrument kinematics data). Unless the reviewer can acquire a meaningful understanding from such disparate data streams, the nature and the cause of many inefficiencies may remain obscured. Similarly, only by recognizing correlative patterns between data streams may the reviewer be able to distinguish anomalous from genuinely inefficient behavior. Consequently, the reviewer may be unable to offer any remediating feedback, or may offer improper feedback, resulting in continuation of the inefficiencies, or the creation of new inefficiencies, respectively, each of which risks a cascade of further downstream adverse consequences.

[0005] Accordingly, there exists a need for systems and methods to overcome challenges and difficulties such as those described above. For example, there exists a need for systems to process disparate forms of surgical theater data and to present the processed data to a reviewer in an insightful manner facilitating the identification of inefficiencies and actionable remediating feedback.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various of the embodiments introduced herein may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements: [0007] FIG. 1A is a schematic view of various elements appearing in a surgical theater during a surgical operation, as may occur in relation to some embodiments;

[0008] FIG. 1 B is a schematic view of various elements appearing in a surgical theater during a surgical operation employing a surgical robot, as may occur in relation to some embodiments;

[0009] FIG. 2A is a schematic representation of a series of surgical procedures and surgical tasks during a surgical procedure, as may occur in some embodiments;

[0010] FIG. 2B is a table depicting example tasks with their corresponding start point and end points as may be used in conjunction with various disclosed embodiments;

[0011] FIG. 3A is a schematic block diagram illustrating relations between various metrics and data structures as may be used in some embodiments;

[0012] FIG. 3B is a schematic depiction of an example raw data input, specifically, a forceps translational movement in three-dimensional space, as may be used to generate one or more objective performance indicators (OPIs) in some embodiments;

[0013] FIG. 3C is a schematic depiction of an example raw data input, specifically, a plurality of rotations in three-dimensional space about a plurality of forceps component axes, as may be used to generate one or more OPIs in some embodiments;

[0014] FIG. 3D is a pair of tables illustrating example OPI to skill and skill to task mappings as may be applied in some embodiments;

[0015] FIG. 4A is a schematic block diagram illustrating a flow of information as may be used for surgical team and team member performance analysis in some embodiments;

[0016] FIG. 4B is a schematic plot of machine learning classifier results, as may be used for inferring, e.g., an OPI or skill score, in some embodiments;

[0017] FIG. 4C is a flow diagram illustrating various operations in an example process for determining a classifier value to score mapping, as may be implemented in some embodiments; [0018] FIG. 4D is a schematic diagram illustrating an example application of a reference population-based score determination, as may be performed in some embodiments;

[0019] FIG. 5 is a flow diagram illustrating various operations in an example high- level data correlation and management process, as may be implemented in some embodiments;

[0020] FIG. 6 is a schematic collection of elements in a surgical review Graphical User Interface (GUI), depicting a first configuration having a task selection element, as may be implemented in some embodiments;

[0021] FIG. 7 is a schematic collection of elements in a surgical review GUI, depicting a second configuration having a theater-wide context view element, as may be implemented in some embodiments;

[0022] FIG. 8 is a flow diagram illustrating various operations in an example process for adaptive data consolidation, as may be implemented in some embodiments;

[0023] FIG. 9A is a schematic depth map rendering from an example theater-wide sensor perspective, as may be used in some embodiments;

[0024] FIG. 9B is a schematic top-down view of objects in the theater of FIG. 9A, with corresponding sensor locations, as may be used in some embodiments;

[0025] FIG. 10A is a schematic timeline depicting example region-of-interest availabilities throughout a procedure, as may occur in some embodiments;

[0026] FIG. 10B is a collection of schematic perspective fields of view of an example region of interest, here, a surgical table, as may occur in some embodiments;

[0027] FIG. 10C is a flow diagram illustrating various operations in an example process for region-of-interest management, as may be implemented in some embodiments;

[0028] FIG. 11 A is a schematic theater visualization with example isolated elements, as may be implemented in some embodiments; [0029] FIG. 11 B is a supplemented rendering of the elements in the schematic theater visualization of FIG. 11 A from a first perspective, as may be implemented in some embodiments;

[0030] FIG. 11 C is a supplemented rendering of the elements in the schematic theater visualization of FIG. 11 A from a second perspective, as may be implemented in some embodiments;

[0031] FIG. 11 D is a flow diagram illustrating various operations in an example isolated virtual element rendering process, as may be implemented in some embodiments;

[0032] FIG. 12A is a schematic screenshot of elements in a first GUI display rendering during surgery, as may be used in some embodiments;

[0033] FIG. 12B is a schematic screenshot of elements in a second GUI display rendering during surgery, as may be used in some embodiments;

[0034] FIG. 12C is a schematic screenshot of elements in a third GUI display rendering during surgery, as may be used in some embodiments;

[0035] FIG. 13 is a flow diagram illustrating various operations in an example process for display GUI-highlighting management (e.g., one of the displays in FIGs. 12A- C), as may be implemented in some embodiments;

[0036] FIG. 14 is a flow diagram illustrating various operations in an example bookmark management and rendering process, as may be implemented in some embodiments;

[0037] FIG. 15 is a flow diagram illustrating various operations in an example scoring and visualization process, as may be implemented in some embodiments;

[0038] FIG. 16 is a flow diagram illustrating various operations in an example OPI value determination process from cross-referenced data, as may be implemented in some embodiments;

[0039] FIG. 17 is a schematic grouping of classes of OPIs, as may be used in some embodiments; [0040] FIG. 18 is a flow diagram illustrating various operations in an example process for distinguishing anomalies from genuine inefficiencies and, in some embodiments, their influencers, as may be implemented in some embodiments; and

[0041] FIG. 19 is a block diagram of an example computer system as may be used in conjunction with some of the embodiments.

[0042] The specific examples depicted in the drawings have been selected to facilitate understanding. Consequently, the disclosed embodiments should not be restricted to the specific details in the drawings or the corresponding disclosure. For example, the drawings may not be drawn to scale, the dimensions of some elements in the figures may have been adjusted to facilitate understanding, and the operations of the embodiments associated with the flow diagrams may encompass additional, alternative, or fewer operations than those depicted here. Thus, some components and/or operations may be separated into different blocks or combined into a single block in a manner other than as depicted. The embodiments are intended to cover all modifications, equivalents, and alternatives falling within the scope of the disclosed examples, rather than limit the embodiments to the particular examples described or depicted.

DETAILED DESCRIPTION

Example Surgical Theaters Overview

[0043] FIG. 1A is a schematic view of various elements appearing in a surgical theater 100a during a surgical operation as may occur in relation to some embodiments. Particularly, FIG. 1A depicts a non-robotic surgical theater 100a, wherein a patient-side surgeon 105a performs an operation upon a patient 120 with the assistance of one or more assisting members 105b, who may themselves be surgeons, physician’s assistants, nurses, technicians, etc. The surgeon 105a may perform the operation using a variety of tools, e.g., a visualization tool 110b such as a laparoscopic ultrasound, visual image acquiring endoscope, etc., and a mechanical instrument 110a such as scissors, retractors, a dissector, etc.

[0044] The visualization tool 110b provides the surgeon 105a with an interior view of the patient 120, e.g., by displaying visualization output from an imaging device mechanically and electrically coupled with the visualization tool 110b. The surgeon may view the visualization output, e.g., through an eyepiece coupled with visualization tool 110b or upon a display 125 configured to receive the visualization output. For example, where the visualization tool 110b is a visual image acquiring endoscope, the visualization output may be a color or grayscale image. Display 125 may allow assisting member 105b to monitor surgeon 105a’s progress during the surgery. The visualization output from visualization tool 110b may be recorded and stored for future review, e.g., using hardware or software on the visualization tool 110b itself, capturing the visualization output in parallel as it is provided to display 125, or capturing the output from display 125 once it appears on-screen, etc. While two-dimensional video capture with visualization tool 110b may be discussed extensively herein, as when visualization tool 110b is an endoscope, one will appreciate that, in some embodiments, visualization tool 110b may capture depth data instead of, or in addition to, two-dimensional image data (e.g., with a laser rangefinder, stereoscopy, etc.). Accordingly, one will appreciate that it may be possible to apply various of the two-dimensional operations discussed herein, mutatis mutandis, to such three-dimensional depth data when such data is available.

[0045] A single surgery may include the performance of several groups of actions, each group of actions forming a discrete unit referred to herein as a task. For example, locating a tumor may constitute a first task, excising the tumor a second task, and closing the surgery site a third task. Each task may include multiple actions, e.g., a tumor excision task may require several cutting actions and several cauterization actions. While some surgeries require that tasks assume a specific order (e.g. , excision occurs before closure), the order and presence of some tasks in some surgeries may be allowed to vary (e.g., the elimination of a precautionary task or a reordering of excision tasks where the order has no effect). Transitioning between tasks may require the surgeon 105a to remove tools from the patient, replace tools with different tools, or introduce new tools. Some tasks may require that the visualization tool 110b be removed and repositioned relative to its position in a previous task. While some assisting members 105b may assist with surgery-related tasks, such as administering anesthesia 115 to the patient 120, assisting members 105b may also assist with these task transitions, e.g., anticipating the need for a new tool 110c. [0046] Advances in technology have enabled procedures such as that depicted in FIG. 1A to also be performed with robotic systems, as well as the performance of procedures unable to be performed in non-robotic surgical theater 100a. Specifically, FIG. 1 B is a schematic view of various elements appearing in a surgical theater 100b during a surgical operation employing a surgical robot, such as a da Vinci™ surgical system, as may occur in relation to some embodiments. Here, patient side cart 130 having tools 140a, 140b, 140c, and 140d attached to each of a plurality of arms 135a, 135b, 135c, and 135d, respectively, may take the position of patient-side surgeon 105a. As before, one or more of tools 140a, 140b, 140c, and 140d may include a visualization tool (here visualization tool 140d), such as a visual image endoscope, laparoscopic ultrasound, etc. An operator 105c, who may be a surgeon, may view the output of visualization tool 140d through a display 160a upon a surgeon console 155. By manipulating a hand-held input mechanism 160b and pedals 160c, the operator 105c may remotely communicate with tools 140a-d on patient side cart 130 so as to perform the surgical procedure on patient 120. Indeed, the operator 105c may or may not be in the same physical location as patient side cart 130 and patient 120 since the communication between surgeon console 155 and patient side cart 130 may occur across a telecommunication network in some embodiments. An electronics/control console 145 may also include a display 150 depicting patient vitals and/or the output of visualization tool 140d

[0047] Similar to the task transitions of non-robotic surgical theater 100a, the surgical operation of theater 100b may require that tools 140a-d, including the visualization tool 140d, be removed or replaced for various tasks as well as new tools, e.g., new tool 165, introduced. As before, one or more assisting members 105d may now anticipate such changes, working with operator 105c to make any necessary adjustments as the surgery progresses.

[0048] Also similar to the non-robotic surgical theater 100a, the output from the visualization tool 140d may here be recorded, e.g., at patient side cart 130, surgeon console 155, from display 150, etc. While some tools 110a, 110b, 110c in non-robotic surgical theater 100a may record additional data, such as temperature, motion, conductivity, energy levels, etc., the presence of surgeon console 155 and patient side cart 130 in theater 100b may facilitate the recordation of considerably more data than is only output from the visualization tool 140d. For example, operator 105c’s manipulation of hand-held input mechanism 160b, activation of pedals 160c, eye movement with respect to display 160a, etc., may all be recorded. Similarly, patient side cart 130 may record tool activations (e.g., the application of radiative energy, closing of scissors, etc.), movement of instruments, etc., throughout the surgery. In some embodiments, the data may have been recorded using an in-theater recording device, which may capture and store sensor data locally or at a networked location (e.g., software, firmware, or hardware configured to record surgeon kinematics data, console kinematics data, instrument kinematics data, system events data, patient state data, etc., during the surgery).

[0049] Within each of theaters 100a, 100b, or in network communication with the theaters from an external location, may be computer systems 190a and 190b, respectively (in some embodiments, computer system 190b may be integrated with the robotic surgical system, rather than serving as a standalone workstation). As will be discussed in greater detail herein, the computer systems 190a and 190b may facilitate, e.g., data collection, data processing, etc.

[0050] Similarly, many of theaters 100a, 100b may include sensors placed around the theater, such as sensors 170a and 170c, respectively, configured to record the surgical procedure from the perspective of their respective fields of view 170b and 170d. Sensors 170a and 170c may be, e.g., visual image sensors (e.g., color or grayscale image sensors), depth-acquiring sensors (e.g., via stereoscopically acquired visual image pairs, via time-of-flight with a laser rangefinder, structural light, etc.), or a combination of visual image and depth-acquiring sensors. In some embodiments, sensors 170a and 170c may also include audio acquisition sensors or sensors specifically dedicated to audio acquisition may be placed around the theater. A plurality of such sensors may be placed within theaters 100a, 100b, possibly with overlapping fields of view and sensing range, to achieve a more holistic assessment of the surgery. For example, depthacquiring sensors may be strategically placed around the theater so that their resulting depth frames at each moment in the surgery may be consolidated into a single three- dimensional virtual element model of the surgical theater. Similarly, sensors may be strategically placed in the theater to focus upon regions of interest. For example, sensors may be attached to display 125, display 150, or patient side cart 130 with fields of view focusing upon the patient 120’s surgical site. Similarly, sensors may be placed upon console 155 to monitor the operator 105c.

[0051] For clarity, one will appreciate that reference herein to “virtual models”, “virtual elements”, “virtual element models”, or to “virtual representations” refers not only to elements which may be rendered in a virtual reality environment (e.g., wherein the elements are not rendered so as to appear within a real-world imaging device capture), but to elements which may be rendered in an augmented reality environment (e.g., wherein the virtual element is, e.g., scaled or synthetically occluded so as to appear to the viewer as if the element resided within a real-world imaging device capture). For example, one will appreciate that a model in a Wavefront™ OBJ 3D geometry file format may be rendered in both virtual and augmented reality environments. Similarly, one will appreciate that there may be many equivalent ways to render a virtual element or representation within a rendering pipeline. For example, where a virtual element or representation is to be depicted as an outline around a portion of a surface of a mesh, the virtual element or representation may be a change in the texture rendering upon that portion of the mesh, a vertex extrusion around that portion of the mesh, a texture rendering on a two-dimensional plane (e.g., a billboard surface) positioned in the three- dimensional rendering pipeline field of view so as to appear to the viewer as an outline upon the mesh, a second mesh of a differentiating texture placed upon the first mesh, etc. Additionally, one will appreciate that as virtual elements and representations appear in data structures (e.g., the Wavefront™ OBJ file format, JavaScript Object Notation (JSON) format, Extensible Markup Language (XML), etc.) a computer system may receive, analyze, or manipulate a representation or a virtual element without necessarily rendering the representation or the virtual element. Similarly, for clarity, a “pose” refers to the translational position and rotational orientation of a body. For example, in a three- dimensional space, one may represent a pose with six total degrees of freedom. One will readily appreciate that poses may be represented using a variety of data structures, e.g., with matrices, with quaternions, with vectors, with combinations thereof, etc. Thus, in some situations, when there is no rotation, a pose may comprise only a translational component. Conversely, when there is no translation, a pose may comprise only a rotational component.

Example Surgical Theater Task Workflow Context Overview

[0052] FIG. 2A schematically depicts a variety of tasks occurring during one of a plurality of surgeries scheduled within a surgical theater, e.g., one of theaters 100a or 100b. Specifically, the theater may be scheduled for surgeries 210a, 210b, and 210c, over time 205 (as well as intervening surgeries represented by ellipsis 210d). Each of surgeries 210a, 210b, and 210c may include a plurality of tasks. For example, surgical procedure 210b includes the tasks 215a, 215b, 215c, and 215d (additional intervening tasks may be present, as indicated by ellipsis 215e). Each of the tasks 215a-d may be associated with a corresponding set of visualization tool data frames 220a, 220b, 220c, 220d (ellipsis 220e indicating the possibility of additional intervening frame sets) and device datasets including operator kinematics data 225a, 225b, 225c, 225d, patient-side kinematics data 230a, 230b, 230c, 230d, and system events data 235a, 235b, 235c, 235d (again, ellipses 225e, 230e, and 235e indicating the possibility of additional intervening data sets).

[0053] Particularly, task 215a may, e.g., be the “insert first imaging device system” task involving the placement of the first port and insertion of an imaging device (e.g., visualization tool 140d) so that the surgeon may acquire an initial view of the surgical site within the patient interior. Image frames (e.g., red-green-blue (RGB) images) captured with one or more such imaging devices (e.g., RGB cameras) may be organized into sets associated with each task. The set 220a is here generated during task 215a, the set 220b is generated during task 215b, the set 220c is generated during task 215c, and the set 220d is generated during task 215d (ellipsis 220e reflecting the existence of intervening datasets). Images from additional imaging devices (e.g., additional imaging devices introduced through additional ports into the patient site) may also be collected during the tasks.

[0054] Operator-side kinematic data 225a, 225b, 225c, 225d from the surgeon console (e.g., console 155), such as the motion of the control inputs by the surgeon, the motion of the surgeon’s eyes, pedal motions by the surgeon, etc., may also be captured in connection with each of tasks 215a-d (with intervening sets represented by ellipsis 225e). Patient-side kinematic data 230a, 230b, 230c, 230d (again, intervening ellipsis 230e indicating the possibility of more data) may also include kinematics data at the patient-side, e.g., from sensors upon the robotics system patient side cart 130, such as arm motion, instrument motion, instrument activation, instrument arm swaps, etc. System event data, such as replacement of an instrument, activation of a tool (such as a cauterizer), activation of a system alarm, energy applications, button presses, etc., may also be collected in connection with the data sets 235a, 235b, 235c, 235d. While the system event datasets are shown here as vectors taking on binary values, one will appreciate variations, as when events are represented by a finite set of states, or take on more continuous values via interpolation (as when a parametric function may be fitted to individually sampled values of a temperature sensor), etc. Similarly, while waveforms are shown here for the kinematics datasets, one will appreciate variations, as when kinematics data take on a finite set of values, binary values (e.g., “moving” or “stationary”), discrete encoder readings of a continuous component position sampled at fixed intervals, etc.

[0055] In some situations, task data may include one or more of frame sets 220a-d, operator-side kinematics 225a-d, patient-side kinematics 230a-d, and system events 235a-d, rather than all four. Similarly, in theater 100a, one will appreciate that datasets analogous to the datasets 225a-d, 230a-d, 235a-d may instead be derived, mutatis mutandis, from the motion and operation of instruments 110a, 110b by patient-side surgeon 105a.

[0056] Finally, as discussed, the sensors 170a, 170c (which, again, may include a suite of sensors distributed about the theater) may likewise acquire datasets 240a, 240b, 240c, 240d (ellipsis 240e again reflecting the possibility of intervening datasets) in association with each respective task 215a-d. Though output from only a single sensor is shown here for clarity, each of the datasets 240a-d may include one or more collections of temporal data captures from the one or more sensors in the theater. Thus, for example, though only one sensor’s output is shown here, dataset 240a may be three separate visual image video feeds, or three separate depth-frame capture feeds, captured during the task 215a from three separate sensor systems distributed within the theater. Thus, where multiple theater sensors are available, each of the datasets 240a-d may include a plurality of perspectives, some of which may overlap, of the surgical theater. For clarity, “theater-wide” sensor data refers herein to data acquired from one or more sensors configured to monitor a specific region of the theater (the region encompassing all or a portion of the theater) exterior to the patient, to personnel, to equipment, or to any other objects in the theater, such that the sensor can perceive the presence within or passage through at least a portion of the region of the patient, personnel, equipment, or other objects, throughout the surgery. Sensors so configured to collect such “theater-wide” data are referred to herein as “theater-wide sensors.” For clarity, one will appreciate that the specific region need not be rigidly fixed throughout the procedure, as, e.g., some sensors may cyclically pan their field of view so as to augment the size of the specific region, even though this may result in temporal lacunae for portions of the region in the sensor’s data (lacunae which may be remedied by the coordinated panning or field of view of other nearby sensors). Similarly, in some cases, personnel or robotics systems may be able to relocate theater-wide sensors, changing the specific region, throughout the procedure, e.g., to better capture different tasks. Accordingly, sensors 170a and 170c are theater-wide sensors configured to produce theater-wide data 240a-d. Conversely, in this example, interior frame sets 220a-d, kinematics 225a-d, 230a-d, and system event 235a-d data are not theater-wide sensor data. “Visualization data” refers herein to visual image or depth image data captured from a sensor. Thus, visualization data may or may not be theater-wide data. For example, visualization data captured at sensors 170a and 170c is theater-wide data, whereas visualization data captured via visualization tool 140d would not be theater-wide data (for at least the reason that the data is not exterior to the patient).

[0057] The discrete sets of data associated with a task may be determined by the task’s start point and end point. Each start point and each end point may be itself determined, e.g., by either a tool action or a tool-effected change of state in the body. Thus, data acquired between these two events may be associated with the task. For example, start and end point actions for task 215b may occur at timestamps associated with locations 250a and 250b respectively. [0058] FIG. 2B is a table depicting example tasks with their corresponding start and end points as may be used in conjunction with various disclosed embodiments. Specifically, data associated with the task “Mobilize Colon” is the data acquired between the time when a tool first interacts with the colon or surrounding tissue and the time when a tool last interacts with the colon or surrounding tissue. Thus, any of frame sets 220a- d, operator-side kinematics 225a-d, patient-side kinematics 230a-d, system events 235a- d, and theater sensor data 240a-d with timestamps between this start and end point may be data associated with the task “Mobilize Colon”. Similarly, data associated the task “Endopelvic Fascia Dissection” is the data acquired between the time when a tool first interacts with the endopelvic fascia (EPF) and the timestamp of the last interaction with the EPF after the prostate is defatted and separated. Data associated with the task “Apical Dissection” corresponds to the data acquired between the time when a tool first interacts with tissue at the prostate and ends when the prostate has been freed from all attachments to the patient’s body. One will appreciate that task start and end times may be chosen to allow temporal overlap between tasks, or may be chosen to avoid such temporal overlaps. For example, in some embodiments, tasks may be “paused” as when a surgeon engaged in a first task transitions to a second task before completing the first task, completes the second task, then returns to and completes the first task. Accordingly, while start and end points may define task boundaries, one will appreciate that data may be annotated to reflect timestamps affiliated with more than one task.

[0059] Additional examples of tasks include a “2-Hand Suture”, which involves completing 4 horizontal interrupted sutures using a two-handed technique (i.e. , the start time is when the suturing needle first pierces tissue and the stop time is when the suturing needle exits tissue with only two-hand, e.g., no one-hand suturing actions, occurring inbetween). A “Uterine Horn” task includes dissecting a broad ligament from the left and right uterine horns, as well as amputation of the uterine body (one will appreciate that some tasks have more than one condition or event determining their start or end time, as here, when the task starts when the dissection tool contacts either the uterine horns or uterine body and ends when both the uterine horns and body are disconnected from the patient). A “1 -Hand Suture” task includes completing four vertical interrupted sutures using a one-handed technique (i.e., the start time is when the suturing needle first pierces tissue and the stop time is when the suturing needle exits tissue with only one-hand, e.g., no two-hand suturing actions occurring in-between). The task “Suspensory Ligaments” includes dissecting lateral leaflets of each suspensory ligament so as to expose ureter (i.e. , the start time is when dissection of the first leaflet begins and the stop time is when dissection of the last leaflet completes). The task “Running Suture” includes executing a running suture with four bites (i.e., the start time is when the suturing needle first pierces tissue and the stop time is when the needle exits tissue after completing all four bites). As a final example, the task “Rectal Artery/Vein” includes dissecting and ligating a superior rectal artery and vein (i.e., the start time is when dissection begins upon either the artery or the vein and the stop time is when the surgeon ceases contact with the ligature following ligation).

Objective Performance Indicators - Application Overview

[0060] FIG. 3A is a schematic block diagram illustrating relations between various metrics and data structures as may be used in some embodiments. Specifically, a type of surgical operation 305a (e.g., the type of surgery performed in surgical operation 210b) may consist of a plurality of tasks e.g., tasks 305b, 305c, and 305d. Each task may itself implicate a number skills. For example, task 305c may depend upon each of skills 305e, 305f, and 305g. In a similar manner, each skill may itself be assessed based upon one or more OPI metric values (though OPI values may be directly related to tasks, without intervening skills, in some embodiments). For example, a score for skill 305f may be determined from scores for OPI metrics 305h, 305i, and 305j (again, one will appreciate that a skill may instead depend upon only one OPI metric, two OPI metrics, or any other arbitrary number). Each OPI metric may be similarly derived from one or more raw data fields (e.g., system event data, kinematics data, visualization data, theater-wide data, etc.). For example, OPI metric 305i may depend upon raw data values 305k, 305I, and 305m. While metrics and scores may be determined using machine learning systems, as disclosed herein, one will appreciate that metric and score values may simply be determined as a function of their constituent data values (e.g., an OPI may be a mean value of raw data over a time interval). Surgeries may be divided into meaningful task divisions so as to assess the skills involved in each task, to determine OPIs and relate them to the various skills, and to define the OPIs from the available data.

[0061] As an example of raw data (specifically, kinematics data), FIG. 3B depicts a forceps 340’s translational movement 345a in three-dimensional space, as may be used to generate one or more OPIs in some embodiments. FIG. 3C is similarly another example raw data input, specifically, a plurality of rotations in three-dimensional space about a plurality of forceps component axes, as may be used to generate one or more OPIs in some embodiments. Forceps 340 may be able to rotate 345b, 345c, 345d various of its components about respective axes 350a, 350b, and 350c. The translations and rotations of FIGs. 3B and 3C may be captured in raw kinematics data over time, forming, e.g., one of raw data values 305k, 305I, and 305m. OPI metric 305i may be a “forceps tip movement speed” OPI and may represent the speed of the forceps tip based upon the raw values 305k, 305I, and 305m (e.g., the OPI may infer the tip speed from a Jacobian matrix derived from the raw data of FIGs. 3B and 3C). OPI metric 305i may then be, e.g., one of several OPI metrics used in a mathematical function, algorithm, or as part of a feature vector in a model (e.g., a machine learning model, as discussed herein) to produce a skill score for skill 305f (or, again, a task score for task 305c). In some embodiments, collections of skill scores may then be used to assess the surgeon’s, other surgical team member, or the team as a whole’s, performance of task 305c, and ultimately, by considering all the tasks, the performance for the entire surgery 305a overall. Again, for clarity, though raw values and skills associated with the surgeon are often discussed here for the reader’s comprehension, one will appreciate that raw data from the theater (e.g., from sensors 170a and 170c) may be used to assess OPIs, skill, and task scores for other members of the team, combinations of team members, or the team as a whole.

[0062] Thus, where one wishes to assess a surgeon’s or surgical team’s performance on one of tasks 305b, 305c, and 305d, some embodiments may score the task by considering the task’s constituent skill scores resulting from skill-based machine learning models or mathematical functions. Alternatively, in some embodiments, one may instead simply assign OPIs to tasks directly and then train task-based (rather than skill- based) machine models, or prepare corresponding mathematical functions using OP Is (e.g., have experts select OPIs for tasks rather than for skills).

[0063] For further clarity, FIG. 3D is a pair of tables 335a, 335b illustrating example OPI to skill and skill to task mappings as may be applied in some embodiments (e.g., as an initial mapping based upon expert intuition and experience). With a plurality of skills 355c, shaded cells of table 335b indicate corresponding OPIs 355b. Similarly, table 335a indicates via shaded cells how tasks 355a may correspond to skills 355c.

[0064] For clarity, in the example correspondence shown in FIG. 3D, e.g., all six of the shown tasks depend upon the “Camera Use” skill, however, only the “Uterine Horn” task depends upon the “2-Hand Arm Retraction” skill. From these tables, one can also make transitive inferences, for example, that the “Rate Camera Control” OPI is relevant to the “Uterine Horn” task (as “camera use” is common to both in each of the tables). Similarly, the “Dominant Arm Wrist Articulation” OPI relates to the “Suture” skill (as well as the “Dissection” and “1 -Hand Dissection” skills), which is in turn used in assessing the tasks “2-Hand Suture”, “1-Hand Suture”, and “Running Suture”. Thus, tables such as FIG. 3D may be used to select OPIs both for skill models or functions and for task models or functions. One will appreciate that more, or other, skills, tasks, and OPIs may apply than those shown in this example. Also note that a single skill may be applicable to multiple tasks and that a single OPI may be applicable to multiple skills or tasks.

Example Relative Performance Assessment Methodologies

[0065] FIG. 4A is a schematic block diagram illustrating a flow of information as may be used for surgical performance analysis in some embodiments. Specifically, “reference” data 405a may be data acquired from only real-world non-robotic surgery theaters, from only simulated non-robotic surgery theaters (e.g., where the “patient” is a synthetic dummy), from only real-world robotic surgery theaters, from only simulated robotic surgery theaters, or from a mixture of these theaters. Reference datasets 405a may include data for both “expert / experienced” (e.g. , operators with more than 100 hours of experience performing a skill or task) and “non-expert I novice” users (e.g., operators with less than 100 hours of experience performing a skill or task). “Subject” datasets 405b may also be acquired from only real-world non-robotic surgery theaters, from only simulated non-robotic surgery theaters (e.g., where the “patient” is again a synthetic dummy), from only real-world robotic surgery theaters, from only simulated robotic surgery theaters, or from a mixture of these theaters. Datasets 405b may thus reflect the data from surgeons or surgical teams the reviewer wishes to assess.

[0066] Processing systems 405c and 405d (which may be the same system) may process reference 405a and subject 405b data (e.g., recognizing distinct surgeries captured in the data stream, separating the surgeries recognized in the stream into distinct datasets, recognizing tasks within the surgeries, determining OPI values from raw data, determining metadata annotations for the provided datasets, etc.). In some embodiments, human annotators may assist, correct, or verify the results of processing systems 405c and 405d.

[0067] Processed referenced data 405a may then be used to train a machine learning model classifier (e.g., to assess a skill or task score from OPI values) as part of performance assessment system 405e to distinguish “expert” data values from “nonexpert” data values. Following such training, processed subject datasets 405b may likewise be provided to the classifier in performance assessment system 405e trained with “reference” data 405a to produce performance metrics 405f (e.g., skill scores, task scores, etc., as inferred from the probability the subject data is that of an expert or a nonexpert) for the subject datasets 405b.

[0068] FIG. 4B is a schematic plot of example machine learning classifier assessment of whether incoming data is that of an expert or a non-expert, as may be used for, e.g., inferring a surgery score, skill score, task score, or an OPI score (e.g., in lieu of a hardcoded function or algorithm). In some embodiments, direct output from the machine learning model may be used to provide feedback to a surgeon or surgical team. For example, knowing that the model considers the surgeon’s performance to be 80% likely to be that of an expert, may be a meaningful statement. However, the nature of some models, e.g., logistic regression classifiers, support vector machines (SVMs), etc., may not provide outputs which directly map to such meaningful feedback.

[0069] For example, as shown in the abstract feature space 410b of FIG. 4B, in a first situation, a group 410a of expert OPI values may be located a great distance 41 Oe in the feature space from a group 41 Oi of nonexpert OPI values. Similarly, in a second situation, a group 410d of expert OPI values may be located a much shorter distance 410g in the feature space from a group 41 Oh of nonexpert OPI values. In both situations, several machine learning models may determine a separator 41 Of. Where the model is an SVM, the distance from the hyperplane separator to a new feature 410c may be used as a proxy for the probability of the point being in a class (one will appreciate that many binary classifiers, such as some SVMs, may typically only output a predicted class without a percentage prediction). However, distance in feature space or in a latent space is not intuitively analogous to one’s performance as a surgeon, team member, team, etc. Similarly, where separator 410f is the separating plane of a logistic regression classifier, distance from the separator 41 Of may correspond to a value of the sigmoid function and the corresponding output probability. Intuitively, one would expect the new feature 410c (e.g., from the subject dataset 405b) to receive a different probability assignment in the first instance of groups (i.e., 410a and 410i) as compared to the second instance of groups (i.e., 410d and 410h). However, one can imagine situations where the nature of the sigmoid function precipitates similar or the same probability value for the feature 410c in each instance. Similarly, the sigmoid function may plateau and provide similar probabilities for new features a great distance from the separator. Accordingly, such a sigmoid mapping may not have an intuitive correspondence to a skill level. While examples have been given here for SVMs and logistic regression classifiers, one can imagine similar discontinuities between scores and prediction probabilities based upon a feature space or latent space for other models.

[0070] To compensate for any discontinuity between prediction probabilities from a model output and meaningful surgical scores, various embodiments contemplate a postprocessing step to map model probabilities to a score, e.g., surgeon skill levels. For example, FIG. 4C is a flow diagram illustrating various operations in an example process 415 for determining a classifier value to score mapping, as may be implemented in some embodiments. At block 415a, the computer system may review outputs from a skill model for a known population of surgeons, e.g., the training data used to train the model, which may be annotated to indicate expert and nonexpert values (though a different, randomized population may be used instead). At block 415b, the component may generate a mapping between the model outputs and the surgical score level (e.g., a skill score, or in some cases, an OP I value). At block 415c, the system may record the mapping, e.g., for future use when considering new subject data. For example, when future results are produced during inference, the system may index the model outputs, interpolating if necessary, based upon the mappings recorded at block 415c.

[0071] To clarify the process 415 of FIG. 4C, consider a hypothetical example population of reference operators 420b in FIG. 4D. Each of these operators 420b may have provided surgical performances captured in raw data, which were used to generate corresponding OPI values, passed through the skill (or task) model, which in turn produced a corresponding probability output 420a of being an expert. In this example, the mapping at block 415b may order the model outputs 420a into an order 420c of decreasing magnitude, and then map a linear scoring metric 420d to the values as shown (though a linear metric is used here to facilitate understanding, one will appreciate that, e.g., a bell curve or mapping corresponding to the proportion of experts and nonexperts in the reference population 420b may be used instead in some embodiments). Values produced during inference falling between the ranked outputs may generate a corresponding value from the scoring metric. For example, if a new surgeon 420e’s performance was applied to the same skill model, it may produce a probability 0.73 of being an expert. As this probability corresponds to a position 420f in the ranked order 420c, the final score for the skill may be output based on the corresponding position in the metric 420d, e.g., the average of 87.5 and 75 (81.25), the average scaled by the linear position between the model output boundary values (i.e., (0.73-0.5) I (0.75 - 0.5) = 0.92 and 0.92 * (87.5 - 75) + 75 = 86.5), etc.

[0072] The above-described approach may be especially effective where the probability distributions of the model outputs are well separated in the ranking data. Where there is a smaller variance in groups of probabilities in the data, some embodiments may employ other approaches. For instance, some embodiments may estimate kernel density to find local maxima for groupings of probabilities and associate those maxima with a rank or set of ranks (e.g., in a single maxima example, the majority of samples may score 50%). Embodiments may estimate the standard deviation of such distributions to determine when a sample has deviated far enough to constitute a significant change in rank. Absent such estimation, a very tiny change in the machine learning output may precipitate an undesirably wide change in the final score. By estimating the mixture of distributions, embodiments may associate clumps of rankings to one another, rendering scores more robust to variations in the model prediction as well as making more intuitive sense to a human interpreter.

Correlated Playback Overview

[0073] Various embodiments provide reviewers with a platform to review correlated surgical theater data, including, e.g., intraoperative video, operating room video, kinematics and system event data, etc. Performance indicators for self-discovery of intraoperative disruptions may be derived from this data (e.g., as skill scores as discussed herein, as OPI values, as task scores, etc.) to further inform the review, thereby improving surgical team efficiency and training. FIG. 5 is a flow diagram illustrating various operations in an example high-level data correlation and management process 505, as may be implemented in some embodiments. In this example, at block 505a, the computer system may receive data other than theater-wide data (e.g., kinematics data from the surgeon console, instrument kinematics data, system event data, computed tomography (CT) or positron emission tomography (PET) scans acquired of the patient before the surgery, billing International Classification of Diseases, Clinical Modification (ICD-10-CM) codes associated with tasks or task actions, etc.).

[0074] At block 505b, the system may receive the theater-wide sensor data. For example, the theater-wide sensor data may include video images captured at one or more sensors 170a and 170c in the theater during the surgery, depth frames acquired at one or more depth sensors 170a and 170c, etc.

[0075] At block 505c, the system may perform an initial scoring assessment, e.g., using the methods of FIG. 4A-C describe above. For example, OPI and score values may be calculated from the raw kinematics data, as well as for any relevant correlated data values (e.g., the theater-wide optical flow and “head out” system event associations). The computer system may generate skill scores, task scores, or OPIs, if they do not already exist, from the initially received non-theater-wide and theater-wide data sets (naturally, one will appreciate that these datasets, though shown separately here, may be received together in a single dataset).

[0076] At block 505d, the system may seek correlations between and within the data acquired at blocks 505a and 505b. For example, where timestamps to a common clock are available for the data of both blocks 505a and 505b the system may begin aligning playback and determining associations between the datasets. Similarly, timestamps shared between visualization and non-visualization data may be noted. The system may also seek correlations between the scores and metrics calculated at block 505c. Additionally, as discussed herein, some OPIs may specifically take as input both theaterwide and non-theater-wide raw data so as to help the system or the reviewer infer inefficiencies via correlations therebetween.

[0077] In some embodiments, at block 505e, the system may perform historical dataset cross-referencing. For example, the system may recognize historical correlations between head out events from the console and large optical flow variations in certain theater-wide sensor data. The system may then search for such historical correlations among the correlations determined at block 505d. Thus, historical surgeries of the same type as that presently considered may be referenced for similar occurrences. Bookmarks, as will be described in greater detail herein, may also be generated for the reviewer following identification of these correlations, so that they reviewer might quickly compare these historical datasets with the currently reviewed surgery, readily identifying the relevant portions of playback in each surgery.

[0078] At block 505f, the user may begin their review, e.g., initiating playback of the surgical data in the interfaces of FIGs. 6 and 7, discussed in greater detail herein. As the user performs the review at block 505g, the system may provide playback of the current surgery (via data from blocks 505a and 505b) at different locations, provide playback of historical surgeries (via historical datasets identified from block 505e), etc. During these playbacks, correlated representations of the visualization and non-visualization data, theater-wide and non-theater-wide data, etc., may be presented at block 505h. In addition to assessing individual data types during their review, at blocks 505i and 505j the user may also interact with the system so as to effect adjustments in how the system manages correlations between the data reviewed. Similarly, at blocks 505k and 5051, the user may adjust scoring metrics to consider the correlations identified in the various types of data, including new correlations created at block 505j. For example, the scores may include correlation values in their determination, may selectively consider raw data depending upon the corresponding state of the correlation, etc.

[0079] When the user’s review completes, at block 505m the system may perform post-processing operations. For example, newly identified correlations at block 505j resulting in supplemental scoring assessments at block 5051 may be recorded for future identification at block 505e in subsequent historical cross-referencing.

Example Correlated Graphical User Interface Elements

[0080] FIG. 6 is a schematic collection of elements in an example surgical review GUI 605 to effect, e.g., aspects of the review in the process 505, as may be implemented in some embodiments. In this example, the interface 605 is generally divided into two elements 610a and 615a. In some embodiments, metadata regarding the procedure may be presented, e.g., the type of procedure, when the data was captured, in which theater the procedure occurred, details regarding the instruments and equipment used, the identities of the theater personnel or patient, the duration of the procedure or dataset, etc. (e.g., here, the interface indicates that this is procedure number seventeen in the dataset and that the surgery lasted for one hour, twenty-three minutes, and thirty seconds). In the configuration of FIG. 6, element 610a may be populated with sub-elements depicting playback of non-theater-wide data, including intraoperative visualization data, non- theater-wide kinematics and system event data, etc. Here, the visualization playback element 610b may depict the field of view from an imaging device (e.g., a depth or visual image sensor), e.g., the same view as seen by the operator 105c within display 160a during the surgery. As shown here, playback control 620a may be rendered over, or otherwise in connection with, the playback element 610b. Here, the video is paused and selecting control 620a will resume playback.

[0081] Below the element 610b in this example are a plurality of non-theater-wide, non-visualization data timeline renderings (though one will appreciate, e.g., that some non-visualization or non-theater-wide OPI values may themselves be derived using visualization or theater-wide data, as when an OPI is generated from optical flow measurements). In this example, the entire length of each of the timeline elements corresponds to the full length of the playback (which will often be the full duration of the surgery, though, as discussed herein, in some embodiments, playback may extend to include time periods before or after the surgical procedure). The timeline element 610c indicates the durations of one or more clinical tasks (or, in some embodiments, more granularly, actions within the tasks) occurring during the surgery. For example, the user may select one or more tasks from the element 615d to populate the timeline 610c. Here, two tasks are selected (e.g., via selection of elements 615e and 615h), and the period 61 Od, represented here by a highlighted portion of the timeline, associated with one task was considerably longer than the period 61 Oe associated with the second task.

[0082] A system events timeline 61 Of may likewise show the times at which one or more system events of interest occurred during the playback. Here, e.g., “head out” system events are shown, wherein the operator 105c removed their head from the console 155 (analogous events may exist in the theater 100a, e.g., when the surgeon 105a looks away from display 125). For example, at the times in playback associated with markers 61 Oh and 610g the operator may have removed or inserted their head into the console. When the reviewer clicks or otherwise selects one of the markers (e.g., one of markers 610h and 610g), or a portion of any of the timelines, the playback (e.g., in the element 610b) may be advanced to the corresponding time.

[0083] As mentioned, while the preliminary correlation at block 505d may include an initial identification of correlations between theater-wide data and other data (e.g., head out system events following increased motion in the theater-wide data), the preliminary correlation may also include the identification of correlations within the theater-wide data and within the non-theater-wide data. Here, for example, the markers 61 Oh and 610g may correspond to the end times of the task periods 61 Od and 61 Oe, respectively. While such correlations may be self-evident via inspection, as in the depicted element layout, in some embodiments these correlated data values may be highlighted to call them to the attention of the reviewer. Here, for example, the markers 61 Oh and 610g may be rendered in a different color than, e.g., the marker 620b which is not correlated with the end of a task period. As typical ranges for certain system events, kinematic and OPI values may be associated with given types of tasks and surgeries (e g., as determined at block 505d), deviations may also be highlighted, such as the “head out” event associated with the marker 620b occurring during the period 61 Od of the task. Head out events in the middle of a task may not be ideal, and so their occurrence, particularly if repeated multiple times during a task as shown here, may be indicative of a problematic condition in the theater (which, e.g., the reviewer may then further investigate using the configuration of FIG. 7).

[0084] Timelines 61 Oi, 61 Oj, 610k may indicate the portions of the surgery during which a given instrument is inserted into the patient, installed upon a surgical robot, etc. Here, the timelines 61 Oi, 61 Oj, 610k indicate whether an instrument has been attached to a first, second, and third robotic arm respectively (different instruments may be indicated by different colors or other unique indicia). Here, as indicated by timeline 61 Oi, the same instrument was attached to the first arm shortly after the surgery began (after the initial period 6101) and remained attached for the duration of the surgery. In contrast, timeline 61 Oj indicates that an instrument was attached throughout the entire surgery. Timeline 610k indicates that two different instruments were variously attached to the arm throughout the surgery, particularly a first instrument at periods 61 On and 61 Or and a second instrument at times 61 Oo, 61 Op, and 61 Oq. As with the correlation of the head out events associated with markers 620b, 61 Oh, 610g with the steps / tasks timeline 610c, periods of instrument attachment or detachment upon various arms may also indicate useful correlations with other of the non-theater-wide data (and theater-wide data, as will be discussed with respect to FIG. 7). For example, the fact that timeline 61 Oi indicates that an instrument was not initially attached during period 6101 may explain the delay in the task associated with period 61 Od beginning (e.g., where the delay is identified based upon historical indications of typical start times for the task). For example, the period 6101 may indicate a period during which a team member was negligent in anticipating the upcoming use of the instrument, thereby resulting in the task’s start time being delayed. Consistent occurrence of such a delay across surgeries (or across instrument attachments for tasks within a single surgery) may reflect an inefficient pattern, rather than a one-time anomalous mistake. Reviewing the various team member’s behavior in the theater via the configuration of FIG. 7, as will be discussed, may help confirm this causal agency.

[0085] In addition to attachment and insertion information, in some embodiments, timelines may also indicate various characteristics of the instruments and state of surgery. For example, colors, textures, or supplemental annotations to periods in the timelines may indicate instrument configurations, such as tool activation. Again, such detailed timeline representations may facilitate the reviewer’s identification of correlations via inspection, though some embodiments may highlight the correlations, e.g., as determined at blocks 505d and 505j for clarity.

[0086] In timeline 670a, a plot 670b of a value of an OPI metric is shown relative to a baseline value. In some embodiments, many different OPIs may be plotted upon the same or different timelines, as bar charts, pie charts, etc., to readily facilitate the reviewer’s comparison. In some embodiments, these OPI values may be overlaid upon one or more of the timelines 610c, 610f, 610i, 610j, 610k to also readily facilitate comparison by the reviewer. Correlation values between various of the data may likewise be plotted in some embodiments. Similarly, thresholds and conditions on such correlations may be specified by the system or by the reviewer to select corresponding portions in time of the theater-wide and non-theater-wide data for further analysis, for scoring, etc.

[0087] In this example’s depicted configuration, while toggle switch 605a is in the off position, as shown in FIG. 6, the element 615a is populated with a collection of the tasks occurring during the surgery, which may appear in chronological order. One may select between tasks in the timeline by selecting “Clinical Tasks” icon 615b (as is presently the case) and types of “System Events” via icon 615c. Selecting “System Events” icon 615c, for example, may populate element 615d with types of system events appearing in the data, the selection of which may be used to populate regions in element 610a, e.g., the timeline 61 Of (which, per block 505j, may precipitate new correlation highlights). In the element 615d, the task steps are presented as shown via icon elements 615e, 615f, 615g, and 615h. Selection of any of icons 615e-h may, e.g., translate playback in element 610a to the corresponding location in the operation timeline, populate the regions of timeline 610c, etc. (one will appreciate that “clicking” via different buttons or different combinations of, e.g., mouse and keyboard, buttons may precipitate different functionality for a single icon). The user may “favorite” an aspect of the data associated with an icon for future consideration. For example, by selecting the “favorite” icon 625a the task associated with element 615e may be readily recalled, e.g., via a bookmarking procedure as described in greater detail herein. In some embodiments, the user may choose to represent a task by its constituent actions, and similarly bookmark the beginning, end, or intermediate time points of such actions for greater granularity.

[0088] In this example, selecting the toggle switch 605a will populate element 615a with theater-wide data. For example, in FIG. 7 the switch 605a has been activated, and now element 615a has been populated with theater-wide data (here, the field of view rendering 710) corresponding to the same time in playback as depicted in element 610b. The field of view 710 rendered in element 615a may be a visual image captured by a visual theater-wide image sensor (e.g., one of sensors 170a and 170c) within the surgical theater during the operation, a plurality of fields of view from multiple such theater-wide sensors, a depth frame captured from a depth-producing theater-wide sensor system deployed in the theater during the surgery, a consolidated depth frame virtual element (e.g., consolidated depth frames from theater-wide sensors 170a and 170c formed into one or more three-dimensional virtual element models), depth views textured with the visual image views, combinations thereof, etc. In some embodiments, the user may be able to adjust the pose of a depth-data derived virtual element or frame in view 710 using mouse or keyboard commands. When element 615a is presenting the view 710, it is referred to as a “theater-wide context” GUI view element herein. Selection of the playback icon control 620a (or other playback initiating action) may initiate simultaneous parallel playback in both element 610b and view 710 of element 615a for the same points in time (in some embodiments, the reviewer may be able to specify a temporal offset between the two playbacks, e.g., to more readily identify causal relations). Analogous to the favorite bookmarks, like the bookmark following selection of icon 625a, bookmarks may also be created for the current time in the playback and view of the theater-wide data via bookmark creation icon element 710a. Creation of a bookmark via icon element 710a may produce a corresponding icon, e.g., as icons 710b, 710c were produced. Selecting either of icons 710b, 710c may begin playback (e.g., in element 610b) from the corresponding point in the timeline, as well as populate element 615a with the bookmarked theater-wide views corresponding to the same point in time (which may, again, be from a single visual image camera, multiple camera views at that point in time, visual image views and depth views at that point in time, a resultant virtual element model from merging the data, etc.). Similarly, in some embodiments, bookmarks may be correlated between the data types. For example, selecting icon 625a may also create a bookmark, corresponding to one of icons 710b, 710c. Selecting this bookmark may begin playback at the clinical task (or a more specific task action) corresponding to element 615e and populate element 615a with a default, or preferred, field of view from visual images, depth cameras, etc. (e.g., using the methods disclosed herein, e.g., with respect to FIG. 10C) if no preferred view was specified by the user. Again, for clarity, when the user clicks other playback associated data representations, such as one of head out event markers 620b, 61 Oh, 610g, the system may begin synchronized playback of non-theater- wide and theater-wide data in elements 610a, 615a and their sub-elements, respectively.

[0089] In some embodiments, the configuration of FIG. 7 may also present additional OPI and other data representations than appeared in the configuration of FIG. 6, e.g., here the timeline 690a is additionally presented indicating the time of occurrence of a system event via marker 690c (e.g., a “head out” event) as well as supplemented data highlighting 690b (e.g., a region of the timeline highlighted by the system or by the user as corresponding to an inefficiency). For clarity, while the timelines in these examples correspond to the duration of the surgical playback, one will appreciate that, in some embodiments, the timelines and data may be extended to include activities taking place in the operating room during phases other than the surgery episode (e.g., data, such as theater-wide data, for analyzing the surgery 210b may be extended to include setup before the procedure began, as well as the period after the surgery completed during which post-surgical cleanup occurred). Indeed, some datasets may simply include all the data collected for an entire day within the theater, e.g., for all of surgeries’ 210a, 210b, 210c and the intervening periods therebetween. Such comprehensive datasets may be useful for analysis as activities, including inefficiencies and anomalies, related to one procedure may impact activities related to a downstream procedure. For example, failure to properly reset a theater following a surgery may itself be an inefficiency. Such a failure may be the cause of an anomaly in a downstream surgery for a different team. If the failure chronically affects performance for the same team or team member in a downstream surgery, it may itself be representative of an influencer for the downstream inefficiency.

[0090] Thus, one will appreciate from the foregoing that a GUI with the elements of FIGs. 6 and 7 may be able to present cross-referenced and correlated representations of data, including both non-theater-wide and theater-wide data, to the reviewer. Timestamps, task start and end times, system events, kinematics values outside thresholds (e.g., determined from historical data or correlation values), arbitrary reviewer bookmarks at a given moment in playback, etc., may all be used as “anchors” with which to simultaneously present visualization data from the surgery (e.g. in element 610a), theater-wide data (e.g., in element 615a), and other data (e.g., via the timelines 610c, 61 Of, 61 Oi, 61 Oj, 610k, 670a) as well as to assess correlations therebetween. Thus, the reviewer may, e.g., distinguish the significance of different occurrences of a system event based upon corresponding task start and stop times, focus playback upon periods around the most significant occurrences, then review the corresponding theater-wide and other data to verify the existence of an inefficient action and its cause. The reviewer can record historical patterns of this nature and create corresponding rules to expedite bookmark creation and future review (e.g., at block 505e). For example, as repeated “head out” events during a task may indicate inefficiencies in the theater outside the surgical field, the system may review future incoming datasets (e.g., at blocks 505c, 505d, and 505e) to recognize such occurrences and automatically create bookmarks to expedite the reviewer’s analysis of the future dataset.

Example Adaptive Data Consolidation

[0091] As discussed, various of the disclosed embodiments may be applied to datasets with disparate sensor information. For example, some datasets may have no theater-wide data, have visual image theater-wide data only, have depth frame theaterwide data only, have both image and depth theater-wide data, etc. Similarly, different of these types of data may be available at different times within a single surgery, in different surgeries, in the period between surgeries, etc. FIG. 8 is a flow diagram illustrating various operations in an example process 805 for adaptively consolidating data in these various formats, e.g., so that they may be cross-referenced with non-theater-wide data despite their disparate character (e.g., as used at blocks 505d and 505e). In this manner, correlated playback, e.g., between element 610b and the view 710 may be possible, even as the character or quality of the theater-wide data available to be rendered in view 710 changes.

[0092] At block 805a, the system may consider whether a dataset includes theaterwide depth data from sensors (e.g., sensors 170a or 170c) during the surgery. If so, the system may determine whether a consolidated virtual element model view during some, or all, of the theater is possible from the acquired data (e.g., confirming that the theaterwide sensor data overlaps spatiotemporally such that each sensor’s cotemporaneous data frames may be registered with one another) at block 805b. If consolidation is possible, then the system may prepare appropriate consolidated virtual elements (e.g., three-dimensional representations of the theater, e.g., as vertex meshes derived from combined Truncated Signed Distance Function representations of the depth values) at block 805c (e.g., for some, or all, of the durations during which the sensor data may be registered). Naturally, in some embodiments, the consolidated representation may already be created and simply referenced at this step. If consolidation is not possible, then individual depth fields of view may be noted for the appropriate time at block 805d. If visual image data is available at block 805e, e.g., where the sensors 170a and 170c acquire both depth and visual images throughout the procedure, then the depth frame or consolidated representations determined at blocks 805d and 805c, respectively, may be textured with these visual images at block 805f. The depth data may then be made available for future consideration at block 805g (e.g., as part of the initial scoring at block 505c)

[0093] In contrast, where depth data is not available at block 805a, but the system determines that only visual image data (e.g., only RGB data) is available at block 805h, the system may consider whether depth data may be inferred form the visual image data at block 805i based upon the poses of the sensors (or, e.g., the time-varying pose of a single sensor). For example, some sensors may acquire stereoscopic images of their fields of view, facilitating depth inference based upon the known separation of their respective poses. Similarly, where the relative poses of the visual image capturing sensors around the theater are known, it may be possible to similarly infer corresponding depth values. For some configurations, motion within the theater may also facilitate depth inference, as the parallax and occlusions produced as team members and equipment move about the theater may facilitate depth inference. Where such depth inferences are possible, the system may determine synthetic depth data at block 805j and then use the data in blocks 805b-d as previously described. Naturally, the visual images may be used at block 805f to texture these presentations.

[0094] If depth data cannot be inferred from the visual images, however, then at block 805k the system may prepare individual field of view renderings (e.g., aligning the image data with timestamps in the surgery if this has not already been done) and publish the results for future consideration at block 805g. Finally, where neither depth nor visual in-theater data is available, then at block 805I the system may note the absence so that downstream processing may consider the dataset accordingly. One will appreciate that the process 805 may be applied repeatedly for different temporal portions of the data as, e.g., some temporal periods may have depth data, some only visual data, some periods may have no theater-wide visualization data at all, or data of unacceptable quality, etc., depending upon the nature of the dataset. This may also be the case when data is available throughout the surgery, but the best view of a region of interest may vary throughout the surgery, and consequently, the most appropriate choice for rendering.

[0095] Similarly, in some embodiments, the computer system may present a default view 710 in element 615a based upon a preferential ordering of the available representations. For example, if a consolidated virtual model is available, the model may be rendered in a desired pose. If the model is not available, but at least one depth frame is available, then the depth frame may be rendered. If the depth frame is not available, but visual image video is available, then the visual image may be rendered. If neither depth nor visual image data is available at that moment in playback, then the system may seek an interpolated rendering (e.g., from nearby preceding or subsequent acceptable data values, which may be particularly suitable for short durations of unavailability), may transition back to the configuration of FIG. 6, or may simply indicate that no theater-wide data is presently available in element 615a.

Example Theater-Wide Sensor Topologies

[0096] As mentioned, the element 615a in the view of FIG. 7 may be populated with depth values, visual images, etc. As discussed above, where multiple data types are available at a given time in the surgical playback, e.g., multiple depth fields of view, multiple visual image camera fields of view, etc., the system may preferentially select from the available viewing options. Various embodiments may additionally, or alternatively, select an appropriate rendering using knowledge of the theater topology, statically or dynamically over time.

[0097] For example, FIG. 9A is a schematic depth map rendering from an example theater-wide sensor perspective 905 (e.g., as may be rendered for the view 710 in element 615a), as may be used in some embodiments. Specifically, this example depicts depth values corresponding to an electronics/control console 905a (e.g., the electronics/control console 145) and a nearby tray 905b, and cabinet 905c. Also within the field of view are depth values associated with a first technician 905d, presently adjusting a robotic arm (associated with depth values 905f) upon a surgical robot system (associated with depth values 905e). Team members, with corresponding depth values 905g, 905h, and 905i likewise appear in the field of view, as does a portion of the surgical table 905j. Depth values 905I corresponding to a cart and a boom with a lighting system’s depth values 905k also appear within the field of view.

[0098] The theater-wide sensor capturing the field of view 905 may be only one of several sensors placed throughout the theater. For example, FIG. 9B is a schematic top- down view of objects in the theater at a given moment during the surgical operation. Specifically, the field of view 905 may have been captured via a theater-wide sensor 920a with corresponding field of view 925a. Thus, for clarity, cabinet depth values 905c may correspond to cabinet 910c, electronics/control console depth values 905a may correspond to electronics/control console 910a, and tray depth values 905b may correspond to tray 910b. Robotic system 91 Oe may correspond to depth values 905e, and each of the individual team members 910d, 910g, 910h, and 910i may correspond to depth values 905d, 905g, 905h, and 905i, respectively. Similarly, cart 9101 may correspond to depth values 9051. Depth values 905j may correspond to table 91 Oj (with an outline of a patient shown here for clarity, though the patient has not yet been placed upon the table corresponding to depth values 905j in the example field of view 905). A top-down representation of the boom corresponding to depth values 905k is not shown for clarity, though one will appreciate that the boom may likewise be considered in various embodiments.

[0099] As indicated, each of the sensors 920a, 920b, 920c is associated with different fields of view 925a, 925b, and 925c, respectively. Generally, the fields of view 925a-c may have complementary characters, some fields of view being more suitable than others for viewing various objects in the theater. Note, as mentioned, that this complementarity may be dynamic both spatially and temporally. Such dynamic character may result from movement of an object being tracked, but also from movement of intervening occluding objects (and, in some cases, movement of the sensors themselves). For example, at the moment depicted in FIGs. 9A and 9B, the field of view 925a has only a limited view of the table 910j, as the electronics/control console 910a substantially occludes that portion of the field of view 925a. Consequently, in the depicted moment, the field of view 925b is better able to view the surgical table 910j. However, neither field of view 925b nor 925a has an adequate view of the operator 91 On in console 910k. To observe the operator 910n (e.g., when they remove their head in accordance with “head out” events), field of view 925c may be more suitable. However, over the course of the operation, these complementary relationships may change. For example, before the procedure begins, electronics/control console 910a may be removed and the robot 91 Oe moved into the position 910m. In this configuration, field of view 925a may instead be much better suited for viewing the patient table 91 Oj than the field of view 925b. As another example, movement of the console 910k to the presently depicted pose of electronics/control console 910a may render field of view 925a more suitable for viewing operator 910n, than field of view 925c. Suitability of a field of view may thus depend upon the number and duration of occlusions, quality of the field of view (e.g., how close the object of interest is to the sensor), and movement of the object of interest within the theater. Such changes may be transitory and short in duration, as when a team member moving in the theater briefly occludes a sensor, or they may be chronic, as when equipment is moved into a fixed position throughout the duration of the procedure.

Example Theater Element Region of Interest Management Methodologies

[0100] To facilitate the reader’s comprehension, FIG. 10A is a schematic timeline depicting region-of-interest availability throughout a procedure, as may occur in some embodiments. For example, surgical table 91 Oj may be a region-of-interest when the reviewer is assessing the quality of the team’s chosen surgical port placements. As indicated in the example perspective of the view 905, the view of surgical table 91 Oj from sensor 920a may be partially occluded by electronics/control console 910a. By comparing the state of occlusion from each of the theater-wide sensor fields of view (e.g., fields of view 925a, 925b, 925c), the system may infer the choice of one or more sensors at a given time in the procedure most suitable for monitoring the region of interest.

[0101] Specifically, in this example, timeline 1010a indicates the visibility of the surgical table 91 Oj from the sensor 920a’s field of view 925a over time 1025, particularly over the course of the surgical procedure. Similarly, the timeline 1010b indicates the visibility of the surgical table 91 Oj from the sensor 920b’s field of view 925b over the course of the surgery and the timeline 1010c indicates the visibility of the surgical table 91 Oj from the sensor 920c’s field of view 925c over the course of the surgery. Though shown here as a binary “visible” or “not visible” value, in some embodiments, visibility may be a more granular numerical “score” value. For example, visibility may be a percentage of the field of view depicting the region of interest, the unoccluded distance from the sensor to the region of interest, a visible area of the region of interest scaled by the distance, a combination of these factors, etc. In some embodiments, visibility may also be a factor of the region of interest’s orientation, e.g., obliqueness, relative to the field of view. For example, a dot product between the field of view and one or more normals upon the region of interest may be used as a measure of visibility (e.g., when it is desirable to view the top of the table 91 Oj or the patient thereon), possibly in combination with the other factors discussed above.

[0102] Where only depth data is available, the system may recognize a region of interest, e.g., from an object’s surface mesh or silhouette, using a machine learning system, such as a neural network, trained to recognize objects from depth data. Conversely, where only visual image data is available, a You-Only-Look-Once “YOLO” or similar network may be used to recognize an object within the field of view, as well as corresponding pixels, boundary boxes, etc. Where both visual images and depth data are available, both approaches may be used, e.g., as part of an ensemble classifier receiving each machine learning model’s respective output, as a weighted vote of each machine learning model’s respective output, etc.

[0103] In this example, all three fields of view initially perceive the surgical table 91 Oj at time 1020a. As mentioned, visibility may not be a binary property. Accordingly, “visible” may here refer to fields of view whose perception of the region of interest exceeds a scoring threshold as specified above. However, between the time 1020b and the time 1020c, the second view 925b, associated with sensor 920b, may no longer have adequate visibility of the surgical table 91 Oj region of interest. For example, the robotic system 91 Oe may now be in position at location 910m, completely, or substantially, occluding surgical table 91 Oj from sensor 920b‘s field of view 925b. Similarly, beginning at time 1020d, the surgical table 91 Oj may no longer be visible from sensor 920a (e.g., as the electronics/control console 910a has been moved directly in front of the of the sensor 920a). Similarly, at time 1020e surgical table 910j may be unsatisfactorily occluded from the perspective of sensor 920c, as team members 91 Od, 910g, 910h, and 910i gather around the console 910k to discuss the state of the procedure. Similarly, electronics/control console 910a may still occlude sensor 920a’s field of view 925a. Consequently, only the sensor 920b may now have an adequate view of the surgical table region of interest 91 Oj (e.g., during the interval 1015c).

[0104] Pursuant to the different available visibilities at a given time, the system may adjust its visualization selection for a given time in playback. For example, during playback in the interval 1015a, where the region of interest is visible from all the theaterwide sensors, the system may render the depth view with the best visibility score, may render a composite object (e.g., a three-dimensional virtual model combining the depth data from the three views) using each of fields of view 925a, 925b, 925c, etc. Similarly, in the interval 1015b, where only data from fields of view 925a and 925c are available, the system may prepare a view based upon only these data streams. For example, the system may select the view with the best visibility score, may generate a composite virtual element from just these two views’ data, rotated and translated appropriately per a corresponding visibility score, etc. However, during interval 1015c, only sensor 920b has an adequate view of the surgical table 91 Oj. Thus, only its depth image or visual image may be presented during playback, rather than a virtual element derived from multiple fields of view. Similarly, where insufficient data for a composite representation that includes the region of interest is available in interval 1015b, the system may show only the best view from one of sensors 920a and 920c.

[0105] For clarity, FIG. 10B is a collection of schematic fields of view 1030a, 1030b, 1030c of an example surgical table region of interest (e.g., as seen directly by a sensor, or following a pose adjustment to a virtual element), as may occur in some embodiments. Particularly, as mentioned, some embodiments may assess a visibility score based upon the size and perspective of the region of interest within the field of view (for visual images, a quality of focus visual definition of the region of interest may be considered in the score). For example, in the view 1030a the system may detect the portion of the visual image or depth frame associated with the region of interest (here, the outlined region 1035a around the surgical table). As mentioned, such detection may occur, e.g., via application of a YOLO neural network to a visual image, a corresponding neural network for depth value classification, a combination of the two, etc. In contrast, in the view 1030b, there is no occlusion, and the entire table is visible, albeit from the perspective of the sensor’s pose, facilitating an outlined region 1035b around the surgical table larger than in the view 1030a. Consequently, the view 1030b may receive a higher visibility score than the view 1030a. For example, where the visibility score is simply the number of pixels in a visual image associated with the region of interest, then more pixels will be available in the view 1030b as compared to the view 1030a (naturally, other conditions may be imposed, such as a minimum distance, to prevent unhelpfully zoomed views of a region of interest from receiving undesirably high scores). However, by a visibility metric which simply considers the number of pixels, the view 1030c will receive a superior visibility score than either of views 1030a or 1030b, as indicated by the large outline 1035c. In addition, where a machine learning detector of the region of interest also facilitates inference of the orientation of an object of interest, the orientation may additionally be considered in the visibility scoring metric (e g., based upon an alignment with a preferred pose). For example, when viewing the surgical table, perspectives facilitating viewing of the surgical area may be especially valuable (e.g., so that the reviewer, or the system, might identify and assess the port placement choices of the surgical team). Consequently, the scoring metric may favor certain perspective views, such as that shown in view 1030c, corresponding side views, etc., which provide larger, more accessible, perspectives of the region of interest in a relevant orientation.

[0106] One will appreciate that where composite views are available, as when the system combines depth frames from depth sensors (or visual image sensors for which depth may be inferred), the resulting composite virtual element may be rotated and translated to achieve the more desired perspective of view 1030c (naturally, scaling of the element may have the same effect as zooming for some graphical pipelines). Thus, view 1030a may be only one of several available depth sensor views, which, when combined, facilitate a composite rendering translated and rotated to accommodate the view 1030c, which will be presented to the reviewer. Once this view has been determined, the system may continue to present this perspective during playback so long as the composite rendering remains viable (e.g., adequate continuity exists between successive composite renderings, the depth sensor fields of view continue to provide adequate visibility scores, etc.)

[0107] FIG. 10C is a flow diagram illustrating various operations in an example process 1050 for region-of-interest management, as may occur in some embodiments. At block 1050a, the system may receive a region of interest designation. For example, the user or a computer system may: mark pixels to be tracked via optical flow; designate a type of object to be recognized and tracked via a neural network, as described herein; provide a template image or depth representation to recognize and track within the sensor fields of view; etc.

[0108] In some embodiments, the process 1050 may iterate over all the time intervals of the procedure, e.g., as described above with respect to FIG. 10A. Here, to facilitate understanding, the computer system receives the specific interval for consideration at block 1050b, e.g., one of the intervals 1015a-c. For the data available during this interval, at blocks 1050c and 1050d the system may iterate over the theaterwide sensor fields of view, determining region of interest availability scores at block 1050e. While a single score for a view may be determined as described above, in some embodiments, multiple scores may be determined using different methods and their result summed, averaged, integrated via a weighted average, etc., for the view. Once all the views have been considered, at block 1050f the system may determine if integrated representations, such as virtual composite elements, are already available, e.g., as determined at block 805c or if they can be created if they do not yet exist. Where the composite representation of the region of interest does not exist, and it is possible for it to be generated from the views, then it may be generated at block 1050g. For example, in some embodiments, the holistic integration of block 805c, using all the sensor data, may not focus upon the specific character of the region of interest and so an adequate virtual element of much of the theater, but not of the region of interest, may have been created. Accordingly, in some embodiments, even if a composite representation is available from block 805c, which includes the region of interest, the system may still prepare another consolidated representation at block 1050g focusing upon the region of interest, specifically (e.g., using only depth frames with visible scores above a threshold, extracting the relevant portion from the block 805c representation, etc.). Similarly, some embodiments may consider if the composite representation is available from block 805c for use in improving presentation at block 1050g. For example, occlusions during the interval of block 1050b may prevent a desired representation of the region of interest. However, in-filling based upon depictions of the region of interest outside the interval, possibly in conjunction with other types of data, may be used to create an improved virtual element representation of the region of interest at block 1050g. For example, where the port placement locations are not visible during the interval of block 1050b, but are not occluded during a preceding or subsequent, and nearby, interval, then the representation at block 1050g may avail itself of this preceding or subsequent data, possibly in combination with kinematics data for instruments involved in the port placements, to interpolate between virtual element representations of the region of interest and prepare a representation at block 1050g. [0109] When the virtual element representation of the region of interest is available, the system may determine a best scoring perspective with which to present the representation to the user. For example, as discussed above, having created a composite representation of the surgical table, the representation may be rotated and translated to a variety of views to achieve the view 1030c. One will appreciate that Monte Carlo methods, particle filters, random walks, gradient descent (where the field of view score is used as part of a metric), etc., may be used to identify a suitable perspective at block 1050h.

[0110] At block 1050i , the system may confirm that the best scoring available view, whether the perspective composite view of block 1050h or one of the individual views scored at block 1050e, is above a threshold. For example, if the region of interest is absent or significantly occluded in all the fields of view, it may be more appropriate for the system to return a non-visible notification at block 1050j rather than return the poorly scoring view. Conversely, where the best scoring view is satisfactory (whether it be from a pose-adjusted virtual element or from an original sensor view), then at block 1050k the system may refer that view for rendering to the user (e.g., in the field of view 710 rendered in element 615a).

[0111] As mentioned, the process 1050 may be applied for a plurality of intervals throughout the playback. In some embodiments, even if a different view than as is presently presented receives a higher score, the system may only transition or select the other view if the other view retains a higher score above a desired threshold for an adequate duration. Such a condition may prevent the system from undesirably transitioning between views for very short intervals, which may be disorienting to the reviewer. One will also appreciate that in some embodiments the system may first filter the theater-wide data generally based upon the process 805, then filter the data again in accordance with visibility of a region of interest, via the process 1050, possibly resulting in a different choice from process 805, as when a visual image presents a better view than a consolidated virtual element. Conversely, in some embodiments, visualization selection via a process similar to process process 1050 may instead first limit the data to be considered by process 805, as when fewer than all the depth theater-wide sensor fields of view are to be considered in forming a virtual element. In still yet other embodiments, one will appreciate that the processes 805 and 1050 may be consolidated into a single process, rather than run separately.

Example Theater Region of Interest Isolation Methodologies

[0112] Following the system’s (e.g., when applying a predefined bookmark from historical data) or user’s identification (e.g., manually selecting an object in the theaterwide data for tracking) of a region of interest, various configurations of certain embodiments may isolate the region of interest for presentation in the element 615a. For example, FIG. 11A is a schematic rendering of a theater view 1105, e.g., from the field of view of a visual image sensor, the field of view of a depth sensor, or a perspective view of a composite representations of the theater, with example isolated elements, as may be implemented in some embodiments. Specifically, as the patient is the region of interest in this example, the system may reduce the opacity, mask, remove, or otherwise de-focus virtual elements, portions of virtual elements, or pixels in the view other than the region of interest. Here, the system continues to render the region of interest 1105c at full opacity, but renders the other objects 1105a, 1105b, and 1105d at a reduced opacity. Where the view 1105 is a virtual element, a compass 1105e may be provided in the element 615a to orient the reviewer (e.g., to the pose of the view or the region of interest relative to the theater as a whole).

[0113] As mentioned, the viewer may adjust the rendering in element 615a during the playback in various manners for various embodiments. For a virtual element rendering (e.g., derived from depth data), the user may rotate the region of interest, e.g., following isolation, as shown in FIGs. 11 B and 11 C (or, as described, the system may rotate the region in considering a best pose and view score). Thus, element 615a may present the side-view perspective 1110 of the patient region of interest 1105c or the angled perspective 1115 from the foot of the surgical table (naturally, where a virtual element is generated from a single sensor point of view, occluded regions of the virtual element may be in-filled or not rendered). In some embodiments, the compass 1105e may be appropriately rotated or translated to help orient the reviewer (e.g., when the reviewer iterates between global views of the composite representation and focused, e.g., bookmarked, local view camera poses, and wishes to appreciate the orientation relative to the theater as a whole).

[0114] The rendering in element 615a may be supplemented with highlights, virtual elements, etc., to correlate the theater-wide data with raw data values, OPI values, with various non-theater-wide data, etc. For example, here, the kinematics data for the visualization tool used to render the field of view in element 610b is used to create an arrow icon virtual element 1120 to quickly inform the reviewer of the correspondence between the perspective rendering of the region of interest and the field of view in in element 610b (e.g., from visualization tool 140d). The pose of the virtual element 1120 may accordingly change throughout playback in accordance with the visualization tool’s pose in the theater. Similarly, in some embodiments the surface of the patient may itself be isolated, as shown by the isolated region of the patient surface 1130, or highlighted, relative to the virtual element 1120 indicating the visualization tool’s pose. As another example, when the reviewer or computer system is interested in assessing the surgical team’s choice of port placements, the locations associated with virtual elements 1125a and 1125b, corresponding to the port placements, may be determined from the kinematics data and/or inspection of the theater-wide data (e.g., as adjustments to the texture rendering of the surface of the patient 1130, perspective billboard outlines around the locations, cylindrical mesh models placed upon the surface of the patient 1130 at the locations and oriented relative to a normal upon the surface of the patient 1130 at that location, etc.). The addition of virtual elements based upon non-theater-wide data, such as instrument kinematics data, e.g., as with virtual elements 1125a, 1125b, and 1120, may provide a direct correspondence between the view in element 610b and the view in element 615a. In combination with OPI data, e.g., as rendered in timeline element 670a, such correlated representations may readily facilitate the identification, by the system, or by the user, of inefficiencies, anomalies, and their causes, as described in greater detail herein (e.g., additional virtual elements depicting corresponding historical positions may be also presented).

[0115] FIG. 11 D is a flow diagram illustrating various operations in an example isolated virtual element rendering process 1150, as may be implemented in some embodiments. At block 1150a, the system may receive the depth or pixel value selection, e.g., the desire to isolate the patient region of interest 1105c (or, more specifically, the surface 1130) as in FIG. 11A (e.g., via preexisting bookmark, selection by the user, tracking via a machine learning model, etc.). Thus, in addition to points in time, and perspectives of theater-wide data, one will appreciate that bookmarks may also be used to specify a region of interest to isolate. Where the theater-wide data is a visual image and a YOLO-style network was used for the region of interest identification, then the isolation may already be output from the network. Similarly, where a machine learning system located the desired region of interest in the depth data, the depth values in the consolidated representation may be readily inferred. Again, in some embodiments, the user may determine the selection as an arbitrary depth value selection or an arbitrary pixel value selection and may bookmark the result for future consideration.

[0116] At block 1150b, the system may track the object through a desired time interval, e.g., by repeated application of the detection method, by assessing continuity using optical flow assessments between consecutive depth frames or visual images, etc. At block 1150c, the system may determine the associated data, including non-theater- wide data, for the selected region. For example, where the depth values are known to correspond to regions on the patient, then the kinematics data associated with rendering the pose indication 1120, port placement virtual elements 1125a, 1125b, etc., may be acquired. At block 1150d, the system may gather data for virtual element rendering (e.g., to render the virtual elements 1125a, 1125b, and 1120), e.g., translating the information from block 1150c into an appropriate pose for rendering. In some embodiments, the user may specify the virtual elements to render from kinematics data regardless of the particular region presently selected or presented (as this may more readily facilitate the recognition of various correlations, anomalies, inefficiencies, causal influencers, etc.).

[0117] At block 1150e, the system may return the rendering information, e.g., the relevant virtual elements and their respective poses throughout the playback, information regarding which depth values or pixels values to focus or de-focus during the playback, etc. In some embodiments, at block 1150f, the system may record the selection as a bookmark (e.g., one of bookmarks associated with icons 710b, 710c). Particularly where there are many bookmarks, this may readily facilitate the reviewer’s cross-referencing information between the theater and non-theater-wide data, OPI values, etc., in the GUI 605 at different points in the surgery. For clarity, as bookmarks may identify both temporal periods as well as distinct objects and perspectives, multiple bookmarks may refer to the same region during an interval of playback, each from different perspectives, with different choices and renderings of associated kinematics data-derived virtual elements, etc.

Example Correlated Heads Up Display (HUD) Rendering and Highlighting

[0118] As mentioned, in some embodiments, the element 610b may not render only the imaging sensor output, but all or a portion of the GUI interface (including any HUD) presented to team members, e.g., via displays 125, 150, or 160a. In some embodiments the GUI may allow the reviewer to select from among such views or to present the views simultaneously. Accordingly, a particular choice of view or views may be included as part of the bookmark data.

[0119] FIG. 12A is a schematic screenshot of elements in a first image sensor GUI rendering during surgery, e.g., upon a team display such as display 150, as may be used in some embodiments. Similarly, FIG. 12B is a schematic screenshot of elements appearing in the HUD of a surgical console system, such as the display 160a of console 155. The view 1215 of FIG. 12C shows a variation of the image on display 160a within console 155. In some embodiments, the user may alternate between different display views in element 610a throughout playback (e.g., between the console 155 view 1210 and the display 150 interface view 1205) in element 610a, or the views may be presented simultaneously. In some embodiments, elements appearing in these interface views 1205, 1210, 1215 during the course of surgery may be recognized and correlated with timelines e.g., 610c, 61 Of, 61 Oi , 61 Oj, 610k, 670a, etc., as well as with objects and virtual elements appearing in the view 710 of element 615a.

[0120] For example, elements appearing in the GUIs may inform the detection, and the distinguishing, of anomalies and inefficiencies in the surgery via the reviewer’s visual inspection. Such visual inspection may complement or better distinguish anomalies from inefficiencies than assessment exclusively via OPI analysis. For example, repeated adjustment of the display settings element 1205g may be indicative of an equipment failure anomaly, explaining what may otherwise appear to be genuinely inefficient behavior by the team members. Similarly, occlusions of the surgical field by one or more of instruments 1215a, 1215b, or 1215c as well as by various icon elements, such as icons 1205g, 1205c, 1205d, 1210h, 1215d, 1215f, may inform the reviewer of the operator or team member’s cognitive state. For example, such occlusions may explain an operator’s delayed reaction to the presentation of an important anatomical artifact (e.g., a tumor, abscess, etc.) within the visualization tool’s 140d field of view. Thus, in some embodiments, rendering the entire GUI may facilitate a better appreciation of the operator and team members’ cognitive states.

[0121] In some embodiments, renderings of the GUI interfaces may also be augmented to call attention to GUI elements associated with aberrant OPI values or other significant events. For example, FIG. 13 is a flow diagram illustrating various operations in an example process 1305 for GU l-highlighting management, as may be implemented in some embodiments. At block 1305a, the system may receive a data type selection, e.g., from the reviewer or from an automated process recognizing historical outliers in the OPI metrics. For example, the user may highlight an instrument attachment GUI icon, activation of an instrument (e.g., a cauterizer) GUI icon, user changes to display settings, motion of an instrument, movement of a specific instrument (e.g., the visualization tool 140d), etc. At block 1305b, the system may determine the theater-wide data corresponding to the selection. For example, where the data type selected at block 1305a concerns an instrument attachment, the system may identify the instrument prior to attachment and the robotic arm to which the instrument will be attached in the theaterwide data (e.g., as recognized by a YOLO, or analogous depth-based neural network and tracking system). Similarly, selecting an icon associated with GUI display settings may precipitate the system’s identifying theater-wide data associated with the device upon whose output the GUI’s HUD was displayed (e.g., surgeon console 155 or electronics/control console 145).

[0122] At block 1305c, the system may determine the GUI elements (e.g., in the interface views 1205, 1210, 1215) corresponding to the data type selected at block 1305a (e.g., based upon a table indicating such relations). For example, an instrument selection may be associated with GUI elements 1215j, 1215i, 1215h, 1215g or with the depictions of the instruments 1215a, 1215b, 1215c themselves. The instruments (e.g., 1215a, 1215b, 1215c) may themselves be highlighted by outlining the instrument’s pixels, introducing augmented reality virtual elements into the rendering, etc. Similarly, where display settings are adjusted, elements such as icon 1205g may be focused, via a change in the pixel opacities, highlighting around the element, etc.

[0123] At block 1305d, the data determined at blocks 1305b and 1305c may be consolidated. For example, the time intervals during which the highlights of the respective instruments appear in both theater-wide and non-theater-wide data may be noted. In some embodiments, each of the elements 610b and 615a may render the highlighted elements simultaneously. During these periods of simultaneity, at block 1305e the system may determine the most appropriate viewing presentation in element 615a of this region of interest in the theater-wide data (e.g., using process 1050).

[0124] Thus, until the review is complete at block 1305f, at block 1305g the system may consider whether the data types selected are “active”, e.g., in accordance with the determinations at 1305b-d (e.g., that an instrument is visible within the theater, that the GUI depicts activation of an instrument, etc.). Where the data type is active, appropriate highlights may be made in blocks 1305h and 1305i within element 610b and the view 710 of element 615a, respectively. At block 1305i, the pose of the theater-wide data derived view may also be adjusted as described herein to focus upon the regions of interest associated with the active data types (though one will appreciate that block 1305i may be ignored, and the view not changed, in some embodiments, to facilitate the user’s comparison with, e.g., an arbitrarily chosen of the theater-wide data).

Example Bookmark Management Methodologies

[0125] FIG. 14 is a flow diagram illustrating various operations in an example bookmark management and rendering process 1405, as may be implemented in some embodiments. At block 1405a the system may receive an instruction to create a new bookmark (e.g., via icon 710a). At blocks 1405b and 1405c the system may associate the new bookmark with the corresponding non-theater-wide and theater-wide data. In some embodiments, the user may specify a timestamp or temporal interval during which the bookmark applies, as well as a perspective view from the theater-wide data presently appearing in the element 615a (e.g., by adjusting the pose of the depth-data derived virtual element using mouse or keyboard commands). At blocks 1405b and 1405c the system may acquire data at the designated time or during the designated interval to render the view, render corresponding virtual elements, render isolated regions of interest in the theater-wide data appropriately, present relevant highlights of GUI and other nontheater-wide data (e.g., as discussed above with respect to FIGs. 12A-C and FIG. 13), etc. The bookmark and its associations may then be stored for future reference at block 1405d, e.g., in any suitable storage format, such as JSON, XML, etc.

[0126] Until the user’s review is complete at block 1405e, whenever the user selects the bookmark at block 1405f (e.g., selecting one of icons 710b, 710c) the system may acquire the relevant data at block 1405g, e.g., as was stored at block 1405d. If the bookmark does not itself identify an appropriate choice of sensor view or virtual element pose for the theater-wide data, then the system may determine an appropriate perspective or pose at block 1405h for presenting the data in element 615a (again, in some embodiments, the view may not be changed and block 1405h not performed, so that arbitrary comparisons between non-theater-wide and theater-wide data may be more readily inferred during the bookmark). At block 1405i, the system may then initiate playback at the bookmarked timestamp or at the beginning of the bookmarked interval, rendering appropriate virtual elements in element 615a, adjusting GUI highlights in element 610b, highlighting of appropriate regions in the timelines 610c, 61 Of, 61 Oi, 610k, 610k, 670a, etc.

[0127] Naturally, the bookmarks may also be stored for use in subsequent reviews, as when historical data is considered. By either manually (e.g., as described in this process 1405) bookmarking or automatically (e.g., based upon the user’s specification of intervals based upon system event and task duration correlations, as discussed elsewhere herein) bookmarking related intervals across multiple surgeries, the reviewer may be able to readily compare surgeries with one another. Such accessible comparison of disparately gathered data may facilitate the reviewer’s recognizing inefficiencies and their causes, as well as distinguishing genuine inefficiencies from one-time anomalies.

Example Correlated Scoring Methodologies

[0128] FIG. 15 is a flow diagram illustrating various operations in an example scoring and visualization process 1505, as may be implemented in some embodiments. At block 1505a, the system may pre-process the theater-wide datasets, non-theater-wide datasets, and any relevant historical datasets. For example, the data in the datasets may be reformatted into a form suitable for comparison. At block 1505b, the system may cross-reference the datasets, e.g., as described above, determining overlapping time periods in the surgical procedure. At block 1505c, any relevant input from the user, such as bookmark selections for specific OPIs, historically specified bookmarks, bookmarks automatically generated per specified criteria, etc., may be considered.

[0129] At block 1505d, the system may perform the initial playback score determination, e.g., calculating OPI, skill, and task metric scores throughout the surgery. These initial OPI values over the duration of the surgical procedure may, e.g., be used to populate bar charts, pie charts, timelines, etc., such as timeline plot 670b in timeline element 670a. To assess correlations between theater-wide and other data, in some embodiments, the system may perform one or more iterations throughout all the data in parallel (without attempting to render the data to the reviewer). Such moment by moment analysis may allow the system to prepare appropriate virtual elements, generate timelines as discussed with respect to FIG. 10A and to determine appropriate corresponding theater-wide views, recognize correlations between kinematics data, system data, and theater-wide data, etc.

[0130] In some embodiments, the system may also consider a rendering priority at block 1505e, such as for representing graphical elements in element 610a (e.g., the choice of OPIs, timelines, and their ordering) and virtual elements in element 615a (e.g., virtual elements to accompany regions of interest). For example, virtual elements in element 615a may be focused or unfocused (e.g., in accordance with the discussion of FIGs. 11A-C) based upon the corresponding object’s association with undesirable or desirable metric scores (e.g., where the score is outside a threshold and identified as being associated with that element or where data associated with the object contributes to the calculation of the metric score). Similarly, the elements populating element 610a may change over the duration of playback to bring OPI and data determinations exhibiting undesirable behavior, or behavior counter to historical patterns, to the reviewer’s attention (e.g., their OPI timeline being featured more prominently in the element 610a). For example, the ordering and arrangement of the timelines 610c, 610f, 610i, 610j, 610k, and 670a, may change depending upon the importance of the respective data. OP I deviations may be normalized relative to corresponding historical data and totally ordered to facilitate the determination of the rendering priority.

[0131] At block 1505f, the user or the system may initiate playback within the review GUI. As playback advances at blocks 1505g and 1505h, the system may update the non-theater-wide data rendering (e.g., in element 610a and, if in the configuration of FIG. 6, task data, system event data, etc. in the element 615a) at block 1505i. Where theaterwide data rendering is desired at block 1505j (e.g., following selection of the toggle switch 605a, so that element 615a displays theater-wide data, as in the view 710) the system may update the rendering at block 1505k.

[0132] At block 15051 the user may adjust a metric, e.g., by adding or removing a task, skill, or OPI score for calculation, changing the parameters of a task, skill, or OPI score calculation, etc. As mentioned, OPIs, tasks, and skills may be directed to non- theater-wide data exclusively (e.g., instrument motion kinematics within the patient), theater-wide data exclusively (e.g., port placement of the instrument as depicted by one of virtual elements 1125a and 1125b), or combinations of the two (e.g., a metric considering both instrument range of motion in connection with the chosen port placement or a metric considering head out events in connection with optical flow and motion in the theater and with a temporal location within a task).

[0133] Where the user performs such a metric adjustment at block 15051, at block 1505m the system may perform the appropriate score metric recalculation or new metric score calculation from the data. In some embodiments, the rendering priority (e.g., from the initial determination at block 1505e) may be adjusted at block 1505n based upon the change in OPI metric values. Once the review concludes at block 1505g, the system may prepare the OPI data at block 1505o for use in future reviews. For example, newly created score metrics, or adjustments thereto, at blocks 15051 and 1505m may be stored for future reference. Similarly, score metrics useful for future comparison, but which may not have been created at block 1505d or 15051, or which were changed from their normal state, may be calculated to facilitate such future comparison (e.g., to better inform the priority ordering determinations at blocks 1505e and 1505n). Example Composite OPI Metric Value Methodologies

[0134] FIG. 16 is a flow diagram illustrating various operations in an example OPI value determination process 1605 from cross-referenced data, as may be implemented in some embodiments. Though performed independently in some embodiments, in some embodiments, the process 1605 may be performed, e.g., at block 505c, at block 1505d during the initial assessment of the original set of OPIs, or following user adjustment to the OPIs at block 15051. At block 1605a, the system may collect the available theaterwide and non-theater-wide data for consideration. At blocks 1605b and 1605c the system may iterate over all the desired composite OPI values whose value determination involves both theater-wide and non-theater-wide data. At block 1605d, the system may determine the available portion of the non-theater-wide data which the OPI score is dependent upon. Similarly, at block 1605e, the system may determine the available portion of the theaterwide data which the OPI score is dependent upon. Each of the components at blocks 1605d and 1605e (each of which may include one or more aspects of the respective datasets) may or may not have conditional temporal limitations. For example, some composite OPIs may require temporal overlap between the non-theater-wide and theaterwide data contributions (as when the score compares camera instrument orientation to the location and speed with which an additional port is created and a second instrument inserted), whereas other composite OPIs may depend upon only partial, or no, temporal overlap between the non-theater-wide and theater-wide data contributions (e.g., an OPI incorporating instrument range of motion following a chosen port location).

[0135] At block 1605f, the system may confirm that the non-theater-wide and theater-wide component data determined at blocks 1605d and 1605e, respectively, are sufficient to fulfill the composite OPI’s calculation. If both sufficient theater-wide and non- theater-wide data is available, then at block 1605h the system may determine the composite OPI value. Conversely, if the original data does not satisfy the conditions, then at block 1605g the system may determine if it is possible, or permitted by the user, to infill or interpolate the available data so as to satisfy the conditions (one will appreciate that lacunae may exist in either, or both, of the theater-wide and non-theater-wide data). For example, if insufficient or inadequate theater-wide depth data exists to determine a port placement (e.g., because of an occlusion) then the system may attempt to infer the port location from available instrument kinematics data for an instrument at that port (e.g., where the location of a port used for an imaging device is known, then comparison of the imaging device’s kinematics with that of the instrument at the new port may, by inference, reveal the location of the occluded port). Similarly, in-filling of missing non-theater-wide and theater-wide data may be possible, as when temporally neighboring data is available. When possible, the system may perform such supplementation at block 1605i, before again seeking to determine the composite OPI value at block 1605h. Conversely, when supplementation is not possible, the system may note that the OPI value could not be calculated (at least with the existing conditions required by the OPI) at block 1605j (as indicated, noting which, or if both, of the non-theater-wide or theater-wide components were inadequate). In some situations, knowledge that the data does not facilitate certain analyses may itself inform the reviewer regarding inefficiencies or anomalies in the sensor deployment, the sensors’ configurations, the team’s use of the deployed sensors, etc.

[0136] Once all the composite OPIs have been considered at block 1605b, the system may return the results at block 1605k, e.g., for presentation in the GUI 605, for adjusting the rendering priority in block 1505n, etc.

Example Theater-Wide, Non-Theater-Wide, and Composite OPI Metrics

[0137] FIG. 17 is a schematic grouping of classes of OPIs, as may be used in some embodiments. Specifically, classes of OPIs availing themselves of theater-wide data, non-theater-wide data, or both theater-wide and non-theater-wide data may be grouped based upon the personnel within the theater whose performance they will be used to assess or who are associated with the data’s acquisition.

[0138] For example, OPIs in the group 1705 may be directed to assessing the performance of operator 105c (or, mutatis mutandis, surgeon 105a). A “head out” class 1705a of OPIs may monitor a system event, particularly the moments when surgeons move their heads outside of the surgical console (e.g., as detected by a limit switch). Some OPIs may monitor any action taken by the surgeon, which concludes the surgeon’s viewing of the surgical field, such as closing their eyes, transitioning to a display of other than the surgical field, etc. Some “head out” OPIs may avail themselves of both theaterwide and non-theater-wide data (e.g., a sensor, such as a limit switch, within the console and a visual or depth theater-wide perspective of the surgeon’s head leaving the console) to provide a composite analysis. Example OPIs in this class include, e.g., the total number of head removals, the median duration of head removals, a frequency of head removals (such as counts per minute), and event timestamps. While some OPIs may be amenable to calculation using only theater-wide or non-theater-wide data, often the OP I may be supplemented with data from both groups to form a composite metric (e.g., the gaze of the operator following a head removal may be derived from the theater-wide data and incorporated into the OPI, which originally only assessed a head-detecting sensor within the console 155). More frequent head out events may indicate intraoperative disruption (which may result from a one-time anomaly or a chronic inefficiency). By assessing both theater-wide and non-theater-wide data, the nature of the disruption may be made more readily apparent to the reviewer. In cases with two or more consoles (e.g., during teaching cases), head out events might also be used to assess the communication efficiency between the teaching surgeon and the learning surgeon (e.g., based upon their respective gazes following their respective head removals, such as whether the gazes meet). Thus, linking OPIs in this class between theater-wide and non-theater-wide data can help users, e.g., identify areas to improve communication, reduce intraoperative disruption, enhance teaching/learning experiences for dual console cases, etc.

[0139] As another example, a “console active time” 1705b class of OPIs may consider the duration during which the surgeon (again, either operator 105c, or, mutatis mutandis, surgeon 105a) actively controls any instrument(s). Composite analysis may be possible as such control may be monitored by both non-theater-wide kinematics data collected by the instruments themselves, as well as via theater-wide data of the robotic arm or instrument motion. Even when these OPIs consist only of non-theater-wide kinematics data, they may be used to supplement OPIs in the “head out” class 1705a, e.g., to ascertain when a surgeon’s heads is inside the console, but the surgeon is not actively controlling an instrument. Such behavior may correlate with certain inefficiencies, as when the surgeon is distracted, overextending their planning period, etc. For cases with two or more consoles, these OPIs may also indicate how long a surgeon observes a case rather than is actually operating (which, may, e.g., correlate with the quality of the teacher, experience gained by the student, etc.). Linking this OPI between theater-wide and non-theater-wide views, as in the GUI of FIG. 7, can help reviewers assess periods during the surgery when the console is active or inactive and why. For example, an active console may imply that the surgeon’s attention is focused upon the surgical field as it appears in video playback element 610b, and consequently, that the surgeon is unaware of the theater-wide dynamics appearing contemporaneously in the view rendering 710 of element 615a.

[0140] A “console handoff’ 1705c class of OP Is may track when one console obtains control from another console (e.g., where only one of the consoles may control one or more of the tools 140a, 140b, 140c, and 140d at a time). For example, OPIs in this class may include counts of the number of handoffs, a frequency of handoffs, a time between successive handoffs, handoff timestamps, etc. Console handoff can be an indicator of mentor/mentee coordination as well as of operator interaction during the procedure. Rendering these OPIs’ values in the element 610a alongside the respective console’s video playbacks (in two playback elements similar to playback element 610b), and also alongside the contemporaneous behavior in the theater via a rendering 710 in element 615a, may enable a reviewer to readily infer causes for console handoffs based upon the collective context or inefficiencies associated with such handoffs (e.g., when assistants regularly fail to adapt to the change of controlling surgeon). Similarly, consideration of these types of data may readily facilitate machine learning based identification of causal patterns. Understanding the causal relationships associated with handoff events may help improve teaching, learning, and the associated behavior of other members of the team.

[0141] As yet another set of examples, OPIs in the group 1710 may be directed to a first surgical assistant managing the patient side cart 130 (or, mutatis mutandis, the assistant 105b), focusing upon the assistant’s actions in connection with the operating surgeon. An “instrument exchange” class 1710a of OPIs in this group may include OPIs monitoring when instruments are installed or uninstalled from a robotic arm and/or replaced with a new instrument during the procedure (again, one will appreciate analogous OPIs for a surgeon 105a picking up / setting down, or inserting I removing, an instrument). Example OPIs in this class may include the count of such exchanges and timestamps, the types of instruments exchanged, the speed with which the exchanges were performed, etc. Instrument exchange timestamps may be used to review both theater-wide (e.g., the view 710) and non-theater-wide views (e.g., the view in element 610b) and to identify potential inefficiencies regarding how / when the assistant (e.g., assisting member 105b or 105d) prepares for the instrument exchange.

[0142] An “arm collision” class 1710b of OPIs may consider when a robotic arm collides with another arm (or an instrument with an instrument), monitoring, e.g., the number of times such collisions occur, the frequency of the collisions, the duration or force of the contact, the place of contact, timestamps at which the collisions occurred, etc. In some embodiments, bookmarks may be automatically generated at times at or near each collision so that the reviewer may readily advance playback to the relevant portion of the surgery to identify the potential cause for the collision. Such bookmark pre-population (which may be used for any of the OPIs described herein), may improve the reviewer’s ability to quickly assess a large number of surgeries in one or more theaters to, e.g., improve the theaters’ configurations so as to avoid inefficiencies or adverse events in future procedures.

[0143] A “port distance” class 1710c of OPIs may consider the coordinates of the ports (e.g., via the locations associated with virtual elements 1125a and 1125b), which may also be used to compute relative distances between ports (both during the surgery and to historical counterparts). One will appreciate that the “distance” may be the straight line distance between the two port locations in three-dimensional space, the shortest contour, or geodesic, over the surface of the patient between the two port locations, the shortest contour of accessible instrument insertion points across the surface of the patient between the two port locations, the combined distance of each port location to a target anatomy, etc. Port distance may be especially useful for recognizing atypical or inefficient procedures. Consequently, in some embodiments the computer system may automatically bookmark and highlight times in the surgery where the relative distance exceeds a threshold (e.g., an upper bound established outside a statistical norm from similar historical surgeries). By considering port placements, reviewers may identify how port placement / distance affects the procedure and identify opportunities for improvement. [0144] As with the group 1710, OPIs in the group 1715 may be directed to a surgical assistant managing the patient side cart 130 (or, mutatis mutandis, the assistant 105b), but here in connection with the rest of the team (OPI groups, such as groups 1710 and 1715, considering member’s actions in connection with one another may particularly benefit from consideration of composite theater-wide and non-theater-wide data). For example, an “instrument clutch” class 1715a of OPIs may monitor when the instrument clutch is pressed to facilitate instrument insertion or arm position adjustment without moving a port. Example OPIs in this class may monitor the number of such clutch activations, the duration of each activation, timestamps when the activation begins and ends, etc. These OPIs may be used to evaluate how efficient the team is during setups and instrument exchanges. Port placement, distances between ports, etc., derived from theater-wide data may also be used to inform these OPI scores.

[0145] As another example class in group 1715, the “port clutch” class 1715b of OPIs may monitor when the port clutch is activated so as to release tension at port sites or to adjust arm positioning and spacing. Example OPIs in this class may monitor the number of such activations, the duration of each activation, timestamps when the activation begins and ends, etc. Similar to instrument clutch OPIs, port clutch OPIs may be used to evaluate how efficient the team is during setup and port placement. Accordingly, theater-wide data indicating member placement and movement may help inform these OPI scores.

[0146] One will appreciate that the OPIs depicted here are merely exemplary, provided to facilitated the reader’s understanding, and that additional OPIs may apply within each class and within each group, as evidenced by ellipses 1725a-c. Similarly, one will appreciate that in some surgeries only one, two, or all three of the assistants associated with the groups 1710 and 1715 may be present in the theater. As evidenced by ellipsis 1750, other OPI class groups may likewise be considered. Indeed, the groups depicted here are merely exemplary and chosen for ease of comprehension, and OPIs may be determined for theater-wide, non-theater-wide, or composite raw data without any explicit group or class association.

Example Automated Anomaly and Genuine Inefficiency Detection Methodology [0147] As mentioned, GUI 605 may readily facilitate the detection of inefficiencies and anomalies by the reviewer via visual inspection, via assessment of existing OPIs, creation of new OPIs including composite OPIs, via assessing correlations between OPIs, via combinations of these approaches, etc. Generally, anomalies are undesirable events not directly associated with the team’s choices, and consequently, not generally appropriate for remediating feedback. Thus, a one-time “freak” power outage or a highly atypical patient reaction are anomalies. In contrast, inefficiencies correspond to undesirable events associated with the team’s choices, and (particularly when chronic) are consequently suitable for remediating feedback. Thus, slow preparation time, poor port placement, and unnecessarily delayed instrument attachments are examples of genuine inefficiencies.

[0148] In many cases, the reasons for an inefficiency, and the ability to distinguish an anomaly from an inefficiency, may be readily evident from visual inspection using the interface 605. However, in some embodiments, the computer system may assist the user by automatically detecting potential inefficiencies or potential anomalies, and flagging them to the user, in lieu of, or in combination with, the user’s manual inspection. For example, FIG. 18 is a flow diagram illustrating various operations in an example computer-implemented process 1805 for distinguishing anomalies from genuine inefficiencies and, in some embodiments, their causal influencers, as may be implemented in some embodiments. Generally, a causal influencer will manifest as a state or event temporally preceding, but highly correlated with, a subsequent inefficiency or anomaly.

[0149] At blocks 1805a, 1805b, and 1805c, the computer system may determine the non-theater-wide, theater-wide, and composite task, skill, and OPI scores. At block 1805d, the system may determine candidate inefficiencies from these scores, such as score values, or collections of score values, departing from a historical baseline more than a threshold amount or which sufficiently adhere to a pattern known historically to correlate with an inefficiency or with an anomaly.

[0150] For each of these identified candidate inefficiencies, at blocks 1805e and 1805f the system may determine causal influencers for the candidate inefficiency at block 1805g (for example, associations previously identified by a reviewer from historical patterns as indicative of a causal connection). Some OPIs may be specifically created to detect a causal agent for other OP I deviations and may consequently be used for the determination at block 1805g (e.g., some OPIs may consider the temporal or spatiotemporal correlation between other OPIs to infer a causal connection). In some embodiments, the GUI system may invite the human user to inspect the identified candidate inefficiency, confirm that the inefficiency is genuine, and to select, or create a new, causal influencer category. In some embodiments network graphs of influencer relations may be used to infer the most likely, or collection of likely, influencers. In some embodiments, machine learning systems trained to recognize causal influencers from inefficiency precipitating score values may likewise be used.

[0151] At block 1805h, the system may determine whether a candidate inefficiency is in fact a genuine inefficiency (and identify the inefficiency as such at block 1805j) or merely an anomaly (and identify it as such at block 1805i). For example, where no influencers could be found at block 1805g or detected with a high enough certainty or probability, then the system may identify the candidate inefficiency as an anomaly at block 1805j. Again, a machine learning system may be used to distinguish between genuine inefficiencies and anomalies at block 1805h (e.g., a binary classifier, such as support vector machine, or an appropriately configured random forest, neural network, logistic regression classifier, etc.). Consultation with historical data may also be used to distinguish anomalies from genuine inefficiencies. For example, an influencer may be associated with a “stability” value indicative of the influencer’s association with similar inefficiencies in historical data (e.g., via temporal correlation manifested above a threshold). Where a strong stability value is associated with candidate inefficiency and its influencer or influencers, then the system may identify the inefficiency as genuine at block 1805j.

[0152] Some embodiments may employ, e.g., combinations of correlation test, multivariable regression, recursive feature selection and ranking measured by operative time to identify inefficiency influencers. As discussed herein, while anomalies may be thus automatically detected in some embodiments, they may also be detected “qualitatively” by user-initiated review or inspection, albeit possibly initiated based upon an initially computer-determined quantitative measure (e.g., a pre-populated bookmark derived from an OPI value, or application of a machine learning anomaly detector to one or more OPI values, may have alerted the reviewer to a moment in the surgery for confirmation as indicating either inefficient or anomalous behavior). Even when anomalies are automatically detected (e.g., when the historical temporal correlation is too weak for an influencer to be identified), human inspection may be invited to confirm the nature of the anomaly, e.g., that it is, or is not, the result of one-time confounding factors, such as unique patient anatomies, one-time OR team variations, teaching versus nonteaching cases, surgeon experience (e.g., the surgeon’s first surgery in this theater), etc. Such granular assessments may be helpful as some anomalies may be partially associated with chronic inefficiencies, e.g., as when a team is already inefficient in preparing the patient, but is particularly inefficient when presented with a patient having an unusually high body mass index (BMI) for the procedure.

[0153] Once all the candidate inefficiencies have been considered at block 1805e, then at block 1805k the system may provide the genuine inefficiency and anomaly determinations to the user or other computer component, e.g., for representation in the GUI and for manual verification by the user during the playback review. Where influencers are associated with objects in the theater-wide data or particular OPI values, the system may highlight those elements in the review GUI (e.g., interface 605) during playback (e.g., via methods analogous to the equipment display GUI element highlighting discussed herein with respect to FIG. 13), bookmark them as regions of interest (and their corresponding times, as well as identifying appropriate perspective views per the methods disclosed herein), or otherwise bring them to the reviewer’s attention during their review.

[0154]

Computer System

[0155] FIG. 19 is a block diagram of an example computer system as may be used in conjunction with some of the embodiments. The computing system 1900 may include an interconnect 1905, connecting several components, such as, e.g., one or more processors 1910, one or more memory components 1915, one or more input/output systems 1920, one or more storage systems 1925, one or more network adaptors 1930, etc. The interconnect 1905 may be, e.g., one or more bridges, traces, busses (e.g., an ISA, SCSI, PCI, I2C, Firewire bus, etc.), wires, adapters, or controllers.

[0156] The one or more processors 1910 may include, e.g., an Intel™ processor chip, a math coprocessor, a graphics processor, etc. The one or more memory components 1915 may include, e.g., a volatile memory (RAM, SRAM, DRAM, etc.), a non-volatile memory (EPROM, ROM, Flash memory, etc.), or similar devices. The one or more input/output devices 1920 may include, e.g., display devices, keyboards, pointing devices, touchscreen devices, etc. The one or more storage devices 1925 may include, e.g., cloud-based storages, removable Universal Serial Bus (USB) storage, disk drives, etc. In some systems memory components 1915 and storage devices 1925 may be the same components. Network adapters 1930 may include, e.g., wired network interfaces, wireless interfaces, Bluetooth™ adapters, line-of-sight interfaces, etc.

[0157] One will recognize that only some of the components, alternative components, or additional components than those depicted in FIG. 19 may be present in some embodiments. Similarly, the components may be combined or serve dual-purposes in some systems. The components may be implemented using special-purpose hardwired circuitry such as, for example, one or more ASICs, PLDs, FPGAs, etc. Thus, some embodiments may be implemented in, for example, programmable circuitry (e.g., one or more microprocessors) programmed with software and/or firmware, or entirely in special-purpose hardwired (non-programmable) circuitry, or in a combination of such forms.

[0158] In some embodiments, data structures and message structures may be stored or transmitted via a data transmission medium, e.g., a signal on a communications link, via the network adapters 1930. Transmission may occur across a variety of mediums, e.g., the Internet, a local area network, a wide area network, or a point-to-point dial-up connection, etc. Thus, “computer readable media” can include computer-readable storage media (e.g., "non-transitory" computer-readable media) and computer-readable transmission media.

[0159] The one or more memory components 1915 and one or more storage devices 1925 may be computer-readable storage media. In some embodiments, the one or more memory components 1915 or one or more storage devices 1925 may store instructions, which may perform or cause to be performed various of the operations discussed herein. In some embodiments, the instructions stored in memory 1915 can be implemented as software and/or firmware. These instructions may be used to perform operations on the one or more processors 1910 to carry out processes described herein. In some embodiments, such instructions may be provided to the one or more processors 1910 by downloading the instructions from another system, e.g., via network adapter 1930.

[0160] For clarity, one will appreciate that while a computer system may be a single machine, residing at a single location, having one or more of the components of FIG. 19, this need not be the case. For example, distributed network computer systems may comprise multiple individual processing workstations, each workstation having some, or all, of the components depicted in FIG. 19. Processing and various operations described herein may accordingly be spread across the one or more workstations of such a computer system. For example, one will appreciate that a process amenable to being run in a single thread upon a single workstation may instead be separated into an arbitrary number of sub-threads across one or more workstations, such sub-threads then run in serial or in parallel to achieve a same, or substantially similar, result as the process run within the single thread. Similarly, one will appreciate that while a non-transitory computer readable medium may stand alone (e.g., in a single USB storage device), or reside within a single workstation (e.g., in the workstation’s random access memory or disk storage), such a medium need not reside at a single geographic location, but may include, e.g., multiple memory storage units residing across geographically separated workstations of a computer system in network communication with one another or across geographically separated storage devices.

Remarks

[0161] The drawings and description herein are illustrative. Consequently, neither the description nor the drawings should be construed so as to limit the disclosure. For example, titles or subtitles have been provided simply for the reader’s convenience and to facilitate understanding. Thus, the titles or subtitles should not be construed so as to limit the scope of the disclosure, e.g., by grouping features which were presented in a particular order or together simply to facilitate understanding. Unless otherwise defined herein, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In the case of conflict, this document, including any definitions provided herein, will control. A recital of one or more synonyms herein does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any term discussed herein is illustrative only and is not intended to further limit the scope and meaning of the disclosure or of any exemplified term.

[0162] Similarly, despite the particular presentation in the figures herein, one skilled in the art will appreciate that actual data structures used to store information may differ from what is shown. For example, the data structures may be organized in a different manner, may contain more or less information than shown, may be compressed and/or encrypted, etc. The drawings and disclosure may omit common or well-known details in order to avoid confusion. Similarly, the figures may depict a particular series of operations to facilitate understanding, which are simply exemplary of a wider class of such collection of operations. Accordingly, one will readily recognize that additional, alternative, or fewer operations may often be used to achieve the same purpose or effect depicted in some of the flow diagrams. For example, data may be encrypted, though not presented as such in the figures, items may be considered in different looping patterns (“for” loop, “while” loop, etc.), or sorted in a different manner, to achieve the same or similar effect, etc.

[0163] Reference herein to "an embodiment" or "one embodiment" means that at least one embodiment of the disclosure includes a particular feature, structure, or characteristic described in connection with the embodiment. Thus, the phrase "in one embodiment" in various places herein is not necessarily referring to the same embodiment in each of those various places. Separate or alternative embodiments may not be mutually exclusive of other embodiments. One will recognize that various modifications may be made without deviating from the scope of the embodiments.

Claims

CLAIMS We claim:

1 . A computer-implemented method for causing surgical data to be displayed, the method comprising: causing a first graphical user interface (GUI) element to display a visualization sensor output depicting at least a portion of a patient interior, the visualization sensor output captured during a surgical procedure in a surgical theater; and causing a second GUI element to display a field of view rendering of the surgical theater based upon theater-wide data acquired during the surgical procedure.

2. The computer-implemented method of Claim 1 , wherein the method further comprises: causing playback of the visualization sensor output in the first GUI element to be temporally synchronized with playback of the theater-wide data in the second GUI element.

3. The computer-implemented method of Claim 2, wherein the method further comprises: causing a third GUI element to display information derived from non-theater-wide kinematics data acquired during the surgical procedure.

4. The computer-implemented method of Claim 3, wherein the method further comprises: causing a fourth GUI element to display information derived from non-theater- wide system events data, the non-theater-wide system events data acquired during the surgical procedure.

5. The computer-implemented method of one of Claims 1-4, wherein the method further comprises: determining a score metric value based, at least in part, upon one or more of: non-theater-wide kinematics data; non-theater-wide system event data; the theater-wide data; and the visualization sensor output.

6. The computer-implemented method of Claim 5, wherein the method further comprises: determining a time to adjust playback displayed in the first GUI element based upon the score metric value; determining a time to adjust playback of the theater-wide data displayed in the second GUI element based upon the score metric value; and determining a view of the theater-wide data displayed in the second GUI element based upon the score metric value.

7. The computer-implemented method of Claim 5, wherein the method further comprises: determining a stability of an influencer based, at least in part, upon the score metric value.

8. The computer-implemented method of Claim 7, wherein assessing the stability of the influencer based upon the score metric value comprises considering the score metric value when performing at least one or more of: a correlation test; a multivariable regression; a recursive feature selection; and a ranking measured by operative time.

9. The computer-implemented method of Claim 8, wherein the score metric value is one of an objective performance indicators (OPI) score, a skill score, or a task score.

10. The computer-implemented method of Claim 8, the method further comprising determining the existence of an inefficiency associated with the surgical procedure based, at least in part, upon: the stability of the influencer; and a determination that the score metric value is not associated with an anomaly.

11 . The computer-implemented method of one of Claims 1-4, wherein the method further comprises: determining an adjustment of the display of the visualization sensor output in the first GUI element based upon the theater-wide data acquired during the surgical procedure.

12. The computer-implemented method of Claim 11 , wherein determining the adjustment of the display of the visualization sensor output comprises: determining an OPI value based, at least in part, upon the theater-wide data; and determining, based at least in part upon the OPI value, a time in the visualization sensor output playback to perform playback of the visualization sensor output.

13. The computer-implemented method of Claim 11 , wherein determining the adjustment of the display of the visualization sensor output comprises: determining a visibility of an object in the theater-wide data, the visibility associated with at least one temporal interval; and determining, based at least in part upon the temporal interval, a time in the visualization sensor output playback to perform playback of the visualization sensor output.

14. The computer-implemented method of Claim 13, wherein the object is a surgical instrument.

15. The computer-implemented method of Claim 11 , wherein determining the adjustment of the display of the visualization sensor output comprises: causing a temporal offset to be applied to playback in the first GUI element relative to playback in the second GUI element.

16. The computer-implemented method of Claim 11 , wherein determining the adjustment of the display of the visualization sensor output comprises: causing an instrument appearing in the first GUI element to be highlighted in correspondence with a highlight associated with the instrument appearing in the second GUI element.

17. The computer-implemented method of one of Claims 1-4, wherein the method further comprises: determining an adjustment of the display of the theater-wide data in the second GUI element based upon non-theater-wide data acquired in association with the surgical procedure.

18. The computer-implemented method of Claim 17, wherein determining the adjustment of the display of the theater-wide data in the second GUI element comprises: determining an OPI value based, at least in part, upon the non-theater-wide data; and determining, based at least in part upon the OPI value, a time in the non-theater- wide data to perform playback of the non-theater-wide data in the second GUI element.

19. The computer-implemented method of Claim 17, wherein determining the adjustment comprises determining a time to highlight a portion of the theater-wide data in the second GUI element.

20. The computer-implemented method of Claim 17, wherein determining the adjustment comprises: receiving a user selection of a heads-up-display GUI element appearing in the first GUI element; and determining a visibility of an object associated with the heads-up-display GUI element in the theater-wide data; and determining, based at least in part upon the visibility of the object, a time in the playback of the theater-wide data to perform the playback of the theater-wide data in the second GUI element.

21 . The computer-implemented method of Claim 20, wherein the object is a surgical instrument.

22. The computer-implemented method of Claim 17, wherein determining the adjustment of the display of the theater-wide data in the second GUI element comprises; determining, at least in part from the non-theater-wide data, a pose of the visualization sensor associated with the visualization sensor output; and determining a pose of a virtual element, the pose of a virtual element corresponding to the pose of the visualization sensor, the virtual element to be displayed in the second GUI element.

23. A non-transitory computer-readable medium comprising instructions configured to cause a computer system to perform a method, the method comprising: causing a first graphical user interface (GUI) element to display a visualization sensor output depicting at least a portion of a patient interior, the visualization sensor output captured during a surgical procedure in a surgical theater; and causing a second GUI element to display a field of view rendering of the surgical theater based upon theater-wide data acquired during the surgical procedure.

24. The non-transitory computer-readable medium of Claim 23, wherein the method further comprises: causing playback of the visualization sensor output in the first GUI element to be temporally synchronized with playback of the theater-wide data in the second GUI element.

25. The non-transitory computer-readable medium of Claim 24, wherein the method further comprises: causing a third GUI element to display information derived from non-theater-wide kinematics data acquired during the surgical procedure.

26. The non-transitory computer-readable medium of Claim 25, wherein the method further comprises: causing a fourth GUI element to display information derived from non-theater- wide system events data, the non-theater-wide system events data acquired during the surgical procedure.

27. The non-transitory computer-readable medium of one of Claims 23-26, wherein the method further comprises: determining a score metric value based, at least in part, upon one or more of: non-theater-wide kinematics data; non-theater-wide system event data; the theater-wide data; and the visualization sensor output.

28. The non-transitory computer-readable medium of Claims 27, wherein the method further comprises: determining a time to adjust playback displayed in the first GUI element based upon the score metric value; determining a time to adjust playback of the theater-wide data displayed in the second GUI element based upon the score metric value; and determining a view of the theater-wide data displayed in the second GUI element based upon the score metric value.

29. The non-transitory computer-readable medium of Claim 27, wherein the method further comprises: determining a stability of an influencer based, at least in part, upon the score metric value.

30. The non-transitory computer-readable medium of Claim 29, wherein assessing the stability of the influencer based upon the score metric value comprises considering the score metric value when performing at least one or more of: a correlation test; a multivariable regression; a recursive feature selection; and a ranking measured by operative time.

31 . The non-transitory computer-readable medium of Claim 30, wherein the score metric value is one of an objective performance indicators (OPI) score, a skill score, or a task score.

32. The non-transitory computer-readable medium of Claim 30, the method further comprising determining the existence of an inefficiency associated with the surgical procedure based, at least in part, upon: the stability of the influencer; and a determination that the score metric value is not associated with an anomaly.

33. The non-transitory computer-readable medium of one of Claims 23-26, wherein the method further comprises: determining an adjustment of the display of the visualization sensor output in the first GUI element based upon the theater-wide data acquired during the surgical procedure.

34. The non-transitory computer-readable medium of Claim 33, wherein determining the adjustment of the display of the visualization sensor output comprises: determining an OPI value based, at least in part, upon the theater-wide data; and determining, based at least in part upon the OPI value, a time in the visualization sensor output playback to perform playback of the visualization sensor output.

35. The non-transitory computer-readable medium of Claim 33, wherein determining the adjustment of the display of the visualization sensor output comprises: determining a visibility of an object in the theater-wide data, the visibility associated with at least one temporal interval; and determining, based at least in part upon the temporal interval, a time in the visualization sensor output playback to perform playback of the visualization sensor output.

36. The non-transitory computer-readable medium of Claim 35, wherein the object is a surgical instrument.

37. The non-transitory computer-readable medium of Claim 33, wherein determining the adjustment of the display of the visualization sensor output comprises: causing a temporal offset to be applied to playback in the first GUI element relative to playback in the second GUI element.

38. The non-transitory computer-readable medium of Claim 33, wherein determining the adjustment of the display of the visualization sensor output comprises: causing an instrument appearing in the first GUI element to be highlighted in correspondence with a highlight associated with the instrument appearing in the second GUI element.

39. The non-transitory computer-readable medium of one of Claims 23-26, wherein the method further comprises: determining an adjustment of the display of the theater-wide data in the second GUI element based upon non-theater-wide data acquired in association with the surgical procedure.

40. The non-transitory computer-readable medium of Claim 39, wherein determining the adjustment of the display of the theater-wide data in the second GUI element comprises: determining an OPI value based, at least in part, upon the non-theater-wide data; and determining, based at least in part upon the OPI value, a time in the non-theater- wide data to perform playback of the non-theater-wide data in the second GUI element.

41 . The non-transitory computer-readable medium of Claim 39, wherein determining the adjustment comprises determining a time to highlight a portion of the theater-wide data in the second GUI element.

42. The non-transitory computer-readable medium of Claim 39, wherein determining the adjustment comprises: receiving a user selection of a heads-up-display GUI element appearing in the first GUI element; and determining a visibility of an object associated with the heads-up-display GUI element in the theater-wide data; and determining, based at least in part upon the visibility of the object, a time in the playback of the theater-wide data to perform the playback of the theater-wide data in the second GUI element.

43. The non-transitory computer-readable medium of Claim 42, wherein the object is a surgical instrument.

44. The non-transitory computer-readable medium of Claim 39, wherein determining the adjustment of the display of the theater-wide data in the second GUI element comprises; determining, at least in part from the non-theater-wide data, a pose of the visualization sensor associated with the visualization sensor output; and determining a pose of a virtual element, the pose of a virtual element corresponding to the pose of the visualization sensor, the virtual element to be displayed in the second GUI element.

45. A computer system, the computer system comprising: at least one computer processor; at least one computer-readable memory, the at least one computer-readable memory comprising instructions configured to cause the computer system to perform a method, the method comprising: causing a first graphical user interface (GUI) element to display a visualization sensor output depicting at least a portion of a patient interior, the visualization sensor output captured during a surgical procedure in a surgical theater; and causing a second GUI element to display a field of view rendering of the surgical theater based upon theater-wide data acquired during the surgical procedure.

46. The computer system of Claim 45, wherein the method further comprises: causing playback of the visualization sensor output in the first GUI element to be temporally synchronized with playback of the theater-wide data in the second GUI element.

47. The computer system of Claim 46, wherein the method further comprises: causing a third GUI element to display information derived from non-theater-wide kinematics data acquired during the surgical procedure.

48. The computer system of Claim 47, wherein the method further comprises: causing a fourth GUI element to display information derived from non-theater- wide system events data, the non-theater-wide system events data acquired during the surgical procedure.

49. The computer system of one of Claims 45-48, wherein the method further comprises: determining a score metric value based, at least in part, upon one or more of: non-theater-wide kinematics data; non-theater-wide system event data; the theater-wide data; and the visualization sensor output.

50. The computer system of Claim 49, wherein the method further comprises: determining a time to adjust playback displayed in the first GUI element based upon the score metric value; determining a time to adjust playback of the theater-wide data displayed in the second GUI element based upon the score metric value; and determining a view of the theater-wide data displayed in the second GUI element based upon the score metric value.

51 . The computer system of Claim 49, wherein the method further comprises: determining a stability of an influencer based, at least in part, upon the score metric value.

52. The computer system of Claim 51 , wherein assessing the stability of the influencer based upon the score metric value comprises considering the score metric value when performing at least one or more of: a correlation test; a multivariable regression; a recursive feature selection; and a ranking measured by operative time.

53. The computer system of Claim 52, wherein the score metric value is one of an objective performance indicators (OPI) score, a skill score, or a task score.

54. The computer system of Claim 52, the method further comprising determining the existence of an inefficiency associated with the surgical procedure based, at least in part, upon: the stability of the influencer; and a determination that the score metric value is not associated with an anomaly.

55. The computer system of one of Claims 45-48, wherein the method further comprises: determining an adjustment of the display of the visualization sensor output in the first GUI element based upon the theater-wide data acquired during the surgical procedure.

56. The computer system of Claim 55, wherein determining the adjustment of the display of the visualization sensor output comprises: determining an OPI value based, at least in part, upon the theater-wide data; and determining, based at least in part upon the OPI value, a time in the visualization sensor output playback to perform playback of the visualization sensor output.

57. The computer system of Claim 55, wherein determining the adjustment of the display of the visualization sensor output comprises: determining a visibility of an object in the theater-wide data, the visibility associated with at least one temporal interval; and determining, based at least in part upon the temporal interval, a time in the visualization sensor output playback to perform playback of the visualization sensor output.

58. The computer system of Claim 57, wherein the object is a surgical instrument.

59. The computer system of Claim 55, wherein determining the adjustment of the display of the visualization sensor output comprises: causing a temporal offset to be applied to playback in the first GUI element relative to playback in the second GUI element.

60. The computer system of Claim 55, wherein determining the adjustment of the display of the visualization sensor output comprises: causing an instrument appearing in the first GUI element to be highlighted in correspondence with a highlight associated with the instrument appearing in the second GUI element.

61 . The computer system of one of Claims 45-48, wherein the method further comprises: determining an adjustment of the display of the theater-wide data in the second GUI element based upon non-theater-wide data acquired in association with the surgical procedure.

62. The computer system of Claim 61 , wherein determining the adjustment of the display of the theater-wide data in the second GUI element comprises: determining an OPI value based, at least in part, upon the non-theater-wide data; and determining, based at least in part upon the OPI value, a time in the non-theater- wide data to perform playback of the non-theater-wide data in the second GUI element.

63. The computer system of Claim 61 , wherein determining the adjustment comprises determining a time to highlight a portion of the theater-wide data in the second GUI element.

64. The computer system of Claim 61 , wherein determining the adjustment comprises: receiving a user selection of a heads-up-display GUI element appearing in the first GUI element; and determining a visibility of an object associated with the heads-up-display GUI element in the theater-wide data; and determining, based at least in part upon the visibility of the object, a time in the playback of the theater-wide data to perform the playback of the theater-wide data in the second GUI element.

65. The computer system of Claim 64, wherein the object is a surgical instrument.

66. The computer system of Claim 61 , wherein determining the adjustment of the display of the theater-wide data in the second GUI element comprises; determining, at least in part from the non-theater-wide data, a pose of the visualization sensor associated with the visualization sensor output; and determining a pose of a virtual element, the pose of a virtual element corresponding to the pose of the visualization sensor, the virtual element to be displayed in the second GUI element.