Atty. Dkt: 135039-0109 (P06771-WO) SYSTEM AND METHOD FOR DETECTING AND REMOVING NON-SURGICAL DATA BACKGROUND [0001] This application claims the benefit of, and priority to, under 35 U.S.C. § 119, U.S. Provisional Patent Application No.63/511,596, filed June 30, 2023, which is hereby incorporated by reference herein in its entirety. BACKGROUND [0002] Surgical procedures may involve capturing imagery such as video feeds from a variety of viewpoints. For example, in some instances, at least part of the surgical procedure may be performed with a computer-assisted robotic medical system. A medical tool, such as an imaging device like an endoscope, may be used in the robotic medical system to provide imagery. Data sources such as cameras, sensors, etc. may also be located at various viewpoints in the surgical facility to capture and provide imagery of various aspects of the surgical procedure. The captured imagery from the surgical procedure may be processed in various ways. SUMMARY [0003] This disclosure is directed to a system and method for recognizing and scrubbing non-surgical information from a data stream, such as one or more surgical videos. In particular, this disclosure uses multiple data stream components (e.g., multiple video streams) and an event stream to accurately detect and scrub non-surgical information from the data stream. The event stream may determine when a medical tool is installed in a computer- assisted system and whether a visualization tool (e.g., an external visualization tool different from the medical tool or a user interface (“UI”) overlay) receiving data stream components from sources other than the medical tool is connected to the computer-assisted system. When the medical tool is installed and the visualization tool is not connected to the computer-assisted system, the data stream may be considered to include only relevant surgical information that may be useful to preserve. However, when the medical tool is installed in the computer- assisted system and the visualization tool is also connected to the computer-assisted system, the data stream may include both surgical and non-surgical information. The non-surgical information may be accurately identified and scrubbed from the data stream, preserving the relevant surgical information, and removing the irrelevant information that may potentially
Atty. Dkt: 135039-0109 (P06771-WO) contain personal or sensitive data. When the medical tool is not installed in the computer- assisted system, a machine learning model may be used to analyze the data stream and classify each frame in the data stream as in-body or out-of-body. Based on the classification and whether the visualization tool is connected to the computer-assisted system or not, any identified non-surgical information may be removed from the data stream. [0004] At least one aspect is directed to a system having one or more processors to receive a data stream associated with an activity at least partially performed with assistance of a robotic medical system, receive an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system, determine, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is present in the data stream, and modify the data stream in response to determining that the non- surgical information is present in the data stream. [0005] At least one aspect is directed to a non-transitory computer-readable media having computer-readable instructions stored thereon. The computer-readable instructions when executed by one or more processors cause the one or more processors to receive a data stream associated with an activity at least partially performed with assistance of a robotic medical system, receive an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system, determine, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is present in the data stream, and modify the data stream in response to determining that the non- surgical information is present in the data stream. [0006] At least one aspect is directed to a system having one or more processors to receive a data stream associated with an activity at least partially performed with assistance of a robotic medical system, receive an event stream from the robotic medical system indicating whether one or more tools are connected to the robotic medical system, determine, based at least on the data stream and whether the one or more tools are connected, that non-surgical information is absent from the data stream, and store the data stream without modification in response to determining that the non-surgical information is absent excluded from the data stream. [0007] These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations and are incorporated in and constitute a part of this specification. The foregoing information and the
Atty. Dkt: 135039-0109 (P06771-WO) following detailed description and drawings include illustrative examples and should not be considered as limiting. BRIEF DESCRIPTION OF THE DRAWINGS [0008] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements. For purposes of clarity, not every component can be labeled in every drawing. In the drawings: [0009] FIG.1 is an example system showing detection and scrubbing out non-surgical information from a data stream. [0010] FIG.2 is an example block diagram showing detection and scrubbing out non- surgical information from a data stream. [0011] FIG.3 is an example data stream. [0012] FIG.4 is an example flowchart outlining operations for detecting and scrubbing out non-surgical information from a data stream. [0013] FIG.5 is another example flowchart outlining operations for detecting and scrubbing out non-surgical information from a data stream. [0014] FIG.6 is a block diagram showing an example surgical environment. [0015] FIG.7 is an example computer system. DETAILED DESCRIPTION [0016] Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems of detection and removal of surgically irrelevant data streams. The various concepts introduced above and discussed in greater detail below can be implemented in any of numerous ways. [0017] In many applications of surgical data science, removal of non-surgical portions of a surgical data stream may be desirable to comply with various regulatory rules, laws, or practices. One approach for removing non-surgical portions of a surgical data stream may involve manually reviewing the data stream and scrubbing out all portions of the data stream that potentially contain non-surgical information. However, manually reviewing and scrubbing the data stream is time consuming and ineffective. Another approach for removing non- surgical portions of a surgical data stream may involve automatically scrubbing out all portions of a data stream that have even slightest potential of showing any non-surgical information. This approach often over-scrubs the data stream, removing relevant surgical portions of the data stream as well, thereby reducing the value of the data stream. This approach is further
Atty. Dkt: 135039-0109 (P06771-WO) limited in that it relies on a single data stream and lacks contextual information, ultimately leading to over-scrubbing. [0018] The present disclosure provides a systematic solution to accurately recognize and scrub non-surgical information from a data stream, while preserving relevant surgical portions of the data stream. During a surgical procedure, imagery may be captured from a plurality of viewpoints. For example, a medical tool such as an imaging device may be installed in a computer-assisted system (e.g., a computer-assisted robotic medical system) such that the medical tool may provide imagery of an interior region of a patient’s body during the surgical procedure. Image capture devices may also be located at various angles in a surgical facility where the surgical procedure is being performed to capture imagery. In some instances, a medical tool (e.g., an endoscope) may be used laparoscopically (e.g., by hand instead of through the robotic medical system) to provide imagery during the surgical procedure. Thus, imagery captured from multiple data sources during a surgical procedure may be available. The present disclosure takes advantage of the multiple sources of imagery in accurately detecting and scrubbing non-surgical information. [0019] Further, the robotic medical system that may be used for at least part of the surgical procedure may provide an event stream indicative of a state of the robotic medical system. For example, a state of the robotic medical system may indicate whether a medical tool is installed in the robotic medical system. A state of the robotic medical system may also indicate whether a visualization tool is connected to the robotic medical system. When the visualization tool is connected to the robotic medical system, imagery from data sources other than the robotic medical system may also be available. Based on the captured imagery and the event stream, the present disclosure provides a system and method to accurately detect and scrub non- surgical information from the imagery while preserving the relevant surgical portions of the imagery. By using multiple sources of imagery and the event stream to detect and scrub non- surgical information, the present disclosure provides a low over-scrubbing rate such that relevant portions of the imagery are preserved. [0020] Specifically, the present disclosure uses the event stream to determine when a medical tool (e.g., an endoscope) is installed in the robotic medical system. The medical tool installed in the robotic medical system may provide imagery related to an interior region of a patient’s body (e.g., in-body). Thus, the imagery provided by the medical tool installed in the robotic medical system may be considered relevant surgical information that may be desired to be preserved. If the visualization tool is not connected to the robotic medical system, indicating that imagery from other data sources is not available, the system and method of the
Atty. Dkt: 135039-0109 (P06771-WO) present disclosure may save the imagery from the medical tool installed in the robotic medical system without any scrubbing. On the other hand, if the visualization tool is connected to the robotic medical system, the captured imagery may include images from the interior region of a patient’s body as provided by the medical tool installed in the robotic medical system and images captured from other viewpoints of the surgical facility as provided by the visualization tool. The imagery from sources other than the medical tool installed in the robotic medical system may potentially include at least some imagery of non-surgical information, which may be desired to be scrubbed. In such instances, the system and method of the present disclosure may identify a subset of the imagery that contains non-surgical information and selectively scrubs out that subset, thereby retaining only the relevant surgical information. In some instances, if the medical tool is used laparoscopically, the imagery may or may not be modified depending on whether the visualization tool is connected to the robotic medical system and whether the imagery includes in-body data or out-of-body data (e.g., data from an exterior of a patient’s body, including patient’s surroundings). [0021] The present disclosure provides several advantages. For example, by combining imagery from multiple sources and an event stream, the present disclosure increases the accuracy of detecting and scrubbing non-surgical information, while preserving relevant surgical portions of the imagery. The present disclosure provides a mechanism to recognize that imagery captured from a medical tool installed within a robotic medical system may contain more relevant information and minimize the amount of information that is scrubbed from such images. The present disclosure may also determine when the medical tool is used laparoscopically and may be indicative of what a surgeon performing the surgical procedure is trying to do. The present disclosure may be used to highlight portions of the imagery where the medical tool is used laparoscopically for future reference. The present disclosure may determine and provide an amount of time that a medical tool (e.g., an endoscope) and/or a visualization tool is either installed in/connected to or uninstalled/disconnected from the robotic medical system. Knowledge of the amount of time that the medical tool/visualization tool is installed/connected or uninstalled/disconnected may provide an indication of the efficiency of the surgical procedure and/or used as a metric of performance for medical personnel (e.g., surgeon). [0022] The present disclosure may also generate confidence scores indicative of a confidence level that the imagery does not contain non-surgical information. The present disclosure may also provide real-time feedback to a surgeon on what portions of the imagery is being scrubbed out. For example, the present disclosure may be able to provide feedback to
Atty. Dkt: 135039-0109 (P06771-WO) surgeon that the imagery includes images from the medical tool when installed within the robotic medical system, which are preserved for later reference. The present disclosure provides options for both real-time feedback and post-processing options, as well as edge based and cloud-based implementations to provide a flexible and efficient system. For example, the present disclosure may provide a mechanism to capture the imagery using one or more edge- based devices and scrubbing of the non-surgical information using one or more cloud-based devices. In some examples, the imagery may be captured using one or more cloud-based devices along with, or instead of, the one or more edge-base devices. In some examples, the scrubbing of the non-surgical information may be performed using one or more edge-based devices along with, or instead of, the one or more cloud-based devices. Thus, the technology can include or allow for a hybrid deployment that can leverage both cloud-based processing and processing on edge-based devices. [0023] Although the present disclosure is discussed in the context of a surgical procedure, in some embodiments, the present disclosure may be applicable to other medical sessions or environments or activities, as well as non-medical activities where removal of irrelevant information is desired. [0024] The imagery that is captured from a surgical procedure may be in the form of a data stream. A data stream may be considered any sequence of digital encoded data and/or analog data captured from a data source. Each unit of transmission of a data stream may be considered a frame. The data stream may be considered to include a sequence of elements, and each element may be considered a frame. [0025] The non-surgical information as used herein may include Protected Health Information (PHI). PHI may include any information, such as private information, that identifies or can be used to identify an individual. PHI may include any type of identifying information related to a patient such as textual, graphical, tabular, visual, or other types of information indicative of medical information, demographic information, mental state information, physical features of the patient, etc. Non-surgical information may generally include information pertaining to out-of-body regions of a patient and out-of-body surroundings of the patient (e.g., information pertaining to the surgical facility. Medical personnel, etc.). Information pertaining to the in-body regions of a patient may be considered surgical information and may be desired to be preserved. [0026] Referring now to FIG.1, an example system 100 is shown, in accordance with some embodiments. The system 100 may be associated with a medical environment 105. The medical environment 105 may be a medical surgical environment. A medical surgical
Atty. Dkt: 135039-0109 (P06771-WO) environment may include a surgical facility such as an operating room in which a surgical procedure, whether, invasive, non-invasive, in-patient, or out-patient, may be performed on a patient. In other embodiments, the system 100 may be associated with other types of medical sessions or activities, or non-medical environments that may require removal of non-surgical information from a data stream captured from that environment. The system 100 may include one or more image capture devices 110A-110B (collectively referred to herein as image capture devices 110). Although two image capture devices are shown in FIG.1, in some embodiments, greater than or fewer than two image capture devices may be provided within the medical environment 105. Each of the image capture devices 110 may be configured to capture images from a particular viewpoint within the medical environment 105. Thus, each of the image capture devices 110 may be positioned, mounted, or otherwise located based on content that is desired to be captured from a particular viewpoint of an image capture device. [0027] For example, in some embodiments, at least one of the image capture devices 110 may be positioned to capture one or more images of an area where a patient is located within the medical environment 105. In some embodiments, at least one of the image capture devices 110 may be positioned to capture one or more images of an area where one or more medical professionals are located within the medical environment 105. In some other embodiments, at least one of the image capture devices 110 may be configured to capture one or more images of other designated areas within the medical environment 105. The image capture devices 110 may include any of a variety of sensors, cameras, video imaging devices, infrared imaging devices, visible light imaging devices, intensity imaging devices (e.g., black, color, grayscale imaging devices, etc.), depth imaging devices (e.g., stereoscopic imaging devices, time-of- flight imaging devices, etc.), medical imaging devices such as endoscopic imaging devices, ultrasound imaging devices, etc., non-visible light imaging devices, any combination or sub- combination of the above mentioned imaging devices, or any other type of imaging devices that may be suitable for the purposes described herein. [0028] The images that are captures by the image capture device 110 may include still images, video images, vector images, bitmap images, other types of images, or combinations thereof. In some embodiments, one or more of the image capture devices 110 may be configured to capture other parameters such as sound, motion, pressure, temperature, etc. within the medical environment 105 as well. The image capture devices 110 may capture the images at any suitable predetermined capture rate and/or frequency. Other settings, such as zoom settings, etc. of each of the image capture devices 110 may vary as desired to capture suitable images from a particular viewpoint. In some embodiments, one or more of the image
Atty. Dkt: 135039-0109 (P06771-WO) capture devices 110 may have fixed locations, positions, and/or orientations. In other embodiments, one or more of the image capture devices 110 may be portable, or otherwise configured to change orientation or telescope in various directions. In some embodiments, one or more of the image capture devices 110 may be part of a multi-sensor architecture including multiple sensors, with each sensor being configured to detect, measure, or otherwise capture a particular parameter (e.g., sound, images, pressure, etc.). [0029] The images captured by the image capture devices 110 may be sent as a data stream component to a visualization tool 115. The visualization tool 115 can include, refer to, or provide a UI overlay. A data stream component may be considered any sequence of digital encoded data or analog data from a data source such as the image capture devices 110. The visualization tool 115 may be configured to receive a plurality of data stream components and combine the plurality of data stream components into a single data stream. The visualization tool 115 may be configured to receive up to five different data stream components (e.g., one from each image capture device). The visualization tool 115 can include a user interface that receives, via an user input device, various types of information, including, for example, PHI. Although a single visualization tool (e.g., the visualization tool 115) is shown in FIG.1, multiple visualization tools or UI overlays may be present in the system 100, with each visualization tool configured to receive one or more data stream components. In addition to receiving the data stream components from the image capture devices 110, the visualization tool 115 may also receive a data stream component from a medical tool 120. The medical tool 120 may be any type and form of tool used for surgery, medical procedures or a tool in an operating room or environment associated with or having an image capture device. In some embodiments, the medical tool 120 may be an endoscope or endoscopic camera for visualizing organs, tissues, etc. within a body of the patient (e.g., inserted into the body of the patient). In some embodiments, the medical tool 120 may include other or additional types of therapeutic and/or diagnostic medical imaging implements. The medical tool 120 may be configured to be installed in a robotic medical system 125. Installing the medical tool 120 in the robotic medical system 125 may include or be associated with or be part of connecting the medical tool to the robotic medical system. The medical tool 120 may be separate from the visualization tool 115. The visualization tool 115 may be connected to a port (e.g., input port) of the robotic medical system 125 to receive data from the medical tool 120, while the medical tool may be connected to a manipulator arm of the robotic medical system to assist with the surgery (e.g., be inserted into the body of the patient and capture images from within the body of the patient). The visualization tool 120 is not intended to be inserted into a body of the patient. The connection
Atty. Dkt: 135039-0109 (P06771-WO) of the visualization tool 120 to the robotic medical system 125 may include, or be associated with or be part of, any installation that may be needed. [0030] The robotic medical system 125 may be a computer-assisted system configured to perform a surgical or medical procedure or activity on a patient via or using or with the assistance of one or more robotic components and/or medical tools. As described in more detail below, the robotic medical system 125 may include one or more manipulator arms that perform one or more computer-assisted medical tasks. The medical tool 120 may be installed on a manipulator arm of the robotic medical system 125 to perform a surgical task. The images (e.g., video images) captured by the medical tool 125 may be sent to the visualization tool 115. The robotic medical system 125 may also include one or more input ports to receive direct or indirect connection of one or more auxiliary devices. For example, in some embodiments, the visualization tool 115 may be connected to the robotic medical system 125 to receive the images from the medical tool when the medical tool is installed in the robotic medical system (e.g., on a manipulator arm of the robotic medical system). The visualization tool 115 may combine the data stream components from the image capture devices 110 and the medical tool 120 into a single combined data stream for presenting on a display 130. In some embodiments, the display 130 may be associated with a user control system or other type of display system, whether within the medical environment 105 or remote, to view the single combined data stream. [0031] The system 100 also includes a data processing system 135 associated with the medical environment 105. The data processing system 135 may include a recording device 140 configured to access the single combined data stream from the visualization tool 115 and/or the display 130. The single combined data stream received by the recording device 140 is referred to as data stream 145. The recording device 140 is also configured to receive an event stream 150 from the robotic medical system 125. The event stream 150 may include a stream of event data or information, such as packets, that identify or convey a state of the robotic medical system 125 or an event that occurred in association with the robotic medical system or surgical or medical surgery being performed with the robotic medical system. An example state of the robotic medical system 125 may indicate whether the medical tool 120 is installed on a manipulator arm of the robotic medical system or not. For example, when the medical tool 120 is installed on a manipulator arm of the robotic medical system 125, a signal or data packet(s) may be generated indicating that the medical tool has been installed on the manipulator arm of the robotic medical system 125. The signal or data packet(s) may be sent to the recording device 140 as the event stream 150. Another example state of the robotic medical system 125
Atty. Dkt: 135039-0109 (P06771-WO) may indicate whether the visualization tool 115 is connected, whether directly to the robotic medical system or indirectly through another auxiliary system that is connected to the robotic medical system. [0032] When the visualization tool 115 is directly plugged into the robotic medical system 125 or plugged into another auxiliary system that is plugged into a port of the robotic medical system, the robotic medical system may generate a signal or data packet(s) that may be sent to the recording device 140 as the event stream 150. Yet another state of the robotic medical system 125 may indicate whether another auxiliary system is connected to (e.g., plugged into) any of the ports of the robotic medical system. The robotic medical system 125 may have other states, which may be detected by the recording device 140. In some embodiments, the recording device 140 may be configured to determine (e.g., record) or otherwise receive the event stream 150 through an Application Programming Interface (API) of the robotic medical system 125. In other embodiments, the recording device 140 may determine or otherwise receive the event stream 150 via other suitable mechanisms. In some embodiments, the recording device 140 may poll the robotic medical system 125 to determine the state of the robotic medical system. [0033] The recording device 140 may receive the data stream 145 and the event stream 150. Based on the data stream 145 and the event stream 150, the recording device 140 detects if the data stream includes any non-surgical information (e.g., PHI). In some embodiments, the recording device 140 may include a detection processor 155 configured to detect non-surgical information within the data stream 145. In some embodiments, the non-surgical information may include, in addition to that described above, any out-of-body information of the patient, including for example, any features or identifiers that may be indicative of an identity of the patient or any features that depict, or may be visible from, an exterior of the body of a patient in a non-surgical state. In contrast, in-body information may be indicative of an interior of the body of the patient and may not be considered non-surgical information. [0034] The recording device 140 may also include a data stream modification processor 160 that may be configured to modify the data stream 145 in response to the detection of non- surgical information by the detection processor 155. Modification of the data stream 145 may include blurring non-surgical information, redacting or blocking non-surgical information, deleting or removing non-surgical information, overwriting non-surgical information, or otherwise scrubbing out the non-surgical information using other ways. The data stream modification processor 160 may store the data stream 145 that has been modified within a database 165. If the detection processor 155 does not detect any non-surgical information in
Atty. Dkt: 135039-0109 (P06771-WO) the data stream 145, the detection processor may store the data stream without any modification within the database 165. The database 165 may include, or be associated with, a local storage attached to the recording device 140 and/or the data processing system 135, a network attached storage such as a cloud storage associated with the recording device and/or the data processing system, or a combination of a local attached storage and a network attached storage. The database 165 may store the data stream 145, with and/or without modification to allow later retrieval, access, viewing, and/or processing of the data stream. [0035] In some embodiments, the recording device 140 may highlight, identify, and/or annotate portions of the data stream 145 before storing in the database 165. For example, in some embodiments, the recording device 140 may highlight the portions of the data stream 145 that are received from the medical tool 120 when installed in the robotic medical system 125. In some embodiments, the recording device 140 may add a notation or indication in real-time on the data stream 145 on portions of the data stream 145 that are from the medical tool 120 and/or from the image capture devices 110. In some embodiments, the recording device 140 may also be configured to send alerts, for example, when the event stream determines that the medical tool 120 is installed in the robotic medical system 125, when the visualization tool 115 is connected to the robotic medical system, when the data stream is modified, etc. [0036] To detect non-surgical information and/or to modify the data stream 145, the recording device 140 may be associated with one or more machine learning models 170. For example, the machine learning models 170 may include a non-surgical information detection model 175 and an in-body/out-of-body detection model 180. The machine learning models 170 may include other or additional type of machine learning models that may be suitable for performing the functions described herein. The non-surgical information detection model 175 may be a deep learning neural network or other computer-vision model trained to receive the data steam 145 as an input and identify presence of non-surgical information within the data stream. For example, the non-surgical information detection model 175 may identify specific frames or locations of the data stream 145 that include non-surgical information, such as PHI. The non-surgical information detection model 175 may also be configured to detect, classify, and/or categorize specific types of the out-of-body non-surgical information and/or PHI within the data stream 145. For example, the non-surgical information detection model 175 may determine whether the non-surgical information is related to physical features of the patient, such as facial features, demographic information, medical information, etc., surroundings of the patient, etc.
Atty. Dkt: 135039-0109 (P06771-WO) [0037] The in-body/out-of-body detection model 180 may be a deep learning neural network or computer-vision model trained to receive the data stream 145 as an input and classify/categorize each frame of the data stream as either in-body frame or an out-of-body frame. Although shown as separate machine learning models, in some embodiments, the non- surgical information detection model 175 and the in-body/out-of-body detection model 180 may be configured as a single machine learning model that may perform the functions of both models. Further, although shown separate from the recording device 140, in some embodiments, the machine learning models 170 may be part of the recording device. In some embodiments, the machine learning models 170 may be located in a remote location (e.g., a cloud location) and accessible to the recording device 140. [0038] The data processing system 135, the recording device 140, and/or components thereof may include a physical or virtual computer system operatively coupled, or associated with, the medical environment 105. In some embodiments, the data processing system 135, the recording device 140, and/or components thereof may be coupled, or associated with, the medical environment 105 via a network, either directly or directly through an intermediate computing device or system. The network may be any type or form of network. The geographical scope of the network may vary widely and may include a body area network (BAN), a personal area network (PAN), a local-area network (LAN) (e.g., Intranet), a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network may assume any form such as point-to-point, bus, star, ring, mesh, tree, etc. The network may utilize different techniques and layers or stacks of protocols, including, for example, the Ethernet protocol, the internet protocol suite (TCP/IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (Synchronous Optical Networking) protocol, the SDH (Synchronous Digital Hierarchy) protocol, etc. The TCP/IP internet protocol suite may include application layer, transport layer, internet layer (including, e.g., IPv6), or the link layer. The network may be a type of a broadcast network, a telecommunications network, a data communication network, a computer network, a Bluetooth network, or other types of wired and wireless networks. [0039] The data processing system 135, the recording device 140, and/or components thereof, may be located at least partially at the location of the surgical facility associated with the medical environment 105 or remotely therefrom. At least some elements of the data processing system 102, the recording device 140, and/or components thereof may be accessible via portable devices such as laptops, mobile devices, wearable smart devices, etc. The data processing system 135, the recording device 140, and/or components thereof may include other
Atty. Dkt: 135039-0109 (P06771-WO) or additional elements that may be considered desirable to have in performing the functions described herein. The data processing system 135, the recording device 140, and/or components thereof, may include, or be associated with, a system processor 185. [0040] The system processor 185 may execute one or more instructions associated with the system 100. The system processor 185 may include an electronic processor, an integrated circuit, or the like including one or more of digital logic, analog logic, digital sensors, analog sensors, communication buses, volatile memory, nonvolatile memory, and the like. The system processor 185 may include, but is not limited to, at least one microcontroller unit (MCU), microprocessor unit (MPU), central processing unit (CPU), graphics processing unit (GPU), physics processing unit (PPU), embedded controller (EC), or the like. The system processor 185 may include, or be associated with, a memory 190 operable to store or storing one or more non-transitory computer-readable instructions for operating components of the system processor and operating components operably coupled to the system processor. The one or more instructions may include at least one of firmware, software, hardware, operating systems, embedded operating systems, and the like. The system processor 185 or the system 100 generally may include at least one communication bus controller to effect communication between the system processor and the other elements of the system 100. [0041] The memory 190 may include one or more hardware memory devices to store binary data, digital data, or the like. The memory 190 may include one or more electrical components, electronic components, programmable electronic components, reprogrammable electronic components, integrated circuits, semiconductor devices, flip flops, arithmetic units, or the like. The memory 190 may include at least one of a non-volatile memory device, a solid-state memory device, a flash memory device, a NAND memory device, a volatile memory device, etc. The memory 190 may include one or more addressable memory regions disposed on one or more physical memory arrays. [0042] One or more aspects, components or functionality performed by the data processing system 135 can leverage cloud-based services or edge-based devices. The data processing system 135 can allow for edge deployment, cloud deployment or a hybrid deployment. The data processing system 135 can allow for real-time and postprocessing. For example, the technology can provide both real-time feedback and post-processing options, as well as edge based and cloud-based implementations to provide a flexible and efficient system. For example, the technology can capture the imagery using one or more edge-based devices, and then scrub the non-surgical information using one or more cloud-based devices or servers. In some examples, the imagery may be captured using one or more cloud-based devices along
Atty. Dkt: 135039-0109 (P06771-WO) with, or instead of, the one or more edge-base devices. In some examples, the scrubbing of the non-surgical information may be performed using one or more edge-based devices along with, or instead of, the one or more cloud-based devices. [0043] Turning to FIG.2, an example block diagram showing a system 200 for detecting and scrubbing non-surgical information in a data stream is shown, in accordance with some embodiments. The recording device 140 receives the data stream 145 and the event stream 150. Based on the event stream 150, the recording device 140, and particularly, the detection processor 155, detects presence of non-surgical information in the data stream 145. For example, based on the event stream 150, the recording device 140 may determine whether the medical tool 120 (e.g., endoscope) is installed in the robotic medical system 125. Based on the event stream 150, the recording device 140 may also determine whether the visualization tool 115 is connected (either directly or indirectly) to the robotic medical system 125. Based on the installation of the medical tool 120 within the robotic medical system 125 and the connection of the visualization tool 115 to the robotic medical system, the recording device 140 may detects the presence of non-surgical information in the data stream. [0044] For example, based on the event stream 150, the recording device 140 may determine that the medical tool 120 is installed in the robotic medical system 125 and the visualization tool 115 is not connected to the robotic medical system, as indicated by path 205. When the visualization tool 115 is not connected to the robotic medical system 125 and the medical tool 120 is installed within the robotic medical system, the recording device 140 may determine that the data stream 145 only includes images from the medical tool and that the data stream includes no images from the image capture devices 110. Because the data stream 145 only includes images from the medical tool 120 and the medical tool upon installation to the robotic medical system 125 only provides in-body images, the recording device 140 may infer that the data stream only includes in-body images. Further, because the in-body images are considered to contain no non-surgical information, the recording device 140 may determine that no modification of the data stream 145 is needed. Thus, the recording device 140 may store the data stream 145 without modification within the database 165. [0045] As another example, the recording device 140 may determine that the medical tool 120 is installed within the robotic medical system 125 and that the visualization tool 115 is also connected to the robotic medical system, as indicated via path 210. Thus, the recording device 140 may determine that because the medical tool 120 is installed within the robotic medical system 125, data stream includes images received from the medical tool. Further, because the visualization tool 115 is also connected to the robotic medical system 125, the recording device
Atty. Dkt: 135039-0109 (P06771-WO) 140 may determine that the data stream 145 includes images from the image capture devices 110. Thus, the recording device 140 may determine that the data stream 145 may include both in-body images (e.g., received from the medical tool 120) and out-of-body images (e.g., received from the image capture devices 110 depending on the viewpoint from which the images are captured). As a result, the recording device 140 may localize the visualization tool 115, as indicated by box 215. [0046] As indicated above, the data stream 145 is a combined data stream (e.g., combined video stream) including data stream components from multiple data sources (e.g., the image capture devices 110). For example, the images from the image capture device 110A may be a first data stream component, the images from the image capture device 110B may be a second data stream component, the images from the medical tool 120 may be a third data stream component, and so on. The visualization tool 115 may display each data stream component on a different portion or region of the display 130. For example, in some embodiments, the first data stream component may be displayed on a first portion (e.g., top right) of the display 130, the second data stream component may be displayed on a second portion (e.g., bottom right) of the display, and so on. The various portions on the display 130 may or may not overlap. In some embodiments, the data stream component from each data source may have a dedicated portion of the display 130 where that data stream component may be displayed. In other embodiments, each data stream component may be displayed in any portion of the display 130. The size of each data stream component on the display 130 may vary depending upon the embodiment. In some embodiments, one or more data stream components may have textual, graphical, or other types of information associated therewith that may be displayed along with the data stream component. In some embodiments, the visualization tool 115 may also present on the display 130 a user interface menu bar showing various options, tabs, widgets, etc. that a user may interact with to receive a variety of information and/or to activate one or more portions of the display. An example of the display 130 showing the data stream 145 is shown in FIG.3. [0047] Referring to FIG.3 in conjunction with FIG.2, FIG.3 shows an example 300 of the display 130 showing an example recorded data stream (e.g., the data stream 145) having a first data stream component 305 (e.g., first video stream) and a second data stream component 310 (e.g., second video stream). The first data stream component 305 may be displayed around a top portion of the display 130 and the second data stream component 310 may be displayed around a bottom portion of the display, although in other embodiments, the first and second data stream components may be displayed in other portions of the display. The first data
Atty. Dkt: 135039-0109 (P06771-WO) stream component 305 contains an endoscopic view from inside the body of a patient and captured by the medical tool 120. Because the first data stream component 305 provides relevant surgical context, and only provides in-body data, the first data stream component may be considered to have no non-surgical information and may be desired to be preserved without modification. The second data stream component 310 may provide images captured by one of the image capture devices 110, such as, an ultrasound device. The second data stream component 310 may contain both in-body and out-of-body images, including additional information such as text, and therefore, is a high risk of including non-surgical information. [0048] By localizing the visualization tool 115, the recording device 140 may identify the portion of the data stream 145 that includes the second data stream component 310. For example, the recording device 140 may identify the region of the display 130 where the second data stream component 310 is displayed. By identifying the region of the display 130 where the second stream component is displayed, the recording device 140 may identify the portion of the data stream 145 that includes the second data stream component 310, and therefore, the non-surgical information. If the visualization tool 115 is receiving a third data stream component from another image capture device, the third data stream component may be displayed in yet another region of the display 130. When localizing the visualization tool 115, the recording device may also identify the region where the third data stream component is being displayed. Thus, when localizing the visualization tool 115, the recording device 140 may identify each region of the display 130 where a data stream component from the image capture devices 110 is being displayed. In some embodiments, the recording device 140 may be made aware of where the data stream component of a particular image capture device is displayed based on information received from the visualization tool or a user. In some embodiments, the recording device 140 may determine which region a particular data stream component is displayed based on a parameter or property of the particular data stream component. In some embodiments, the recording device 140 may use a machine learning model to identify which region a particular data stream component is displayed. In yet other embodiments, the recording device 140 may use a combination of the mechanisms described above or another mechanism to localize the visualization tool 115. [0049] In addition to the first data stream component 305 and the second data stream component 310, the display 130 may present a user interface menu 315. As shown in FIG.3, the user interface menu 315 includes four tabs, with one tab associated with the first data stream component 305, one tab associated with the second data stream component, one tab providing various settings and options, and one tab that is unassigned (but could be assigned,
Atty. Dkt: 135039-0109 (P06771-WO) for example, to a third data stream component when available). In other embodiments, greater than or less than four tabs may be provided on the user interface menu 315. The user interface menu 315 may also provide information about the tools (e.g., information related to the medical tool 120, the visualization tool 115, etc.) that are installed in and/or connected to the robotic medical system 125. The user interface menu 315 may also display, for example, in real-time, information related to the capture of the data stream 145 (e.g., the data sources), the scrubbing of the data stream (e.g., whether the data stream is being scrubbed, what information is being scrubbed, etc.), and/or any other information that may be considered useful to provide to a user. The location of the user interface menu 315 may also vary on the display 130 from one embodiment to another. [0050] In response to localizing the visualization tool 115, the recording device 140 may modify the data stream 145, as indicated in box 220 in FIG.2. For example, in some embodiments, to modify the data stream 145, the recording device 140 may scrub out the data stream components that originate from the image capture devices 110. For example, with respect to the example 300, the recording device 140 may scrub out the second data stream component 310, including the images and the textual information included in the second data stream component. To scrub out the second data stream component 310, the recording device 140 may blur the second data stream component, redact the second data stream component (e.g., by blocking or covering up the second data stream component), delete or remove the second data stream component, etc. In particular, the recording device 140 may scrub out the portion of the data stream 145 displayed in the region that has been localized. For example, if the localization of the visualization tool 115 determines that the portion of the data stream that is coming from the image capture devices 110 is displayed in the bottom left of the display 130, the bottom left portion of the data stream 145 may be scrubbed out as having non-surgical information. In some embodiments, the recording device 140 may use the non-surgical information detection model 175 to identify specific portions of the second data stream component 310 to scrub out only those portions of the second data stream component that the non-surgical information detection model detects as containing non-surgical information. For example, the non-surgical information detection model 175 may classify portions of the data stream 145 as surgically relevant or irrelevant. For example, by classifying the data stream components as surgically relevant or irrelevant, the recording device 140 may preserve at least some portion of the second data stream component 310 and fully preserve the first data stream component 305. Thus, the recording device 140 may preserve the first data stream component 305 and either conceal the entire second data stream component 310 or conceal the portions of
Atty. Dkt: 135039-0109 (P06771-WO) the second data stream component that contain non-surgical information. The data stream 145 that has been so modified may be saved within the database 165. [0051] As another example, the recording device 140 may determine that the medical tool 120 is not installed within the robotic medical system 125, as indicated by path 225. When the medical tool 120 is not installed within the robotic medical system 125, in some embodiments, the medical tool 120 may still be used, for example, laparoscopically. When used laparoscopically, an incision may be made on a patient and the medical tool 120 (e.g., an endoscope) may be inserted through the incision manually. Thus, even when not installed within the robotic medical system 125, the medical tool 120 may still provide a data stream component. However, because the medical tool 120 is used laparoscopically, the data stream 145 from the medical tool may include both in-body images and out-of-body images. Thus, the recording device 140 may use the in-body/out-of-body detection model 180 to classify each frame of the data stream 145 as either an in-body frame or an out-of-body frame, as indicated by box 230. In some embodiments, a frame of the data stream 145 may include both in-body information and out-of-body information. In such cases, the in-body/out-of-body detection model 180 may be configured to classify such frames as out-of-body frames. [0052] In response to classifying each frame of the data stream 145 as in-body or out-of- body, the recording device 140, and particularly, the detection processor 155, may remove the out-of-body frames from the data stream 145 or remove only portions of the out-of-body frames that contain potential non-surgical information, as indicated by box 235. Responsive to removing the out-of-body frames from the data stream 145, the data stream may contain only in-body frames, which may be stored in the database 165. In some embodiments, in response to removing the out-of-body frames and/or for each in-body frame identified by the in- body/out-of-body detection model 180, the recording device 140 may determine, from the event stream 150, whether the visualization tool 115 is connected to the robotic medical system 125 or not. If the recording device 140 determines that the visualization tool 115 is not installed, as indicated by path 240, the recording device may infer that the data stream 145 only includes in-body frames where the risk of containing non-surgical information is low. Thus, the data stream 145 containing only in-body frames is stored within the database 165 without modification (or further modification after removal of the out-of-body frames or removal of non-surgical information from the out-of-body frames). On the other hand, if the recording device 140 determines that the visualization tool is connected to the robotic medical system 125, as indicated by path 245, the recording device 140 may localize the visualization tool 115,
Atty. Dkt: 135039-0109 (P06771-WO) as indicated by the box 215, modify the data stream 145 as indicated by the box 220 before storing the modified data stream within the database 165. [0053] Turning now to FIG.4, an example flowchart outlining the operations of a process 400 is shown, in accordance with some embodiments of the present disclosure. The process 400 may be performed by a system having one or more processors executing computer- readable instructions stored on a memory. The process 400 may be performed by the recording device 140, and particularly, by one or more processors of the recording device that execute non-transitory computer-readable instructions stored on a memory (e.g., the memory 190). Although the process 400 is described below as being performed by the recording device 140, in other embodiments, the process may be performed by the system having the one or more processors. [0054] The process 400 may be used to detect and scrub non-surgical information (e.g., PHI) in a data stream before storing the data stream to the database 165. Thus, at operation 405, the recording device 140 receives a data stream (e.g., the data stream 145). The data stream may include one or more data stream components from one or more of the image capture devices 110 and/or the medical tool 120. At operation 410, the recording device 140 receives an event stream (e.g., the event stream 150). The recording device 140 may use the event stream to determine a state of the robotic medical system 125. For example, the recording device 140 may use the event stream 150 to determine whether the medical tool 150 is installed within the robotic medical system 125 and whether the visualization tool 115 is connected (directly or indirectly) to the robotic medical system. Although the operation 405 is described as occurring before the operation 410, in some embodiments, the operations 405 and 410 may occur simultaneously or the operation 410 may occur before the operation 405. [0055] At operation 415, the recording device 415 detects whether the data stream (e.g., the data stream 145) contains non-surgical information. In some embodiments, the recording device 415 may determine a likelihood of presence of non-surgical information in the data stream based on the event stream and the data stream. For example, if the recording device 140 determines that the medical tool 120 is installed within the robotic medical system 125 and the visualization tool 115 is not connected to the robotic medical system, the recording device may determine that the data stream is highly likely to include only in-body frames, indicating a low likelihood of non-surgical information. Likewise, if the recording device 140 determines that the medical tool 120 is installed within the robotic medical system 125 and the visualization tool 115 is also connected to the robotic medical system, the recording device may determine that the data stream may include out-of-body frames from data stream components received
Atty. Dkt: 135039-0109 (P06771-WO) from the image capture devices 110 (e.g., from data sources other than the medical tool 120). On the other hand, if the recording device 140 determines that the medical tool 120 is not installed within the robotic medical system, the recording device may classify each frame of the data stream as an out-of-body frame or in-body frame to detect non-surgical information. For each in-body frame, the recording device 140 may determine whether the visualization tool 115 is connected to determine whether non-surgical information may be present in the data stream. Thus, at the operation 415, the recording device 140 may determine whether the data stream is expected to include non-surgical information based on the event stream. [0056] At operation 420, if the recording device 140 determines that the data stream contains no non-surgical information (or the risk of non-surgical information is low), for example, because the visualization tool 115 is not connected, the recording device stores the data stream (e.g., the data stream 145) without modification at operation 425. On the other hand, if the recording device 140 determines that the data stream is likely to contain non- surgical information, for example, because the visualization tool 115 is connected to the robotic medical system 125 or the data stream includes out-of-body frames, the recording device modifies the data stream at operation 430. Modification of the data stream may include blurring non-surgical information, blocking or redacting non-surgical information, overwriting non-surgical information, removing or deleting non-surgical information, blurring, blocking, redacting, or overwriting entire data stream components either from the image capture devices 110 and/or those containing out-of-body frames. The data stream may be modified in other ways. At operation 435, the modified data stream is stored within the database 165. [0057] Turning now to FIG.5, an example flowchart outlining the operations of a process 500 is shown, in accordance with some embodiments of the present disclosure. The process 500 may be performed by a system having one or more processors executing computer- readable instructions stored on a memory. For example, the process 500 may be performed by the recording device 140, and particularly, by one or more processors of the recording device that execute non-transitory computer-readable instructions stored on a memory (e.g., the memory 190). Although the process 500 is described below as being performed by the recording device 140, in other embodiments, the process may be performed by the system having the one or more processors. The process 500 may be used to detect and scrub out non- surgical information from a data stream before storing the data stream to the database 165. [0058] At operation 505, the system such as the recording device 140 receives a data stream (e.g., the data stream 145) similar to operation 405. At operation 510, the recording device 140 receives an event stream (e.g., the events stream 150), similar to the operation 410.
Atty. Dkt: 135039-0109 (P06771-WO) Like the operations 405 and 410, the operations 505 and 510 may occur simultaneously or the operation 510 may occur before the operation 505. At operation 515, the recording device 140 determines, based on the data stream of the operation 505 and the event stream of the operation 510, whether a medical tool (e.g., the medical tool 120, for example, an endoscope) is installed in the robotic medical system (e.g., the robotic medical system 125). If the recording device 140 determines that a medical tool is installed in the robotic medical system, the recording device 140 determines if a visualization tool (e.g., the visualization tool 115) is connected to the robotic medical system (e.g., the robotic medical system 125) at operation 520. [0059] If, at the operation 520, the recording device 140 determines that the visualization tool is not connected to the robotic medical system, the recording device determines that the data stream includes images from the medical tool only. Further, because the data stream is received from the medical tool when the medical tool is installed to the robotic medical system, the recording device 140 determines that the data stream includes only in-body images, which contain no non-surgical information. Thus, at operation 525, the recording device 140 stores the data stream of the operation 505 within a database (e.g., the database 165) without modification. [0060] On the other hand, if at the operation 520, the recording device 140 determines that the visualization tool is connected to the robotic medical system, the recording device infers that the data stream from the operation 505 includes images from the medical tool (which is installed to the robotic medical system) and from other data sources (e.g., the image capture devices 110). The recording device 140 may also determine that because the data stream includes images from the other data sources, the images from these other data sources may include out-of-body images, which may include non-surgical information. Thus, at the operation 530, the recording device 140 localizes the visualization tool. Specifically, the recording device 140 may identify portions of the data stream of the operation 505 that includes images from data sources other than the medical tool. When the data stream of the operation 505 is displayed on a display (e.g., the display 130), data stream components from various data sources, including the medical tool, may be displayed on various portions of the recording device 140. By localizing the visualization tool, the recording device 140 may identify those portions of the data stream where the data stream components from data sources other than the medical tool are displayed. Thus, by localizing the visualization tool, the recording device 140 may identify the data stream components that are from data sources other than the medical tool. [0061] At operation 535, the recording device 140 modifies the data stream. Specifically, the recording device 140 may modify the data stream to scrub out the data stream components
Atty. Dkt: 135039-0109 (P06771-WO) that are from data sources other than the medical tool. Scrubbing out the data stream components may include blurring, blocking, redacting, overwriting, removing/deleting, or the like. In some embodiments, the recording device 140 may implement a machine learning model (e.g., the non-surgical information detection model 175) to specifically identify non- surgical information (e.g., the out-of-body images) from the data stream components to be concealed and conceal only conceal the identified non-surgical information (as opposed to concealing the entire data stream components). Modification of the data stream may preserve the images (e.g., the data stream components) that are from the medical tool (e.g., the medical tool 120). At operation 540, the recording device 140 stores the modified data stream within a data base (e.g., the database 165). [0062] If, at the operation 515, the recording device 140 determines that the medical tool is not installed in the robotic medical system (e.g., the robotic medical system 125), the recording device analyzes the data stream at operation 545. Specifically, if the medical tool is not installed in the robotic medical system, the medical tool may still be used laparoscopically, and therefore, provide a data stream. However, the data stream in this case from the medical tool may include out-of-body images. Thus, at the operation 545, the recording device 140 may analyze the data stream using a machine learning model (e.g., the in-body/out-of-body detection model 180) to classify each frame of the data stream of the operation 505 as either an in-body frame or an out-of-body frame. [0063] At operation 550, for each out-of-body frame identified, the recording device 140 modifies the data stream to conceal or scrub out the out-of-body frames from the data stream, preserving the in-body frames at the operation 535. Thus, upon scrubbing, only in-body frames may be left, which may be stored within a database (e.g., the database 165) at the operation 540. However, at the operation 550, if only in-body frames are found in the data stream (or after all the out-of-body frames) have been scrubbed, the recording device 140 determines if the visualization tool (e.g., the visualization tool 115) is connected to the robotic medical system at operation 555. If the visualization tool is not connected to the robotic medical system, the recording device 140 may determine that the data stream only contains in-body frames and store the data stream within a database (e.g., the database 145) at the operation 525 without modification (or without further modification). However, if the visualization tool is connected to the robotic medical system, the recording device 140 localizes the visualization tool at the operation 530 and modifies the data stream at the operation 535 before storing the modified data stream in the database at the operation 540.
Atty. Dkt: 135039-0109 (P06771-WO) [0064] In some embodiments, the recording device 140 may generate confidence scores indicating a likelihood or probability of having non-surgical information (e.g., PHI) in the data stream. In some embodiments, the recording device 140 may determine what proportion of time in the data stream 145 was the medical tool 120 installed in the robotic medical system 125 and what proportion of the time the visualization tool 115 was connected to the robotic medical system during a surgical procedure. Higher the proportion of time when the medical tool 120 is connected to the robotic medical system 125, higher the confidence that the data stream includes no non-surgical information. If the medical tool 120 is installed for 100% of the time in the robotic medical system 125 and the visualization tool 120 is connected for no time to the robotic medical system, the recording device 140 may produce a high confidence score (e.g., score of 100) that the data stream contains no non-surgical information. If the medical tool 120 is connected to the robotic medical system for at least some time and the visualization tool is also connected to the robotic medical system for 100% of the time, the recording device 140 may generate a score (e.g., a score of 0) indicating a low confidence that the data stream contains no non-surgical information. Thus, based on a proportion of time the medical tool 120 and the visualization tool 115 are connected to the robotic medical system 125, the recording device 140 may generate confidence scores. [0065] Referring to FIG.6, a surgical system 600 is shown, in accordance with some embodiments. The surgical system 600 may be an example of the medical environment 105. The surgical system 600 may include a robotic medical system 605 (e.g., the robotic medical system 125), a user control system 610, and an auxiliary system 615 communicatively coupled one to another. A visualization tool 620 (e.g., the visualization tool 115) may be connected to the auxiliary system 615, which in turn may be connected to the robotic medical system 605. Thus, when the visualization tool 620 is connected to the auxiliary system 615 and this auxiliary system is connected to the robotic medical system 605, the visualization tool may be considered connected to the robotic medical system. In some embodiments, the visualization tool 620 may additionally or alternatively be directly connected to the robotic medical system 605. [0066] The surgical system 600 may be used to perform a computer-assisted medical procedure on a patient 625. In some embodiments, surgical team may include a surgeon 630A and additional medical personnel 630B-630D such as a medical assistant, nurse, and anesthesiologist, and other suitable team members who may assist with the surgical procedure or medical session. The medical session may include the surgical procedure being performed on the patient 625, as well as any pre-operative (e.g., which may include setup of the surgical
Atty. Dkt: 135039-0109 (P06771-WO) system 600, including preparation of the patient 625 for the procedure), and post-operative (e.g., which may include clean up or post care of the patient), and/or other processes during the medical session. Although described in the context of a surgical procedure, the surgical system 600 may be implemented in a non-surgical procedure, or other types of medical procedures or diagnostics that may benefit from the accuracy and convenience of the surgical system. [0067] The robotic medical system 605 may include a plurality of manipulator arms 635A- 635D to which a plurality of medical tools (e.g., the medical tool 120) may be coupled or installed. Each medical tool may be any suitable surgical tool (e.g., a tool having tissue- interaction functions), imaging device (e.g., an endoscope, an ultrasound tool, etc.), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or other suitable instrument that may be used for a computer-assisted surgical procedure on the patient 625 (e.g., by being at least partially inserted into the patient and manipulated to perform a computer- assisted surgical procedure on the patient). Although the robotic medical system 605 is shown as including four manipulator arms (e.g., the manipulator arms 635A-635D), in other embodiments, the robotic medical system may include greater than or fewer than four manipulator arms. Further, not all manipulator arms may have a medical tool installed thereto at all times of the medical session. Moreover, in some embodiments, a medical tool installed on a manipulator arm may be replaced with another medical tool as suitable. [0068] One or more of the manipulator arms 635A-635D and/or the medical tools attached to manipulator arms may include one or more displacement transducers, orientational sensors, positional sensors, and/or other types of sensors and devices to measure parameters and/or generate kinematics information. One or more components of the surgical system 600 may be configured to use the measured parameters and/or the kinematics information to track (e.g., determine poses of) and/or control the medical tools, as well as anything connected to the medical tools and/or the manipulator arms 635A-635D. [0069] The user control system 610 may be used by the surgeon 630A to control (e.g., move) one or more of the manipulator arms 635A-635D and/or the medical tools connected to the manipulator arms. To facilitate control of the manipulator arms 635A-635D and track progression of the medical session, the user control system 610 may include a display (e.g., the display 130) that may provide the surgeon 630A with imagery (e.g., high-definition 3D imagery) of a surgical site associated with the patient 625 as captured by a medical tool (e.g., the medical tool 120, which may be an endoscope) installed to one of the manipulator arms 635A-635D. The user control system 610 may include a stereo viewer having two or more displays where stereoscopic images of a surgical site associated with the patient 625 and
Atty. Dkt: 135039-0109 (P06771-WO) generated by a stereoscopic imaging system may be viewed by the surgeon 630A. In some embodiments, the user control system 610 may also receive images from the auxiliary system 615 and the visualization tool 620. [0070] The surgeon 630A may use the imagery displayed by the user control system 610 to perform one or more procedures with one or more medical tools attached to the manipulator arms 635A-635D. To facilitate control of the manipulator arms 635A-635D and/or the medical tools installed thereto, the user control system 610 may include a set of controls. These controls may be manipulated by the surgeon 630A to control movement of the manipulator arms 635A-635D and/or the medical tools installed thereto. The controls may be configured to detect a wide variety of hand, wrist, and finger movements by the surgeon 630A to allow the surgeon to intuitively perform a procedure on the patient 625 using one or more medical tools installed to the manipulator arms 635A-635D. [0071] The auxiliary system 615 may include one or more computing devices configured to perform processing operations within the surgical system 600. For example, the one or more computing devices may control and/or coordinate operations performed by various other components (e.g., the robotic medical system 605, the user control system 610) of the surgical system 600. A computing device included in the user control system 610 may transmit instructions to the robotic medical system 605 by way of the one or more computing devices of the auxiliary system 615. The auxiliary system 615 may receive and process image data representative of imagery captured by one or more imaging devices (e.g., medical tools) attached to the robotic medical system 605, as well as other data stream sources received from the visualization tool. For example, one or more image capture devices (e.g., the image capture devices 110) may be located within the surgical system 600. These image capture devices may capture images from various viewpoints within the surgical system 600. These images (e.g., video streams) may be transmitted to the visualization tool 620, which may then passthrough those images to the auxiliary system 615 as a single combined data stream. The auxiliary system 615 may then transmit the single video stream (including any data stream received from the medical tool(s) of the robotic medical system 605) to present on a display (e.g., the display 130) of the user control system 610. [0072] In some embodiments, the auxiliary system 615 may be configured to present visual content (e.g., the single combined data stream) to other team members (e.g., the medical personnel 630B-630D) who may not have access to the user control system 610. Thus, the auxiliary system 615 may include a display 640 configured to display one or more user interfaces, such as images of the surgical site, information associated with the patient 625
Atty. Dkt: 135039-0109 (P06771-WO) and/or the surgical procedure, and/or any other visual content (e.g., the single combined data stream). In some embodiments, display 640 may be a touchscreen display and/or include other features to allow the medical personnel 630A-630D to interact with the auxiliary system 615. [0073] The robotic medical system 605, the user control system 610, and the auxiliary system 615 may be communicatively coupled one to another in any suitable manner. For example, in some embodiments, the robotic medical system 605, the user control system 610, and the auxiliary system 615 may be communicatively coupled by way of control lines 645, which may represent any wired or wireless communication link as may serve a particular implementation. Thus, the robotic medical system 605, the user control system 610, and the auxiliary system 615 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc. [0074] It is to be understood that the surgical system 600 may include other or additional components or elements that may be needed or considered desirable to have for the medical session for which the surgical system is being used. [0075] Turning to FIG.7, an example block diagram of an example computer system 700 is shown, in accordance with some embodiments. The computer system 700 may be any computing device used herein and may include or be used to implement a data processing system or its components. The computer system 700 includes at least one bus 705 or other communication component or interface for communicating information between various elements of the computer system. The computer system further includes at least one processor 710 or processing circuit coupled to the bus 705 for processing information. The computer system 700 also includes at least one main memory 715, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 705 for storing information, and instructions to be executed by the processor 710. The main memory 715 may be used for storing information during execution of instructions by the processor 710. The computer system 700 may further include at least one read only memory (ROM) 720 or other static storage device coupled to the bus 705 for storing static information and instructions for the processor 710. A storage device 725, such as a solid-state device, magnetic disk or optical disk, may be coupled to the bus 705 to persistently store information and instructions. [0076] The computer system 700 may be coupled via the bus 705 to a display 730, such as a liquid crystal display, or active-matrix display, for displaying information. An input device 735, such as a keyboard or voice interface may be coupled to the bus 705 for communicating information and commands to the processor 710. The input device 735 may include a touch
Atty. Dkt: 135039-0109 (P06771-WO) screen display (e.g., the display 730). The input device 735 may also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 710 and for controlling cursor movement on the display 730. [0077] The processes, systems and methods described herein may be implemented by the computer system 700 in response to the processor 710 executing an arrangement of instructions contained in the main memory 715. Such instructions may be read into the main memory 715 from another computer-readable medium, such as the storage device 725. Execution of the arrangement of instructions contained in the main memory 715 causes the computer system 700 to perform the illustrative processes described herein. One or more processors in a multi- processing arrangement may also be employed to execute the instructions contained in the main memory 715. Hard-wired circuitry can be used in place of or in combination with software instructions together with the systems and methods described herein. Systems and methods described herein are not limited to any specific combination of hardware circuitry and software. [0078] Although an example computing system has been described in FIG.7, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. [0079] The herein described subject matter sometimes illustrates different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are illustrative, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable,” to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable and/or physically interacting components and/or wirelessly interactable and/or wirelessly interacting components and/or logically interacting and/or logically interactable components.
Atty. Dkt: 135039-0109 (P06771-WO) [0080] With respect to the use of plural and/or singular terms herein, those having skill in the art can translate from the plural to the singular and/or from the singular to the plural as is appropriate to the context and/or application. The various singular/plural permutations may be expressly set forth herein for sake of clarity. [0081] It will be understood by those within the art that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.). [0082] Although the figures and description may illustrate a specific order of method steps, the order of such steps may differ from what is depicted and described, unless specified differently above. Also, two or more steps may be performed concurrently or with partial concurrence, unless specified differently above. Such variation may depend, for example, on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods can be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps. [0083] It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to inventions containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” and/or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations). [0084] Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in
Atty. Dkt: 135039-0109 (P06771-WO) the art would understand the convention (e.g., “a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general, such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word and/or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.” [0085] Further, unless otherwise noted, the use of the words “approximate,” “about,” “around,” “substantially,” etc., mean plus or minus ten percent. [0086] The foregoing description of illustrative implementations has been presented for purposes of illustration and of description. It is not intended to be exhaustive or limiting with respect to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the disclosed implementations. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.