EP4673077A1

EP4673077A1 - Systems and methods for calibrating an image sensor in relation to a robotic instrument

Info

Publication number: EP4673077A1
Application number: EP24713316.8A
Authority: EP
Inventors: A. Jonathan MCLEOD; Maximilian H. ALLAN
Original assignee: Intuitive Surgical Operations Inc
Current assignee: Intuitive Surgical Operations Inc
Priority date: 2023-03-01
Filing date: 2024-02-26
Publication date: 2026-01-07
Also published as: WO2024182294A1; CN120769731A

Abstract

An illustrative system includes a memory storing instructions and a processor communicatively coupled to the memory. The processor may be configured to execute the instructions to perform a process comprising obtaining scanning data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scanning data including images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement; determining, based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor; determining, based on the tracking data for the robotic instrument, second motion data for the robotic instrument; and determining, based on the first motion data and the second motion data, a calibration of the image sensor in relation to the robotic instrument in the imaging space.

Description

SYSTEMS AND METHODS FOR CALIBRATING AN IMAGE SENSOR IN RELATION TO A ROBOTIC INSTRUMENT

RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63/449,253, filed March 1 , 2023, the content of which is hereby incorporated by reference in its entirety.

BACKGROUND INFORMATION

[0002] A computer-assisted surgical system that employs robotic and/or teleoperation technology typically includes a stereoscopic image viewer configured to provide, for display to a surgeon, images of an imaging space (e.g., a surgical space) as captured by an imaging device such as an endoscope. While the surgeon’s eyes are positioned in front of viewing lenses of the stereoscopic image viewer, the surgeon may view the images of the surgical space while remotely manipulating one or more surgical instruments located within the surgical space. The surgical instruments are attached to one or more manipulator arms of a surgical instrument manipulating system included as part of the computer-assisted surgical system.

[0003] In addition to the surgical instruments that are attached to the one or more manipulator arms, additional instruments may be inserted into the surgical space to facilitate the surgeon performing procedures within the surgical space. For example, image sensors such as sub-surface sensing devices (e.g., ultrasound devices) may be provided within the surgical space to improve the surgeon’s perception of the surgical space and improve an outcome of a procedure. However, such additional instruments are not typically integrated into a module that attaches to a manipulator arm of a computer-assisted surgical system. In view of this, such additional instruments may only be available as drop-in instruments that rely on, for example, a grasper surgical instrument attached to a manipulator arm of a computer-assisted surgical system to grasp and move the drop-in instruments within the surgical space. Operation of a teleoperated grasper surgical instrument to interact with a drop-in instrument requires a surgeon to perform complex maneuvers to pick up and use the drop-in instrument within the surgical space. In addition, because there are numerous ways (e.g., grasping positions, grasping angles, etc.) that a robotic instrument such as a teleoperated grasper surgical instrument may engage with a drop-in instrument, it is difficult to determine a position and/or orientation of the drop-in instrument in relation to the robotic instrument.

SUMMARY

[0004] An example system comprises a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: obtaining scanning data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scanning data including images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement; determining, based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor; determining, based on the tracking data for the robotic instrument, second motion data for the robotic instrument; and determining, based on the first motion data and the second motion data, a calibration of the image sensor in relation to the robotic instrument in the imaging space.

[0005] An example computer program product embodied in a non-transitory computer readable storage medium comprises computer instructions for: obtaining scanning data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scanning data including images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement; determining, based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor; determining, based on the tracking data for the robotic instrument, second motion data for the robotic instrument; and determining, based on the first motion data and the second motion data, a calibration of the image sensor in relation to the robotic instrument in the imaging space.

[0006] An example method comprises obtaining, by a calibration system, scanning data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scanning data including images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement; determining, by the calibration system and based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor; determining, by the calibration system and based on the tracking data for the robotic instrument, second motion data for the robotic instrument; and determining, by the calibration system and based on the first motion data and the second motion data, a calibration of the image sensor in relation to the robotic instrument in the imaging space.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings illustrate various embodiments and are a part of the specification. The illustrated embodiments are merely examples and do not limit the scope of the disclosure. Throughout the drawings, identical or similar reference numbers designate identical or similar elements.

[0008] FIG. 1 illustrates an example computer-assisted surgical system according to principles described herein.

[0009] FIG. 2 illustrates an example view of an imaging space according to principles described herein.

[0010] FIG. 3 illustrates examples of various different ways that a robotic instrument may engage with example image sensors according to principles described herein.

[0011] FIG. 4 illustrates an example calibration system according to principles described herein.

[0012] FIG. 5 illustrates an example flow chart depicting various operations that may be performed by the calibration system illustrated in FIG. 4 according to principles described herein.

[0013] FIG. 6 illustrates an example image of an imaging space according to principles described herein.

[0014] FIGS. 7A and 7B illustrate additional example images of an imaging space according to principles described herein.

[0015] FIGS. 8A and 8B illustrate additional example images of an imaging space according to principles described herein.

[0016] FIG. 9 illustrates an example method for calibrating an image sensor in relation to a robotic instrument according to principles described herein.

[0017] FIG. 10 illustrates an example computing device according to principles described herein. DETAILED DESCRIPTION

[0018] Systems and methods for calibrating an image sensor in relation to a robotic instrument are described herein. As will be described in more detail below, an illustrative system includes a memory that stores instructions and a processor communicatively connected to the memory. The processor is configured to execute the instructions to perform a process comprising obtaining scanning data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scanning data including images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement; determining, based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor; determining, based on the tracking data for the robotic instrument, second motion data for the robotic instrument; and determining, based on the first motion data and the second motion data, a calibration of the image sensor in relation to the robotic instrument in the imaging space.

[0019] Various advantages and benefits are associated with systems and methods described herein. For example, systems and methods such as those described herein may facilitate quick and/or convenient calibration of an image sensor in relation to a robotic instrument regardless of the manner in which the image sensor is engaged by (e.g., grasped by) the robotic instrument. Such a calibration may result in improved imaging provided by way of the image sensor during operation of the image sensor in an imaging space. In addition, the systems and methods for calibration described herein may be based on uncharacterized data associated with an imaging space and do not require fiducials and/or supplemental information associated with phantom objects that may be used in conventional calibration methods. Moreover, systems and methods such as those described herein may reduce the mental and/or physical workload required for a user of a computer-assisted surgical system (e.g., a surgeon and/or another user associated with a computer-assisted surgical system) to use (e.g., teleoperate) a robotic instrument to interact with an image sensor located in a surgical space, such as by the systems and methods autonomously or semi-autonomously facilitating the robotic instrument performing a scanning movement and/or providing guidance to the user to help the user perform the scanning movement. In so doing, systems and methods such as those described herein may simplify procedures performed within the surgical space and/or improve usability of a computer-assisted surgical system. These and other benefits that may be realized by the systems and methods described herein will be evident from the disclosure that follows.

[0020] Example systems described herein may be configured to operate as part of or in conjunction with a plurality of different types of computer-assisted surgical systems. The different types of computer-assisted surgical systems may include any type of computer-assisted surgical system as may serve a particular implementation, such as a computer-assisted surgical system designed for use in minimally-invasive medical procedures, for example. In certain examples, a type of computer-assisted surgical system may include a system in which one or more surgical devices (e.g., surgical instruments) are manually (e.g., laparoscopically) controlled by a user. In certain examples, a type of computer-assisted surgical system may include a robotic surgical system configured to facilitate operation one or more smart instruments (e.g., smart subsurface imaging devices) that may be manually and/or robotically controlled by a user. In certain implementations, the plurality of different types of computer-assisted surgical systems may be of different types at least because they include different types of surgical instrument manipulating systems. For example, a first computer-assisted surgical system may include a first type of surgical instrument manipulating system, a second computer-assisted surgical system may include a second type of surgical instrument manipulating system, and a third computer-assisted surgical system may include a third type of surgical instrument manipulating system.

[0021] Each type of surgical instrument manipulating system may have a different architecture (e.g., a manipulator arm architecture), have a different kinematic profile, and/or operate according to different configuration parameters. An illustrative computer- assisted surgical system with a first type of surgical instrument manipulating system will now be described with reference to FIG. 1. The described computer-assisted surgical system is illustrative and not limiting. Systems such as those described herein may operate as part of or in conjunction with the described computer-assisted surgical system and/or any other suitable computer-assisted surgical system.

[0022] FIG. 1 illustrates an example computer-assisted surgical system 100 (“surgical system 100”). As shown, surgical system 100 may include a surgical instrument manipulating system 102 (“manipulating system 102”), a user control system 104, and an auxiliary system 106 communicatively coupled one to another.

[0023] Surgical system 100 may be utilized by a surgical team to perform a computer-assisted surgical procedure on a patient 108. As shown, the surgical team may include a surgeon 110-1 , an assistant 110-2, a nurse 110-3, and an anesthesiologist 110-4, all of whom may be collectively referred to as “surgical team members 110.” Additional or alternative surgical team members may be present during a surgical session as may serve a particular implementation.

[0024] While FIG. 1 illustrates an ongoing minimally invasive surgical procedure, surgical system 100 may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of surgical system 100. Additionally, it will be understood that the surgical session throughout which surgical system 100 may be employed may not only include an operative phase of a surgical procedure, as is illustrated in FIG. 1 , but may also include preoperative, postoperative, and/or other suitable phases of the surgical procedure. A surgical procedure may include any procedure in which manual and/or instrumental techniques (e.g., teleoperated instrumental techniques) are used on a patient to investigate, diagnose, or treat a physical condition of the patient. Additionally, a surgical procedure may include any procedure that is not performed on a live patient, such as a calibration procedure, a simulated training procedure, and an experimental or research procedure.

[0025] As shown in FIG. 1 , surgical instrument manipulating system 102 may include a plurality of manipulator arms 112 (e.g., manipulator arms 112-1 through 112-4) to which a plurality of robotic surgical instruments (“robotic instruments”) (not shown) may be coupled. As used herein, a “robotic instrument” refers to any instrument that may be directly attached to (e.g., plugged into, fixedly coupled to, mated to, etc.) a manipulator arm (e.g., manipulator arm 112-1) such that movement of the manipulator arm directly causes movement of the instrument. Each robotic instrument may be implemented by any suitable therapeutic instrument (e.g., a tool having tissue-interaction functions), imaging device (e.g., an endoscope), diagnostic instrument, or the like that may be used for a computer-assisted surgical procedure (e.g., by being at least partially inserted into patient 108 and manipulated to perform a computer-assisted surgical procedure on patient 108). In some examples, one or more of the robotic instruments may include force-sensing and/or other sensing capabilities.

[0026] In the example shown in FIG. 1 , manipulator arms 112 of manipulating system 102 are attached on a distal end of an overhead boom that extends horizontally. However, manipulator arms 112 may have other configurations in certain implementations. In addition, while manipulating system 102 is depicted and described herein as including four manipulator arms 112, it will be recognized that manipulating system 102 may include only a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation.

[0027] Manipulator arms 112 and/or robotic instruments attached to manipulator arms 112 may include one or more displacement transducers, orientational sensors, and/or positional sensors (hereinafter “surgical system sensors”) used to generate raw (e.g., uncorrected) kinematics information. One or more components of surgical system 100 may be configured to use the kinematics information to track (e.g., determine positions of) and/or control the robotic instruments.

[0028] In addition, manipulator arms 112 may each include or otherwise be associated with a plurality of motors that control movement of manipulator arms 112 and/or the surgical instruments attached thereto. For example, manipulator arm 112-1 may include or otherwise be associated with a first internal motor (not explicitly shown) configured to yaw manipulator arm 112-1 about a yaw axis. In like manner, manipulator arm 112-1 may be associated with a second internal motor (not explicitly shown) configured to drive and pitch manipulator arm 112-1 about a pitch axis. Likewise, manipulator arm 112-1 may be associated with a third internal motor (not explicitly shown) configured to slide manipulator arm 112-1 along insertion axis. Manipulator arms 112 may each include a drive train system driven by one or more of these motors in order to control the pivoting of manipulator arms 112 in any manner as may serve a particular implementation. As such, if a robotic instrument attached, for example, to manipulator arm 112-1 is to be mechanically moved, one or more of the motors coupled to the drive train may be energized to move manipulator arm 112-1.

[0029] Robotic instruments attached to manipulator arms 112 may each be positioned in an imaging space. An “imaging space” as used herein may refer to any space or location where an imaging operation may be performed by an imaging device such as described herein. In certain examples, an imaging space may correspond to a surgical space. A “surgical space” may, in certain examples, be entirely disposed within a patient and may include an area within the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed. For example, for a minimally invasive surgical procedure being performed on tissue internal to a patient, the surgical space may include the tissue, anatomy underlying the tissue, as well as space around the tissue where, for example, robotic instruments and/or other instruments being used to perform the surgical procedure are located. In other examples, a surgical space may be at least partially disposed external to the patient at or near where a surgical procedure is planned to be performed, is being performed, or has been performed on the patient. For instance, surgical system 100 may be used to perform an open surgical procedure such that part of the surgical space (e.g., tissue being operated on) is internal to the patient while another part of the surgical space (e.g., a space around the tissue where one or more instruments may be disposed) is external to the patient. A robotic instrument may be referred to as being positioned or located at or within a surgical space when at least a portion of the robotic instrument (e.g., a distal portion of the robotic instrument) is located within the surgical space. Example imaging spaces and/or images of imaging spaces will be described herein. [0030] User control system 104 may be configured to facilitate control by surgeon 110-1 of manipulator arms 112 and robotic instruments attached to manipulator arms 112. For example, surgeon 110-1 may interact with user control system 104 to remotely move, manipulate, or otherwise teleoperate manipulator arms 112 and the robotic instruments. To this end, user control system 104 may provide surgeon 110-1 with one or more images (e.g., high-definition three-dimensional (3D) images) of a surgical space associated with patient 108 as captured by an imaging device. In certain examples, user control system 104 may include a stereoscopic image viewer having two displays where stereoscopic images (e.g., 3D images) of a surgical space associated with patient 108 and generated by a stereoscopic imaging system may be viewed by surgeon 110-1. Surgeon 110-1 may utilize the images to perform one or more procedures with one or more robotic instruments attached to manipulator arms 112.

[0031] To facilitate control of robotic instruments, user control system 104 may include a set of master controls (not shown). These master controls may be manipulated by surgeon 110-1 to control movement of robotic instruments (e.g., by utilizing robotic and/or teleoperation technology). The master controls may be configured to detect a wide variety of hand, wrist, and finger movements by surgeon 110-1. In this manner, surgeon 110-1 may intuitively perform a surgical procedure using one or more robotic instruments.

[0032] User control system 104 may further be configured to facilitate control by surgeon 110-1 of other components of surgical system 100. For example, surgeon 110- 1 may interact with user control system 104 to change a configuration or operating mode of surgical system 100, to change a display mode of surgical system 100, to generate additional control signals used to control surgical instruments attached to manipulator arms 112, to facilitate switching control from one robotic instrument to another, to facilitate interaction with other instruments and/or objects within the surgical space, or to perform any other suitable operation. To this end, user control system 104 may also include one or more input devices (e.g., foot pedals, buttons, switches, etc.) configured to receive input from surgeon 110-1.

[0033] Auxiliary system 106 may include one or more computing devices configured to perform primary processing operations of surgical system 100. The one or more computing devices included in auxiliary system 106 may control and/or coordinate operations performed by various other components (e.g., manipulating system 102 and/or user control system 104) of surgical system 100. For example, a computing device included in user control system 104 may transmit instructions to manipulating system 102 by way of the one or more computing devices included in auxiliary system 106. As another example, auxiliary system 106 may receive, from manipulating system 102, and process image data representative of images captured by an imaging device attached to one of manipulator arms 112.

[0034] In some examples, auxiliary system 106 may be configured to present visual content to surgical team members 110 who may not have access to the images provided to surgeon 110-1 at user control system 104. To this end, auxiliary system 106 may include a display monitor 114 configured to display one or more user interfaces, such as images (e.g., 2D images) of the surgical space, information associated with patient 108 and/or the surgical procedure, and/or any other visual content as may serve a particular implementation. For example, display monitor 114 may display images of the surgical space together with additional content (e.g., representations of target objects, graphical content, contextual information, etc.) concurrently displayed with the images. In some embodiments, display monitor 114 is implemented by a touchscreen display with which surgical team members 110 may interact (e.g., by way of touch gestures) to provide user input to surgical system 100.

[0035] Manipulating system 102, user control system 104, and auxiliary system 106 may be communicatively coupled one to another in any suitable manner. For example, as shown in FIG. 1 , manipulating system 102, user control system 104, and auxiliary system 106 may be communicatively coupled by way of control lines 116, which may represent any wired or wireless communication link as may serve a particular implementation. To this end, manipulating system 102, user control system 104, and auxiliary system 106 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0036] FIG. 2 illustrates a view 200 of an imaging space (e.g., a surgical space) in which various robotic instruments are attached to manipulator arms 112 of surgical system 100. As shown, the robotic instruments may include an imaging device 202 and one or more other robotic instruments 204 (e.g., robotic instruments 204-1 through 204- 3) in the form of one or more surgical tools. While FIG. 2 shows one imaging device 202 and three other robotic instruments 204 located at the imaging space, any number, type, and/or combination of robotic instruments may be at the imaging space during a surgical procedure. In the example shown in FIG. 2, robotic instruments 204-1 and 204-3 are shown as grasping-type robotic instruments whereas robotic instrument 204-2 is shown as a cutting-type robotic instrument. It is understood that other types of robotic instruments (e.g., diagnostic tools, therapeutic tools, etc.) different than those shown in FIG. 2 may additionally or alternatively be provided within the imaging space during the surgical procedure in certain implementations. Tissue 206 represents anatomical tissue at the imaging space.

[0037] Imaging device 202 may capture one or more images at the imaging space. Any of robotic instruments 204 and/or tissue 206 that are within a field of view of imaging device 202 may be depicted in the image(s) captured by imaging device 202. [0038] Imaging device 202 may provide data representing visible light data of an imaging space. For example, imaging device 202 may capture visible light images of the surgical space that represent visible light sensed by imaging device 202. Visible light images may include images that use any suitable color and/or grayscale palette to represent a visible light-based view of the imaging space.

[0039] Imaging device 202 may also provide data representing depth data of an imaging space or data that may be processed to derive depth data of the imaging space. For example, imaging device 202 may capture images of the imaging space that represent depth sensed by imaging device 202. Alternatively, imaging device 202 may capture images of the imaging space that may be processed to derive depth data of the imaging space. The depth information may be represented as depth images (e.g., depth map images obtained using a Z-buffer that indicates distance from imaging device 202 to each pixel point on an image of an imaging space), which may be configured to visually indicate depths of objects in the imaging space in any suitable way, such as by using different greyscale values to represent different depth values. Images captured by an imaging device (e.g., by imaging device 202) and/or derived from images captured by the imaging device (e.g., visible light images and depth images) may be used to facilitate detecting a robotic instrument (e.g., robotic instruments 204-1 through 204-3) and/or one or more objects within an imaging space, such as described herein. [0040] During a procedure such as a surgical procedure, it may be desirable to use a robotic instrument (e.g., robotic instrument 204-1 or 204-3) to teleoperate an image sensor that is provided in addition to imaging device 202 but that is not directly attached to one of manipulator arms 112. In this scenario, such an image sensor may only be movable within an imaging space by either being manually manipulated by a user (e.g., surgeon 110-1 , assistant 110-2, etc.) or by being moved by a user through teleoperation of a robotic instrument directly attached to one of manipulator arms (e.g., by being grasped or otherwise engaged by robotic instrument 204-3). In this regard, “teleoperation of an image sensor” as described herein refers to the indirect teleoperation of an image sensor by way of a robotic instrument attached to a computer- assisted surgical system. As such, such an image sensor may be referred to as a drop- in image sensor. Examples of image sensors may include, but are not limited to, a drop- in ultrasound probe and/or any other suitable image sensor that may be provided in an imaging space.

[0041] Image sensors such as those described herein that are not directly attached to one of manipulator arms 112 may be engaged with a computer-assisted surgical system in any suitable manner. In certain examples, such image sensors may be engaged by a computer-assisted surgical system by being communicatively coupled, in any suitable manner, to the computer-assisted surgical system. In certain examples, such image sensors may be physically coupled to a component of a computer-assisted surgical system and/or by a component attached to a computer-assisted surgical system, such as a robotic instrument (e.g., robotic instrument 204-1 or 204-3) attached to a computer-assisted surgical system. This may be accomplished in any suitable manner. For example, in certain implementations, an image sensor may be configured to be grasped by a grasper robotic instrument such as robotic instrument 204-1. To that end, in certain examples, an image sensor may include one or more graspable portions (e.g., protrusions, loops, etc.) that a robotic instrument may grasp to facilitate user teleoperation of the image sensor. Alternatively, a robotic instrument may generally grasp a casing or housing of an image sensor. Example image sensors will be described further herein.

[0042] There is a large amount of variability in how image sensors may be physically coupled to one of robotic instruments 204. FIG. 3 illustrates a diagram 300 that depicts a plurality of different example grasping states that robotic instrument 204-1 may assume when grasping image sensors 302 (e.g., image sensors 302-1 and 302-2). As shown in FIG. 3, image sensor 302-1 may be grasped in a first grasp state 304-1 in which robotic instrument 204-1 grasps a central region of image sensor 302-1. Alternatively, image sensor 302-1 may be grasped in a second grasp state 304-2 in which robotic instrument 204-1 grasps image sensor 302-1 at a position closer to a distal end of image sensor 302-1. Image sensor 302-2 is different from image sensor 302-1 in that image sensor 302-2 includes a protrusion 306 that is configured to be grasped by robotic instrument 204-1. As shown in FIG. 3, protrusion 306 of image sensor 302-2 may be grasped in a first grasp state 308-1 where protrusion 306 is deeply seated within the jaws of robotic instrument 204-1. Alternatively, protrusion 306 may be grasped in a second grasp state 308-2 where protrusion 306 is only partially seated within the jaws of robotic instrument 204-1. The example grasp states depicted in FIG. 3 are provided for illustrative purposes only. It is understood that robotic instrument 204-1 may grasp image sensors 302-1 and 302-2 at any other suitable location and/or at any other suitable angle with respect to image sensors 302 in certain implementations.

[0043] Due to the variability in ways that image sensors may be physically coupled to a robotic instrument, it is difficult to use kinematics information of a robotic instrument directly to perform operations (e.g., for three-dimensional ultrasound reconstruction, augmented reality, and/or other forms of guidance) associated with an image sensor in an imaging space. This is because each time a robotic instrument engages with an image sensor, there may be an unknown linkage between the pose (e.g., the 3D position in 3D space and/or the orientation in 3D space) of the image sensor and the pose the robotic instrument that is engaged with the image sensor. Accordingly, it is desirable to perform a calibration to determine the linkage between the position of an image sensor and the position of a robotic instrument upon the image sensor being physically coupled to the robotic instrument.

[0044] FIG. 4 shows an example calibration system 400 that may be implemented according to principles described herein to calibrate an image sensor in relation to a robotic instrument. As shown in FIG. 4, calibration system 400 (e.g., system 400) may include, without limitation, a memory 402 and a processor 404 selectively and communicatively coupled to one another. Memory 402 and processor 404 may each include or be implemented by hardware and/or software components (e.g., processors, memories, communication interfaces, instructions stored in memory for execution by the processors, etc.). In some examples, memory 402 and processor 404 may be implemented by a single device (e.g., a single computing device). In certain alternate examples memory 402 and processor 404 may be distributed between multiple devices and/or multiple locations as may serve a particular implementation. [0045] Memory 402 may maintain (e.g., store) executable data used by processor 404 to perform any of the operations described herein. For example, memory 402 may store instructions 406 that may be executed by processor 404 to perform any of the operations described herein. Instructions 406 may be implemented by any suitable application, software, code, and/or other executable data instance.

[0046] Memory 402 may also maintain any data received, generated, managed, used, and/or transmitted by processor 404. For example, memory 402 may maintain any suitable data associated with calibrating an image sensor. Such data may include, but is not limited to, data associated with scanning movements, depth map information associated with an imaging space, scanning data (e.g., images captured by an image sensor, kinematics data for robotic instruments and/or manipulator arms, etc.), pose information associated with image sensors and/or additional objects located in a surgical space, endoscopic images of an imaging space, data defining guidance content associated with an image sensor, augmented images of an imaging space, composite images, motion path data, user interface content (e.g., graphical objects, notifications, etc.), ultrasound reconstructed volumes, and/or any other suitable data.

[0047] Processor 404 may be configured to perform (e.g., execute instructions 406 stored in memory 402) various processing operations associated with calibrating an image sensor. For example, processor 404 may determine, based on first motion data for an image sensor and second motion data for a robotic instrument, a calibration of the image sensor in relation to the robotic instrument in an imaging space. These and other operations that may be performed by processor 404 are described herein.

[0048] FIG. 5 illustrates a flow diagram 500 depicting various operations that may be performed by system 400 (e.g., processor 404) to determine a calibration such as described herein. At operation 502, system 400 may obtain scanning data. Such scanning data may be collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space.

[0049] The scanning data may include any suitable information that may be accessed, obtained, or generated by system 400 during a scanning movement. For example, the scanning data may include images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement. The images may include any suitable number of images captured by the image sensor during the scanning movement. For example, the images may include a first image captured by the image sensor at a first time point during the scanning movement, a second image captured by the image sensor at a second time point during the scanning movement, a third image captured by the image sensor at a second time point during the scanning movement, and so forth. In examples where the image sensor corresponds to a drop-in ultrasound image sensor, the images may correspond to subsurface images of tissue captured during the scanning movement.

[0050] The tracking data for the robotic instrument may include any suitable data associated with the robotic instrument. For example, the tracking data may include kinematics data indicative of at least one of a position or an orientation of the robotic instrument during the scanning movement. In certain examples, system 400 may map kinematics data at a particular time point during a scanning movement to a corresponding image captured by the image sensor at that particular time point. For example, a first set of kinematics data for the robotic instrument at the first time point may be mapped to the first image captured by the image sensor, a second set of kinematics data for the robotic instrument at the second time point may be mapped to the second image captured by the image sensor, a third set of kinematics data for the robotic instrument may be mapped to the third image captured by the image sensor, and so forth.

[0051] The scanning movement may include any suitable movement of a robotic instrument in an imaging space. In certain examples, the scanning movement may be performed by a user (e.g., surgeon 110-1) teleoperating a robotic instrument to move an image sensor in the imaging space.

[0052] To illustrate an example, FIG. 6 shows an image 600 that may be captured by imaging device 202 and that includes an example image sensor that may be teleoperated during a surgical procedure in a surgical space according to principles described herein. As shown in FIG. 6, image 600 illustrates a surgical space in which image sensor 302-2 and robotic instruments 204-1 through 204-3 are disposed in relation to a kidney 602 of a patient (e.g., patient 108). As shown in FIG. 6, image sensor 302-2 includes protrusion 306 that is grasped by robotic instrument 204-1. As such, teleoperation of robotic instrument 204-1 by a user (e.g., by surgeon 110-1 manipulating master controls of user control system 104) results in teleoperation of robotic instrument 204-1. In the example shown in FIG. 6, an arrow 604-1 represents an example motion path that robotic instrument 204-1 may take to reach a surface of kidney 602. An arrow 604-2 represents an example scanning movement that robotic instrument 204-1 may take to move image sensor 302-2 along a surface of kidney 602. Although the example scanning movement depicted in FIG. 6 has a curved shape, it is understood that a scanning movement may have any suitable shape and/or attribute as may serve a particular implementation. For example, a scanning movement may be performed in a straight line in certain examples. Alternatively, a scanning movement may include one or more changes of direction. For example, the scanning movement may have a zig zag shape and/or may retrace over a previously traveled portion of the scanning movement in certain implementations.

[0053] In certain implementations, system 400 may facilitate user teleoperation of an image sensor within an imaging space with various levels of autonomy. For example, in certain examples, system 400 may assist a user in teleoperating a robotic instrument to perform the scanning movement. For example, system 400 may provide haptic feedback, audio feedback, visual feedback, and/or any other suitable notification or guidance to facilitate a user in moving a robotic instrument such as robotic instrument 204-1 along a motion path associated with a scanning movement.

[0054] In certain alternative examples, system 400 may automatically perform the scanning movement in the imaging space. As used herein, the expression “automatically” means that an operation (e.g., performing a scanning movement) or series of operations are performed without requiring further input from a user. For example, system 400 may analyze depth data, images, and/or any suitable information associated with an imaging space. Based on such information, system 400 may automatically control a robotic instrument to move toward a starting position for a scanning movement and/or automatically move the robotic instrument along a surface in the imaging space during the scanning movement, without requiring that the user provide further input.

[0055] Returning to FIG. 5, at operation 504, system 400 may determine, based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor. The first motion data may be indicative of a trajectory of the image sensor during the scanning movement (e.g., sets of positions and/or orientations of the image sensor over time during the scanning movement). System 400 may determine the first motion data in any suitable manner. For example, in certain implementations, system 400 may determine the first motion data by determining a spatial relationship between the images captured by the image sensor using temporal changes of a pattern in the captured images.

[0056] To illustrate, FIGS. 7A and 7B show images 700 (e.g., images 700-1 and 700- 2) of a surgical space in which image sensor 302-2 is being used to capture ultrasound images 702 (e.g., ultrasound images 702-1 and 702-2) at different positions within the surgical space. As shown in FIG. 7A, image sensor 302-2 is positioned at a first position along the scanning movement represented by arrow 604-2. Ultrasound image 702-1 includes a first pattern representative of a subsurface image of kidney 602. On the other hand, in FIG. 7B, image sensor 302-2 is positioned at a second position along the scanning movement represented by arrow 604-2. Ultrasound 702-2 includes a second pattern representative of a subsurface image of kidney 602. The temporal changes between the first pattern in image 702-1 and the second pattern in image 702-2 may be indicative of a spatial relationship between the images and may be used by system 400 in any suitable manner to determine the first motion data. This is one example of how the images captured by an image sensor such as image sensor 302-2 may be used. Other suitable ways of using such images may be used in other examples.

[0057] Ultrasound images 702 are shown to the side of images 700 in FIGS. 7 A and 7B for illustrative purposes. It is understood that ultrasound images 702 may be provided for display in any suitable manner as may serve a particular implementation. In certain alternative implementations, ultrasound images may be provided as an augmentation to an image of a surgical space (e.g., as an overlay over an endoscopic image of a surgical space). For example, ultrasound image 702-1 may be overlaid over a portion of image 700-2 in certain implementations so that a user (e.g., surgeon 110-1) may view one or more captured ultrasound images concurrently and in place while teleoperating robotic instrument 204-1 to move image sensor 302-2. Additionally or alternatively, ultrasound images 700 may be provided for display at any other location relative to an image of a surgical space and/or by way of any other suitable display device (e.g., display monitor 114) associated with a computer-assisted surgical system. In certain alternative examples, ultrasound images associated with a scanning movement may not be provided for display. In such examples, the determining of the first motion data based on images captured by an image sensor may be transparent to a user (e.g., the determination may be performed as a background process).

[0058] In certain examples, the determining of the spatial relationship between ultrasound images may include using speckle decorrelation. In such examples, a distance that the image sensor moves during a scanning movement may be estimated based on an amount of correlation between a first speckle pattern in a first image captured by the image sensor during the scanning movement and second speckle pattern in a second image captured by the image sensor after the first image.

[0059] Additionally or alternatively, system 400 may determine the spatial relationship by implementing machine learning operations. Any suitable machine learning operation may be used as may serve a particular implementation. In certain examples, a pre-trained machine learning algorithm may be implemented that is trained based on different surface contours that may be present in an imaging space. Additionally or alternatively, the machine learning algorithm may be trained based on one or more previous procedures performed in an imaging space. For example, information obtained during one or more previous surgical procedures associated with different patients may be used to train the machine learning algorithm in certain implementations. In certain examples, such a machine learning algorithm may additionally or alternatively be configured to filter out portions of images captured by the image sensor that are not suitable for speckle decorrelation.

[0060] Returning to FIG. 5, at operation 506, system 400 may determine, based on the tracking data for the robotic instrument, second motion data for the robotic instrument. This may be accomplished in any suitable manner. For example, system 400 may determine the second motion data based on kinematics data of the robotic instrument as the robotic instrument moves during the scanning movement. The kinematics data may specify any suitable information associated with the robotic instrument as the robotic instrument moves during the scanning movement. For example, the kinematics data may include information regarding the position, orientation, trajectory, etc. of the robotic instrument at any given time point during the scanning movement. In certain examples, the second motion data may further be based on endoscopic images captured by an imaging device such as imaging device 202. In such examples, the endoscopic images may be analyzed in any suitable manner to confirm and/or adjust the second motion data and/or position, orientation, and/or trajectory data associated with the movement of a robotic instrument during a scanning movement.

[0061] At operation 508, system 400 may determine a calibration of the image sensor in relation to the robotic instrument in the imaging space. This may be accomplished in any suitable manner. For example, system 400 may determine a calibration transform between the first motion data and the second motion data. Such a calibration transform may be calculated in any suitable manner. For example, system 400 may formulate the calibration as an AX = XB problem and estimate the calibration of the image sensor in relation to the robotic instrument based on the solution to the AX = XB problem. In such examples, A, X, and B may each represent homogeneous transformations (e.g., rigid-body motions) with A representing the first motion data and B representing the second motion data. With such an equation, it is possible to solve for X and determine the current spatial relationship between the robotic instrument and the physically coupled image sensor based on the correspondence between the first motion data and the second motion data.

[0062] In certain examples, system 400 may determine whether the calibration determined at operation 508 is within a predefined threshold confidence level. If it is determined that the calibration is not within the predefined threshold confidence level, system 400 may perform an additional scanning movement and repeat operations 502- 508 until the predefined threshold confidence level is achieved.

[0063] At operation 510, system 400 may use the calibration in one or more processes associated with the image sensor in the imaging space. This may be accomplished in any suitable manner. For example, based on the calibration, system 400 may facilitate the image sensor maintaining (e.g., automatically or through haptic feedback) a predefined contact angle (e.g., a 90° contact angle) with respect to tissue during a surgical procedure. Additionally or alternatively, the calibration may be used by system 400 to process images captured by the image sensor during a surgical procedure and/or to generate or refine a 3D reconstructed volume of an object (e.g., kidney 602) in an imaging space.

[0064] At operation 512, system 400 may determine whether there has been a change in the physical coupling of the robotic instrument to the image sensor. For example, system 400 may detect a change in the grasp state of the image sensor in relation to the robotic instrument. System 400 may determine the change in any suitable manner. For example, system 400 may determine that there has been a change in the grasp state based on kinematics information associated with the robotic instrument. For example, the kinematics information may indicate that the jaws of a robotic instrument 204-1 opened more than a predefined threshold amount, which may be indicative of a change in the grasp state. Additionally or alternatively, system 400 may use endoscopic images, depth map images, and/or any other suitable image(s) of the imaging space to determine whether there has been a change in the grasp position, angle, etc. of the robotic instrument in relation to the image sensor. In certain additional or alternative implementations, system 400 may detect the change based on an unexpected difference between kinematics data and images captured by the image sensor. For example, system 400 may detect that there is a change in the grasp state in instances where one of the kinematics data or the images change without a corresponding expected change in the other. For example, system 400 may detect the change based on the kinematics data indicating movement of a robotic instrument without the images changing in a manner that is expected based on the movement (e.g., which may be indicative of the image sensor not being currently grasped by the robotic instrument and thus stationary). If the answer at operation 512 is “NO,” the flow returns to operation 510 and system 400 may continue to use the calibration determined at operation 508 in one or more processes associated with the image sensor. On the other hand, if the answer at operation 512 is “Yes,” the flow may return to operation 502 and system 400 may repeat operations 502-508 to determine an updated calibration of the image sensor in relation to the robotic instrument.

[0065] In certain examples, system 400 may be further configured to generate an ultrasound reconstructed volume based on images captured by an ultrasound image sensor. The ultrasound reconstructed volume may represent a three-dimensional representation of the subsurface structure of an object (e.g., tissue such as a kidney or any other anatomical structure) in an imaging space. The ultrasound reconstructed volume may be generated in any suitable manner. For example, the ultrasound reconstructed volume may be generated based on the calibration determined at operation 508.

[0066] In certain examples, a scanning movement such as described herein may be performed by system 400 in parallel with the generating of an ultrasound reconstructed volume. In such examples, the images captured during the scanning movement may be used both for calibration as well as for generating the ultrasound reconstructed volume. [0067] In certain examples, the ultrasound reconstructed volume may be a previously generated ultrasound reconstructed volume. In such examples, system 400 may be configured to refine, based on the calibration, the previously generated ultrasound reconstructed volume.

[0068] In certain examples, system 400 may be configured to generate augmented images or composite images of an imaging space based on endoscopic images and images captured by an image sensor such as an ultrasound image sensor. To that end, system 400 may obtain endoscopic images of the imaging space in any suitable manner. For example, system 400 may access images captured by imaging device 202. System 400 may register, based on the calibration, the images captured by the image sensor with the endoscopic images. Based on the registering, system 400 may generate an augmented image or a composite image associated with the imaging space. Such an augmented image or composite image may be represented in any suitable manner to a user during a procedure. For example, an augmented image or a composite image may be provided for display to surgeon 110-1 by way of a display device of user control system 104 or may overlaid over a live image of the imaging space provided for display to surgeon 110-1 by way of a display device of user control system 104.

[0069] In certain examples, system 400 may generate guidance content associated with an image sensor such as a drop-in ultrasound sensor. As used herein, “guidance content” may include any content that may be used by a computer-assisted surgical system to facilitate guided teleoperation of an image sensor in an imaging space. The generating of such guidance content by system 400 may include generating instructions and/or other guidance content for use by a computer-assisted surgical system, such as by generating computer-readable instructions for processing by the computer-assisted surgical system, and/or may include generating and/or accessing any suitable content to be presented by the computer-assisted surgical system (e.g., via a user interface associated with the computer-assisted surgical system).

[0070] Examples of guidance content may include, but are not limited to, notifications, virtual pointers, animations, instructions, audible guidance, visual guidance, haptic feedback guidance, graphical depictions of motion paths for an robotic instrument to follow during a scanning movement, content configured to indicate a contact state of an image sensor with respect to an object in the surgical space, instructions usable by the computer-assisted surgical system to provide guidance content, and/or any combination thereof. Examples of guidance content that may be generated by system 400 to be presented by a computer-assisted surgical system may include, but are not limited to, motion paths for a robotic instrument to follow within an imaging space, content configured to indicate a contact state of an image sensor with respect to an object in the imaging space, and/or any other generated content that may facilitate guided teleoperation of an image sensor. Specific examples of guidance content are described herein.

[0071] System 400 may generate guidance content at any suitable time. For example, system 400 may generate guidance content prior to a surgical procedure, during a surgical procedure, and/or at any other suitable time.

[0072] In certain examples, system 400 may generate at least some guidance content by accessing the guidance content from a storage device (e.g., memory 402) associated with a computer assisted surgical system (e.g., surgical system 100). Examples of guidance content that may be accessed from a storage device may include, but are not limited to, graphical depictions of robotic instruments, image sensors, and/or other instruments that are engaged by (e.g., grasped by) robotic instruments, audible notifications, visual notifications, etc. [0073] Guidance content may be generated based on any suitable parameters associated with a surgical space. For example, guidance content may be generated based on one or more of a procedural context associated with the surgical space, parameters of an image sensor (e.g., an identified type of image sensor, a pose of the image sensor, etc.), parameters of a robotic instrument (e.g., an identified type of robotic instrument, a pose of the robotic instrument, etc.), an indicated or a predicted use or operation of the image sensor, and/or any other suitable parameter or combination of parameters.

[0074] Guidance content generated by system 400 for presentation by a computer- assisted surgical system may be configured to be presented in any suitable manner. For example, in certain implementations, the guidance content may be configured to be presented by way of a user interface associated with a computer-assisted surgical system. To illustrate, system 400 may provide the guidance content for presentation by way of user control system 104 of surgical system 100 to facilitate a user, such as surgeon 110-1 , teleoperating an image sensor. Additionally or alternatively, the guidance content may be provided for presentation by way of any other suitable user interface that may be associated with a computer-assisted surgical system. For example, guidance content may be provided to a user by way of a user interface associated with display monitor 114 of auxiliary system 106 in certain implementations. [0075] In certain examples, system 400 may provide guidance content as visual guidance to facilitate a user (e.g., surgeon 110-1 ) of a computer-assisted surgical system teleoperating an image sensor in an imaging space. Such visual guidance may be provided in any suitable manner. For example, system 400 may instruct a computer- assisted surgical system to provide a blinking light and/or any suitable graphical object or augmented overlay for display to a user (e.g., to surgeon 110-1 by way of user control system 104) that guides the user in teleoperation of the image sensor in an imaging space.

[0076] Additionally or alternatively, system 400 may provide guidance content as audible guidance to facilitate a user of a computer-assisted surgical system teleoperating an image sensor in a surgical space. Such an audible guidance may be provided in any suitable manner. For example, an audible notification may include a “beep,” playback of an audio clip with spoken language, and/or any other suitable audible guidance.

[0077] Additionally or alternatively, system 400 may be provide guidance content as haptic feedback guidance to facilitate a user of a computer-assisted surgical system teleoperating a robotic instrument to move an image sensor. Such haptic feedback guidance may be provided in any suitable manner. For example, system 400 may instruct a computer-assisted surgical system to cause one of the master controls of user control system 104 to vibrate to inform the user regarding where or how to move an image sensor in an imaging space.

[0078] In certain implementations, guidance content generated by system 400 may facilitate an image sensor making contact with an object, maintaining a predefined amount of contact with the object, and/or maintaining a predefined contact angle with respect to the surface of the object. Accordingly, in such examples, the guidance content may indicate at least one of a contact pressure or a contact angle of an image sensor with respect to a surface of an object and/or may indicate one or more operations to be performed to obtain and/or maintain a certain contact angle and/or contact pressure (e.g., within certain ranges of contact angles and/or contact pressures) between the image sensor and the object.

[0079] In certain examples, guidance content may facilitate teleoperation of an image sensor to capture images during a surgical procedure prior to, during, or after calibration. For example, system 400 may generate any suitable guidance content to be provided by way of a computer-assisted surgical system to facilitate teleoperation of image sensor 302-2 with respect to kidney 602 to obtain one or more ultrasound images. For example, system 400 may obtain a first parameter that indicates a relative position between image sensor 302-2 and kidney 602, a second parameter that indicates an attribute (e.g., a signal strength) of images captured by image sensor 302- 2, a third parameter that indicates contact pressure between image sensor 302-2 and kidney 602, and/or any other suitable parameter to determine whether image sensor 302-2 is in contact with kidney 602. If, based on such parameters, system 400 determines that image sensor 302-2 is not in contact with kidney 602, system 400 may generate any suitable guidance content that instructs the user to move image sensor 302-2 towards kidney 602 to capture ultrasound images. For example, system 400 may generate a text notification to be provided for display that instructs a user to move image sensor 302-2 toward kidney 602. If system 400 determines that image sensor 302-2 is in contact with kidney 602 but not at enough contact pressure, system 400 may generate a visual notification in the form of, for example, a downwardly oriented arrow icon overlaid over an image of the surgical space indicating that the user needs to increase the contact pressure of image sensor 302-2 with respect to a surface of kidney 602. If system 400 determines that image sensor 302-2 is in contact with kidney 602 but at too much contact pressure, system 400 may generate a visual notification in the form of, for example, an upwardly oriented arrow icon indicating that the user needs to increase the contact pressure of image sensor 302-2 with respect to a surface of kidney 602. System 400 may generate any other suitable guidance content in other implementations.

[0080] In certain examples, guidance content generated by system 400 may include a motion path for an image sensor to follow during a scanning movement in an imaging space while the image sensor is engaged by a robotic instrument. In certain examples, system 400 may be configured to generate a plurality of motion paths for an image sensor to follow in an imaging space. For example, a first motion path may start at a current position of an image sensor in an imaging space and may end at a first position on a surface of an object in the imaging space. A second motion path may start at the first position on the surface of the object and extend to a second position on the surface of the object.

[0081] System 400 may generate guidance content in the form of a motion path in any suitable manner. For example, in certain implementations, system 400 may automatically generate a motion path based on one or more parameters associated with an imaging space. For example, system 400 may analyze image 600 shown in FIG. 6 and determine based on image 600, the presence of kidney 602, image sensor 302-2, and/or any other suitable parameters associated with the imaging space, that a procedural context is associated with image sensor 302-2 capturing images of kidney 602. Based on such a procedural context, system 400 may automatically generate guidance content in the form of a motion path for a robotic instrument to follow, without requiring that the user provide further input.

[0082] In certain alternative examples, system 400 may generate a motion path for an image sensor to follow for a scanning movement based on input provided by a user. To that end, system 400 may be configured to facilitate a user defining at least some portions of a motion path prior to system 400 generating the motion path. System 400 may facilitate a user providing input to define at least a part of a motion path in any suitable manner. For example, system 400 may facilitate a user defining a start position of a motion path and a stop position of the motion path. System 400 may facilitate a user selecting the start position and the stop position in any suitable manner. For example, a user (e.g., surgeon 110-1) may be able to move a cursor by manipulating master controls of user control system 104 to position virtual pointers with respect to an object in a surgical space to define the start position and the stop position. Alternatively, a user (e.g., assistant 110-2) may define virtual pointers through any suitable input (e.g., mouse cursor input, touch input, etc.) entered by way of any suitable display (e.g., display monitor 114) associated with a computer-assisted surgical system.

[0083] To illustrate an example, FIGS. 8A and 8B show images 800 (e.g., images 800-1 and 800-2) that may be provided for display when virtual pointers are used as guidance content to facilitate defining a motion path for a scanning movement. As shown in FIG. 8A, image 800-1 includes a first virtual pointer 802-1 that is indicative of a start point of a motion path and a second virtual pointer 802-2 that is indicative of an end point of a motion path. A user may designate the position of virtual pointers 802 in any suitable manner. For example, a user may provide any suitable user input by way of master controls of user control system 104 to select the position of first virtual pointer 802-1. System 400 may use depth data associated with the surgical space to project first virtual pointer 802-1 so as to virtually appear at a depth position within the surgical space that corresponds to a surface of kidney 602. Similarly, the user may provide any suitable user input by way of master controls of user control system 104 to select the position of second virtual pointer 802-2. System 400 may use depth data associated with the surgical space to project second virtual pointer 802-2 so as to virtually appear at a depth position within the surgical space that corresponds to a surface of kidney 602.

[0084] Virtual pointers 802 may be provided for display to a user by way of a user interface in any suitable manner. For example, virtual pointers may be provided for display as overlays over a stereoscopic image displayed by user control system 104. [0085] In the example shown in FIGS. 8A and 8B, system 400 may use virtual pointers 802, depth data associated with the surgical space, and/or any other suitable information as parameters to facilitate generating a motion path 804 shown in FIG. 8B for image sensor 302-2 to follow during a scanning movement. This may be accomplished in any suitable manner. For example, system 400 may use depth data to generate a smooth curve between first virtual pointer 802-1 and second virtual pointer 802-2 that follows the surface of kidney 602. System 400 may then project the generated smooth curve onto a depth map of the surgical space to generate motion path 804. In such examples, system 400 may be configured to discard depth outliers due to, for example, specular reflection or other visual effects to maintain a smooth curve for motion path 804. In so doing, motion path 804 may facilitate image sensor 302-2 being moved, via teleoperation, optimally with respect to kidney 602 during a scanning movement.

[0086] In the example shown in FIGS. 8A and 8B, virtual pointers 802 are shown as “X”s. However, it is understood that virtual pointers may be represented by other shapes, icons, graphical objects, etc. in other implementations. In addition, motion path 804 is shown in FIG. 8B as a dotted line for illustrative purposes. It is understood that in certain implementations, guidance content provided to facilitate guided teleoperation of an image sensor may not include specifically displaying a motion path to a user.

[0087] In addition to a graphical depiction being provided as part of guidance content, system 400 may provide additional guidance content associated with a motion path, such as by concurrently providing additional guidance content to facilitate an image sensor moving along a motion path. For example, in certain implementations such additional guidance content may include providing a notification to the user of a computer-assisted surgical system that requests user confirmation that the motion path indicated by, for example, a graphical depiction is acceptable. Such a notification may be provided to a user in any suitable manner. For example, system 400 may access an audible notification from a storage device associated with a computer-assisted surgical system. System 400 may instruct the computer-assisted surgical system to display the graphical depiction of the motion path and playback an audio clip with the expression “please confirm that the motion path is acceptable.” The user may then visually examine the motion path represented by the graphical depiction to determine whether the motion path is free of obstructions and/or is otherwise acceptable. If the user determines that the motion path is acceptable, the user may provide any suitable response to the audio clip. For example, the user may say “yes” out loud to indicate that the motion path represented by the graphical depiction is acceptable. In such an example, system 400 may use any suitable speech recognition algorithm to detect the response of the user. Additionally or alternatively, system 400 may access any suitable text notification that a computer-assisted surgical system may provide for display to a user to request user confirmation that a motion path is acceptable.

[0088] Additionally or alternatively, guidance content provided by system 400 may include content that facilitates a user moving an image sensor along a motion path. For example, in certain implementations, system 400 may be configured to provide virtual guidance to facilitate a user moving an image sensor along a motion path. In certain examples, such virtual guidance may include system 400 providing haptic feedback guidance. Such haptic feedback guidance may be provided in any suitable manner. For example, such haptic feedback guidance may correspond to a virtual fixture such as a haptic feedback tunnel in the surgical space that is configured to guide control of the image sensor and/or the robotic instrument engaging the image sensor along a motion path in the surgical space. With such a haptic feedback tunnel, as the user moves an image sensor along a motion path, system 400 may provide haptic feedback in the form of vibration of the master controls of user control system 104 whenever the image sensor and/or the robotic instrument engaging the image sensor deviates from the motion path by more than some predefined threshold amount.

[0089] In certain implementations, while a user moves a robotic instrument within a surgical space to control, for example, a grasped image sensor, system 400 may provide guidance content to automatically adjust a pose of the image sensor to improve performance of a surgical procedure. For example, system 400 may perform an imagebased visual servoing operation in certain examples to automatically make adjustments to the pose of an image sensor to improve image quality. Such an image-based visual servoing operation may help ensure that an image sensor such as a drop-in ultrasound probe maintains a desired position and/or orientation with respect to an object in the surgical space. In certain examples, the maintaining of a desired position and/or orientation may include maintaining an amount of pressure and/or a desired contact angle with respect to an object in a surgical space (e.g., to capture adequate images).

[0090] FIG. 9 illustrates an example method of calibrating an image sensor in relation to a robotic instrument. While FIG. 9 illustrates example operations according to one embodiment, other embodiments may omit, add to, reorder, and/or modify any of the operations shown in FIG. 9. One or more of the operations shown in FIG. 9 may be performed by a system such as system 400, any components included therein, and/or any implementation thereof.

[0091] At operation 902, a calibration system (e.g., calibration system 400) may obtain scanning data collected during a scanning movement. The scanning movement may correspond to a movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space. The scanning data may include images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement. Operation 902 may be performed in any of the ways described herein.

[0092] At operation 904, the calibration system may determine, based on images captured by an image sensor during the scanning movement, first motion data for the image sensor. Operation 904 may be performed in any of the ways described herein. [0093] At operation 906, the calibration system may determine, based on tracking data for a robotic instrument, second motion data for the robotic instrument. Operation 906 may be performed in any of the ways described herein. [0094] At operation 908, the calibration system may determine, based on the first motion data and the second motion data, a calibration of the image sensor in relation to the robotic instrument in the imaging space. Operation 908 may be performed in any of the ways described herein.

[0095] In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided in accordance with the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and/or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and/or transmitted using any of a variety of known computer-readable media.

[0096] A non-transitory computer-readable medium as referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that may be read and/or executed by a computing device (e.g., by a processor of a computing device). For example, a non-transitory computer-readable medium may include, but is not limited to, any combination of non-volatile storage media and/or volatile storage media. Illustrative non-volatile storage media include, but are not limited to, read-only memory, flash memory, a solid-state drive, a magnetic storage device (e.g., a hard disk, a floppy disk, magnetic tape, etc.), ferroelectric random-access memory (“RAM”), and an optical disc (e.g., a compact disc, a digital video disc, a Blu-ray disc, etc.). Illustrative volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0097] FIG. 10 illustrates an example computing device 1000 that may be specifically configured to perform one or more of the processes described herein. As shown in FIG. 10, computing device 1000 may include a communication interface 1002, a processor 1004, a storage device 1006, and an input/output (“I/O”) module 1008 communicatively connected one to another via a communication infrastructure 1010. While an example computing device 1000 is shown in FIG. 10, the components illustrated in FIG. 10 are not intended to be limiting. Additional or alternative components may be used in other embodiments. Components of computing device 1000 shown in FIG. 10 will now be described in additional detail.

[0098] Communication interface 1002 may be configured to communicate with one or more computing devices. Examples of communication interface 1002 include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio/video connection, and any other suitable interface.

[0099] Processor 1004 generally represents any type or form of processing unit capable of processing data and/or interpreting, executing, and/or directing execution of one or more of the instructions, processes, and/or operations described herein.

Processor 1004 may perform operations by executing computer-executable instructions 1012 (e.g., an application, software, code, and/or other executable data instance) stored in storage device 1006.

[0100] Storage device 1006 may include one or more data storage media, devices, or configurations and may employ any type, form, and combination of data storage media and/or device. For example, storage device 1006 may include, but is not limited to, any combination of the non-volatile media and/or volatile media described herein. Electronic data, including data described herein, may be temporarily and/or permanently stored in storage device 1006. For example, data representative of computer-executable instructions 1012 configured to direct processor 1004 to perform any of the operations described herein may be stored within storage device 1006. In some examples, data may be arranged in one or more databases residing within storage device 1006.

[0101] I/O module 1008 may include one or more I/O modules configured to receive user input and provide user output. One or more I/O modules may be used to receive input for a single virtual experience. I/O module 1008 may include any hardware, firmware, software, or combination thereof supportive of input and output capabilities. For example, I/O module 1008 may include hardware and/or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., touchscreen display), a receiver (e.g., an RF or infrared receiver), motion sensors, and/or one or more input buttons.

[0102] I/O module 1008 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I/O module 1008 is configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and/or any other graphical content as may serve a particular implementation.

[0103] In some examples, any of the systems, computing devices, and/or other components described herein may be implemented by computing device 1000. For example, memory 402 may be implemented by storage device 1006, and processor 404 may be implemented by processor 1004.

[0104] In the preceding description, various example embodiments have been described with reference to the accompanying drawings. It will, however, be evident that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the claims that follow. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. The description and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.

Claims

CLAIMS What is claimed is:

1. A system comprising: a memory storing instructions; and one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: obtaining scanning data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scanning data including images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement; determining, based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor; determining, based on the tracking data for the robotic instrument, second motion data for the robotic instrument; and determining, based on the first motion data and the second motion data, a calibration of the image sensor in relation to the robotic instrument in the imaging space.

2. The system of claim 1 , wherein the tracking data includes kinematics data indicative of at least one of a position or an orientation of the robotic instrument during the scanning movement.

3. The system of claim 1 , wherein the determining of the first motion data includes determining a spatial relationship between the images captured by the image sensor using temporal changes of a pattern in the captured images.

4. The system of claim 3, wherein the determining of the spatial relationship includes using at least one of speckle decorrelation or machine learning operations.

5. The system of claim 1 , wherein the determining of the calibration includes determining a calibration transform between the first motion data and the second motion data.

6. The system of claim 1 , wherein the image sensor is physically coupled to the robotic instrument by being grasped by the robotic instrument.

7. The system of claim 6, wherein the process further comprises: detecting a change in a grasp state of the image sensor in relation to the robotic instrument; obtaining, based on the change, additional scanning data collected during an additional scanning movement in which the image sensor physically coupled to the robotic instrument is moved within the imaging space, the additional scanning data including additional images captured by the image sensor during the additional scanning movement and additional tracking data for the robotic instrument during the additional scanning movement; determining, based on the additional images captured by the image sensor during the additional scanning movement, third motion data for the image sensor; determining, based on the additional tracking data for the robotic instrument, fourth motion data for the robotic instrument; and determining, based on the third motion data and the fourth motion data, an updated calibration of the image sensor in relation to the robotic instrument in the imaging space.

8. The system of claim 1 , wherein: the image sensor is a drop-in ultrasound probe; and the images include ultrasound images captured by the drop-in ultrasound probe during the scanning movement.

9. The system of claim 8, wherein the process further comprises generating an ultrasound reconstructed volume.

10. The system of claim 9, wherein the ultrasound reconstructed volume is generated based on the calibration.

11. The system of claim 9, wherein the scanning movement is performed in parallel with the generating of the ultrasound reconstructed volume.

12. The system of claim 9, wherein: the ultrasound reconstructed volume is a previously generated ultrasound reconstructed volume; and the process further comprises refining, based on the calibration, the previously generated ultrasound reconstructed volume.

13. The system of claim 1 , wherein the process further comprises generating a motion path for the robotic instrument to follow during the scanning movement.

14. The system of claim 13, wherein: the robotic instrument is coupled to a computer-assisted surgical system; and the process further comprises assisting a user of the computer-assisted surgical system in moving the robotic instrument along the motion path.

15. The system of claim 14, wherein the process further comprises providing at least one of haptic feedback, audible feedback, or visual feedback to assist the user in moving the robotic instrument along the motion path.

16. The system of claim 13, wherein: the process further comprises facilitating a user of a computer-assisted surgical system defining a start position of the motion path and a stop position of the motion path; and the generating of the motion path is based on the start position and the stop position defined by the user.

17. The system of claim 13, wherein the process further comprises directing a computer-assisted surgical system to automatically move the robotic instrument along the motion path.

18. The system of claim 1 , wherein the process further comprises: obtaining endoscopic images of the imaging space; registering, based on the calibration, the images captured by the image sensor with the endoscopic images; and generating, based on the registering, an augmented image or a composite image associated with the imaging space.

19. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for performing a process comprising: obtaining scanning data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scanning data including images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement; determining, based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor; determining, based on the tracking data for the robotic instrument, second motion data for the robotic instrument; and determining, based on the first motion data and the second motion data, a calibration of the image sensor in relation to the robotic instrument in the imaging space.

20. The computer program product of claim 19, wherein the tracking data includes kinematics data indicative of at least one of a position or an orientation of the robotic instrument during the scanning movement.

21. The computer program product of claim 19, wherein the determining of the first motion data includes determining a spatial relationship between the images captured by the image sensor using temporal changes of a pattern in the captured images.

22. The computer program product of claim 21 , wherein the determining of the spatial relationship includes using at least one of speckle decorrelation or machine learning operations.

23. The computer program product of claim 19, wherein the determining of the calibration includes determining a calibration transform between the first motion data and the second motion data.

24. The computer program product of claim 19, wherein the image sensor is physically coupled to the robotic instrument by being grasped by the robotic instrument.

25. The computer program product of claim 24, wherein the process further comprises: detecting a change in a grasp state of the image sensor in relation to the robotic instrument; obtaining, based on the change, additional scanning data collected during an additional scanning movement in which the image sensor physically coupled to the robotic instrument is moved within the imaging space, the additional scanning data including additional images captured by the image sensor during the additional scanning movement and additional tracking data for the robotic instrument during the additional scanning movement; determining, based on the additional images captured by the image sensor during the additional scanning movement, third motion data for the image sensor; determining, based on the additional tracking data for the robotic instrument, fourth motion data for the robotic instrument; and determining, based on the third motion data and the fourth motion data, an updated calibration of the image sensor in relation to the robotic instrument in the imaging space.

26. The computer program product of claim 19, wherein: the image sensor is a drop-in ultrasound probe; and the images include ultrasound images captured by the drop-in ultrasound probe during the scanning movement.

27. The computer program product of claim 26, wherein the process further comprises generating an ultrasound reconstructed volume.

28. The computer program product of claim 27, wherein the ultrasound reconstructed volume is generated based on the calibration.

29. The computer program product of claim 27, wherein the scanning movement is performed in parallel with the generating of the ultrasound reconstructed volume.

30. The computer program product of claim 27, wherein: the ultrasound reconstructed volume is a previously generated ultrasound reconstructed volume; and the process further comprises refining, based on the calibration, the previously generated ultrasound reconstructed volume.

31. The computer program product of claim 19, wherein the process further comprises generating a motion path for the robotic instrument to follow during the scanning movement.

32. The computer program product of claim 31 , wherein: the robotic instrument is coupled to a computer-assisted surgical system; and the process further comprises assisting a user of the computer-assisted surgical system in moving the robotic instrument along the motion path.

33. The computer program product of claim 32, wherein the process further comprises providing at least one of haptic feedback, audible feedback, or visual feedback to assist the user in moving the robotic instrument along the motion path.

34. The computer program product of claim 31 , wherein: the process further comprises facilitating a user of a computer-assisted surgical system defining a start position of the motion path and a stop position of the motion path; and the generating of the motion path is based on the start position and the stop position defined by the user.

35. The computer program product of claim 31 , wherein the process further comprises directing a computer-assisted surgical system to automatically move the robotic instrument along the motion path.

36. The computer program product of claim 19, wherein the process further comprises: obtaining endoscopic images of the imaging space; registering, based on the calibration, the images captured by the image sensor with the endoscopic images; and generating, based on the registering, an augmented image or a composite image associated with the imaging space.

37. A method comprising: obtaining, by a calibration system, scanning data collected during a scanning movement in which an image sensor physically coupled to a robotic instrument is moved within an imaging space, the scanning data including images captured by the image sensor during the scanning movement and tracking data for the robotic instrument during the scanning movement; determining, by the calibration system and based on the images captured by the image sensor during the scanning movement, first motion data for the image sensor; determining, by the calibration system and based on the tracking data for the robotic instrument, second motion data for the robotic instrument; and determining, by the calibration system and based on the first motion data and the second motion data, a calibration of the image sensor in relation to the robotic instrument in the imaging space.

38. The method of claim 37, wherein the tracking data includes kinematics data indicative of at least one of a position or an orientation of the robotic instrument during the scanning movement.

39. The method of claim 37, wherein the determining of the first motion data includes determining a spatial relationship between the images captured by the image sensor using temporal changes of a pattern in the captured images.

40. The method of claim 39, wherein the determining of the spatial relationship includes using at least one of speckle decorrelation or machine learning operations.

41. The method of claim 37, wherein the determining of the calibration includes determining a calibration transform between the first motion data and the second motion data.

42. The method of claim 37, wherein the image sensor is physically coupled to the robotic instrument by being grasped by the robotic instrument.

43. The method of claim 42, further comprising: detecting, by the calibration system, a change in a grasp state of the image sensor in relation to the robotic instrument; obtaining, by the calibration system and based on the change, additional scanning data collected during an additional scanning movement in which the image sensor physically coupled to the robotic instrument is moved within the imaging space, the additional scanning data including additional images captured by the image sensor during the additional scanning movement and additional tracking data for the robotic instrument during the additional scanning movement; determining, by the calibration system and based on the additional images captured by the image sensor during the additional scanning movement, third motion data for the image sensor; determining, by the calibration system and based on the additional tracking data for the robotic instrument, fourth motion data for the robotic instrument; and determining, by the calibration system and based on the third motion data and the fourth motion data, an updated calibration of the image sensor in relation to the robotic instrument in the imaging space.

44. The method of claim 37, wherein: the image sensor is a drop-in ultrasound probe; and the images include ultrasound images captured by the drop-in ultrasound probe during the scanning movement.

45. The method of claim 44, further comprising generating, by the calibration system, an ultrasound reconstructed volume.

46. The method of claim 45, wherein the ultrasound reconstructed volume is generated based on the calibration.

47. The method of claim 45, wherein the scanning movement is performed in parallel with the generating of the ultrasound reconstructed volume.

48. The system of claim 45, wherein: the ultrasound reconstructed volume is a previously generated ultrasound reconstructed volume; and the method further comprises refining, based on the calibration, the previously generated ultrasound reconstructed volume.

49. The method of claim 37, further comprising generating a motion path for the robotic instrument to follow during the scanning movement.

50. The method of claim 49, wherein: the robotic instrument is coupled to a computer-assisted surgical system; and the method further comprises assisting a user of the computer-assisted surgical system in moving the robotic instrument along the motion path.

51. The method of claim 50, further comprising providing at least one of haptic feedback, audible feedback, or visual feedback to assist the user in moving the robotic instrument along the motion path.

52. The method of claim 49, wherein: the method further comprises facilitating a user of a computer-assisted surgical system defining a start position of the motion path and a stop position of the motion path; and the generating of the motion path is based on the start position and the stop position defined by the user.

53. The method of claim 49, further comprising directing a computer-assisted surgical system to automatically move the robotic instrument along the motion path.

54. The method of claim 37, further comprising: obtaining endoscopic images of the imaging space; registering, based on the calibration, the images captured by the image sensor with the endoscopic images; and generating, based on the registering, an augmented image or a composite image associated with the imaging space.