WO2025199062A1

WO2025199062A1 - Visualization views for lung biopsy and treatment procedures

Info

Publication number: WO2025199062A1
Application number: PCT/US2025/020290
Authority: WO
Inventors: Daniel OVADIA; Ruth LIBKIND; Guy Alexandroni; Patrick L. Lukasak; Irina SHEVLEV
Original assignee: Covidien LP
Current assignee: Covidien LP
Priority date: 2024-03-17
Filing date: 2025-03-17
Publication date: 2025-09-25
Anticipated expiration: 2026-09-17

Abstract

Systems and methods for planning a lung navigation pathway including a computing device including a processor and a memory, the memory storing therein an application that when executed by the processor, accesses image data stored in the memory, analyzes the image data to identify a pleura, a tumor, and airways, and determine a pathway through the airways through the airways to the tumor, generates a 3D model of the lungs, the 3D model including a representation of the pleura, the airways, the tumor, and the pathway through the airways to the tumor, wherein the identification of the pleura, the tumor, the airways, and the determination of the pathway is executed by one or more neural network algorithms associated with the application, and presents the 3D model and pathway in a user interface on a display in communication with the computing device.

Description

VISUALIZATION VIEWS FOR LUNG BIOPSY AND TREATMENT PROCEDURES BACKGROUND

Technical Field

[0001] This disclosure relates to the field of endoluminal navigation and particularly to visualization of navigation of pathways and targets within the lungs of a patient.

Description of Related Art

[0002] There are several commonly applied medical methods, such as endoscopic procedures or minimally invasive procedures, for treating various maladies affecting organs including the liver, brain, heart, lungs, gall bladder, kidneys, and bones. Often, one or more imaging modalities, such as magnetic resonance imaging (MRI), positron emissions tomography, (PET) ultrasound imaging, computed tomography (CT), cone-beam computed tomography (CBCT), or fluoroscopy may be employed by clinicians to identify and navigate to areas of interest within a patient and ultimately a target for biopsy or treatment. In some procedures, pre-operative scans may be utilized for target identification and intraoperative guidance. Real-time imaging may also be used to obtain a more accurate and current image of the target area. Furthermore, real-time image data displaying the current location of a medical device with respect to the target and its surroundings may be used to navigate the medical device to the target in a safe and accurate manner (e.g., without causing damage to other organs or tissue).

[0003] For example, an endoscopic approach may be useful in navigating to areas of interest within a patient, and particularly so for areas within luminal networks of the body such as the lungs, blood vessels, colorectal cavities, and the renal ducts. To enable the endoscopic approach, navigation systems may use previously acquired MRI data or CT image data to generate a three-dimensional (3D) rendering, model, or volume of a particular body part.

[0004] The resulting volume generated from the MRI scan or CT scan may be utilized to create a navigation plan to facilitate the advancement of a navigation catheter (or other suitable medical device) through a bronchoscope and a branch of the bronchus of a patient to an area of interest. A locating or tracking system, such as, for example, an electromagnetic (EM) tracking system, may be utilized in conjunction with, for example, CT data, to facilitate guidance of the navigation catheter through the branch of the bronchus to the area of interest. In certain instances, the navigation catheter may be positioned within one of the airways of the branched luminal networks adjacent to, or within, the area of interest to provide access for one or more medical instruments.

[0005] As will be appreciated, accurate placement of the catheter and therewith the medical instrument is important to ensure successful therapy. Improvements to the current navigation catheter systems are desired.

SUMMARY

[0006] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions. One general aspect of the disclosure includes a system for planning a lung navigation pathway a computing device including a processor and a memory, the memory storing therein an application that when executed by the processor: accesses image data stored in the memory; analyzes the image data to identify a pleura, a tumor, and airways, and determine a pathway through the airways through the airways to the tumor; generates a 3D model of the lungs, the 3D model including a representation of the pleura, the airways, the tumor, and the pathway through the airways to the tumor, where the identification of the pleura, the tumor, the airways, and the determination of the pathway is executed by one or more neural network algorithms associated with the application; and presents the 3D model and pathway in a user interface on a display in communication with the computing device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0007] Implementations may include one or more of the following features. The system where the application, when executed by the processor identifies a geometric center of the tumor as a first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface, where the pathway. The application when executed by the processor receives a second biopsy location, determines a pathway through the airways to the tumor to access the second biopsy location, displays an indicator of the second biopsy location and the pathway in the user interface. The application determines multiple pathways to each biopsy location and presents a score for each pathway in the user interface. The level of detail of the 3D model is adjusted based on a level of zoom. The application when executed by the processor displays the determined pathway on live endoluminal images. The application when executed by the processor generates a timeline view of the lung navigation pathway. The timeline view includes a representation of each bifurcation of the airways along the pathway. The timeline view depicts an indicator of a location of a catheter being navigated along the pathway. The timeline view depicts a representation of the tumor and a biopsy location within the tumor. A portion of the timeline changes color as navigation proceeds. The system may include a timer indicating a duration of a procedure. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0008] A further aspect of the disclosure includes a system including includes a catheter configured for navigation within lungs of a patient; a tracking system configured for detecting a location of the catheter; and a computing device including a processor and a memory, the memory storing therein an application that when executed by the processor: accesses image set stored in the memory; analyzes the image data to identify a pleura, a tumor, and airways, and determine a pathway through the airways through the airways to the tumor; generates a 3D model of the lungs, the 3D model including a representation of the pleura, a representation of the airways, a representation of the tumor, and a representation of the pathway through the airways to the tumor; and presents the 3D model and pathway in a user interface on a display in communication with the computing device, where the identification of the pleura, identification of the tumor, identification of the airways, and the determination of the pathway is executed by one or more neural network algorithms associated with the application. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0009] Implementations may include one or more of the following features. The system where the application when executed by the processor depicts an exit angle cone on the tumor. The application, when executed by the processor displays or removes elements of the 3D model based on a level of zoom. The application, when executed by the processor, displays the pathway on live bronchoscopic images in the user interface. The application when executed by the processor generates and presents a target alignment view of the 3D model and a live fluoroscopic image in the user interface. The application when executed by the processor generates a timeline view of the pathway, where the timeline view includes a representation of each bifurcation of the airways along the pathway. The application, when executed by the processor displays tumor dimensional data on the user interface. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Various aspects and embodiments of the disclosure are described hereinbelow with references to the drawings, wherein:

[0011] FIG. 1A is a schematic view of a robotic surgical system in accordance with the disclosure.

[0012] FIG. IB is a schematic view of a luminal network navigation system in accordance with the disclosure;

[0013] FIG. 2 is a 3D model generated in accordance with the disclosure;

[0014] FIG. 3 is a user interface for planning an endoluminal navigation in accordance with the disclosure;

[0015] FIG. 4 is a user interface for planning an endoluminal navigation to a selected tumor in accordance with the disclosure;

[0016] FIG. 5 is a user interface depicting a planned pathway through a 3D model to a tumor in accordance with the disclosure;

[0017] FIG. 6A is a user interface showing a planned pathway zoomed-in to the area around the tumor in accordance with the disclosure;

[0018] FIG. 6B depicts a pop-out feature of the user interface in accordance with the disclosure;

[0019] FIG. 7 is a user interface of an endoluminal navigation in accordance with the disclosure;

[0020] FIG. 8 is a user interface of an endoluminal navigation indicating a recommendation of local registration in accordance with the disclosure;

[0021] FIG. 9 depicts a timeline feature which is incorporated on the user interface of FIGS 7 and 8 in accordance with the disclosure;

[0022] FIG. 10 is a user interface for target alignment in accordance with the disclosure; and

[0023] FIG. 11 is a schematic depiction of a system for generating and displaying the user interfaces and conducting aspects of the endoluminal navigation in accordance with the disclosure. DETAILED DESCRIPTION

[0024] Catheters and catheter-like devices, such as endoscopes, biopsy tools, and treatment tools, are used in a myriad of endoluminal medical procedures. These flexible devices are navigated through luminal networks of the body including the vasculature, airways, and digestive systems. In accordance with one aspect of the disclosure, preprocedural, or intra-procedural images (e.g., CT or PET, MRI, or CBCT images) may be automatically analyzed with a neural network algorithm to develop a deep airway tree (3D model of the airways). The neural network algorithms enable a pixel-by-pixel segmentation identifying and distinguishing fluid, air, luminal tissue (e.g., the tissue of the airways), bone, and others. A deep airway tree refers to a 3D model of the luminal network (e.g., airways) which approaches the pleura boundary encompassing substantially all the luminal network. Many 3D modeling applications not employing neural networks tend to output shallower 3D models (e.g., modeling only the larger airways) which can result in an inferior or an incomplete 3D model.

[0025] A further aspect of the disclosure is directed to use of the neural network algorithms in identifying a tumor or lesion within the pre-procedural or intra-procedural images. Neural network algorithms are trained specifically to identify tumors and lesions within the pre-procedural or intra-procedural images. The neural networks analyze the pre- procedural or intra-procedural images on a pixel-by-pixel basis to separate tumors and lesions from other tissue or air based on a value (e.g., Hounsfield unit) of each pixel. Groups of pixels having a common or substantially common value are grouped and compared to expected structures (e.g., trachea and carina) and shapes (e.g., tubes of varying diameters) within the pre-procedural or intra-procedural images to distinguish air within the airways, the airways themselves, healthy tissue, and the diseased tissue (e.g., tumors or lesions).

[0026] Yet a further aspect of the disclosure is directed to neural network algorithms that generate a pathway within the luminal network from the identified tumor or lesion back to a starting location for navigation (e.g., the trachea of the lungs). To determine a pathway the neural network can consider a variety of factors including proximity of luminal structures to the tumor or lesion, angles at which a tool (e.g., biopsy tool) exits the luminal network to reach the target, radii of the luminal network (e.g., increasing radius along a pathway for target to starting point), tissue densities, mechanical properties of biopsy tools and therapeutic tools, and others. [0027] Another feature of the disclosure is directed to a user interface displaying zoombased filtering. Zoom-based filtering changes the level of detail of a 3D model displayed based on where, within the luminal network, the navigation has progressed. Where appropriate, the entire 3D model and one or more targets may be displayed on the user interface. As planning or navigation continues, a portion of the 3D model is displayed in greater detail and may include depictions the portion of the pathway already navigated, more details of the target, and for example other physical structures of the patient in proximity to the target (e.g., the pleura or blood vessels). Once in proximity of the target further details are depicted in the user interface including further details of the proximity of the pleura boundaries, blood vessels, relative position of a pathway for a catheter or other tool with respect to the target and other details. Changing the details of the 3D model displayed on the user interface reduces the cognitive load allowing a user to focus on only what is necessary based on the position of the catheter within the luminal network. These and other aspects of the disclosure are described in greater detail below.

[0028] Still another aspect of the disclosure is directed to a user interface that enhances visualization features and promotes efficient procedures. In a first feature of this aspect of the disclosure, the pathway generated by the neural network algorithm is displayed on live intraluminal (e.g., endoscopic or bronchoscopic) images acquired during the navigation of a catheter or other tool. A second feature of the user interface is a schematic navigation view. As will be appreciated, luminal navigation can occasionally be disorienting for the clinician and assessing exactly where a catheter is within the luminal network is not always readily apparent. The schematic navigation view provides a timeline view depicting the navigational pathway to reach the target (e.g., a tumor or lesion) and the progress of the navigation along the pathway as well as other features.

[0029] Still another aspect of the disclosure is the incorporation of the above aspects into a robotic surgical system, wherein the catheter is robotically drive to one or more targets within the patient.

[0030] FIG. 1A depicts a robotic surgical system 10 including a control tower 20, which is connected to all of the components of the robotic surgical system 10 including a surgeon console 30 and one or more mobile carts 60. Each of the mobile carts 60 includes a robotic arm 40 having a surgical instrument 50 removably coupled thereto. The robotic arms 40 also couple to the mobile carts 60. The robotic surgical system 10 may include any number of mobile carts 60 and/or robotic arms 40. [0031] The surgical instrument 50 is configured for use during minimally invasive surgical procedures (e.g., laparoscopic) or for catheter based intraluminal procedures (described in greater detail in connection with FIG. IB). The surgical instrument 50 may include an end effector 49 such as an electrosurgical forceps configured to seal tissue by compressing tissue between jaw members and applying electrosurgical current thereto, a surgical stapler including a pair of jaws configured to grasp and clamp tissue while deploying a plurality of tissue fasteners, e.g., staples, and cutting stapled tissue, a surgical clip applier including a pair of jaws configured apply a surgical clip onto tissue or other end effectors without departing from the scope of the disclosure. Alternatively, the surgical instrument 50 may include a robotic actuated catheter including an articulation mechanism (e.g., one or more pull-wires or tendons) effective to alter the shape of the catheter. In addition, the robotic arm 40 can be employed to advance or retract the catheter during a laparoscopic or endoluminal (or combined) procedure.

[0032] One of the robotic arms 40 may include a laparoscopic camera 51 configured to capture video of the surgical site. The laparoscopic camera 51 may be a stereoscopic endoscope configured to capture two side-by-side (i.e., left and right) images of the surgical site to produce a video stream of the surgical scene. The laparoscopic camera 51 is coupled to an image processing device, which may be disposed within the control tower 20. The image processing device may be any computing device configured to receive the image feed from the laparoscopic camera 51 and output the processed images or video stream.

[0033] The surgeon console 30 includes a first screen 32, which displays a video feed of the surgical site provided by camera 51 of the surgical instrument 50 disposed on the robotic arm 40, and a second screen 34, which displays a user interface for controlling the robotic surgical system 10. The first screen 32 and second screen 34 may be touchscreens allowing for displaying various graphical user inputs.

[0034] The surgeon console 30 also includes a plurality of user interface devices, such as foot pedals 36 and a pair of hand controllers 38a and 38b which are used by a user to remotely control robotic arms 40. The surgeon console further includes an armrest 33 used to support clinician's arms while operating the hand controllers 38a and 38b.

[0035] The control tower 20 includes a screen 23, which may be a touchscreen, and outputs on the graphical user interfaces (GUIs). The control tower 20 also acts as an interface between the surgeon console 30 and one or more robotic arms 40. In particular, the control tower 20 is configured to control the robotic arms 40, such as to move the robotic arms 40 and the corresponding surgical instrument 50, based on a set of programmable instructions and/or input commands from the surgeon console 30, in such a way that robotic arms 40 and the surgical instrument 50 execute a desired movement sequence in response to input from the foot pedals 36 and the hand controllers 38a and 38b. The foot pedals 36 may be used to enable and lock the hand controllers 38a and 38b, repositioning camera movement and electrosurgical activation/deactivation. In particular, the foot pedals 36 may be used to perform a clutching action on the hand controllers 38a and 38b. Clutching is initiated by pressing one of the foot pedals 36, which disconnects (i.e., prevents movement inputs) the hand controllers 38a and/or 38b from the robotic arm 40 and corresponding instrument 50 or camera 51 attached thereto. This allows the user to reposition the hand controllers 38a and 38b without moving the robotic arm(s) 40 and the instrument 50 and/or camera 51. This is useful when reaching control boundaries of the surgical space.

[0036] Each of the control tower 20, the surgeon console 30, and the robotic arm 40 includes a respective computer 21, 31, 41. The computers 21, 31, 41 are interconnected to each other using any suitable communication network based on wired or wireless communication protocols. The term “network,” whether plural or singular, as used herein, denotes a data network, including, but not limited to, the Internet, Intranet, a wide area network, or a local area network, and without limitation as to the full scope of the definition of communication networks as encompassed by the present disclosure. Suitable protocols include, but are not limited to, transmission control protocol/internet protocol (TCP/IP), datagram protocol/internet protocol (UDP/IP), and/or datagram congestion control protocol (DC). Wireless communication may be achieved via one or more wireless configurations, e.g., radio frequency, optical, Wi-Fi, Bluetooth (an open wireless protocol for exchanging data over short distances, using short length radio waves, from fixed and mobile devices, creating personal area networks (PANs), ZigBee® (a specification for a suite of high level communication protocols using small, low-power digital radios based on the IEEE 122.15.4- 1203 standard for wireless personal area networks (WPANs)).

[0037] The computers 21, 31, 41 may include any suitable processor (not shown) operably connected to a memory (not shown), which may include one or more of volatile, non-volatile, magnetic, optical, or electrical media, such as read-only memory (ROM), random access memory (RAM), electrically-erasable programmable ROM (EEPROM), nonvolatile RAM (NVRAM), or flash memory. The processor may be any suitable processor (e.g., control circuit) adapted to perform the operations, calculations, and/or set of instructions described in the present disclosure including, but not limited to, a hardware processor, a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), a microprocessor, and combinations thereof. Those skilled in the art will appreciate that the processor may be substituted for by using any logic processor (e.g., control circuit) adapted to execute algorithms, calculations, and/or set of instructions described herein.

[0038] FIG. IB is a perspective view of an exemplary system for facilitating endoluminal navigation of a medical device, e.g., a catheter, to a soft-tissue target via airways of the lungs. System 100 may be further configured to construct fluoroscopic based three-dimensional volumetric data of the target area from 2D fluoroscopic images to confirm navigation to a desired location. Other intraprocedural imaging modalities may also be employed including CBCT, ultrasound, laparoscopic cameras, and others. System 100 may be further configured to facilitate the approach of a medical device to the target area by using, for example, Electromagnetic Navigation (EMN) and for determining the location of a medical device with respect to the target. The EMN system may employ a variety of sensor technologies including without limitation air-coil sensors, tunnel magnetoresistance (TMR), and others without departing from the scope of the disclosure. Though described in connection with EMN, other systems for intraluminal and lung navigation are considered within the scope of the disclosure including shape sensing technology (e.g., Fiber-bragg gratings) which detect the shape of the distal portion of the catheter and match that shape to the shape of the luminal network in a 3D model.

[0039] In FIG. IB, a catheter 102 is part of a catheter guide assembly 106. In one embodiment, catheter 102 is inserted into a bronchoscope 108 for access to a luminal network of the patient P. Specifically, catheter 102 of catheter guide assembly 106 may be inserted into a working channel of bronchoscope 108 for navigation through a patient’s luminal network. The catheter 102 may itself include imaging capabilities and the bronchoscope 108 is not strictly required. A locatable guide (LG) 110 (a second catheter), including a sensor 104 may be inserted into catheter 102 and locked into position such that sensor 104 extends a desired distance beyond the distal tip of catheter 102. The position and orientation of sensor 104 relative to a reference coordinate system, and thus the distal portion of catheter 102, within an electromagnetic field can be derived. Catheter guide assemblies 106 are currently marketed and sold by Medtronic PLC under the brand names SUPERDIMENSION® Procedure Kits, or EDGE™ Procedure Kits, and are contemplated as useable with the disclosure. Additionally or alternatively, the catheter 102 may be part of a robotic system, part of a robotic assisted system, or part of a robot and configured for performance of one or more of the methods and execution of the algorithms and applications described herein.

[0040] System 100 generally includes an operating table 112 configured to support a patient P, a bronchoscope 108 configured for insertion through patient P’s mouth into patient P’s airways; monitoring equipment 114 coupled to bronchoscope 108 or catheter 102 (e.g., a video display, for displaying the video images received from the video imaging system of bronchoscope 108 or the catheter 102); a locating or tracking system 115 including a locating module 116, a plurality of reference sensors 118 and a transmitter mat 120 including a plurality of incorporated markers; and a computing device 122 including software and/or hardware used to facilitate identification of a target, pathway planning to the target, navigation of a medical device to the target, and/or confirmation and/or determination of placement of catheter 102, or a suitable device therethrough, relative to the target.

[0041] In accordance with aspects of the disclosure, the visualization of intra-body navigation of a medical device (e.g., a catheter, LG, biopsy or therapy tool), towards a tumor or lesion, may be a portion of a larger workflow of a navigation system. An imaging device 124 capable of acquiring images or video of the patient P is also included in this particular aspect of system 100. The images, sequence of images, or video captured by imaging device 124 may be stored within imaging device 124 or transmitted to computing device 122 for storage, processing, and display. Additionally, imaging device 124 may move relative to the patient P so that images may be acquired from different angles or perspectives relative to patient P to create a sequence of images, such as a fluoroscopic video. The pose of imaging device 124 relative to patient P while capturing the images may be estimated via markers incorporated with the transmitter mat 120. The markers are positioned under patient P, between patient P and operating table 112 and between patient P and a radiation source or a sensing unit of imaging device 124. The markers incorporated with the transmitter mat 120 may be two separate elements which may be coupled in a fixed manner or alternatively may be manufactured as a single unit. Imaging device 124 may include a single imaging device or more than one imaging device. The imaging device 124 may be for example a fluoroscopic imaging, an ultrasound imaging device, or an intraprocedural CBCT imaging device.

[0042] Computing device 122 may be any suitable computing device including a processor and storage medium, wherein the processor is capable of executing instructions stored on the storage medium. Computing device 122 may further include a database configured to store patient data, image data sets including CT images, CBCT images, fluoroscopic images and video, fluoroscopic 3D reconstruction, navigation plans, and other such image data. Although not explicitly illustrated, computing device 122 may include inputs, or may otherwise be configured to receive, image data sets and other data described herein. Additionally, computing device 122 includes a display configured to display images, 3D models, and other data in one or more graphical user interfaces. Computing device 122 may be connected to one or more networks through which one or more databases (e.g., image databases) may be accessed.

[0043] For use in a navigation phase, a six degrees-of-freedom electromagnetic locating or tracking system 114, or other suitable system for determining position and orientation of a distal portion of the catheter 102, is utilized for performing registration of the images and the pathway for navigation. Tracking system 114 may include the location module 116, a plurality of reference sensors 118, and the transmitter mat 120 (including the markers). Tracking system 114 is configured for use with a locatable guide 110 and particularly sensor 104. As described above, locatable guide 110 and sensor 104 are configured for insertion through catheter 102 into patient P’s airways (either with or without bronchoscope 108) and are selectively lockable relative to one another via a locking mechanism.

[0044] Transmitter mat 120 is positioned beneath patient P. Transmitter mat 120 generates an electromagnetic field around at least a portion of the patient P within which the position of a plurality of reference sensors 118 and the sensor 104 can be determined with use of a tracking module 116. A second electromagnetic sensor 126 may also be incorporated into the end of the catheter 102. The second electromagnetic sensor 126 may be a five degree-of-freedom sensor or a six degree-of-freedom sensor. One or more of reference sensors 118 are attached to the chest of the patient P. Registration is generally performed to coordinate locations of the three-dimensional model and two-dimensional images from the planning phase, with the patient P’s airways as observed through the bronchoscope 108 and allow for the navigation phase to be undertaken with knowledge of the location of the sensor 104.

[0045] In accordance with a further aspect of the disclosure, the catheter 102 may be navigated through the luminal network of the patient P relying on the sensor 126 to without employing a locatable guide 110. In aspects, the catheter 102 is navigated using functionality from the robotic surgical system 10 described in connection with FIG. 1A. Still further, the catheter 102 may include an endoluminal camera. The endoluminal camera captures images of the endoluminal pathway as the catheter 102 is advanced towards a target. The endoluminal camera may be a permanent feature of the catheter 102, or a removable camera that is advanced into the working channel of the catheter 102 similar to the locatable guide 110. One or more light pipes may be employed to carry light to the end of the catheter 102 from a light source that remains outside of the patient, and to capture light reflected by the patient’s tissues and carry the reflected light to image forming components, also outside of the patient.

[0046] Registration of the patient P’s location on the transmitter mat 120 may be performed by moving sensor 104 through the airways of the patient P. More specifically, data pertaining to locations of sensor 104, while locatable guide 110 is moving through the airways, is recorded using transmitter mat 120, reference sensors 118, and tracking system 114. A shape resulting from this location data is compared to an interior geometry of passages of the three-dimensional model generated in the planning phase, and a location correlation between the shape and the three-dimensional model based on the comparison is determined, e.g., utilizing the software on computing device 122. In addition, the software identifies non-tissue space (e.g., air filled cavities) in the three-dimensional model. The software aligns, or registers, an image representing a location of sensor 104 with the three-dimensional model and/or two-dimensional images generated from the three-dimension model, which are based on the recorded location data and an assumption that locatable guide 110 remains located in non-tissue space in patient P’s airways. Alternatively, a manual registration technique may be employed by navigating the bronchoscope 108 with the sensor 104 to pre-specified locations in the lungs of the patient P, and manually correlating the images from the bronchoscope to the model data of the three-dimensional model.

[0047] Though described herein with respect to EMN systems using EM sensors, the instant disclosure is not so limited and may be used in conjunction with flexible sensor, ultrasonic sensors, or without sensors. Shape sensing technology (e.g., Fiber-bragg gratings) may also be used for intraluminal and lung navigation, which detect the shape of the distal portion of the catheter and match that shape to the shape of the luminal network in a 3D model.

[0048] Additionally, the methods described herein may be used in conjunction with robotic systems such that robotic actuators may drive the catheter 102 or bronchoscope 108 proximate the target. [0049] One aspect of the system 100 is a software component stored or accessible from (e.g., cloud) a computing device 122 and configured for processing computed image data. Though many aspects of image data processing have previously been performed at least partially manually, aspects of this disclosure are directed to automated image processing techniques and systems (e.g., using neural network algorithms). These aspects of the disclosure are described in greater detail below in connection with target identification and pathway planning. Some of the aspects described herein, particularly features displayed on a user interface may be presented during the planning phase or during the navigation phase of a procedure where a medical device, such as a biopsy tool or treatment tool, may be inserted into catheter 102 to obtain a tissue sample from or to treat the target.

[0050] In accordance with the disclosure, a 3D model of a luminal network (e.g., the patient’s lungs) or another suitable portion of the anatomy, may be generated from previously acquired scans, such as CT, CBCT, PET or MRI scans. Tumors and lesions within the scan data are detected and pathways through the 3D model to arrive at the tumors or lesions are generated.

[0051] Once the pathway plan is generated and accepted by a clinician, that pathway plan may be utilized by a navigation system to drive a catheter or catheter like device along the pathway plan through the anatomy and particularly the luminal network (e.g., airways) to reach the tumor or lesion. The driving of the catheter along the pathway plan may be manual or it may be robotic, or a combination of both. In a single procedure planning, registration of the pathway plan to the patient, and navigation are performed to enable a medical device, e.g., a catheter to be navigated along the planned path to reach the target or lesion, so that a biopsy or treatment of the target can be completed.

[0052] As described above, in connection with aspects of the disclosure, neural network algorithms are employed by computing device 122 to automatically analyze image data (e.g., CT, CBCT, PET, or MRI image data). The image data may be pre-procedure image data (e.g., acquired days or weeks before an endoluminal navigation procedure) or image data that is acquired intra-procedurally (e.g., 3D fluoroscopic or CBCT image data). One or more neural network algorithms analyze the image data for multiple purposes. In accordance with one aspect of the disclosure, a first purpose of the neural network algorithm is to identify and segment the pleura of a patient’s lungs in the image data. As will be appreciated, the location of the pleura is a procedurally significant portion of the patient’s anatomy. Generally, it is desirable to avoid piercing of the pleura as piercing of the pleura can lead to air (and blood) entering chest cavity and can lead to pneumothorax or other complications. Further, in instances where piercing of the pleura is unavoidable, prior knowledge of the pleura’s location and the probability of its being pierced during the procedure allows the clinician to plan for and be ready to mitigate the effects of the piercing the pleura.

[0053] Neural network algorithms are configured to analyze the image data on a pixel-by- pixel basis. This may be based on a variety of factors including the density of the tissue in each pixel of the image data. The harder or denser a tissue is, the brighter pixels of that tissue appear in an image. For example, bone tissue which is very dense will appear brighter (e.g., white) in the image data, having a higher Hounsfield value, than any other tissue. In contrast, air which has little density typically appears very dark (e.g., black) in image data. Soft tissues have varying levels of density, and thus express varying levels of brightness ranging from dense cartilaginous tissue (e.g., trachea and central airways) to less dense and elastic tissues (e.g., the alveoli) and are depicted in the images in a range of brightnesses. Using segmenting and thresholding techniques the neural network algorithms can define the pixels of the pleura as the outermost portions of the lung tissue and separate them from the pixels that make up the muscles and bones of the rib cage.

[0054] A second purpose of the neural network algorithms is to identify tumors or lesions within the image data. As with the pleura boundary, segmentation techniques are employed to define a grouping of pixels that define the tumor or lesion. The tumor or lesion will be denser than the healthy soft tissue of the lungs and will have a shape not associated with harder tissues of the lungs. Harder tissues within the lungs are generally a result of cartilaginous tissue associated with the airways themselves. By distinguishing the grouping of pixels of tissue both on density and shape false positive detections of a tumor or lesion can be avoided.

[0055] A third purpose of the neural network algorithms is to generate 3D models of the airways and in some instances blood vessels of the lungs. Airways and blood vessels have a density that is relatively uniform, or at least within a range. The neural network algorithms can detect these tissues and group like density tissues to form shapes that are generally tubular in nature. The tubular shapes, which are generally connected to a common source (e.g., trachea, pulmonary artery, pulmonary vein), by grouping the pixels together through multiple slices of the image data and in multiple direction (axial, coronal, sagittal) a 3D model of the airways and blood vessels can be generated. As noted above, through the use of neural network algorithms and their ability to detect minute differences in brightness of pixels, 3D models can be generated which reach deep into the airways and approach the pleura of the lungs.

[0056] A fourth purpose of the neural network algorithms is to identify pathways through the 3D model to the target so that a catheter 102 can be navigated through the airways to arrive at the target (e.g., a tumor or lesion). This pathway starts at the target and is defined in the direction of the trachea. In one example, the target center (center of the tumor) is used as the starting point of the pathway. The neural network seeks the closest airways in the 3D model to the center of the target. Proximity of an airway is not the only factor for consideration. Among the factors the neural network considers when determining a pathway are an angle at which a tool (e.g., biopsy tool) exits the luminal network to reach the target. As will be appreciated, it is difficult if not impossible for most endoluminal tools to exit an airway at angles approaching a right angle to the longitudinal direction of the airway. Most endoluminal tools are straight and somewhat rigid on their distal end portions, thus incapable of being maneuvered to make such sharp turns to exit the narrow airways of the patient. Accordingly, the determination of the nearest airway must be made in conjunction with an angle of exit of the catheter 102 or tool to reach the center of the tumor. Another factor which is part of the pathway generation is the radii of the airways. For example, determining whether the airway closest to the target large enough to receive the catheter 102. If not an airway further away may be a better choice for navigation of the catheter, or another location at which the catheter will have to leave the airway to reach the target may be identified, one where the angle to the center of the target allows for substantially straight movement of the biopsy or therapy tool. In some instances, multiple pathways may be generated by the neural network algorithm and a suggested pathway identified for presentation to the user. Additionally, the neural network algorithm may analyze a potential pathway as it proceeds from the target back to the trachea to ensure that the airways being traversed have an increasing radius along the pathway. Other factors considered including tissue densities, mechanical properties of biopsy tools and therapeutic tools, and others may also be factors in the pathway generation.

[0057] In accordance with one aspect of the disclosure, an application on computing device 122 may present to a user, via a user interface, a number of image data sets associated with different patients, from which a selection can be made. The user selects an image data set for a patient requiring generation of a pathway plan. Once selected via the user interface, the application launches set of algorithms including neural network algorithms to perform the aspects described above including segmentation of the pleura boundaries, identification of tumors or lesions (targets) within the image data set, generation of 3D models of the airways and blood vessels, and generation of pathways from the trachea to the tumor or lesion. Following these steps, the 3D model 202 is displayed in the user interface 204 as shown in FIG. 2. The 3D model 202 includes the airway 206 tree, the blood vessels 208, the pleura boundary 210, and the tumors or lesions 212. Accordingly, the 3D model 202 provides a substantially more accurate model of the anatomy of the patient because the neural network algorithms are capable of generating 3D models of the airways much deeper than prior systems. Further the relative positions of the airways 206, blood vessels 208, pleura boundary 210 and the tumors or lesions 212 are much more accurate than prior 3D models employed in luminal navigation, and particularly lung navigation. The planning of a procedure may take place on any computing device capable of receiving and displaying the image data and running the planning application including the computing device 122 or a computing device entirely separate from the system 100.

[0058] FIG. 3 depicts a first planning screen 214 on the UI 204. The planning screen 214 identifies the individual tumors 212 and labels them. A table 216 is displayed including information regarding each identified tumor 212 such as its location (e.g., right-upper lobe, left-upper lobe, etc.), a thumbnail image 218 of the shape of the tumor, and a size 220 of the tumor (e.g., its longest dimension). Buttons 222 allow for manipulation of the 3D model 202 including, for example, centering a field of view of the 3D model, zooming in or out on the 3D model, measuring tools, increasing, or decreasing contrast of portions of the 3D model, and rotation of the 3D model. In addition, the image data from which the 3D model 202 was generated can be selected via tabs 224 (e.g., CT or PET).

[0059] FIG. 4 depicts the UI 204 following receipt of a selection of a tumor 212 (e.g., tumor 1). The selection may be made on the 3D model 202 or in the table 216. The information regarding this tumor 212 is highlighted in the table 216, and the image data from which the 3D model was generated is displayed in side panel 226. Side panel 226 includes images from three views, axial 228, coronal 230, and sagittal 232, each of which includes a scroll bar 234 allowing the images to be changed in accordance with the scroll (e.g., the next image along that axis). The three scroll bars 234 may be interconnected such that movement of one scrolls all three, or they may be selectively individually scrolled. The images depict the location of the tumor 212 identified by the neural network algorithms (described above) which are found in the 3D model 202. [0060] Receipt of a selection of a continue to plan button 236 advances the user interface 204 to FIG. 5, in which the 3D model depicts planned pathways 238 within the airways 206 to arrive at the tumor or lesion 212. As shown in FIG. 5, two pathways 238 are depicted to reach the tumor 212. A recommended pathway (e.g., one with the highest score in the table 216) is presented in a distinct fashion (e.g., highlighted) relative to other pathways. In the plan tab of the table 216, each pathway 238 is given a score by the neural network algorithm which takes into account factors such as exit angle of a tool (e.g., biopsy tool) from the catheter 102 to reach the biopsy location 240 within the tumor 212. As will be appreciated, the smaller the angle the more direct the path from an opening at the distal end of the catheter 102 to the biopsy location 240. As many biopsy and therapy tools have rigid end portions, a straighter path to the biopsy location 240 from the catheter 102 is easier to achieve. While an articulated catheter 102 can point its opening towards the target 212, the rigidity of the tool passing through the catheter 102 will generally cause the tool and the catheter 102 to straighten as the tool exits the catheter 102. Another factor is the distance that the tool must travel from the airway (e.g., through the airway wall and parenchyma) to reach the biopsy location 240. A shorter distance of travel from the airway to the tumor 212 results in less interaction with intervening tissue, which may include for example blood vessels. Accordingly a shorter distance of travel from the airway 206 is often, but not always, preferred.

[0061] In FIG. 5, the biopsy location 240 is the geometric center of the tumor 212. This may be a default position for the pathway plan proposed by the neural network algorithm. Using various pointing tools (e.g., mouse, touchscreen, etc.) additional biopsy locations 240 can be added to the tumor 212. By adding (or moving) a biopsy location 240 additional or altered pathways 238 may be generated by the neural network algorithms of the application. Each biopsy location 240 may receive an identity (e.g., number) and each biopsy location 240 is separately listed in the table 216 under the heading “Biopsy,” and may have its own pathway 238. Though not expressly shown in FIG. 5, the neural network algorithm may also determine multiple different biopsy locations within the tumor 212 and the pathway to each of the biopsy locations 240. The location of these biopsy locations 240 may be based on historical patterns for effective biopsy (or therapy) when considering the size, shape, location, and other information about the tumor 212. Alternatively, the dispersion of the biopsy locations 240 determined by the neural network may be based on empirical algorithms derived from data collected from multiple prior biopsies and therapies, the data from which is saved and analyzed (e.g., by a further neural network algorithm) to identify likely effective biopsy locations 240 within a tumor 212. Further the biopsy locations may be determined based on the position of the tumor within the patient, taking into account features such as the pleura, blood vessels, the shape and position of the exit cone, and other factors to minimize any challenges in collecting the biopsy and to optimize the opportunity to collect significant and useful samples from the tumor 212.

[0062] In order to accurately place each biopsy location 240, zooming in on the tumor 212 using the buttons 222 can result in display of a portion of the 3D model 202 near the tumor 212 as depicted in FIG. 6 A. As can be seen in FIG. 6 A, the initial biopsy location 240 is in the center of the tumor 212. An exit angle cone 242 is projected from the end of the pathway 238 and defines an expected potential trajectory of a tool passed through the catheter 102 navigated to that location within the 3D model 202 (and the patient). This exit angle cone 242 represents the potential volume of the tumor 212 from which a biopsy may be collected from this position of the catheter 102. While the exit angle cone 242 encompasses the biopsy location 240, based on the mechanical properties of the catheter 102 and the biopsy or therapy tools, the exact biopsy location 240 may not be achieved in the patient. Importantly in FIG. 6A (also shown in FIG. 5) shows a representation of a portion of the pleura 210. As noted above, the pleura 210 represents a boundary that is generally avoided, if possible, to prevent pneumothorax. As such, with the biopsy locations 240 identified, and the pathways 238 to those biopsy locations 240 identified, and the exit angle cone 242 displayed relative to the tumor 212 and the pleura 210, either the clinician manually or a neural network algorithm may make a determination whether there is any likelihood that the pleura 210 could be breached. When it is determined that such an occurrence might happen, an alternative pathway 238 may be selected or identified, or the biopsy location 240 can be altered to avoid the potential of piercing the pleura 210.

[0063] A tumor 212 may have multiple 2, 5, 10, etc. biopsy locations 240 to ensure that adequate sampling has been achieved. Each biopsy location 240 may have its own pathway 232, exit angle cone 242. Once satisfied with the plan for a tumor 212, the process may be repeated for all tumors 212 in the 3D model 202. Following identification of all biopsy locations 240 for all tumors 212 and the pathways 238 to all biopsy locations 240, a selection of the review plan button 244 allows for review of all pathways 238, leading to all tumors 212, and biopsy locations 240. If accepted, the plan is stored and can be utilized in a procedure, either immediately or at a later date. Those of skill in the art will recognize that the pathways 238 and the biopsy locations 240 may additionally or alternatively be employed in therapy planning and the biopsy locations 240 can be therapy application locations without departing from the scope of the disclosure.

[0064] In one aspect of the disclosure, the review of the pathway plan provides a virtual bronchoscopy view which simulates the navigation of a catheter 102 through the luminal network on the UI 204. The UI 204 may also depict the 3D model 202, and as the simulated navigation proceeds the level of zoom of the 3D model can be altered (e.g., Figs. 4-6) to provide the clinician with information necessary for review at that point of the simulation navigation.

[0065] As will be appreciated, the features of the 3D model 202 depicted in FIG. 2 are not all depicted in the subsequent FIG. 4-6. The reduction of the features of the 3D model reduces the cognitive load on the clinician and allows the clinician to focus on the relevant aspects of the 3D model and the patient’s anatomy. Using the buttons 222 various of these features may be selectively toggled on or off by the clinician. For example, on arriving at the UI 204 as shown in FIG. 6A, a clinician may be interested in blood vessels 208 around the tumor 212 so that the pathway 238 can be adjusted to avoid the intersection of the blood vessels 208. Accordingly, the relevant blood vessels can be manually or automatically toggled on such that they appear in the UI 204. Additionally, some of these features may be automatically removed in the planning software or ghosted (e.g., made translucent or transparent) to reduce the appearance of intervening portions of the anatomy. By reducing the displayed portions of the 3D model to only those with clinical relevance to the cognitive load is reduced, and a clearer understanding of the planned procedure is generated.

[0066] FIG. 6B depicts a further aspect of the disclosure in which a pop-up 246 is presented in the UI 204. In the pop-up a variety of selections are presented to the clinician allowing for biopsy locations 240 to be automatically generated in the tumor 212. Buttons 248 in the pop-up allow for selection center biopsy locations, PET biopsy locations, a geometric biopsy location, or custom biopsy location to be added to a tumor 212.

[0067] While the preceding aspects of the disclosure focus on planning of pathways and collection of biopsies (or application of therapy), the disclosure is not so limited. The following features are primarily directed to aspects of the UI 204 during the intraluminal navigation procedure. FIG. 7 depicts a side-by-side view of two visualization features of the disclosure. On one side of the US 204 is displayed a live bronchoscopic image 302. The live bronchoscopic image is acquired by a bronchoscope associated with (either inserted endoluminally within or a component of) catheter 102. The pathway 238 is overlaid on the live bronchoscopic image 302. This overlay of the pathway 238 provides guidance to a clinician when navigating the airways of the patient and provides directional assistance on which airway to follow to arrive at the tumor 212 at the end of the pathway 238, as determined during the planning (described above). Simultaneously with bronchoscopic image 302, the 3D model 202 is also displayed on the other side. The display of the 3D model is connected to the view in the bronchoscopic image 302. Thus as navigation of the catheter 102 is undertaken a location along the pathway 238 of the catheter 102 may be depicted in the 3D model 202. Alternatively, or additionally, the level of zoom of the 3D model 202 may be adjusted (similar to that depicted in Figs 4-6) as the catheter 102 is navigated to different portions of the luminal network. The 3D model 212 includes at least the airways 206, the tumor 212 and a portion of the pleura 210 relevant for the navigation.

[0068] Though described as wholly separate portions of the application, where necessary or appropriate the planning can be reset or modified either prior to or as part of the procedure. In addition, the plans may be modified and targets added or deleted from the pathway plan or a new target selected during the navigation utilizing methods substantially similar to those described herein above.

[0069] FIG. 8 depicts a pop-up 303 which may be optionally presented on the UI 204 alerting the clinician that a local registration process could be undertaken to reduce or eliminate CT-body-divergence. The local registration process may acquire fluoroscopic or cone-beam CT (CBCT) images. In one local registration process fluoroscopic images are acquired and a relative position of the catheter 102 and the tumor is determined. The determined relative positions are then used to update the displayed relative positions of the catheter 102 and the tumor 212 in the 3D model. Alternatively, where CBCT images are acquired the CBCT images are processed (e.g., using the neural network algorithms) to identify the relative positions of the catheter 102 and the tumor, and the image data from the CBCT is used to update the 3D model. The CBCT image data may not be images of the entirety of the lungs (or another luminal network) but may rather be a small portion near the catheter 102 and the tumor 212. A 3D model may be generated from the CBCT image data and replace portions of the original 3D model. Alternatively, the biopsy locations 240 and pathways 238 from the original 3D model can be applied to the 3D model from the CBCT data to allow for accurate “last mile” navigation to the biopsy locations 240. [0070] Another feature of the UI 204 is the timeline feature 304 which appears along the bottom of the UI 204. A more detailed view of the timeline 304 can be found in FIG. 9. The timeline is a linear representation of the pathway plan 238 (e.g., from trachea to tumor) and provides a type of turn-by-tum navigation instructions. The timeline 304 provides an abstract representation of the pathway 238. The timeline 304 highlights information needed for navigation such as the bifurcations along the way and shows them visually with the relevant lumen clearly marked. In addition, the view’s timeline represents the distance until reaching target, and thus can be used as a simple way to evaluate progress to the target. As the catheter 102 is navigated within the luminal network (e.g., the lungs) at each bifurcation where a decision must be made as to which airway to follow and indicator 306 is depicted. Each indicator includes the number of airways at the bifurcation and one of the airways or a side of the bifurcation is highlighted indicating the direction navigation should continue. The timeline changes color (e.g., from dark blue to light blue) as portions of the pathway 238 are navigated along and the progress of the catheter 102 is detected. A gap 308 between the dark blue of the as yet traversed pathway 238 and the tumor 212 represents the distance outside of the airway the biopsy or therapy tool must travel to reach the tumor 212. The tumor 212 is represented as another color along the timeline 304. The biopsy location 240 may be depicted within the tumor 212 portion of the timeline 304. Though not shown in FIG. 8, a representation of the pleura 210 may be shown on the timeline 304 to the right of the tumor 212 portion, if relevant to the specific pathway 238 being navigated. A timer 310 displays the time required to traverse the pathway 238 as part of the procedure. The procedure may be recorded and played back using buttons 312 on the timeline 304.

[0071] Yet a further feature of the disclosure is directed to an alignment view 400 displayed on UI 204 as shown in FIG. 10. In FIG. 10 live fluoroscopic images are depicted on the UI 204 alongside portions of the 3D model 202. By adjusting the position of the catheter 102 and observing the differences in location in the fluoroscopic images as well as in the 3D model 202 the clinician is able to observe the interaction of the biopsy tool with the tumor 212, even if the tumor is not entirely visible in the fluoroscopic images. The movements shown in the 3D model help the clinician confirm that the biopsy tool or therapy tool is aligned with and interacting with the tumor 212 as planned.

[0072] Reference is now made to FIG. 11, which is a schematic diagram of a system 400 configured for use with the features described hereinabove. System 400 may include a workstation 401, and optionally an imaging device 415 (e.g., a fluoroscope, CBCT scanner, or an ultrasound device, etc.). In some embodiments, workstation 401 may be coupled with imaging device 415, directly or indirectly, e.g., by wireless communication. Workstation 401 may include a memory 402, a processor 404, a display 406 and an input device 410. Processor 404 may include one or more hardware processors. Workstation 401 may optionally include an output module 412 and a network interface 408. Memory 402 may store an application 418 and image data 414. Application 418 may include instructions executable by processor 404 for executing the methods of the disclosure (e.g., the image processing neural network algorithms described herein), and presentation on one or more displays (e.g., screens 23, 32, or 34).

[0073] Application 418 may further include a user interface 416. Image data 414 may include the CT or CBCT scans, 3D models, fluoroscopic 3D reconstructions of the target area and/or any other fluoroscopic image data, ultrasound image data, magnetic resonance imaging (MRI) data, positron emissions tomography (PET) image data and the like. Processor 404 may be coupled with memory 402, display 406, input device 410, output module 412, network interface 408 and imaging device 415. Workstation 401 may be a stationary computing device, such as a personal computer, or a portable computing device such as a tablet computer. Workstation 401 may include a plurality of computer devices. Those of skill in the art will recognize that the methods and systems described herein may be incorporated and executed on any computing device capable of receiving the image data of the luminal network including computers 21, 31, 41 of the robotic surgical system 10.

[0074] Memory 402 may include any non-transitory computer-readable storage media for storing data and/or software including instructions that are executable by processor 404 and which control the operation of workstation 401 and, in some embodiments, may also control the operation of imaging device 415. Memory 402 may include one or more storage devices such as solid-state storage devices, e.g., flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, memory 402 may include one or more mass storage devices connected to the processor 404 through a mass storage controller (not shown) and a communications bus (not shown).

[0075] Although the description of computer-readable media contained herein refers to solid-state storage, it should be appreciated by those skilled in the art that computer-readable storage media can be any available media that can be accessed by the processor 404. That is, computer readable storage media may include non-transitory, volatile, and non-volatile, removable, and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media may include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium (e.g., cloud based storage) which may be used to store the desired information, and which may be accessed by workstation 401.

[0076] Application 418 may, when executed by processor 404, cause display 406 (e.g., to present user interface 416. User interface 416 may be configured to present to the user with the various UI 204 (e.g., on screens 23, 32, or 34) described herein and others without departing from the scope of the disclosure.

[0077] Network interface 408 may be configured to connect to a network such as a local area network (LAN) consisting of a wired network and/or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and/or the Internet. Network interface 408 may be used to connect between workstation 401 and imaging device 415. Network interface 408 may also be used to receive image data 414. Input device 410 may be any device by which a user may interact with workstation 401, such as, for example, a mouse, keyboard, foot pedal, touch screen, and/or voice interface. Output module 412 may include any connectivity port or bus, such as, for example, parallel ports, serial ports, universal serial busses (USB), or any other similar connectivity port known to those skilled in the art. From the foregoing and with reference to the various figures, those skilled in the art will appreciate that certain modifications can be made to the disclosure without departing from the scope of the disclosure.

EXAMPLES

Example 1 - includes a system for planning a lung navigation pathway comprising, a computing device including a processor and a memory, the memory storing therein an application that when executed by the processor, accesses image data stored in the memory, analyzes the image data to identify a pleura, a tumor, and airways, and determine a pathway through the airways through the airways to the tumor, generates a 3D model of the lungs, the 3D model including a representation of the pleura, the airways, the tumor, and the pathway through the airways to the tumor, wherein the identification of the pleura, the tumor, the airways, and the determination of the pathway is executed by one or more neural network algorithms associated with the application, and presents the 3D model and pathway in a user interface on a display in communication with the computing device. Example 2 - The system of example 1, wherein the application when executed by the processor identifies a geometric center of the tumor as a first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface, wherein the pathway .

Example 3 - The system of example 2, wherein the application when executed by the processor receives a second biopsy location, determines a pathway through the airways to the tumor to access the second biopsy location, displays an indicator of the second biopsy location and the pathway in the user interface.

Example 4 - The system of any of examples 2 or 3, wherein the application determines multiple pathways to each biopsy location and presents a score for each pathway in the user interface.

Example 5 - The system of any of the preceding examples, wherein a level of detail of the 3D model is adjusted based on a level of zoom.

Example 6 - The system of any of the preceding examples wherein the application when executed by the processor displays the determined pathway on live endoluminal images.

Example 7 - The system of any of the preceding examples, wherein the application when executed by the processor generates a timeline view of the lung navigation pathway.

Example 8 - The system of example 7, wherein the timeline view includes a representation of each bifurcation of the airways along the pathway.

Example 9 - The system of any of examples 7 or 8, wherein the timeline view depicts an indicator of a location of a catheter being navigated along the pathway.

Example 10 - The system of any of examples 7-9, wherein the timeline view depicts a representation of the tumor and a biopsy location within the tumor.

Example 11 -The system of any of examples 7-10, wherein a portion of the timeline changes color as navigation proceeds.

Example 12 - The system of any of examples 7-11, further comprising a timer indicating a duration of a procedure.

Example 13 - A system including a catheter configured for navigation within lungs of a patient, a tracking system configured for detecting a location of the catheter; and a computing device including a processor and a memory, the memory storing therein an application that when executed by the processor: accesses image set stored in the memory, analyzes the image data to identify a pleura, a tumor, and airways, and determine a pathway through the airways through the airways to the tumor, generates a 3D model of the lungs, the 3D model including a representation of the pleura, a representation of the airways, a representation of the tumor, and a representation of the pathway through the airways to the tumor, and presents the 3D model and pathway in a user interface on a display in communication with the computing device, wherein the identification of the pleura, identification of the tumor, identification of the airways, and the determination of the pathway is executed by one or more neural network algorithms associated with the application.

Example 14 - The system of example 13, wherein the application when executed by the processor identifies a geometric center of the tumor as a first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface.

Example 15 - The system of any of examples 13-14, wherein the application when executed by the processor depicts an exit angle cone on the tumor.

Example 16 - The system of any of examples 13-15, wherein the application when executed by the processor displays or removes elements of the 3D model based on a level of zoom.

Example 17 - The system of any of examples 13-16, wherein the application when executed by the processor displays the pathway on live bronchoscopic images in the user interface.

Example 18 - The system of any of examples 13-17, wherein the application when executed by the processor generates and presents a target alignment view of the 3D model and a live fluoroscopic image in the user interface.

Example 19 - The system of any of examples 13-18, wherein the application when executed by the processor generates a timeline view of the pathway, wherein the timeline view includes a representation of each bifurcation of the airways along the pathway.

Example 20 - The system of any of examples 13-19, wherein the application when executed by the processor displays tumor dimensional data on the user interface.

[0078] While detailed embodiments are disclosed herein, the disclosed embodiments are merely examples of the disclosure, which may be embodied in various forms and aspects. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the disclosure in virtually any appropriately detailed structure.

Claims

We claim:

1. A system for planning a lung navigation pathway comprising: a computing device including a processor and a memory, the memory storing therein an application that when executed by the processor: accesses image data stored in the memory; analyzes the image data to identify a pleura, a tumor, and airways, and determine a pathway through the airways through the airways to the tumor; generates a 3D model of the lungs, the 3D model including a representation of the pleura, the airways, the tumor, and the pathway through the airways to the tumor; and presents the 3D model and pathway in a user interface on a display in communication with the computing device.

2. The system of claim 1, wherein the application when executed by the processor identifies a geometric center of the tumor as a first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface, wherein the pathway.

3. The system of claim 2, wherein the application when executed by the processor receives a second biopsy location, determines a pathway through the airways to the tumor to access the second biopsy location, displays an indicator of the second biopsy location and the pathway in the user interface.

4. The system of any of claim 2, wherein the application determines multiple pathways to each biopsy location and presents a score for each pathway in the user interface.

5. The system of claim 2, wherein a level of detail of the 3D model is adjusted based on a level of zoom.

6. The system of claim 2, wherein the identification of the pleura, the tumor, the airways, and the determination of the pathway is executed by one or more neural network algorithms associated with the application.

7. The system of claim 2, wherein the application when executed by the processor generates a timeline view of the lung navigation pathway.

8. The system of claim 7, wherein the timeline view includes a representation of each bifurcation of the airways along the pathway.

9. The system of claim 7, wherein the timeline view depicts an indicator of a location of a catheter being navigated along the pathway.

10. The system of claim 7, wherein the timeline view depicts a representation of the tumor and a biopsy location within the tumor.

11. The system of claim 7, wherein a portion of the timeline changes color as navigation proceeds.

12. The system of claim 7, further comprising a timer indicating a duration of a procedure.

13. A system comprising: a catheter configured for navigation within lungs of a patient; a tracking system configured for detecting a location of the catheter; and a computing device including a processor and a memory, the memory storing therein an application that when executed by the processor: accesses image set stored in the memory; analyzes the image data to identify a pleura, a tumor, and airways, and determine a pathway through the airways through the airways to the tumor; generates a 3D model of the lungs, the 3D model including a representation of the pleura, a representation of the airways, a representation of the tumor, and a representation of the pathway through the airways to the tumor; and presents the 3D model and pathway in a user interface on a display in communication with the computing device.

14. The system of 13, wherein the application when executed by the processor identifies a geometric center of the tumor as a first biopsy location and displays an indicator of the first biopsy location in the tumor in the user interface.

15. The system of claim 13, wherein the application when executed by the processor depicts an exit angle cone on the tumor.

16. The system of claim 13, wherein the application when executed by the processor displays or removes elements of the 3D model based on a level of zoom.

17. The system of claim 13, wherein the application when executed by the processor displays the pathway on live bronchoscopic images in the user interface.

18. The system of claim 13, wherein the application when executed by the processor generates and presents a target alignment view of the 3D model and a live fluoroscopic image in the user interface.

19. The system of claim 13, wherein the identification of the pleura, identification of the tumor, identification of the airways, and the determination of the pathway is executed by one or more neural network algorithms associated with the application.

20. The system of claim 13, wherein the application when executed by the processor displays tumor dimensional data on the user interface.