Classifications

• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
  • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
  • G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
  • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/044—Recurrent networks, e.g. Hopfield networks
  • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/045—Combinations of networks
  • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/0464—Convolutional networks [CNN, ConvNet]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/09—Supervised learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H30/00—ICT specially adapted for the handling or processing of medical images
  • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
  • G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
  • G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/50—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for simulation or modelling of medical disorders
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
  • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
  • A61B2034/2046—Tracking techniques
  • A61B2034/2051—Electromagnetic tracking systems
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
  • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
  • A61B2034/2046—Tracking techniques
  • A61B2034/2061—Tracking techniques using shape-sensors, e.g. fiber shape sensors with Bragg gratings
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
  • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
  • A61B2034/2046—Tracking techniques
  • A61B2034/2063—Acoustic tracking systems, e.g. using ultrasound
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
  • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
  • A61B2034/2046—Tracking techniques
  • A61B2034/2065—Tracking using image or pattern recognition
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
  • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
  • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
  • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
  • A61B90/37—Surgical systems with images on a monitor during operation
  • A61B2090/378—Surgical systems with images on a monitor during operation using ultrasound
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/044—Recurrent networks, e.g. Hopfield networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/045—Combinations of networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/084—Backpropagation, e.g. using gradient descent
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10132—Ultrasound image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20081—Training; Learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30004—Biomedical image processing
  • G06T2207/30061—Lung

Definitions

• Various embodiments of this disclosure relate generally to machine- learning-based techniques for in vivo navigation, and, more particularly, to systems and methods for determining registration between non-optical image data, e.g., ultrasound imaging data, and medical imaging data.
• non-optical image data e.g., ultrasound imaging data, and medical imaging data.
• a medical device is advanced, at least in part, into the body of a patient.
• an ablation device is advanced into the peripheral portion of the lung having the undesirable tissue.
• techniques using direct insertion e.g., via a needle, have been used, such techniques generally have a high risk of complications.
• Less invasive techniques have been developed, such as techniques utilizing a bronchoscope. However, such techniques may also have disadvantages.
• an exemplary embodiment of a system for providing in vivo navigation of a medical device may include a memory, a display, and a processor operatively connected to the display and the memory.
• the memory may store instructions and a trained machine-learning model.
• the machine-learning model may have been trained, based on (i) training medical imaging data and training non-optical in vivo image data of at least a portion of an anatomy of one or more individuals and (ii) registration data associating the training non-optical in vivo image data with locations in the training medical imaging data as ground truth.
• the training may have been configured to cause the trained machine-learning model to learn associations between the training non-optical in vivo image data and the training medical imaging data.
• the processor may be configured to execute the instructions in the memory to perform operations.
• the operations may include: receiving input medical imaging data associated with at least a portion of an anatomy of a patient; receiving input non-optical in vivo image data from a sensor positioned on a distal end of a medical device that is advanced into the portion of the anatomy of the patient; using the learned associations to determine a location of the distal end of the medical device in the input medical imaging data; modifying the input medical imaging data to include a location indicator indicative of the determined location of the distal end of the medical device; and causing the display to output the modified input medical imaging data including the location indicator.
• the operations may further include: upon the medical device moving within the portion of the anatomy of the patient, receiving further non-optical in vivo image data from the sensor; using the learned associations to determine an updated location of the distal end of the medical device based on the further non-optical in vivo image data; updating the input medical imaging data to adjust the location indicator based on the updated location of the distal end of the medical device; and causing the display to output the updated input medical imaging data.
• determining of the updated location, the updating of the input medical imaging data, and the output via the display of the updated input medical imaging data may occur in real-time or near real-time such that the display is configured to output a live location of the distal end of the medical device.
• the trained machine-learning model may be configured to learn associations between a sequence of non-optical in vivo images of the training non-optical in vivo image data and a path of travel within the training medical imaging data.
• the trained machine learning-model may be configured to determine the location of the distal end of the medical device in the input medical imaging data by predicting a path of travel of the distal end of the medical device from a previous location in the input medical imaging data using the input non-optical in vivo image data.
• the operations may further include: extracting at least one three-dimensional structure from the input non-optical in vivo image data; and registering the at least one three-dimensional structure with geometry of the at least portion of the anatomy from the input medical imaging data.
• the determination of the location of the distal end of the medical device may be further based on the registration of the at least one three-dimensional structure with the geometry.
• the trained machine-learning model may include one or more of a long short term memory network or a sequence-to- sequence model.
• the operations may further include receiving a position signal from a position sensor positioned proximate to the distal end of the medical device. In some embodiments, the determination of the location of the distal end of the medical device may be further based on the position signal.
• the operations may further include using the position signal to localize a position of the distal end of the medical device to a region within the portion of the anatomy of the patient.
• using the learned associations to determine the location of the distal end of the medical device in the input medical imaging data may include using the learned associations to identify the location of the distal end within the localized region.
• the input non-optical in vivo image data may include 360 degree image data from a phased transducer array.
• the training may be configured to associate the training non-optical in vivo image data with diameters of interior portions of anatomy.
• determining the location of the distal end of the medical device using the learned associations may include: using the learned associations to determine a diameter of an interior portion of the anatomy of the patient at a current location of the distal end of the medical device; and comparing the current diameter with geometry of the input medical imaging data to identify a location in the input medical imaging data matching the determined diameter.
• the trained machine-learning model may be configured to learn associations between a sequence of diameters determined based on the training non-optical in vivo image data and a path of travel within the training medical imaging data.
• the trained machine learning- model may be configured to determine the location of the distal end of the medical device in the input medical imaging data by predicting a path of travel of the distal end of the medical device from a previous location in the input medical imaging data using the input non-optical in vivo image data.
• the portion of the anatomy of the patient may include a peripheral portion of a lung of the patient.
• the input non-optical in vivo image data may include ultrasound data.
• the trained machine learning-model may be configured to determine the location of the distal end of the medical device in the input medical imaging data based on shape information associated with the medical device received from a further sensor of the medical device.
• an exemplary embodiment of a method for providing in vivo navigation of a medical device may include: receiving input medical imaging data associated with at least a portion of an anatomy of a patient; receiving input non-optical in vivo image data from a sensor positioned on a distal end of a medical device that is advanced into the portion of the anatomy of the patient; using a trained machine-learning model to determine a location of the distal end of the medical device in the input medical imaging data, wherein: the trained machine learning model is trained, based on (i) training medical imaging data and training non-optical in vivo image data of at least a portion of an anatomy of one or more individuals and (ii) registration data associating the training non-optical in vivo image data with locations in the training medical imaging data as ground truth; the training is configured to cause the trained machine-learning model to learn associations between the training non-optical in vivo image data and the training medical imaging data, and the trained machine-learning model is configured to use the learned associations to determine the
• the input non-optical in vivo image data may include ultrasound data.
• the portion of the anatomy of the patient may include a peripheral portion of a lung of the patient.
• an exemplary embodiment of a method of training a machine-learning model to determine an output location of a distal end of a medical device in an anatomy of a patient within input medical imaging data in response to receiving the input medical imaging data and receiving, from a sensor positioned on the distal end of a medical device, input non-optical in vivo image data, may include: inputting training data into the machine-learning model, the training data including training medical imaging data and training non-optical in vivo image data of at least a portion of an anatomy of one or more individuals; inputting ground truth into the machine-learning model that includes registration data associating the training non-optical in vivo image data with locations in the training medical imaging data; and using the training data and the ground truth with the machine-learning model to learn associations between the training non-optical in vivo image data and the training medical imaging data that are usable by the machine-learning model to determine the output location of the distal end of the medical device.
• the method may further include using the training data and the ground truth with the machine-learning model to learn associations between a sequence of training non-optical in vivo images and a path of travel within the training medical imaging data, such that the machine learning-model is configured to determine the location of the distal end of the medical device in the input medical imaging data by predicting a path of travel of the distal end of the medical device from a previous location in the input medical imaging data using the input non-optical in vivo image data.
• the training non-optical in vivo image data is ultrasound data.
• FIG. 1 depicts an exemplary environment for training and/or using a machine-learning model to provide in vivo navigation of a medical device, according to one or more embodiments.
• FIG. 2A depicts an exemplary embodiment of a medical device usable with the environment of FIG. 1 , according to one or more embodiments.
• FIG. 2B depicts an exemplary ultrasound image produced by a transducer operating in air.
• FIG. 2C depicts another exemplary ultrasound image produced by a transducer operating in vivo with a gap between the transducer and surrounding tissue.
• FIG. 3 depicts a flowchart of an exemplary method of training a machine-learning model to provide in vivo navigation of a medical device, according to one or more embodiments.
• FIG. 4A depicts a flowchart of an exemplary method of using a trained machine-learning model to provide in vivo navigation of a medical device, according to one or more embodiments.
• FIGs. 4B and 4C depict exemplary embodiments of a navigational output generated by a navigation system, according to one or more embodiments.
• FIG. 5 depicts an example of a computing device, according to one or more embodiments.
• a medical device e.g., an ablation device to be navigated within the periphery of a lung of a patient.
• conventional navigation techniques may not be suitable.
• conventional techniques may not be sufficiently accurate for navigating to a target site and/or confirming that a target site has been reached.
• Conventional navigation techniques may also rely on inclusion of a light source, camera, and/or lens, which may result in a medical device of a size that may be too large to be navigated to some target sites.
• systems and methods are described for using machine learning to locate a distal end of a medical device, with reference to medical imaging data, e.g., pre-operative CT scan data, based on non-optical in vivo image data received from a sensor disposed on the distal end of the medical device.
• medical imaging data e.g., pre-operative CT scan data
• non-optical in vivo image data received from a sensor disposed on the distal end of the medical device.
• the trained machine-learning model may be usable to provide navigation information for the medical device, e.g., a position indicator in the medical imaging data indicating a live location of the distal end of the medical device.
• proximal and distal portions portions of the device and/or its components are referred to as proximal and distal portions.
• proximal is intended to refer to portions closer to a user of the device, and the term “distal” is used herein to refer to portions further away from the user.
• extends “distally” indicates that a component extends in a distal direction, and extends “proximally” indicates that a component extends in a proximal direction.
• the term “based on” means “based at least in part on.”
• the singular forms “a,” “an,” and “the” include plural referents unless the context dictates otherwise.
• the term “exemplary” is used in the sense of “example” rather than “ideal.”
• the terms “comprises,” “comprising,” “includes,” “including,” or other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, or product that comprises a list of elements does not necessarily include only those elements, but may include other elements not expressly listed or inherent to such a process, method, article, or apparatus.
• medical imaging data generally encompass data associated with and/or indicative of a geometry and/or physiology of a patient, e.g., that may be generating via medical imaging and/or that may be represented as an image of the anatomy of the patient, e.g., a two- dimensional image, a three-dimensional image or model, a video, a time-varying image, etc.
• Medical imaging generally encompasses techniques whereby a signal (light, electromagnetic energy, radiation, etc.) is generated, and measurements are taken that are indicative of how that signal interacts with and/or is affected by, transmitted through, or the like, the patient.
• Medical imaging technologies include CT scans, MRI scans, X-ray scans, or any other suitable modality, e.g., that may be used to visualize an interior of at least a portion of the patient’s anatomy.
• Medical imaging data may include, for example, two-dimensional data and/or images, three-dimensional data and/or images, voxel data, a geometric model of at least a portion of patient anatomy, a solid model of the portion of patient anatomy, a mesh of nodes or points representative of the portion of the anatomy and/or characteristics of the portion of the anatomy, and/or any other suitable data associated with the patient and/or medical imaging.
• non-optical image data generally encompasses data that is indicative of, associated with, and/or usable to generate an image, and that was generated using a non-optical signal, e.g., via a signal generated by an ultrasound transducer.
• a “machine-learning model” generally encompasses instructions, data, and/or a model configured to receive input, and apply one or more of a weight, bias, classification, or analysis on the input to generate an output.
• the output may include, for example, a classification of the input, an analysis based on the input, a design, process, prediction, or recommendation associated with the input, or any other suitable type of output.
• a machine-learning model is generally trained using training data, e.g., experiential data and/or samples of input data, which are fed into the model in order to establish, tune, or modify one or more aspects of the model, e.g., the weights, biases, criteria for forming classifications or clusters, or the like.
• Aspects of a machine-learning model may operate on an input linearly, in parallel, via a network (e.g., a neural network), or via any suitable configuration.
• the execution of the machine-learning model may include deployment of one or more machine learning techniques, such as linear regression, logistical regression, random forest, gradient boosted machine (GBM), deep learning, and/or a deep neural network.
• Supervised and/or unsupervised training may be employed.
• supervised learning may include providing training data and labels corresponding to the training data.
• Unsupervised approaches may include clustering, classification or the like.
• K-means clustering or «-Nearest Neighbors may also be used, which may be supervised or unsupervised.
• Combinations of «-Nearest Neighbors and an unsupervised cluster technique may also be used. Any suitable type of training may be used, e.g., stochastic, gradient boosted, random seeded, recursive, epoch or batch-based, etc.
• a medical device In certain medical procedures, it may be desirable to navigate a medical device to a target site within the body of a patient, e.g., into the periphery of a lung of the patient for a procedure such as ablation of undesirable tissue.
• conventional in vivo navigation techniques e.g., conventional bronchoscope techniques
• a concern for bronchoscopic techniques is the ability to not only accurately navigate to a target for ablation, but also to confirm that a target has been reached.
• medical imaging such as a CT image of the patient taken prior or during a procedure, may be used as a passive map in conjunction with active navigation via a bronchoscope.
• Bronchoscopic techniques for such purpose include navigational (electromagnetic and/or video) bronchoscopy, radial-probe endobronchial ultrasound, and robotic bronchoscopy.
• Electromagnetic navigation alone is generally insufficient to provide fine grained detail and/or confirming that a target site is reached.
• Video navigation may be used to confirm arrival at a target site, but provides limited information, and moreover the requirement for a light source, camera, and lens with the bronchoscope may increase device diameter to a size that is too big to reach desired portions of the periphery of the lung.
• Radial probes require rotation of the device for imaging, which may present risks to the patient and/or may not be possible depending on the physiology of the patient and/or the position of the device.
• robotic bronchoscopy generally involves large and/or complex machines external to the patient operating in concert with the procedure, such as a CT imaging machine or the like, which can be cumbersome and/or costly.
• Conventional techniques may result in a medical device that is too large to navigate to the target site, that may present risk to the patient, and/or that may require large, complex, and/or expensive external machinery to operate in conjunction with the procedure. Accordingly, improvements in technology relating to in vivo navigation for a medical device are needed.
• a medical procedure includes introducing and/or advancing a medical device to a target site within a patient’s body.
• Medical imaging such as a CT scan or the like, may be taken of at least a portion of the patient’s body that includes the target site, e.g., prior to and/or during the procedure.
• a location of the target site in the patient’s body may be identified in the medical imaging.
• the medical device may be introduced into the patient’s body so as to be advanced toward the target site.
• the medical device may be introduced via the patient’s airway so as to be advanced toward a target site in the patient’s lung.
• the medical device may include an end effector, such as an ablation device, positioned on a distal end of the medical device, e.g., for performing a therapeutic procedure.
• the medical device may further include a sensor positioned on the distal end of the medical device, such as a transducer configured to generate a signal indicative of an ultrasound medical image.
• the medical device may, in some instances, not include one or more of a camera, a light source, or a lens.
• a navigation system may be configured to receive the signal generated by the sensor, e.g., receive non-optical in vivo image data from the sensor.
• the navigation system may include a trained machine-learning model that is configured to determine a location of the distal end of the medical device within the patient’s body in the medical imaging data based on the received non-optical in vivo image data from the sensor.
• the navigation system may modify the medical imaging data so as to include a location indicator indicative of a location of the distal end of the medical device within the anatomy of the patient depicted by the medical imaging data, and may cause a display to output the modified medical imaging data.
• the display may depict a live location of the distal end of the medical device within the medical imaging data as it travels within the anatomy of the patient.
• a machine-learning model may be trained to determine a location, within medical imaging data, of a distal end of a medical device advanced into anatomy of a patient.
• Training data that includes medical imaging data and non-optical in vivo image data of at least a portion of an anatomy of one or more individuals may be input into the machine-learning model.
• Ground truth that includes registration data associating the non-optical in vivo image data with locations in the medical imaging data may also be input into the machine learning model.
• the training data and the ground truth may be used with the machine-learning model to develop associations between non-optical in vivo image data and medical imaging data usable by the machine-learning model to determine the output location of the distal end of the medical device.
• the ground truth may be at least partially developed with an additional navigation technique.
• the training data and ground truth may be obtained using a medical device that includes a sensor, e.g., such as an ultrasound transducer, and that also includes an optical sensor such as a camera.
• Video bronchoscopy may be used to determine and/or verify a location of the medical device in order to generate a ground truth association between the signal from the sensor and a location of the medical device in the medical imaging data.
• the training of the machine-learning model may be validated by comparing a location determined via the trained model with a location determined via the video bronchoscopy.
• a pressure sensor, medical imaging technique, or the like may be used to determine pressures, temperatures, or other biological or physiological characteristics within the body of a patient.
• a medical device may include, instead of or in addition to the sensor described above, a further sensor configured to sense one or more of such biological or physiological characteristics. The one or more determined characteristics within the body and the one or more determined characteristics sensed by the further sensor may be used as inputs for the machine-learning model.
• bronchoscopy and/or navigation in and/or ablation of tissue within the periphery of the lungs it should be understood that techniques according to this disclosure may be adapted to any suitable procedure involving in vivo navigation of a medical device including, for example, a cardiac or heart valve procedure, any procedure in the pulmonary, gastrointestinal, urinary, or other body tract, any procedure using an endoscope, bronchoscope, colonoscope, ureteroscope, or other like devices, and/or any therapeutic or diagnostic procedure including, for example, biopsy, ablation, resection, dissection, injection, application of a drug or therapeutic, etc., or combinations thereof. It should also be understood that the examples above are illustrative only. The techniques and technologies of this disclosure may be adapted to any suitable activity.
• machine learning techniques that may be adapted to in vivo navigation of a medical device.
• machine learning techniques adapted to determining a location and/or path of travel of a medical device within anatomy of a patient, with reference to medical imaging data may include one or more aspects according to this disclosure, e.g., a particular selection of training data, a particular training process for the machine-learning model, operation of a particular device suitable for use with the trained machine-learning model, operation of the machine-learning model in conjunction with particular data such as medical imaging data, modification of such particular data by the machine-learning model, etc., and/or other aspects that may be apparent to one of ordinary skill in the art based on this disclosure.
• FIG. 1 depicts an exemplary environment 100 that may be utilized with techniques presented herein.
• One or more user device(s) 105, one or more medical device(s) 110, one or more display(s) 115, one or more medical provider(s) 120, and one or more data storage system(s) 125 may communicate across an electronic network 130.
• one or more navigation system(s) 135 may communicate with one or more of the other components of the environment 100 across electronic network 130.
• the one or more user device(s) 105 may be associated with a user 140, e.g., a user associated with one or more of generating, training, or tuning a machine-learning model for providing in vivo navigation of a medical device, generating, obtaining, or analyzing medical imaging data, and/or performing a medical procedure.
• a user 140 e.g., a user associated with one or more of generating, training, or tuning a machine-learning model for providing in vivo navigation of a medical device, generating, obtaining, or analyzing medical imaging data, and/or performing a medical procedure.
• the components of the environment 100 are associated with a common entity, e.g., a hospital, facility, or the like. In some embodiments, one or more of the components of the environment is associated with a different entity than another.
• the systems and devices of the environment 100 may communicate in any arrangement. As will be discussed herein, systems and/or devices of the environment 100 may communicate in order to one or more of generate, train, or use a machine-learning model to provide in vivo navigation for the medical device 110, among other activities.
• the user device 105 may be configured to enable the user 140 to access and/or interact with other systems in the environment 100.
• the user device 105 may be a computer system such as, for example, a desktop computer, a mobile device, a tablet, etc.
• the user device 105 may include one or more electronic application(s), e.g., a program, plugin, browser extension, etc., installed on a memory of the user device 105.
• the electronic application(s) may be associated with one or more of the other components in the environment 100.
• the electronic application(s) may include one or more of system control software, system monitoring software, software development tools, etc.
• FIG. 2A depicts an exemplary embodiment of the medical device 110.
• the medical device 110 may include a distal end 205 connected to a proximal end 210 via a tube 215.
• the distal end 205 may include one or more portions 220 configured to one or more of receive a component or communicate with a lumen disposed in the tube 215.
• at least one sensor 225 may be disposed in one of the portions 220.
• a tool having an end effector 230 may be disposed in another of the portions 220, e.g., an ablation device, forceps, a net, an orifice for taking in or outputting fluid and/or material, etc.
• the sensor 225 may include, for example, a transducer, an electromagnetic position sensor, a fiber-optic position sensor, or the like. In the embodiment depicted in FIG. 2, the sensor 225 includes a transducer array, but it should be understood that any suitable type of non-optical sensor may be used.
• the tube 215 may, in some embodiments, be formed from a flexible material.
• the tube 215 may include one or more lumens (not shown) that communicate between the distal end 205 and the proximal end 210.
• the tube 215 may further include and/or house other elements such as a wire connector configured to communicate data between a component at the distal end 205, e.g., the sensor 225, and the proximal end 210.
• the proximal end 210 may include, for example, a handle portion 245 that enable an operator to manipulate, advance, retract, and/or orient the distal end 205.
• the proximal end 210 may further include one or more interface 250, e.g., an umbilicus to output data, send or receive electrical signals, and/or communicate a fluid or material into or out from the medical device 110.
• An interface for data may include one or more of a wired or wireless connection.
• the interface 250 may also be configured to receive power for operating the sensor 225 or the end effector 230.
• the medical device 110 does not include a visual navigation element such as a fiber-optic line and lens, a camera, or the like.
• the distal end 205, and in some embodiments, the tube 215, may have an outer diameter that is small relative to a conventional medical device such as a bronchoscope.
• the medical device 110 may have an outer diameter suitable for navigation into a periphery of a lung, e.g., a diameter of 3 millimeters or less.
• the medical device 110 is configured to be disposable, e.g., a single-use device.
• a cost due to disposal of the medical device 110 may be reduced relative to conventional medical devices.
• the display 115 may be configured to output information received from other systems in the environment 100.
• the display 115 may be a monitor, a tablet, a television, a mobile device, etc.
• the display 115 may be integrated into another component of the environment, e.g., the user device 105.
• the medical provider 120 may include and/or represent a person using a computer system, the computer system, and/or an entity that uses the computer system.
• the medical provider 120 may include a medical imaging device such as a CT scanner, an entity such as a hospital or outpatient facility that uses a medical imaging device, a medical data exchange system, or the like.
• the medical provider 120 may generate or otherwise obtain medical imaging data, e.g., by performing medical imaging on a patient and/or perform analysis of the obtained medical imaging data.
• the medical provider 120 may perform a CT scan on a patient, and generate a three-dimensional model and/or two- dimensional image of at least a portion of an anatomy of the patient.
• the medical provider 120 may also obtain any suitable patient-specific information, such as age, medical history, etc.
• the medical provider 120 may provide and/or provide access to medical imaging data and/or any other data to one or more of the other components of the environment 100, e.g., the navigation system 135 as discussed in further detail below.
• the data storage system 125 may include a server system, an electronic medical data system, computer-readable memory such as a hard drive, flash drive, disk, etc.
• the data storage system 125 includes and/or interacts with an application programming interface for exchanging data to other systems, e.g., one or more of the other components of the environment.
• the data storage system 125 may include and/or act as a repository or source for medical imaging data. For example, medical imaging data resulting from a CT scan may be stored by the data storage system 125 and/or provided by the data storage system 125 to the navigation system 135 as discussed in more detail below.
• the electronic network 130 may be a wide area network (“WAN”), a local area network (“LAN”), personal area network (“PAN”), or the like.
• electronic network 130 includes the Internet, and information and data provided between various systems occurs online. “Online” may mean connecting to or accessing source data or information from a location remote from other devices or networks coupled to the Internet. Alternatively, “online” may refer to connecting or accessing an electronic network (wired or wireless) via a mobile communications network or device.
• the Internet is a worldwide system of computer networks — a network of networks in which a party at one computer or other device connected to the network can obtain information from any other computer and communicate with parties of other computers or devices.
• a “website page” generally encompasses a location, data store, or the like that is, for example, hosted and/or operated by a computer system so as to be accessible online, and that may include data configured to cause a program such as a web browser to perform operations such as send, receive, or process data, generate a visual display and/or an interactive interface, or the like.
• the navigation system 135 may one or more of (i) generate, store, train, or use a machine-learning model configured to determine a location of the distal end 205 of the medical device 110, adjust medical imaging data for a patient, e.g., based on a determined location of the distal end 205, to include a visual indicator of that location, operate the display 115 to display the adjusted medical imaging data, among other activities.
• the navigation system 135 may include a machine-learning model and/or instructions associated with the machine-learning model, e.g., instructions for generating a machine-learning model, training the machine-learning model, using the machine-learning model etc.
• the navigation system 135 may include instructions for retrieving medical imaging data, adjusting medical imaging data, e.g., based on the output of the machine learning model, and/or operating the display 115 to output medical imaging data, e.g., as adjusted based on the machine-learning model.
• the navigation system 135 may include training data, e.g., medical imaging data and non-optical in vivo image data from one or more individuals, and may include ground truth, e.g., registration data associating the non-optical in vivo image data with locations in the medical imaging data.
• the non-optical image data includes ultrasound data.
• Ultrasound data generally includes data associated with the internal structure of a portion of the patient’s anatomy that was generated via the application of ultrasound to the anatomy of the patient, whereby pulses of high frequency vibration are transmitted into tissue using a probe, e.g., an ultrasound transducer.
• the vibrations reflect, at least partially, from surfaces that represent a change in acoustic impedance within the body, e.g., a geometry of a structure or tissue. Reflected vibrations that return to the transducer may be transmitted, e.g., via a wire in the tube 215 to a connector on the proximal end 210 and/or to the medical provider system 120 for processing into image data.
• the generation of the image data is based on the time taken by the reflections to return to the transducer after application of the vibrations, and the intensity of the returned reflections.
• a conventional transducer generally is configured to receive variance in signal response across only one dimension. In other words, for a static location of a transducer, only one column of pixel data for an ultrasound image may be received.
• the transducer is generally swept over a field of view, e.g., rotated back and forth, in order to successively add and/or refresh columns of values to the data.
• FIG. 2B depicts an exemplary ultrasound image for a transducer operating in air, such that an essentially blank ultrasound image is formed.
• FIG. 2C depicts an exemplary ultrasound image for a transducer operating in an airway whereby a gap exists between the transducer and surrounding tissue over at least a portion of the sweep of the transducer.
• This gap results in an artifact 280 in the image data known as a “ringdown” artifact, which is generally considered to reduce the diagnostic use of the image data.
• image data even with the presence of such artifacts, may be used for the purpose of navigation.
• the presence of a ringdown artifact may be an indication that the diameter of the lumen in which the medical device is travelling is too large for the device, which may operate as an indication that the device may not have been advanced sufficiently into the periphery of the lung where lumen diameters decrease. Further uses for such imaging data is discussed in more detail below,
• the medical device 110 includes a transducer array.
• a transducer array may include, for example, a plurality of transducers arranged in parallel with each other, e.g., distributed over at least a portion of an outer circumference of the distal end. As a result, multiple columns of data may be sensed at once without rotation of the sensor 225.
• any suitable transducers in parallel may be used. Additional transducers effectively increases the static field of view of the sensor 225.
• the sensor 225, the medical device 110, and/or another system are configured to control the acquisition of data using a sensor with transducers in parallel and/or control the combining of signals from the parallel transducers into combined medical imaging data.
• a system or device other than the navigation system 135 is used to generate and/or train the machine-learning model.
• a system may include instructions for generating the machine learning model, the training data and ground truth, and/or instructions for training the machine-learning model.
• a resulting trained-machine-learning model may then be provided to the navigation system 135.
• a machine-learning model includes a set of variables, e.g., nodes, neurons, filters, etc., that are tuned, e.g., weighted or biased, to different values via the application of training data.
• variables e.g., nodes, neurons, filters, etc.
• training may proceed by feeding a sample of training data into a model with variables set at initialized values, e.g., at random, based on Gaussian noise, a pre-trained model, or the like.
• the output may be compared with the ground truth to determine an error, which may then be back- propagated through the model to adjust the values of the variable.
• Training may be conducted in any suitable manner, e.g., in batches, and may include any suitable training methodology, e.g., stochastic or non-stochastic gradient descent, gradient boosting, random forest, etc.
• a portion of the training data may be withheld during training and/or used to validate the trained machine-learning model, e.g., compare the output of the trained model with the ground truth for that portion of the training data to evaluate an accuracy of the trained model.
• the training of the machine-learning model may be configured to cause the machine-learning model to learn associations between non-optical in vivo image data and medical imaging data, such that the trained machine-learning model is configured to determine an output location within input medical imaging data in response to the input medical imaging data and input non-optical in vivo image data based on the learned associations.
• the machine-learning model may be configured to receive as input medical imaging data and non-optical in vivo image data.
• data may generally be expressed as an array of pixels or voxels.
• a monochromatic two-dimensional image may be represented as a two-dimensional array of values corresponding to intensities of pixels of the image.
• Three- dimensional imaging data e.g., resulting from a CT scan or the like, may be represented as a three-dimensional array.
• the variables of the machine-learning model perform operations on the input data in order to generate an output.
• the output may be a location within the medical imaging data, e.g., three-dimensional coordinates or data indicative thereof. It should be understood that the number of dimensions of the foregoing data is exemplary, and that any suitable type of data may be used e.g., data that has a time component.
• the variables of a machine-learning model may be interrelated in any suitable arrangement in order to generate the output.
• the machine-learning model may include image- processing architecture that is configured to identify, isolate, and/or extract features, geometry, and or structure in one or more of the medical imaging data and/or the non-optical in vivo image data.
• the machine-learning model may include one or more convolutional neural network (“CNN”) configured to identify features in the medical imaging data and/or the non-optical in vivo image data, and may include further architecture, e.g., a connected layer, neural network, etc., configured to determine a relationship between the identified features in order to determine a location in the medical imaging data.
• CNN convolutional neural network
• different samples of training data and/or input data may not be independent.
• the non-optical in vivo image data sensed by the sensor 225 at a current location may be related to the non-optical in vivo image data sensed by the sensor 225 at a previous location.
• factors such as one or more of the geometric characteristics of the patient’s anatomy and the successive travel of the distal end 225 may result in the non-optical in vivo image data sensed by the sensor 225 as successive instances to be related.
• the machine-learning model may be configured to account for and/or determine relationships between multiple samples.
• the machine-learning model of the navigation system 135 may include a Recurrent Neural Network (“RNN”).
• RNNs are a class of feed-forward neural networks that may be well adapted to processing a sequence of inputs.
• the machine learning model may include a Long ShorTerm Memory (“LSTM”) model and/or Sequence to Sequence (“Seq2Seq”) model.
• LSTM model may be configured to generate an output from a sample that takes at least some previous samples and/or outputs into account.
• a Seq2Seq model may be configured to, for example, receive a sequence of non-optical in vivo images as input, and generate a sequence of locations, e.g., a path, in the medical imaging data as output.
• a component or portion of a component in the environment 100 may, in some embodiments, be integrated with or incorporated into one or more other components.
• a portion of the display 115 may be integrated into the entity user device 105 or the computer system associated with a medical provider 120.
• the navigation system 135 may be integrated with the medical provider system 120 and/or the data storage system 125.
• operations or aspects of one or more of the components discussed above may be distributed amongst one or more other components. Any suitable arrangement and/or integration of the various systems and devices of the environment 100 may be used.
• FIG. 3 illustrates an exemplary process for training a machine learning model to determine an output location of a distal end 205 of a medical device 110 in an anatomy of a patient within first medical imaging data in response to input of the first medical imaging data and input of first non-optical in vivo image data received from a sensor positioned on the distal end of a medical device, such as in the various examples discussed above.
• the medical provider 120 may obtain medical imaging data for one or more individuals.
• the medical provider 120 may perform a CT scan of a portion of the one or more individuals’ anatomy, e.g., the peripheral portion of the individuals’ lungs, and or may retrieve such medical imaging data from another source, e.g., the data storage system 115, or another entity such as a hospital or outpatient facility, e.g., via an electronic medical database.
• the individuals may be categorized based on one or more criteria, e.g., age, gender, height, weight, and/or any other suitable demographic data.
• the individuals may not be human.
• training data may be generated from animal studies using species with at least some anatomical similarity to humans, e.g., pigs or the like.
• the obtained medical imaging data is usable to depict a visual representation of the portion of the anatomy of each individual.
• the medical provider 120 may obtain non-optical in vivo image data of the at least portion of the anatomy of the one or more individuals.
• the medical provider 120 e.g., a physician or operator or the like, may introduce a medical device, e.g., the medical device 110 or the like, into the body of the one or more individuals, and capture non-optical in vivo image data as a distal end of the medical device is navigated to a target site within a respective individual.
• the non-optical in vivo image data is ultrasound data.
• the medical device includes a transducer array, such that the ultrasound data is received without requiring sweeping or rotation of the distal end.
• the non-optical in vivo image data includes image data associated with at least a portion of a circumference of an interior of the anatomy.
• the field of view of the non-optical in vivo image data may be 30 degrees, 90 degrees, 180 degrees, 360 degrees, etc.
• the transducer array is configured to continuously capture data, e.g., such that a continuous sequence of data values is captured for each segment of the transducer array as the transducer array travels within the anatomy of the individual.
• the medical provider 120 may obtain location information associated with a location of the distal end of the medical device as the non-optical image data is captured. Any suitable type of location information may be used.
• the distal end of the medical device 110 may include an electromagnetic position sensor, e.g., that uses one or more electromagnetic signals to determine a three-dimensional location of the position sensor.
• the distal end of the medical device may include an optical navigation element, e.g., a camera, optical fiber, lens, or the like, that enables the medical provider 120 to visually inspect the location of the distal end of the medical device within the anatomy of the patient, and enter such data, e.g., via a user device 105.
• the medical device 110 may include a fiber optic shape sensing mechanism.
• the location information includes shape information associated with a shape of the medical device 110.
• an external scanner e.g., a CT scanner, X-ray scanner, or the like, may be operated in conjunction with the travel of the medical device within the individual, and may be used to determine a location of the distal end within the individual. It should be understood that, while the medical device in this method may utilize some of the aforementioned optical navigation elements and techniques for the purpose of generating training data, as discussed in further detail below, there is no requirement that such elements or techniques are used during a procedure using a trained machine-learning model, even when training data used to train the model was collected with a medical device using optical navigation.
• the medical provider 120 may obtain additional data in addition to and/or based on the medical imaging data and/or non- optical in vivo image data. For example, in some embodiments, the medical provider 120 may extract at least one three-dimensional structure from the medical imaging data and/or the non-optical in vivo imaging data. For example, the medical provider 120 may generate a three-dimensional model based on the medical imaging data.
• the navigation system 135 may receive the obtained medical imaging data, non-optical in vivo image data, the location information, and optionally the additional data, and may generate registration data associating the locations at which the non-optical in vivo image data was captured with locations in the medical imaging data.
• generating the registration data may include registering the anatomy of the individual with the medical imaging data and/or the generated three-dimensional model, and then associating the location at which the non-optical in vivo image data was captured with a corresponding location in the registered medical imaging data.
• the medical provider may register a location of a structure extracted from the medical imaging data with a similar structure extracted from the non-optical in vivo imaging data.
• the medical provider 120 may be configured to receive user input, e.g., to set, adjust, or fine tune location information relative to the medical imaging data.
• the display 115 may output the medical imaging data in conjunction with the output of the optical navigation element, and enable the user to set, select, adjust, or tune a location for the current position of the distal end of the medical device in the medical imaging data.
• the shape of the medical device 110 may be registered with geometry of the medical imaging data.
• the navigation system 135 may input the medical imaging data and the non-optical in vivo image data of the at least a portion of the anatomy of the one or more individuals into a machine-learning model as training data.
• the training data is inputted in batches.
• at least a portion of the training data is withheld from the machine learning model to be used as validation data.
• training data is input as a respective sequence corresponding to each one of the one or more individual.
• the navigation system 135 may input the registration data into the machine-learning model as ground truth. In some embodiments, step 330 is performed concurrently, in parallel, or in sequence, e.g., alternatingly, with step 325.
• the navigation system 135 may use the training data and the ground truth with the machine-learning model to develop associations between non-optical in vivo image data and medical imaging data usable by the machine learning model to determine the output location of the distal end of the medical device.
• the navigation system 135 may, e.g., for each sample of training data, batch of training data, or the like, use the machine-learning model to determine an error between the training data and the ground truth, and back- propagate the error in order to adjust one aspect of the machine-learning model.
• the machine-learning model is trained to learn the associations between non-optical in vivo image data and medical imaging data usable by the machine-learning model to determine the output location of the distal end of the medical device.
• the machine learning model may be configured to determine the location of the distal end 205 of the medical device 110 in the first medical imaging data by predicting a path of travel of the distal end 205 of the medical device 110 from a previous location in the first medical imaging data using the first non-optical in vivo image data.
• the machine-learning model and/or the navigation system 135 may be configured to track and or store the location(s) of the distal end 205 over time and/or determine the current location of the distal end 205 based on the previous location(s).
• the machine-learning model may include one or more of a long short term memory network or a sequence-to- sequence model, e.g., as discussed in one or more of the examples above.
• the machine-learning model is configured to learn associations between the shape of the medical device 110, e.g., as the shape changes over time as the medical device 110 is moved, and a position of the distal end 205 within the medical imaging data.
• the machine-learning model is configured to learn associations between a sequence of dimensions or measurements, e.g., diameters such as cross-sectional diameters of a body lumen, determined based on of the non-optical in vivo image data and a path of travel within the medical imaging data.
• the medical provider 120 and/or the navigation system 135 may determine diameters at the locations in the medical imaging data, and/or may determine diameters for the non-optical in vivo image data, and may use such determined diameters as further inputs to the machine-learning model.
• diameters of the locations in the medical imaging data may be determined based on a geometry of the portion of the anatomy associated with the medical imaging data.
• the diameters of the locations in the medical imaging data and the non-optical in vivo image data are used as training data and the determined diameters of the locations are used as ground truth for the machine-learning model and/or another machine-learning model that is configured to output diameters in response to input of non-optical in vivo image data.
• Any suitable technique for determining diameters for interior portions of the anatomy based on non-optical in vivo image data may be used. While some of the embodiments above relate to diameters, it should be understood that, in at least some embodiments, the dimensions or measurements are not limited to circles or approximations of circles, and that any suitable dimension, measurement, and/or geometry may be used.
• the navigation system 135 may validate the trained machine-learning model.
• the navigation system 135 may input training data, e.g., a portion of the training data withheld from the machine-learning model during training, as validation data and use the trained machine-learning model to generate output location(s) of within the medical imaging data.
• the navigation system 135 may then compare locations of the generated output with the locations from the registration data of the ground truth corresponding to the input validation data to generate an accuracy of the trained machine-learning model.
• the navigation system 135 may determine an accuracy based on an average distance between each location in the output and a corresponding location in the registration data. Any suitable accuracy measure may be used.
• the navigation system 135 may validate or reject the training of the machine-learning model based on whether the accuracy is above or below a predetermined threshold, respectively.
• FIG. 4 illustrates an exemplary process for providing in vivo navigation of a medical device, e.g., by utilizing a trained machine-learning model such as a machine-learning model trained according to one or more embodiments discussed above.
• the navigation system 135 may receive first medical imaging data associated with at least a portion of an anatomy of a patient.
• the first medical imaging data may be associated with a CT scan of the patient, or the like.
• the first medical imaging data may be received from a data storage system 125.
• the first medical imaging data may have been obtained, e.g., via medical imaging of the patient, at a previous time, e.g., pre-operatively.
• the first medical imaging data may be received from a medical provider 120, e.g., from a medical imaging scanning device such as a CT scanner or the like that is operated in conjunction with the method.
• the at least portion of the anatomy may be a periphery of a lung of the patient.
• the first medical imaging data may identify a target site within the anatomy of the patient, e.g., a location of undesirable tissue to be ablated, a location of an illness or malady such as a lesion, a foreign mass, or any other suitable medically relevant location.
• the medical provider 120 may insert a distal end 205 of a medical device 110 into the body of the patient, and advance the distal end 205 toward the target site.
• the medical provider 120 may insert the distal end 205 into the body of the patient, bronchially, endoscopically, laparoscopically, or via any other suitable technique.
• the navigation system 135 may receive first non-optical in vivo image data from a sensor positioned on the distal end 205 of the medical device 110.
• the first non-optical in vivo image data may include ultrasound data.
• the sensor may include an ultrasound transducer.
• the ultrasound transducer may be a transducer array.
• the first non-optical in vivo image data may include non- optical image data that extends over a sweep of a circumference, e.g., that has a field of view of 30 degrees, 90 degrees, 180 degrees, 360 degrees, etc., such that the field of view may be obtained without sweeping or rotating the sensor.
• the first non-optical image data may be received, for example, via an interface on a proximal end 210 of the medical device 110.
• the navigation system 135 may receive a position signal from a position sensor positioned proximate to the distal end 205 of the medical device 110, e.g., via the interface.
• the position signal may include information usable to localize a location of the positon sensor to a predetermined region.
• the position signal may include three-dimensional location information accurate to about six inches, three inches, one inch, etc.
• the position sensor may include, for example, an electromagnetic position sensor, a fiber optic shape sensing mechanism, or the like, or combinations thereof.
• the position signal includes information associated with a shape of the medical device 110, or the like.
• the navigation system 135 may extract one or more three-dimensional structure from one or more of the first medical imaging data or the first non-optical in vivo image data.
• the data received at step 405 may include one or more extracted structure, e.g., a geometrical three- dimensional model of the anatomy of the patient, or the like.
• the extracted structure(s) includes a diameter of an interior portion of the anatomy of the patient.
• the navigation system 135 may use a trained machine learning model, e.g., a model trained according to the method of FIG. 3 and/or other embodiments discussed above, to determine a location of the distal end 205 of the medical device 110 in the first medical imaging data.
• the trained machine-learning model may have been trained, based on (i) second medical imaging data and second non-optical in vivo image data of at least a portion of an anatomy of one or more individuals as training data and (ii) registration data associating the second non-optical in vivo image data with locations in the second medical imaging data as ground truth.
• the training may have been configured to cause the trained machine-learning model to learn associations between non-optical in vivo image data and medical imaging data, such that the trained machine-learning model is configured to determine an output location within input medical imaging data in response to the input medical imaging data and input non-optical in vivo image data based on the learned associations.
• the trained machine-learning model includes one or more of a long short term memory network or a sequence-to-sequence model.
• the navigation system 135 may use the position signal to localize a position of the distal end of the medical device to a region within the portion of the anatomy of the patient. In some embodiments, the navigation system 135 restricts the first medical imaging data input into the trained machine-learning model to only the localized region. In some embodiments, the navigation system 135 inputs the localized region as a further input of the trained- machine-learning model. In some embodiments, the trained machine-learning model is further configured to receive the position signal as an input. In some embodiments, the position signal includes one or more of a three-dimensional co ordinate, a three-dimensional region or volume, a shape of the medical device 110, e.g., associated with a fiber optic shape sensor, etc.
• the navigation system 135 is configured to register at least one structure extracted from the first non-optical in vivo image data with geometry of the at least portion of the anatomy of the patient in the first medical imaging data, e.g., with at least one structure extracted from the first medical imaging data.
• the determination of the location of the distal end 205 of the medical device 110 is further based on the registration of the at least one three-dimensional structure with the geometry.
• the registration and/or the one or more extracted structure(s) may be used as further input(s) to the trained machine-learning model.
• the registration may be used to identify a localized region of the anatomy for determining the location.
• the trained machine-learning model was trained to learn associations between non-optical in vivo image data and dimensions or measurements, e.g., diameters of interior portions of anatomy.
• the navigation system 135 may use the trained machine-learning model to determine a diameter of an interior portion of the anatomy of the patient at a current location of the distal end 205 of the medical device 110.
• the navigation system 135 may compare the current diameter with geometry of the medical imaging data to identify a location in the medical imaging data matching the determined diameter, e.g., in order to determine the location of the distal end 205 of the medical device 110.
• the trained machine-learning model was trained to learn associations between a sequence of non-optical in vivo images of the non-optical in vivo image data and a path of travel within the medical imaging data.
• the trained machine learning-model is configured to determine the location of the distal end 205 of the medical device 110 in the first medical imaging data by predicting a path of travel of the distal end 205 of the medical device from a previous location in the first medical imaging data using the first non-optical in vivo image data.
• the trained machine-learning model may be configured to accept the first medical imaging data and a sequence of non-optical in vivo images in the first non-optical in vivo imaging data as input, and generate a sequence of locations, e.g., a path, in the first medical imaging data as output.
• the trained machine-learning model was trained to learn associations between a sequence of dimensions or measurements such as diameters, e.g., that were determined based on the non-optical in vivo image data, and a path of travel within the medical imaging data.
• the trained machine-learning model may be configured to determine the location of the distal end 205 of the medical device 110 in the first medical imaging data by predicting sequence indicative of a path of travel of the distal end 205 of the medical device 110 from a previous location in the first medical imaging data using the first non- optical in vivo image data.
• the navigation system 135 may modify the first medical imaging data to include a location indicator indicative of the determined location of the distal end of the medical device.
• the location indicator may include one or more of a graphic or object indicative of a location of the distal end 205, e.g., a geometric shape such as an arrow, circle, etc., a graphic or object indicative of a path of the distal end 205, e.g., a solid line, dashed line, coloration of portion(s) of the first medical imaging data that has been passed through, etc.
• the location indicator includes a representation of the medical device 110 and/or the distal end 205 within the medical imaging data.
• the first medical imaging data may be further adjusted to include a depiction of at least one structure extracted from one or more of the first medical imaging data or the first non-optical in vivo imaging data. In some embodiments, the first medical imaging data may be further adjusted to include a rendering or three- dimensional model of the anatomy of the patient at the current location of the distal end 205. In some embodiments, the first medical imaging data may be further adjusted to include an image generated based on the first non-optical in vivo image data.
• first medical imaging data may be further adjusted to include a visual depiction of additional data, such as a distance between the distal end 205 and the target site or a confirmation that the target site has been reached by the distal end 205, e.g., as determined by the navigation system 135.
• additional data such as a distance between the distal end 205 and the target site or a confirmation that the target site has been reached by the distal end 205, e.g., as determined by the navigation system 135.
• the navigation system 135 may cause the display 115 to output the modified first medical imaging data including the location indicator.
• the display 115 may show the first medical imaging data as a map of the anatomy of the patient, with the location indicator identifying the current location of the distal end 205 of the medical imaging device.
• FIGs. 4B and 4C depict different exemplary embodiments of outputs 475 that may be produced by the navigation system 135.
• an output 475 produced by the navigation system 135 may include one or more of medical imaging data 477 depicting at least a portion of an anatomy of a patient, location information 479 and/or path information 481 overlaid over the medical imaging data 477 and indicating a current and/or past location of the medical device 110, ultrasound imaging data 483 overlaid at corresponding location(s) of the medical imaging data, or the like.
• the navigation system 135, user device 105, touch-screen input of the display 115 or the like is configured to receive an input from a user 140, e.g., to manipulate a perspective of view for the first medical imaging data, to include, move, adjust, and/or remove in the output further information such as the image generated based on the first non-optical in vivo image data or the additional data discussed above.
• the medical provider 120 may move the medical device 110, e.g., such that the position of the distal end 205 within the anatomy of the patient changes.
• steps 415 to step 440 may be iterated, e.g., so as to account for the new position of the distal end 205 of the medical device 110.
• the navigation system 135 may receive further non-optical in vivo image data from the sensor, and may use the trained machine learning model to determine an updated location of the distal end 205 of the medical device 110 based on the further non-optical in vivo image data.
• the navigation system 135 may update the first medical imaging data to adjust the location indicator based on the updated location of the distal end 205 of the medical device 110, and may update the display 115 to output the updated first medical imaging data. In some embodiments, such iteration may occur in real-time or near real-time such that the display 115 is configured to output a live location of the distal end 205 of the medical device 110.
• the navigation may be configured to cause the display 115 to output a target site confirmation upon the navigation system 135 determining that the distal end 205 has reached the target site.
• the medical provider 120 may perform a procedure at the target site using the medical device 110.
• the medical provider may activate an end effector 230, e.g., an ablation device, in order to ablate tissue at the target site.
• an end effector 230 e.g., an ablation device
• the navigation system 135 and/or the medical provider may confirm completion of the procedure based on further non-optical in vivo image data received from the sensor at the distal end 205.
• the procedure may involve a modification to the geometry of the anatomy of the patient.
• the navigation system 135 may be configured to extract one or more modified structures of the anatomy of the patient, and compare the modified structure(s) with previously extracted structure(s).
• the first medical imaging data may have identified tissue to be ablated, and the navigation system 135 may be configured to identify whether said tissue has been ablated or still persists within the patient.
• the medical provider 120 may retract the medical device 110 out from the body of the patient. In some embodiments, the medical provider 120 may dispose of the medical device 110.
• the medical device 110 includes a fiber-optic light and a receiver fiber-optic that is usable for position sensing of the distal end 205.
• any process or operation discussed in this disclosure that is understood to be computer-implementable, such as the processes illustrated in FIGs. 3 and 4, may be performed by one or more processors of a computer system, such any of the systems or devices in the environment 100 of FIG. 1 , as described above.
• a process or process step performed by one or more processors may also be referred to as an operation.
• the one or more processors may be configured to perform such processes by having access to instructions (e.g., software or computer-readable code) that, when executed by the one or more processors, cause the one or more processors to perform the processes.
• the instructions may be stored in a memory of the computer system.
• a processor may be a central processing unit (CPU), a graphics processing unit (GPU), or any suitable types of processing unit.
• a computer system such as a system or device implementing a process or operation in the examples above, may include one or more computing devices, such as one or more of the systems or devices in FIG. 1.
• One or more processors of a computer system may be included in a single computing device or distributed among a plurality of computing devices.
• a memory of the computer system may include the respective memory of each computing device of the plurality of computing devices.
• FIG. 5 is a simplified functional block diagram of a computer 500 that may be configured as a device for executing the methods of FIGs. 3 and 4, according to exemplary embodiments of the present disclosure.
• the computer 500 may be configured as the navigation system 135 and/or another system according to exemplary embodiments of this disclosure.
• any of the systems herein may be a computer 500 including, for example, a data communication interface 520 for packet data communication.
• the computer 500 also may include a central processing unit (“CPU”) 502, in the form of one or more processors, for executing program instructions.
• CPU central processing unit
• the computer 500 may include an internal communication bus 508, and a storage unit 506 (such as ROM, HDD, SDD, etc.) that may store data on a computer readable medium 522, although the computer 500 may receive programming and data via network communications.
• the computer 500 may also have a memory 504 (such as RAM) storing instructions 524 for executing techniques presented herein, although the instructions 524 may be stored temporarily or permanently within other modules of computer 500 (e.g., processor 502 and/or computer readable medium 522).
• the computer 500 also may include input and output ports 512 and/or a display 510 to connect with input and output devices such as keyboards, mice, touchscreens, monitors, displays, etc.
• the various system functions may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load. Alternatively, the systems may be implemented by appropriate programming of one computer hardware platform.
• Storage type media include any or all of the tangible memory of the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming. All or portions of the software may at times be communicated through the Internet or various other telecommunication networks.
• Such communications may enable loading of the software from one computer or processor into another, for example, from a management server or host computer of the mobile communication network into the computer platform of a server and/or from a server to the mobile device.
• another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links.
• the physical elements that carry such waves, such as wired or wireless links, optical links, or the like, also may be considered as media bearing the software.
• terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.
• the disclosed methods, devices, and systems are described with exemplary reference to transmitting data, it should be appreciated that the disclosed embodiments may be applicable to any environment, such as a desktop or laptop computer, an automobile entertainment system, a home entertainment system, etc. Also, the disclosed embodiments may be applicable to any type of Internet protocol.

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Biomedical Technology (AREA)
• Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• General Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• General Physics & Mathematics (AREA)
• Public Health (AREA)
• Medical Informatics (AREA)
• Molecular Biology (AREA)
• Data Mining & Analysis (AREA)
• Biophysics (AREA)
• General Engineering & Computer Science (AREA)
• Computing Systems (AREA)
• Mathematical Physics (AREA)
• Software Systems (AREA)
• Evolutionary Computation (AREA)
• Computational Linguistics (AREA)
• Artificial Intelligence (AREA)
• Epidemiology (AREA)
• Primary Health Care (AREA)
• Surgery (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Business, Economics & Management (AREA)
• General Business, Economics & Management (AREA)
• Robotics (AREA)
• Heart & Thoracic Surgery (AREA)
• Animal Behavior & Ethology (AREA)
• Veterinary Medicine (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Radiology & Medical Imaging (AREA)
• Databases & Information Systems (AREA)
• Pathology (AREA)
• Image Analysis (AREA)
• Ultra Sonic Daignosis Equipment (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=82786882

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (1)

Citations (3)

Family Cites Families (15)

• 2022
  • 2022-06-21 EP EP22750975.9A patent/EP4360014A1/en active Pending
  • 2022-06-21 WO PCT/US2022/073050 patent/WO2022272239A1/en not_active Ceased
  • 2022-06-21 JP JP2023578787A patent/JP2024523449A/ja active Pending
  • 2022-06-21 US US17/807,893 patent/US20220401154A1/en active Pending
  • 2022-06-21 CN CN202280056666.1A patent/CN117941008A/zh active Pending
  • 2022-06-21 AU AU2022299436A patent/AU2022299436A1/en active Pending

Patent Citations (3)

Also Published As

Similar Documents

Legal Events