JP7282685B2 - A robotic system for navigation of luminal networks with compensation for physiological noise - Google Patents

Description

This application claims the benefit of U.S. Provisional Patent Application No. 62/480,257, entitled "Robotic System for Navigation of Luminal Networks Compensating for Physiological Noise," filed March 31, 2017, and The entire disclosure is incorporated by reference.

The systems and methods of the present disclosure relate to intraluminal procedures and, more particularly, to intraluminal navigation.

Bronchoscopy is a medical procedure that allows a practitioner to examine the internal condition of a patient's pulmonary airways, such as the bronchi and bronchioles. In a bronchoscopic procedure, a flexible tubular instrument called a bronchoscope is inserted into the patient's mouth, down the patient's throat and into the lung airways, where it is identified for later diagnosis and treatment. Guided to the tissue site. A bronchoscope can include an internal lumen (a "working channel") that provides a pathway to a tissue site through which catheters and various medical devices can be inserted into the tissue site.

Pulmonologists can avoid surgical trauma by relying on the patient's breathing cycle for decisions and actions. An example of such an action includes inserting a biopsy device to collect a sample of tissue, eg, via a bronchoscope. Around and adjacent to the lung airways, the circumference of the airways changes according to the respiratory phases of the lungs. When the patient inhales during the inspiratory phase of the respiratory cycle, the airway diameter expands, and when the patient exhales during the expiratory phase of the cycle, the airway diameter contracts. During the procedure, the pulmonologist observes the patient to determine whether the patient is in the inspiratory or expiratory phase and whether a particular fixed-diameter instrument or endoscope can enter the airway. to judge whether The airway can be wrapped around the device without causing trauma during exhalation, but attempting to pass the device through the constricted airway during exhalation can cause significant trauma, e.g., piercing a blood vessel.

The above problems, among others, are solved in some embodiments by the luminal network navigation system and method of the present disclosure. The disclosed luminal network navigation systems and techniques incorporate respiratory rate and/or respiratory volume into the navigation framework to implement patient safety measures (e.g., instrument control techniques, user interface alerts, notifications, etc.). ). As an example of an instrument control approach, the robotic system of the present disclosure can automatically perform respiratory gating to prevent the user (practitioner) from unintentionally traumatizing the patient's airway. Here, "respiratory gating" means synchronizing the operation of an instrument within the patient's airway with the patient's breathing. In some instances, device control techniques have the potential for trauma from device insertion during expiration, e.g., small-diameter airways in the lung periphery, where the diameter of the airway during inspiration is approximately the diameter of the device. Identification of regions (“warning zones”) in the patient's airways is included. The robotic system can perform respiratory gating to control the instrument when it is determined that the instrument is located within one of the identified regions. As an example of a user interface warning, the robotic system can provide visual and/or audible indications of inspiration, expiration, and/or instrument positioning within specified warning zones during a bronchoscopy procedure. can. The user can perform instrument control based on a warning from the user interface, for example, by performing respiratory gating manually or by controlling the robot system to perform respiratory gating.

Some embodiments of the disclosed bronchoscopy navigation systems and techniques relate to identifying and compensating for movement caused by a patient's breathing to provide more accurate localization of instruments within the patient's actuations. For example, electromagnets (electromagnetic sensors) are provided as instruments located within the patient's actuations. The navigation system may apply a filter to the instrument's position information from the electromagnetic sensor to remove signal noise due to the periodic movement of the respiratory tract caused by respiration. The number of periodic respiratory movements can be obtained from data from one or more additional sensors. In some implementations, the inspiratory and expiratory cycles can be determined, in one example, based on data from additional electromagnetic sensors, accelerometers, and/or acoustic respiration sensors placed on the patient's body. . In some implementations, the rate can be obtained from other types of sensors or systems, such as breath rate information from a ventilator used to control a patient's breathing, or a breathing rate information placed to monitor the patient. respiration rate information extracted from automated analysis of images received from the optical sensor.

Accordingly, one aspect is a system for navigating a patient's luminal network comprising a generator for generating an electromagnetic field and a set of one or more electromagnetic sensors provided at the distal end of a steerable instrument. , a set of one or more respiratory sensors, at least one computer readable memory having executable instructions stored thereon, and one or more processors in communication with the at least one computer readable memory. executing the instructions to access at least a pre-operative model representing the luminal network in the system; and a mapping between the coordinate system of the electromagnetic field and the coordinate system of the pre-operative model. calculating the position of at least one of the set of electromagnetic sensors within the electromagnetic field based on data signals from the set of electromagnetic sensors; and based on data signals from the set of respiratory sensors. calculating the respiration rate of the patient; and determining the steerable relative to the pre-operative model based on the registration mapping, the respiration rate, and the position of at least one of the set of electromagnetic sensors within the electromagnetic field. and determining the position of the distal end of an instrument.

In some implementations, each electromagnetic sensor of the set of electromagnetic sensors produces a signal indicative of the distance and angle between the electromagnetic sensor and the generator, and the electrical signal is the electromagnetic sensor within the electromagnetic field. It can be used to determine one or both of the position and orientation of the sensor.

In some implementations, the one or more processors execute the instructions to provide the system with one or more data signals from the set of respiratory sensors, one or more of the one or more data signals. Transforming to a frequency domain representation and determining a respiration rate from the frequency domain representation of the one or more data signals are performed. In some implementations, the one or more processors execute the instructions to cause the system to apply a prediction filter to one or more data signals from at least the set of electromagnetic sensors. wherein the prediction filter predicts respiratory motion due to the respiration; and removing components of the one or more data signals attributable to the predicted respiratory motion to render the steerable relative to the preoperative model. and determining the position of the distal end of the instrument.

In some implementations, the one or more processors execute the instructions to instruct the system to detect at least one of the set of respiration sensors during an inspiratory phase and an expiratory phase of the respiration of the patient. Calculating the magnitude of one displacement is performed. In some implementations, the one or more processors execute the instructions to cause the system to determine the position of at least one of the set of electromagnetic sensors relative to at least the set of respiratory sensors; (1) the determined at least one position of the set of electromagnetic sensors relative to the set of respiration sensors; and (2) the at least one displacement of the set of respiration sensors between inspiratory and expiratory phases. calculating displacement of at least one position of said set of electromagnetic sensors between said inspiratory phase and said expiratory phase based on the magnitude; and determining the position of the distal end of the steerable instrument relative to the pre-operative model based on the calculated positional displacement of at least one of a set of electromagnetic sensors. In some implementations, the set of respiration sensors includes a first additional electromagnetic sensor positioned at a first location on the body surface and a first additional electromagnetic sensor employed at a second location on the body surface. and a second additional electromagnetic sensor being used as a sensor, wherein a first magnitude of displacement of said first additional electromagnetic sensor between said inspiratory phase and said expiratory phase is determined by said second The second position is spaced from the first position such that the second displacement magnitude of the additional electromagnetic sensor is greater than the second displacement magnitude. In some implementations, the one or more processors execute the instructions to instruct the system to detect at least the set of electromagnetic sensors for the first additional electromagnetic sensor and the second additional electromagnetic sensor. and interpolating between the first displacement magnitude and the second displacement magnitude based on the determined relative positions of the set of electromagnetic sensors; said calculation of the displacement of the position of said set of electromagnetic sensors between an inspiratory phase and said expiratory phase is performed based on said interpolated magnitudes. In some implementations, the one or more processors execute the instructions to instruct the system to determine at least one position of the pre-operative model based on the calculated at least one displacement magnitude. estimating a motion vector for , translating the pre-operative model in the coordinate system of the electromagnetic field based on the estimated motion vector, and based on the translated pre-operative model, the steering and determining said position of said distal end of a possible instrument. In some implementations, to translate the pre-operative model within the coordinate system of the electromagnetic field, the one or more processors execute the instructions to instruct the system to perform at least the first displacement. moving a first portion of the model to a first new coordinate based on the magnitude; and moving a second portion of the model to a second new coordinate based on the magnitude of the second displacement. and let it run.

Some implementations further include a robotic system having an instrument driver that performs movement of the steerable instrument. In some implementations, the one or more processors execute the instructions to instruct the system to perform the maneuver based at least on the position of the distal end of the steerable instrument relative to the pre-operative model. Identifying a possible next movement of the instrument and directing operation of the instrument driver to perform said next movement. Some implementations further include a display, wherein the one or more processors execute the instructions to instruct the system to at least indicate the position of the distal end of the steerable instrument relative to the pre-operative model. Generating a graphical representation and displaying the generated graphical representation on the display are performed. In some implementations, the robotic system has an input device that controls movement of the steerable instrument based on user manipulation of the input device.

In some implementations, the pre-operative model comprises a 3D computed tomography model of the luminal network of the patient.

Another aspect is an apparatus for determining navigation of a luminal network of a patient, comprising at least one computer readable memory having executable instructions stored thereon and in communication with said at least one computer readable memory 1 one or more processors that execute the instructions to access to the system at least a model representing the luminal network; accessing a mapping to and from a coordinate system of an electromagnetic field; receiving data from an electromagnetic sensor at the distal end of a steerable instrument being used and inserted into the luminal network; calculating the position of the electromagnetic sensor within the electromagnetic field based on data from the electromagnetic sensor, and receiving data from at least one additional sensor that detects movement of the luminal network. calculating a period of periodic movement of the luminal network based on data from the at least one additional sensor; mapping, the period, and the electromagnetic sensor within the electromagnetic field; and determining the position of the distal end of the steerable instrument relative to the model based on the position.

In some implementations, the at least one additional sensor comprises one or more electromagnetic sensors.

In some implementations the at least one additional sensor comprises an accelerometer.

In some implementations, the at least one additional sensor comprises an acoustic respiration sensor, wherein the acoustic respiration sensor detects the periodic movements during patient respiration.

In some implementations, the luminal network comprises a respiratory airway, and the one or more processors execute the instructions to cause the device to steer the steerable instrument through the luminal network. Make it do what it induces.

In some implementations, the one or more processors execute the instructions to provide the system with one or more data signals from at least the at least one additional sensor, the one or more Transforming to a frequency domain representation of a data signal and identifying the period of periodic movement from the frequency domain representation of the one or more data signals are performed. In some implementations, the one or more processors execute the instructions to cause the system to apply a filter to one or more data signals from at least the electromagnetic sensor, wherein the filter attenuates a portion of the one or more data signals at the specified period; and based on the one or more data signals from the electromagnetic sensor to which the filter is applied, for the model and determining the position of the distal end of the steerable instrument.

In some implementations, the luminal network comprises a respiratory airway, and the one or more processors execute the instructions to cause at least a period between inspiratory and expiratory phases of the patient's respiration. Calculating a magnitude of said at least one displacement of said at least one additional sensor is performed. In some implementations, the one or more processors execute the instructions to determine the position of the electromagnetic sensor relative to the at least one additional sensor; (1) the at least one additional sensor; and (2) the magnitude of the at least one displacement of the at least one additional sensor between the inspiratory phase and the expiratory phase. calculating a positional displacement of the electromagnetic sensor during an exhalation phase; and based on the calculated positional displacement of the electromagnetic sensor between the inspiratory phase and the expiratory phase, relative to the preoperative model. and determining the position of the distal end of the steerable instrument. In some implementations, the at least one additional sensor comprises a first additional electromagnetic sensor used positioned at a first location on the body surface and a first additional electromagnetic sensor positioned at a second location on the body surface. and a second additional electromagnetic sensor being used as a sensor, wherein a first magnitude of displacement of said first additional electromagnetic sensor between said inspiratory phase and said expiratory phase is determined by said first The second position is spaced from the first position such that the second displacement magnitude of the two additional electromagnetic sensors is greater. In some implementations, the one or more processors execute the instructions to provide the system with the position of the electromagnetic sensor relative to at least the first additional electromagnetic sensor and the second additional electromagnetic sensor. and interpolating between the first displacement magnitude and the second displacement magnitude based on the determined position of the electromagnetic sensor, wherein the inspiratory phase and the said calculation of the displacement of said electromagnetic sensor's position between exhalation phases is performed based on said interpolated magnitude. In some implementations, the 1
The one or more processors instruct the system by executing the instructions to at least estimate a motion vector for at least one position of the model based on the calculated at least one displacement magnitude; Translating the model in the coordinate system of the electromagnetic field based on the estimated motion vector and determining the position of the distal end of the steerable instrument based on the translated model. make things happen. In some implementations, to translate the pre-operative model within the coordinate system of the electromagnetic field, the one or more processors execute the instructions to instruct the system to perform at least the first displacement. moving a first portion of the model to a first new coordinate based on the magnitude; and moving a second portion of the model to a second new coordinate based on the magnitude of the second displacement. to move to and run.

Another aspect is a non-transitory computer readable storage medium having instructions stored therein for use in at least one computing device inserted into at least a tissue site of a patient when the instructions are executed. receiving first data from an electromagnetic sensor of an instrument and second data from at least one additional sensor that detects movement of the tissue site; and based on the first data, measuring the tissue site calculating a position of an electromagnetic sensor within an electromagnetic field located around a; and calculating a period of periodic movement of the tissue site based on second data; (1) a period of the tissue site and (2) determining the position of the instrument relative to the tissue site based on the period of motion and (2) the position of the electromagnetic sensor within the field.

In some implementations, execution of the instructions causes the at least one computing device to transform the second data into a frequency domain representation and determine the period of the periodic movement from the frequency domain representation. to do and to do. In some implementations, when the instructions are executed, the at least one computing device applies a filter to the first data, the filter filtering the first data at the specified periodicity. weakening a portion; and determining the position of the instrument based on the filtered first data.

In some implementations, the tissue site comprises a respiratory airway, and execution of the instructions instructs the at least one computing device to perform the at least one addition between the inspiratory and expiratory phases of the respiration of the patient. calculating the magnitude of the displacement of at least one of the sensors. In some implementations, when the instructions are executed, the at least one computing device determines the position of the electromagnetic sensor relative to the at least one additional sensor; based on the determined position of the electromagnetic sensor and (2) the magnitude of the at least one displacement of the at least one additional sensor between the inspiratory phase and the expiratory phase; and calculating the displacement of the position of the electromagnetic sensor between the inspiratory phase and the expiratory phase, based on the calculated positional displacement of the electromagnetic sensor between and determining the position of the distal end of the instrument. In some implementations, the at least one additional sensor comprises a first additional electromagnetic sensor positioned at a first location on the patient and a first additional electromagnetic sensor positioned at a second location on the patient. and a second additional electromagnetic sensor being used, wherein a first magnitude of displacement of said first additional electromagnetic sensor between said inspiratory phase and said expiratory phase is equal to said second The second position is spaced from the first position to be greater than the second displacement magnitude of the additional electromagnetic sensor, and the at least one computing device is instructed to execute the instructions. determining the position of the electromagnetic sensor relative to one additional electromagnetic sensor and the second additional electromagnetic sensor; and based on the determined position of the electromagnetic sensor, determining the magnitude of the first displacement and the first displacement. performing an interpolation between two displacement magnitudes, wherein the calculation of the displacement of the position of the electromagnetic sensor between the inspiratory phase and the expiratory phase is performed on the basis of the interpolated magnitudes. to be done.

In some implementations, execution of the instructions causes the at least one computing device to access data, the data being a model representing a topography of the tissue site, a coordinate system of the field and the and a mapping to and from a coordinate system of a model, wherein the determination of the position of the instrument is performed based on the mapping, the period, and the position of the electromagnetic sensor within the field. In some implementations, the tissue site comprises a respiratory airway, and execution of the instructions instructs the at least one computing device to perform the at least one addition between the inspiratory and expiratory phases of the respiration of the patient. calculating a magnitude of at least one displacement of the sensor of; estimating a motion vector for at least one position of the model based on the calculated at least one magnitude of displacement; Translating the model in a coordinate system based on the obtained motion vector; and Determining the position of the instrument based on the translated model. In some implementations, to translate the pre-operative model within the coordinate system of the electromagnetic field, upon execution of the instructions, instruct the at least one computing device, based on the magnitude of the first displacement, to: moving a first portion of the model to a first new coordinate; moving a second portion of the model to a second new coordinate based on the magnitude of the second displacement; to run.

Another aspect provides first data from a first sensor of an instrument being used inserted into a tissue site of a patient and second data from at least one additional sensor that detects movement of the tissue site. receiving data; calculating a position of the first sensor within a volume around the tissue site based on the first data; and determining a position of the tissue site based on second data. and determining the position of the instrument relative to the tissue site based on the period and the position of the first sensor within the volume.

Some implementations are performed by one or more hardware processors.

Some implementations further include transforming the second data to a frequency domain representation and identifying the period of the periodic movement from the frequency domain representation.

Some implementations are applying a filter to the first data, the filter weakening a portion of the first data at the specified period; determining the position of the instrument based on the first data;
further includes

In some implementations, the tissue site comprises a respiratory airway, and the method comprises measuring a magnitude of at least one displacement of the at least one additional sensor between an inspiratory phase and an expiratory phase of the respiration of the patient. further comprising calculating Some implementations comprise determining a position of the first sensor relative to the at least one additional sensor; (1) the determined position of the first sensor relative to the at least one additional sensor; (2) the position of the first sensor during the inspiratory phase and the expiratory phase based on the magnitude of the at least one displacement of the at least one additional sensor between the inspiratory and expiratory phases; and determining the position of the device based on the calculated positional displacement of the first sensor between the inspiratory phase and the expiratory phase. In some implementations, the at least one additional sensor comprises a first additional sensor positioned and used at a first location on the patient and a first additional sensor positioned and used at a second location on the patient. wherein a first displacement magnitude of said first additional sensor between said inspiratory phase and said expiratory phase is determined by said second additional sensor wherein the second position is spaced from the first position to be greater than a second displacement magnitude of the method, wherein the method comprises: for the first additional sensor and the second additional sensor Determining a position of a first sensor and interpolating between the first displacement magnitude and the second displacement magnitude based on the determined position of the first sensor. and wherein the calculation of the displacement of the position of the first sensor between the inspiratory phase and the expiratory phase is performed based on the interpolated magnitude. Some implementations further include accessing data, the data representing a model representing the topography of the tissue site and a mapping between the field coordinate system and the model coordinate system. , the method further includes determining the position of the instrument based on the mapping, the period, and the position of the electromagnetic sensor within the field. In some implementations, the tissue site comprises a respiratory airway, and the method determines a magnitude of at least one displacement of the at least one additional sensor between an inspiratory phase and an expiratory phase of the respiration of the patient. estimating a motion vector for at least one position of the model based on the calculated at least one displacement magnitude; and based on the estimated motion vector, a coordinate system Further comprising translating the model and determining the position of the instrument based on the translated model. Some implementations are translating the model within the coordinate system, moving a first portion of the model to a first new coordinate based on the magnitude of the first displacement. and moving a second portion of the model to a second new coordinate based on the magnitude of the second displacement; and translating the model based on.

Another aspect is a system for navigating a patient's luminal network comprising: a generator for generating an electromagnetic field; a set of one or more electromagnetic sensors provided at the distal end of a steerable instrument; at least one respiration sensor, at least one computer readable memory having executable instructions stored thereon, and one or more processors in communication with the at least one computer readable memory to execute the instructions. accessing the system with at least a pre-operative model representing the luminal network; accessing a mapping between the coordinate system of the electromagnetic field and the coordinate system of the pre-operative model; calculating the position of one of the set of electromagnetic sensors within the electromagnetic field based on the data signals from the electromagnetic sensors of the mapping and the position of the set of electromagnetic sensors within the electromagnetic field determining the position of the distal end of the steerable instrument relative to the pre-operative model; and determining the data signals from the set of electromagnetic sensors based on data from the at least one respiratory sensor. determining whether the respiratory phase of the patient at the time of acquisition corresponds to an inspiratory phase or an expiratory phase; and based on the position of the distal end of the steerable instrument relative to the model and the respiratory phase. and determining whether to activate a safe mode for subsequent movement of the steerable instrument.

In some implementations, the one or more processors execute the instructions to cause the system to enter the safe mode, and in the safe mode, cause the system to synchronize the next movement of the respiratory phase. implement one or more safety features that induce

In some implementations, the one or more processors execute the instructions to cause the system to access information regarding a navigational path of the luminal network to a target tissue site; Using the position of the distal end of the steerable instrument relative to the pre-operative model, the distal end of the steerable instrument is positioned within a predefined safe area of the luminal network. and activating the safe mode based on determining that the steerable instrument is located within the predefined safe area. In some implementations, the navigation path includes multiple regions, wherein the safe region is the lumen where the difference between the diameter of the airway and the diameter of the distal end of the steerable instrument is less than a predetermined value. Deployed as part of a network.

In some implementations, in the secure mode, the one or more processors:
Execution of the instructions causes the system to output information indicative of the respiratory phase to a user.

In some implementations, the system further comprises a robotic system, the robotic system being a display and an input device for transmitting signals to control movement of the steerable instrument in response to user manipulation of the input device. An input device for generating and an instrument driver for effecting movement of the steerable instrument based on the signal from the input device. In some implementations, the one or more processors execute the instructions to instruct the system to activate the instrument driver during the exhalation phase of the patient's respiration in response to activation of the safe mode. deter. In some implementations, the one or more processors execute the instructions to instruct the system to override the attempted actuation of the instrument driver based on user manipulation of the input device. , inhibiting said actuation of said instrument driver. In some implementations, the one or more processors execute the instructions to cause the system to output graphical representations of the inspiratory and expiratory phases of the breath that can be displayed on the display.

In some implementations, the pre-operative model comprises a 3D computed tomography model of the luminal network of the patient.

Another aspect is an apparatus for guiding navigation of a luminal network of a patient, comprising at least one computer readable memory having executable instructions stored thereon and in communication with said at least one computer readable memory 1 One or more processors that access at least data to the device by executing the instructions, the data being a model representing the luminal network, a coordinate system of the model and the luminal a mapping between a coordinate system of an electromagnetic field generated around a network, a signal from an electromagnetic sensor at the distal end of a steerable instrument used and inserted into said luminal network, and said luminal network; and accessing data corresponding to the signals from at least one additional sensor detecting movement of the electromagnetic sensor within the electromagnetic field based on the data corresponding to the signal from the electromagnetic sensor calculating the position of the steerable instrument based on the position of the distal end of the steerable instrument relative to the model; calculating the next movement of the steerable instrument based on the position of the distal end of the steerable instrument relative to the model; determining whether the patient's respiratory phase at the time of acquisition of the signal from the first sensor corresponds to an inspiratory phase or an expiratory phase, based on the data corresponding to the signal from determining whether to activate a safety mode for the next movement of the steerable instrument based on the respiratory phase.

In some implementations, the at least one additional sensor comprises one or more electromagnetic sensors.

In some implementations the at least one additional sensor comprises an accelerometer.

In some implementations, the at least one additional sensor comprises an acoustic respiration sensor, wherein the acoustic respiration sensor detects the periodic movements during patient respiration.

In some implementations, the one or more processors execute the instructions to cause the system to enter the safe mode and, in the safe mode, instruct the system to perform the next shift of the respiratory phase. Implement one or more safety features that induce synchronization.

In some implementations, the one or more processors execute the instructions to cause the system to access information regarding a navigational path of the luminal network to a target tissue site; Using the position of the distal end of the steerable instrument relative to the pre-operative model, the distal end of the steerable instrument is positioned within a predefined safe area of the luminal network. and activating the safe mode based on determining that the steerable instrument is located within the predefined safe area. In some implementations, the navigation path includes multiple regions, wherein the safe region is the lumen where the difference between the diameter of the airway and the diameter of the distal end of the steerable instrument is less than a predetermined value. Deployed as part of a network. In some implementations, in the safe mode, the one or more processors execute the instructions to cause the system to output information indicative of the respiratory phase to a user.

Some implementations further comprise a robotic system, wherein the robotic system is a display and an input device that generates signals that control movement of the steerable instrument in response to user manipulation of the input device. and an instrument driver that effects movement of the steerable instrument based on the signal from the input device. In some implementations, the one or more processors execute the instructions to instruct the system, in response to activation of the safe mode, during one or more exhalation phases of the breathing of the patient. Inhibit the operation of the instrument driver. In some implementations, the one or more processors execute the instructions to instruct the system to override the attempted actuation of the instrument driver based on user manipulation of the input device. , inhibiting said actuation of said instrument driver. In some implementations, the one or more processors execute the instructions to cause the system to output graphical representations of the inspiratory and expiratory phases of the breath that can be displayed on the display.

Another aspect is a non-transitory computer readable storage medium having instructions stored thereon, the instrument being used inserted into at least a patient's luminal network in a processor of a device when the instructions are executed. receiving first data from a first sensor of and second data from at least one additional sensor that detects movement of said luminal network; and based on said first data, said calculating the position of the first sensor within a field located around a tissue site; and based on the second data, the patient's position at the time of acquisition of the first data from the first sensor. determining whether a respiratory phase corresponds to an inspiratory phase or an expiratory phase; determining a position of the device based on the mapping and the position of the first sensor within the field; Determining whether to activate a safety mode based on the position of the device and the respiratory phase.

In some implementations, when the instructions are executed, at least the processor receives image data from the at least one additional sensor; and based on the image data, determines whether the respiratory phase is the inspiratory phase or the and determining which of the expiratory phases corresponds to.

In some implementations, when the instructions are executed, the processor at least receives accelerometer data from the at least one additional sensor; corresponds to said inspiratory phase or said expiratory phase.

In some implementations, when the instructions are executed, the processor is provided with (1) data corresponding to a model representing the topography of the luminal network and (2) a coordinate system of the field and a coordinate system of the model. and determining the position of the instrument relative to the model based on the mapping and the position of the first sensor within the field. Including determining the position of the instrument.

In some implementations, one or more of, upon execution of the instructions, causing the processor to at least activate the safe mode and, in the safe mode, to induce the processor to synchronize the next movement of the respiratory phase. implementation of safety functions.

In some implementations, when the instructions are executed, the processor accesses information regarding at least a navigational path of the luminal network to a target tissue site; , identifying that the instrument is being used placed within a predefined safe area of the lumen network; and determining that the instrument is located within the predefined safe area. activating the safe mode based on . In some implementations, the navigation path includes multiple regions, wherein the safe region is the lumen where the difference between the diameter of the airway and the diameter of the distal end of the steerable instrument is less than a predetermined value. Deployed as part of a network.

In some implementations, execution of the instructions causes the processor to at least output information indicative of the respiratory phase to a user in response to a determination to activate the safe mode.

In some implementations, execution of the instructions causes the processor to at least inhibit activation of a robotic instrument driver during the exhalation phase of the patient's respiration in response to a determination to activate the safe mode. and the robotic instrument driver performs movement of the instrument through the luminal network.

Another aspect provides first data from a first sensor of an instrument being used inserted into a patient's luminal network and second data from at least one additional sensor that detects movement of the luminal network. receiving data from 2; calculating, based on the first data, the position of the first sensor within a field located around the tissue site; and based on the second data, determining whether the respiratory phase of the patient at the time of acquisition of the first data from the first sensor corresponds to an inspiratory phase or an expiratory phase; determining a position of the device based on the position; determining a next movement of the device based on the position; and activating a safe mode for the next movement based on the respiratory phase. determining whether to activate.

Some implementations are performed by one or more hardware processors.

Some implementations further include activating the safety mode and implementing one or more safety features in the safety mode to induce synchronization of the subsequent movement of the respiratory phase. .

Some implementations access data corresponding to (1) a model representing the luminal network topography and (2) a mapping between the field coordinate system and the model coordinate system. and determining the position of the instrument includes determining the position of the instrument relative to the model based on the mapping and the position of the first sensor within the field.

Some implementations include: accessing information regarding a navigational path of the luminal network to a target tissue site; identifying that the instrument is positioned within a defined safe area and being used; and activating the safe mode based on determining that the instrument is positioned within the predefined safe area. and further including.

Some implementations further include having a user output information representative of the respiratory phase in response to a determination to activate the safe mode.

Some implementations are responsive to a decision to activate the safe mode to inhibit activation of a robotic instrument driver during the exhalation phase of the patient's respiration, wherein the robotic instrument driver activates the luminal network. further comprising moving the instrument through. In some implementations, inhibiting actuation of the robotic instrument driver includes disabling user input to perform the next movement during the exhalation phase or a subsequent exhalation phase.

Aspects of the present disclosure are described below in conjunction with the accompanying drawings and tables, which are illustrative and not limiting of aspects of the disclosure, and like components are labeled with like names.

1 illustrates an example operating environment for implementing the disclosed navigation system and techniques; FIG. 1B illustrates an example lumen network navigated in the environment of FIG. 1A; FIG. 1B illustrates an example robotic arm guiding movement of an instrument through the luminal network of FIG. 1B. FIG. FIG. 12 illustrates an example command console of an example surgical robotic system, according to one embodiment. 1 illustrates an example endoscope having image generation and EM sensing capabilities of the present disclosure; FIG. 1 shows a schematic block diagram of a navigation system of the present disclosure; FIG. FIG. 10 is a flowchart of an example process for filtering noise caused by luminal network movement from instrument position estimation results of the present disclosure; FIG. 6 shows a flowchart of various example processes that may be used in the adjustment block of FIG. 5; FIG. 6 shows a flowchart of various example processes that may be used in the adjustment block of FIG. 5; FIG. 6 shows a flowchart of various example processes that may be used in the adjustment block of FIG. 5; FIG. 10 is a flowchart of an example process for enabling safe mode when navigating a luminal network of the present disclosure; [0014] Fig. 4 illustrates an example user interface that may be presented to a user when navigating a lumenal network in the secure mode of the present disclosure; [0014] Fig. 4 illustrates an example user interface that may be presented to a user when navigating a lumenal network in the secure mode of the present disclosure;

Disclosed embodiments analyze multiple navigation-related data sources to refine the estimation of the position and orientation of a medical device in a luminal network, such as a lung airway, and apply a filter to one data of the device to Assist navigation in the luminal network by removing noise from periodic motion of the luminal network and/or activating respiratory gating or other types of safety features that adjust navigation control based on the cyclic motion Regarding systems and methods.

A bronchoscope can include a light source and a miniature camera to allow the practitioner to examine the patient's trachea and airways. Not knowing the exact position of the bronchoscope within the patient's airway can result in trauma to the patient. To locate the bronchoscope, image-based bronchoscope guidance systems use data from the bronchoscope's camera to register at bifurcations of the patient's airways (e.g., at specific locations within the luminal network). ), which is advantageously less susceptible to position errors due to patient respiratory motion. However, because image-based guidance methods rely on bronchoscope images, the bronchoscope images can be affected by artifacts in the images, such as patient coughing and mucus blockages.

Electromagnetic navigation-guided bronchoscopy (EMN bronchoscopy) is a type of bronchoscopic procedure that implements electromagnetic (EM) technology to locate and guide endoscopic instruments and catheters within the airways of the lungs. be. An EMN bronchoscopy system may employ an electromagnetic field generator that generates a low intensity, variable electromagnetic field to establish the location of a tracked site around the patient's luminal network. An electromagnetic field is a physical field produced by charged objects that affects charged objects in the vicinity of the electromagnetic field. Electromagnetic sensors attached to objects located within the generated field can be used to track the position and orientation of these objects within the electromagnetic field. A small current is induced in the electromagnetic sensor by the changing electromagnetic field. The properties of these electrical signals depend on the distance and angle between the sensor and the electromagnetic field generator. Thus, an EMN bronchoscopy system can have an electromagnetic field generator, a steerable channel with an electromagnetic sensor at or near its tip, and a guidance computer system. An electromagnetic field generator generates an electromagnetic field around the luminal network of the patient being navigated, eg, airway, gastrointestinal tract, circulatory pathways. A steerable channel is inserted into the working channel of the bronchoscope and tracked in the electromagnetic field via an electromagnetic sensor.

Prior to beginning an EMN bronchoscopy procedure, a virtual three-dimensional (3D) bronchial map of a patient's specific airway structures can be obtained, for example, by preoperative computed tomography (CT) chest scan. Using this map and the EMN bronchoscopy system, practitioners can navigate to desired locations within the lungs to biopsy diseased tissue, stage lymph nodes, guide radiotherapy or guide brachytherapy. Marker insertion etc. for can be performed. For example, registration can be performed at the start of the procedure to generate a mapping between the electromagnetic field coordinate system and the model coordinate system. This tracks the steerable channel during bronchoscopy so that the position of the steerable channel in the model coordinate system is nominally known based on the position data from the electromagnetic sensors. However, patient breathing can cause chest motion, leading to mis-association of the position of the operable instrument and/or model with the coordinate system of the electromagnetic field. These errors are magnified in the peripheral airway when the bifurcation of the airway becomes smaller and more subject to movement due to patient breathing.

Here, a coordinate system is a reference system in a specific sensing mode. For example, for electromagnetic data, the electromagnetic coordinate system is the frame of reference defined by the electromagnetic field source (field generator). For CT images and segmented 3D models, this frame of reference is based on the system defined by the scanner. The navigation system represents (registers) these different data sources (within their respective frames of reference) to a 3D model (such as a CT system), e.g. solve the problem.

Thus, as described in more detail below, the disclosed luminal network navigation systems and techniques combine input from image-based, robotic, and electromagnetic navigation systems, as well as input from other patient sensors. Combinations can reduce navigational problems and enable more efficient endoscopic procedures. For example, a navigation fusion framework can analyze image information received from the instrument's camera, positional information from electromagnetic sensors at the tip of the instrument, and robot positional information from the robotic system that guides the movement of the instrument. Based on the analysis, the navigation fusion framework can make instrument localization and/or navigation decisions based on one or more of these types of navigation data. Some implementations of the navigational fusion framework can also identify the instrument position relative to the 3D model of the luminal network. In some embodiments, a filter is applied to the instrument position information from the electromagnetic sensors, e.g. Signal noise due to periodic motion of the network can be removed. The periodic motion frequency can be obtained from data from one or more additional sensors. For example, respiratory rate may be data from additional electromagnetic sensors, accelerometers, and/or acoustic respiratory sensors placed on the patient's body, and/or optical sensors placed at locations that have a field of view to observe patient movement. can be specified based on Some embodiments may implement navigational safety features based on one or both of instrument position and luminal network periodic motion. For example, in bronchoscopy implementations, safety features include display of respiratory rate information and/or restrictions imposed on instrument insertion during exhalation.

The disclosed systems and techniques can provide advantages in bronchoscopy guidance systems and other applications, including other types of endoscopic procedures for navigation of luminal networks. In anatomy, "lumen" can mean the space or cavity within a tubular organ such as airways, blood vessels, or intestines. As used herein, "luminal network" means a sculptural structure having at least one lumen extending toward a target tissue site, such as pulmonary airways, circulatory system, digestive system. Thus, although the present disclosure presents examples of navigation systems related to bronchoscopy, the safety and data filtering aspects of the disclosure may be applied to other medical systems for navigating a moving luminal network of a patient. can also be applied.

Various embodiments are described below with reference to figures for illustrative purposes. Many other implementations of the disclosed concepts are possible and various advantages can be obtained with the disclosed implementations. Headings are included for reference and to facilitate identification of the various sections. These headings are not intended to limit the technical ideas of this disclosure with respect to the headings. Those technical ideas may be applied throughout this specification.

(Overview of example navigation system)
FIG. 1A illustrates an exemplary operational environment 100 for implementing one or more aspects of the disclosed navigation systems and techniques. The operating environment 100 includes a patient 101, a platform 102 that supports the patient 101, a surgical robotic system 110 that guides movement of the endoscope 115, a command center 105 that controls the operation of the surgical robotic system 110, an electromagnetic controller 135, and an electromagnetic field generator. 120, including electromagnetic sensors 125,130. FIG. 1A also outlines a region of luminal network 140 within patient 101, details of which are shown in FIG. 1B.

Surgical robotic system 110 includes one or more robotic arms that position endoscope 115 and guide movement of endoscope 115 in luminal network 140 of patient 101 . Command center 105 may be communicatively connected to surgical robotic system 110 for receiving position data and/or providing control signals from a user. Here, "communicatively connected" means a wireless WAN (Wide Area Network) (eg, one or more cellular networks), a wireless LAN (Local Area Network) (eg, IEEE 802.11 (Wi-Fi ( Any wired and/or wireless data transmission medium including, but not limited to, those configured for one or more standards such as (registered trademark)), Bluetooth (registered trademark), data transfer cables, etc. means. Surgical robotic system 110 is described in detail with reference to FIG. 1C, and command center 105 is described in detail with reference to FIG.

The endoscope 115 is inserted into the patient's body to acquire images of the body (such as body tissue) and to provide a working channel for inserting other medical instruments to the target tissue site. It is a tubular flexible surgical instrument. In some implementations, endoscope 115 may be a bronchoscope. Endoscope 115 has one or more image generating devices (eg, cameras or other types of optical sensors) at its distal end. An image generating device may have one or more optical components such as optical fibers, fiber arrays, photosensitive substrates, and/or lenses. The optics move with the distal end of the endoscope 115, and as the distal end of the endoscope 115 moves, the field of view of the image acquired by the image generating device correspondingly changes. The distal end of endoscope 115 may be provided with one or more electromagnetic sensors 125 for tracking the position of the distal end within the electromagnetic field generated surrounding luminal network 140 . The distal end of endoscope 115 is further described below with reference to FIG.

Electromagnetic controller 135 can control electromagnetic field generator 120 to generate a variable electromagnetic field. Electromagnetic fields can be time-varying and/or spatially-varying depending on the environment. In some embodiments, the electromagnetic field generator 120 can be an electromagnetic field generating substrate. Some embodiments of the disclosed patient navigation system can employ an electromagnetic field generator substrate positioned between the patient and the platform 102 supporting the patient, the electromagnetic field generator substrate having an underlying Thin film barriers can be incorporated to mitigate tracking distortion caused by deposited conductive or magnetic materials. In other embodiments, the electromagnetic field generator board can be mounted, for example, on a robotic arm similar to that shown in surgical robotic system 110 to provide flexible configuration options around the patient.

The electromagnetic space measurement system incorporated into command sensor 105, surgical robotic system 110, and/or electromagnetic controller 135 is provided by an electromagnetic field or coils of electromagnetic sensors, such as electromagnetic sensors 125, 130, embedded in coils of electromagnetic sensors. It can locate objects within an electromagnetic field. When an electromagnetic sensor is placed in a controlled and variable electromagnetic field as described herein, a voltage is induced in the coil of the sensor. This induced voltage can be used by electromagnetic space measurement systems to calculate the position and orientation of electromagnetic sensors, as well as objects with electromagnetic sensors. Because the strength of the magnetic field is weak and can safely pass through human tissue, object localization is possible without the line-of-sight limitations of optical spatial measurement systems.

It can be coupled to the distal end of an endoscope 115 to track its position within the electromagnetic field. The electromagnetic field is stationary with respect to the field generator, and the coordinate system of the 3D model of the lumen network can be mapped to the coordinate system of the electromagnetic field. However, the patient's airway, and thus the distal end of the endoscope 115 placed in the airway, may exhibit movement relative to the electromagnetic field generator 120 due to the patient's respiratory rate, and the endoscope 115 relative to the model. introduces potential error in determining the position of the distal end of the .

Accordingly, a number of additional electromagnetic sensors 130 may be provided on the patient's body surface (eg, within the region of the luminal network 140) to track breathing-induced displacements. A number of different electromagnetic sensors 130 can be spaced on the body surface to track different displacements at these locations. For example, the periphery of the lung may exhibit greater motion due to respiration than the central airways, and providing several electromagnetic sensors allows for a more accurate analysis of these motion effects. For example, the distal end of the endoscope 115 travels through different regions of the luminal network 140 and thus experiences varying degrees of displacement due to patient respiration as it travels through these different regions. can receive The disclosed position filtering technique can correlate the approximate position of the distal end of the endoscope 115 with a plurality of additional electromagnetic sensors, and determine the magnitude of the identified displacement of these particular additional electromagnetic sensors. It can be used, for example, to correct for noise or artifacts in the endoscope position signal due to airway motion via filtering/removal of the respiratory motion artifact component of the endoscope position signal.

In other embodiments, other types of sensors configured to detect motion of the patient's luminal network can be used in place of or in addition to the additional electromagnetic sensors 130. can. For example, one or more inertial sensors (eg, accelerometers, gyroscopes, etc.) can be placed on the patient's body surface and used to estimate chest surface displacement during respiration. In another example, an acoustic respiration sensor may be placed on the patient's body surface within a region of the airway (eg, lumen network region 140) and used to measure the inspiratory and expiratory phases of the respiratory cycle. good. In another example, an optical sensor (eg, an imaging device) can capture a series of images of the patient's body, and these images can be analyzed to identify respiratory phase and/or displacement. In some embodiments, the patient 101 is breathing with assistance from a ventilator during the procedure, and the ventilator (and/or a device communicatively coupled to the ventilator) Data representing inspiratory and expiratory phases can be provided.

FIG. 1B shows an exemplary lumen network 140 that can be navigated in the operational environment 100 of FIG. 1A. The luminal network 140 includes the branching structures of the patient's airway 150 and lesions 155 that can be accessed as described herein for diagnosis and/or treatment. As shown, lesion 155 is located in the periphery of airway 150 . Endoscope 115 has a first diameter so that its distal end cannot be positioned through a smaller diameter airway around lesion 155 . Thus, steerable catheter 155 extends the remaining distance from endoscope 115 working channel to lesion 155 . Steerable catheter 145 may have a lumen through which instruments such as biopsy needles, cytology brushes, and/or tissue sampling forceps may be passed to the target tissue site of lesion 155 . In such embodiments, both the distal end of endoscope 115 and the distal end of steerable catheter 145 may be provided with electromagnetic sensors for tracking their position within airway 150 . In other embodiments, the overall diameter of endoscope 115 may be small enough to reach the periphery without steerable catheter 155, or for deploying medical devices through non-steerable catheters. may be small enough (eg, within 2.5-3 cm) to be close to the perimeter. Medical instruments deployed through endoscope 115 can be equipped with electromagnetic sensors, and the positional filtering and safe mode navigation techniques described below can be applied to such medical instruments.

In some embodiments, a 2D display of the 3D luminal network model of the present disclosure, or a cross-section of the 3D model may resemble FIG. 1B. Navigation safe areas and/or navigation route information can be overlaid on such representations.

FIG. 1C shows an exemplary robotic arm 175 of surgical robotic system 110 for guiding instrument movement through lumen network 140 of FIG. 1B. Surgical robotic system 110 includes a base 180 coupled to one or more robotic arms (eg, robotic arm 175). The robotic arm 175 includes a plurality of arm segments 170 jointed at joints 165, which provide the robotic arm 175 with multiple degrees of freedom. As an example, one embodiment of robotic arm 175 may have seven degrees of freedom corresponding to seven arm segments. In some embodiments, the robotic arm 175 includes set-up joints that use a combination of brakes and counterweights to maintain the position of the robotic arm 175 . The counterweight can include gas springs or coil springs. Brakes, eg, fail-safe brakes, can include mechanical and/or electrical components. Additionally, robotic arm 175 may be a gravity-assisted passive support robotic arm.

The robotic arm 175 can be coupled to an Instrument Device Manipulator (IDM) 190 using a Mechanism Changer Interface (MCI) 160 . IDM 190 can be removed and replaced with a different type of IDM, such as a first type of IDM configured to operate an endoscope or a second type of IDM configured to operate a laparoscope. . MCI 160 includes connectors for communicating pneumatic, power, electrical and optical signals from robotic arm 175 to IDM 190 . The MCI 160 can be a setscrew or baseplate connection. IDM 190 manipulates surgical instruments, such as endoscope 115, using techniques including direct drives, harmonic drives, gear drives, belts and pulleys, magnetic drives, and the like. The MCI 160 is interchangeable based on the type of IDM 190 and can be customized for specific types of surgical procedures. The arm of robotic system 175 can include joint-level torque sensing and a wrist at the distal end.

The robotic arm 175 of the surgical robotic system 110 can manipulate the endoscope 115 using elongated moving members. Elongated moving members can include pull wires, also called pull wires or push wires, cables, fibers, or flexible shafts. For example, robotic arm 175 can actuate multiple pull wires coupled to endoscope 115 to redirect the tip of endoscope 115 . Pull wires may include both metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. Endoscope 115 can exhibit non-linear motion in response to forces applied by the elongate moving member. Non-linear motion can be based on the stiffness and compressibility of the endoscope 115 and variability in slack or stiffness between different elongate moving members.

Base 180 is positioned so that a user, such as a physician, can comfortably use a command console to control surgical robotic system 110 while robotic arm 175 has access to perform or assist in performing surgical procedures on a patient. can be In some embodiments, base 180 may be coupled to an operating table or bed for supporting the patient. Base 180 can be communicatively coupled to command console 105 shown in FIG. 1A.

The base 180 includes a power source 182, pneumatic pressure 186, and control and sensor electronics 184 including components such as a central processing unit, data buses, control circuits, and memory, and associated actuators such as motors for moving the robotic arm 175. can include Electronics 184 may implement the navigation control techniques, safety modes, and/or data filtering techniques of the present disclosure. Electronics 184 in base 180 can also process and transmit control signals communicated from the command console. In some embodiments, base 180 includes wheels 188 for carrying surgical robotic system 110 and wheel locks/brakes (not shown) for wheels 188 . The mobility of the surgical robotic system 110 facilitates adaptation to space constraints within the operating room and facilitates proper positioning and movement of surgical instruments. Additionally, being movable allows the robotic arm 175 to be configured such that it does not interfere with the patient, physician, anesthesiologist, or any other equipment. During operation, a user can control robotic arm 175 using a control device, such as a command console.

FIG. 2 illustrates an exemplary command console 200 that can be used as command console 105 in exemplary operating environment 100 . Command console 200 includes a console base 201 , a display module 202 such as a monitor, and control modules such as keyboard 203 and joystick 204 . In some embodiments, one or more of the functionality of command console 200 may be integrated into base 180 of surgical robotic system 110 or another system communicatively coupled to surgical robotic system 110 . A user 205, eg, a physician, uses command console 200 to remotely control surgical robotic system 110 from an ergonomic position.

Console base 201 includes a central processing unit, storage, data bus, and associated data responsible for interpreting and processing signals such as camera images and tracking sensor data from endoscope 115 shown in FIGS. 1A-1C. Communication ports can be included. In some embodiments, both console base 201 and base 180 perform signal processing for load balancing. The console base 201 can also process commands and instructions provided by the user 205 via the control modules 203,204. In addition to the keyboard 203 and joystick 204 shown in FIG. 2, the control module captures other devices such as computer mice, trackpads, trackballs, control pads, controllers such as handheld remote controls, and hand and finger gestures. Sensors (eg, motion sensors or cameras) can be included. The controller can include a set of user inputs (eg, buttons, joysticks, directional pads, etc.) mapped to instrument movements (eg, articulation, actuation, water flushing, etc.).

A user 205 can control a surgical instrument such as an endoscope 115 using command console 200 in velocity mode or position control mode. In velocity mode, the user 205 directly controls the pitch and yaw motion of the distal end of the endoscope 115 under direct manual control using the control module. For example, movement on joystick 204 can be mapped to yaw and pitch movement at the distal end of endoscope 115 . Joystick 204 can provide tactile feedback to user 205 . For example, joystick 204 can vibrate to convey that endoscope 115 cannot translate or rotate further in a particular direction. Command console 200 also provides visual feedback (e.g., pop-up messages) and/or audio feedback (e.g., beeps) to indicate that endoscope 115 has reached maximum translation or rotation. can also Haptic and/or visual feedback may be provided by the system operating in a safe mode during patient exhalation, as described in more detail below.

In position control mode, command console 200 uses a three-dimensional (3D) map of the patient lumen network and input from the navigation sensors of the present disclosure to control surgical instruments, such as endoscope 115 . Command console 200 provides control signals to robotic arm 175 of surgical robotic system 110 to maneuver endoscope 115 to a target location. Due to its reliance on 3D maps, the position control mode may require accurate mapping of the patient's anatomy.

In some embodiments, the user 205 can manually operate the robotic arm 175 of the surgical robotic system 110 without using the command console 200 . During setup in the operating room, user 205 operates robotic arm 175, endoscope 115 (or an endoscope), and other surgical equipment to access the patient. Surgical robotic system 110 determines the appropriate configuration of robotic arm 175 and apparatus based on force feedback and inertial control from user 205 .

Display 202 may include an electronic monitor (eg, LCD display, LED display, touch-sensitive display), virtual reality viewing device (eg, goggles or glasses), and/or other display device. In some embodiments, the display module 202 is integrated with the control module, eg, as a tablet device with a touch screen. In some embodiments, one display 202 displays a 3D model of the patient's luminal network and virtual navigation information (eg, a virtual representation of the end of the endoscope in the model based on electromagnetic sensor positions), and the other A display 202 may display image information received from a camera or another sensing device at the end of endoscope 115 . In some implementations, the integrated display 202 and control module can be used by the user 205 to both review data and enter commands to the surgical robotic system 110 . The display 202 can display 3D images and/or 2D renderings of 3D images using a stereoscopic device, eg, a visor or goggles. The 3D images provide an "endoscopic view" (ie, an endoscopic view), which is a computer 3D model of the patient's anatomy. An "endoscopic view" provides a virtual environment of the expected position of the endoscope 115 in and within a patient. The user 205 compares the "endoscope view" model to the actual image captured by the camera to mentally determine or confirm that the endoscope 115 is in the correct or nearly correct position within the patient. Useful. An “endoscopic view” provides information about the anatomy, such as the shape of the airway, circulation, or patient's bowel or colon around the distal end of the endoscope 115 . Display module 202 can simultaneously display a 3D model and a CT scan of the anatomy around the distal end of endoscope 115 . Additionally, the display module 202 can overlay the already determined navigation path of the endoscope 115 on the 3D model and CT scan.

In some embodiments, a model of endoscope 115 is displayed along with the 3D model to help illustrate surgical conditions. For example, CT scans identify lesions within the anatomy that may require biopsies. During operation, display module 202 can display a reference image acquired by endoscope 115 corresponding to the current position of endoscope 115 . Display module 202 can automatically display different views of a model of endoscope 115 depending on user settings and the particular surgical procedure. For example, the display module 202 shows a fluoroscopic overhead view of the endoscope 115 during the navigation steps as the endoscope 115 approaches the patient's surgical field.

FIG. 3 shows a distal end 300 of an exemplary endoscope, such as endoscope 115 of FIGS. 1A-1C, having imaging and electromagnetic sensing capabilities of the present disclosure. In FIG. 3, the endoscope distal end 300 includes an imaging device 315 , an illumination source 310 , and the end of an electromagnetic sensor coil 305 . Distal end 300 is the working channel of the endoscope through which surgical instruments such as biopsy needles, cytology brushes, and forceps are inserted along the endoscope shaft to allow access to areas near the endoscope tip. Further includes an opening to 320 .

Illumination source 310 provides light for illuminating a portion of the anatomical space. The illumination sources can each be one or more light emitting devices configured to emit light at a selected wavelength or range of wavelengths. This wavelength can be any suitable wavelength, such as visible spectrum light, infrared, X-ray (eg, fluoroscopy), to name a few. In some embodiments, illumination source 310 can include a light emitting diode (LED) located at distal end 300 . In some embodiments, the illumination source 310 is one or more optical fibers that extend throughout the endoscope to transmit light from a remote light source, e.g., an x-ray generator, through the distal end 300. can include Where distal end 300 includes multiple illumination sources 310, each of these can be configured to emit light of the same or different wavelengths as one another.

Imaging device 315 is any photosensitive substrate or structure configured to convert energy representing received light into an electrical signal, such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) image sensor. can contain. Some examples of imaging devices 315 include one or more sensors configured to transmit images from the distal end 300 of the endoscope to an eyepiece and/or image sensor at the proximal end of the endoscope. It may include optical fibers, such as fiber optic bundles. Imaging device 315 may further include one or more lenses and/or wavelength pass or cutoff filters as required by various optical designs. Light emitted from illumination source 310 enables imaging device 315 to acquire images of the interior of the patient's luminal network. These images can then be sent as individual frames or a series of consecutive frames (eg, video) to a computer system, such as command console 200, for processing in the present disclosure.

An electromagnetic coil 305 disposed on the distal end 300 may be used in conjunction with an electromagnetic tracking system to detect the position and orientation of the distal end 300 of the endoscope while positioned within the anatomical system. . In some embodiments, the angles may be set so that the coils 305 provide sensitivity to electromagnetic fields along different axes, giving the disclosed navigation system a full six degrees of freedom: three positions and three angles. The ability to measure degrees can be given. In other embodiments, a single coil may be disposed on or within distal end 300 with its axis oriented along the endoscope shaft of the endoscope. Due to the rotational symmetry of such a system, it is insensitive to rotation about its axis, and in such an embodiment only 5 degrees of freedom can be detected.

FIG. 4 shows a schematic block diagram of an exemplary navigational fusion system 400 of the present disclosure. As described in detail below, using box 400, data from a number of different sources are combined with real-time movement and position/orientation information of surgical instruments (eg, endoscopes) within the patient's luminal network. are combined and iteratively analyzed during the surgical procedure to provide an estimate of and make navigational decisions. System 400 includes respiratory sensor data repository 405, endoscopic electromagnetic sensor data repository 415, registration data repository 475, model data repository 425, endoscopic imaging data repository 480, navigation path data repository 445, safe area data repository 455. , and several data repositories including a robot position data repository 470 . Although shown separately in FIG. 4 for clarity of the following discussion, some or all of the data repositories may be stored together in a single memory or set of memories. The system 400 also includes a respiratory rate and/or phase identifier 410, an endoscope position estimator 420, a registration calculator 465, a position calculator 430, an image analyzer 435, a state estimator 440, a safe mode controller 450, and several processing modules including navigation controller 460 . Each module represents a set of computer-readable instructions stored in memory and one or more processors configured by the instructions to together perform the functions described below. For example, navigation fusion system 400 may be implemented as one or more data storage devices and one or more hardware processors in control and sensor electronics 184 and/or console base 201 described above.

Respiratory sensor data repository 405 is a data store that stores data derived from respiratory sensors. As described above, the respiratory sensors may comprise electromagnetic sensors, acoustic respiratory sensors, image sensors having a field of view positioned to acquire an image of the luminal network, and ventilator inflation/deflation information. In some embodiments, the respiratory sensor may comprise several electromagnetic sensors, and the data in respiratory sensor data repository 405 includes, for each sensor, time-dependent position data representing the sensor's position within the electromagnetic field over time. be able to. For example, each sensor's data can be stored as a set of the form (x, y, z, t _n ), where x, y, z are the coordinates of the sensor in the region at time period t _n . show. Respiratory sensor data repository 405 can store many such tuples for each sensor corresponding to many different times.

Respiration rate and/or phase identifier 410 is a module configured to receive data from respiration sensor data relocation 405 and analyze such data to calculate respiration rate and/or respiration phase. Respiratory rate refers to the time interval between successive phases, eg, between successive cycles of inspiration and expiration. Phase refers to whether the respiratory cycle is in the inspiratory phase (eg, while the patient is inhaling) or the expiratory phase (while the patient is exhaling). Some embodiments may use data from one or all of the sensors in various implementations to extract respiration rate from respiration sensor data using a Fourier transform.

Endoscope electromagnetic sensor data repository 415 is a data storage device that stores data derived from electromagnetic sensors at the distal end of the endoscope. As noted above, such sensors can include electromagnetic sensor coils 305, and the resulting data can be used to determine the position and orientation of the sensor within the electromagnetic field. Similar to data from electromagnetic respiratory sensors, endoscopic electromagnetic sensor data can be stored as a set of the form (x, y, z, t _n ), where x, y, and z are time periods represents the coordinates of the sensor in the electromagnetic field at t _n .
Some embodiments can further include instrument roll, pitch, and yaw within the electromagnetic sensor tuple. Endoscope electromagnetic sensor data repository 415 can store many such tuples for each endoscope-based sensor corresponding to many different times.

In some embodiments, the endoscope position estimator 420 receives data from the endoscope electromagnetic sensor data repository 415 and from the respiratory rate and/or phase identifier 410 and uses that data to A module that reduces "noise" in the signal received from the endoscopic electromagnetic sensor due to periodic motion of the luminal network. For example, in one embodiment, endoscope position estimator 420 may generate a filter based on the determined respiration rate and apply the filter to the endoscope electromagnetic sensor data. In another embodiment, the endoscopic position estimator 420 can identify the magnitude of displacement of one or more respiratory sensors during respiration, and the displacement value can be converted to the position indicated by the endoscopic electromagnetic sensor data. can be applied as a bias to This involves, for example, identifying the respiratory sensor displacement at time period _tn and applying it as a bias to the endoscope position at time period _tn , identifying the next respiratory sensor displacement at time period tn ₊₁ and applying it as It can be done dynamically, such as by applying a bias to the endoscope position at time period t _n+1 .

Model data repository 425 is a data storage device that stores data representing a model of a patient's luminal network. As an example, a preoperative procedure can be performed to perform CT scans of the patient's lungs, and the computer system can use the data from these scans to construct a 3D model of the patient's lungs. Such models can provide 3D information regarding the structure and connectivity of the luminal network, including in some examples the topography and/or diameter of the patient's airways. Several CT scans are performed during breath-hold so that the patient's airways are dilated to their full diameter.

Alignment calculator 465 performs an alignment between the coordinate system of the 3D model (eg, the coordinate system of the CT scanner used to generate the model) and the coordinate system of the electromagnetic field (eg, the electromagnetic field of electromagnetic field generator 120). Or a module that can identify a mapping. In order to track the sensors through the patient's anatomy, the navigation fusion system 400 may require a process known as "registration," whereby the registration calculator 465 uses different coordinate systems. Identify a geometric transformation that aligns a single object between. For example, certain anatomical regions of a patient can have representations in 3D model coordinates and also in electromagnetic sensor coordinates. To perform the initial alignment calculations, one embodiment of the alignment calculator 465 uses the Tubular Network Navigation Registration can be performed as described in US patent application Ser. No. 15/268,238 entitled "Navigation of Tubular Networks". As an example of one possible registration technique, the registration calculator 465 collects endoscopic imaging data as the endoscope is inserted into the patient's airway, e.g., as the endoscope reaches various bifurcations. Data can be received from repository 480 and electromagnetic sensor data repository 415 at several different points. The image data can be used to identify when the distal end of the endoscope has reached the bifurcation, eg, via automated feature analysis. Alignment calculator 465 receives data from endoscope electromagnetic sensor data repository 415 and can identify electromagnetic sensor locations at the distal end of the endoscope when the endoscope is positioned at the bifurcation. . In some examples, other points in the patient's airway can be used in addition to bifurcations, and such points can be mapped to corresponding points in a "skeletal" model of the airway. Alignment calculator 465 can use data linking at least three of the electromagnetic locations to points in the model to identify geometric transformations between the electromagnetic domain and the model. Another embodiment can include manual registration, for example, by taking at least three from the first branch of the patient's airway and from two or more branches of the left and right lungs, corresponding Alignment calculations can be performed using the points. This data (also called alignment data) for performing geometric transformations can be stored in alignment data repository 475 as alignment data.

After the initial registration is complete, the registration calculator 465 provides an estimate of the registration transform based on the received data to improve the transform accuracy and compensate for variations in the navigation system, such as changes due to patient movement. can be updated. In some aspects, the registration calculator 465 continuously estimates the registration transformation at defined intervals and/or based on the position of the endoscope (or parts thereof) within the luminal network. can be updated.

Alignment data repository 475 is a data store that stores alignment data that can be used to perform geometric transformations from the coordinate system of the electromagnetic field to the coordinate system of the model, as described herein. Also, as noted above, alignment data may be generated by alignment calculator 465 and may be updated continuously or periodically in some embodiments.

Position calculator 430 is a module that receives data from model data repository 425, alignment data repository 475, and scope position estimator 420 and converts electromagnetic sensor coordinates to 3D model coordinates. Scope position estimator 420 calculates the initial position of the electromagnetic sensor relative to the position of the electromagnetic field generator, as described above. This position also corresponds to a position in the 3D model. To transform the electromagnetic sensor initial position from the electromagnetic coordinate system to the model coordinate system, the position calculator 430 performs the conversion between the electromagnetic coordinate system stored in the alignment data repository 475 and the model coordinate system (eg, alignment data). You can access the mapping. To convert the scope position to the 3D model coordinate system, the position calculator 430 takes as inputs data representing the topography of the 3D model from the model data repository 425 and the electromagnetic fields and coordinate system of the 3D model from the alignment data repository 475. and the position of the scope within the electromagnetic field from scope position estimator 420 . Also, in some embodiments, pre-estimated state data may be received from state estimator 440 . Based on the received data, position calculator 430 can, for example, perform on-the-fly conversion of the electromagnetic sensor position data to positions within the 3D model. This may represent a preliminary estimate of the position of the distal tip of the scope within the topography of the 3D model, and the state estimator 440 for producing a final estimate of scope position, as described in more detail below. can be provided as one input to

Endoscopic imaging data repository 480 is a data storage device that stores image data received from an endoscope's camera, eg, imaging device 315 . In various implementations, the image data may be separate images or a series of image frames within a video sequence.

Image analyzer 435 is a module that can receive data from endoscope imaging data repository 480 and model data repository 425 and compare this data to determine endoscope positioning. For example, the image analyzer 435 can access volume-rendered or surface-rendered intraluminal images of the airway system from the model scan and compare the rendered images to real-time images or video frames from the imaging device 315. . For example, the images can be registered (e.g., using Powell's optimization, simplex or gradient methods, normalized cross-correlation or gradient descent algorithms with mutual information as cost), then weighted normalization of the squared error Summed and normalized mutual information can be used to compare registered images obtained from two sources. A similarity between the 2D image from the scan and the 2D image received from the endoscope can indicate that the endoscope is positioned near the location of the image from the scan. Such image-based navigation can perform local registration at the bifurcation of the patient's airway and can be less susceptible to noise from patient respiratory motion than electromagnetic tracking systems. However, because the image analyzer 435 relies on endoscopic video, the analysis can be affected by artifacts in the image caused by patient coughing or mucosal obstruction.

The image analyzer 435 may implement object recognition techniques that, in some embodiments, allow the image analyzer 435 to detect objects present within the field of view of the image data, such as bifurcation openings, lesions, or particles. can. Using object recognition, the image analyzer can output information about which objects have been identified, as well as object data indicating the object's position, orientation, and/or size expressed as a probability. As an example, object recognition can be used to detect objects that can indicate branching points in a luminal network and then determine their location, size and/or orientation. In one embodiment, in a given image within a luminal network, each branch typically appears as a dark, roughly elliptical region, and these regions are treated as objects by region detection algorithms such as MSER (Maximally Stable Extremal Region). can be used and automatically detected by the processor. Image analyzer 435 can use light reflection intensity combined with other techniques to identify the airway. In addition, the image analyzer 435 tracks the detected object over a set of consecutive image frames to detect which of the set of possible branches in the luminal network has entered. can be done.

Robot position data repository 470 receives robot position data from surgical robotic system 110, e.g. A data storage device that stores data related to movement. Exemplary robot position data includes reaching a particular anatomical site within the luminal network and/or orientation within the luminal network (e.g., relative to one or both of the endoscopic instrument leader and sheath). command data to command the instrument tip to change (e.g., in specific pitch, roll, yaw, insertion and retraction), insertion data representing the insertion movement of a portion of the medical device (e.g., instrument tip or sheath), IDM data, e.g., mechanical movement of elongated members of the medical device, such as movement of one or more pull wires, tendons, or shafts of the endoscope that drive the actual movement of the endoscope within the luminal network; It contains mechanical data representing

Navigation path data repository 445 is a data storage device that stores data representing pre-planned navigation paths through the luminal network to target tissue sites. Navigation to a specific location within the luminal network of the patient's body is performed prior to surgery to generate a 3D model of the tubular network and the information necessary to determine the navigational path within the tubular network. It may be necessary to perform the steps of As described above, a 3D model of the airway topography and structure of a particular patient can be generated. A target can be selected, eg, a lesion to biopsy or a portion of organ tissue to be surgically repaired. In one embodiment, the location of interest may be selected by the user pointing to a computer display capable of showing the 3D model, such as by clicking with a mouse or touching a touch screen. In some embodiments, a navigational path can be programmatically identified by analysis of a modeled and identified lesion site to derive the shortest navigational path to the lesion. In some embodiments, the route may be identified by the physician, or the automatically identified route may be modified by the physician. The navigation path can identify a series of branches within the luminal network that travel to reach the identified target site.

State estimator 440 is a module that receives inputs and performs an analysis of the inputs to determine the state of the medical device. For example, state estimator 440 may incorporate data from respiratory rate and/or phase identifier 410, scope position estimator 420, position calculator 430, image analyzer 435, navigation path data repository 445, and robot position data repository 470. can be received as input. State estimator 440 can perform probabilistic analysis to determine states and corresponding probabilities of medical devices within the lumen network, given the inputs provided. The estimated state is (1) the x, y, z position of the instrument relative to the coordinate system of the model of the luminal network, and (2) the instrument is located in a specific region of the model, e.g. It can refer to one or more of (3) pitch, roll, yaw, insertion, and/or retraction of the instrument, and (4) distance to the target site, whether located in the area. A state estimator 440 can provide an estimated state of the instrument (or the distal tip of the instrument) as a function of time.

In some embodiments, state estimator 440 may implement a Bayesian framework to determine states and corresponding probabilities. Bayesian statistical analysis starts with a belief, called the prior, and updates that belief with observed data. A priori represents an estimate of what the Bayesian model parameters are and can be represented as a parameterized distribution. Observed data can be collected to provide evidence for the actual value of the parameter. The result of Bayesian analysis is called the posterior, which represents a probability distribution that describes an event with confidence. If more data is acquired, the ex post can be treated as ex ante and updated with the new data. This process uses conditional probabilities, eg, Bayes' theorem, which indicates how likely event A is if event B occurs.

With respect to the disclosed navigational fusion system 400, the state estimator 440 can use previously estimated state data as a priori, and the respiratory rate and/or phase identifier 410, scope position estimator 420, position calculator 430, Inputs from image analyzer 435, navigation path data repository 445, and/or robot position data repository 470 can be used as observation data. At the beginning of the procedure, a visual-based initialization technique can be used to estimate the initial depth and roll within the trachea, and this estimate can be used in advance. State estimator 440 may perform Bayesian statistical analysis of prior and observed data to generate posterior distributions representing probabilities and confidence values for each of multiple possible states.

As used herein, the "probability" of a "probability distribution" means the likelihood that an estimate of possible positions and/or orientations of a medical device is correct. For example, different probabilities can be calculated by one of the algorithm modules that indicate the relative likelihood that the medical device is at one of several different possible branches within the luminal network. In one embodiment, the type of probability distribution (e.g., discrete or continuous distribution) depends on the features of the estimated state (e.g., the type of estimated state, e.g., continuous location information and discrete branch selection). selected to match. As an example, the estimated state for identifying which segment the medical device is in because it is at the trifurcate bifurcation may be represented by a discrete probability distribution, determined by one of the algorithm modules. As such, it may include three discrete values of 20%, 30%, and 50% representing the likelihood of being in each of the three branches. As another example, the estimated conditions may include a medical device roll angle of 40 ± 5 degrees, and the segment depth of the instrument tip within the bifurcation may be 4 ± 1 mm, each with a continuous probability distribution It is represented by a kind of Gaussian distribution.

In contrast, "confidence value" as used herein reflects a measure of confidence in the state estimation provided by one of the modules of FIG. 4 based on one or more factors. . In electromagnetic-based modules, factors such as electromagnetic field distortion, electromagnetic alignment inaccuracy, patient shifting or movement, and patient respiration can affect the reliability of state estimation. In particular, the confidence value in the state estimates provided by the electromagnetic-based module may depend on the patient's particular respiration rate, the movement of the patient or the electromagnetic field generator, and the position within the anatomy in which the instrument tip is located. For the image analyzer 435, examples of factors that can affect the confidence value in the state estimate include lighting conditions at a point in the anatomy where the image is acquired, the optical sensor that acquires the image, the presence of fluid, tissue, or other obstructions to or in front of, the patient's respiration, the condition of the patient's own tubular network (e.g., the lungs) such as general fluids within the tubular network, and obstruction of the tubular network; Also included are specific motion techniques used, for example, in navigation or image acquisition.

For example, one factor is that certain algorithms may have different levels of accuracy for each depth of the patient's lung, such as locations relatively close to the airway opening, and certain algorithms may vary depending on the location of the medical device. and orientation estimation, but the confidence value may decrease as one progresses toward the lower end of the lung. In general, the confidence value is based on one or more systematic factors associated with the process leading to the outcome, and the probability is. It is a relative measure that arises when trying to get the correct result from multiple possibilities using a single algorithm based on the underlying data.

As an example, the formula for computing the outcome of an estimated state represented by a discrete probability distribution (e.g., branch/segment identification for a tri-branch with three values of participating estimated states) is as follows: can be done.

In the exemplary equations above, S _i (i=1, 2, 3) are the possible exemplary values of the estimated state in the case where three possible segments are identified or exist in the 3D model. , C _EM , C _image , and C _robot denote confidence values corresponding to electromagnetic-based, image-based, and robot-based algorithms, and P _i,EM , P _i,image , and P _{i, robot} represents the probability of segment i. Due to the stochastic nature of such fusion algorithms, breathing can be tracked over time and even predicted to overcome latency and outlier disturbances. can also

In some embodiments, confidence values for data from scope position estimator 420, registration calculator, and image analyzer 435 are adaptively determined based on respiratory rate and/or respiratory phase from phase identifier 410. can do. For example, robot position data and image data may be affected differently than electromagnetic sensor data by respiratory motion. In some embodiments, visual data obtained from the endoscopic imaging data repository 430 is used to detect certain types of respiratory motion that are not detectable via sensors external to the luminal network, e.g., by visual processing. Airway movement can be detected in head-to-tail (back and forth) movements that can be detected.

Safe area data repository 455 is a data storage device that stores data representing areas and/or conditions to which particular attention should be paid during instrument insertion. For example, as described above, the 3D model can include information regarding airway diameter. Branches of the luminal network with diameters less than or equal to the diameter of the endoscope, or within a predetermined threshold of the diameter of the endoscope (eg, 1-2 mm, about 4 mm, or any other threshold distance) are designated as safe regions. can be specified. In some embodiments, such designation may allow the diameter comparison to be programmatically performed by a processor. As another example, a particular phase of a patient's respiratory cycle is designated as the expiratory safe "zone" of the patient's breathing, or as the transition phase that begins with exhalation and ends mid-inspiration when the patient's airway is expected to constrict. can do. In some embodiments, the threshold can be configured based on factors including instrument dimensions, controlled motion control tolerances, user-configurable preferences, and the like. In some embodiments, the safe area data repository 455 may store instructions regarding the operation of the robotic system and/or limitations in various safe areas.

Safety mode controller 450 is a module that receives various inputs and determines whether to activate safety mode. For example, safe mode controller 450 may receive as inputs data from safe region data repository 455, respiratory rate and/or respiratory phase data from phase identifier 410, and estimated state output from state estimator 440. can. The safe mode controller 450 can compare the respiratory phase and estimated state to data from the safe area repository to determine whether safe mode should be activated.

Navigation controller 460 is a module that receives data from safe mode controller 450 and uses this data to guide further operation of surgical robotic system 110 . For example, when safe mode is activated, navigation controller 460 may receive data from safe mode controller 450 regarding specific display instructions and/or IDM operating instructions. When safe mode is not activated, navigation controller 460 may receive data from safe mode controller 450 regarding the estimated state and the next move identified in the navigation route data.

(Overview of example navigation techniques)
According to one or more aspects of the present disclosure, FIG. 5 illustrates an exemplary system for filtering out noise due to luminal network motion from an instrument position estimate, as described herein. 5 shows a flow chart of a simple process 500 . Process 500 may be implemented in navigation fusion system 400 of FIG. 4, control and sensor electronics 184 of FIG. 1, and/or console base 201 of FIG. 2, or components thereof.

At block 505 , the position calculator 430 may access a model of the patient's luminal network, eg, from the model data repository 425 . For example, the model can be a segmented map of the patient's airway generated from a CT scan in some embodiments. The model can be any two-dimensional or three-dimensional representation of the patient's actual luminal network (or part of a luminal network).

At block 510, endoscope position estimator 420 may receive data from instrument sensors, and respiratory rate/phase identifier 410 may, for example, from respiratory sensor data repository 405 and endoscope electromagnetic sensor data repository 410, respectively. , can receive data respiration sensors. As described above, the endoscope sensor data can be derived from the endoscope's electromagnetic sensors, which indicate the position and/or orientation of the distal end of the endoscope within the electromagnetic field generated around the luminal network. can be provided and the respiratory sensor data can be generated by a sensor positioned to detect motion of the luminal network.

At block 515, position calculator 430 may estimate the position of the device relative to the model. For example, the coordinate system of the model may be mapped to the coordinate system of the electromagnetic field at the beginning of the medical procedure during registration (see discussion above regarding registration calculator 465 and registration data repository 475). The position calculator 430 can use this mapping (via the alignment data) along with the coordinates of the sensor position within the electromagnetic field to generate an initial estimate of the position of the device sensor within the model. However, as noted above, due to movement of the patient's airway during breathing, the initial alignment of the model to the coordinate system of the electromagnetic field may not accurately reflect the actual dynamic position of the patient's airway within the electromagnetic field. There is Since the instrument is located in one of the dynamically moving airways, when the airway position in the electromagnetic field changes from that same airway's mapped position in the model, the position estimated in block 515 will be, for example, , may be inaccurate for respiratory motion artifacts/components of the estimated position of the instrument.

At block 520, the respiration rate and/or phase identifier 410 may extract the respiration rate from the data from the respiration sensor, such as by using a Fourier transform to extract the respiration rate. A Fourier transform may be applied to data from one or more sensors in embodiments having multiple respiratory sensors.

At block 525, the position calculator 430 and/or the endoscope position estimator 420 adjusts the instrument and/or model position based on the determined respiration rate to compensate for periodic motion of the luminal network. A filtering stage 535 can be implemented to adjust. Various embodiments of filtering stage 535 are described in detail with reference to FIGS. 6A-6C.

At block 530, state estimator 440 may output an indication of instrument position. Output may be provided to a navigation system, eg, surgical robotic system 110, a user interface, eg, display 202, or both. In some embodiments, the indicators can be output to state estimator 440 for use in identifying possible states of the device.

6A-6C show flowcharts of various exemplary processes that may be used for filtering stage 535 of FIG. The processes of FIGS. 6A-6C may be performed by navigation fusion system 400 of FIG. 4, control and sensor electronics 184 of FIG. 1, and/or console base 201 of FIG. 2, or components thereof.

Referring to FIG. 6A, one exemplary process 600A that can be used for filtering stage 535 is shown. At block 605, endoscope position estimator 420 may design a filter based on the determined respiration rate. As noted above, in some embodiments, a model of the patient's airway is generated during breath-holding. Thus, the filter can be a bandpass or bandstop filter designed to select the device electromagnetic sensor data during peak inspiration conditions corresponding to breath-hold conditions during which the model was generated.

At block 610, endoscope position estimator 420 may apply the filter designed at block 605 to the instrument electromagnetic sensor data to filter out a portion of the data. By doing so, process 600A can filter out portions of the electromagnetic sensor signal that are typically considered "noise" and lead to inaccurate alignment with the 3D model. Since the electromagnetic sensor positions are registered to the static 3D model, it is possible to improve the accuracy of registration by filtering portions of the signal that occur during respiratory states that differ from those from which the model was generated. can.

Referring to FIG. 6B, another exemplary process 600B that may be used for filtering stage 535 is shown. At block 615, the respiratory rate and/or phase identifier 410 may identify the magnitude of displacement for each respiratory sensor. Magnitude can be measured relative to the "baseline" position of each sensor. A reference line can be set when calibrating the model coordinates to the coordinates of the electromagnetic field by recording the position of each sensor during calibration. In embodiments having multiple electromagnetic sensors placed on the patient's chest, sensors mounted closer to the sternum exhibit lower magnitudes of displacement than sensors mounted closer to the lower border of the lungs. .

At block 620, endoscope position estimator 420 may identify the relative position of the instrument sensor to the respiratory sensor. For example, the x- and y-coordinates (representing length and width positions in the electromagnetic field) can be compared to determine the relative distance between each respiration sensor and the nearest respiration sensor and/or device sensor.

At block 625, the process endoscope position estimator 420 may interpolate the displacement of the instrument sensor based on the displacement of the respiratory sensor and the relative positions of the instrument sensor and the respiratory sensor.

At block 630 , endoscope position estimator 420 may adjust the estimated instrument position calculated by the interpolated displacements at block 515 . Therefore, the adjusted position can represent a more accurate position of the instrument within the model by compensating for the displacement of the airway with respect to the model coordinate system.

Referring to FIG. 6C, another exemplary process 600C that may be used for filtering stage 535 is shown. At block 630, the respiration rate and/or phase identifier 410 may identify the magnitude of displacement for each respiration sensor. This can be performed similarly to block 615 of process 600B described above.

At block 635, the position calculator 430 may access the mapping of the 3D model to respiratory sensor positions. For example, each respiratory sensor can be mapped to x and y coordinates in the model.

At block 640, the position calculator 430 may transform the model to new coordinates within the coordinate system of the electromagnetic field based on this mapping and the magnitude of the displacement. For example, for each (x,y) coordinate to which a sensor is mapped, process 600C may adjust the z value of the (x,y,z) model coordinate based on the displacement magnitude of the mapped sensor. can. For (x,y) coordinates between mapped sensors, the z value can be adjusted based on the interpolated magnitude based on magnitude and distance from neighboring sensors. Accordingly, the position of the model within the coordinate system of the electromagnetic field can be dynamically adjusted to reflect movement of the patient's airway.

At block 645, position calculator 430 may register the instrument position to the translated model. For example, the position calculator 430 can access the (x, y, z) coordinate data of the instrument sensor within the electromagnetic field and identify the corresponding position within the transformed model.

Some embodiments of process 500 may use one or more of processes 600A, 600B, 600C to calculate the instrument position relative to the model in filtering stage 535 .

According to one or more aspects of the present disclosure, FIG. 7 depicts a flowchart of an example process 700 for activating safe mode during luminal network navigation of the present disclosure. Process 700 may be implemented by navigation fusion system 400 of FIG. 4, control and sensor electronics 184 of FIG. 1, and/or console base 201 of FIG. 2, or components thereof.

At block 705 , state estimator 440 may access a 3D model of the patient's luminal network, eg, from model data repository 425 .

At block 710, the state estimator 440 may receive data from one or both of the instrument sensors and respiratory sensors, for example, from the respiratory sensor data repository 405 and the endoscope electromagnetic sensor data repository 415, or modules 410, 420. , and/or analysis of such data from 430 . As described above, the endoscope sensor data can provide the position and/or orientation of the distal end of the endoscope within the electromagnetic field generated around the luminal network, and the respiratory sensor data can provide the luminal It can be generated by a sensor positioned to detect network motion. After block 710, process 700 can be split into two sub-processes that can be run separately or together to determine whether to activate secure mode. These sub-processes, when executed together, can be executed in parallel or serially.

If state estimator 440 receives instrument sensor position data at block 710 , process 700 may proceed to block 715 . At block 715, state estimator 440 may estimate the position of the device relative to the model. This estimation may be performed by any of the processes 600A-600C described above in some embodiments to compensate for periodic motion.

At decision block 720, the safe mode controller 450 determines whether the position of the instrument is within the predetermined safe area of the 3D model. As noted above, a safety zone can be predefined based on a comparison of airway and device diameters. If the instrument position does not fall within the safe area, process 700 loops back to block 710 to receive new data from the sensors. In other embodiments, process 700 may proceed to block 725 .

If the position of the instrument is within the safe area, process 700 proceeds to block 735 where safe mode controller 450 activates safe mode for further navigation of the instrument (eg, the next or subsequent movement of the instrument).

Moving to block 725 , if the process 700 received respiratory sensor location data at block 710 , the process 700 may proceed to block 725 . At block 725, the respiratory rate and/or phase identifier 410 may identify the respiratory phase as either inspiration or expiration.

At decision block 730, safe mode controller 450 determines whether the respiratory phase corresponds to a predetermined safe condition. In some embodiments, all expiratory phases can be identified as safe conditions. In some embodiments, exhalation in certain branches of the airway may correspond to safety conditions. In some embodiments, the safe mode controller 450 can analyze historical respiratory rate data to predict whether the next movement of the instrument will fall within the respiratory phase that correlates with the safe state. If the respiratory phase (or predicted phase) does not correspond to predetermined safety conditions, process 700 loops back to block 710 to receive new data from the sensors. In other embodiments, process 700 may proceed to block 715 .

If the respiratory phase (or predicted phase) corresponds to a predetermined safety condition, process 700 proceeds to block 735, where safe mode controller 450 performs further navigation (e.g., the next movement of the instrument, or all subsequent movement of the instrument). Invoke Safe Mode for Moves).

If process 700 advances to block 735 because the instrument position has entered the safe area, then in some embodiments the airway diameter tends to decrease further toward the lung periphery and the navigation path is to narrow the airway. The safe mode may be activated for all subsequent insertions as there is a tendency to move from the center towards the outer lung periphery. If the process 700 proceeds to block 735 for a respiratory phase (eg, an expiratory phase) to cause the process to activate the safe mode, the safe mode is for the duration of the predicted expiratory cycle or for the next detected inspiratory cycle. may be operated up to

In safe mode, some embodiments of process 700 may implement navigation controller 460 to impose restrictions on instrument movement. For example, in safe mode, the navigation controller 460 can prevent the surgical robotic system 110 from activating the instrument driver. In such embodiments, the navigation controller 460 can disable user input to the robotic system to guide instrument insertion while the safety mode is activated, e.g., during patient exhalation. .

In some embodiments, the safety mode controller 450 may determine that the diameter of the airway located down the navigation path from the calculated instrument position is smaller than the diameter of the instrument. This allows the navigation controller 460 to inhibit further insertion in the safe mode, allowing the user to insert a smaller diameter steerable channel through the working channel of the endoscope for further navigation. can be prompted to

In the safe mode, some embodiments of process 700 may provide an output to the user indicating reminders when moving the instrument instead of imposing restrictions on movement of the instrument. Such outputs include graphical (eg, on a display), audio, or tactile (eg, tactile feedback via input device 204) alerts.

(Overview of example navigation user interface)
8A and 8B illustrate exemplary user interfaces 800A, 800B that may be presented to a user during safe mode lumen network navigation of the present disclosure. For example, in some embodiments, user interfaces 800A, 800B may be displayed on display 202 of FIG.

FIG. 8A shows an exemplary user interface 800A that may be presented to a user during the expiratory phase of patient breathing. Exemplary user interface 800 includes alerts 805 , virtual navigation section 810 , and respiration tracking section 820 .

A virtual navigation section 810 includes a visualization of the patient's airway 812 and a visualization of the navigational path 814 through the airway. As noted above, this can be based on a 3D model in some embodiments. In some embodiments, virtual navigation section 810 may alternatively or additionally display images received from an endoscopic camera.

The respiration tracking section 820 includes a patient respiration waveform 822 and a marker 824 that indicates the current point in the respiration cycle. In waveform 822, the portion of the waveform with a positive slope represents inspiration and the portion with a negative slope represents expiration. Some embodiments may further display a predictive waveform for future breaths, for example, based on respiration rate analysis of previous respiration rate and/or ventilator cycle data. As shown, the current point in the respiratory cycle corresponds to the expiratory phase.

Alert 805 alerts the system operator to pause navigation during exhalation. In some embodiments, warning 805 can be accompanied by an additional warning 816 superimposed on virtual navigation section 810 . In other embodiments, the display may change color, an alarm may sound, the input joystick may vibrate, or to alert the user that the device is operating in safe mode. Other visual, audio, or tactile indications may be presented. In some embodiments, user control of robotic system 110 may be disabled in this state to reduce trauma to the patient's airway.

Referring to FIG. 8B, another exemplary user interface 800B that may be presented to a user during navigation through the predetermined safe area described above is shown. User interface 800B includes warning 805, virtual navigation portion 810, and model display portion 830. FIG.

As noted above, the virtual navigation section 810 includes a visualization of the patient's airway 812 and a visualization of the navigational path 814 through the airway.

Model display section 830 includes a graphical representation of the 3D model with current position marker 832 and safe area marker 834 . As shown, current position 832 is within safe area 834 . Warning 805 thus alerts the operator of the system that he is navigating a warning area. A further warning may be presented to assist the user to pause navigation during exhalation in this region.

(alternative filtering technique)
As noted above, some embodiments (a) receive raw sensor data over a given time period and (b) calculate respiration rate on the raw sensor data to determine respiration rate over the given time period. Applying a determining function (e.g., a Fourier transform) and (c) applying a filter to the raw sensor data to remove components of the raw sensor data attributable to the determined respiration rate can be utilized. can. However, in these approaches, (a)-(c)
may cause unnecessary delays due to To reduce the delays due to (a)-(c), some embodiments may utilize prediction techniques to predict respiration rate for future time periods. According to one prediction approach, a non-linear Kalman filter (such as an extended Kalman filter (EKF), a non-Saint Kalman filter (UKF), or any other suitable approach that applies a Kalman filter to a non-linear function) is used to perform near real-time or substantially can predict breathing motion in real time. As used herein, "real time" refers to processing applied immediately after acquisition of sensor data, e.g., within a sufficiently short period of time so that the processed data can be used for instrument navigation. means the processing on the sensor data completed in . An EKF or multiple EKFs (one per patch, one per range) can identify respiratory amplitude, command and phase in real time. According to embodiments, the EKF or respiratory motion detected by the EKF can be removed from the raw electromagnetic sensor data generated by the electromagnetic sensor, or any other position sensor. The EKF can process historical raw sensor data to predict respiratory motion for the current period. The predicted respiratory motion is then used to filter out respiratory components in the raw sensor data. An EKF or multiple EKFs (one per patch, one per range) can identify respiratory amplitude, command and phase in real time. Other exemplary embodiments may use other prediction techniques such as alpha-beta filtering, Bayesian filtering, particle filtering, and the like.

(Compensation for command by robot)
In some embodiments, movement of the device may exhibit movement similar to breathing. To compensate for these movements, control (e.g., insert, retract, articulate) the instrument in embodiments to detect and compensate for respiration rate (or any other physiologically induced movement) Using the commanded motion data used for breathing can avoid detecting that motion as respiration rate. For example, if the instrument movement is at a given rate (which can be determined by the commanded data), the embodiments described above apply a filter to the sensor data to remove data due to that movement. can be done.

(alternative sensor type)
In addition to using electromagnetic sensors to determine the position of the instrument, as described above, other embodiments may use other suitable sensor types. Such position sensors may include shape sensing fibers, accelerometers, visual detection algorithms, gyroscopes, or any other suitable sensor capable of detecting motion characteristics.

(compensation for other physiological noise)
While many of the embodiments of the present disclosure detect and compensate for noise resulting from patient respiration rate, other embodiments detect noise caused by heart rate or any other detectable characteristic, other physiological characteristic of the patient. can be detected and compensated for. In such cases, when the heart rate can generate electromagnetic data noise, in these embodiments the techniques described above are used to detect the heart rate and remove the noise generated by the heart rate. be able to. Other noise artifacts can also be detected, which can occur when a patient exhibits periodic tremors or body movements.

(implementation system and terminology)
Embodiments of the present disclosure provide systems, methods, and apparatus for improved navigation of luminal networks.

As used herein, the terms "link", "link", "coupled" or other variations of the word "linked" refer to either indirect or direct linkage. Note that it is possible to show For example, if a first component is "coupled" to a second component, the first component is indirectly coupled to the second component through another component, or It can be directly connected to the second component.

The robotic motion activation functions described herein can be stored as one or more instructions on a processor-readable medium or computer-readable medium. The term "computer-readable medium" means any available medium that can be accessed by a computer or processor. By way of example, and not limitation, such media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium containing any desired instruction or data structure. It may comprise any other computer-accessible medium that may be used to store program code. Note that the computer-readable medium may be tangible and non-transitory. As used herein, the term "code" may refer to software, instructions, code, or data executable by a computing device or processor.

The methods of the disclosure comprise one or more steps or actions for achieving the described method. The method steps and/or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is required for proper operation of the described method, the order and/or use of a specific step and/or action may be used without departing from the scope of the claims. can be changed.

As used herein, the term "plurality" means two or more. For example, a plurality of components means two or more components. The term "determining" encompasses a variety of actions, thus "determining" includes calculating, computing, processing, deriving, examining, retrieving (e.g., searching a table, database, or another data structure), checking, etc. Also, "determining" includes receiving (eg, receiving information), accessing (eg, accessing data in memory), and the like. Also, "determining" includes resolving, choosing, selecting, establishing and the like.

The phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" is a description of both "based only on" and "based at least on."

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these implementations will be apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the scope of the invention. For example, those skilled in the art will be familiar with equivalent methods of securing, attaching, coupling, or engaging instrument components, equivalent mechanisms for producing a particular actuation motion, and equivalent mechanisms for delivering electrical energy. It will be understood that a number of corresponding alternative and equivalent arrangements, such as, may be employed. Accordingly, this invention is not intended to be limited to the implementations shown herein, but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein. .

Claims (11)

A system configured to navigate a patient's luminal network, comprising:
an electromagnetic field generator configured to generate an electromagnetic field;
a set of one or more electromagnetic sensors provided at the distal end of the steerable instrument;
a set of one or more respiratory sensors configured to be placed on the patient;
at least one computer readable memory in which executable instructions are stored;
one or more processors in communication with the at least one computer readable memory for executing the instructions to access to the system a pre-operative model representing at least the luminal network;
accessing a registration mapping between the coordinate system of the electromagnetic field and the coordinate system of the pre-operative model;
receiving electromagnetic data signals from the set of one or more electromagnetic sensors within the electromagnetic field;
calculating a respiratory rate of the patient based on respiratory data signals from the set of one or more respiratory sensors;
applying to the electromagnetic data signal and the respiratory data signal a prediction filter configured to predict respiratory motion due to respiration;
removing components of the electromagnetic data signal attributable to the predicted respiratory motion;
calculating the position of at least one of the set of one or more electromagnetic sensors within the electromagnetic field based on the electromagnetic data signal with the component removed;
The distal end of the steerable instrument relative to the pre-operative model by translating the position of the at least one of the set of one or more electromagnetic sensors within the electromagnetic field using the registration mapping. determining the position of the
determining the at least one position of the set of one or more electromagnetic sensors relative to the set of one or more respiratory sensors;
(i) the determined at least one position of the set of one or more electromagnetic sensors relative to the set of one or more respiratory sensors; and (ii) the position between inspiratory and expiratory phases. said set of one or more electromagnetic sensors between said inspiratory and said expiratory phases of said breathing of said patient based on a magnitude of displacement of at least one of said set of one or more respiratory sensors; calculating at least one position displacement of
Based on the calculated at least one positional displacement of the set of one or more electromagnetic sensors between the inspiratory phase and the expiratory phase, the distal position of the steerable instrument relative to the pre-operative model determining the position of the edge;
a processor configured to run
a system. each electromagnetic sensor of the set of one or more electromagnetic sensors configured to generate an electrical signal indicative of the distance and angle between each of the electromagnetic sensors and the electromagnetic field generator, the electrical signal comprising: 2. The system of claim 1, wherein the system can be used to determine one or both of the position and orientation of each electromagnetic sensor within the electromagnetic field. A system configured to navigate a patient's luminal network, comprising:
a generator configured to generate an electromagnetic field;
a set of one or more electromagnetic sensors provided at the distal end of the steerable instrument;
a set of one or more respiratory sensors;
at least one computer readable memory in which executable instructions are stored;
one or more processors in communication with the at least one computer readable memory for executing the instructions to access to the system a pre-operative model representing at least the luminal network;
accessing a registration mapping between the coordinate system of the electromagnetic field and the coordinate system of the pre-operative model;
calculating the position of at least one of the set of one or more electromagnetic sensors within the electromagnetic field based on data signals from the set of one or more electromagnetic sensors;
calculating the patient's respiration rate based on data signals from the set of one or more respiration sensors;
of the distal end of the steerable instrument relative to the pre-operative model based on the registration mapping, the respiration rate, and the position of at least one of the set of one or more electromagnetic sensors within the electromagnetic field; determining a position;
a processor configured to run
has
The one or more processors execute the instructions to instruct the system to at least displace at least one of the set of one or more respiration sensors between inspiratory and expiratory phases of respiration of the patient. configured to perform computing the magnitude of
The one or more processors execute the instructions to instruct the system to at least determine the position of at least one of the set of one or more electromagnetic sensors relative to the set of one or more respiratory sensors. and
(1) the determined at least one position of the set of one or more electromagnetic sensors relative to the set of one or more respiratory sensors; and (2) between the inspiratory phase and the expiratory phase. at least one of said set of one or more electromagnetic sensors between said inspiratory phase and said expiratory phase based on the magnitude of said at least one displacement of said set of one or more respiratory sensors; calculating a displacement of the position;
Based on the calculated displacement of the at least one position of the set of one or more electromagnetic sensors between the inspiratory phase and the expiratory phase, determining the position of the extremity; and
The set of one or more respiration sensors includes a first additional electromagnetic sensor positioned and used at a first location on the body surface and a first additional electromagnetic sensor positioned and used at a second location on the body surface. wherein a first magnitude of displacement of said first additional electromagnetic sensor between said inspiratory phase and said expiratory phase is determined by said second additional electromagnetic sensor; wherein said second position is spaced from said first position such that said second displacement magnitude of said electromagnetic sensor is greater than said first position. The one or more processors execute the instructions to instruct the system to detect at least the set of one or more electromagnetic sensors for the first additional electromagnetic sensor and the second additional electromagnetic sensor. determining a relative position;
interpolating between the first displacement magnitude and the second displacement magnitude based on the determined relative positions of the set of one or more electromagnetic sensors;
performing the calculation of the displacement of the position of the set of one or more electromagnetic sensors between the inspiratory phase and the expiratory phase based on the interpolated magnitude;
4. The system of claim 3, wherein the system is configured to perform The one or more processors estimate a motion vector for at least one position of the pre-operative model based on the at least one displacement magnitude calculated by the system by executing the instructions. and
translating the pre-operative model within the coordinate system of the electromagnetic field based on the estimated motion vector;
and determining the position of the distal end of the steerable instrument based on the translated pre-operative model. To translate the pre-operative model within the coordinate system of the electromagnetic field, the one or more processors execute the instructions to cause the system to: moving a first portion of the pre-operative model to first new coordinates;
moving a second portion of the pre-operative model to a second new coordinate based on the magnitude of the second displacement;
6. The system of claim 5, wherein the system is configured to perform 3. The system of claim 1, further comprising a robotic system having an instrument driver configured to effect movement of the steerable instrument. Execution of the instructions causes the one or more processors to instruct the system to determine at least the position of the distal end of the steerable instrument relative to the pre-operative model. identifying movement;
8. The system of claim 7, configured to: direct operation of the instrument driver to perform the next movement. The system further has a display, and the one or more processors execute the instructions to provide the system with at least a graphical representation of the position of the distal end of the steerable instrument relative to the pre-operative model. and
8. The system of claim 7, configured to: display the generated graphical representation on the display. 8. The system of claim 7, wherein the robotic system comprises an input device, the input device configured to control movement of the steerable instrument based on user manipulation of the input device. 3. The system of claim 1, wherein the pre-operative model comprises a three-dimensional computed tomography model of the luminal network of the patient.

Images