JP2022547132A - System, device and method for turbidity analysis - Google Patents

Abstract

An endoscopic system including an endoscopic imager configured to capture images of an in vivo target site and determining one or more image metrics for each image frame of a plurality of image frames captured over a period of time. and a processor configured to analyze changes in the image metric over time and determine a turbidity metric for the target site based on the analyzed changes in the image metric. [Selection drawing] Fig. 5

Description

[Priority claim]
This disclosure claims priority to U.S. Provisional Patent Application No. 62/904,882, filed September 24, 2019, the disclosure of which is incorporated herein by reference.

The present disclosure relates to endoscopic procedures, and more particularly to systems, devices and methods for performing turbidity analysis of an endoscopic imaging environment.

Endoscopic imagers may be used during various medical interventions. Haze or turbidity in the imaging environment limits the view of anatomy provided by the imager. Turbidity may be caused by blood, urine or other particles. In some endoscopic procedures (eg, ureteroscopic procedures), a turbid imaging environment can be managed by a fluid management system that circulates fluid within the imaging cavity.

The present disclosure relates to an endoscopic system that includes an endoscopic imager configured to capture image frames of an in-vivo target site, and a processor. A processor determines one or more image metrics for each image frame of a plurality of image frames captured over a period of time, analyzes changes in the image metrics over the period of time, and based on the analyzed image metric changes , configured to determine a turbidity metric of the target site.

In some embodiments, the image metric is an image entropy metric including a red entropy metric and a cyan entropy metric.

In some embodiments, the processor estimates the blood content in the current image frame and modifies the current image frame to reduce the visual effects of the blood content and enhance the rest of the current image frame. further configured to modify.

In some embodiments, the processor is further configured to identify and classify particles within the current image frame.

In an embodiment, the endoscopic system further includes a display configured to annotate the current image frame with the turbidity metric.

In an embodiment, the display is further configured to bracket the identified particles and annotate the current image frame with the particle classification.

In some embodiments, the identified particles are kidney stones and the classification relates to kidney stone size.

The present disclosure provides an endoscopic imager configured to capture an image frame of an in-vivo target site, a fluid delivery mechanism for supplying irrigation fluid to the target site to define the field of view of the endoscopic imager, determining a turbidity metric of at least one image from the imager; determining a fluid delivery adjustment for cleaning fluid based on the turbidity metric; and fluid delivery provided by a fluid delivery mechanism based on the determined fluid delivery adjustment. and a processor configured to control the fluid delivery mechanism to regulate the .

In an embodiment, the processor is further configured to determine the type of interventional activity based on feature detection identifying the interventional instrument.

In an embodiment, the processor is further configured to determine a phase of interventional activity and adjust fluid delivery provided by the fluid delivery mechanism based on the phase of interventional activity.

In an embodiment, the processor is further configured to identify and classify particles in the current image frame and adjust fluid delivery provided by the fluid delivery mechanism based on the particle classification.

In some embodiments, the particle is a kidney stone and the particle classification relates to kidney stone size.

In an embodiment, the processor is further configured to determine a blood metric for the at least one image and adjust fluid delivery provided by the fluid delivery mechanism based on the blood metric.

In an embodiment, the processor is further configured to determine an image entropy metric for the at least one image.

In some embodiments, the turbidity metric and blood metric are determined based in part on the image entropy metric.

The present disclosure also includes determining one or more image metrics for each image frame of a plurality of image frames of an in vivo target site captured over a period of time and analyzing changes in the image metrics over the period of time. and determining a turbidity metric of the target site based on changes in the analyzed image metric.

In some embodiments, the image metric is an image entropy metric including a red entropy metric and a cyan entropy metric.

In one embodiment, a method includes estimating blood content in a current image frame and reducing the visual effects of the blood content to enhance the remainder of the current image frame. and modifying to.

In some embodiments, the method further includes identifying and classifying particles within the current image frame.

In some embodiments, the method further includes annotating the current image frame with the turbidity metric.

1 illustrates a system for performing an endoscopic procedure, according to various exemplary embodiments of the present disclosure; FIG. FIG. 5 illustrates a method of managing cleaning fluid flow in a closed-loop feedback system, according to various exemplary embodiments of the present invention; FIG. 4 illustrates a method of enhancing visibility through a blood-clouded liquid medium according to a first exemplary embodiment; FIG. 5 illustrates a method of enhancing visibility through a blood-clouded liquid medium according to a second exemplary embodiment; Figure 5 is a flow chart representing an embodiment combining the methods of Figures 2-4;

The present disclosure may be further understood with reference to the following description and accompanying drawings, in which like elements are referenced with the same reference numerals. Exemplary embodiments describe algorithmic improvements to manage turbidity within an endoscopic imaging environment. A typical urological procedure includes an imaging device (e.g., a ureteroscope or other endoscopic imager), a fluid delivery mechanism (to clear the field of view and/or widen or distend the body cavity), and A therapeutic mechanism (eg, Boston Scientific's LithoVue™ device, and/or a laser or RF energy source, etc.) is utilized.

Improvements described herein include, for example, methods of analyzing endoscopic images to determine turbidity and providing this information to a fluid management system to manage clarity within the field of view. Other improvements include algorithmically subtracting blood features from endoscopic images and amplifying blood-obscured image features, particularly for urological procedures. Some common urological procedures include kidney stone management (eg, lithotripsy), BPH (ie, benign prostatic hyperplasia) treatment (eg, GreenLight™ laser surgery), prostatectomy, bladder tumor resection. surgery, uterine fibroid management, diagnostics, etc., but those skilled in the art will recognize that imaging-enhancing devices and techniques are used in a variety of procedures where turbidity is a concern (i.e., including non-urological procedures). You will understand that you can.

FIG. 1 illustrates a system 100 for performing endoscopic procedures, according to various exemplary embodiments of the present disclosure. System 100 includes an endoscope 102 having an imager 104 that acquires image frames of an in-vivo anatomy during an endoscopic procedure, and an imager 104 for removing blood and debris that may obscure the view of the imager 104 . and a fluid delivery mechanism 106 for supplying fluid (eg, saline) to the anatomy. The fluid delivery mechanism 106 also simultaneously provides suction to remove fluid from the anatomy. In this manner, the anatomy is continuously refreshed with substantially transparent fluid to produce clearer images.

System 100 may further include a therapeutic device 108 selected according to the nature of the endoscopic procedure. Treatment device 108 may operate through endoscope 102 or may reside external to endoscope 102 . For example, the treatment device 108 can be a laser or shock wave generator, eg, to break up kidney stones, or a resectoscope to remove prostate tissue. Treatment devices may not be used when the urological procedure is intended to be diagnostic, ie, to examine anatomy rather than treat symptoms. Although exemplary embodiments are described with respect to urological imaging, they are not so limited. Some embodiments are equally applicable to a wide range of procedures, such as endoscopic procedures in the digestive system, including endoscopic procedures that do not involve fluid delivery mechanisms.

System 100 includes a computer 110 that processes image frames provided by imager 104 and provides processed images to display 112 . Computer 110 and display 112 are provided in an integrated station, such as an endoscope console in this embodiment. The endoscope console may also implement other functions for performing urological procedures including, for example, actuators that control the flow rate, pressure, or method of delivery of fluid delivered through fluid delivery mechanism 106 . Exemplary embodiments describe an algorithmic process that alters and enhances a displayed image generally continuously or in any other desired manner.

In one embodiment, image metrics determined from captured image frames are used to estimate the degree to which a video sequence is obscured or occluded by a turbid imaging or imaging environment. Image metrics include image entropy metrics. Image entropy is generally defined as a measure of the amount of information or amount of information in an image and can be approximated by evaluating the frequency of intensity values in the image. High entropy may reflect, for example, a large amount of anatomical detail associated with clear images, or may be associated with non-clear images, such as swirl clouds of particles that obscure anatomy. Changes in entropy measurements over time can help distinguish between high-entropy clear images and high-entropy not-clear images, for example. For example, the time course of entropy measurements of a series of images distinguishes between these two types of images through entropy variability in non-clear images.

On the other hand, low entropy reflects loss of contrast and/or blurring of details. Low entropy can result from very simple scenes (eg, plain white walls). The amount of image entropy can be measured using image entropy metrics, including the total entropy in the image, or the ratio of red to cyan entropy in the image. In some embodiments, the ratio of red to cyan entropy can emphasize the contribution of blood (represented by red entropy) to other fluids (represented by cyan entropy) to emphasize the blood contribution. In some embodiments, the image entropy metric can further include entropy variation in specific temporal frequency bands within the sequence of images. As noted above, entropy variation can be measured over time to distinguish between clear and non-clear images. Specifically, abrupt changes in high-entropy measurements of the observed image reflect the chaotic entropy associated with swirling particles clouds, and thus imply that the observed image is not a clear image.

Optical flow analysis is used to characterize the changing scene between images. In general, optical flow analysis identifies neighborhoods in a given image that correspond to neighborhoods in the previous image, and identifies differences in these locations. One such algorithm for estimating such misalignment is the Farneback algorithm. Such analysis provides information about stationary and moving objects in the field of view, as well as systematic field motions, ie camera pans, rotations, advances and retreats.

Machine learning systems can also be employed to characterize and classify video segments according to occlusion level or scene content. For example, machine learning methods such as neural networks, convolutional neural networks, optimizers, linear regression and/or logistic regression classifiers are used to discover novel image analysis and/or combinations of entropy metrics as described above to characterize video segments. and classification can be performed. Machine learning methods can generate scalar estimates of turbidity, as well as blood and particle-field probabilities.

In another embodiment, other metrics such as spatial and temporal frequency decomposition can also be used to classify the video segments. Spatial frequency analysis can effectively measure image sharpness to directly reveal image clarity. An image containing a cloudy field of particles may exhibit relatively high spatial frequencies, which are similar to those of images containing high anatomical detail. Distinguishing between high spatial frequency images containing cloudy vision and images containing high anatomical detail can be done using temporal frequency analysis. As mentioned above, time-frequency analysis means tracking changes in metrics over time.

As described in detail below, a machine learning system can optimize the non-linear combiner and various assessments including particle/stone assessment, blood assessment and clarity assessment. Various metrics can be derived through linear combinations of pixel values, intensity histogram values, nonlinear functions (eg, logarithms) of these, and sequences of these values over time. Machine learning systems can use algorithms that find other combinations of the above values that do not directly correspond to spatial frequency or entropy, but effectively correspond to the viewer's impression of turbidity.

Machine learning systems operate on the feature spaces they present (ie, the system's values determine the classification). In one embodiment, a video machine learning system has video pixels directly input into the system. A typical machine learning model computes an image metric (eg, spatial frequency spectrum) from these values. The feature space can be augmented by providing system analysis that is useful but not suitable for computation in machine learning systems. For example, entropy, which is considered useful for this computation, cannot be easily computed in many common video processing neural network architectures. As a result, machine learning systems can perform better when presented with other analyzes in addition to raw pixels. Providing intensity histogram data with logarithms allows the neural network to incorporate entropy and entropy-like measurements, thereby converging faster and better results. The system can identify mathematical variations of metrics as superior. Therefore, careful enhancement of the feature space can dramatically improve performance.

Certain metrics that are directly derivable from image data do not by themselves do a good job of estimating turbidity in clinical settings where the image background is constantly changing. Exemplary embodiments describe methods for informing and adjusting neural networks with entropy-related metrics that estimate turbidity levels, for example, from image data.

The metrics and analysis described above feed into well-designed machine learning algorithms that perform multiple evaluations that can be output to the display to annotate the displayed image frames and/or inform further analysis. be able to.

In a first example, a particle or stone assessment is performed. Particle/calculus evaluation determines the properties of particles suspended in a fluid. Feature detection can be used to determine the properties of particles. For example, kidney stones can be identified as such and metrics such as stone size, as well as depth estimates (ie, distance from the imager) can be determined. Stone motion can also be estimated based in part on the optical flow analysis described above. Other particles can include, for example, proteins, tissue, protrusions, thrombi, and the like. Particle/stone assessment identifies a particle, segments it within an image frame, and sizes it.

This particle identification and classification can be used to annotate the current image frame. For example, particles can be bracketed to display relevant metrics to inform the operating physician and enable quick decision-making. Particle/concrete analysis can also be used to manage the delivery of cleaning fluid by fluid delivery mechanism 106 . For example, cloudy vision usually suggests the need for increased fluid flow, whereas identification of mobile stones may suggest reduced fluid flow to maintain the stone's position within the imager's field of view. can. Particle/stone assessment may also directly or indirectly affect the use of treatment device 108 depending on the nature of the treatment. For example, in lithotripsy, the classification of a stone as, for example, too large to be retrieved, too large to pass spontaneously, or small enough to pass spontaneously influences operation of the system during the remainder of the intervention. It can be automatically (or under medical supervision) driven or influenced in motion.

In a second example, an intelligibility assessment is performed. The clarity rating determines the total turbidity measure within the image. As noted above, a machine learning process using, for example, image entropy metrics as input can generate new turbidity measures or measurements. The turbidity measure can be, for example, a scalar estimate of turbidity or some other metric developed by a machine learning algorithm. The clarity rating can be used to annotate the currently displayed image, or it can be used to manage the supply of cleaning fluid.

In a third example, a blood evaluation is performed. Blood assessment estimates a total measurement of blood volume or blood content in the cavity, eg, in parts per million (PPM), although other metrics can be used. The blood assessment can be used to annotate the currently displayed image and can be used to manage the supply of cleaning fluid. Blood assessment can also inform, for example, thrombus analysis, laser settings for BPH treatment, and the like.

In addition to the particle evaluation, clarity evaluation and blood evaluation described above, a fourth evaluation can also be used to manage fluid circulation within the imaged cavity in vivo. A fourth assessment relates to the type of intervention performed during the endoscopic procedure. For example, the intervention can be kidney stone management, prostatectomy, fibroid management, and the like. A feature detector can be applied to the current image frame to determine the type of interventional activity in progress. For example, during laser prostatectomy, a laser device resides within the FOV of the imager and can be identified as the laser device by the feature detector.

In another example, the stone retrieval basket device can be identified by a feature detector or the like. Once identified, the phase of intervention is assessed. For example, the system can measure the degree to which a stone has been crushed or otherwise reduced in size, and/or can determine when a stone has been received in a basket for retrieval, and/or the like. Therefore, intervention assessment should be conducted to identify phases of intervention that may affect desirable flow characteristics (e.g., no stone in field, stone in field, laser on), at least particle size during laser lithotripsy procedures. / Can be used in combination with calculus evaluation and the like. The type of intervention may affect the optimal flow rate of fluid in the cleaning system. For example, laser prostatectomy may require higher flow rates than, for example, kidney stone procedures due to tissue heating. Heating of the tissue may cause complementary heating of the fluid within the cavity. Faster fluid inflow and outflow can serve to maintain a more stable temperature within the cavity and prevent damage to healthy tissue.

FIG. 2 illustrates a method 200 of managing the supply of cleaning fluid with a closed-loop feedback system, according to various exemplary embodiments of the invention. Method 200 may employ some or all of the metrics described above. Some of the metrics described can be derived from a single image frame, while others can be derived from a sequence of image frames. Accordingly, closed-loop adjustment of fluid delivery can be performed based on multiple image frames. However, given the fast frame rates of imagers commonly used in the art, fluid delivery adjustments are applied quickly enough to appear substantially continuous from the perspective of the physician during surgery.

At 205, imager 104 captures a series of image frames. As noted above, the total number of images required to perform a given computation can vary, and so can the number of images in the sequence.

At 210, some or all of the aforementioned metrics are determined from one or more image frames in the sequence. For example, image entropy metrics can be derived from each image and determined from these as the images are captured. In another example, pixel metrics are determined directly from these images.

At 215, machine learning combiner algorithms/functions are used to process various metrics and analytics to characterize and classify image or video segments according to occlusion level and scene content. As noted above, machine learning algorithms can combine metrics in various ways and self-adjust the weighting or use of metrics depending on the analysis discovered. It should be noted that all of the metrics/analyses described above, with the exception of treatment feature detection and intervention phase assessment, are input into the machine learning combiner algorithm, whereas treatment features/phase detection/assessment are determined directly from the image frames and particle assessment , does not notify blood evaluation or clarity evaluation.

At 220, the flow management algorithm processes the particle evaluation, the blood evaluation, the clarity evaluation and the intervention phase evaluation to determine if a fluid delivery adjustment of the wash fluid is required and, if so, the adjustment value. to decide.

At 225, the adjustment values are fed back to the processor to adjust the flow delivered by the fluid delivery mechanism.

Method steps 205-225 are performed substantially continuously to provide a closed loop feedback system for managing fluid flow during an endoscopic intervention.

Of the metrics described above, those particularly related to blood assessment and intelligibility assessment are used to algorithmically provide image enhancements to compensate for bloody fields of view in endoscopic interventions. be able to. Blood turbidity can be managed by algorithmically subtracting blood features from the image and amplifying blood-hidden features. The methods 300 and 400 described below can be applied singly or in combination.

FIG. 3 illustrates a method 300 of enhancing visibility through a blood-clouded liquid medium, according to a first exemplary embodiment. Method 300 includes several steps of method 200 (or steps similar to those of method 200) for managing the supply of cleaning fluid, and can be used alone or in combination.

At 305 , imager 104 captures a series of images according to step 205 of method 200 . At 310, each image is divided into regions of fixed size. Each region can be, for example, 20×20 pixels.

At 315, similar to steps 210-215 of method 200, the machine learning combiner algorithm described above is used to compute and process some of the metrics described above for each region. In this way, each region is evaluated for blood volume or blood content to generate a "blood map" that characterizes each region according to the presence of blood. Specifically, we determine the metrics to provide to the machine learning combiner algorithm for blood assessment, which we process to classify each region. There is no need to perform a particle/concrete assessment or clarity assessment, or to characterize any interventional device for method 300 . However, in another embodiment, a clarity assessment can be performed, for example, when a cloudy field is caused by particles other than blood.

At 320, the blood map values are multiplied at the combiner with the optical flow adjusted for the frame-to-frame local variation of each metric associated with blood detection. Thus, individual image sub-regions can be utilized to identify regions obscured by blood and selectively or proportionally apply image enhancements to mitigate the loss of image detail caused by blood. .

At 325, the values obtained from 320 are used to scale the convolution kernel or color transform computed to complement each metric and apply it in the region with sigmoidal reduction at the region boundaries. The entropy-related kernel uses either a low-pass filter kernel (for positive entropy correlation) or a high-pass filter kernel (for negative entropy correlation) applied to the associated color channel.

At 330, the image frames are reconstructed from the spatial frequency representations of each region of phase and varied amplitude to emphasize the background of interest while de-emphasizing the foreground containing blood. The visual effect of blood can be reduced to varying degrees throughout the image depending on the scaling of the convolution kernel. After this enhancement is applied to the spatial frequency magnitude components, it can be recombined with the phase components and transformed back to a conventional image.

FIG. 4 illustrates a method 400 of enhancing visibility through a blood-clouded liquid medium according to a second exemplary embodiment.

At 405, a deformable nonlinear transform is used to correlate the current image frame to the super-resolution image map. A super-resolution image map can generally be defined as an image of improved resolution produced from the fusion of low-resolution images. A current image can be fused with a super-resolution image map to increase the resolution of the current image.

At 410, based on the blood detection-related metrics described above, a "blood mask" is created, which reflects the estimated spatial distribution of blood turbidity in a given image frame.

At 415, the current image frame is fused with the modified super-resolution map to produce an enhanced frame that is a weighted average of the current frame and the super-resolution map. When the blood mask value is low, the pixels are pushed towards the pixels of the original image frame, and when the mask values are high, the pixels are pushed towards the pixels of the super-resolution map.

FIG. 5 is a flow chart 500 representing a combined embodiment including elements of methods 200, 300 and 400. As shown in FIG. Those skilled in the art will appreciate that the various metrics and analyzes described above can be used singly or in combination in various ways.

At 505, images are extracted from the endoscopic imaging feed. As mentioned above, some metrics/analyses, such as optical flow analysis, require many images for a decision. However, flowchart 500 can assume that enough previous images have been captured to perform each calculation.

At 510, the image is separated into red and cyan components for calculating various metrics. For example, at 515, a red entropy value and a cyan entropy value are calculated. Metrics such as the ratio of red to cyan entropy can be calculated, or red-cyan values can be used directly. At 520, a spatial frequency analysis is performed on the red spectral component and the cyan spectral component. At 525, the red/cyan entropy values and spatial frequency analysis are provided to temporal frequency analysis. Those skilled in the art will appreciate that time-frequency analysis requires a series of photographs in order to analyze the differences between the photographs. For example, entropy variation is measured over a period of time. At 530, a color balance is performed that compares the red and cyan components. At 535, a pixel change rate analysis is performed that compares pixel values between images. At 540, optical flow analysis is performed that correlates successive images to derive information such as object or tissue motion over time.

At 545, the machine learning system combines the metrics/analysis described above in various ways to provide particle information (particle/stone rating 550), blood volume or blood content information (blood rating 555), and turbidity information (clear generate a gender assessment 560). An intervention assessment is also performed that identifies intervention features in the image (treatment device feature detection 565) and assesses the phase of the intervention (treatment phase assessment 570). Particle, blood, clarity and intervention analysis are used to control the fluid flow 575 of the cleaning system. As noted above, fluid flow management 575 can include regulation of flow rates, pressures, or fluid delivery methods.

At 580 , the current image can be enhanced by removing or reducing the blood component of the image according to method 300 or 400 . At 585 , the current image can be annotated according to method 200 . The current image can be further annotated with blood or turbidity metrics derived in the blood/clarity assessment. At 590, the enhanced/annotated image is displayed. As additional images are captured, the steps/metrics/analyses described above are performed or determined.

In another embodiment, a computer-readable medium includes instructions that, when executed by a computer, cause the computer to perform the various image processing steps/analyses described above, such as determining the turbidity metric.

Those skilled in the art will appreciate that changes can be made to the above-described embodiments without departing from the inventive concept. Furthermore, it should be understood that structural features and methods associated with one embodiment may also be incorporated into other embodiments. It is therefore understood that the invention is not limited to the particular embodiments disclosed, but rather modifications are included within the scope of the invention as defined by the appended claims.

Claims (15)

An endoscope system,
an endoscopic imager configured to capture image frames of an in vivo target site;
a processor;
The processor
determining one or more image metrics for each image frame of a plurality of image frames captured over a period of time;
analyzing changes in said image metrics over said time period;
determining a turbidity metric for the target site based on changes in the analyzed image metric;
An endoscope system configured to: 2. The endoscope system of claim 1, wherein the image metrics are image entropy metrics including a red entropy metric and a cyan entropy metric. The processor
estimating the blood content in the current image frame;
3. The endoscopic device of claim 2, further configured to alter the current image frame to reduce the visual effects of the blood content and enhance the remainder of the current image frame. mirror system. The endoscopic system of any one of claims 1-3, wherein the processor is further configured to identify and classify particles within a current image frame. 5. The endoscopic system of claim 4, further comprising a display configured to annotate the current image frame with the turbidity metric. The display is
enclosing the identified particles in parentheses;
annotating the current image frame with a particle classification;
6. The endoscopic system of claim 5, further configured to: 7. The endoscopic system of claim 6, wherein the identified particles are kidney stones and the classification relates to size of the kidney stones. An endoscope system,
an endoscopic imager configured to capture image frames of an in vivo target site;
a fluid delivery mechanism that supplies irrigation fluid to the target site to define the field of view of the endoscopic imager;
a processor;
The processor
determining a turbidity metric of at least one image from the endoscopic imager;
determining a fluid delivery adjustment for the cleaning fluid based on the turbidity metric;
controlling the fluid delivery mechanism to adjust the fluid delivery provided by the fluid delivery mechanism based on the determined fluid delivery adjustment;
An endoscope system configured to: 9. The endoscopic system of Claim 8, wherein the processor is further configured to determine a type of interventional activity based on feature detection identifying an interventional instrument. The processor
determining the phase of said intervention activity;
10. The endoscopic system of claim 9, further configured to adjust fluid delivery provided by the fluid delivery mechanism based on the phase of the interventional activity. The processor
identify and classify particles in the current image frame;
The endoscopic system of any one of claims 8-10, further configured to adjust fluid delivery provided by the fluid delivery mechanism based on particle classification. 12. The endoscopic system of claim 11, wherein said particle is a kidney stone and said particle classification relates to size of said kidney stone. The processor
determining a blood metric for the at least one image;
The endoscopic system of any one of claims 8-11, further configured to adjust fluid delivery provided by the fluid delivery mechanism based on the blood metric. 14. The endoscopic system of Claim 13, wherein the processor is further configured to determine an image entropy metric for the at least one image. 15. The endoscopic system of claim 14, wherein the turbidity metric and the blood metric are determined based in part on the image entropy metric.

Images