CA3140678A1 - System and method for detection of floaters - Google Patents

Abstract

images of a patient's eye may be imaged and the images processed to detect and track floaters within the patient's eye. The floater detection and tracking may be used to identify characteristics of the floaters as well as possibly perform laser treatment of the floaters.

Description

SYSTEM AND METHOD FOR DETECTION OF FLOATERS
TECHNICAL FIELD
[0001] The current disclosure relates to a system and method for the detection of floaters and in particular to the detection of floaters for subsequent treatment using lasers.
BACKGROUND

[0002] Floaters in a patient's eye can impact the patient's vision and/or comfort. Floaters are microscopic fibers that can tend to clump together within the vitreous of the eye that cast shadows over the patients retina. Current treatment for floaters incudes removing the vitreous fluid that has the floaters and replacing it with a solution. New treatments may use lasers to breakup the debris within the vitreous. The lasers may be targeted at the debris by an ophthalmologist using a targeting laser. The manual targeting process may risk targeting non-floater elements within the patient's eye. Further, the manual targeting limits the minimum size of the floaters that can be targeted and treated using existing techniques.

[0003] An additional, alternative and or improved system and method for the treatment of floaters is desirable.
SUMMARY

[0004] In accordance with the present disclosure there is provided a system for use in treatment of floaters in an eye of a patient comprising: a first imaging system for capturing real-time images of the patient's eye; a laser treatment system for focusing and firing a treatment laser; and a controller for controlling the first imaging system and the laser treatment system, the controller configured to: detect a floater in an image captured by the first imaging system; track a position of the detected floater across images subsequently captured by the first imaging system; and focus the treatment laser of the laser treatment system at the tracked position of the detected floater for subsequent firing of treatment laser to treat the floater.

[0005] In a further embodiment of the system, the first imaging system comprises a scanning laser ophthalmoscopy imaging system.

[0006] In a further embodiment of the system, the treatment laser comprises a femtosecond laser.

Date recue / Date received 2021-11-30

[0007] In a further embodiment of the system, detecting the floater is done using a machine learning algorithm using large kernels for object detection.

[0008] In a further embodiment of the system, detecting the floater further comprises removing non-floater features of the eye from the image prior to using the machine learning algorithm.

[0009] In a further embodiment of the system, the non-floater features comprise veins in the eye.

[0010] In a further embodiment of the system, the system further comprises a second imaging system for capturing real-time images of the patient's eye.

[0011] In a further embodiment of the system, the second imaging system comprises an optical coherence tomography (OCT) imaging system.

[0012] In a further embodiment of the system, a location within the eye that the OCT imaging system images is adjusted based on the tracked location of the floater.

[0013] In a further embodiment of the system, the OCT imaging system is used to determine a depth of the floater.

[0014] In a further embodiment of the system, tracking the position of the detected floater comprises stabilizing images subsequently captured by the first imaging system.

[0015] In a further embodiment of the system, the controller determines one or more of: a number of floaters; a surface area of floaters; a volume of floaters; a location of floaters; an opacity of floaters; a refractive index of floaters; a speed of movement of floaters; a direction of movement of floaters; and a concentration of floaters.

[0016] In a further embodiment of the system, detecting the floater uses a convolutional neural network (CNN) that takes as input a sequence of a number (M) of image frames captured by the first imaging system and determines a sequence of M floater detection masks corresponding to floater locations in each image frame of the input sequence.

[0017] In a further embodiment of the system, detecting the floater comprises:
applying the CNN to a plurality of input sequences of M image frames, each of the plurality of input Date recue / Date received 202 1-1 1-30 sequences including a frame of interest to provide a plurality of floater mask sequences each including a floater detection mask for the frame of interest; and summing the floater detection masks for the frame of interest from each of the plurality of floater mask sequences.

[0018] In a further embodiment of the system, detecting the floater further comprises: applying a threshold value to the summation of the floater detection masks.

[0019] In accordance with the present disclosure there is further provided a system for use in treatment of floaters in an eye of a patient comprising: a first imaging system for capturing real-time images of the patient's eye; a laser treatment system for focusing and firing a treatment laser; and a controller for controlling the first imaging system and the laser treatment system, the controller configured to: send an image captured by the first imaging system to a remote server for detecting a floater in the image; buffer subsequently captured images from the first imaging system; receive a position of the floater detected in the image by the remote server; track a position of the detected floater across the buffered images; and focus the treatment laser of the laser treatment system at the tracked position of the detected floater for subsequent firing of treatment laser to treat the floater.

[0020] In a further embodiment of the system, the first imaging system comprises a scanning laser ophthalmoscopy imaging system.

[0021] In a further embodiment of the system, the treatment laser comprises a femtosecond laser.

[0022] In a further embodiment of the system, detecting the floater is done using a machine learning algorithm using large kernels for object detection.

[0023] In a further embodiment of the system, detecting the floater further comprises removing non-floater features of the eye from the image prior to using the machine learning algorithm.

[0024] In a further embodiment of the system, the non-floater features comprise veins in the eye.

[0025] In a further embodiment of the system, the system further comprises a second imaging system for capturing real-time images of the patient's eye.

Date recue / Date received 202 1-1 1-30

[0026] In a further embodiment of the system, the second imaging system comprises an optical coherence tomography (OCT) imaging system.

[0027] In a further embodiment of the system, a location within the eye that the OCT imaging system images is adjusted based on the tracked location of the floater.

[0028] In a further embodiment of the system, the OCT imaging system is used to determine a depth of the floater.

[0029] In a further embodiment of the system, tracking the position of the detected floater comprises stabilizing images subsequently captured by the first imaging system.

[0030] In a further embodiment of the system, the controller determines one or more of: a number of floaters; a surface area of floaters; a volume of floaters; a location of floaters; an opacity of floaters; a refractive index of floaters; a speed of movement of floaters; a direction of movement of floaters; and a concentration of floaters.

[0031] In a further embodiment of the system, detecting the floater uses a convolutional neural network (CNN) that takes as input a sequence of a number (M) of image frames captured by the first imaging system and determines a sequence of M floater detection masks corresponding to floater locations in each image frame of the input sequence.

[0032] In a further embodiment of the system, detecting the floater comprises:
applying the CNN to a plurality of input sequences of M image frames, each of the plurality of input sequences including a frame of interest to provide a plurality of floater mask sequences each including a floater detection mask for the frame of interest; and summing the floater detection masks for the frame of interest from each of the plurality of floater mask sequences.

[0033] In a further embodiment of the system, detecting the floater further comprises: applying a threshold value to the summation of the floater detection masks.

[0034] In accordance with the present disclosure there is further provided a method for use in treatment of a floater, the method comprising: detecting a floater in a captured image;
tracking a position of the detected floater across subsequently captured images; and focusing a treatment laser at the tracked position of the detected floater for subsequent firing of a treatment laser to treat the floater.

Date recue / Date received 2021-11-30

[0035] In a further embodiment of the method, detecting the floater is performed at a controller of an imaging system.

[0036] In a further embodiment of the method, detecting the floater is performed at remote server separate from a controller of an imaging system.

[0037] In a further embodiment of the method, the method further comprises buffering the subsequently captured images.

[0038] In a further embodiment of the method, the method further comprises capturing real-time images of the patient's eye using a second imaging system.

[0039] In a further embodiment of the method, the second imaging system comprises an optical coherence tomography (OCT) imaging system.

[0040] In a further embodiment of the method, the method further comprises adjusting a location within the eye that the OCT imaging system images based on the tracked location of the floater.

[0041] In a further embodiment of the method, the method further comprises using the OCT
images to determine a depth of the floater.

[0042] In a further embodiment of the method, the position of the detected floater comprises stabilizing images subsequently captured by the first imaging system.

[0043] In a further embodiment of the method, the controller determines one or more of: a number of floaters; a surface area of floaters; a volume of floaters; a location of floaters; an opacity of floaters; a refractive index of floaters; a speed of movement of floaters; a direction of movement of floaters; and a concentration of floaters.

[0044] In a further embodiment of the method, detecting the floater uses a convolutional neural network (CNN) that takes as input a sequence of a number (M) of image frames captured by the first imaging system and determines a sequence of M floater detection masks corresponding to floater locations in each image frame of the input sequence.

[0045] In a further embodiment of the method, detecting the floater comprises:
applying the CNN to a plurality of input sequences of M image frames, each of the plurality of input Date recue / Date received 202 1-1 1-30 sequences including a frame of interest to provide a plurality of floater mask sequences each including a floater detection mask for the frame of interest; and summing the floater detection masks for the frame of interest from each of the plurality of floater mask sequences.

[0046] In a further embodiment of the method, detecting the floater further comprises:
applying a threshold value to the summation of the floater detection masks.

[0047] In accordance with the present disclosure there is further provided a non-transitory computer readable medium having stored thereon instructions, which when executed by a processor of a computing device, configure the device to provide a method as described above.
BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Further features and advantages of the present disclosure will become apparent from the following detailed description, taken in combination with the appended drawings, in which:

[0049] FIG. 1 depicts a system for the treatment of floaters;

[0050] FIG. 2 depicts a method for targeting a laser for use in the treatment of floaters;

[0051] FIG. 3 depicts a floater detection process;

[0052] FIG. 4 depicts a further floater detection process;

[0053] FIG. 5 depicts a distributed system for the treatment of floaters;

[0054] FIG. 6 depicts a further method for targeting a laser for use in the treatment of floaters;

[0055] FIG. 7 depicts a distributed system for the detection of floaters;

[0056] FIG. 8A depicts an image of an eye with a floater; and

[0057] FIG. 8B depicts the image of the eye of FIG. 8A with the floater identified.
DETAILED DESCRIPTION

[0058] Floaters in a patient's eye may be tracked in real-time using a combination of imaging devices. A first imaging device, such as a scanning laser ophthalmoscopy (SLO) imaging device, may capture an image of the eye or portion of the eye within which a floater is visible.

Date recue / Date received 2021-11-30 The image from the first imaging device may provide an X-Y image that allows a position of the floater to be partially determined, although no depth information about the position of the floater may not be determined by first imaging device. The X-Y position of the floater may be used to control an imaging location of a second imaging device capable of capturing depth information, such as an optical coherence tomography (OCT) imaging device. The images from the first and second imaging devices allow the 3D location of floaters within the eye can be determined. The imaging devices may capture images in real-time which may allow the tracking of floaters to be done in real-time. The tracking information can be used for various purposes including for example measuring details of the floater(s) as well as possibly treating the floater(s) with a laser.

[0059] FIG. 1 depicts a system for the treatment of floaters. The system comprises an imaging and treatment device 102 that can be used for imaging a patient's eye, depicted as eye 104. The patient's eye may have one or more floaters 106. The imaging and treatment device 102 is depicted a single device in FIG. 1, however it will be appreciated that the components may be provided in multiple separate devices. International patent application No. PCT/CA2021/051451 filed October 15, 2021 entitled "OPHTHALMOLOGICAL
IMAGING
AND LASER DELIVERY DEVICE, SYSTEM AND METHODS," which is incorporated herein by reference in its entirety describes an imaging and treatment device that could be used as the imaging and treatment device 102. The imaging and treatment device 102 comprises an SLO imaging device 108 that can capture a X-Y image 110 of the patients eye and an OCT
imaging device 112 that captures a depth image 114 of the patient's eye. The OCT imaging device 112 may capture a depth 'slice' image at a particular horizontal location in the eye.
Both of the imaging devices 108, 112 may be able to capture multiple frames of images to provide real-time images, or videos of the patient's eye.

[0060] Imaging and treatment device 102 may also include a treatment laser 116 that can be targeted and fired at a particular location within the patient's eye, such as at a floater. The laser may be one of various known treatment lasers, including for example a femtosecond laser. Other lasers may be used including for example nanosecond lasers, picosecond lasers, microsecond lasers, milisecond lasers, or cw lasers. The SLO imaging device 108, the OCT
imaging device 110 and the treatment laser 116 can be calibrated so that all of the coordinate systems of devices are aligned and a location in one of the device's coordinate system can Date recue / Date received 202 1-1 1-30 be aligned with the same location in the coordinate system of the other devices. Although not depicted in detail in FIG. 1, it will be appreciated that each of the imaging devices 108, 112 as well as the treatment laser 116 will include an optical pathway and other components, such as light sources, sensors, etc. The optical pathways of the imaging devices and treatment later may include at least a portion of the optical pathways that are common to all of the devices. For example, the last portion of the optical pathway before the patient's eye may be common to all of the devices.

[0061] The imaging and treatment components 108, 112, 116 may be controlled by a controller 118 that is configured to provide various functionality including floater detection functionality 120, floater tracking functionality and floater treatment functionality 124. The floater detection functionality 120 uses image processing techniques to detect floaters within the SLO images. Floater detection can be difficult using current techniques.
Current object detection techniques perform well when detecting object with relative sharp edges. The object detection techniques typically use kernels for feature extraction/detection with a relatively small kernel size, such as 3x3 or 4x4. The floaters in the captured SLO images are shadows of the actual floaters and typically do not include sharp edges. In order to improve the floater detection, the object detection may be modified to use relatively large kernel sizes of for example, 8x8, 16x16, 32x32, and larger.

[0062] Additionally, floater detection may be further complicated by other features within the image. For example, features such as veins within the eye may make the floater detection difficult. It is possible to identify the non-floater features within the images and then remove or mask those features from within the images prior to attempting to detect the floaters. The non-floater features may be detected using various image processing techniques including machine learning image classification techniques and/or object detection techniques.

[0063] The controller 118 may further include floater tracking functionality 122. Regardless of the particular details on the image processing used to detect floaters, once detected the floaters can be tracked across subsequently captured images. The tracking can be done using conventional image processing or tracking techniques such as optical flow. These conventional techniques may be modified to use additional information from previous tracking. For example, the floater tracking may be used to predict floater locations in future frames, with the predicted locations used to speed detection/tracking of the floaters.

Date recue / Date received 202 1-1 1-30

[0064] The tracking functionality 122 may track the floater's X-Y position across the SLO
images. The OCT image, or images may be used to track the depth, or Z, position information of the floater. The tracked X-Y position of the floater may be used to control the location that is imaged by the OCT imaging device. The OCT imaging device may provide a depth window that is insufficient to image the entire depth of the patient's eye and as such multiple OCT images may need to be captured covering different depths in order to detect the depth of the floater. Once detected, the depth of the floater may be tracked and predicted.
The predicted floater depth location may be used to control, at least the initial, imaging depth of the OCT images to increase the likelihood that the floater is captured by the OCT images.
Further, multiple OCT images of adjacent depth slices may be captured to capture depth information for the entire extent of the floater.

[0065] As described above, the SLO and OCT imaging devices 108,112 may be used to detect and track one or more floaters in both the X-Y image plane of the SLO
imaging device as well as the X-Z, or depth, image plane of the OCT imaging device. It will be appreciated that reference to the X-Y and X-Z image planes are used only for explanation and other relative axes and coordinate systems could be used to provide information about the physical location of the floater. The tracked floater location may be used by treatment functionality 124 of the controller to target the treatment laser 116 at an appropriate location for treating the floater with the laser. Prior to firing the treatment laser, it is possible for the treatment functionality 124 to verify the safety of the possible treatment location. For example, if the floater is in front of and close to the retina, it may be determined that firing the treatment laser pose too big of a risk for hitting the retina and so may not fire the laser.
Additionally or alternatively, it is possible for the treatment functionality to adjust laser parameters based on a safety level of the treatment location. For example, if there are no other features close to the treatment location, it may be possible to increase a power level, or firing duration of the laser without causing risk to the patient's eye.

[0066] It will be appreciated that the detection, treatment and tracking of floaters may be performed repeatedly. That is, the detection process may be continually performed in order to detect floaters. Similarly the tracking process may be performed constantly to continually track floaters. Alternatively, the detection process may be performed periodically to detect all floaters and begin tracking the floaters. The periodic detection may be used to update the Date recue / Date received 202 1-1 1-30 tracking and/or detect new floaters. If the detection is performed periodically, the detection may be performed during the floater treatment which may break up the floater into additional smaller floaters.

[0067] FIG. 2 depicts a method for targeting a laser for use in the treatment of floaters. The method 200 begins with detecting a floater (202) in an image. The image captures a plane of the patient's eye, and may be for example a SLO image or a regular camera image. The floater detection from the SLO image identifies a location of the floater but does not include the depth information. Once the floater location is detected (202), its position can be tracked across multiple images (204). The floater tracking (204) can provide the location, including depth information, of the floaters. The floater tracking may use images captured using both the first imaging device (i.e. the SLO imaging device) and the second imaging device (i.e. the OCT imaging device). With the floater location tracked, the floater may be treated (206) by targeting a laser at the tracked location.

[0068] The floater detection (202) can be performed in various ways. For example, as depicted in FIG. 2, the detection may begin with detecting and removing, or masking, non-floater features in the SLO image (208). The non-floater features may be for example veins or other structures of the eyes. The non-floater features may be detected using imaging recognition functionality. The image with the non-floater features removed or masked, may be processed using machine learning (ML) object detection for detecting the floaters (210).
The floater detection may be based on existing ML object detection processes, which typically rely on relatively small kernels for feature detection/identification. The ML
object detection may be modified to use a relatively large kernel size, such as for example 16x16, 32x32 or larger. The larger kernel size improves the detecting of floaters which do not have well defined edges in the images.

[0069] Once the initial location of a floater is detected in the SLO image, its position may be tracked across multiple frames of the SLO images. In addition to tracking the position of the floater in the SLO images, the tracking may also be performed on the OCT
images in order to track the depth of the floater. The tracking may be performed in various ways.
As depicted, the tracking may begin with stabilizing SLO image frames (212). The stabilization may be done by registering stationary features within the eye across different frames. The floater may be tracked across different frames of the stabilized images(214) using known techniques Date recue / Date received 202 1-1 1-30 such as optical flow. Further, the tracking may make use of previous tracking information, for example to predict a likely location of the floater in a current frame in order to accelerate the tracking process. With the location of the floater tracked in the SLO image frames, the OCT
imaging location may be adjusted to capture depth strips at the floater location (216). With the OCT imaging location adjusted, the OCT imaging may capture one or more OCT
images which may are then processed to determine the depth of the floater (218). The OCT imaging device may only be able to capture the depth slice images over a particular window depth size, which may not cover the entire depth of the patient's eye. Accordingly, a single OCT
image may not capture the floater and as such the depth window may be adjusted until the floater is captured. The OCT imaging device may allow the depth of focus to be adjusted in order to change the window depth until the floater is detected in the OCT
image. The depth of the floater may be used as a starting depth for subsequent OCT imaging.

[0070] With the depth and position of the floater tracked, the floater can be treated (206).
Although the floater may be treated in various ways, as depicted, the treatment may be performed using a laser. The treatment includes targeting, including focusing, the treatment laser at the tracked position/depth of the floater (220). The safety of firing the laser at the target location may be verified (222) and assuming that the treatment location is safe, the laser may be fired at the floater (224) to break it up. Verifying the safety of the target may include determining the proximity to other features of the eye that could be damaged by the laser. If the features are within a path of the laser, or within a threshold distance of the path of the laser, the location may be deemed unsafe for treatment. As will be appreciated, floaters are moving within the eye and as such the tracking may continue unit the floater is determined to be in a 'safe' location for treatment. Verifying the safety of the treatment location may consider the treatment location relative to other features of the eye as well as possibly other factors such as the power and duration of the treatment laser.

[0071] FIG. 3 depicts a floater detection process. The process 300 uses a convolutional neural network (CNN) 302 to process a sequence of SLO images 304. The CNN 302 outputs a sequence of masks 306 providing detected locations of floaters. The floaters within the captured images are typically out of focus, and more so the closer they are to the front of the eye, with very blurry edges and typically just vague gradients providing low contrast.
Conventional image tracking and objected detection typically relies either on (i) landmarks, Date recue / Date received 2021-11-30 which are areas of high contrast to track over time or (ii) edges. In the detection of floaters, the areas of interest have very low contrast, even compared to other features in the SLO such as the optic disk, and also have no defined edges. Accordingly, the conventional image tracking processes tend to fail when detecting/tracking floaters.

[0072] The detection process 300 uses a convolutional neural network 302 in a configuration similar to U-Net. Rather than using as inputs the individual color channels of an image such as RGB, the input to the CNN 302 comprises an image with resolution Wi x Hi with M
channels, where M is the number of frames in the SLO sequence. The input can therefore be considered the sequence of frames of one channel each as captured by the SLO
imaging device. The output comprises of segmentation masks 306 showing the location of floaters.
The output masks also have M channels, each with as resolution of W2 X H2 which need not be the same as the input resolution Wi x Hi.

[0073] The CNN model may be trained on a collection of SLO image sequence/videos in which floaters have been labelled. The kernels of the convolutional layers may have larger sizes than typically found in CNNs such as 8x8,16x16, 32x32 to accommodate the detection of larger feature sizes specific to floaters.

[0074] FIG. 4 depicts a further floater detection process. To increase accuracy of the floater detection process 300 described above, as well as to have an adjustable "sensitivity" metric, the process 400 uses multiple image sequences 402 to identify floaters in a single frame. For example, to detect floaters on frame N=20, with a sequence length of M=6, the floater detection on frame 20 can be performed with frame sequences, include frame sequences 17 to 22, frame sequences 18 to 23, etc. Each of these sequences will produce floater mask predictions for frame 20 using CNNs 404, which may be the same as that described above in FIG. 3. By predicting across some or all of the frame sequences which include frame 20, a number of prediction mask sequences 406 is obtained with each sequence including a mask for the frame of interest, i.e. frame 20. The masks of the frame of interest can then be added together 408. If, for example, 5 sequences of images were used, resulting in 5 different prediction masks for frame 20, with each mask consists of values ranging from 0 to 1, the sum of the masks will range from 0 to 5. A sensitivity threshold 410 can then be specified between 0 and 5 to fine tune performance parameters such as false positive detection and output the smoothed floater mask for the frame of interest 412.

Date recue / Date received 2021-11-30

[0075] The machine learning based floater detection may be combined with classical tracking method. After detecting the floater using a ML model as described above, the predicted location can be passed to a classical image processing-based approach for object tracking such as optical flow. The predicted motion of the classical image processing-based object tracker can be used to limit the search area for subsequent ML-based detection of the floater.
Additionally or alternatively, after the classical image processing-based object tracker is activated on a detected floater, the ML-based detection method can periodically be activated to re-estimate the location of the floater and ensure continued tracking accuracy.

[0076] FIG. 5 depicts a distributed system for the treatment of floaters. The floater imaging and treatment device described above has been described as having a single controller that detects, tracks and treats the floaters. The image processing techniques may require a large amount of processing to perform quickly enough to make the real-time tracking and treating of floaters possible and practical. The system 500 may use a remote server, or other remote processing device to provide the required processing requirements of the image processing.
While the remote processing may be faster, or make possible improved image processing, the additional communication and possibly processing time, make it difficult to provide real-time detection and tracking of floaters. The system 500 described above makes use of an image buffer to make the detection/tracking possible. The system 500 is similar to the floater imaging and treatment device 102 described above, and as such similar elements are not described in further detail.

[0077] The system 500 may send the captured images to a remote server 528 via a communication network 530 for processing. The remote server 528 may provide image detection functionality 520, which may perform the floater detection and returns the results back to the imaging and treatment device 502. There may be a delay in receiving the detected floater location information from the remote server, which would make the detected location unsuitable for use in subsequent tracking in the most recent images.
In order to deal with the delay, the device uses an image buffer that can temporarily store the images captured subsequent to sending the images to the remote server for detection.
Upon receiving the detection results from the remote server 528, the buffered images are used to track the floater from detected location to the current image frames. The controller 518 may use tracking functionality 522 that may be substantially similar to the tracking 122 described Date recue / Date received 202 1-1 1-30 above; however the tracking may be performed on the buffered images. The tracking may be performed relatively quickly so that the tracking across the buffered images can be 'fast-forwarded', or performed faster than real-time, to the current frames and the tracking continued in real-time.

[0078] Although the above has described the detection as being done at a remote server, a similar buffering and fast-forward tracking may be used even if the detection is not performed at a remote server. That is, if the detection process performed takes a length of time that makes it difficult or impossible to use the detected location as a starting point for tracking in the current images, the same process of buffering images and then fast-forwarding the tracking of the detected location across the buffered images may be applied.

[0079] FIG. 6 depicts a further method for targeting a laser for use in the treatment of floaters.
The method 600 may be used to track floater locations from an initial detected location using a detection process that may take a length of time that makes using the detected location as an initial tracking location difficult. The method 600 passes an initial image, such as an SLO
image, to floater detection functionality (602). The floater detection functionality may be performed locally or remotely. While the initial floater location is determined, newly captured SLO image frames are buffered (604). The detected floater location is received (606) and then used as the initial location for tracking the floater location across the buffered images (608). The tracking of the floaters across the buffered images may be performed relatively quickly, allowing the tracking across the buffered images to catch up to the currently captured images.

[0080] FIG. 7 depicts a distributed system for the detection of floaters. The system 700 is similar to those described with reference to FIGs. 1 and 5. Similar features and functionality will not be described again in detail. The system 700 may include an imaging and treatment device 702 that includes the first (i.e. SLO) imaging device 108, and the second (i.e. OCT) imaging device 112; however, unlike the devices of FIGs. 1 and 5, the device 702 may omit a floater treatment laser 116, and similarly the controller 718 may omit the treatment functionality 124. The controller may include local detection functionality 720a and possibly local tracking functionality 722a that perform floater detection and tracking respectively. The local detection and local tracking may work in conjunction with, or be replaced by, remote detection functionality 720b, and remote tracking functionality 722b provided by a remote Date recue / Date received 202 1-1 1-30 server 728 in communication with the device 702 via a communication network 730. It will be appreciated that although the server is remote from the device, it does not need to be physically distant from the device 702. The controller may also include an image frame buffer 726 and a fast-forward tracking functionality 728 to track a floater from a detected location across buffered images in the buffer 726.

[0081] While the above has described tracking floaters and using the tracked location for targeting a treatment laser, it is possible to use the tracked floater information for other purposes. For example, the floater images and locations may be processed in order to identify and/or determine characteristics about the floater(s). This information may include for example a number of floaters, surface area of individual floaters, total surface area of all floaters, volume of individual floaters, total volume of all floaters, locations of floaters, opacity of floaters, refractive index of floaters, speed of movement of floaters, direction of movement of floaters, concentration of floaters, etc.. These characteristics may be used for various purposes including for example determining a severity of the patient's floater condition, determining a possible likelihood of successfully treating floaters with lasers, etc.

[0082] FIG. 8A depicts an image of an eye with a floater. The captured image is a single frame image captured from an SLO imaging device. The image 800 includes at least one floater along with additional features of the eye, such as the retina, veins, etc. FIG. 8B depicts the image of the eye of FIG. 8A with the floater identified. The floater is identified with a bounding box 802. The location may be used to control the imaging location of the OCT
imaging device. For example, depth slices may be captured by the OCT imaging device between the region identified by lines 804a, 804b.

[0083] It will be appreciated by one of ordinary skill in the art that the system and components shown in FIGs. 1 -8B may include components not shown in the drawings. For simplicity and clarity of the illustration, elements in the figures are not necessarily to scale, are only schematic and are non-limiting of the elements structures. It will be apparent to persons skilled in the art that a number of variations and modifications can be made without departing from the scope of the invention as defined in the claims.

[0084] Although certain components and steps have been described, it is contemplated that individually described components, as well as steps, may be combined together into fewer Date recue / Date received 202 1-1 1-30 components or steps or the steps may be performed sequentially, non-sequentially or concurrently. Further, although described above as occurring in a particular order, one of ordinary skill in the art having regard to the current teachings will appreciate that the particular order of certain steps relative to other steps may be changed. Similarly, individual components or steps may be provided by a plurality of components or steps. One of ordinary skill in the art having regard to the current teachings will appreciate that the components and processes described herein may be provided by various combinations of software, firmware and/or hardware, other than the specific implementations described herein as illustrative examples.

[0085] The techniques of various embodiments may be implemented using software, hardware and/or a combination of software and hardware. Various embodiments are directed to apparatus, e.g. a node which may be used in a communications system or data storage system. Various embodiments are also directed to non-transitory machine, e.g., computer, readable medium, e.g., ROM, RAM, CDs, hard discs, etc., which include machine readable instructions for controlling a machine, e.g., processor to implement one, more or all of the steps of the described method or methods.

[0086] Some embodiments are directed to a computer program product comprising a computer-readable medium comprising code for causing a computer, or multiple computers, to implement various functions, steps, acts and/or operations, e.g. one or more or all of the steps described above. Depending on the embodiment, the computer program product can, and sometimes does, include different code for each step to be performed.
Thus, the computer program product may, and sometimes does, include code for each individual step of a method, e.g., a method of operating a communications device, e.g., a wireless terminal or node. The code may be in the form of machine, e.g., computer, executable instructions stored on a computer-readable medium such as a RAM (Random Access Memory), ROM

(Read Only Memory) or other type of storage device. In addition to being directed to a computer program product, some embodiments are directed to a processor configured to implement one or more of the various functions, steps, acts and/or operations of one or more methods described above. Accordingly, some embodiments are directed to a processor, e.g., CPU, configured to implement some or all of the steps of the method(s) described herein.

Date recue / Date received 202 1-1 1-30 The processor may be for use in, e.g., a communications device or other device described in the present application.

[0087] Numerous additional variations on the methods and apparatus of the various embodiments described above will be apparent to those skilled in the art in view of the above description. Such variations are to be considered within the scope.

Date recue / Date received 2021-11-30

Claims (44)

WHAT IS CLAIMED IS:

1. A system for use in treatment of floaters in an eye of a patient comprising:
a first imaging system for capturing real-time images of the patient's eye;
a laser treatment system for focusing and firing a treatment laser; and a controller for controlling the first imaging system and the laser treatment system, the controller configured to:
detect a floater in an image captured by the first imaging system;
track a position of the detected floater across images subsequently captured by the first imaging system; and focus the treatment laser of the laser treatment system at the tracked position of the detected floater for subsequent firing of treatment laser to treat the floater.

2. The system of claim 1, wherein the first imaging system comprises a scanning laser ophthalmoscopy imaging system.

3. The system of claim 1, wherein the treatment laser comprises a femtosecond laser.

4. The system of claim 1, wherein detecting the floater is done using a machine learning algorithm using large kernels for object detection.

5. The system of claim 4, wherein detecting the floater further comprises removing non-floater features of the eye from the image prior to using the machine learning algorithm.

6. The system of claim 5, wherein the non-floater features comprise veins in the eye.

7. The system of claim 1, further comprising:
a second imaging system for capturing real-time images of the patient's eye.

8. The system of claim 7, wherein the second imaging system comprises an optical coherence tomography (OCT) imaging system.

9. The system of claim 8, wherein a location within the eye that the OCT
imaging system images is adjusted based on the tracked location of the floater.

10.The system of claim 9, wherein the OCT imaging system is used to determine a depth of the floater.

11.The system of claim 1, wherein tracking the position of the detected floater comprises stabilizing images subsequently captured by the first imaging system.

12.The system of claim 1, wherein the controller determines one or more of:

a number of floaters;
a surface area of floaters;
a volume of floaters;
a location of floaters;
an opacity of floaters;
a refractive index of floaters;
a speed of movement of floaters;
a direction of movement of floaters; and a concentration of floaters.

13.The system of claim 1, wherein detecting the floater uses a convolutional neural network (CNN) that takes as input a sequence of a number (M) of image frames captured by the first imaging system and determines a sequence of M floater detection masks corresponding to floater locations in each image frame of the input sequence.

14.The system of claim 13, wherein detecting the floater comprises:
applying the CNN to a plurality of input sequences of M image frames, each of the plurality of input sequences including a frame of interest to provide a plurality of floater mask sequences each including a floater detection mask for the frame of interest; and summing the floater detection masks for the frame of interest from each of the plurality of floater mask sequences.

15.The system of claim 14, wherein detecting the floater further comprises:
applying a threshold value to the summation of the floater detection masks.

16.A system for use in treatment of floaters in an eye of a patient comprising:
a first imaging system for capturing real-time images of the patient's eye;
a laser treatment system for focusing and firing a treatment laser; and a controller for controlling the first imaging system and the laser treatment system, the controller configured to:
send an image captured by the first imaging system to a remote server for detecting a floater in the image;
buffer subsequently captured images from the first imaging system;
receive a position of the floater detected in the image by the remote server;
track a position of the detected floater across the buffered images; and focus the treatment laser of the laser treatment system at the tracked position of the detected floater for subsequent firing of treatment laser to treat the floater.

17.The system of claim 16, wherein the first imaging system comprises a scanning laser ophthalmoscopy imaging system.

18.The system of claim 16, wherein the treatment laser comprises a femtosecond laser.

19.The system of claim 16, wherein detecting the floater is done using a machine learning algorithm using large kernels for object detection.

20.The system of claim 19, wherein detecting the floater further comprises removing non-floater features of the eye from the image prior to using the machine learning algorithm.

21.The system of claim 20, wherein the non-floater features comprise veins in the eye.

22.The system of claim 16, further comprising:
a second imaging system for capturing real-time images of the patient's eye.

23.The system of claim 22, wherein the second imaging system comprises an optical coherence tomography (OCT) imaging system.

24.The system of claim 23, wherein a location within the eye that the OCT
imaging system images is adjusted based on the tracked location of the floater.

25.The system of claim 24, wherein the OCT imaging system is used to determine a depth of the floater.

26.The system of claim 16, wherein tracking the position of the detected floater comprises stabilizing images subsequently captured by the first imaging system.

27.The system of claim 16, wherein the controller determines one or more of:
a number of floaters;
a surface area of floaters;
a volume of floaters;
a location of floaters;
an opacity of floaters;
a refractive index of floaters;
a speed of movement of floaters;
a direction of movement of floaters; and a concentration of floaters.

28.The system of claim 16, wherein detecting the floater uses a convolutional neural network (CNN) that takes as input a sequence of a number (M) of image frames captured by the first imaging system and determines a sequence of M floater detection masks corresponding to floater locations in each image frame of the input sequence.

29.The system of claim 28, wherein detecting the floater comprises:
applying the CNN to a plurality of input sequences of M image frames, each of the plurality of input sequences including a frame of interest to provide a plurality of floater mask sequences each including a floater detection mask for the frame of interest; and summing the floater detection masks for the frame of interest from each of the plurality of floater mask sequences.

30.The system of claim 29, wherein detecting the floater further comprises:
applying a threshold value to the summation of the floater detection masks.

31.A method for use in treatment of a floater, the method comprising:
detecting a floater in a captured image;
tracking a position of the detected floater across subsequently captured images; and focusing a treatment laser at the tracked position of the detected floater for subsequent firing of a treatment laser to treat the floater.

32.The method of claim 31, wherein detecting the floater is performed at a controller of an imaging system.

33.The method of claim 31, wherein detecting the floater is performed at remote server separate from a controller of an imaging system.

34.The method of claim 33, further comprising buffering the subsequently captured images.

35.The method of claim 34, further comprising capturing real-time images of the patient's eye using a second imaging system.

36.The method of claim 35, wherein the second imaging system comprises an optical coherence tomography (OCT) imaging system.

37.The method of claim 36, further comprising adjusting a location within the eye that the OCT imaging system images based on the tracked location of the floater.

38.The method of claim 37, further comprising using the OCT images to determine a depth of the floater.

39.The method of claim 31, wherein tracking the position of the detected floater comprises stabilizing images subsequently captured by the first imaging system.

40.The method of claim 31, wherein the controller determines one or more of:
a number of floaters;
a surface area of floaters;
a volume of floaters;
a location of floaters;
an opacity of floaters;
a refractive index of floaters;
a speed of movement of floaters;
a direction of movement of floaters; and a concentration of floaters.

41.The method of claim 31, wherein detecting the floater uses a convolutional neural network (CNN) that takes as input a sequence of a number (M) of image frames captured by the first imaging system and determines a sequence of M floater detection masks corresponding to floater locations in each image frame of the input sequence.

42.The method of claim 41, wherein detecting the floater comprises:
applying the CNN to a plurality of input sequences of M image frames, each of the plurality of input sequences including a frame of interest to provide a plurality of floater mask sequences each including a floater detection mask for the frame of interest; and summing the floater detection masks for the frame of interest from each of the plurality of floater mask sequences.

43.The method of claim 42, wherein detecting the floater further comprises:
applying a threshold value to the summation of the floater detection masks.

44.A non-transitory computer readable medium having stored thereon instructions, which when executed by a processor of a computing device, configure the device to provide a method according to any one of claims 31 to 44