CN115699108A - Method for detecting white blood cells and/or white blood cell subtypes from non-invasive capillary vessel video - Google Patents

Method for detecting white blood cells and/or white blood cell subtypes from non-invasive capillary vessel video Download PDF

Info

Publication number
CN115699108A
CN115699108A CN202180038518.2A CN202180038518A CN115699108A CN 115699108 A CN115699108 A CN 115699108A CN 202180038518 A CN202180038518 A CN 202180038518A CN 115699108 A CN115699108 A CN 115699108A
Authority
CN
China
Prior art keywords
images
blood cells
white blood
optical absorption
detected
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202180038518.2A
Other languages
Chinese (zh)
Inventor
C·C·冈萨雷斯
I·巴特沃斯
A·波夸德
A·S·费罗
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Luco Laboratories Inc
Original Assignee
Luco Laboratories Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Luco Laboratories Inc filed Critical Luco Laboratories Inc
Publication of CN115699108A publication Critical patent/CN115699108A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N33/00—Investigating or analysing materials by specific methods not covered by groups G01N1/00 - G01N31/00
    • G01N33/48—Biological material, e.g. blood, urine; Haemocytometers
    • G01N33/483—Physical analysis of biological material
    • G01N33/487—Physical analysis of biological material of liquid biological material
    • G01N33/49—Blood
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume, or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N15/14—Electro-optical investigation, e.g. flow cytometers
    • G01N15/1429—Electro-optical investigation, e.g. flow cytometers using an analyser being characterised by its signal processing
    • G01N15/1433—
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume, or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N15/14—Electro-optical investigation, e.g. flow cytometers
    • G01N15/1468—Electro-optical investigation, e.g. flow cytometers with spatial resolution of the texture or inner structure of the particle
    • G01N15/147—Electro-optical investigation, e.g. flow cytometers with spatial resolution of the texture or inner structure of the particle the analysis being performed on a sample stream
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
    • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
    • G01N21/31—Investigating relative effect of material at wavelengths characteristic of specific elements or molecules, e.g. atomic absorption spectrometry
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G06T7/0016—Biomedical image inspection using an image reference approach involving temporal comparison
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/38—Registration of image sequences
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/693—Acquisition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/695—Preprocessing, e.g. image segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/698—Matching; Classification
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
    • G01N2015/012—
    • G01N2015/016—
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume, or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N2015/1006—Investigating individual particles for cytology
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume, or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N15/14—Electro-optical investigation, e.g. flow cytometers
    • G01N15/1434—Electro-optical investigation, e.g. flow cytometers using an analyser being characterised by its optical arrangement
    • G01N2015/144—Imaging characterised by its optical setup
    • G01N2015/1443—Auxiliary imaging
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N15/00—Investigating characteristics of particles; Investigating permeability, pore-volume, or surface-area of porous materials
    • G01N15/10—Investigating individual particles
    • G01N15/14—Electro-optical investigation, e.g. flow cytometers
    • G01N2015/1486—Counting the particles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10056—Microscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30101—Blood vessel; Artery; Vein; Vascular
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30242—Counting objects in image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images

Abstract

A method includes acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from a non-invasive capillary video taken with an optical device, processing the first plurality of images to determine one or more optical absorption gaps located in the capillaries, and annotating the first plurality of images with an indication of any optical absorption gaps detected in the first plurality of images. The method further includes acquiring a second plurality of images of the same region of interest of the same capillary vessel using advanced optics capable of distinguishing cellular structures of white blood cells and subtypes of white blood cells, and spatiotemporal annotation of the second plurality of images with indications of any white blood cells detected and/or any detected subtype of white blood cells in the second plurality of images.

Description

Method for detecting white blood cells and/or white blood cell subtypes from non-invasive capillary vessel video
RELATED APPLICATIONS
The application claims the benefit and priority of U.S. serial No. 17/331, 893 filed in 35u.s.c. § 119, 120, 363, 365 and 37c.f.r. § 1.55 and 1.78 at 27/2021, the application and the present application also claims the benefit and priority of U.S. provisional application serial No. 63/031, 117 filed in 35u.s.c. § 119, 120, 363, 365 and 37c.f.r. § 1.55 and 1.78 at 28/2020, U.S. provisional application serial No. 17/331, 893 and U.S. provisional application No. 63/031, 117, which are incorporated herein by reference.
Technical Field
The invention relates to a method for detecting white blood cells and/or white blood cell subtypes from a non-invasive capillary video. The invention also relates to a method for determining the density of red blood cells from a non-invasive capillary video.
Background
There is a clinically urgent need for an improved, non-invasive, rapid, accurate and reliable method for measuring leukocytes and leukocyte subtypes in patients, including the identification of patients with low to dangerous leukocyte levels. Leukocytes (also known as leukocytes) include, among other things, subsets of leukocytes such as neutrophils, lymphocytes, monocytes, eosinophils, and basophils. According to the centers for preventive and control of diseases, 11 ten thousand of 65 cancer patients receiving chemotherapy each year in the united states were hospitalized with febrile neutropenia (clinically low levels of neutrophils) caused by chemotherapy. Of the 65 ten thousand cancer patients receiving chemotherapy each year in the united states, 11 thousand are hospitalized for chemotherapy-induced febrile neutropenia (a disease with clinically low neutrophil levels). See, for example, cost of Cancer-Related Neutropenia or river Hospitations, journal of Oncology Practice,13 (6) (2017) by Tai et al, incorporated herein by reference. This hospitalization typically averages 8.5 days, costs about $ 2.5 million, and has a mortality rate of about 7%, which makes neutropenia one of the most serious side effects of chemotherapy. See, e.g., truong et al, intervening Febrile Neutropenia Rates From Randomized Controlled Trials for correlation of Primary prophyphylaxis in The Real World: a Systematic Review and Meta-Analysis, annals of Oncology,27 (4) (2015) and Lyman et al, cost of localization in Patients With Cancer and textile Neutropenia and Impact of regulatory Conditions, am. Soc. Hematology (2015), both of which are incorporated herein by reference. There are also many other diseases and conditions associated with low to dangerous levels of white blood cells, including Acquired Immune Deficiency Syndrome (AIDS), autoimmune diseases, organ transplantation, patients treated with immunosuppressant drugs for various conditions, and the like.
One conventional technique that may be used to identify patients with dangerously low levels of leukocytes is Complete Blood Count (CBC). The CBC can monitor leukocyte differential and neutropenia. Invasive CBC requires the withdrawal of more than about 3 ml of blood in a clinical setting. Subsequent laboratory analyses typically take several hours to several days to produce results. The implementation of CBC is challenging and costly, and may require immunocompromised patients to visit a hospital, increasing their risk of infection. See, e.g., nosocomial Infection Update, ignition Infection Diseases,4 (3), (1998) by Weinstein, r.a, which is incorporated herein by reference.
Alternative conventional techniques based on finger lancing may have fundamental limitations due to the lack of repeatability between successive drops of blood, the white blood cell count of blood from the tip of the lancing site increases, and the blood obtained in this way may contain interstitial fluid. See, for example, bond et al Drop-to-Drop Variation in the Cellular Components of fingerprint Blood: the formulations for the Point-of-are Diagnostic Development, am.J. Clin.Pathol, 144 (6) (2015), the company of Yang et al, company of Blood Counts in Variation live and Artificial Blood and the theory of Measurement Variation, clin.Lab. Haematol.23 (3) (2001), A company of Daae et al, the company of Beiweisen, haematological Parameters in 'Capillary' and the company of Blood waste additives, 48 (7) (1988), all of which are incorporated herein by reference. Because of these limitations, finger stick methods can be difficult to represent whole body blood cell counts when performed outside of a clinical setting. See, for example, the compact of Venous and cervical Differential interference counters Using a Standard Hematology Analyzer and a Novel Microfluidic Impedence Cytometer, ploS one,7 (9) (2012) and the A Textbook of Practical Physiology, J.P.medical Ld. (2012) of Ghai, C.L., both of which are incorporated herein by reference. Thus, there is currently no device for self-monitoring of white blood cell counts (e.g., neutrophil counts) at home.
Conventional in vivo cell imaging systems and methods that may be portable, inexpensive, and practical for point of care typically do not have sufficient depth of focus, contrast, or visionDetecting leukocyte subtype in wild. A conventional angioscope can be used to collect video or images of the nail follicle capillaries of healthy subjects. See, for example, maidonado et al, nailfold Capillarioscopy in Diabetes mellitis, microvasculator Research,112.41-46 (2017), and Mengko et al, morphological Characterization of Nailfold Capillaries, intelligent Technology and Its Applications (ISIA), international sensor, IEEE (2016), both of which are incorporated herein by reference. These conventional systems and methods may allow imaging of capillary geometry and Optical Absorption Gaps (OAGs) in the microcirculation, but may have technical limitations including, among others, depth of focus, contrast with neutrophils, and stability, which may hinder subsequent analysis of the acquired video. See, e.g., bourquard et al Analysis of White Blood Cell Dynamics in Nailfold Capillaries,37 th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, (2015) and Bourquard et al Non-Invasive Detection of Server Neutrition in chemitherapy Patients By Optical Imaging of Nailfold Microcirculation, sci. Rep,8 (1): 5301 (2018), both of which are incorporated herein by reference. As defined herein, an "optical absorption gap" (OAG) refers to a region within a capillary vessel where red blood cells are depleted and which does not absorb light of the wavelengths at which absorption occurs in hemoglobin (e.g., about 400nm to about 600 nm). OAGs can be created by the presence of any leukocyte subtype or by the plasma space. See, for example, U.S. patent No. 9,984,277 and U.S. patent publication No. 2019/0139221, both of which are incorporated herein by reference. As disclosed in the '277 patent and the' 221 patent application, the video or image of one or more capillaries may be used to show that the frequency of OAGs flowing in the capillaries is related to the white blood cells flowing in the capillaries, and may be used to determine a white blood cell count. However, the '277 patent and' 221 patent applications are limited to using absorption signals, and are unable to identify leukocyte subsets within OAGs, and plasma gaps may also lead to false positives or inaccurate quantitative measurements of leukocyte counts. See, e.g., pablo-Trinidad et alThe Automated Detection of neutring Noninival Video Microcopy of Superficial Capillaries, american Journal of Hematology,94 (8) (2019), the Visualization of Blood Cell Contrast in Nailfold Capillaries With High-speed Reverse lines Mobile Microcopy of McKay et al, biological Optical Express,11 (4) (2020), and Optimizing White Blood Cell Contrast in Graded-Field-Capillary Using Capillary Tissue Phantoms, imaging, management, and Analysis of biomodules, cells, and Tissues, international Society for Optics and Photonics XVIII, vol.11243 (2020), all of which are incorporated herein by reference.
The conventional in vivo cell imaging systems and methods discussed above, the '277 patent and the' 221 patent application, also fail to determine the red blood cell density that can be used to non-invasively determine RBC counts.
Disclosure of Invention
In one aspect, a method of detecting leukocytes and/or leukocyte subsets from non-invasive capillary video is presented. The method includes acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from a non-invasive capillary video taken with an optical device, processing the first plurality of images to determine one or more optical absorption gaps located in the capillaries, and annotating the first plurality of images with an indication of any optical absorption gaps detected in the first plurality of images. The method further includes acquiring a second plurality of images of the same region of interest of the same capillary vessel with advanced optics capable of distinguishing cellular structures of white blood cells and subtypes of white blood cells, and spatiotemporal annotation of the second plurality of images with an indication of any white blood cells and/or any subtype of white blood cells detected in the second plurality of images. The method also includes inputting the first plurality of images and annotation information from the second plurality of images of the spatiotemporal annotation into a machine learning subsystem configured to determine the presence of white blood cells in one or more optical absorption gaps in the first plurality of images and/or the subset of any white blood cells present in one or more optical absorption gaps in the first plurality of images.
In one embodiment, the machine learning subsystem may be further configured to determine a subset of white blood cells of any optical absorption gaps detected in the first plurality of images. The machine learning subsystem may also be configured to determine a whole white blood cell difference measure and/or a partial white blood cell difference measure. The method may also include temporally aligning the first plurality of images with a second plurality of images of the spatio-temporal annotation. Temporally aligning may include creating the region of interest and the same region of interest by using the same objective lens on the optics and the advanced optics. Temporally aligning may include creating the region of interest and the same region of interest by focusing the optics and the advanced optics at the same location in the capillary vessel. The method may further include generating optical absorption gap reference data including the frame identifier and an indication of any optical absorption gaps detected in the first plurality of images. The method may further include generating spatio-temporally annotated lookup data that includes a frame identifier and an indication of any existing subset of white blood cells. Temporally aligning the first plurality of images with the second plurality of images of the spatiotemporal annotation may include temporally aligning frame identifiers of the first plurality of images with frame identifiers of the second plurality of images of the spatiotemporal annotation visually. The method may further include inputting the first plurality of images, the optical absorption gap reference data, and the spatiotemporal annotated lookup data into a machine learning subsystem. The machine learning subsystem may be configured to output result data for any detected white blood cells and/or any detected subset of white blood cells and compare the result table to the underlying truth data. The machine learning subsystem may be configured to output, for each optical absorption gap in the first plurality of images, result data of any detected white blood cells and/or any detected subset of white blood cells and compare the result data with underlying truth data. Spatio-temporally annotating the second plurality of images may further comprise indicating one or more of: the size, granularity, brightness, velocity, elongation and/or margin of the white blood cells and/or the density of the red blood cells located upstream or downstream of the location of the detected white blood cells. The subset of leukocytes can include granulocytes, neutrophils, lymphocytes, monocytes, eosinophils, or basophils. The optical device may include a high resolution camera. The advanced imaging apparatus may include, among other things, one or more of the following: a Spectrally Encoded Confocal Microscope (SECM) device, a Scanning Confocal Alignment Plane Excitation (SCAPE) microscope device, a scattering confocal alignment bevel imaging (SCOPI) device, or an oblique back-lit microscope (OBM) device. The predetermined region of the human subject may include one or more of the following, among others: fingers, nail capsules, toes, tongue, gums, lips, retina and/or earlobe. The optical device may be configured to output at least one optical absorption gap signal. The advanced optical apparatus may be configured to output an advanced optical signal. The spatiotemporal annotation of the second plurality of images may be performed by a human. Spatiotemporal annotation of the second plurality of images may be performed by the processing subsystem. The method may further include using the first plurality of images and annotation information from the first plurality of images and information from a machine learning subsystem that has learned and determined the presence of white blood cells in the one or more optical absorption gaps and/or the subset of white blood cells present in the one or more optical absorption gaps using annotation information from the second plurality of images acquired by the advanced optical device.
In another aspect, a method of detecting leukocytes and/or leukocyte subsets from non-invasive capillary video is presented. The method includes acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from a non-invasive capillary video taken with an optical device, processing the first plurality of images to determine one or more optical absorption gaps located in the capillaries, and annotating the first plurality of images with an indication of any optical absorption gaps detected in the first plurality of images. The method also includes determining the presence of white blood cells in the one or more optical absorption gaps and/or a subset of any white blood cells present in the one or more optical absorption gaps using the first plurality of images and annotation information from the first plurality of images and information from a machine learning subsystem that has learned and determined the presence of white blood cells in the one or more optical absorption gaps and/or a subset of white blood cells present in the one or more optical absorption gaps using annotation information from a second plurality of images acquired using advanced optics.
In yet another aspect, a method of determining red blood cell density from non-invasive capillary video is presented. The method includes acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from a non-invasive capillary video taken with an optical device, processing the first plurality of images to determine one or more hemoglobin optical absorption regions located in the capillaries, and annotating the first plurality of images with an indication of any regions of hemoglobin optical absorption detected in the first plurality of images. The method also includes acquiring a second plurality of images of the same region of interest of the same capillary vessel with advanced optics capable of resolving cellular structures of red blood cells, spatio-temporally annotating the second plurality of images with an indication of density of any red blood cells detected in the second plurality of images, and inputting the first plurality of images and annotation information from the spatio-temporally annotated second plurality of images into a machine learning subsystem configured to determine the density of any red blood cells present in one or more optical absorption gaps in the first plurality of images.
In one embodiment, the red blood cell count is determined based on the density of red blood cells.
Drawings
Other objects, features and advantages will become apparent to those skilled in the art from the following description of the preferred embodiments and the accompanying drawings, in which:
FIG. 1 is a flow chart illustrating the main steps of one embodiment of a method for detecting leukocytes and/or leukocyte subsets from non-invasive capillary video;
FIG. 2 shows in more detail an example of frame images of a first plurality of images and a second plurality of images that may be used by the method shown in FIG. 1, along with additional elements;
FIG. 3 is a schematic diagram illustrating one example of a nail sac of a finger of a human subject for a predetermined area of the human subject for use in the methods illustrated in one or more of FIGS. 1 and 2;
FIG. 4 is a diagram illustrating in more detail an example of an additional element used by one or more of the illustrated methods of FIGS. 1-3 and an example of a region of interest;
FIG. 5 illustrates an example of additional regions of a predetermined region of a human subject for use with the methods illustrated in one or more of FIGS. 1-4;
FIG. 6 shows in more detail an example of one or more capillaries of the nail sac shown in FIG. 3 that may be detected by an optical device for use in one or more of the methods shown in FIGS. 1-5;
FIG. 7 illustrates an example of an OAG reference table with optics detecting then annotated images and an example of a spatiotemporal annotation look-up table with advanced optics detecting then spatiotemporal annotated images for the methods illustrated in one or more of FIGS. 1-6;
FIG. 8 shows an example of a plot of an OAG signal output by an optical device and a plot of an advanced optical output signal output by an advanced optical device for use in the methods shown in one or more of FIGS. 1-7;
FIG. 9 illustrates in more detail an example of a first plurality of images having frame identifiers and a second plurality of images having frame identifiers temporally aligned for use in one or more of the methods illustrated in FIGS. 1-8;
FIG. 10 is a flow chart illustrating another embodiment of a method of detecting leukocytes and/or leukocyte subsets from non-invasive capillaries; and
FIG. 11 is a flow chart illustrating one embodiment of a method of determining the density of any red blood cells present in one or more absorption gaps.
Detailed Description
Aside from the preferred embodiment or embodiments disclosed below, this invention is capable of other embodiments and of being practiced or being carried out in various ways. Therefore, it is to be understood that the invention is not limited in its application to the details of construction and the arrangements of components set forth in the following description or illustrated in the drawings. If only one embodiment is described herein, the claims hereof are not to be limited to that embodiment. Furthermore, the claims hereof are not to be read restrictively unless there is clear and convincing evidence manifesting a certain exclusion, restriction, or disclaimer.
As shown in fig. 1, one embodiment of the method detects leukocytes and/or leukocyte subsets from non-invasive capillary video. In step 10 of fig. 1, the method includes acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from a non-invasive capillary video taken with an optical device. The first plurality of images 12 of fig. 2 are preferably derived from non-invasive capillary vessel video acquired or captured using an optical device 14, such as a high resolution camera, imager or imaging device disclosed in the '277 patent and/or the' 221 patent applications, or similar types of devices, cited above and incorporated by reference herein. In this example, the first plurality of images 12 includes images or frames 16, 18, 20, 22, and 24 of a region of interest (ROI) 26 in fig. 3 and 4, the region of interest (ROI) 26 including one or more capillaries, such as capillary 28 in fig. 2 and 4, of a predetermined region of a human subject. In this example, the first plurality of images 12 includes five images or frames 16, 18, 20, 22, and 24. In other examples, the first plurality of images 12 may include more or less than the five images or frames 16-24 shown in this example. In one example, the predetermined area of the human subject may be a nail sac of a finger, such as nail sac 40 of fig. 3 of finger 42 of human subject 44 in fig. 5. The nail 40 of fig. 3 is a preferred region of a human subject because the capillaries are in a more longitudinal position, one or more of which are more readily detected by the optical device 14, e.g., as shown by the capillaries 28 of fig. 6. In other examples, the predetermined region of human subject 44 of fig. 5 may include a region defined by a toe, tongue, gum, lip, retina, earlobe, or any similar body part of human subject 44.
Fig. 4 shows in more detail one example of an ROI 26 of a predetermined region of a human subject, in which an image of one or more capillaries may be acquired or captured using the optical device 14. As disclosed in the '221 patent application and/or the' 277 patent, in one design, light 52 emitted by the light source 50 is reflected by the mirror 54 such that the light 52 penetrates the nail sac 56 in the ROI 26, and the reflected light 60 is detected by the optics 14 coupled to the processing subsystem 70 in fig. 2 and 4 to create a non-invasive capillary video including the first plurality of images 12 in fig. 2 with images or frames 16-24.
In step 60 of fig. 1, the method of detecting leukocytes and/or leukocyte subsets from non-invasive capillary video further comprises processing the first plurality of images 12 to determine one or more OAGs located in capillaries. As described in the background section above, OAGs are a region of capillary blood vessel in which red blood cells are depleted and do not absorb light of the wavelengths absorbed in hemoglobin (e.g., about 400nm to about 600 nm) and indicate the presence of one or more white blood cells, such as disclosed in the '221 patent application and/or the' 277 patent. In one example, a processing subsystem 70 (similar to the processor disclosed in the '221 patent application and/or the' 277 patent or similar type of processing subsystem) coupled to the optical device 14 processes the first plurality of images 12 and detects one or more OAGs, such as OAG 64 of fig. 2, in the images or frames 16, 18, 20, 22, and 24 in the capillaries 28.
In step 62 of fig. 1, the method of detecting leukocytes and/or leukocyte subsets from a non-invasive capillary video further comprises annotating the first plurality of images 12 with an indication of any OAGs detected in the plurality of images. In one example, the first plurality of FIG. 2The individual images 12 are input to a processing subsystem 70, and the processing subsystem 70 outputs annotation information 72 relating to any detected OAGs. In one example, the annotation information 72 preferably includes OAG reference data, such as the OAG reference table 74 of fig. 7 or similar type of OAG reference data, which preferably includes a frame identifier 76 and an indication of any optical gaps detected in each image or frame 16, 18, 20, 22 and 24 of the first plurality of images 12, indicated at 78. In this example, the OAG reference table 74 includes a frame identifier t for each image or frame 0 、t 1 、t 2 、t 3 、t 4 ...t n And an OAG identifier for each image or frame, e.g., 1 indicates that an OAG has been detected. Annotating the first plurality of images 12 with indications of any OAGs detected in the first plurality of images 12 of fig. 2 and 7 to generate annotation information 72 may be performed by a trained human operator or by the processing subsystem 70 of fig. 2 and 4.
In step 80 of fig. 1, the method of detecting leukocytes and/or leukocyte subsets from a non-invasive capillary video further comprises acquiring a second plurality of images of the same region of interest of the same capillary vessel using advanced optics capable of distinguishing cellular structures of leukocytes and leukocyte subsets.
Fig. 2 shows one example of a second plurality of images 82 comprising images or frames 84, 86, 88, 90, 92 and 94 of the same ROI 26 of fig. 3 and 4, the ROI comprising one or more capillaries, such as capillaries 28 of a predetermined region of a human subject, such as the nail sac 40, acquired or captured by the advanced optics 96 of fig. 2 and 6 capable of resolving cell structures of leukocytes and leukocyte subtypes. In one example, the advanced Optics 96 may include a Spectrally-Encoded Confocal microscope (SECM) device, a scanning Confocal alignment Plane Excitation (SCAP) microscope device, a Scattering Confocal alignment bevel Imaging (SCOPI) device, or an Oblique Backlighting Microscope (OBM) device, e.g., as disclosed In the following documents, gold et al, noninational Imaging of Flowing Blood Cells Using Label-Free coded Flow Cytometry, biomedical Optics Express, vol.3No.6 (2012), ucbohard et al, swept Confonnal-Aligned Planar Excitation (SCAP) Microscopy for High-specific Imaging of Behaving images, nature Photonics, vol.9 (2015), mckay et al High-Speed Imaging of carving Particles With Through turbo Media With Forming alignment, outline plate Illumination, SPIE Bios, san Francisco, CA (2019), mckay et al Imaging Human Blood Cells In Vivo With outline Back-Illumination Capilaroscopy, biomedical Optics Express, vol.11 (5) (2020), and Ford, T.N. and Mertz, J. Video-ray Imaging of Microsimulation With outline Illumination (18, 18).
In one example, the cellular structure of the white blood cells and subsets of white blood cells resolved by the advanced optics 96 may include any subset of white blood cells detected, such as granulocytes, neutrophils, lymphocytes, monocytes, eosinophils, or basophils. Image 100 of fig. 2 shows an example of the cellular structure of the white blood cells and/or subsets of white blood cells in images 86 and 88 resolved by advanced optics 96, for example, in this example, by a Spectrally Encoded Confocal Microscope (SECM) or similar type of advanced optics.
In step 102 of fig. 1, the method of detecting leukocytes and/or a subtype of leukocytes from a non-invasive capillary vessel video further comprises spatiotemporally annotating the second plurality of images with indications of any leukocytes and/or any detected subtypes of leukocytes detected in the second plurality of images. Spatiotemporal annotation of the second plurality of images 82 of fig. 2 may include an indication of one or more of: the size, granularity, brightness, velocity, elongation and/or margin of the white blood cells and/or the density change of the red blood cells located upstream or downstream of the location where the white blood cells are detected. In one example, the second plurality of images 82 of FIG. 2 are spatially annotated with spatiotemporal annotation data, such as the spatiotemporal annotation lookup table 108 of FIG. 7, which includes a frame identifier 110 (e.g., t) for each image or frame 84, 86, 88, 90, 92 0 、t 1 、t 2 、t 3 、t 4 ...t n ) And an indication of the subset of leukocytes associated with each frame identifier, such as granulocytes, neutrophils, lymphocytes, monocytes, eosinophils, or basophils, illustratively indicated at 112. Spatio-temporally annotating the second plurality of images 82 with an indication of any white blood cells detected and/or any detected subset of white blood cells in the second plurality of images 82 may be performed by a trained human operator or by the processing subsystem 70 of fig. 2 and 4 and preferably generates the annotation information 120 of fig. 2 and the spatio-temporally annotated second plurality of images 122 in relation to the second plurality of images 82.
In step 128 of fig. 1, the method of detecting leukocytes and/or a subtype of leukocytes further comprises inputting the first plurality of images 12, the annotation information 72 from the first plurality of images 12, and the annotation information 120 from the spatio-temporally annotated second plurality of images 122 of fig. 2 into a machine learning subsystem 124, the machine learning subsystem 124 configured to determine the presence of leukocytes and/or the subtype of any leukocytes present in the one or more optical absorption gaps in the first plurality of images 12. In one example, the machine learning subsystem 124 may be a neural network, a support vector machine, a machine learning subsystem utilizing a random forest learning method, an adaptive Boost meta-algorithm, a naive bayes classifier, or deep learning known to those skilled in the art. The machine learning subsystem 122 may preferably be configured to determine the presence of white blood cells in the OAG and determine a whole white blood cell difference measure and/or a partial white blood cell difference measure.
In one example, the first plurality of images 12 of FIG. 2 are preferably temporally aligned with the second plurality of images 122 of the spatiotemporal annotation. In this example, temporally aligning includes using the same objective lens 170 on the optics 14 and the superior optics 96 of fig. 4 to create the same region of interest, such as the ROI 26 of fig. 2 and 4. In other examples, temporally aligning the first plurality of images 12 with the second plurality of images 122 of the spatiotemporal annotation includes creating the same ROI 26 for the optics 14 and the advanced optics 96 of fig. 4, for example, by focusing the optics 28 and the advanced optics 90 at the same location in the capillary vessel, for example, on the ROI 26 and the capillary vessel 28, as shown. In other examples, temporally aligning the first plurality of images 12 with the second plurality of spatio-temporally annotated images 122 may use image alignment processing methods, such as registration or similar image alignment processing methods known to those skilled in the art. See, e.g., medical Image Registration by Oliveira, F.P. and Travares, J.M.R. et al: a Review, computer Methods in biometry and Biomedical Engineering,17 (2) (2014), incorporated herein by reference.
In one embodiment, the method of detecting leukocytes and/or leukocyte subsets from non-invasive capillary video preferably comprises aligning the first plurality of images 12 of fig. 2 with the spatiotemporally annotated second plurality of images 122. In one example, temporally aligning the first plurality of images 12 with the second plurality of spatio-temporal annotated images 122 includes temporally aligning each frame identifier 76 of FIG. 7 (e.g., t in the optical absorption gap reference table 74 0 、t 1 、t 2 、t 4 ...t n ) With each frame identifier 110 (e.g., t in the spatiotemporal annotation look-up table 108) 0 、t 1 、t 2 、t 3 、t 4 ...t n ) And (6) carrying out alignment.
Plot 129 of fig. 8 shows an example of an OAG signal 132 output by the optical device 14 of fig. 2 and 4 and input to the processing subsystem 70. In this example, the OAG signal 132 includes peaks 140, 142, and 144, each of which indicates the presence of an OAG indicative of white blood cells in a capillary vessel, such as the OAG 64 of fig. 2 and 9 in the capillary vessel 28. Plot 129 of fig. 8 also shows an example of the advanced optical output signals 134, 136 and 138 output by the advanced optical device 96 (in this example, the SECM device) that are input to the processing subsystem 70 of fig. 2 and 4. Each of the high-level optical signals 134, 136 and 138 preferably includes a peak indicative of a subset of white blood cells corresponding to OAGs present or detected in the capillaries. For example, peak 146 of the high-level optical signal 134 indicates a leukocyte subset of monocytes, peak 148 indicates a leukocyte subset of lymphocytes, and peak 150 indicates a leukocyte subset of granulocytes, neutrophils. The peaks 146, 148, and 150 of the advanced optical signals 134, 136, and 138, respectively, are for illustrative purposes only, as the advanced optical signals 134, 136, and 138 may have peaks that represent other types of leukocyte subsets. Curve 129 may also include additional high-level optical signals whose peaks indicate additional leukocyte subsets, such as eosinophils, basophils, or other leukocyte structures. In this example, as shown, the processing subsystem 70 time aligns the peak 146 of the advanced optical signal 134 with the peak 140 of the OAG signal 132, which indicates the presence of a monocyte in the OAG 64 of fig. 2 and 9 in the capillary 28. Similarly, as shown, the processing subsystem 70 temporally aligns the peak 148 of the advanced optical signal 136 with the peak 142 of the OAG signal 132, which in this example indicates that lymphocytes are present in the OAG 64 in the capillary 28. As shown, the processing subsystem 70 also temporally aligns the peak 150 of the advanced optical signal 138 with the peak 144 of the OAG signal 132, indicating the presence of neutrophils in the OAG 64 of the capillary 28. In a similar manner, the processing subsystem 70 may temporally align the peaks of one or more additional high-level optical signals, each having a peak indicative of an additional leukocyte subset (e.g., granulocytes, eosinophils, basophils, or other leukocyte structures), with additional peaks on the OAG signal 132.
FIG. 9 shows a frame having a respective frame identifier t 0 、t 0 +68ms、t 0 +136ms、t 0 +204ms and t 0 One example of a first plurality of images 12 of images for frames 16, 18, 20, 22 and 24 at +272ms and having a frame identifier t, respectively 0 +68ms、t 0 +136ms、t 0 +204ms and t 0 The images or frames 84, 86, 88, 90 and 92 at +272ms, the second plurality of images 82. In this example, the advanced optics 96 of fig. 2 and 4 acquire the second plurality of images 82 of fig. 9 by linear scanning (indicated at 180) of the capillary vessel 28 using a SECM device. As mentioned above, other advanced optical devices may be used.
In one example, the first plurality of images 12 of fig. 2, annotation information 72 from the first plurality of images (e.g., the OAG reference data 74 of fig. 7, such as a table or similar type of data sum), and annotation information such as from the second plurality of images 120 of fig. 2 (e.g., the spatiotemporal annotated lookup data 108 of fig. 7, such as a table or similar type of data) are input to the machine learning subsystem 124 of fig. 2, which outputs result data 170 (e.g., a table of similar type of result data) that indicates any white blood cells detected and/or subtypes of any white blood cells detected. The machine learning subsystem 124 then preferably compares the resultant data 170 with underlying truth data 172 (e.g., tables or similar types of data) to determine and improve the accuracy of the detected leukocytes and/or the determined leukocyte subsets. As known to those skilled in the art, "ground truth" is a term relative to true knowledge about an ideal intended result.
In one embodiment, the machine learning subsystem 122 may output result data 174 (e.g., a table of similar types of data) including any white blood cells detected and/or any subset of white blood cells detected from each OAG in the first plurality of images 12, and compare the result data 174 to the underlying truth data 172 data to determine and improve the accuracy of the detected white blood cells and/or determined subset of white blood cells.
Once the machine learning subsystem 124 of fig. 2 has efficiently and effectively learned and determined the presence of white blood cells in one or more optical absorption gaps and/or a subset of white blood cells present in one or more optical absorption gaps using annotation information from the second plurality of images acquired using advanced optics, in step 190 of fig. 10, a method of detecting white blood cells and/or a subset of white blood cells from non-invasive capillaries of another embodiment using similar techniques as discussed above with reference to one or more of fig. 1-9 can include acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from a non-invasive capillary video taken using optics. In step 192, the method may further include processing the first plurality of images to determine one or more optical gaps located in the capillary vessel. The method may further include annotating the first plurality of images with an indication of any optical gaps detected in the first plurality of images in step 92, and determining in step 194 the presence of white blood cells in the one or more optical absorption gaps and/or a subset of any white blood cells present in the one or more optical absorption gaps using the first plurality of images and annotation information from the first plurality of images and information from a machine learning subsystem that has learned and determined the presence of white blood cells in the one or more optical absorption gaps and/or a subset of white blood cells present in the one or more optical absorption gaps using annotation information from the second plurality of images acquired using advanced optics.
The result is a method for detecting leukocytes and/or leukocyte subtypes from non-invasive capillary video that accurately, efficiently, and quantitatively determines leukocyte differential measurements and/or fractional leukocyte differential measurements to assist medical personnel in treating various diseases and conditions associated with low to dangerous levels of leukocytes, such as neutropenia, AID, autoimmune disease, organ transplantation, patients treated with immunosuppressant drugs for various conditions, and the like. Once the machine learning subsystem efficiently and effectively learns and determines the presence of white blood cells in one or more optical absorption gaps and/or the subset of white blood cells present in one or more optical absorption gaps using annotation information from the second plurality of images acquired using advanced optics, the claimed method may determine the presence of white blood cells and the subset of white blood cells in an OAG using a simple, portable, and cost-effective imaging device, such as a angioscope, and without also using advanced and expensive optical imaging systems, such as SECM, SCAP, SCOPI, OPBM, and the like.
Using techniques similar to those discussed above with reference to one or more of fig. 1-9, in step 200 of fig. 10, a method of determining red blood cell density from non-invasive capillary video of one embodiment of the present invention includes acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from non-invasive capillary video taken with an optical device. In step 202, the first plurality of images is processed to determine one or more hemoglobin optical absorption regions located in the capillaries. In step 204, the first plurality of images is annotated with an indication of any regions of hemoglobin optical absorption detected in the first plurality of images. In step 206, a second plurality of images of the same region of interest of the same capillary vessel is acquired using advanced optics capable of resolving the cellular structure of the red blood cells. In step 208, the second plurality of images is spatiotemporally annotated with an indication of the density of any red blood cells detected in the second plurality of images. In step 210, the first plurality of images and annotation information from the second plurality of images of spatiotemporal annotations are input into a machine learning subsystem configured to determine the density of any red blood cells present in the one or more optical absorption gaps.
In one example, the red blood cell count is determined from red blood cell density.
Although specific features of the invention are shown in some drawings and not in others, this is for convenience only as each feature may be combined with any or all of the other features in accordance with the invention. The words "including," "comprising," "having," and "with" as used herein are to be interpreted broadly and comprehensively and are not limited to any physical interconnection. Moreover, any embodiments disclosed in the subject application are not to be taken as the only possible embodiments. Other embodiments will occur to those skilled in the art and are within the scope of the following claims.
Moreover, any amendment presented during the prosecution of the patent application for this patent is not a disclaimer of any claim element presented in the application as filed: those skilled in the art cannot reasonably be expected to draft a claim that would literally encompass all possible equivalents, many equivalents will be unforeseeable at the time of the amendment and are beyond a fair interpretation of what is to be surrendered (if anything), the rationale underlying the amendment may bear no more than a tangential relation to many equivalents, and/or the applicant can not be expected to describe certain insubstantial substitutes for any claim element amended for many other reasons.

Claims (24)

1. A method of detecting leukocytes and/or leukocyte subsets from non-invasive capillary video, the method comprising:
acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from a non-invasive capillary video taken with an optical device;
processing the first plurality of images to determine one or more optical absorption gaps located in the capillary vessel;
annotating the first plurality of images with an indication of any optical absorption gaps detected in the first plurality of images;
acquiring a second plurality of images of the same region of interest of the same capillary vessel using advanced optics capable of resolving cellular structures of leukocytes and leukocyte subtypes;
spatio-temporally annotating the second plurality of images with indications of any white blood cells and/or any subset of white blood cells detected in the second plurality of images; and
inputting the first plurality of images and annotation information from a second plurality of images of spatiotemporal annotations into a machine learning subsystem configured to determine the presence of white blood cells in the one or more optical absorption gaps and/or a subset of any white blood cells present in the one or more optical absorption gaps.
2. The method of claim 1, wherein the machine learning subsystem is further configured to determine a leukocyte subset of any optical absorption gaps detected in the first plurality of images.
3. The method of claim 2, wherein the machine learning subsystem is further configured to determine a whole white blood cell difference measure and/or a partial white blood cell difference measure.
4. The method of claim 1, further comprising temporally aligning the first plurality of images with a second plurality of images of spatiotemporal annotations.
5. The method of claim 4, wherein the temporally aligning comprises creating the region of interest and a same region of interest by using a same objective lens on the optical device and the advanced optical device.
6. The method of claim 4, wherein the temporally aligning comprises creating the region of interest and a same region of interest by focusing the optical device and the advanced optical device at a same location in the capillary vessel.
7. The method of claim 4, further comprising generating optical absorption gap reference data comprising a frame identifier and an indication of any optical absorption gaps detected in the first plurality of images.
8. The method of claim 7, further comprising generating spatiotemporal annotated lookup data comprising a frame identifier and an indication of any existing subset of white blood cells.
9. The method of claim 8, wherein temporally aligning the first plurality of images with a second plurality of images of spatiotemporal annotations comprises temporally aligning frame identifiers of the first plurality of images with frame identifiers of the second plurality of images of visually spatiotemporal annotations.
10. The method of claim 9, further comprising inputting the first plurality of images, the optical absorption gap reference data, and spatiotemporal annotated lookup data into the machine learning subsystem, the machine learning subsystem configured to output result data of any detected white blood cells and/or any detected subset of white blood cells and to compare the result table to underlying truth data.
11. The method of claim 9, wherein the machine learning subsystem is configured to output, for each optical absorption gap in the first plurality of images, result data of any detected white blood cells and/or any detected subset of white blood cells and compare the result data to underlying truth data.
12. The method of claim 1, wherein the spatio-temporally annotating the second plurality of images further comprises indicating one or more of: the size, granularity, brightness, velocity, elongation and/or margin of the white blood cells and/or the density change of the red blood cells located upstream or downstream of the location where the white blood cells are detected.
13. The method of claim 1, wherein the subset of leukocytes comprises granulocytes, neutrophils, lymphocytes, monocytes, eosinophils, or basophils.
14. The method of claim 1, wherein the optical device comprises a high resolution camera.
15. The method of claim 1, wherein the advanced imaging apparatus comprises one or more of: a Spectrally Encoded Confocal Microscope (SECM) device, a Scanning Confocal Alignment Plane Excitation (SCAPE) microscope device, a scattering confocal alignment bevel imaging (SCOPI) device, or an oblique back-lighting microscope (OBM) device.
16. The method of claim 1, wherein the predetermined region of the human subject comprises one or more of: fingers, nail capsules, toes, tongue, gums, lips, retina and/or earlobe.
17. The method of claim 1, wherein the optical device is configured to output at least one optical absorption gap signal.
18. The method of claim 1, wherein the advanced optical device is configured to output an advanced optical signal.
19. The method of claim 1, wherein spatiotemporal annotation of the second plurality of images is performed by a human.
20. The method of claim 1, wherein spatiotemporal annotation of the second plurality of images is performed by a processing subsystem.
21. The method of claim 1, further comprising using the first plurality of images and annotation information from the first plurality of images and information from the machine learning subsystem to determine the presence of white blood cells in the one or more optical absorption gaps and/or a subset of any white blood cells present in the one or more optical absorption gaps, the machine learning subsystem having learned and determined the presence of white blood cells in one or more optical absorption gaps and/or a subset of white blood cells present in one or more optical absorption gaps using annotation information from the second plurality of images acquired with the advanced optics.
22. A method of detecting leukocytes and/or leukocyte subsets from non-invasive capillary video, the method comprising:
acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from a non-invasive capillary video taken with an optical device;
processing the first plurality of images to determine one or more optical absorption gaps located in the capillary vessel;
annotating the first plurality of images with an indication of any optical absorption gaps detected in the first plurality of images; and
determining the presence of white blood cells in the one or more optical absorption gaps and/or a subset of any white blood cells present in the one or more optical absorption gaps using the first plurality of images and annotation information from the first plurality of images and information from a machine learning subsystem that has learned and determined the presence of white blood cells in the one or more optical absorption gaps and/or a subset of white blood cells present in the one or more optical absorption gaps using annotation information from a second plurality of images acquired with the advanced optics.
23. A method of determining red blood cell density from non-invasive capillary video, the method comprising:
acquiring a first plurality of images of a region of interest including one or more capillaries of a predetermined region of a human subject from a non-invasive capillary video taken with an optical device;
processing the first plurality of images to determine one or more hemoglobin optical absorption regions located in the capillary vessel;
annotating the first plurality of images with an indication of any regions of hemoglobin optical absorption detected in the first plurality of images;
acquiring a second plurality of images of the same region of interest of the same capillary vessel using advanced optics capable of resolving the cellular structure of the red blood cells;
spatio-temporally annotating the second plurality of images with an indication of the density of any red blood cells detected in the second plurality of images; and
inputting the first plurality of images and annotation information from a second plurality of images of spatiotemporal annotations into a machine learning subsystem configured to determine a density of any red blood cells present in the one or more optical absorption gaps.
24. The method of claim 23, wherein the red blood cell count is determined from red blood cell density.
CN202180038518.2A 2020-05-28 2021-05-27 Method for detecting white blood cells and/or white blood cell subtypes from non-invasive capillary vessel video Pending CN115699108A (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US202063031117P 2020-05-28 2020-05-28
US63/031,117 2020-05-28
PCT/US2021/034455 WO2021242983A1 (en) 2020-05-28 2021-05-27 A method to detect white blood cells and/or white blood cell subtypes form non-invasive capillary videos

Publications (1)

Publication Number Publication Date
CN115699108A true CN115699108A (en) 2023-02-03

Family

ID=78705183

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202180038518.2A Pending CN115699108A (en) 2020-05-28 2021-05-27 Method for detecting white blood cells and/or white blood cell subtypes from non-invasive capillary vessel video

Country Status (6)

Country Link
US (2) US11860154B2 (en)
EP (1) EP4157088A4 (en)
JP (1) JP2023528806A (en)
CN (1) CN115699108A (en)
CA (1) CA3185035A1 (en)
WO (1) WO2021242983A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230032932A1 (en) * 2021-08-02 2023-02-02 Leuko Labs, Inc. Automated system for acquiring images of one or more capillaries in a capillary bed
CN116630294B (en) * 2023-06-08 2023-12-05 南方医科大学南方医院 Whole blood sample detection method and device based on deep learning and storage medium

Family Cites Families (73)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US534211A (en) 1895-02-12 Charles h
US2588528A (en) 1948-07-27 1952-03-11 California Thimble Corp Finger shaped thimble
US4998533A (en) 1986-07-15 1991-03-12 Winkelman James W Apparatus and method for in vivo analysis of red and white blood cell indices
US5596987A (en) 1988-11-02 1997-01-28 Noninvasive Technology, Inc. Optical coupler for in vivo examination of biological tissue
US5068536A (en) 1989-01-19 1991-11-26 Futrex, Inc. Method for providing custom calibration for near infrared instruments for measurement of blood glucose
US5086229A (en) 1989-01-19 1992-02-04 Futrex, Inc. Non-invasive measurement of blood glucose
US5218207A (en) 1989-01-19 1993-06-08 Futrex, Inc. Using led harmonic wavelengths for near-infrared quantitative
US5204532A (en) 1989-01-19 1993-04-20 Futrex, Inc. Method for providing general calibration for near infrared instruments for measurement of blood glucose
US5077476A (en) 1990-06-27 1991-12-31 Futrex, Inc. Instrument for non-invasive measurement of blood glucose
US5237178A (en) 1990-06-27 1993-08-17 Rosenthal Robert D Non-invasive near-infrared quantitative measurement instrument
US5365066A (en) 1989-01-19 1994-11-15 Futrex, Inc. Low cost means for increasing measurement sensitivity in LED/IRED near-infrared instruments
US5362966A (en) 1990-06-27 1994-11-08 Rosenthal Robert D Measurement of finger temperature in near-infrared quantitative measurement instrument
US5436455A (en) 1990-06-27 1995-07-25 Futrex Inc. Non-invasive near-infrared quantitative measurement instrument
US5638818A (en) 1991-03-21 1997-06-17 Masimo Corporation Low noise optical probe
IL107396A (en) 1992-11-09 1997-02-18 Boehringer Mannheim Gmbh Method and apparatus for analytical determination of glucose in a biological matrix
US5452717A (en) 1993-07-14 1995-09-26 Masimo Corporation Finger-cot probe
CA2131060C (en) 1993-09-03 2005-03-15 Hiroshi Yamamoto Non-invasive blood analyzer and method using the same
JP3364323B2 (en) 1994-05-17 2003-01-08 謙 石原 Non-invasive blood analyzer
US5582705A (en) 1995-05-19 1996-12-10 Iowa State University Research Foundation, Inc. Multiplexed capillary electrophoresis system
KR100269563B1 (en) 1995-10-23 2000-12-01 사이토메트릭스, 인코오포레이티드 Apparatus for reflected imaging analysis
US6041247A (en) 1995-11-29 2000-03-21 Instrumentarium Corp Non-invasive optical measuring sensor and measuring method
JP4136017B2 (en) 1996-09-19 2008-08-20 シスメックス株式会社 Particle analyzer
MY117121A (en) 1997-01-09 2004-05-31 Nec Corp Finger fixing apparatus.
US6179159B1 (en) 1998-01-26 2001-01-30 Mariruth D. Gurley Communicable disease barrier digit cover and dispensing package therefor
US6424851B1 (en) 1998-10-13 2002-07-23 Medoptix, Inc. Infrared ATR glucose measurement system (II)
US6438396B1 (en) 1998-11-05 2002-08-20 Cytometrics, Inc. Method and apparatus for providing high contrast imaging
US6358208B1 (en) 1998-11-21 2002-03-19 Philipp Lang Assessment of cardiovascular performance using ultrasound methods and devices that interrogate interstitial fluid
WO2001022741A2 (en) 1999-09-23 2001-03-29 Nadeau Richard G Medical applications of orthogonal polarization spectral imaging
US6213952B1 (en) 1999-09-28 2001-04-10 Orsense Ltd. Optical device for non-invasive measurement of blood related signals utilizing a finger holder
US20090018417A1 (en) 2001-01-19 2009-01-15 Wei-Kung Wang Apparatus monitoring signal in situ
US6563849B2 (en) 2000-07-19 2003-05-13 Hamamatsu Photonics K.K. Drive circuit for light emitting device
CN1166951C (en) 2002-08-28 2004-09-15 彭黎明 Woundless blood cell parameter measuring method
GB0225791D0 (en) 2002-11-05 2002-12-11 Kratos Analytical Ltd Charged particle spectrometer and detector therefor
US7333186B2 (en) 2004-03-17 2008-02-19 Matsushita Electric Industrial Co., Ltd. Method and device for measuring biological information
US7251836B2 (en) 2004-06-08 2007-08-07 Wanda Santiago Finger cover
WO2006069379A2 (en) 2004-12-22 2006-06-29 Bio-Tree Systems, Inc. Medical imaging methods and apparatus for diagnosis and monitoring of diseases and uses therefor
US20060161063A1 (en) 2005-01-19 2006-07-20 Yio-Wha Shau System and method for real-time microcirculation diagnosis
US20070092115A1 (en) 2005-10-26 2007-04-26 Usher David B Method and system for detecting biometric liveness
US7991210B2 (en) 2005-11-23 2011-08-02 Vital Images, Inc. Automatic aortic detection and segmentation in three-dimensional image data
JP5095986B2 (en) 2005-11-30 2012-12-12 学校法人慶應義塾 Non-nail invasive blood substance measuring device and nail plate transpiration device
EA013659B1 (en) 2006-03-10 2010-06-30 Хадас Леви Automated sampling and analysis using a personal sampler device
US7477924B2 (en) 2006-05-02 2009-01-13 Nellcor Puritan Bennett Llc Medical sensor and technique for using the same
US9646415B2 (en) 2006-05-16 2017-05-09 Underground Imaging Technologies, Inc. System and method for visualizing multiple-sensor subsurface imaging data
US8366652B2 (en) 2007-08-17 2013-02-05 The Invention Science Fund I, Llc Systems, devices, and methods including infection-fighting and monitoring shunts
US8842900B2 (en) 2008-10-28 2014-09-23 Sysmex Corporation Specimen processing system and blood cell image classifying apparatus
US10489673B2 (en) 2009-04-03 2019-11-26 Siemens Healthcare Gmbh System and method for detecting landmarks in a three-dimensional image volume
WO2010117576A2 (en) 2009-04-07 2010-10-14 Virginia Commonwealth University Image processing and machine learning for diagnostic analysis of microcirculation
US20110134426A1 (en) 2009-12-04 2011-06-09 Life Technologies Corporation Apparatuses, systems, methods, and computer readable media for acoustic flow cytometry.
US9706138B2 (en) 2010-04-23 2017-07-11 Flir Systems, Inc. Hybrid infrared sensor array having heterogeneous infrared sensors
EP2385474A1 (en) 2010-05-07 2011-11-09 TomTec Imaging Systems GmbH Method for analysing medical data
EP2676121A1 (en) 2011-02-15 2013-12-25 Microbix Biosystems Inc. Methods, systems, and apparatus for performing flow cytometry
US9202127B2 (en) 2011-07-08 2015-12-01 Qualcomm Incorporated Parallel processing method and apparatus for determining text information from an image
KR101273692B1 (en) 2011-09-09 2013-06-17 스타링포스 주식회사 Method for anaiysis of nail capillary image
US8908797B2 (en) 2012-03-07 2014-12-09 Samsung Electronics Co., Ltd. Apparatus and method for time alignment of an envelope tracking power amplifier
WO2013173446A1 (en) 2012-05-15 2013-11-21 1087 Systems, Inc. Cytometry system with interferometric measurement
US8855430B1 (en) 2012-05-30 2014-10-07 Google Inc. Refining image annotations
CN112557301A (en) 2012-07-25 2021-03-26 赛拉诺斯知识产权有限责任公司 Image analysis and measurement of biological samples
US8957374B2 (en) 2012-09-28 2015-02-17 Corning Incorporated Systems and methods for measuring birefringence in glass and glass-ceramics
US9217714B2 (en) 2012-12-06 2015-12-22 Seagate Technology Llc Reflective surfaces for surface features of an article
US9445057B2 (en) 2013-02-20 2016-09-13 Magna Electronics Inc. Vehicle vision system with dirt detection
JP6200168B2 (en) 2013-02-28 2017-09-20 キヤノン株式会社 Image processing apparatus and image processing method
US9424395B2 (en) 2013-03-04 2016-08-23 Heartflow, Inc. Method and system for sensitivity analysis in modeling blood flow characteristics
KR102055474B1 (en) 2013-03-15 2019-12-12 아이리스 인터내셔널 인크. Flowcell systems and methods for particle analysis in blood samples
EP4129061A3 (en) 2014-11-24 2023-04-19 Massachusetts Institute of Technology Systems, apparatus and methods for analyzing blood cell dynamics
US9772282B2 (en) * 2015-11-12 2017-09-26 Massachusetts Institute Of Technology System for wide field-of-view, highly oblique illumination microscopy for scatter-based discrimination of cells
CN114569062A (en) 2016-01-21 2022-06-03 美国印第安纳大学研究和技术公司 Treatment and prevention of retinal vascular disease by photocoagulation
EP3452825A4 (en) 2016-05-04 2020-01-08 Labthroughput LLC System and method for distinguishing blood components
BR112019003144A2 (en) * 2016-08-22 2019-05-21 Iris International, Inc. biological particle classification system and method
EP3602065A4 (en) * 2017-03-20 2020-12-23 Spectral Platforms, Inc. Spectroscopic methods to detect and characterize microorganisms
EP4292520A2 (en) 2017-10-16 2023-12-20 Massachusetts Institute of Technology Systems, devices and methods for non-invasive hematological measurements
US11690540B2 (en) * 2019-01-25 2023-07-04 The General Hospital Corporation Systems, methods, and apparatus for differential phase contrast microscopy by transobjective differential EPI-detection of forward scattered light
CN116211296A (en) 2019-07-24 2023-06-06 麻省理工学院 Finger insert for plication imaging device
US20230032932A1 (en) * 2021-08-02 2023-02-02 Leuko Labs, Inc. Automated system for acquiring images of one or more capillaries in a capillary bed

Also Published As

Publication number Publication date
JP2023528806A (en) 2023-07-06
US20210374963A1 (en) 2021-12-02
EP4157088A1 (en) 2023-04-05
WO2021242983A1 (en) 2021-12-02
US11860154B2 (en) 2024-01-02
US20240085399A1 (en) 2024-03-14
CA3185035A1 (en) 2021-12-02
EP4157088A4 (en) 2024-04-24

Similar Documents

Publication Publication Date Title
Rumyantsev et al. Fundamental bounds on the fidelity of sensory cortical coding
US11850065B2 (en) Vessel imaging system and method
US20240085399A1 (en) Method to detect white blood cells and/or white blood cell subtypes from non-invasive capillary videos
Golan et al. Noninvasive imaging of flowing blood cells using label-free spectrally encoded flow cytometry
Novak et al. In vivo flow cytometer for real-time detection and quantification of circulating cells
Joseph et al. Imaging single-cell blood flow in the smallest to largest vessels in the living retina
Palochak et al. Retinal blood velocity and flow in early diabetes and diabetic retinopathy using adaptive optics scanning laser ophthalmoscopy
Tan et al. In vivo flow cytometry of extremely rare circulating cells
Guevara-Torres et al. Label free measurement of retinal blood cell flux, velocity, hematocrit and capillary width in the living mouse eye
EP2153773A1 (en) Vessel imaging system and method
Pera et al. Diffuse fluorescence fiber probe for in vivo detection of circulating cells
Choo et al. Recent advances in intravital microscopy for investigation of dynamic cellular behavior in vivo
Meng et al. Ultrafast two-photon fluorescence imaging of cerebral blood circulation in the mouse brain in vivo
Gao et al. Smart and fast blood counting of trace volumes of body fluids from various mammalian species using a compact, custom-built microscope cytometer
Pizzagalli et al. Leukocyte Tracking Database, a collection of immune cell tracks from intravital 2-photon microscopy videos
Winer et al. In vivo noninvasive microscopy of human leucocytes
Mautuit et al. Comparing measurements of vascular diameter using adaptative optics imaging and conventional fundus imaging
US9844320B2 (en) System and method for observing an object in a blood vessel
Mohanarathinam et al. Diabetic Retinopathy Detection and Classification using Hybrid Multiclass SVM classifier and Deeplearning techniques
Olayah et al. Blood slide image analysis to classify WBC types for prediction haematology based on a hybrid model of CNN and handcrafted features
Markovic et al. Performance of computer vision in vivo flow cytometry with low fluorescence contrast
CN1166951C (en) Woundless blood cell parameter measuring method
Leon et al. Hyperspectral imaging for in-vivo/ex-vivo tissue analysis of human brain cancer
Niehorster et al. Microsaccade detection using pupil and corneal reflection signals
Wu et al. In vivo imaging flow cytometry of human leukocytes

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination