EP3837525A1 - Dosage à base d'image utilisant des structures de surveillance intelligentes - Google Patents

Dosage à base d'image utilisant des structures de surveillance intelligentes

Info

Publication number
EP3837525A1
EP3837525A1 EP19860313.6A EP19860313A EP3837525A1 EP 3837525 A1 EP3837525 A1 EP 3837525A1 EP 19860313 A EP19860313 A EP 19860313A EP 3837525 A1 EP3837525 A1 EP 3837525A1
Authority
EP
European Patent Office
Prior art keywords
sample
image
processing device
plate
sample card
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP19860313.6A
Other languages
German (de)
English (en)
Other versions
EP3837525A4 (fr
Inventor
Stephen Y. Chou
Wei Ding
Wu Chou
Jun Tian
Yuecheng Zhang
Mingquan Wu
Xing Li
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Essenlix Corp
Original Assignee
Essenlix Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Essenlix Corp filed Critical Essenlix Corp
Publication of EP3837525A1 publication Critical patent/EP3837525A1/fr
Publication of EP3837525A4 publication Critical patent/EP3837525A4/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N1/00—Sampling; Preparing specimens for investigation
    • G01N1/28—Preparing specimens for investigation including physical details of (bio-)chemical methods covered elsewhere, e.g. G01N33/50, C12Q
    • G01N1/2813—Producing thin layers of samples on a substrate, e.g. smearing, spinning-on
    • G01N2001/282—Producing thin layers of samples on a substrate, e.g. smearing, spinning-on with mapping; Identification of areas; Spatial correlated pattern
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
    • G16H10/40—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for data related to laboratory analysis, e.g. patient specimen analysis

Definitions

  • the present invention is related to devices and methods of performing biological and chemical assays, computational imaging, artificial intelligence and machine learning.
  • the present invention provides, among other thing, devices and methods for improving the accuracy of the image-based biological/chemical assaying in a sample.
  • One aspect of the present invention is a method of improving the imaging-based assays using a sample holding device with specially designed monitoring structures for assaying accuracy, efficiency, error detection, monitoring or any combination thereof.
  • Another aspect of the present invention is a method of micro-selective-image- assaying (MSIA) that can perform multiple assaying applications on a single image of the sample taken by the imager.
  • MSIA micro-selective-image- assaying
  • Another aspect of the present invention is a method to detect monitoring structures and locate their centers from the image of the sample for assaying, wherein the method combines machine learning with the error correction using the predetermined structural properties associated with plurality of the monitoring structures.
  • Another aspect of the present invention is a method to estimate the true-lateral- dimension (TLD) in the image-based assaying, from which the actual dimension or morphological features of the analytes in the image of the sample can be determined.
  • TLD true-lateral- dimension
  • Another aspect of the present invention is a method to determine the actual area and associated sample volume for the area-of-interest in the image of the sample, by which the image-based assaying can be conducted on any selected sub-area for flexibility and granularity.
  • Another aspect of the present invention is a method of monitoring the quality of the sample holding device and the quality of the sample preparation in the image-based assaying from the image of monitoring structures in the sample holding device.
  • Another aspect of the present invention is a method of using the predesigned monitoring structures in the sample holding device to adjust the operation of the imager in the image-based assaying.
  • Another aspect of the present invention is a method of removing defects or foreign objects, such as air bubbles, dusts, and so forth, in the image-based assaying from the image of the sample taken by the imager.
  • Another aspect of the present invention is a method of building machine learning models for image-based assay using intelligent monitoring structures, wherein two approaches based on the original image of the sample and on the transformed sample images are described.
  • Another aspect of the present invention is an actual use case of image-based assaying for red blood cells in complete-blood-count (CBC), using the methods and algorithms described herein.
  • CBC complete-blood-count
  • a method of using an apparatus to improve imaging-based assays comprising:
  • a sample holder comprising the first plate and second plates that face each other and a plurality of monitoring structures on a sample contact surface of one or both plates, wherein the monitoring structures have at least one predesigned and predetermined parameter of geometry and/or an optical property, wherein the sample contact surface contacts a sample;
  • step (d) analyzing the images taken in step (c), using an algorithms, to detect a parameter related to the analyte, wherein the analysis comprises comparing the images with the at least one predesigned and predetermined parameter and detecting and/or correcting the defect and errors using the at least one predesigned and predetermined parameter.
  • Figs. 1A and 1 B show side views of the device for use in an imaged-based assay.
  • Fig. 1A illustrates a solid-phase surface with protrusion-type monitoring marks.
  • Fig. 1 B illustrates a solid-phase surface with trench-type monitoring marks.
  • Characteristics corresponding to the monitoring marks e.g., pitch and distance, can be used, in an algorithm, to determine a property of an analyte in a sample.
  • Figs. 2A and 2B show side views of the device for use in an imaged-based assay.
  • Figs. 2A illustrates how the monitoring marks (e.g., protrusion-type) can be separate structures from the spacers.
  • Fig. 2B illustrates how the monitoring marks can be the same structures as the spacers. Characteristics corresponding to the monitoring marks can be used, in an algorithm, to determine a property of an analyte in the sample.
  • Fig. 2C shows a side view of the device for use in an image-based assay.
  • Figs. 2C illustrates how the monitoring marks (e.g., trench-type) can be separate structures from the spacers. Characteristics corresponding to the monitoring marks can be used, in an algorithm, to determine a property of an analyte in a sample.
  • Figs. 2D and 2E show side views of the device for use in an image-based assay.
  • Figs. 2D and 2E illustrate how the monitoring marks can be separate structures from the spacers and disposed on both sample contact areas of the device. Characteristics corresponding to the monitoring marks can be used, in an algorithm, to determine a property of an analyte in a sample.
  • Fig. 5 is exemplary diagram and workflow of an algorithm for using monitoring marks e.g. pillars, together with imaging process and/or machine learning.
  • Fig. 6-A shows analyte detection and localization workflow, which consists of two stages, training and prediction, according to some embodiments of the present invention.
  • Fig. 6-B shows the process to remove one item from an ordered list, according to some embodiments of the present invention.
  • Fig. 7 -A shows an embodiment of a QMAX card used in image-based assaying.
  • Fig. 8 shows a flow diagram of image-based assay with the sample holding device of QMAX card.
  • Fig. 9 shows the flow diagram of TLD/FoV estimation based on pillars or monitoring marks in the image-based assay
  • Fig. 10 shows flow diagram of training the machine learning model for pillar or monitoring mark detection
  • Fig. 11 shows the flow diagram of generating training database for Image transform based machine learning model.
  • Fig. 12 shows the flow diagram of training the image transform model for image transform based pillar or monitoring mark detection.
  • Fig. 13 is a sample image for image-based assay with pillars in the sample holding device
  • Fig. 14 shows the detected pillars in the transformed image used in pillar or monitoring mark detection
  • Fig. 15 illustrates the defects of air bubbles and dusts in the sample for assaying
  • Fig. 16 is a three-side view of the sample holding device, QMAX card with pillars or monitoring marks in the image-based assay using an imager
  • Fig. 17 is an Image of the sample with large air bubbles.
  • Fig. 18 illustrates the detected pillars in the image of the sample.
  • Fig. 19 is a plot of the light intensity curve versus the locations.
  • an imager In an image-based assay for assaying an analyte in a sample, an imager is used to create an image of the sample which is in a sample holder, and the image is used in a determination of a property of the analyte or the sample for assaying.
  • the image distortion can lead to inaccuracy in a determination of a property of the analyte.
  • one fact is poor focusing, since a biological sample itself does not have a sharp edge that is preferred in a focusing.
  • the object dimension will be different from the real one, and other object (e.g. blood cells) can become unidentifiable.
  • a lens might be not perfect, causing different locations of the sample having different degrees of distortion.
  • the sample holder is not in the same plane of the optical imaging system, causing a good focus in one area and poor focusing in another.
  • the present invention is related to the devices and methods that can get a“true” image from a distorted image in the image-based assaying, hence improving the accuracy of an assay.
  • One aspect of the present invention is the devices and methods that use monitoring marks that has an optical observable flat surface that is parallel to neighboring surface
  • Another aspect of the present invention is the devices and methods that use a QMAX card to make at least a part of the sample forming a uniform layer of thickness in the sample holding area of the QMAX card and use monitoring marks on the card to improve the assay accuracy
  • Another aspect of the present invention is the devices and methods that use monitoring marks together with computational imaging, artificial intelligence, and/or machine learning in the image-based assay.
  • lateral dimension refers to the linear dimension in the plane of a thin sample layer that is being imaged.
  • TLD true lateral dimension
  • FoV Field of view
  • micro-feature in a sample can refer to analytes, microstructures, and/or micro-variations of a matter in a sample.
  • Analytes refer to particles, cells, macromolecules, such as proteins, nucleic acids and other moieties.
  • Microstructures can refer to microscale difference in different materials.
  • Micro- variation refers to microscale variation of a local property of the sample.
  • Example of micro- variation is a variation of local optical index and/or local mass.
  • cells examples include blood cells, such as white blood cells, red blood cells, and platelets.
  • sample as used herein relates to a material or mixture of materials containing one or more analytes or entity of interest.
  • the samples are a bodily fluid sample from the subject.
  • solid or semi-solid samples can be provided.
  • the sample can include tissues and/or cells collected from the subject.
  • the sample can be a biological sample.
  • biological samples can include but are not limited to, blood, serum, plasma, a nasal swab, a nasopharyngeal wash, saliva, urine, gastric fluid, spinal fluid, tears, stool, mucus, sweat, earwax, oil, a glandular secretion, cerebral spinal fluid, tissue, semen, vaginal fluid, interstitial fluids derived from tumorous tissue, ocular fluids, spinal fluid, a throat swab, exhaled condensates (e.g.
  • the samples may include nasopharyngeal wash.
  • Nasal swabs, throat swabs, stool samples, hair, fingernail, ear wax, breath, and other solid, semi-solid, or gaseous samples may be processed in an extraction buffer, e.g., for a fixed or variable amount of time, prior to their analysis.
  • the extraction buffer or an aliquot thereof may then be processed similarly to other fluid samples if desired.
  • tissue samples of the subject may include but are not limited to, connective tissue, muscle tissue, nervous tissue, epithelial tissue, cartilage, cancerous sample, or bone.
  • a sample may be obtained from a subject, e.g., a human, and it may be processed prior to use in the subject assay.
  • the protein/nucleic acid may be extracted from a tissue sample prior to use, methods for which are known.
  • the sample may be a clinical sample, e.g., a sample collected from a patient.
  • the samples also can be the sample of food, environments, and others. Some of the samples have a shape deformable but not free-flowable (e.g. sputum).
  • analyte refers to any substance that is suitable for testing in the present invention.
  • An analyte includes, but limited to, atoms, molecules (e.g., a protein, peptides, DNA, RNA, nucleic acid, or other molecule), cells A tissues, viruses, bacteria, and nanoparticles with different shapes.
  • a biomarker is an analyte.
  • the terms“determining,”“measuring,” and“assessing,” and“assaying” are used interchangeably and include both quantitative and qualitative determinations.
  • the term“light-emitting label” refers to a label that can emit light when under an external excitation. This can be luminescence. Fluorescent labels (which include dye molecules or quantum dots), and luminescent labels (e.g., electro- or chemi-luminescent labels) are types of light-emitting label.
  • the external excitation is light (photons) for fluorescence, electrical current for electroluminescence and chemical reaction for chemi-luminescence.
  • An external excitation can be a combination of the above.
  • labeled analyte refers to an analyte that is detectably labeled with a light emitting label such that the analyte can be detected by assessing the presence of the label.
  • a labeled analyte may be labeled directly (i.e. , the analyte itself may be directly conjugated to a label, e.g., via a strong bond, e.g., a covalent or non-covalent bond), or a labeled analyte may be labeled indirectly (i.e. , the analyte is bound by a secondary capture agent that is directly labeled).
  • hybridizing and “binding”, with respect to nucleic acids, are used interchangeably.
  • Hybridization refers to a reaction in which one or more polynucleotides react to form a complex that is stabilized via hydrogen bonding between the bases of the nucleotide residues.
  • the hydrogen bonding may occur by Watson-Crick base pairing, Hoogstein binding, or in any other sequence-specific manner.
  • the complex may comprise two strands forming a duplex structure, three or more strands forming a multi-stranded complex, a single self-hybridizing strand, or any combination of these.
  • hybridization can be performed under conditions of various stringency. Suitable hybridization conditions are such that the recognition interaction between a capture sequence and a target nucleic acid is both sufficiently specific and sufficiently stable. Conditions that increase the stringency of a hybridization reaction are widely known and published in the art. See, for example, Green, et al., (2012), infra.
  • spacer and “optical calibration marks” and “optical calibration marks” and “pillars” are interchangeable.
  • test refers to an investigative (analytic) procedure in and not limited to laboratory, medicine, pharmacology, environmental biology, healthcare, and molecular biology - for and not limited to qualitatively assessing or quantitatively measuring the presence, amount, concentration, or functional activity of a target entity (I.e. the analyte).
  • the analyte can be a drug, a biochemical substance, or a cell in an organism or organic sample such as human blood.
  • image-based assay refers to an assaying procedure that utilizes the image of the sample taken by an imager, wherein the sample can be and not limited to medical, biological and chemical sample.
  • imager refers to any device that can take image of the objects it includes and not limited to cameras in the microscope, smartphone, or special device that can take image at various wavelength.
  • sample feature refers to some property of the sample that represents a potentially interesting condition.
  • a sample feature is a feature that appears in an image of a sample and can be segmented and classified by a machine learning model or some other algorithms.
  • sample features include and not limited to analyte types in the sample, e.g. red blood cells, white blood cells, and tumor cells, and it includes analyte shape, count, size, volume, concentration and the like.
  • defects in the sample refers to foreign objects and artifacts that should not exist in an ideal sample condition or should not be considered in the sample. They can come from and not limited to pollutants, e.g. dusts, air bobbles, etc., and from the peripheral objects, including structural objects in the sample, e.g. monitor marks (such as pillars) in the sample holding device. Defects in the sample can be of significant size and take significant amount of volume in the sample for assaying, e.g. air bubbles, wherein they can appear in different shapes, sizes, amounts, and concentrations in the sample, and they also sample dependent varying from sample to sample.
  • pollutants e.g. dusts, air bobbles, etc.
  • monitor marks such as pillars
  • morphological feature of the analytes refers to the appearance (e.g. shape, color, size, etc.) and the structure of the analyte.
  • the term“homographic transform” refers to a class of collineation transforms induced by an isomorphism of the projective spaces. It is known in the field of image processing and it is applied in camera models to characterize the image plane and the corresponding physical plane in the real world.
  • machine learning refers to algorithms, systems and apparatus in the field of artificial intelligence that often use statistical techniques and artificial neural networks to give computer the ability to "learn” (i.e., progressively improve performance on a specific task) from data without being explicitly programmed.
  • artificial neural network refers to a layered connectionist system inspired by the biological networks that can“learn” to perform tasks by considering examples, generally without being programmed with any task-specific rules.
  • neural network refers to a class of multilayer feed-forward artificial neural networks most commonly applied to analyzing visual images.
  • Deep learning refers to a broad class of machine learning methods in artificial intelligence (Ai) that iearn from data with some deep network structures.
  • machine learning model refers to a trained computational model that is built from a training process in the machine learning from the data.
  • the trained machine learning model is applied during the inference stage by the computer that gives computer the capability to perform certain tasks (e.g. detect and classify the objects) on its own.
  • machine learning models include ResNet, DenseNet, etc. which are also named as“deep learning models” because of the depth in their layered network structure.
  • image segmentation refers to an image analysis process that partitions a digital image into multiple image patch segments (sets of pixeis, often with a set of bit-map masks that cover the image segments enclosed by their segment boundary contours) image segmentation can be achieved through the image segmentation algorithms in image processing, such as watershed, grabcuts, mean-shift, etc., and it can aiso be achieved through dedicated machine iearning algorithms, such as MaskRCNN, etc.
  • a device for assaying a micro-feature in a sample using an imager the device
  • ii. are inside the sample during an assaying the microstructure, wherein the sample forms, on the sample contact area, a thin layer of a thickness less than 200 urn;
  • iii. have their lateral linear dimension of about 1 urn (micron) or larger, and iv. have at least one lateral linear dimension of 300 urn or less; and wherein during the assaying at least one monitoring mark is imaged by the imager wherein used during assaying the analyte; and a geometric parameter (e.g. shape and size) of the monitoring mark, and/or a pitch between monitoring marks are (a) predetermined and known prior to assaying of the analyte, and (b) used as a parameter in an algorithm that determines a property related to the micro-feature.
  • a geometric parameter e.g. shape and size
  • a device for assaying a micro-feature in a sample using an imager the device
  • a solid-phase surface comprising a sample contact area for contacting a sample which contains a micro-feature
  • each monitoring mark comprises either a
  • the protrusion or the trench comprises a flat surface that is substantially parallel to a neighbor surface that is a portion of the solid-phase surface adjacent the protrusion or the trench;
  • a distance between the flat surface and the neighboring surface is about
  • the flat surface an area that has (a) a linear dimension is at least about 1 urn or larger, and (b) at least one linear dimension 150 urn or less;
  • the flat surface of at least one monitoring mark is imaged by an imager used during assaying the micro-feature
  • v. a shape of the flat surface, a dimension of the flat surface, a distance between the flat surface and the neighboring surface, and/or a pitch between monitoring marks are (a) predetermined and known prior to assaying of the micro-feature, and (b) used as a parameter in an algorithm that determines a property related to the micro-feature.
  • a device for assaying a micro-feature in a sample using an imager comprising:
  • the first plate and the second plate are movable relative to each other into different configurations
  • each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample that contains a micro-feature
  • one or both of the first plate and the second plate comprises the spacers that are permanently fixed on the inner surface of a respective plate, iv. the spacers have a substantially uniform height that is equal to or less than 200 microns, and a fixed inter-spacer-distance (ISD);
  • ISD inter-spacer-distance
  • the monitoring marks are made of a different material from the sample; vi. the monitoring marks are inside the sample during an assaying the
  • the sample forms, on the sample contact area, a thin layer of a thickness less than 200 urn; and vii. the monitoring marks have their lateral linear dimension of about 1 urn
  • At least one monitoring mark is imaged by the imager wherein used during assaying the micro-feature; and a shape of the flat surface, a dimension of the flat surface, a distance between the flat surface and the neighboring surface, and/or a pitch between monitoring marks are (a) predetermined and known prior to assaying of the micro feature, and (b) used as a parameter in an algorithm that determines a property related to the micro-feature;
  • one of the configurations is an open configuration, in which: the two plates are partially or completely separated apart, the spacing between the plates is not regulated by the spacers, and the sample is deposited on one or both plates;
  • another of the configurations is a closed configuration which is configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying the pressing force on the force area; and in the closed configuration: at least part of the sample is compressed by the two plates into a layer of highly uniform thickness and is substantially stagnant relative to the plates, wherein the uniform thickness of the layer is confined by the sample contact areas of the two plates and is regulated by the plates and the spacers; and
  • a monitoring mark is (i) a different structure from the spacers, or (ii) the same structure that is used as a spacer.
  • A2-2 A device for assaying a micro-feature in a sample using an imager, the device
  • the first plate and the second plate are movable relative to each other into different configurations
  • each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample that contains a micro-feature
  • one or both first plate and the second plate comprises the spacers that are
  • the spacers have a substantially uniform height that is equal to or less than 200 microns, and a fixed inter-spacer-distance (ISD);
  • each monitoring mark comprises either a protrusion or a trench on one or both sample contact areas;
  • the protrusion or the trench comprises a flat surface that is substantially parallel to a neighbor surface that is a portion of the solid-phase surface adjacent the protrusion or the trench;
  • a distance between the flat surface and the neighboring surface is about 200
  • the flat surface an area that has (a) a linear dimension is at least about 1 urn or larger, and (b) at least one linear dimension 150 urn or less;
  • the flat surface of at least one monitoring mark is imaged by an imager used during assaying the micro-feature
  • x. a shape of the flat surface, a dimension of the flat surface, a distance between the flat surface and the neighboring surface, and/or a pitch between monitoring marks are (a) predetermined and known prior to assaying the micro-feature, and (b) used as a parameter in an algorithm that determines a property related to the micro feature.
  • one of the configurations is an open configuration, in which: the two plates are partially or completely separated apart, the spacing between the plates is not regulated by the spacers, and the sample is deposited on one or both plates;
  • another of the configurations is a closed configuration which is configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying a pressing force on the force area; and in the closed configuration: at least part of the sample is compressed by the two plates into a layer of highly uniform thickness and is substantially stagnant relative to the plates, wherein the uniform thickness of the layer is confined by the sample contact areas of the two plates and is regulated by the plates and the spacers; and
  • a monitoring mark is (i) a different structure from the spacers, or (ii) the same structure that is used as a spacer.
  • a device for image-based assay comprising:
  • the device has at least five monitoring marks wherein at least three of the monitoring marks are not aligned on a linear line.
  • An apparatus for assaying an analyte in a sample using an imager comprising:
  • a system for performing an imaging-based assay comprising:
  • the thickness of the thin layer is configured, so that analytes form a monolayer in the sample holder.
  • the term“monolayer” means that in the thin sample layer inside the sample holder, there is no substantial overlap between two neighboring analytes in the direction normal to the plane of the sample layer.
  • Another aspect of the present invention is to utilize the pillars or monitoring marks of the sample holding device with computational imaging, artificial intelligence and/or machine learning. It has a process of forming the images from measurements, using algorithms to process the image and map the objects in the image to their actual dimensions in real world.
  • Machine learning (ML) is applied in the present invention to learn the salient features of the objects in the sample, captured by the ML models, built and trained from the images of the sample taken by the imager.
  • Intelligent decision logic is built into and applied in the inference process of the present invention to detect and classify the target objects in the sample according to the knowledge embedded in the ML models.
  • Fig. 9 shows a block diagram of process 100 for TLD (true-literal-dimension)/FoV (field- of-view) estimation in the present invention.
  • actions may be removed, combined, or broken up into sub-actions.
  • the process begins at the action module 101 , where the process captures an image of the sample for assaying, wherein the image is taken by an imager over the sample holding device, e.g. QMAX card, that has pillars or monitoring marks and a structure described in the previous section of monitoring marks on QMAX card.
  • the captured image from action module 101 is fed to the action module 102 as input and in the action module 102, the image for assaying is processed and searched to detect and locate the centers of pillars or monitoring marks.
  • the process 102 has a pre- trained machine learning model, M_Pillar, to detect pillar or monitoring mark in the image for assaying.
  • pillars or monitoring marks in the present invention have a known shape and dimension, fabricated by high precision nano-imprint fabrication process. Moreover, these pillars or monitoring marks are distributed in a pre-defined periodic pattern in the sample holding device and have a known pitch distance between them. These features in the present invention become critical for the process 100, because pillars or monitoring marks are small, around ⁇ 30 microns, surrounded by samples in the sample holding device, and subjected to strong light scattering and diffractions among particles in the sample. In addition, they are often not in the focus of the imager as the imager is focused on the analytes in the sample for the image-based assay, not on the structures of the sample holder.
  • the action module 103 takes the detected pillars and their centers from the action module 102 in the process 100, and it performs post error correction and estimation refinement based on the properties of pillars or monitoring marks described in the present invention.
  • the detected pillars should be aligned in a pre-defined known periodic pattern and the distance between two adjacent pillar centers should subject to a known pitch distance, etc.
  • the action taken by the action module 103 is to eliminate false detections, align the detected pillars in the design pattern known from fabrication, using the periodicity of their distribution to find others, and adjust the position of the pillar centers according to the known periodicity and pitch distance between them.
  • the combination of the actions taken by the action module 102 and 103 in present invention makes pillar or monitoring mark detection robust with precision for TLD/FoV estimation.
  • the action module 103 of the process 100 estimates a homographic transform based on the detected pillar centers from action module 102 and 103.
  • the homographic transform is known in the art of image processing where the image taken by the imager on the sample is modeled by a perspective projection between the objects in the image being taken and the actual objects in real world being imaged upon. This perspective projection can be characterized by a homographic transform (a.k.a. perspective transform) described by a transformation matrix H described below:
  • H a 3x3 non-singular homogenous matrix that maps objects in the image taken by the imager to the objects in the actual sample plan which is being imaged upon, from which the length or the area of the objects in the image taken by the imager is mapped to its actual size in real world, and by which the TLD/FoV of the objects and their actual size in the image can be determined.
  • the transformation matrix H it requires to know at least 4 pair of points corresponding to the mapping of H from the image taken by the imager to the actual sample plan being imaged upon. And these 4 pair of points are used as anchors that bind the image taken by the imager and the actual sample plan being imaged upon through a homographic transform between them, and it needs at least 4 non-colinear pillar centers to make the H matrix non singular.
  • the action module 104 of the process 100 estimates the homographic transform matrix H for TLD/FoV estimation based on the detected pillar centers from the action module 103. Moreover, in action module 104, it uses the estimated transformation matrix H to do TLD/FoV estimation in the subsequent image-based assaying process.
  • dedicated machine learning model for pillar or monitoring mark detection is built from the training data. In some embodiment of the present invention, the pillar and monitoring mark detection is performed directly on the image for assaying.
  • Fig. 10 shows a block diagram of process 200 for building a machine learning model for pillar or monitoring mark detection on the original input image for assaying.
  • the process begins at the action module 201 , where the process obtains a set of images for assaying from the imager in a training database DB0. These images are collected by taking image on the samples in the sample holding device, e.g. QMAX card in the action module 202, it takes each training image from DB0 and label the pillars or monitoring marks in each image. The labeled images are saved in a second training database DB1 for machine learning model training. This training dataset DB1 is dedicated for training machine leaning model to detect pillars or monitoring marks, which is different from the typical training database for detecting analytes in the sample.
  • the action module 203 it takes the new training database DB1 from the action module 202, and select a machine learning model structure, in a form of deep neural network, to train the model against the training database DB1.
  • a machine learning model of RetinaNet is used, and in some other embodiments, a machine learning model of Fast-RCNN is selected.
  • Tensorflow and PyTourch are used to train the machine learning model using the training database DB1 for pillar or monitoring mark detection.
  • the process 200 ends with the action module 205, in which the machine learning model obtained from the action module 204 are verified and saved for assaying applications.
  • Fig. 11 is a block diagram of creating a special training database for the image transform- based machine learning approach for pillar and monitoring mark detection.
  • the process 300 starts with the action module 301 that takes a training image database DB1A consisting of images for assaying taken by the imager.
  • the images from DB1A are labeled for pillars or monitoring marks.
  • the labeled images in DB1A are the input to the action module 303, where they are transformed by covering the labeled pillar or monitoring mark with a white mask along its contour.
  • the transformed images from the action module 303 are inputs to the action module 304, where they are further transformed by applying the black mask on areas in the image not covered by the white masks from the action module 303.
  • Using white masks for pillars and black masks for areas not covered by white masks in the present invention has two advantages. One it maximizes the contrast between the pillars and other areas, and second, it suppresses the noise in the background, making the subsequent detection of the pillars or monitoring marks and their centers more robust.
  • action module 304 the transformed image by 302 and 303 are verified and saved in a new target image training database DB2A for training the image transformation model in pillar or monitoring mark detection.
  • Fig. 12 shows a block diagram of building an image transform based machine learning model for pillar and monitor mark detection.
  • the process 400 starts with the action module 401.
  • the action module 401 load a training image database DB1A consisting of the images taken by the imager for assaying.
  • the action module 402 load the target image training database DB2A obtained by transforming the images in DB1A using process 300 described previously.
  • it takes DB1A the training image database and the paired target image training database DB2A from the action module 402 as training target. It selects a machine learning framework to train a machine learning model that transforms the original images taken by the imager for assaying to new images of white pillar or monitoring masks and black background in the target image training database.
  • the machine learning framework of Pixel-to- Pixel (P2P) transform is selected, and in some other embodiment of the present invention, the machine learning framework of CycleGAN is used.
  • the action module 403 trains a machine learning model TModel using the training image database DB1A and the paired target image transform database DB2A from the process 300.
  • the pillar and monitoring mark detection comprises the following actions:
  • Fig. 17 is an actual image of the sample taken by an imager for assaying the red blood cell. It is apparent that the noise level in the image of the sample is quite high, and in the image of the sample, the focus on the monitoring structures, e.g. pillars or monitoring marks in the sample holder, is poor, because in the image-based assaying, the focus is on the analytes not on the prereferals. This makes the detection and location of the pillars or monitoring marks in the image of the sample challenging.
  • Fig. 18 shows the detection results of the pillars of monitoring marks from the image of the sample in assaying red blood cell based on the transformed image using the system and methods described herein.
  • the image for assaying are partitioned into disjoint patches, and patch dependent holomorphic transform is estimated for each image patch if there are at least 4 detected non-collinear pillar centers in that patch. Otherwise, it uses the holomorphic transform estimated from all detected pillar centers in the image to determine the TLD/FoV in the image-based assaying.
  • the method of TLD/FoV estimation described herein opens the possibility for other applications in image-based assaying.
  • it is applied to remove the defects, e.g. air bubbles, dusts, etc. in the image-based assaying.
  • Air bubbles and dusts can occur in samples, depending on the environment, assaying operation, and the type of the sample being used for assaying.
  • These defects in the sample for assaying should be taken away from the sample for better assaying accuracy. However, this can be extremely difficult once these defects occur and are trapped in the closed space of the sample holding device. And moreover, in many cases, the device is press sealed and should not be opened again.
  • Fig. 15 depicts the situation that both air bubbles and dusts occur in the sample.
  • it performs the micro-selective-image-assaying, comprising:
  • the area removed from the image of the sample is larger than the area of the detected defects with a margin D, and consequently, a larger volume is virtually removed in the image-based assaying according to the enlarged defects area.
  • One benefit of this operation is to further reduce the negative impact of the defects on the assaying results, because analytes can attach to the defects and the local uniformity of the analytes distribution can be influenced.
  • the present invention makes a critical use of the structure of the sample holding device, e.g. QMAX card, and the pillars or monitoring marks therein.
  • the image of the sample taken by the imager on the sample holding device, e.g. QMAX card is a pseudo-3D image, because the height of the sample in the sample holding device is known priori and uniform.
  • the sample volume corresponding to the surface area of the objects in the image of the sample can be determined, once its actual area size in the original sample plan can be obtained.
  • the structure of the pillars or monitoring marks in the sample holding device are utilized for more reliable TLD/FoV estimation.
  • the present invention selects certain area where analytes are not forming clusters with the sample volume of the selected area determined from the image of the sample and the uniform height of the sample in the sample holding device. In some embodiment, it selects the area in the image of the sample based on less defects, better signal-to-noise ratio, focus condition, etc. for better assaying accuracy.
  • the combination of machine learning based pillar and monitoring mark detection for FoV estimation and the machine learning based defects detection and segmentation to reduce the variations in the sample makes the described approach in the present invention flexible and resilient in the image-based assaying.
  • the thickness of the sample is configured to a thin thickness, so that the objects (e.g. cells) of interests forming a monolayer (i.e. there is no significant overlap between the object in the direction normal to the sample layer).
  • a method for determining a fabrication quality of a QMAX card using an imager comprising:
  • a method for determining a fabrication quality of a QMAX card using an imager comprising:
  • determining the fabrication quality comprises measuring a characteristic (e.g., a length, width, pitch, webbing) of one or more monitoring marks, and comparing the measured characteristic with a reference value to determine a fabrication quality of the QMAX card.
  • a characteristic e.g., a length, width, pitch, webbing
  • determining the fabrication quality comprises measuring a first characteristic (e.g., an amount, a length, width, pitch, webbing) of one or more first monitoring marks, and comparing the measured first characteristic with a second characteristic (e.g., a number, a length, width, pitch, webbing) of one or more second monitoring marks to determine a fabrication quality of the QMAX card.
  • a first characteristic e.g., an amount, a length, width, pitch, webbing
  • a second characteristic e.g., a number, a length, width, pitch, webbing
  • monitoring marks placed inside a thin sample can be used to monitor the operation conditions for the QMAX card.
  • the operation conditions can include whether the sample is loaded properly, whether the two plates are closed properly, whether the gap between the two plates is the same or approximately the same as a predetermined value.
  • the operation conditions of the QMAX card based assay is monitored by taking the images of the monitoring mark in a closed configuration. For example, if the two plates are not closed properly, the monitoring marks will appear differently in an image from the situation that the two plates are closed properly. A monitoring mark surrounded by a sample (properly closed) will have a different appearance than a monitoring mark not surrounded (not properly closed) by the sample. Hence, it can provide information on the sample loading conditions.
  • a device for using a monitoring mark to monitor an operation condition of the device comprises of a first plate, a second plate, spacers, and one or more monitoring marks, wherein:
  • the first plate and the second plate are movable relative to each other into different
  • each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample being analyzed
  • one or both first plate and the second plate comprises the spacers that are permanently fixed on the inner surface of a respective plate;
  • the monitoring mark has at least one of its dimensions that (a) is predetermined and known, and (b) is observable by an imager;
  • the monitoring mark is a microstructure that has at least one lateral linear dimension of
  • the monitoring mark is inside the sample
  • one of the configurations is an open configuration, in which: the two plates are partially or completely separated apart, the spacing between the plates is not regulated by the spacers, and the sample is deposited on one or both plates;
  • another configuration is a closed configuration, which is configured after the sample is deposited in the open configuration and the plates are forced to the closed
  • the monitoring mark is imaged to determine (i) whether the two plates have reached the intended closed configuration thereby regulating the sample thickness to be approximately a
  • the image of the monitoring mark is used to determine whether the two plates have reached the intended closed configuration, wherein the sample is regulated to have a thickness of approximately a predetermined thickness.
  • the image of the monitoring mark is used to determine whether a sample has been loaded as desired.
  • the monitoring mark is imaged to determine whether the two plates have reached the intended closed configuration wherein the sample thickness is regulated to be a predetermined thickness, and to determine whether a sample has been loaded as desired.
  • the spacers serve as the monitoring marks.
  • the system comprises the device and a computational device and a non-transitory computer readable medium having instructions that, when executed, it performs the determination process in assaying.
  • a non-transitory computer readable medium having instructions that, when executed, perform a method comprising using one or more images of a thin sample layer together with monitoring marks to determine (i) whether the two plates have reached the intended closed configuration thereby regulating the sample thickness to be approximately a predetermined thickness, or (ii) whether a sample has been loaded as desired.
  • the system comprises a non-transitory computer readable medium having instructions that, when executed, perform any method of the present disclosure.
  • a method for using a monitoring mark to monitor an operating condition of the device comprising: a) obtaining a device of any prior embodiment, wherein the device comprises two movable plates, spacers, and one or more monitoring marks where the monitoring marks are in the sample contact area;
  • a device for assaying a micro-feature in a thin sample using an imager comprising:
  • I. has a sharp edge that (i) has predetermined and known shape and
  • a device for assaying a micro-feature in a thin sample using an imager comprising:
  • i. comprises either a protrusion or a trench from the solid-phase surface ii. has a sharp edge that (i) has predetermined and known shape and
  • iii. is a microstructure that at least one lateral linear dimension of 300 urn or less;
  • a device for assaying a micro-feature in a thin sample using an imager comprising:
  • each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample that comprises or is suspected to comprise a micro-feature
  • the monitoring mark has a sharp edge that (a) has predetermined and known shape and dimension, and (b) is observable by an imager that images the micro-feature;
  • the monitoring mark is a microstructure that at least one lateral linear dimension of 300 urn or less; and v. the monitoring mark is inside the sample;
  • AA-3 A device for assaying a micro-feature in a thin sample using an imager, the device comprising:
  • the first plate and the second plate are movable relative to each other into different configurations
  • each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample that comprises or is suspected to comprise a micro-feature
  • one or both of the first plate and the second plate comprises the spacers that are
  • the monitoring mark has a sharp edge that (a) has predetermined and known shape and dimension, and (b) is observable by an imager that images the micro-feature;
  • the monitoring mark is a microstructure that at least one lateral linear dimension of 300 urn or less;
  • the monitoring mark is inside the sample
  • one of the configurations is an open configuration, in which the two plates are partially or completely separated apart, the spacing between the plates is not regulated by the spacers, and the sample is deposited on one or both plates;
  • another of the configurations is a closed configuration which is configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying the imprecise pressing force on the force area; and in the closed configuration: at least part of the sample is compressed by the two plates into a layer of highly uniform thickness and is substantially stagnant relative to the plates, wherein the uniform thickness of the layer is confined by the sample contact areas of the two plates and is regulated by the plates and the spacers; and
  • a monitoring mark is (i) a different structure from the spacers, or (ii) the same structure that is used as a spacer.
  • An apparatus for improving image-taking of a micro-feature in a sample comprising:
  • the imager takes images, and wherein at least one image comprises both a portion of sample and the monitoring.
  • a system for improving image-taking of a micro-feature in a sample comprising:
  • CC-1 An apparatus for improving analysis of an image of a micro-feature in a sample, the apparatus comprising:
  • the computation device runs an algorithm that utilizes the mark as a parameter together with an imaging processing method to improve the image quality in the image.
  • a system for improving analysis of images of a micro-feature in a sample comprising:
  • an imager being used in assaying a sample of comprises or is suspected to comprise a micro-feature by taking one or multiple images of the sample and the mark; and
  • CC-3 A computer program product for assaying a micro-feature in a sample, the program comprising computer program code applied and adapted for, in at least one image:
  • CC-4 A computing devices for assaying a micro-feature in a sample, the computation device comprising computing devices that operate the algorithms in any of embodiments of the present invention.
  • CC-5 The method, device, computer program product, or system of any prior embodiment, wherein the improvement of the image quality comprises at least one selected from the group consisting of denoising, image normalization, image sharpening, image scaling, alignment, super resolution, deblurring, and any combination of thereof.
  • the imaging processing method comprises at least one selected from the group consisting of a histogram-based operation, a mathematics-based operation, a convolution-based operation, a smoothing operation, derivative-based operation, a morphology-based operation, shading correction, image enhancement and/or restoration, segmentation, feature extraction and/or matching, object detection and/or classification and/or localization, image understanding, and any combination of thereof.
  • CC-6.1 The method, device, computer program product, or system of any prior embodiment, wherein the histogram-based operation comprises at least one selected from the group consisting of contrast stretching, equalization, minimum filtering, median filtering, maximum filtering, and any combination thereof.
  • CC-6.2 The method, device, computer program product, or system of any prior embodiment, wherein the mathematics-based operation comprises at least one selected from the group consisting of binary operation (e.g., NOT, OR, AND, XOR, and SUB) arithmetic-based operations (e.g., ADD, SUB, MUL, DIV, LOG, EXP, SORT, TRIG, and INVERT), and any combination thereof.
  • CC-6.3 The method, device, computer program product, or system of any prior embodiment, wherein the convolution-based operation comprises at least one selected from the group consisting of an operation in the spatial domain, Fourier transform, DCT, integer transform, an operation in the frequency domain, and any combination thereof.
  • CC-6.4 The method, device, computer program product, or system of any prior embodiment, wherein the smoothing operation comprises at least one selected from the group consisting of a linear filter, a uniform filter, a triangular filter, a Gaussian filter, a non-linear filter, a medial filter a kuwahara filter, and any combination thereof.
  • the derivative-based operation comprises at least one selected from the group consisting of a first-derivative operation, a gradient filter, a basic derivative filter, a Prewitt gradient filters, a Sobel gradient filter, an alternative gradient filter, a Gaussian gradient filter, a second derivative filter, a basic second derivative filter, a frequency domain Laplacian, a Gaussian second derivative filter, an Alternative Laplacian filter, a Second-Derivative-in-the-Gradient-Direction (SDGD) filter, a third derivative filter, a higher derivative filter (e.g., a greater than third derivative filter), and any combination thereof.
  • SDGD Second-Derivative-in-the-Gradient-Direction
  • CC-6.6 The method, device, computer program product, or system of any prior embodiment, wherein the morphology-based operation comprises at least one selected from the group consisting of dilation, erosion, Boolean convolution, opening and/or closing, hit-and-miss operation, contour, skeleton, propagation, gray-value morphological processing, Gray-level dilation, gray-level erosion, gray-level opening, gray-level closing, morphological smoothing, morphological gradient, morphological Laplacian, and any combination thereof.
  • CC-6.7 The method, device, computer program product, or system of any prior embodiment, wherein the image enhancement and/or restoration comprises at least one selected from the group consisting of sharpening, unsharpening, noise suppression, distortion suppression, and any combination thereof.
  • CC-6.8 The method, device, computer program product, or system of any prior embodiment, wherein the segmentation comprises at least one selected from the group consisting of thresholding, fixed thresholding, Histogram-derived thresholding, Isodata algorithm, background- symmetry algorithm, Triangle algorithm, Edge finding, Gradient-based procedure, zero-crossing based procedure, PLUS-based procedure, Binary mathematical morphology, salt-or-pepper filtering, Isolate objects with holes, filling holes in objects, removing border-touching objects, Exo skeleton, Touching objects, Gray-value mathematical morphology, Top-hat transform, thresholding, Local contrast stretching, and any combination thereof.
  • the segmentation comprises at least one selected from the group consisting of thresholding, fixed thresholding, Histogram-derived thresholding, Isodata algorithm, background- symmetry algorithm, Triangle algorithm, Edge finding, Gradient-based procedure, zero-crossing based procedure, PLUS-based procedure, Binary mathematical morphology, salt-or-pepper filtering, Isolate objects with holes, filling holes in
  • CC-6.9 The method, device, computer program product, or system of any prior embodiment, wherein the feature extraction and/or matching comprises at least one selected from the group consisting of Independent component analysis, ISO map, Kernel Principal Component Analysis, Latent semantic analysis, partial least squares, principal component analysis, multifactor dimensionality reduction, nonlinear dimensionality reduction, multilinear principal component Analysis, Multilinear subspace learning, Semidefinite embedding, Autoencoder, and any combination thereof.
  • a device for assaying a micro-feature in a thin sample using an imager comprising:
  • ii. is a microstructure that at least one lateral linear dimension of 300 urn or less;
  • a device for assaying a micro-feature in a thin sample using an imager comprising:
  • i. comprises either a protrusion or a trench from the solid-phase surface ii. has a sharp edge that (i) has predetermined and known shape and
  • iii. is a microstructure that at least one lateral linear dimension of 300 urn or less;
  • a device for assaying a micro-feature in a thin sample using an imager comprising:
  • each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample that comprises or is suspected to comprise a micro-feature
  • the monitoring mark has a sharp edge that (a) has predetermined and known shape and dimension, and (b) is observable by an imager that images the micro-feature;
  • the monitoring mark is a microstructure that at least one lateral linear dimension of 300 urn or less;
  • AA-3 A device for assaying a micro-feature in a thin sample using an imager, the device
  • each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample that comprises or is suspected to comprise a micro-feature
  • one or both of the first plate and the second plate comprises the spacers that are permanently fixed on the inner surface of a respective plate
  • the monitoring mark has a sharp edge that (a) has predetermined and known shape and dimension, and (b) is observable by an imager that images the micro feature;
  • the monitoring mark is a microstructure that at least one lateral linear dimension of 300 urn or less;
  • At least one of the marks is imaged by the imager during the assaying wherein one of the configurations is an open configuration, in which: the two plates are partially or completely separated apart, the spacing between the plates is not regulated by the spacers, and the sample is deposited on one or both of the plates;
  • another of the configurations is a closed configuration which is configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying the imprecise pressing force on the force area; and in the closed configuration: at least part of the sample is compressed by the two plates into a layer of highly uniform thickness and is substantially stagnant relative to the plates, wherein the uniform thickness of the layer is confined by the sample contact areas of the two plates and is regulated by the plates and the spacers; and
  • a monitoring mark is (a) a different structure from the spacers, or (b) the same structure that is used as a spacer.
  • An apparatus for improving image-taking of a micro-feature in a sample comprising:
  • CB-2 A system for improving image-taking of a micro-feature in a sample, the system comprising: i. a device of any prior device embodiment;
  • iii a non-transitory computer readable medium having instructions that, when executed, utilize the mark as a parameter together with an imaging processing method to adjust the setting of the imager for the next image.
  • CC-1 An apparatus for improving analysis of an image of a micro-feature in a sample, the apparatus comprising:
  • a computation device being used in receiving an image of a mark and a sample that comprises or is suspected to comprise a micro-feature
  • the computation device runs an algorithm that utilizes the mark as a parameter together with an imaging processing method to improve the image quality in the image.
  • a system for improving analysis of images of a micro-feature in a sample comprising:
  • an imager being used in assaying a sample of comprises or is suspected to comprise a micro-feature by taking one or multiple images of the sample and the mark;
  • (c) a non-transitory computer readable medium having instructions that, when executed, utilize the mark as a parameter together with an imaging processing method to improve the image quality in at least one image taken in (c).
  • CC-3 A computer program product for assaying a micro-feature in a sample, the program comprising computer program code means applied and adapted for, in at least one image:
  • CC-4 A computing devices for assaying a micro-feature in a sample, the computation device comprising a computing device that operate the algorithms in any of embodiments of the present invention.
  • CC-5 The method, device, computer program product, or system of any prior embodiment, wherein the improvement of the image quality comprises at least one selected from the group consisting of denoising, image normalization, image sharpening, image scaling, alignment (e.g., for face detection), super resolution, deblurring, and any combination of thereof.
  • the imaging processing method comprises at least one selected from the group consisting of a histogram-based operation, a mathematics-based operation, a convolution-based operation, a smoothing operation, derivative-based operation, a morphology-based operation, shading correction, image enhancement and/or restoration, segmentation, feature extraction and/or matching, object detection and/or classification and/or localization, image understanding, and any combination of thereof.
  • CC-6.1 The method, device, computer program product, or system of any prior embodiment, wherein the histogram-based operation comprises at least one selected from the group consisting of contrast stretching, equalization, minimum filtering, median filtering, maximum filtering, and any combination thereof.
  • CC-6.2 The method, device, computer program product, or system of any prior embodiment, wherein the mathematics-based operation comprises at least one selected from the group consisting of binary operation (e.g., NOT, OR, AND, XOR, and SUB) arithmetic-based operations (e.g., ADD, SUB, MUL, DIV, LOG, EXP, SORT, TRIG, and INVERT), and any combination thereof.
  • CC-6.3 The method, device, computer program product, or system of any prior embodiment, wherein the convolution-based operation comprises at least one selected from the group consisting of an operation in the spatial domain, Fourier transform, DCT, integer transform, an operation in the frequency domain, and any combination thereof.
  • CC-6.4 The method, device, computer program product, or system of any prior embodiment, wherein the smoothing operation comprises at least one selected from the group consisting of a linear filter, a uniform filter, a triangular filter, a Gaussian filter, a non-linear filter, a medial filter a kuwahara filter, and any combination thereof.
  • the derivative-based operation comprises at least one selected from the group consisting of a first-derivative operation, a gradient filter, a basic derivative filter, a Prewitt gradient filters, a Sobel gradient filter, an alternative gradient filter, a Gaussian gradient filter, a second derivative filter, a basic second derivative filter, a frequency domain Laplacian, a Gaussian second derivative filter, an Alternative Laplacian filter, a Second-Derivative-in-the-Gradient-Direction (SDGD) filter, a third derivative filter, a higher derivative filter (e.g., a greater than third derivative filter), and any combination thereof.
  • SDGD Second-Derivative-in-the-Gradient-Direction
  • CC-6.6 The method, device, computer program product, or system of any prior embodiment, wherein the morphology-based operation comprises at least one selected from the group consisting of dilation, erosion, Boolean convolution, opening and/or closing, hit-and-miss operation, contour, skeleton, propagation, gray-value morphological processing, Gray-level dilation, gray-level erosion, gray-level opening, gray-level closing, morphological smoothing, morphological gradient, morphological Laplacian, and any combination thereof.
  • CC-6.7 The method, device, computer program product, or system of any prior embodiment, wherein the image enhancement and/or restoration comprises at least one selected from the group consisting of sharpening, noise suppression, distortion suppression, and any combination thereof.
  • CC-6.8 The method, device, computer program product, or system of any prior embodiment, wherein the segmentation comprises at least one selected from the group consisting of thresholding, fixed thresholding, Histogram-derived thresholding, ISO data algorithm, background-symmetry algorithm, Triangle algorithm, Edge finding, Gradient-based procedure, zero-crossing based procedure, PLUS-based procedure, Binary mathematical morphology, salt- or-pepper filtering, Isolate objects with holes, filling holes in objects, removing border-touching objects, Exo-skeleton, Touching objects, Gray-value mathematical morphology, Top-hat transform, thresholding, Local contrast stretching, and any combination thereof.
  • the segmentation comprises at least one selected from the group consisting of thresholding, fixed thresholding, Histogram-derived thresholding, ISO data algorithm, background-symmetry algorithm, Triangle algorithm, Edge finding, Gradient-based procedure, zero-crossing based procedure, PLUS-based procedure, Binary mathematical morphology, salt- or-pepper filtering, Isolate objects with holes, filling holes in objects, removing border-
  • CC-6.9 The method, device, computer program product, or system of any prior embodiment, wherein the feature extraction and/or matching comprises at least one selected from the group consisting of Independent component analysis, Isomap, Kernel Principal Component Analysis, Latent semantic analysis, Partial least squares, Principal component analysis, Multifactor dimensionality reduction, Nonlinear dimensionality reduction, Multilinear principal component Analysis, Multilinear subspace learning, Semidefinite embedding, Autoencoder, and any combination thereof.
  • Independent component analysis Isomap
  • Kernel Principal Component Analysis Latent semantic analysis
  • Partial least squares Principal component analysis
  • Multifactor dimensionality reduction Multifactor dimensionality reduction
  • Nonlinear dimensionality reduction Multilinear principal component Analysis
  • Multilinear subspace learning Multilinear subspace learning
  • Semidefinite embedding Autoencoder, and any combination thereof.
  • T1 A method for determining, from a distorted image, a true-lateral-dimension (TLD) of a sample on a sample holder, the method comprising:
  • the algorithm is a computer code that is executed on a computer system
  • the algorithm uses an image of the monitoring marks as parameters.
  • each monitoring mark comprises either a protrusion or a trench from the solid-phase surface.
  • T3 The method, device, computer program product, or system of any prior embodiment, wherein the microstructure does not have a sharp edge.
  • T4 The method, device, computer program product, or system of any prior embodiment, wherein the sample is selected from the group consisting of a biological sample, a chemical sample, and a sample that does not have a sharp edge.
  • T5. The method, device, computer program product, or system of any prior embodiment, wherein the monitoring mark is used as a parameter together with an imaging processing method in an algorithm that (i) adjusting the imagine, (ii) processing an image of the sample, (iii) determining a property related to the micro-feature, or (iv) any combination of the above.
  • T6 The method, device, computer program product, or system of any prior embodiment, wherein the spacers have a substantially uniform height that is equal to or less than 200 microns, and a fixed inter-spacer-distance (ISD);
  • ISD inter-spacer-distance
  • a method for improving an imaging of a thin layer of sample comprising:
  • the algorithm is a computer code that is executed on a computer system
  • the algorithm uses an image of the monitoring marks as parameters.
  • T1 A method for determining, from a distorted image, a true-lateral-dimension (TLD) of a sample on a sample holder, the method comprising:
  • the device comprises one or more monitoring marks in the sample contact area
  • the algorithm is a computer code that is executed on a computer system
  • a device for assaying a micro-feature in a sample using an imager the device
  • ii. are inside the sample during an assaying the microstructure, wherein the sample forms, on the sample contact area, a thin layer of a thickness less than 200 urn;
  • iii. have their lateral linear dimension of about 1 urn (micron) or larger, and iv. have at least one lateral linear dimension of 300 urn or less; and wherein during the assaying at least one monitoring mark is imaged by the imager wherein used during assaying the analyte; and a geometric parameter (e.g. shape and size) of the monitoring mark, and/or a pitch between monitoring marks are (a) predetermined and known prior to assaying of the analyte, and (b) used as a parameter in an algorithm that determines a property related to the micro-feature.
  • a geometric parameter e.g. shape and size
  • a device for assaying a micro-feature in a sample using an imager the device
  • a solid-phase surface comprising a sample contact area for contacting a sample which comprises a micro-feature
  • each monitoring mark comprises either a
  • the protrusion or the trench comprises a flat surface that is substantially parallel to a neighbor surface that is a portion of the solid-phase surface adjacent the protrusion or the trench;
  • a distance between the flat surface and the neighboring surface is about 200 micron (urn) or less;
  • the flat surface an area that has (a) a linear dimension is at least about 1 urn or larger, and (b) at least one linear dimension 150 urn or less;
  • a shape of the flat surface, a dimension of the flat surface, a distance between the flat surface and the neighboring surface, and/or a pitch between monitoring marks are (a) predetermined and known prior to assaying of the micro-feature, and (b) used as a parameter in an algorithm that determines a property related to the micro-feature.
  • a device for assaying a micro-feature in a sample using an imager comprising: a first plate, a second plate, spacers, and one or more monitoring marks, wherein:
  • each of the first plate and the second plate comprises an inner surface
  • one or both of the first plate and the second plate comprises the spacers that are permanently fixed on the inner surface of a respective plate, (d) the spacers have a substantially uniform height that is equal to or less than 200 microns, and a fixed inter-spacer-distance (ISD);
  • the monitoring marks are made of a different material from the sample
  • microstructure wherein the sample forms, on the sample contact area, a thin layer of a thickness less than 200 urn;
  • the monitoring marks have their lateral linear dimension of about 1 urn (micron) or larger, and have at least one lateral linear dimension of 300 urn or less;
  • At least one monitoring mark is imaged by the imager wherein used during assaying the micro-feature; and a shape of the flat surface, a dimension of the flat surface, a distance between the flat surface and the neighboring surface, and/or a pitch between monitoring marks are (a) predetermined and known prior to assaying of the micro feature, and (b) used as a parameter in an algorithm that determines a property related to the micro-feature;
  • one of the configurations is an open configuration, in which: the two plates are partially or completely separated apart, the spacing between the plates is not regulated by the spacers, and the sample is deposited on one or both of the plates;
  • another of the configurations is a closed configuration which is configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying the imprecise pressing force on the force area; and in the closed configuration: at least part of the sample is compressed by the two plates into a layer of highly uniform thickness and is substantially stagnant relative to the plates, wherein the uniform thickness of the layer is confined by the sample contact areas of the two plates and is regulated by the plates and the spacers; and
  • a monitoring mark is (i) a different structure from the spacers, or (ii) the same structure that is used as a spacer.
  • a device for assaying a micro-feature in a sample using an imager the device
  • each of the first plate and the second plate comprises an inner surface comprising a sample contact area for contacting a sample that comprises a micro-feature;
  • one or both of the first plate and the second plate comprises the spacers that are permanently fixed on the inner surface of a respective plate,
  • the spacers have a substantially uniform height that is equal to or less than 200 microns, and a fixed inter-spacer-distance (ISD);
  • each monitoring mark comprises either a protrusion or a trench on one or both of the sample contact areas
  • the protrusion or the trench comprises a flat surface that is substantially parallel to a neighbor surface that is a portion of the solid-phase surface adjacent the protrusion or the trench;
  • a distance between the flat surface and the neighboring surface is about 200 micron (urn) or less;
  • the flat surface an area that has (a) a linear dimension is at least about 1 urn or larger, and (b) at least one linear dimension 150 urn or less;
  • a shape of the flat surface, a dimension of the flat surface, a distance between the flat surface and the neighboring surface, and/or a pitch between monitoring marks are (a) predetermined and known prior to assaying the micro-feature, and (b) used as a parameter in an algorithm that determines a property related to the micro-feature
  • one of the configurations is an open configuration, in which: the two plates are partially or completely separated apart, the spacing between the plates is not regulated by the spacers, and the sample is deposited on one or both of the plates;
  • another of the configurations is a closed configuration which is configured after the sample is deposited in the open configuration and the plates are forced to the closed configuration by applying the imprecise pressing force on the force area; and in the closed configuration: at least part of the sample is compressed by the two plates into a layer of highly uniform thickness and is substantially stagnant relative to the plates, wherein the uniform thickness of the layer is confined by the sample contact areas of the two plates and is regulated by the plates and the spacers; and
  • a monitoring mark is (i) a different structure from the spacers, or (ii) the same structure that is used as a spacer.
  • a device for imaging based assay comprising: a device of any prior device embodiment, wherein the device has at least four monitoring marks not aligned on a linear line.
  • An apparatus for assaying a micro-feature in a sample using an imager comprising:
  • a system for performing an imaging-based assay comprising:
  • a system for assaying a micro-feature in a sample using an imager comprising: (a) a device of any prior device embodiment;
  • a non-transitory computer readable medium comprising instructions that, when executed, utilize monitoring marks of the device to assay a property related to the micro-feature, wherein the instructions comprise machine learning.
  • a method for assaying a micro-feature in a sample using an imager comprising:
  • a method for assaying a micro-feature in a sample using an imager comprising:
  • T1 A method for determining, from a distorted image, a true-lateral-dimension (TLD) of a sample on a sample holder, the method comprising:
  • the algorithm is a computer code that is executed on a computer system
  • the algorithm uses an image of the monitoring marks as parameters.
  • T2 A method for determining, from a distorted image, the true-lateral-dimension (TLD) of a sample on a sample holder, the method comprising:
  • the algorithm is a computer code that is executed on a computer system
  • the algorithm uses an image of the monitoring marks as parameters.
  • T3 The device, method, or system of any prior embodiment, wherein micro-features from the sample and monitoring marks are disposed within the sample holding device.
  • the determining comprises detecting and locating the monitoring marks in the image of the sample taken by the imager.
  • T5 The device, method, or system of any prior embodiment, wherein the determining comprises generating a monitoring mark grid based on the monitoring marks detected from the image of the sample taken by the imager.
  • T6 The device, method, or system of any prior embodiment, wherein the determining comprises calculating a homographic transform from the generated monitoring mark grid.
  • the determining comprises estimating the TLD from the homographic transform, and determining the area, size, and concentration of the detected micro- features in the image-based assay.
  • T8 The method, device or system of any prior embodiment, wherein the TLD estimation is based on regions in a sample image taken by the imager, comprising:
  • a sample holding device e.g. QMAX device, wherein there are monitoring marks, wherein the monitoring marks are not submerged in the sample and reside in the device that can be imaged from the top by an imager in the image- based assay;
  • T9 The method, device or system of any prior embodiment, wherein the monitoring marks in the sample holding device are distributed according to a periodic pattern with a defined pitch period.
  • T10 The method, device or system of any prior embodiment, wherein the said monitoring marks are detected and applied as detectable anchors for calibration and improving the measurement accuracy in the image-based assay.
  • T11 The method, device or system of any prior embodiment, wherein the detection of the monitoring marks in the sample image taken by the imager utilizes the periodicity of the monitoring mark distribution in the sample holding device for error correction and/or the reliability of the detection.
  • identification, area and/or shape contour estimation of the said monitoring marks in image- based assay are through machine learning (ML) with ML based monitoring mark detection models and apparatus built or trained from the image taken by the imager on the said device in the image-based assay.
  • ML machine learning
  • T 14 The method, device or system of any prior embodiment, wherein the detected monitoring marks are applied to TLD estimation in the image-based assay to calibrate the system and/or improve the measurement accuracy in the imaged-based assay.
  • T15 The method, device or system of any prior embodiment, wherein the detected monitoring marks are applied and not limited to micro-feature size, volume and/or concentration estimation in image-based assay to calibrate the system and/or improve the measurement accuracy.
  • T16 The method, device or system of any prior embodiment, wherein the detection of the monitoring marks and/or TLD estimation are applied to the fault detection in image-based assay, including and not limited to detecting defects in the sample holding device, mis-placement of the sample holding device in the imager, and/or the focusing fault of the imager.
  • T17 The method, device or system of any prior embodiment, wherein the said monitoring marks are detected as anchors to apply in a system to estimate the area of an object in image-based assay, comprising:
  • detecting the monitoring marks in the image of the sample taken by the imager on the sample holding device determine the TLD and calculate the area estimation in the image-based assay to determine the size of the imaged object from pixels in the image to its physical size of micrometers in the real world.
  • T18 The method, device or system of any prior embodiment, wherein the system comprises: i. detecting the monitoring mark in a digital image;
  • iii calculating the image transform based on the monitoring mark grid; and iv. estimating the area of the object in image of the sample and its physical size in the real world in image-based assay.
  • T19 The method, device or system of any prior embodiment, wherein the generated monitoring mark grid from the detected monitoring marks is used to calculate a homographic transform to estimate TLD, the area of the object in the image of the sample taken by the imager, and the physical size of the object in the real world.
  • T20 The method, device or system of any prior embodiment, wherein the method comprises: i. partitioning the image of the sample taken by the imager in image-based assay into nonoverlapping regions;
  • T21 The method, device or system of any prior embodiment, wherein the assay is a medical, a diagnostic, a chemical or a biological test.
  • T22 The method, device or system of any prior embodiment, wherein said micro-feature is a cell.
  • T23 The method, device or system of any prior embodiment wherein said micro-features is a blood cells.
  • T24 The method, device or system of any prior embodiment wherein said micro-feature is a protein, peptide, DNA, RNA, nucleic acid, small molecule, cell, or nanoparticle.
  • T25 The method, device or system of any prior embodiment wherein said micro-feature comprises a label.
  • annotated data set i. feeding an annotated data set to a convolutional neural network, wherein the annotated data set is from samples that are the same type as the test sample and for the same micro-feature;
  • inference pattern in the image of the sample and generate the contour masks enclosing the object identified by inference module where the object is in focus, (c) identifying at least one portion of the image of the sample for at least one object in the selected portion of the image of the sample, and
  • the method, device or system of any prior embodiment further comprises computer readable storage medium or memory storage unit comprising a computer program of any prior embodiment.
  • the method, device or system of any prior embodiment further comprises a computing arrangement or mobile apparatus comprising the calculation device of any prior embodiment.
  • the method, device or system of any prior embodiment further comprises a computing arrangement or mobile apparatus comprising the computer program product of any prior embodiment.
  • the method, device or system of any prior embodiment further comprises a computing arrangement or mobile apparatus comprising the computer readable storage medium or storage unit of any prior embodiment.
  • a device for analyzing a sample comprising:
  • the surface amplification layer is on one of the sample contact areas
  • the capture agent is immobilized on the surface amplification layer, wherein the capture agent specifically binds the target analyte
  • the surface amplification layer amplifies an optical signal from the target analyte or a label attached to the target analyte when they are is in proximity of the surface amplification layer much stronger than that when they are micron or more away,
  • one of the configurations is an open configuration, in which the average spacing between the inner surfaces of the two plates is at least 200 urn;
  • another of the configurations is a close configuration, in which, at least part of the sample is between the two plates and the average spacing between the inner surfaces of the plates is less than 200 urn.
  • a device for analyzing a sample comprising:
  • the surface amplification layer is on one of the sample contact areas
  • the capture agent is immobilized on the surface amplification layer, wherein the capture agent specifically binds the target analyte
  • the surface amplification layer amplifies an optical signal from a label attached to the target analyte when it is in proximity of the surface amplification layer much stronger than that when it is micron or more away,
  • one of the configurations is an open configuration, in which the average spacing between the inner surfaces of the two plates is at least 200 urn;
  • another of the configurations is a close configuration, in which, at least part of the sample is between the two plates and the average spacing between the inner surfaces of the plates is less than 200 urn; wherein the thickness of the sample in the closed configuration, the concentration of the labels dissolved in the sample in the closed configuration, and the amplification factor of the surface amplification layer are configured such that any the labels that are bound directly or indirectly to the capture agents are visible in the closed configuration without washing away of the unbound labels.
  • An apparatus comprising a device of any prior embodiment and a reader for reading the device.
  • a homogeneous assay method using a device of any prior embodiment wherein the thickness of the sample in a closed configuration, the concentration of labels, and amplification factor of the amplification surface are configured to make the label(s) bound on the amplification surface visible without washing away of the unbound labels.
  • the method is a homogeneous assay in which the signal is read without using a wash step to remove any biological materials or labels that are not bound to the amplification surface.
  • the device or method of any prior embodiment wherein the labels bound to the amplification surface are read by a lump-sum reading method.
  • the signal amplification layer comprises a D2PA.
  • the signal amplification layer comprises a layer of metallic material.
  • the signal amplification layer comprises a continuous metallic film that is made of a material selected from the group consisting of gold, silver, copper, aluminum, alloys thereof, and combinations thereof.
  • the signal amplification layer comprises a layer of metallic material and a dielectric material on top of the metallic material layer, wherein the capture agent is on the dielectric material.
  • the metallic material layer is a uniform metallic layer, nanostructured metallic layer, or a combination.
  • assay comprises detecting the labels by Raman scattering.
  • capture agent is an antibody.
  • the capture agent is a polynucleotide.
  • the device further comprise spacers fixed on one of the plate, wherein the spacers regulate the spacing between the first plate and the second plate in the closed configuration.
  • the amplification factor of the surface amplification layer is adjusted to make the optical signal from a single label that is bound directly or indirectly to the capture agents visible.
  • the amplification factor of the surface amplification layer is adjusted to make the optical signal from a single label that is bound directly or indirectly to the capture agents visible, wherein the visible single labels bound to the capture agents are counted individually.
  • the spacing between the first plate and the second plate in the closed configuration is configured to make saturation binding time of the target analyte to the capture agents 300 sec or less.
  • the spacing between the first plate and the second plate in the closed configuration is configured to make saturation binding time of the target analyte to the capture agents 60 sec or less.
  • the amplification factor of the surface amplification layer is adjusted to make the optical signal from a single label visible.
  • the capture agent is a nucleic acid.
  • the capture agent is a protein.
  • the capture agent is an antibody.
  • sample contact area of the second plate has a reagent storage site, and the storage site is approximately above the binding site on the first plate in the closed configuration.
  • the reagent storage site comprises a detection agent that binds to the target analyte.
  • the detection agent comprises the label
  • the signal amplification layer comprises a layer of metallic material.
  • the signal amplification layer comprises a layer of metallic material and a dielectric material on top of the metallic material layer, wherein the capture agent is on the dielectric material.
  • the metallic material layer is a uniform metallic layer, nanostructured metallic layer, or a combination.
  • the amplification layer comprises a layer of metallic material and a dielectric material on top of the metallic material layer, wherein the capture agent is on the dielectric material, and the dielectric material layer has a thickness of 0.5 nm, 1 nm, 5 nm, 10 nm, 20 nm, 50 nm, 00 nm, 200 nm, 500 nm, 1000 nm, 2um, 3um, 5um, 10 urn, 20 urn, 30 urn, 50 urn, 100 urn, 200 urn, 500 urn, or in a range of any two values.
  • the method further comprises quantifying a signal in an area of the image to providing an estimate of the amount of one or more analytes in the sample.
  • the method comprises identifying and counting individual binding events between an analyte with the capture agent in an area of the image, thereby providing an estimate of the amount of one or more analytes in the sample.
  • identifying and counting steps comprise: (1) determining the local intensity of background signal, (2) determining local signal intensity for one label, two labels, three labels, and four or more labels; and (3) determining the total number of labels in the imaged area.
  • the identifying and counting steps comprises: (1) determining the local spectrum of background signal, (2) determining local signal spectrum for one label, two labels, three labels, and four or more labels; and (3) determining the total number of labels in the imaged area.
  • the identifying and counting steps comprise: (1) determining the local Raman signature of background signal, (2) determining local signal Raman signature for one label, two labels, three labels, and four or more labels; and (3) determining the total number of labels in the imaged area.
  • the identifying and counting step comprises determining one or more of the local intensities, spectrum, and Raman signatures.
  • the method comprises quantifying a lump-sum signal in an area of the image, thereby providing an estimate of the amount of one or more analytes in the sample.
  • the sample contact area of the second plate has a reagent storage site, and the storage site is, in a closed configuration, approximately above the binding site on the first plate.
  • the method further comprises a step of labeling the target analyte with a detection agent.
  • the detection agent comprises a label
  • the method further comprises measuring the volume of the sample in the area imaged by the reading device.
  • the target analyte is a protein, peptide, DNA, RNA, nucleic acid, small molecule, cell, or nanoparticle.
  • the signals are luminescence signals selected from the group consisting of fluorescence, electroluminescence,
  • the signals are the forces due to local electrical, local mechanical, local biological, or local optical interaction between the plate and the reading device.
  • inter spacer distance SD is equal or less than about 120 urn (micrometer).
  • inter spacer distance SD is equal or less than about 100 urn (micrometer).
  • the fourth power of the inter- spacer-distance (ISD) divided by the thickness (h) and the Young’s modulus (E) of the flexible plate (ISD 4 /(hE)) is 5x10 6 um 3 /GPa or less.
  • the fourth power of the inter- spacer-distance (ISD) divided by the thickness (h) and the Young’s modulus (E) of the flexible plate (ISD 4 /(hE)) is 5x10 5 um 3 /GPa or less.
  • the spacers have pillar shape, a substantially flat top surface, a predetermined substantially uniform height, and a predetermined constant inter-spacer distance that is at least about 2 times larger than the size of the analyte, wherein the Young’s modulus of the spacers times the filling factor of the spacers is equal or larger than 2 MPa, wherein the filling factor is the ratio of the spacer contact area to the total plate area, and wherein, for each spacer, the ratio of the lateral dimension of the spacer to its height is at least 1 (one).
  • the spacers have pillar shape, a substantially flat top surface, a predetermined substantially uniform height, and a predetermined constant inter-spacer distance that is at least about 2 times larger than the size of the analyte, wherein the Young’s modulus of the spacers times the filling factor of the spacers is equal or larger than 2 MPa, wherein the filling factor is the ratio of the spacer contact area to the total plate area, and wherein, for each spacer, the ratio of the lateral dimension of the spacer to its height is at least 1 (one), wherein the fourth power of the inter-spacer-distance (ISD) divided by the thickness (h) and the Young’s modulus (E) of the flexible plate (ISD 4 /(hE)) is 5x10 6 um 3 /GPa or less.
  • ISD inter-spacer-distance
  • E Young’s modulus
  • analytes is proteins, peptides, nucleic acids, synthetic compounds, or inorganic compounds.
  • the sample is a biological sample selected from amniotic fluid, aqueous humour, vitreous humour, blood (e.g., whole blood, fractionated blood, plasma or serum), breast milk, cerebrospinal fluid (CSF), cerumen (earwax), chyle, chime, endolymph, perilymph, feces, breath, gastric acid, gastric juice, lymph, mucus (including nasal drainage and phlegm), pericardial fluid, peritoneal fluid, pleural fluid, pus, rheum, saliva, exhaled breath condensates, sebum, semen, sputum, sweat, synovial fluid, tears, vomit, and urine.
  • blood e.g., whole blood, fractionated blood, plasma or serum
  • CSF cerebrospinal fluid
  • cerumen earwax
  • chyle chime
  • endolymph perilymph
  • perilymph perilymph
  • feces breath
  • the spacers have a shape of pillars and a ratio of the width to the height of the pillar is equal or larger than one.
  • the spacers have a shape of pillar, and the pillar has substantially uniform cross-section.
  • samples is for the detection, purification and quantification of chemical compounds or biomolecules that correlates with the stage of certain diseases.
  • samples is related to infectious and parasitic disease, injuries, cardiovascular disease, cancer, mental disorders, neuropsychiatric disorders, pulmonary diseases, renal diseases, and other and organic diseases.
  • samples is related to virus, fungus and bacteria from environment, e.g., water, soil, or biological samples.
  • samples is related to the detection, quantification of chemical compounds or biological samples that pose hazard to food safety or national security, e.g. toxic waste, anthrax.
  • samples is related to glucose, blood, oxygen level, total blood count.
  • samples is related to the detection and quantification of specific DNA or RNA from biosamples.
  • samples is related to the sequencing and comparing of genetic sequences in DNA in the chromosomes and mitochondria for genome analysis.
  • samples is cells, tissues, bodily fluids, and stool.
  • sample is the sample in the fields of human, veterinary, agriculture, foods, environments, and drug testing.
  • sample is a biological sample is selected from hair, finger nail, ear wax, breath, connective tissue, muscle tissue, nervous tissue, epithelial tissue, cartilage, cancerous sample, or bone.
  • inter-spacer distance is in the range of 5 urn to 120 urn.
  • inter-spacer distance is in the range of 120 urn to 200 urn.
  • the flexible plates have a thickness in the range of 20 urn to 250 urn and Young’s modulus in the range 0.1 to 5 GPa.
  • the thickness of the flexible plate times the Young’s modulus of the flexible plate is in the range 60 to 750 GPa-um.
  • the layer of uniform thickness sample is uniform over a lateral area that is at least 1 mm 2 .
  • the layer of uniform thickness sample is uniform over a lateral area that is at least 3 mm 2 .
  • the layer of uniform thickness sample is uniform over a lateral area that is at least 5 mm 2 .
  • the layer of uniform thickness sample is uniform over a lateral area that is at least 10 mm 2 .
  • the layer of uniform thickness sample is uniform over a lateral area that is at least 20 mm 2 .
  • the layer of uniform thickness sample has a thickness uniformity of up to +1-5% or better.
  • the layer of uniform thickness sample has a thickness uniformity of up to +/-10% or better.
  • the layer of uniform thickness sample has a thickness uniformity of up to +/- 20% or better.
  • the layer of uniform thickness sample has a thickness uniformity of up to +/- 30% or better.
  • the method, device, computer program product, or system of any prior embodiment having five or more monitoring marks, wherein at least three of the monitoring marks are not in a straight line.
  • each of the plates comprises, on its respective outer surface, a force area for applying an imprecise pressing force that forces the plates together;
  • the algorithm is stored on a non-transitory computer-readable medium, and wherein the algorithm comprises instructions that, when executed, perform a method that utilizes monitoring marks of the device to determine a property corresponding to the analyte.
  • the marks have the same shapes as the spacers.
  • the marks is periodic or aperiodic.
  • the distance between two marks are predetermined and known, but the absolution coordinates on a plate are unknown.
  • the marks have predetermined and know shapes.
  • the marks is configured to have a distribution in a plate, so that regardless the position of the plate, there are always the marks in the field of the view of the imaging optics.
  • the marks is configured to have a distribution in a plate, so that regardless the position of the plate, there are always the marks in the field of the view of the imaging optics and that the number of the marks are sufficient to for local optical information.
  • the marks are used to control the optical properties of a local area of the sample, whereas the area size is 1 um A 2, 5 um A 2, 10 um A 2, 20 um A 2, 50 um A 2, 100 um A 2, 200 um A 2, 500 um A 2, 1000 um A 2, 2000 um A 2, 5000 um A 2, 10000 um A 2, 100000 um A 2, 500000 um A 2, or a range between any of two values.
  • the optical system for imaging the assay have“limited imaging optics”.
  • Some embodiments of limited imaging optics include, but not limited to:
  • the limited imaging optics system comprising:
  • the imaging sensor is a part of the camera of a smartphone; wherein at least one of the imaging lenses is a part of the camera of smartphone;
  • resolution by physics is worse than 1 um, 2um, 3um, 5um, 10um, 50um, or in a range between any of the two values.
  • resolution per physics is worse than 1 um, 2um, 3um, 5um, 10um, 50um, or in a range between any of the two values.
  • optical resolution per physics is between 1 um and 3um;
  • the numerical aperture is less than 0.1 , 0.15, 0.2, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, or in a range between any of the two values.
  • numerical aperture is between 0.2 and 0.25.
  • distance is 0.2mm, 0.5mm, 1 mm, 2mm, 5mm, 10mm, 20mm, or in a range between any of the two values.
  • distance is 0.2mm, 0.5mm, 1 mm, 2mm, 5mm, 10mm, 20mm, or in a range between any of the two values.
  • the working distance is between 0.5mm to 1 mm. 10.
  • the focal depth is 100nm, 500nm, 1 um, 2um, 10um, 100um, 1mm, or in a range between any of the two values.
  • the focal depth is 100nm, 500nm, 1 um, 2um, 10um, 100um, 1mm, or in a range between any of the two values.
  • the diagonal length of the image sensor is less than 1 inch, 1/2 inch, 1/3 inch, 1 ⁇ 4 inch, or in a range between any of the two values;
  • the imaging lenses comprises at least two lenses, and one lens is a part of the camera module of a smartphone.
  • the optical axis of external lens is aligned with the with the internal lens of smartphone, the alignment tolerance is less than 0.1mm, 0.2mm, 0.5mm, 1mm, or in a range between any of the two values.
  • the height of the external lens is less than 2mm, 5mm, 10mm, 15mm, 20m, or in a range between any of the two values.
  • height of the external lens is between 3mm to 8mm.
  • height of the external lens is between 3mm to 8mm.
  • the diameter of the external lens is less than 2mm, 4mm, 8mm, 10mm, 15mm, 20mm, or in a range between any of the two values.
  • magnification per physics is less than 0.1X, 0.5X, 1X, 2X, 4X, 5X, 10X, or in a range between any of the two values. 22.
  • the preferred optical magnification per physics is less than 0.1X, 0.5X, 1X, 2X, 4X, 5X, 10X, or in a range between any of the two values.
  • the sample position system for imaging the assay have“limited sample manipulation”. Some embodiments of limited sample
  • the limited sample manipulation system comprising:
  • sample holder has a receptacle for taking in the sample card.
  • An example of sharp edge is the edge of a pillar spacer that has a flat top and nearly 90-degree sidewall.
  • An Example of the objects that do not have a sharp edge is spheres.
  • Fig. 7-A shows an embodiment of the sample holding device, QMAX device, and its monitor marks, pillars, used in some embodiments of the present invention. Pillars in QMAX device make the gap between two parallel plates of the sample holding device uniform. The gap is narrow and relevant to the size of the analytes where analytes form a monolayer in the gap. Moreover, the monitoring marks in QMAX device are in the special form of pillars, and consequently, they are not submerged by the sample and can be imaged with the sample by the imager in image based assay.
  • the monitor marks are used as detectable anchors.
  • detecting monitoring marks with an accuracy suitable for TLD estimation in image-based assay is difficult. This is because these monitoring marks are permeated and surrounded by the analytes inside the sample holding device. They are distorted and blurred in the image due to the distortion from the lens, light diffraction from microscopic objects in the sample, defects at microscopic level, mis-alignment in focusing, noise in the image of the sample, etc.
  • imagers are cameras from commodity devices (e.g. cameras from smart phones), since those cameras are not calibrated by the dedicated hardware once they left the manufacture.
  • the detection and locating the monitor marks as detectable anchors for TLD estimation is formulated in a machine-learning framework and dedicated machine-learning model is built/trained to detect them in microscopic imaging.
  • the distribution of the monitor marks in some embodiments of the present invention is intentionally made to being periodic and distributed in a predefined pattern. This makes the approach in the present invention more robust and reliable.
  • sample holding device e.g. QMAX device, wherein there are monitor marks with known configuration residing in the device that are not submerged in the sample and can be imaged by an imager;
  • region based TLD estimation and calibration are employed in image-based assay. It comprises:
  • sample holding device e.g. QMAX device
  • monitor marks in the device - not submerged in the sample can be imaged by an imager in the image-based assay
  • monitor marks When the monitor marks are distributed in a pre-defined periodic pattern, such as in QMAX device, they occur and distribute periodically with a certain pitch, and as a result, detection the monitor marks become more robust and reliable in the procedures described above. This is because with periodicity, all monitor marks can be identified and determined from just few detected ones, and detection errors can be corrected and eliminated, should the detected location and configuration do not follow the pre-defined periodic pattern.
  • imager and “camera” are interchangeable in the description of the present invention.
  • denoise refers to a process of removing noise from the received signal.
  • An example is to remove the noise in the image of sample as the image from the imager/camera can pick up noise from various sources, including and not limited to white noise, salt and pepper noise, Gaussian noise, etc.
  • Methods of denoising include and not limited to: linear and non linear filtering, wavelet transform, statistic methods, deep learning etc.
  • image normalization refers to algorithms, methods and apparatus that change the range of pixel intensity values in the processed image. For example, it includes and not limited to increasing the contrast by histogram stretching, subtract the mean pixel value from each image, etc.
  • methods and algorithms are devised to take advantage of the monitoring marks in the sample holding device, e.g. QMAX device. This includes and not limited to the estimation and adjustment of the following parameters in the imaging device:
  • the image processing/analyzing are applied and strengthened with the monitoring marks in the present invention. They include and not limited to the following image processing algorithms and methods:
  • Histogram-based operations include and not limited to:
  • arithmetic-based operations ADD, SUB, MUL, DIV, LOG, EXP, SORT, TRIG, INVERT, etc.
  • Convolution-based operations in both spatial and frequency domain include and not limited to Fourier transform, DOT, Integer transform, wavelet transform, etc.
  • linear filters uniform filter, triangular filter, gaussian filter, etc.
  • non-linear filters medial filter, kuwahara filter, etc.
  • Derivative-based operations include and not limited to:
  • first derivatives gradient filters, basic derivative filters, prewitt gradient filters, sobel gradient filters, alternative gradient filters, gaussian gradient filters, etc.
  • second derivatives basic second derivative filter, frequency domain Laplacian, Gaussian second derivative filter, alternative Laplacian filter, second-derivative- in-the-gradient-direction (SDGD) filter, etc., and c. other filters with higher derivatives, etc.
  • Morphology-based operations include and not limited to:
  • gray-value morphological processing Gray-level dilation, gray-level erosion, gray-level opening, gray-level closing, etc.;
  • image processing/analyzing algorithms are used together with and enhanced by the monitoring marks. They include and not limited to the following:
  • Image enhancement and restoration include and not limited to
  • Image segmentation include and not limited to:
  • thresholding - fixed thresholding, histogram-derived thresholding, Isodata algorithm, background-symmetry algorithm, triangle algorithm, etc.;
  • Feature extraction and matching include and not limited to:
  • monitoring marks in the present invention is used to improve focus in microscopic imaging.
  • monitoring marks with sharp edge will provide detectable (visible features) for the focus evaluation algorithm to analyze the focus conditions of certain focus settings, especially in low lighting environment and in microscopic imaging.
  • focus evaluation algorithm is at the core part in the auto-focus implementations.
  • detectable features provided by the analyte in the image of the sample is often not enough for the focus evaluation algorithm to run accurately and smoothly.
  • Monitoring marks with sharp edges e.g. the monitoring marks in QMAX device, provide additional detectable features for the focus evaluation program to achieve the accuracy and reliability required in the image-based assay.
  • analytes in the sample are distributed unevenly. Purely relying on features provided by analytes tends to generate some unfair focus setting that gives high weight of focusing on some local high concentration regions and low analyte concentration regions are off target. In some embodiments of the current invention, this effect is controlled with the focusing adjustments from the information of the monitoring marks which have strong edges and are distributed evenly with a accurately processed periodic pattern.
  • each imager has an imaging resolution limited in part by the number of pixels in its imager sensor that varies from one million to multimillion pixels.
  • analytes are of small or tiny size in the sample, e.g. the size of platelets in human blood has a dimeter about 1.4um.
  • the limited resolution in the image sensors put a significant constraint on the capability of the device in the image-based assay, in addition to the usable size of FOV, when certain number of pixels is required by the target detection programs.
  • Single Image Super Resolution is a technique to use image processing and/or machine learning techniques to up-sample the original source image to a higher resolution and remove as much blur caused by interpolation as possible, such that the object detection program can run on the newly generated images as well. This will significantly reduce the constraints mentioned above and enable some otherwise impossible applications.
  • Monitoring marks with known shape and structure e.g. the monitor marks in QMAX card
  • image fusion is performed to break the physical SNR (signal-to-noise) limitation in image based assay.
  • Signal to noise ratio measures the quality of the image of the sample taken by the imager in microscopic imaging.
  • an imaging device due to the cost, technology, fabrication, etc.
  • the application requires higher SNR than the regular imaging device can provide.
  • multiple images are taken and processed (with same and/or different imaging setting, e.g. an embodiment of a 3D fusion to merge multiple images focused at different focus depth into one super focused image) to generate output image(s) with higher SNR to make such applications possible.
  • monitoring marks in the sample holding device e.g. the QMAX card, are used for enhanced solutions.
  • One such embodiment is to handle distortions in the image of the sample taken by the imager.
  • the situation is relatively simple if the distortion parameter is known (most manufacture gives a curve/table for their lens to describe ratio distortion, other distortions can be measured in well-defined experiments).
  • the distortion parameters are unknown ( it can change with the focus location and even the sample), with the monitoring marks, a new algorithm can iteratively estimate the distortion parameters using regularly and even periodically placed monitoring marks of the sample holding device (e.g. QMAX card) without requiring a single coordinate references.
  • the sample holding device e.g. QMAX card
  • the sample holding device has a flat surface with some special monitoring marks for the purpose of analyzing the microscopic features in the image-based assay.
  • TLD True-lateral-dimension estimation for the microscopic image of the sample in the image-based assay.
  • TLD True Lateral Dimension
  • the monitoring marks can be used as detectable anchors to determine the TLD and improve the accuracy in the image-based assay.
  • the monitoring marks are detected using machine-learning model, from which the TLD/FoV of the image of the sample is derived. Moreover, if the monitoring marks have a periodic distribution pattern on the flat surface of the sample holding device, the detection of monitoring marks and the per-sample based TLD/FoV estimation can become more reliable and robust in image-based assay.
  • Analyzing the analytes using the measured response from the analyte compound at a specific wavelength of light or at multiple wavelength of light to predict the analyte concentration can be used to determine the light absorption of the background corresponding to zero concentration - to determine the analyte concentration through light absorption, and this approach is to HgB test in present invention.
  • each monitoring mark can act as an independent detector for the background absorption to make the concentration estimatiom robust and reliable.
  • Evenly distributed monitoring marks can be used to improve the focus accuracy
  • Monitoring marks can be used as references to detect and/or correct image imperfection caused by but not limited to: unevenly distributed illumination, various types of image distortions, noises, and imperfect image pre-processing operations.
  • positions of the marks can be used to detect and/or correct the ratio distortion when the straight line in 3D world is mapped in the image of the sample into a curve.
  • Ratio distribution parameters of the entire image can be estimated based on the position changes of the monitoring marks of the sample holding device described herein.
  • the value of ratio distortion parameters can be iteratively estimated by linear testing of horizontal/vertical lines in reproduced images with distortion removal based on assumed ratio distortion parameters.
  • One way of using machine learning is to detect the analytes in the image of the sample and calculate the bounding boxes that covering them for their locations, is performed using trained machine-learning models in the inference process of the processing.
  • Another way of using machine learning method to detect and locate analytes in the image of the sample is to build and train a detection and segmentation model which involving the annotation of the analytes in the sample image at pixel level. In this approach, analytes in the image of the sample can be detected and located with a tight binary pixel masks covering them in image-based assay.
  • a device for biological analyte detection and localization comprising a QMAX device, an imager, and a computing unit.
  • a biological sample is suspected on the QMAX device.
  • the count and location of an analyte contained in the sample are obtain by the disclosure.
  • the imager captures an image of the biological sample.
  • the image is submitted to a computing unit.
  • the computing unit can be physically directly connected to the imager, connected through network, or in-directly through image transfer.
  • the disclosed analyte detection and localization employ machine learning deep learning.
  • a machine learning algorithm is an algorithm that is able to learn from data.
  • a more rigorous definition of machine learning is“A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.” It explores the study and construction of algorithms that can learn from and make predictions on data - such algorithms overcome following strictly static program instructions by making data driven predictions or decisions, through building a model from sample inputs.
  • Deep learning is a specific kind of machine learning based on a set of algorithms that attempt to model high level abstractions in data.
  • the input layer receives an input, it passes on a modified version of the input to the next layer.
  • the layers are not made of neurons but it can help to think of it that way), allowing the algorithm to use multiple processing layers, composed of multiple linear and non-linear transformations.
  • the disclosed analyte detection and localization workflow consists of two stages, training and prediction, as in Fig. 6-A. We describe training and prediction stages in the following paragraphs.
  • Convolutional neural network a specialized kind of neural network for processing data that has a known, grid-like topology. Examples include time-series data, which can be thought of as a 1 D grid taking samples at regular time intervals, and image data, which can be thought of as a 2D grid of pixels. Convolutional networks have been tremendously successful in practical applications. The name“convolutional neural network” indicates that the network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers. Training data are annotated for the analyte to be detect.
  • Annotation indicates whether or not an analyte presents in a training data.
  • Annotation can be done in the form of bounding boxes which fully contains the analyte, or center locations of analytes. In the latter case, center locations are further converted into circles covering analytes.
  • training data When the size of training data is large, it presents two challenges: annotation (which is usually done by person) is time consuming, and the training is computing expensive. To overcome these challenges, one can partition the training data into patches of small size, then annotate and train on these patches, or a portion of these patches.
  • Annotated training data is fed into a convolutional neural network for model training.
  • the output is a model that can be used to make pixel-level prediction on an image.
  • FCN fully convolutional network
  • Other convolutional neural network architecture can also be used, such as TensorFlow.
  • the training stage generates a model that will be used in the prediction stage.
  • the model can be repeatedly used in the prediction stage for input images.
  • the computing unit only needs access to the generated model. It does not need access to the training data, nor the training stage has to be run on the computing unit.
  • a detection component is applied to the input image, which is followed by a localization component.
  • the output of the prediction stage is the count of analytes contained in the sample, along with the location of each analyte.
  • an input image along with the model generated from the training stage, is fed into a convolutional neural network.
  • the output of the detection stage is a pixel-level prediction, in the form of a heatmap.
  • the heatmap can have the same size as the input image, or it can be a scaled down version of the input image.
  • Each pixel in the heatmap has a value from 0 to 1 , which can be considered as the probability (belief) whether a pixel belongs to an analyte. The higher the value, the bigger the chance it belongs to an analyte.
  • the heatmap is the input of the localization component.
  • One embodiment of the localization algorithm is to sort the heatmap values into a one dimensional ordered list, from the highest value to the lowest value. Then pick the pixel with the highest value, remove the pixel from the list, along with its neighbors. Iterate the process to pick the pixel with the highest value in the list, until all pixels are removed from the list.
  • heatmap heatmap ⁇ D // remove D from the heatmap
  • heatmap is a one-dimensional ordered list, where the heatmap value is ordered from the highest to the lowest. Each heatmap value is associated with its corresponding pixel coordinates.
  • the first item in the heatmap is the one with the highest value, which is the output of the pop(heatmap) function.
  • One disk is created, where the center is the pixel coordinate of the one with highest heatmap value.
  • all heatmap values whose pixel coordinates resides inside the disk is removed from the heatmap.
  • the algorithm repeatedly pops up the highest value in the current heatmap, removes the disk around it, till the items are removed from the heatmap.
  • the localization algorithm is complete.
  • the number of elements in the set loci will be the count of analytes, and location information is the pixel coordinate for each s in the set loci.
  • Another embodiment searches local peak, which is not necessary the one with the highest heatmap value. To detect each local peak, we start from a random starting point, and search for the local maximal value. After we find the peak, we calculate the local area surrounding the peak but with smaller value. We remove this region from the heatmap and find the next peak from the remaining pixels. The process is repeated only all pixels are removed from the heatmap.
  • a method of deep learning for data analysis comprising:
  • annotated data set is from samples that are the same type as the test sample and for the same analyte;
  • a system for data analysis comprising:
  • the QMAX device is configured to compress at least part of a test sample into a layer of highly uniform thickness
  • the imager is configured to produce an image of the sample at the layer of uniform thickness, wherein the image includes detectable signals from an analyte in the test sample;
  • the computing unit is configured to:
  • annotated data set is from samples that are the same type as the test sample and for the same analyte
  • a method for improving the reliability of the assay comprising:
  • the error risk factor is one of the following factors or any combination thereof.
  • the factors are, but not limited to, (1) edge of blood, (2) air bubble in the blood, (3) too small blood volume or too much blood volume, (4) blood cells under the spacer, (5) aggregated blood cells, (6) lysed blood cells, (7) over exposure image of the sample, (8) under exposure image of the sample, (8) poor focus of the sample, (9) optical system error as wrong lever position, (10) not closed card, (1 1) wrong card as card without spacer, (12) dust in the card, (13) oil in the card, (14) dirty out of the focus plane one the card, (15) card not in right position inside the reader, (16) empty card, (17) manufacturing error in the card, (18) wrong card for other application, (19) dried blood, (20) expired card, (21) large variation of distribution of blood cells, (22) not blood sample or not target blood sample and others.
  • the error risk analyzer is able to detect, distinguish, classify, revise and/or correct following cases in biological and chemical application in device: (1) at the edge of sample, (2) air bubble in the sample, (3) too small sample volume or too much sample volume, (4) sample under the spacer, (5) aggregated sample, (6) lysed sample, (7) over exposure image of the sample, (8) under exposure image of the sample, (8) poor focus of the sample, (9) optical system error as wrong lever, (10) not closed card, (1 1) wrong card as card without spacer, (12) dust in the card, (13) oil in the card, (14) dirty out of the focus plane one the card, (15) card not in right position inside the reader, (16) empty card, (17) manufacturing error in the card, (18) wrong card for other application, (19) dried sample, (20) expired card, (21) large variation of distribution of blood cells, (22) wrong sample and others.
  • the threshold is determined from a group test.
  • the threshold is determined from machine learning.
  • monitoring marks are used as comparison to identify the error risk factor.
  • monitoring marks are used as comparison to assess the threshold of the error risk factor.
  • Example A1 is a method using an apparatus to improve imaging-based assays.
  • the method may include receiving a sample image of a sample holder comprising a plurality of monitoring structures integrated on a contact surface of at least one plate of the sample holder, wherein the plurality of monitoring structures are placed according to a pattern, and wherein the contact surface contacts a sample that contains a plurality of analytes; detecting, using a machine learning model, the plurality of monitoring structures in the sample image; optionally performing error correction of the detected plurality of monitoring structures using predetermined structural properties associated with the plurality of monitoring structures; determining, based on the detected plurality of monitoring structures, a true-literal-dimension value associated with the sample image; determining, based on the true-literal dimension value, a homographic transform between locations of the detected plurality of monitoring structures in the sample image and a predetermined distribution pattern for the plurality of monitoring structures in an actual image plan associated with the sample holder; transforming, based on the homographic transform, the sample image containing the pluralit
  • Example A3 the method of Example A1 may further provide that the sample image is partitioned into non-overlapping image patches, each of the non-overlapping image patches comprising at least 4 non-colinear detected centers of the plurality of monitoring structures, wherein a patch specific homographic transform is estimated and applied to compensate the said image patch, and for other image patches, a global homographic transform is estimated and applied based on the detected centers of the plurality of monitoring structures in the whole image.
  • the method of Example A1 may further provide that the monitoring structures are pillars or monitoring marks.
  • the pillars may be nanostructures substantially perpendicularly integrated to one or two contract surfaces of the at least one plate of the sample holder.
  • the monitoring marks are marked regions on the one or two contact surfaces.
  • the marked regions may have optical properties (e.g., transparency) different than unmarked regions on the one or two contact surfaces.
  • the marked regions can be surface areas that are painted or engraved a thin layer of nanomaterials while the unmarked regions are not covered by any nanomaterials.
  • Example A5 the method of Example A1 may further provide that the machine learning model is trained using a labeled training image set, and wherein the machine learning model can be a RetinerNet or a convolutional neural network (CNN) whose parameters are trained using the labeled training image set.
  • the machine learning model can be a RetinerNet or a convolutional neural network (CNN) whose parameters are trained using the labeled training image set.
  • CNN convolutional neural network
  • the method of Example A1 may further provide that the predetermined structural properties comprise at least one of a periodicity, a shape, or a size associated with the plurality of monitoring structures.
  • the monitoring structures can be arranged according to an organized pattern.
  • the periodicity may refer the number of monitoring structures within a measurement unit (e.g., a linear measurement such as, for example an inch or a millimeter, or an area measurement such as, for example, a square inch or a square millimeter).
  • the shape may refer to the geometric configuration of each monitoring structure.
  • the monitoring structure can be a cylindrical structure with a cross-section of a triangle, a rectangle, a square, a circle, a polygon, or any suitable 2D shapes.
  • the size may refer to the area of the cross-section of a monitoring structure.
  • the monitoring structures may have substantially identical shape and size.
  • the monitoring structures may have a variety of shapes and sizes while the locations of different monitoring structures are pre-determined during the manufacturing of the sample holder.
  • the method of Example A1 may further provide that the at least one morphological property comprises at least one of a size or a length of one of the plurality of analyte.
  • the size can be an area measurement.
  • the length can be a linear measurement along an axis. For example, for a circle, the length can be the diameter; for a rectangular, the length can be the height, the width, or the diagonal length.
  • Example A8 is a method micro-selective-image-assaying (MSIA) in the image-based assay.
  • the method includes capturing an image of a sample for assaying in the sample holding device, wherein the sample holding device can be the sample holder as described in Example A1 and the sample in the sample holding device comprising a known uniform height and analytes forms a mono-layer in an area of interest; estimating the TLD or FoV of the image of the sample to determine an estimation of area, size and volume in the image-based assaying; detecting defects including air bubbles or dusts in the image of the sample for assaying and segmenting these defects in the image of the sample by a trained machine learning model; estimating total areas of segmented defects in the image of the sample and calculating their actual area size using the estimated TLD/FoV from (b); estimating the actual volume of the sample corresponding to the total surface area of the detected defects in the image of the sample for assaying, according to the area estimate from (d) and the known
  • Example A9 the method of Example A8 may further provide that the micro-selective- image-assaying (MSIA) utilizes the monitoring structures in the sample holding device of A4 to estimate the TLD/FoV of the image of the sample and map the image of the sample to its actual dimensions through an embodiment of A1.
  • MSIA micro-selective- image-assaying
  • Example A10 the method of Example A8 may further provide that the micro-selective- image-assaying (MSIA) is based on other selective criterions including: a) distributions of defects in the sample, including air bubbles and dusts; b) locations of pillars and monitoring marks and other artifacts in the image of the sample for assaying; and c) distribution and the condition of the analytes in the image of the sample for assaying, including conditions of analytes clustering and focusing condition.
  • MSIA micro-selective- image-assaying
  • Example A1 1 the method of Example A8 may further provide that the micro-selective- image-assaying (MSIA) performs multi-target assaying from the image of the sample, in which the multi-target assaying is based on area/zone selection with areas defined by different reagents or sample heights, for multiple assaying applications from one single image of the sample.
  • MSIA micro-selective- image-assaying
  • Example A12 the method of any of Example A8 or A1 1 may further provide that a machine learning model is trained on the labeled training image samples captured by an imager for defects detection and segmentation in the image-based assay.
  • Example A13 the method of any of Example A8 or A1 1 may further provide that a machine learning model is trained to detect the analytes in the image of the sample and determine the size of the detected analytes using the method of Example A1 and the structure of the sample holding device as described in Example A8.
  • Example A14 is a method of monitoring the image-based assaying using the pillars or monitoring marks using the image of the pillar or monitoring marks in the image of the sample for assaying to determine the quality of the image holding device and the quality of the sample preparation, the method including detecting regions corresponding to missing pillars or broken pillars in the sample holding device that affect the effective sample volume for the assaying; and detecting air bubbles and dusts in the image of the sample, indicating flaws in the assaying operation or defects of in the sample holding device.
  • Example A15 the method of Example A14 may further include using the image of the pillars or monitoring marks in the image of the sample for assaying to detect and adjust the operation of the imager in the image-based assaying, including: focusing, contrast stretching, iso adjustment, and filtering.
  • Example BA-1 is an intelligent assay monitor method, the method including receiving, by a processing device, an image encoding first information of a biological sample deposited in a sample card and second information of a plurality of monitor marks; determining, by the processing device executing a first machine learning model on the image, a measurement of a geometric feature associated with the plurality of monitor marks; determining, by the processing device, a variation between the measurement of the geometric feature with a ground truth value of the geometric feature provided with the sample card; correcting, by the processing device based on the variation, the image encoding the first information and the second information; and determining, by the processing device using the corrected image, a biological property of the biological sample.
  • Example BA-2 the method of Example BA-1 may further provide that the sample card comprises a first plate, a plurality of pillars that are substantially perpendicularly integrated to a surface of the first plate, and a second plate capable of enclosing the first plate to form a thin layer in which the biological sample is deposited.
  • the sample card comprises a first plate, a plurality of pillars that are substantially perpendicularly integrated to a surface of the first plate, and a second plate capable of enclosing the first plate to form a thin layer in which the biological sample is deposited.
  • Example BA-3 the method of any of Example BA-1 or BA-2 may further provide that the plurality of monitor marks corresponds to the plurality of pillars.
  • Example BA-4 the method of Example BA-3 may further provide that at least two of the plurality of pillars are separated by a true-lateral-dimension (TLD), and wherein determining, by the processing device executing a first machine learning model on the image, a measurement of a geometric feature associated with the plurality of monitor marks comprises determining, by the processing device executing the first machine learning model on the image, the TLD.
  • TLD true-lateral-dimension
  • Example BB-1 is an image system including a sample card comprising a first plate, a plurality of pillars substantially perpendicularly integrated to a surface of the first plate, and a second plate capable of enclosing the first plate to form a thin layer in which the biological sample is deposited; and a computing device comprising: a processing device, communicatively coupled to an optical sensor, to receive, from the optical sensor, an image encoding first information of a biological sample deposited in the sample card and second information of a plurality of monitor marks, determine, using a first machine learning model on the image, a measurement of a geometric feature associated with the plurality of monitor marks, determine a variation between the measurement of the geometric feature with a ground truth value of the geometric feature provided with the sample card, correct, based on the variation, the image encoding the first information and the second information, and determine, based on the corrected image, a biological property of the biological sample.
  • a processing device communicatively coupled to an optical sensor, to receive, from the optical
  • Example DA-1 is a method for measuring a volume of a sample in a thin-layered sample card, the method including receiving, by a processing device of an image system, an image of a sample card comprising a sample and a monitor standard, wherein the monitor standard comprises a plurality of pillars perpendicularly integrated to a first plate of the sample, and each of the plurality of pillars has a substantially identical height (H); determining, by the processing device using a machine learning model, a plurality of non-sample sub-regions, wherein the plurality of non-sample sub-regions correspond to at least one of a pillar, an air bubble, or an impurity element; calculating, by the processing device, an area occupied by the sample by removing the plurality of non-sample sub-regions from the image; calculating, by the processing device, a volume of the sample based on the calculated area and the height (H); and determining, by the processing device based on the volume, a biological property of the
  • a method of using an apparatus to improve imaging-based assays comprising: a) receiving a sample image of a sample holder comprising a plurality of monitoring structures integrated on a contact surface of at least one plate of the sample holder, wherein the plurality of monitoring structures are placed according to a pattern, and wherein the contact surface contacts a sample that contains a plurality of analytes;
  • A2 The method of A1 , further comprising:
  • determining the homographic transform based on the detected centers of the plurality of monitoring structures comprising of at least 4 non-colinear points.
  • A3 The method of A1 , wherein the sample image is partitioned into non-overlapping image patches, each of the non-overlapping image patches comprising at least 4 non- colinear detected centers of the plurality of monitoring structures, wherein a patch specific homographic transform is estimated and applied to compensate the said image patch, and for other image patches, a global homographic transform is estimated and applied based on the detected centers of the plurality of monitoring structures in the whole image.
  • D1 The method of A1 , wherein the monitoring structures are pillars or monitoring marks.
  • D2 The method of A1 , wherein the machine learning model is trained using a labeled training image set, and wherein the machine learning model is a RetinerNet whose parameters are trained using the labeled training image set.
  • D3 The method of A1 , wherein the predetermined structural properties comprise at least one of a periodicity, a shape, or a size associated with the plurality of monitoring structures.
  • D4 The method of A1 , wherein the at least one morphological property comprises at least one of a size or a length of one of the plurality of analyte.
  • A4 A method of micro-selective-image-assaying (MSIA) in the image-based assay, comprising:
  • A5 A method of A4, wherein the micro-selective-image-assaying (MSIA) utilizes the monitoring structures in the sample holding device of A4 to estimate the TLD/FoV of the image of the sample and map the image of the sample to its actual dimensions through an embodiment of A1.
  • MSIA micro-selective-image-assaying
  • A6 A method of A4, wherein the micro-selective-image-assaying (MSIA) is based on other selective criterions, including:
  • A7 A method of A4, wherein the micro-selective-image-assaying (MSIA) performs multi- target assaying from the image of the sample, in which the multi-target assaying is based on area/zone selection with areas defined by different reagents or sample heights, for multiple assaying applications from one single image of the sample.
  • MSIA micro-selective-image-assaying
  • A8 A method of any of A4 or A7, wherein a machine learning model is trained on the labeled training image samples captured by an imager for defects detection and segmentation in the image-based assay.
  • A9 A method of any of A4 or A7, wherein a machine learning model is trained to detect the analytes in the image of the sample and determine the size of the detected analytes using A1 and the structure of the sample holding device described in A4.
  • A10 A method of monitoring the image-based assaying using the pillars or monitoring marks using the image of the pillar or monitoring marks in the image of the sample for assaying to determine the quality of the image holding device and the quality of the sample preparation, the method including:
  • A12 A method of A10, further comprising using the image of the pillars or monitoring marks in the image of the sample for assaying to detect and adjust the operation of the imager in the image-based assaying, including: focusing, contrast stretching, iso adjustment, and filtering.
  • QMAX quantification; M: magnifying, A. adding reagents, X: acceleration; also termed as self-calibrated compressed open flow (SCOF)
  • SPF compressed open flow
  • sample as used herein relates to a material or mixture of materials containing one or more analytes or entity of interest.
  • the sample may be obtained from a biological sample such as cells, tissues, bodily fluids, and stool.
  • Bodily fluids of interest include but are not limited to, amniotic fluid, aqueous humour, vitreous humour, blood (e.g., whole blood, fractionated blood, plasma, serum, etc.), breast milk, cerebrospinal fluid (CSF), cerumen (earwax), chyle, chime, endolymph, perilymph, feces, gastric acid, gastric juice, lymph, mucus (including nasal drainage and phlegm), pericardial fluid, peritoneal fluid, pleural fluid, pus, rheum, saliva, sebum (skin oil), semen, sputum, sweat, synovial fluid, tears, vomit, urine and exhaled condensate.
  • blood e.g., whole blood, fractionated blood, plasma, serum, etc.
  • CSF cerebrospinal fluid
  • cerumen earwax
  • chyle e.g., chyle
  • chime endolymph
  • a sample may be obtained from a subject, e.g., a human, and it may be processed prior to use in the subject assay.
  • the protein/nucleic acid may be extracted from a tissue sample prior to use, methods for which are known.
  • the sample may be a clinical sample, e.g., a sample collected from a patient.
  • an“analyte” refers to a molecule (e.g., a protein, peptides, DNA, RNA, nucleic acid, or other molecule), cells, tissues, viruses, and nanoparticles with different shapes.
  • an“analyte,” as used herein is any substance that is suitable for testing in the present method.
  • a“diagnostic sample” refers to any biological sample that is a bodily byproduct, such as bodily fluids, that has been derived from a subject.
  • the diagnostic sample may be obtained directly from the subject in the form of liquid, or may be derived from the subject by first placing the bodily byproduct in a solution, such as a buffer.
  • exemplary diagnostic samples include, but are not limited to, saliva, serum, blood, sputum, urine, sweat, lacrima, semen, feces, breath, biopsies, mucus, etc.
  • an“environmental sample” refers to any sample that is obtained from the environment.
  • An environmental sample may include liquid samples from a river, lake, pond, ocean, glaciers, icebergs, rain, snow, sewage, reservoirs, tap water, drinking water, etc.; solid samples from soil, compost, sand, rocks, concrete, wood, brick, sewage, etc.; and gaseous samples from the air, underwater heat vents, industrial exhaust, vehicular exhaust, etc.
  • samples that are not in liquid form are converted to liquid form before analyzing the sample with the present method.
  • a“foodstuff sample” refers to any sample that is suitable for animal consumption, e.g., human consumption.
  • a foodstuff sample may include raw ingredients, cooked food, plant and animal sources of food, preprocessed food as well as partially or fully processed food, etc.
  • samples that are not in liquid form are converted to liquid form before analyzing the sample with the present method.
  • diagnosis refers to the use of a method or an analyte for identifying, predicting the outcome of and/or predicting treatment response of a disease or condition of interest.
  • a diagnosis may include predicting the likelihood of or a predisposition to having a disease or condition, estimating the severity of a disease or condition, determining the risk of progression in a disease or condition, assessing the clinical response to a treatment, and/or predicting the response to treatment.
  • A“biomarker,” as used herein, is any molecule or compound that is found in a sample of interest and that is known to be diagnostic of or associated with the presence of or a predisposition to a disease or condition of interest in the subject from which the sample is derived.
  • Biomarkers include, but are not limited to, polypeptides or a complex thereof (e.g., antigen, antibody), nucleic acids (e.g., DNA, miRNA, mRNA), drug metabolites, lipids, carbohydrates, hormones, vitamins, etc., that are known to be associated with a disease or condition of interest.
  • A“condition” as used herein with respect to diagnosing a health condition refers to a physiological state of mind or body that is distinguishable from other physiological states.
  • a health condition may not be diagnosed as a disease in some cases.
  • Exemplary health conditions of interest include, but are not limited to, nutritional health; aging; exposure to environmental toxins, pesticides, herbicides, synthetic hormone analogs; pregnancy; menopause; andropause; sleep; stress; prediabetes; exercise; fatigue; chemical balance; etc.
  • the terms“adapted” and“configured” mean that the element, component, or other subject matter is designed and/or intended to perform a given function. Thus, the use of the terms“adapted” and“configured” should not be construed to mean that a given element, component, or other subject matter is simply“capable of” performing a given function. Similarly, subject matter that is recited as being configured to perform a particular function may additionally or alternatively be described as being operative to perform that function.
  • phrases“at least one of” and“one or more of,” in reference to a list of more than one entity means any one or more of the entity in the list of entity, and is not limited to at least one of each and every entity specifically listed within the list of entity.
  • “at least one of A and B” (or, equivalently,“at least one of A or B,” or, equivalently,“at least one of A and/or B”) may refer to A alone, B alone, or the combination of A and B.
  • the term“and/or” placed between a first entity and a second entity means one of (1) the first entity, (2) the second entity, and (3) the first entity and the second entity.
  • Multiple entity listed with“and/or” should be construed in the same manner, i.e.,“one or more” of the entity so conjoined.
  • Other entity may optionally be present other than the entity specifically identified by the “and/or” clause, whether related or unrelated to those entities specifically identified.

Abstract

L'invention concerne la correction d'erreurs dans des instruments, des opérations etc. à l'aide de structures de surveillance intelligentes et d'un apprentissage automatique.
EP19860313.6A 2018-08-16 2019-08-16 Dosage à base d'image utilisant des structures de surveillance intelligentes Pending EP3837525A4 (fr)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US201862719129P 2018-08-16 2018-08-16
US201862764887P 2018-08-16 2018-08-16
US201862764886P 2018-08-16 2018-08-16
US201862719201P 2018-08-17 2018-08-17
PCT/US2019/046971 WO2020055543A1 (fr) 2018-08-16 2019-08-16 Dosage à base d'image utilisant des structures de surveillance intelligentes

Publications (2)

Publication Number Publication Date
EP3837525A1 true EP3837525A1 (fr) 2021-06-23
EP3837525A4 EP3837525A4 (fr) 2023-03-08

Family

ID=69776888

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19860313.6A Pending EP3837525A4 (fr) 2018-08-16 2019-08-16 Dosage à base d'image utilisant des structures de surveillance intelligentes

Country Status (4)

Country Link
EP (1) EP3837525A4 (fr)
JP (1) JP2021535369A (fr)
CN (1) CN113227754A (fr)
WO (1) WO2020055543A1 (fr)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114364308A (zh) * 2019-06-02 2022-04-15 Essenlix 公司 通过图像处理和人工智能增强的生物材料的快速染色
WO2022076920A1 (fr) * 2020-10-08 2022-04-14 Essenlix Corporation Réduction d'erreur de dosage
WO2022125594A1 (fr) * 2020-12-07 2022-06-16 Essenlix Corporation Carte de socle et procédés de contrôle et de dosage d'échantillon liquide
KR102509215B1 (ko) * 2021-02-08 2023-03-14 국민대학교산학협력단 인쇄회로기판의 코팅 영역 검출 방법 및 장치
CN113344783B (zh) * 2021-06-08 2022-10-21 哈尔滨工业大学 一种热力图感知的金字塔人脸超分辨率网络
CN116109982B (zh) * 2023-02-16 2023-07-28 哈尔滨星云智造科技有限公司 一种基于人工智能的生物样本采集有效性检验方法
CN117556245B (zh) * 2024-01-04 2024-03-22 信联电子材料科技股份有限公司 一种四甲基氢氧化铵生产过滤杂质检测方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6905881B2 (en) * 2000-11-30 2005-06-14 Paul Sammak Microbead-based test plates and test methods for fluorescence imaging systems
MXPA03010039A (es) * 2001-05-04 2004-12-06 Legend Films Llc Sistema y metodo para mejorar la secuencia de imagen.
GB201105474D0 (en) * 2011-03-31 2011-05-18 Albagaia Ltd Testing apparatus
US8774515B2 (en) * 2011-04-20 2014-07-08 Xerox Corporation Learning structured prediction models for interactive image labeling
US9060685B2 (en) * 2012-03-26 2015-06-23 Nec Laboratories America, Inc. Whole tissue classifier for histology biopsy slides
WO2017027643A1 (fr) * 2015-08-10 2017-02-16 Essenlix Corp. Dispositifs et procédés de dosages biochimiques pour des applications à étapes simplifiées, sur petits échantillons, à vitesse accélérée et faciles à utiliser
WO2018107105A1 (fr) * 2016-12-08 2018-06-14 Essenlix Corporation Cartes et adaptateurs d'échantillons d'analyse et leur utilisation
US9756324B1 (en) * 2017-04-10 2017-09-05 GELT Inc. System and method for color calibrating an image

Also Published As

Publication number Publication date
WO2020055543A1 (fr) 2020-03-19
JP2021535369A (ja) 2021-12-16
CN113227754A (zh) 2021-08-06
EP3837525A4 (fr) 2023-03-08

Similar Documents

Publication Publication Date Title
US11674883B2 (en) Image-based assay performance improvement
US11719618B2 (en) Assay accuracy improvement
EP3837525A1 (fr) Dosage à base d'image utilisant des structures de surveillance intelligentes
US20230127698A1 (en) Automated stereology for determining tissue characteristics
US10755406B2 (en) Systems and methods for co-expression analysis in immunoscore computation
CN110088804B (zh) 基于原色和免疫组织化学图像的计算机评分
US11733151B2 (en) Assay detection, accuracy and reliability improvement
CN111512383B (zh) 用于图像处理的自动测定评估和归一化
JP7197584B2 (ja) デジタル病理学分析結果の格納および読み出し方法
US20190042826A1 (en) Automatic nuclei segmentation in histopathology images
WO2019048954A1 (fr) Détermination de qualité de coloration de tissu
CN112689757A (zh) 使用crof和机器学习的基于图像测定的系统和方法
US20140348410A1 (en) Methods for obtaining and analyzing images
US20230408534A1 (en) Assay Error Reduction
EP3241181B1 (fr) Analyse de vaisseaux dans des images multiplexées
Qu et al. Segmentation of Hematoxylin-Eosin stained breast cancer histopathological images based on pixel-wise SVM classifier
Uslu et al. Automated quantification of immunomagnetic beads and leukemia cells from optical microscope images
CN114945941A (zh) 用于支持肿瘤检测和分析的非肿瘤分割

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20210324

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G01N0001280000

Ipc: G16H0030200000

A4 Supplementary search report drawn up and despatched

Effective date: 20230206

RIC1 Information provided on ipc code assigned before grant

Ipc: G01N 15/06 20060101ALI20230131BHEP

Ipc: G01N 15/14 20060101ALI20230131BHEP

Ipc: G02B 21/34 20060101ALI20230131BHEP

Ipc: G01N 33/483 20060101ALI20230131BHEP

Ipc: G01N 21/69 20060101ALI20230131BHEP

Ipc: G01N 1/30 20060101ALI20230131BHEP

Ipc: G01N 1/28 20060101ALI20230131BHEP

Ipc: G01N 33/49 20060101ALI20230131BHEP

Ipc: G16H 30/20 20180101AFI20230131BHEP