EA044111B1 - METHODS AND DEVICES FOR NON-DESTRUCTIVE DETECTION OF UNDISSOLVED PARTICLES IN A FLUID - Google Patents

Description

Prior Art

To determine the quality of a drug formulation, differentiation of different types of particles is important. For example, due to low specificity during differentiation, objects such as glass plates may be mistaken for solid protein particles. High specificity of the differentiation system is necessary to make accurate formulation decisions. In the absence of data on the species or species of particles in a particular drug, selecting the exact formulation of the drug may be difficult.

Unfortunately, current particle detection techniques are not suitable for detecting protein aggregates and other small and/or fragile particles. Particles smaller than approximately 100 microns typically cannot be manually detected by humans. Automated research techniques are usually destructive; in other words, these techniques involve removing the test fluid from its container, after which it is generally unsuitable for therapeutic use. Additionally, prior art non-destructive testing systems use only a single photographic image of a container to determine the presence or absence of particles, which often results in inaccurate measurements of particle size and/or number. Known assay techniques may also involve breaking up more fragile particles such as protein aggregates. For example, high-speed (eg, 2000 rpm or higher for several seconds) centrifugation of a tube containing fluid may result in rupture of protein aggregates in the fluid.

The essence of the invention

One embodiment of the technology disclosed herein relates to a device for non-destructively detecting a particle (ie, an undissolved particle) in a vessel at least partially filled with a fluid, such as an aqueous fluid, an emulsion, a petroleum product, and an organic solvent. As used herein, the term detect or detect includes detecting, characterizing, differentiating, distinguishing or identifying the presence, number, location, nature, size, shape (eg elongation or roundness), color, fluorescence, contrast, light absorption coefficient, reflectance light or other characteristic, or combination of two, three, four, five, six, seven, eight, nine, ten, eleven, twelve or more of these characteristics of the particle. In the illustrated embodiments, the device includes an imaging device for the purpose of obtaining time series data depicting the trajectory of a particle in a fluid. The time series data is stored in a memory in operative communication with the imager, and a processor in operative communication with the memory detects and/or identifies the particle. Specifically, the processor reverses the temporal order of the time series data to produce reverse time series data, estimates a particle trajectory based on the reverse time series data, and determines the presence or type of a particle based on its trajectory. As used herein, reverse time series data includes frames of time series data presented in reverse chronological order such that the event that occurred last is displayed first (and vice versa).

Other embodiments include a method and associated computer program product for non-destructively detecting an undissolved particle in a vessel at least partially filled with a fluid. Implementation of the method includes reversing the temporal order of time series data representing the trajectory of a particle in a fluid to obtain reverse time series data, for example, by means of a processor executing commands encrypted in the non-volatile memory of a computer program product. The method also includes estimating a particle's trajectory based on the reverse time series data and then detecting and/or identifying the particle based on its trajectory.

Another embodiment is a device for non-destructively detecting an undissolved particle in a vessel at least partially filled with a fluid, comprising:

(a) at least two imaging devices arranged to form images of the particle from different angles, each imaging device configured to obtain one or more two-dimensional images of the particle in a fluid;

(b) a memory in operative communication with the imaging device and configured to store a time series; and (c) a processor in operative communication with the memory and configured to detect the particle by:

(i) combining two-dimensional images obtained from at least three imaging devices to determine three-dimensional data indicative of the location of the particle in the container; and (ii) detecting the particle based at least in part on the three-dimensional data.

Also disclosed is a method for non-destructively detecting an undissolved particle in a vessel at least partially filled with a fluid, comprising:

- 1 044111 (a) using at least two imaging devices to image the particle from different angles such that each imaging device respectively obtains one or more two-dimensional images of the particle in the fluid;

(b) combining two-dimensional images obtained from at least two imaging devices to determine three-dimensional data indicative of the location of the particle in the container; and (c) detecting the particle based at least in part on the three-dimensional data.

Still other embodiments of the present invention include an apparatus, method, and computer program product for non-destructively detecting (one or more) transparent or reflective objects (eg, glass plates) in a container at least partially filled with a fluid. The imaging apparatus receives data representing light reflected from multiple spatial locations in the vessel as a function of time, and stores the data in a memory in operative communication with the imaging apparatus. The processor, in operative communication with the memory, detects objects (for example, glass plates), possibly in response to instructions encoded in the data-based computer program product, by identifying the corresponding maximum amount of reflected light for each point among the plurality of points represented in the data. The processor then determines the presence or absence of objects (eg, glass slabs) in the container based on the number of spatial locations in which the corresponding maximum amount of reflected light exceeds a predetermined value.

Another embodiment of the invention is a method for non-destructively counting and determining the size of undissolved particles in a vessel at least partially filled with a fluid. The method includes:

(a) obtaining at least one image of particles in the vessel obtained under specified imaging conditions;

(b) detecting particles and determining information indicative of the apparent size of the detected particles in the image based on the at least one image;

(c) determining apparent size particle population information indicative of the apparent size distribution of detected particles; and (d) determining actual size particle population information indicative of the actual size distribution of detected particles based on:

(i) visible size particle population information, and (ii) calibration population data indicating the apparent size distribution of one or more populations of standard size particles imaged under conditions corresponding to specified imaging conditions.

Another embodiment is a device for counting and determining the size of undissolved particles in a vessel at least partially filled with a fluid, containing at least one processor configured to:

(a) obtaining at least one image of particles in the vessel obtained under specified imaging conditions;

(b) detecting particles and determining information indicative of the apparent size of the detected particles in the image based on the at least one image;

(c) determining apparent size particle population information indicative of the apparent size distribution of the detected particles; and (d) determining actual size particle population information indicative of the actual size distribution of the detected particles based on (i) apparent size particle population information, and (ii) calibration population information indicative of the apparent size distribution of one or more populations particles of a standard size, the image of which was obtained under conditions corresponding to the specified imaging conditions.

Another embodiment of the invention is a computer program product for non-destructively counting and determining the size of undissolved particles in a vessel at least partially filled with a fluid, containing non-volatile machine-readable instructions that, when executed by a processor, cause the processor to do the following:

(a) obtaining at least one image of particles in the vessel obtained under specified imaging conditions;

(b) detecting particles and determining data indicative of the apparent size of the detected particles in the image based on the at least one image;

(c) determining apparent size particle population information indicative of the apparent size distribution of the detected particles; and (d) determining actual size particle population information indicative of the actual size distribution of detected particles based on

- 2 044111 (i) visible size particle population information, and (ii) calibration population data indicating the apparent size distribution of one or more populations of standard size particles imaged under conditions corresponding to specified imaging conditions.

Another embodiment of the invention is a method for non-destructively detecting an undissolved particle in a vessel at least partially filled with a fluid, comprising:

(a) using at least one imaging device to form an image of the particle;

(b) processing the image to determine location data indicative of the location of the particle in the container;

(c) detecting the particle at least in part based on the location data, wherein detecting the particle at least in part based on the location data includes determining the presence of the particle at a site of the vessel;

(d) using a sensor to determine a particle characteristic when the particle is located at a specified area of the vessel, (e) generating particle characteristic data indicative of the particular characteristic; and (e) relating the particle characterization data to the particle identification data.

Another embodiment of the invention is a device for non-destructively detecting an undissolved particle in a vessel at least partially filled with a fluid containing:

(a) at least one imaging device positioned to form an image of the particle;

(b) at least one sensor configured to determine the characteristics of the particle when the particle is located in a specified area of the vessel;

(c) at least one processor in operative communication with each of the at least one imaging device and sensor, and configured to:

processing the image to determine location data indicative of the location of the particle in the container;

detecting the particle at least in part based on the location data, and determining the presence of the particle at a portion of the vessel;

using a signal from the sensor to determine a particle characteristic when the particle is located at a specified portion of the vessel, generating particle characteristic data indicative of the particular characteristic; and the relationship of particle characterization data with particle identification data.

Another embodiment of the invention is an apparatus for non-destructively detecting an undissolved particle in a vessel at least partially filled with a fluid, wherein the vessel comprises a transparent tubular vessel wall extending along a longitudinal axis, and the device comprises: an imaging device configured to obtain one or more images of a particle in a fluid, the imaging device comprising at least one optical imaging element positioned to form an image of the particle onto the sensor; a light source located at least partially in a plane passing through the vessel and substantially orthogonal to the longitudinal axis of the vessel, wherein the light source is configured to substantially completely eliminate rays of light emitted by the source and reflected or refracted from the surface of the wall of the vessel, the image of which at least one optical element forms per sensor.

Another embodiment of the invention is a method for non-destructively detecting an undissolved particle in a vessel at least partially filled with a fluid, wherein the vessel comprises a transparent tubular vessel wall extending along a longitudinal axis, and the method includes: using an imaging device to obtain one or more imaging a particle in a fluid, the imaging device comprising at least one optical imaging element positioned to form an image of the particle onto the sensor; and illuminating the vessel by means of a light source located at least partially in a plane passing through the vessel and substantially orthogonal to the longitudinal axis of the vessel, wherein the illumination source is configured to substantially completely eliminate rays of light emitted by the source and reflected or refracted from the wall surface vessels, the image of which is formed by at least one optical element on the sensor.

Unlike other particle detection systems and technologies, the systems and technologies of the present invention are non-destructive - there is no need to remove fluid from the vessel to detect, count and identify particles in the vessel. As a result, the systems and technologies of the present invention can be used to study changes and interactions between particles, fluid and vessel over long periods of time, for example, minutes, hours, days, months or years. In addition, the systems and technologies of the present invention do not necessarily imply or result in the destruction of even more fragile particles, such as small protein aggregates, in the vessel. These systems and technologies also make it possible to obtain time series data, i.e. data representing the trajectories of particles in a moving fluid. Because the present systems use time series data instead of single-frame photographic images of the vessel, the present systems can provide more accurate estimates of the number of particles in the vessel and particle sizes. The present systems also allow additional data to be obtained about each of the particles, such as particle morphology and particle composition, based on particle movement data. For example, sinking particles usually have a higher density compared to floating particles.

This summary is for illustrative purposes only and should not be considered limiting in any way. In addition to the above-described illustrative aspects, embodiments and features, further aspects, embodiments and features will be apparent to those skilled in the art as described with reference to the following drawings and detailed description.

List of drawing figures and other materials

The accompanying drawings, included in and forming a part of this specification, show examples of implementations of the disclosed technology; The drawings, together with the description, are intended to illustrate the principles of the technology disclosed.

In fig. 1A-1C show a visual inspection unit, a visual inspection imaging module, and a visual inspection platform, respectively; each of these elements can be used to detect and identify particles in a container at least partially filled with fluid.

In fig. 2A shows sample preparation, loading, and operation of the visual inspection systems of FIG. 1A1S.

In fig. 2B shows processed images of particles and their trajectories in a moving fluid in a vessel obtained by an illustrative visual inspection system.

In fig. 3A-3C show three types of shaking of a vessel containing a fluid and one or more particles in preparation for detection and identification of particles: rotation of the cylindrical vessel (Fig. 3A), inversion and rotation of the syringe (Fig. 3B), and shaking of the syringe ( Fig. 3C).

In fig. Figure 4 shows the optical ray diagram of a telecentric lens used to form an image of a cylindrical vessel.

In fig. 5A shows a fluid meniscus and a recording volume in a cylindrical vessel containing a fluid.

In fig. 5B shows distortion and blind spots in a cylindrical container caused by the shape of the container.

In fig. 5C and 5D show techniques for compensating for distortion and blind spots when imaging cylindrical vessels.

In fig. 5E shows distortion and blind spots in a cylindrical container due to the shape of the container for particles at various points within the container.

In fig. 5F shows theoretical models of distortion caused by a cylindrical capacitance; each model corresponds to the same container filled with a fluid with a different refractive index. The figure also shows the corresponding experimental measurements that validate the theoretical models.

In fig. 5G shows the use of a correction optical element to correct distortion in a cylindrical container formed due to the shape of the container.

In fig. 5H is a detailed view of the correction optical element of FIG. 5G.

In fig. 5I shows an apparatus for selecting one or more correction optical elements.

In fig. 6A-6D show particle tracking systems with multiple imagers to obtain time series data for moving particles from multiple angles (FIGS. 6A and 6B), at high frame rates from one angle (FIG. 6C), and at different spatial resolutions from one angle (Fig. 6D).

In fig. 7A and 7B illustrate the initiation of the imaging and illumination process for particle imaging by dual-sensor imaging devices.

In fig. 8 shows a schematic diagram of an adaptable multi-purpose lighting configuration comprising light sources positioned in front of, behind, and below the vessel of interest.

In fig. 9A-9C illustrate illumination at various angles to distinguish different types of particles when using the light sources of FIGS. 8.

In fig. 9D shows a diagram of the lighting and timing sequence when using the configurations of FIG. 9A-9C to distinguish between different types of particles.

- 4 044111

In fig. 10A-10C show the glare from a vessel partially filled with fluid (FIG. 10A) and the placement of light sources outside the area defined by rotation of the imaging device about the longitudinal axis of the vessel (FIGS. 10B and 10C).

In fig. 10D-10E show an alternative lighting design to reduce or eliminate glare from the vessel.

In fig. 11 shows a schematic diagram of a configuration suitable for imaging polarizing (eg, chiral) particles.

In fig. 12 is a schematic diagram of a configuration suitable for exciting and imaging fluorescent particles.

In fig. 13A and 13B show maximum brightness projection images of glass slabs (FIG. 13A) and protein (FIG. 13B) obtained by the disclosed visual inspection system.

In fig. 14 shows flow charts illustrating the overall particle detection and identification process, as well as the image preprocessing, particle tracking, and statistical analysis subprocesses.

In fig. 15A and 15B show a frame of time series data before (FIG. 15A) and after (FIG. 15B) background subtraction.

In fig. 16A shows a time series data frame showing the particle as an eight-bit grayscale image (left).

In fig. 16B is a close-up view of the time series data frame shown in FIG. 16V.

In fig. 16C and 16D illustrate embodiments with defined frame boundaries of the time series data shown in FIG. 16A and 16B, respectively.

In fig. 17A-17D show how two consecutive frames of time series data (FIG. 17A) can be used to perform predictive tracking (FIGS. 17B-17D).

In fig. 18A shows a half-tone frame of time series data in which several particles are visible.

In fig. 18B shows an embodiment with defined image boundaries of FIG. 18A, used to determine the location of the geometric center of the particle.

In fig. 19 shows successive frames of time series data illustrating particle collision/occlusion.

In fig. 20A shows a time series data frame showing two adjacent particles in a highlighted area.

In fig. 20B-20E show successive frames of time series data showing particle occlusion clearly visible as particles pass through the highlighted portion of FIG. 20A past each other.

In fig. 21A-21C show apparent occlusion of a moving particle due to subtraction of background artifact, such as a scratch or dirt particles, on the vessel wall for straight paths (FIG. 21A), curved paths (FIG. 21B), and parabolic paths (FIG. 21C).

In fig. 22A-22C show determining the location of the center of mass of irregularly shaped particles using reverse time series data (FIGS. 22B and 22C), and using the center of mass location data to determine the trajectory of the particle (FIGS. 22A).

In fig. 23A-23D show fluid dynamics mapped and simulated in cylindrical vessels. In fig. 23 A shows changes in the shape of the meniscus. In fig. 23B and 23C show the formation of vortices in a fluid-filled vessel, and FIG. 23D shows particle trajectories in an illustrative vortex.

In fig. 24A and 24B show close-ups of successive frames of reverse time series data in which particle collisions were not correctly detected (FIG. 24A), and show the same plot after error correction (FIG. 24B).

In fig. 25A-25E show the time course of particle size measurement due to particle movement.

In fig. 25F is a plot of Feret diameter versus time for the particle shown in FIG. 25C.

In fig. 26A shows frames of processed time series data at various intervals, with lines indicating different particle trajectories.

In fig. 26B shows exemplary measurement data of several time-dependent particle properties corresponding to the particle trajectories of FIG. 26A.

In fig. 27A-27F show detection of a region of interest by angled backlighting. In fig. 27A shows the original image (time series data frame) subjected to edge extraction (FIG. 27B), halftone boundary detection (FIG. 27C), meniscus and tube base identification (FIG. 27D), region of interest (limited by dotted line in FIG. 27E) ), and cropping (Fig. 27F) to form an image of the visible fluid in the container.

In fig. 28A-28C show the determination of the fill volume of a backlit test tube. In fig. 28A shows a raw image of a test tube. In fig. 28B shows the area of interest

- 5 044111 (limited by a dotted line), defined by defining boundaries and highlighting contours. Defects on the surface of the tube (shown in Fig. 28C) can make it difficult to determine the fill volume.

In fig. 29A-29D show the determination of the fill volume of a test tube illuminated from below. In fig. 29A and 29B show pseudo-color images of a partially filled container (FIG. 29A) and an empty container (FIG. 29B). In fig. 29C and 29D show automatic detection of the meniscus of partially filled, empty, and partially filled vessels.

In fig. 30 shows a processor configured to process time series data.

In fig. 31 shows an example of detecting halftone boundaries of an image including a bright particle and a dim particle.

In fig. 32 shows a histogram of apparent particle sizes for a population of standard size particles (100 µm).

In fig. 33 shows apparent particle size data curves for two populations of particles, each population containing particles of the specified standard size (μm).

In fig. 34 shows calibration curves for apparent particle size data for four particle populations, each population containing particles of the specified standard size (µm).

In fig. Figure 35 shows the fit of the overlay of the calibration curves onto the apparent sample particle size data curve.

In fig. Figure 36 shows a comparison of the results using two particle counting and sizing techniques: raw binning and LENS.

In fig. 37 shows the process of counting and sizing particles using various techniques to determine the size of particles below and above a size threshold.

In fig. 38A-38C illustrate particle tracking systems with multiple imagers to obtain time series data for moving particles at multiple angles.

In fig. 39 shows the movement through the container of light beams received by each of the two imagers (left panel) and each of the three imagers (right panel) of the particle tracking systems of FIG. 38A-C.

In fig. 40 shows the particle detection results of an automatic particle detection system (denoted as APT) compared to the results of human visual inspection.

In fig. 41 shows the detection and classification results of particles by the automatic particle detection system.

In fig. 42 shows a table summarizing the linearity of particle counts as a function of sample dilution for an automated particle detection system.

In fig. 43 illustrates the accuracy of an automated particle detection system used to detect and count protein aggregate particles.

In fig. 44 shows the results of particle detection of protein aggregates by an automatic particle detection system (denoted as APT) compared to the results of human visual inspection.

In fig. 45 shows a spectrometer used with a visual inspection unit.

Information confirming the possibility of implementing the invention

In fig. 1A shows an exemplary embodiment of an automated visual inspection unit 100 configured to non-destructively detect and/or identify particles in a transparent container 10 at least partially filled with a fluid such as a protein-containing drug, drugs, biotechnology products, beverages, and other translucent fluids. media regulated by the US Food and Drug Administration.

Although detecting the presence or absence of a particle can be accomplished by inspecting portions of the container that have an irregularly shaped exterior (e.g., the back), in standard embodiments, particle characterization measurements, such as counting and sizing, may be it is necessary to inspect the particles through a substantially uniform vertical wall of the container to reduce distortion. This approach imposes restrictions on the minimum fill volume due to the fact that the apparent two-dimensional cross-section of the fluid in the container 10, accessible to inspection by the block 100, must be of a suitable area to provide acceptable statistics. The required filling volume depends on the circumferential diameter of the container (smaller containers require less filling volume). In various embodiments, the interior volume of the container may be filled with fluid to at least 1%, at least 5%, at least 10%, at least 20%, at least 30%, at least by at least 40%, by at least 50%, by at least 60%, by at least 70%, by at least 80%, by at least 90%, or by at least 100%.

In various embodiments, the particle detection techniques disclosed herein are optical in nature. Accordingly, in some embodiments, the walls of the container 10

- 6 044111 are sufficiently transparent at the wavelength of illumination to provide visualization of the fluid located inside the container. For example, in some embodiments, container 10 may be made of clear borosilicate glass, but other suitable materials may be used. The turbidity of the fluid contained within the vessel is also important and must be low enough to provide the desired degree of visualization. In some embodiments, the turbidity of the fluid is in the range of 0 to 100 NTU, preferably in the range of 0 to 20 NTU, and more preferably in the range of 0 to 10 NTU. Standard techniques for measuring turbidity are given, for example, in the Environmental Protection Agency's Guide, Turbidity Regulations, Chapter 3 (April 1999).

The exemplary systems are configured to detect and identify transparent and/or translucent particles that refract and/or scatter light (for example, protein aggregates, glass fragments or platelets, and oil droplets), particles that reflect light (for example, metal particles) , and/or light-absorbing particles (eg, carbon black and plastic particles), based on differences in their optical characteristics. Some of the disclosed visual inspection units 100 are configured to detect all three types of particles by using illumination sequences similar to those disclosed in the description below. The exposed visual inspection units 100 may also be configured to specifically detect, identify, and/or track proteins, which may be present as tightly bound aggregates, loosely bound cotton wool-like structures with high water content, (reflective) crystals, gelatinous structures, and/or amorphous units.

The term protein, as well as its synonymous term polypeptide, in its broadest sense refers to a substance consisting of two or more subunits - amino acids, amino acid analogues or peptidomimetics. The subunits can be linked through peptide bonds. In yet another embodiment, the subunits may be linked through other linkages, such as ester linkages, ether linkages, etc. As used herein, the term amino acid refers to natural and/or synthetic amino acids, including glycine and optical isomers D and L, amino acid analogues and peptidomimetics. A peptide consisting of three or more amino acids is often called an oligopeptide if it has a short peptide chain. When there is a long peptide chain, the peptide is called a polypeptide or protein. The term peptide fragment as used herein also refers to a peptide chain.

The container 10 may be a rectangular or cylindrical vessel made of glass or plastic (eg, a cuvette, bottle, cartridge, tube or syringe); the container may also have a different shape and/or be made of a different material, as long as the contents of the container can be visualized at the imaging wavelength. While certain embodiments provide clear and undisturbed visualization of the contents of the container, in still other embodiments the imaging periods may coincide with periods during which the container is undisturbed and/or post-processing may be used to compensate for corruption of the recorded data.

Unit 100 includes an imaging device 110 with collecting optics that projects images of the contents of the container onto a sensor. In this case, the collecting optics comprises a telecentric lens 114 and the sensor is a charge-coupled device (CCD) 112. A memory 140 coupled to the CCD 112 records and stores a stream of images representing the contents of the container, and a processor 130 coupled to the memory 140 , analyzes the recorded sequence of images as described below to detect and identify particles in the container 10. It will be apparent to one skilled in the art that the processor 130 may be configured in conjunction with a suitably configured mainframe computer (for example, using an Intel® Core™ i5 or Advanced Micro Devices processor Athlon™), field programmable gate array (eg, Altera® Stratix® or Xilinx® Spartan®-6 FPGA), or application specific integrated circuit. Memory 140 may be a solid-state memory (e.g., flash memory), optical disk (e.g., CD or DVD), or magnetic media, and may be of any selected size (e.g., 1 GB, 10 GB, 100 GB, or larger) .

A lighting system 120, comprising one or more light sources 122a and 122b located around the container 10, illuminates the container 10 and its contents during image formation. The visual inspection unit 100 may be incorporated into the inspection module 160, which also includes a spindle 150, a shaker, an ultrasonic vibrator, or other shaker for the purpose of rotating, shaking, or otherwise agitating the contents of the container prior to imaging, as well as for holding the container 10 during image formation according to FIG. 1(b).

In fig. 1(c) shows a medium/high speed visual inspection platform 170 comprising one or more inspection modules 160-1 through 160-5 (generally referred to as inspection modules 160), an automat 180, and a container tray 172, in individual container recesses. which contains uninspected and/or inspected containers 10. Upon receipt of appropriate instructions from the user or an automatic controller (not shown), the machine 180 moves the container 10 from the container tray 172 to the inspection module 190, which receives and records time series data for particles moving around the container 10. The machine 180 then returns the container 10 to the container tray 172.

In some embodiments, the top layer of the container tray 172 and/or the edges of the container wells are made of Delrin® polyformaldehyde resin or other similar material, and the inner edges of the container wells are beveled to prevent scratching the containers 10 when inserted into the container wells. and removal from them. The vessel tray 172 may include a base layer made of aluminum or other similar material that is resistant to deformation or cracking. The walls of the container wells are typically thick to provide secure support for the containers when the tray 172 is carried (eg, by a person) to and from the viewing platform 170. Depending on the design of the vessel tray 170, it may hold the containers 10 in predetermined positions within a tolerance on the order of microns to facilitate removal and insertion of the containers by the machine 180, which can operate with an accuracy of the order of microns.

The machine 180 is a lift-and-transport system that retrieves containers from a tray 172, moves each container 10 along a rail 182 extending from an area above the tray 172 to an area above the spindles 160, and places the container 10 on a specific spindle 160. Some machines may also be configured with the ability to rotate the container 10 prior to positioning the container 10, thereby eliminating the need for the spindle 160. In yet another embodiment, the machine 180 may include a six-axis mechanical arm configured to provide rotation, oscillation, and/or shaking (i.e., performing the below-described shaking of the needle back and forth) of the container 10, which also eliminates the need for the spindle 160. One skilled in the art will appreciate that other loading and shaking mechanisms and sequences may be used in the disclosed visual inspection systems and processes.

The platform 170 operates as shown in FIG. 2(a). At step 202, the containers 10 to be inspected are cleaned (eg, by hand using suitable solvents), and then placed on the tray 172 at step 204. The machine 180 removes the container 10 from the tray 172 and places it on the spindle 160. Then, at step 206, processor 130 determines the size and location of the meniscus and/or region of interest (eg, the fluid-filled portion of container 10) based on the image of fixed container 10 captured by imaging device 110. In yet another embodiment, the user can indicate the location of the meniscus and/or region of interest if the fill volume, shape and volume of the container are known with a reasonable degree of accuracy. After the region of interest is detected by the processor 130, the spindle 160 rotates and stops the rotation of the container 10 at step 208, which causes the fluid to move and particles to be suspended in the moving fluid in the container 10. At step 210, the imaging device 110 writes the time series data to memory. 140 as a sequence of static images (called frames) displaying photographic images of an area of interest taken at regular intervals.

After imaging device 110 receives a sufficient amount of time series data, processor 130 subtracts background data, which may be dirt and/or scratches on one or more surfaces of the container. The processor may also filter noise from the time series data in a manner apparent to one skilled in the art, and may also perform brightness threshold determination as described below. Processor 130 also reverses the temporal order of the time series data. That is, if each frame of time series data has a label 1, 2, ..., n - 1, n, indicating the order in which the specified frame was received, then the labels of reverse time series data frames are ordered as follows: n, n -1, . .., 2, 1. If necessary, processor 130 also selects the start and end points of the analyzed data as described below. (It will be apparent to one skilled in the art that processor 130 may perform background subtraction, noise filtering, luminance threshold determination, time series data reversal, and/or start/end point determination in any order.) Processor 130 tracks particles moving in or with a fluid. , at step 214, and then sizes, counts, and/or otherwise characterizes the particles based on their trajectories at step 214.

Each inspection module 160 can perform one general type of inspection, allowing for parallel inspection of containers 10; the number of 160 modules can be adjusted depending on the required throughput. In other implementations, each module 160 may be configured to perform different types of inspection. For example, each module 160 may inspect particles at different wavelengths of radiation: module 160-1 may search for particles that respond to light in the visible spectrum (i.e., wavelengths from about 390 nm to about 760 nm), module 160 -2 can examine containers using radiation in the near-infrared range (760-1400 nm), module 160-3 can examine containers using infrared radiation in the short-wave region of the spectrum (1.4-3.0 µm), module 160-4

- 8 044111 can examine particles at ultraviolet wavelengths (10-390 nm), and module 160-5 can probe particles at X-ray wavelengths (less than 10 nm). In yet another implementation, one or more modules 160 may be used to search for particle polarization and/or fluorescence effects.

In embodiments with different kinds of modules 160, the first module 160-1 may perform a preliminary inspection, and a subsequent inspection depends on the results of the preliminary inspection. For example, the first module 160-1 may perform a visible inspection that indicates that a particular container contains polarization-sensitive particles. Processor 130 may then command polarization-based measurement module 160-2 to inspect the container to confirm (or deny) the presence of polarization-sensitive particles. The visible time series data received by module 160-1 may indicate the presence of multiple particles in a particular container 10, but not the type of particles, causing processor 130 to command module 160-3 for infrared inspection.

Shaking the container to ensure particle movement.

As described above, mechanical shaking of the container 10 causes particles to be suspended at the bottom of the container 10 or on the inner walls of the container in the fluid within the container. In certain embodiments, the user and/or the visual inspection system selects and implements an agitation sequence that brings the fluid in the container into a laminar flow regime in which the fluid flows in parallel layers without eddies, turbulent motion, or disruption of the layers. In fluid dynamics, laminar flow is a flow regime characterized by high momentum dissipation and low momentum turnover; in other words, laminar flow is the opposite of rough, turbulent flow. Shaking also causes particles to be suspended in the moving fluid. Friction subsequently causes the fluid to stop moving, after which particles may adhere to the walls of the container or settle to the bottom of the container.

Compared to turbulent flow, laminar flow allows for smoother particle movement, making it easier to estimate particle trajectories. (Of course, the processor can also be configured to estimate particle trajectories in some turbulent flow regimes, provided the sensor frame rate is fast enough to capture images of smooth portions of particle trajectories.) If necessary, the container can be shaken to provide substantially laminar flow . For example, the spindle may rotate the container at a certain speed (or speed profile) for a certain period of time, determined based on measurements of the behavior of the fluid in containers of different sizes and shapes and/or different degrees of filling of the container and fluid viscosity.

In one embodiment, a servo motor or stepper motor drives a spindle holding the cylindrical container, causing the container to rotate about its central axis as shown in FIG. 3(a). Rotating the container 10 at a sufficient speed causes even heavy particles (such as metal particles) to rise from the bottom of the container 10 and penetrate into the liquid. When used with many liquids and particles, the motor drives the spindle holding the container 10 at a rotation speed of 300 rpm for approximately three seconds. (Higher rotation speeds may be necessary to excite heavy particles). After completing rotation for three seconds, the motor stops abruptly, and the fluid can flow freely in the again stationary container. At this time, the imaging device 110 begins to record a video image of the rotating fluid flowing. Memory 140 stores a video image for up to approximately 7-15 seconds, depending on the size of the container being tested (memory 140 stores a video image of shorter duration when testing fluid in smaller containers due to the fact that the fluid in smaller containers slows down faster due to the greater effect of friction on wall).

In yet another embodiment, the spindle rotates the container 10 in a two-phase shaking/imaging sequence. During the first phase, the spindle rotates the container 10 at 300 rpm for three seconds, which causes less dense (and smaller) particles, such as proteins, to be suspended in the moving fluid. Next, the imaging device 110 records a video image of the proteins in the moving fluid. Once imaging device 110 has acquired sufficient time series data, the second phase begins: The spindle rotates container 10 at approximately 1600-1800 rpm, causing denser particles, such as metal particles, to be suspended in the moving fluid, and device 110 The imaging device obtains time series data depicting the movement of denser particles in the vessel 10. The high-speed rotation in the second phase may be sufficiently intense to temporarily dissolve or denature protein aggregates, which may re-form after the fluid flow is slowed or stopped. The two-phase operating principle allows the detection of dense particles that may not be excited by low-speed rotation, and proteins that may be denatured by high-speed rotation.

Other rotation sequences may be used in the disclosed systems depending on (but not limited to): fluid viscosity, degree of fluid filling of the vessel, type of fluid, surface tension, container shape, container size, container material, container texture, dimensions particles, particle shapes, particle types and particle densities. For example, the disclosed systems may rotate larger containers for longer periods of time before obtaining an image of the contents of the container. The exact shaking profile for a given fluid/container combination can be calculated, characterized and/or determined experimentally.

In the event that the visual inspection module uses a predetermined shaking sequence for a sufficiently known fluid/container combination, the module can only trigger the acquisition process when the fluid (and suspended particles) are in laminar flow. In yet another embodiment, the module may receive additional time series data and the processor may automatically select start and end frames based on a selected fluid/container combination and/or shaking sequence.

In some embodiments, the data acquisition process may be triggered based on a detected characteristic of the fluid flow in the container. For example, as described in detail below, in some embodiments it is possible to detect a meniscus of a fluid in a container and monitor the movement of the meniscus to determine the point in time at which the vortex in the fluid weakens after rotation. In some of these cases, the data acquisition process may begin when the detected meniscus movement returns to a substantially stable state.

Any of the above-described visual inspection systems may also be used to detect and/or identify intrinsic and foreign particles in a syringe 12 at least partially filled with drug 32 or other fluid, as shown in FIG. 3B. Syringes 12 are often stored with the needle down. Accordingly, particles may settle in the syringe needle 34. To ensure visualization of these particles, the machine or person turns the syringe 12 over - i.e. the machine or person rotates the syringe 12 180° around an axis perpendicular to its longitudinal axis, resulting in the needle 34 facing upward. Particles deposited in the needle 34 fall in a vertical direction, allowing them to be imaged by the imaging device 110. The machine or person may also rotate the syringe as it is inverted to fully excite the fluid.

Many syringes 12 have barrels with relatively small internal diameters (eg, about 5 mm), which greatly increases the effect of wall friction. For many drugs 32, wall friction causes the entire rotational motion of the fluid to stop in about one second. For particle analysis in practice, such a time window is extremely short. However, gentle shaking of the syringe 12 about an axis perpendicular to its longitudinal axis, according to FIG. 3(c) provides particle movement lasting more than one second. Lateral shaking, which can be performed automatically or manually, excites particles as a result of the movement of the syringe 12 and the movement of the air bubble or bubbles 30 oscillating within the barrel of the syringe 12. The above-described visual inspection modules, blocks and platforms are reconfigurable and can be used with the alternative shaking method indicated.

Once shaking is complete, the visual inspection system must remain stationary to complete the video recording phase. Due to the high resolution of commonly used imaging devices, the spatial resolution of images is very precise (eg, about 10 μm or less) and can be at least as accurate as the diffraction limit. In some configurations, a small (eg 10 µm) probe movement corresponds to an entire pixel of movement in the acquired image. Such movement negatively affects the efficiency of static element removal (background subtraction), which in turn degrades the performance of the analysis tools and the accuracy of the output data.

With the above in mind, vibration elimination is a key design consideration. In certain embodiments, the base of the exemplary visual inspection system is mechanically isolated from the laboratory environment, such as through vibration dampers, platforms, and/or pads. In addition, within the unit, vibration sources such as computers and automated controllers can be mechanically isolated from the rest of the system. In yet another embodiment, data acquisition may be synchronized with the residual movement of the capacitor relative to the imaging device, or may be accomplished by a camera implementing pixel shift or some other motion-compensating measures. This residual motion may also be captured for subsequent processing to eliminate the negative effects of image motion.

- 10 044111

Imaging device configurations.

Exemplary visual inspection systems may use standard available imagers with any suitable sensor, including, but not limited to, charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) arrays. The choice of sensor is quite wide and depends to some extent on the requirements imposed by the particular application. For example, high frame rate sensors provide the ability to accurately map the trajectories of fast-moving particles (for example, in low-viscosity liquids). Sensitivity and noise performance are also important due to the fact that many protein particles in solution are transparent and weakly scatter light, resulting in their images being poorly visible. To improve noise characteristics, the sensor can be cooled in a manner known in the art. For most applications, monochrome sensors provide the best performance, providing slightly greater resolution than color cameras and also having higher sensitivity. However, for a small subset of applications, color sensors may be preferred due to the fact that they capture the color of a particle, which can be extremely important in identifying its source (for example, when examining clothing fibers). For example, in product quality testing (also called technical testing), color sensors can be used to distinguish between different types of materials (eg, fibers) that could potentially contaminate a drug in a manufacturing facility.

To perform a complete inspection of the container, the inspection area of the imaging device must cover the entire volume of the fluid. At the same time, the imaging device must be capable of distinguishing small particles. Visual inspection systems provide large inspection areas and high resolution with large format, high resolution sensors such as the 8 megapixel Allied Vision Technologies (AVT) Prosilica GX3300 CCD sensor with 3296x2472 pixel resolution. Other suitable sensors include the 5MP ACT Pike F505-B and Basler Pilot piA2400-17gm cameras. After selecting optical means to obtain a complete image of the liquid-carrying body of the BD Nurak 1 ml syringe, the AVT Prosilica GX3300 CCD sensor acquires time series data with a spatial resolution of approximately 10 µm per pixel in both lateral dimensions. The combination of high speed and high resolution suggests that high data rates and large file sizes may be necessary to record time series data. Because of the above, the video compression techniques described below are specifically designed to reduce data storage requirements while maintaining the fidelity of the fine particle details captured in the image.

The collecting optics that transmit the image of the area of interest to the sensor must be selected so that it provides clear images of the entire volume with a minimum spot size equal to or less than the pixel size of the sensor to ensure the system operates at the highest resolution. In addition, the collection optics preferably have a depth of field large enough to cover the entire sample volume.

Telecentric lenses, such as lens 114 of FIG. 4 are particularly well suited for visual inspection of fluid volumes due to the fact that they are specifically designed to be insensitive to depth of field. One skilled in the art will appreciate that a telecentric lens is a multi-component lens in which the primary rays are collimated and parallel to the optical axis in the image and/or object space, resulting in a constant degree of magnification regardless of the image and/or location of the object. In other words, for an object located within a certain range of distances from the telecentric lens imaging apparatus, the image of the object obtained by the imaging apparatus is clear and has a constant magnification ratio regardless of the distance of the object from the imaging apparatus. This allows for images to be obtained in which particles at the back of container 10 appear similar to particles at the front of container 10. The use of a telecentric lens also reduces ambient light detection as long as a uniformly dark background is used. Suitable telecentric lenses 114 include the Edmund Optics NT62-901 Wide Format Telecentric Lens and the Edmund Optics NT56-675 TECHSPEC Silver Series 0.16x Telecentric Lens.

Inherent blind spots in containers.

One of the goals of almost any visual inspection system is to ensure 100% inspection of the entire tank volume. However, in practice the container may have fixed areas where particles cannot be detected, as shown in FIG. 5A. First, it may be difficult to include fluid around the meniscus in the analysis due to the fact that the meniscus itself scatters light in such a way that the detector in that area may be saturated and any particles or other elements of interest may be obscured. Secondly, in the case of a bottle, the base of the container is usually curved at the corners (this area is usually called the stem). Curved leg

- 11 044111 distorts and essentially hides any particles that are sufficiently close to the bottom of the bottle.

Thirdly, when using a syringe, the rubber plug contains a central cone that protrudes slightly into the volume of the container. Although the tip of said cone is quite small, it has the potential to hide particles. The least noticeable blind spots are caused by the curvature of the vessel.

Cylindrical containers can also cause the lens effect shown in FIG. 5B (indicated by curved beams 18), which negatively affects the operation of the telecentric lens. The curved walls of the container also form blind spots 14.

In fig. 5E shows an example of the lens effect caused by the cylindrical container 10. The camera/observer is located at the bottom of the drawing. As described above, a telecentric lens can be used when imaging particles in the container 10 to provide a uniform appearance of the particles in the image regardless of their position in the container, i.e. their distances from the camera. To this end, in some embodiments, the selected depth of field of the telecentric lens is greater than the diameter of the fluid volume. In some embodiments, the curvature of the capacitance violates this principle in the absence of a corrective optical element.

According to the drawing, the shape and degree of magnification of the depicted particle in container 10 depends on the position of the particle in the container. Particle 510, located in the center of the container in the foreground, is absolutely not distorted (top inset). Identical particle 502, located to the side in the background, is distorted to the greatest extent (lower inset). It should be noted that when using a cylindrical container, distortion occurs only along the horizontal axis (as can be seen on the second inset).

To reduce these effects, optional correction optics such as correction lens 116 of FIG. 5C. Additional spatial correction optics 118 may provide additional compensation for distortion caused by the capacitance shape, as shown in FIG. 5D. In various embodiments, any suitable correction optical elements may be used in addition to or instead of the correction lens 116 and optics 118, such as those specifically selected for the curvature of the container 10 and/or the refractive index of the fluid.

For example, in some embodiments, a model of the lens effect caused by the cylindrical container 10 may be developed. The model may be based on a suitable set of parameters characterizing optical distortion, including, for example, the outer diameter of the container, the inner diameter of the container, the refractive index of the container, the refractive index fluid, and the wavelength of light radiation. The model may be developed using any techniques known in the art, including, for example, ray tracing techniques. In fig. Figure 5F shows examples of theoretical models of the lens effect for two different groups of capacitance parameters (top left, bottom left), as well as experimental data for the corresponding physical conditions (top right, bottom right). It can be seen from the drawing that the theoretical model exactly matches the experimental data.

According to FIG. 5G and 5H, a correction optical element 503 (represented by a lens in the drawing) is used to correct the above-described lens effect. The design of a corrective optical element can be based on a theoretical optical model of the capacitance, on experimental data describing the optical properties of the capacitance, or on combinations of the two. According to the drawing, the correction optical element 503 is made of a refractive material, and has front and rear surfaces of a cylindrical shape. In some embodiments, the design of the lens may be determined using free parameters including the radius of the front and rear surfaces, the thickness of the lens, the refractive index of the lens, and the location of the lens relative to the capacitance.

In some embodiments, the front and rear surfaces of the lens may have other shapes, such as parabolic or any desired shape. In some embodiments, relaxing the cylindrical surface requirements allows the size of the parameter space for the design of the correction optical element 503 to be increased, allowing for improved distortion correction.

In some embodiments, the correction optical element 503 may comprise multiple elements, further increasing the design parameter space. In some embodiments, correction optical element 503 may correct other types of optical distortion, aberration, or other effects. For example, when using multi-wavelength lighting, the correction optical element 503 can be used to correct chromatic aberration.

In some embodiments, the correction optical element 503 may be configured to correct distortion caused by a particular type of capacitance and/or fluid. Because a single automated visual inspection unit 100 may be used with multiple types of containers, in some embodiments it may be desirable to be able to selectively replace the correction optical element 503 to match a particular container 10 being examined. For example, in FIG. 51 shows the rack 504, on

- 12 044111 which houses several correction optical elements 503. The stand can be moved (manually or automatically) to accommodate the selected element in the optical chain for the image forming device 110. It should be understood that while a stand is shown in the drawing, in various embodiments, any other mechanism suitable for selecting one optical element from a set of multiple optical elements may be used.

Alternative visual inspection systems may include adaptive optics to compensate for distortion caused by the curvature of the container. For example, the telecentric lens 114 may be configured to form an image of the capacitance 10 reflected from a deformable mirror, such as a microelectromechanical system (MEMS) mirror. Sensor 112 uses background data to infer the nature and extent of aberrations caused by surface curvature, surface imperfections, and other defects in capacitance 10. Sensor 112 sends this information back to the deformable mirror, which responds by adjusting its surface to compensate for the aberrations. For example, a deformable mirror may bend or bend in one direction to compensate for the curvature of the container. Because the deformable mirror responds dynamically, it can be used to compensate for aberrations inherent in each individual capacitance 10.

In addition, particle tracking can be configured to detect the disappearance of particles in known areas corresponding to specified blind spots, allowing the program to predict the possibility and timing of a possible subsequent reappearance of a particle in the video image as described below.

Additional techniques for solving blind spot problems (eg, using multiple imaging devices) are described below.

Camera frame rate.

The efficient particle tracking described below using the closest match algorithm (absorbing algorithm) can be viewed as a function of three main factors: the imaging speed (frame rate) of the camera, the density of the particle (in a 2D image), and the standard speed of the particle. To achieve practically effective tracking using the closest match algorithm, the camera must have a frame rate fast enough to meet the following criterion:

Camera frame rate > Maximum particle speed / Minimum distance between particles.

In practice, when projecting a three-dimensional volume onto a two-dimensional image, it is possible that the particles appear as close to each other as possible (or even block each other), while in reality they are sufficiently spaced apart in space in the container. Taking into account this fact, it seems more appropriate to consider the average distance from a nearby particle, rather than the obvious minimum distance between particles. It should be noted that the nearby particle distance represents the distance between neighboring particles in a single frame of time series data, and the closest match distance refers to the distance between positions marked for the same particle in successive frames of time series data. Recalculating the camera speed criterion according to the distance of the closest match gives:

Camera frame rate > Maximum particle speed / Minimum distance between particles.

Alternative visual inspection systems may use predictive tracking techniques instead of nearest-match particle tracking techniques (absorption techniques). In predictive techniques, data about the known trajectory of a particle, coupled with data about the spatial constraints of the container and the expected behavior of the fluid, are used to estimate the most likely position of the particle in the next frame. If executed without error, this approach can allow more accurate tracking of particles moving at speed in intense images.

When attempting to detect and measure extremely small particles in relatively large containers, it is preferable to maximize the spatial resolution of the image sensor. In general, the immediate effect of this increase is to lower the maximum possible frame rate of the sensor.

Visual inspection using multiple imaging devices.

When using a single camera, the quality of the inspection may be reduced due to known blind spots. In addition, superimposing a 3D image of a particle distribution onto a 2D image can lead to ambiguities due to particle occlusion (eg, according to Fig. 5E, in which a particle centered in a container in the background is occluded by a particle centered in the foreground). In alternative visual inspection systems (eg, according to FIG. 6), this problem can in principle be solved by correlating the results obtained from two or more imaging systems. By correlating positional trajectory information from two or more cameras, detailed 3D trajectory maps can be constructed which can be more reliable and less prone to occlusion errors (as described below) than 2D trajectory maps.

Increasing the spatial resolution of the imaging device also limits the data acquisition rate (frame rate) for a given particle concentration and particle velocity. When examining unknown containers, there can be no guarantee that the particle concentration will be sufficiently low. At the same time, to ensure suspension of heavy particles, such as glass or metal, in a fluid medium, a fairly high rotation speed in the container may be necessary, which leads to high particle speed in the resulting video stream. One way to resolve this controversy is to use the innovative imaging hardware disclosed below. By using the best sensors available on the market and allowing enough light to be scattered by the particles in the container, the data acquisition rate can be further increased by condensing the data from two or more sensors with constant reliable triggering via a dedicated trigger source.

In addition, exemplary visual inspection systems may be designed to provide spatial resolution greater than 10 μm by eliminating the need for a complete inspection of the container and viewing only a portion of the volume. In general, this technique is suitable for studying invisible particles, especially protein aggregates, due to the fact that smaller particles are usually present in larger quantities and more uniformly distributed throughout the volume. In yet another embodiment, exemplary visual inspection systems can provide a complete view of the tank and high spatial resolution by using multiple imaging devices at varying degrees of magnification to simultaneously acquire time series data over a wide area and at high resolution.

Different magnification levels can be used simultaneously, for example according to FIG. 6A, in which one imager 1102 is used to view the entire container, and a second imager 1104 with a higher magnification (e.g., a microscope lens with a long focal length) is used to image a finer subvolume in detail and examine, for example, extremely small particles (e.g. particles with a diameter of approximately 10 µm, 5 µm, 1 µm or less). Other visual inspection systems may include multiple imaging devices 1102, 1104, and 1106 located around a container illuminated by one or more rings of LEDs 1120 mounted above and below the container 10 of FIG. 6B. Identical imagers 1102 placed at different locations provide binocular viewing. An imager 1104 with a long focal length microscope lens provides high resolution for a subvolume of the container 10, and an imager 1106 with an alternative sensor (eg, an infrared sensor, a bolometer, etc.) provides additional time series data.

In fig. 6C and 6D show alternative imaging configurations utilizing telecentric imaging properties. At the rear aperture of the telecentric lens, a 50/50 beam splitter cube 1202 splits the projected image into two separate image taps. Each of the image fingers may include a low-speed, high-resolution sensor 1222 that operates alternately with sensor 1222 in the other finger, as shown in FIG. 6C, with the goal of doubling the frame rate. In other words, using two 1222 sensors simultaneously with a relative half-cycle phase shift increases the time resolution by a factor of two. The image streams can then be combined to provide a single video image at twice the sensor's nominal frame rate.

In yet another embodiment, each tap may include a different sensor, as shown in FIG. 6D, for example, to compensate for the trade-off associated with the image sensor array: the higher the camera resolution, the lower the maximum possible frame rate of the camera (for example, 10-50 or 15-25 fps at full resolution, and 50- 200 fps at low resolution, etc.). The most important sensor performance parameter for accurate particle tracking is high time resolution (high frame rate). However, to accurately determine particle sizes, the most important sensor performance parameter is high spatial resolution (the largest possible number of pixels in the image). Currently, the main limiting factor for spatial resolution and data rate is the data bus. Conventional imagers can acquire time series data for a 4 cm tall container with a spatial resolution of approximately 10 μm per pixel and a data rate of approximately 25 frames per second on a standard personal computer bus (eg, dual GigE bus or CameraLink).

In fig. 6D shows one way to achieve high frame rates and high resolution: the fluid is imaged by a low speed, high resolution sensor 1222 and by a lower spatial resolution but higher frame rate sensor 1224. External triggering can provide commensurate synchronization of the two cameras. Because the cameras capture copies of a single image, camera data can be directly correlated to provide improved particle analysis.

In fig. 7A and 7B illustrate the timing and control of lighting sources 120 and multiple cameras. In fig. 7A and FIG. 7B, trigger controller 702 emits two trigger signals, designated ARM 1 and ARM 2 in FIG. 7A and 7B, the signals being output by decimating the main pulse signal. The trigger signal ARM 1 drives the first chamber (1102a in FIG. 7A, 1222a in FIG. 7B) and the trigger signal ARM 2 drives the second chamber (1102b in FIG. 7A, 1222b in FIG. 7B) in turn. In other words, under the influence of trigger signals, the first and second cameras receive alternate sequences of frames. The trigger controller 702 may also drive the light source 120 via a light signal, upon receipt of which the light source 120 illuminates the container whenever an image is acquired by the first or second camera. Other startup sequences are also possible; for example, trigger controller 702 may drive additional cameras and/or combinations of high- and low-resolution cameras that capture images at different frame rates.

It will be apparent to one skilled in the art that other configurations are equally possible. For example, the imaging sensors on each finger may be equivalent, but the collection optics may be different. One bend may contain additional optics to magnify the image to detail a specific area of the image, providing wide-field and magnified views simultaneously.

Lighting configurations.

The claimed visual inspection systems use the interaction of various particles with light to detect and identify particles in containers containing liquid. The interaction of a particle with light is a complex function of a number of factors including particle size, shape, refractive index, reflectance and opacity. Protein-containing particles can mainly scatter light by refraction, and plate glass particles can mainly reflect light. Some particles, such as collagen fibers, can change the inherent physical properties of light, such as rotation and polarization. High-precision detection and differentiation can be achieved by tailoring the detector, particle, and light geometry to maximize the contrast between different particle species.

In fig. 8-12 show various lighting configurations adapted or configured to switch/drive different types of lighting for specific types of particles, containers and/or fluids. For example, light sources may illuminate the particles to maximize the amount of light they reflect or refract towards a detector, while leaving the background dark to maximize the contrast between the particle images and the background. In addition, the sources may emit radiation at any suitable wavelength or range of wavelengths. For example, the sources may emit broadband white light (390-760 nm), narrow-band beam (eg, 632 nm), or even ultraviolet or x-ray radiation. Suitable ranges include: 10-3000 nm, 100-390 nm (ultraviolet), 390760 nm (visible), 760-1400 nm (near-infrared), and 1400-3000 nm (mid-wave infrared). X-ray radiation (<10 nm) is also possible. Taken together, the scope of illumination options disclosed in the present invention allows the inventive visual inspection systems to detect and identify the full range of particles that may be present in drug products.

Due to the fact that some particles scatter light very poorly, it is often preferable to irradiate the sample with the greatest amount of light possible. The upper limit of possible sample irradiation primarily depends on the photosensitivity of the product being tested. Careful selection of wavelength may also be necessary, particularly for biological products; the specific choice depends on the product being irradiated. Monochrome red light with a wavelength of about 630 nm represents the sweet spot, and emission of this wavelength is widely available in inexpensive light sources.

LED arrays, such as CCS Lighting's range of LDL2 LED arrays, are effective at illuminating particles found in pharmaceutical products; however, collimated laser beams can also be used. In some cases, light optics can structure or shape a beam of radiation that is collimated within a volume of fluid (rather than outside the container). In the event that heating of the light source is not desired when using alternative light sources, light may be delivered to the inspection area through the use of optical waveguides or optical fibers 124 as shown in FIG. 8.

The wavelength of illumination can be selected based on the light absorption and/or reflectivity of the fluid and/or the particles being analyzed; this process is especially important when studying light-sensitive pharmaceutical products. Red light (630 nm) provides an acceptable balance between the low light absorbance of protein and the low light absorbance of water. Light gating, synchronized with time series data, helps preserve the integrity of light-sensitive pharmaceutical products by minimizing the impact of incident light on the products. Gating has two additional benefits: LEDs operate more economically in this mode, and gating reduces the effect of motion-induced blur, which, if uncorrected, can negatively affect particle size measurements as described below.

In fig. 8 illustrates an exemplary variable lighting system 120 comprising multiple light sources 122a-122f (collectively referred to as light sources 122), which may be LEDs, lasers, fluorescent or incandescent lamps, flash lamps, or any other suitable light source or combination of suitable sources. Sveta. Light sources 122 may emit visible, infrared, and/or ultraviolet radiation. The light sources may be narrowband or broadband as desired, and may be filtered by suitable optical filters or polarizers. In fig. 8, for example, polarizer 126 polarizes light emitted by light source 122f providing background illumination of the container. In addition to the background light 122f, the lighting system 120 includes four light sources 122a-122d located at the corners of a rectangular prism around the container 10. Another light source 122e illuminates the container 10 from below through an optical fiber 124 coupled to a collimator 126 directed toward the bottom of the container 10. In some cases, the fiber 124 and collimator 126 may be placed within a hollow spindle shaft 128 used to rotate the vessel.

The multiple light sources 122 shown in FIG. 8 can be used to determine the optical properties of a particular particle to provide differentiation based on the particle's interaction with light. One skilled in the art will recognize that different particles interact with light in different ways. Common types of interaction include scattering, reflection, occlusion, or reversal of light polarization, as shown in the table below, in which the X symbol indicates the visibility of a given particle type under a given type of illumination; examples are shown in Figs. 9A-9D and FIGS. 11 (described below). The symbol M indicates the possible visibility of particles of a given type when using a given type of lighting, but particles can also be detected/differentiated by image segmentation in post-processing and through object identification techniques.

Interaction of different types of particles with light

Particle type

Protein Plates Lightproof Cellulose Air particles

Dominant interaction

Method Diffusing Reflection Occlusion Shift Diffusing polarization lighting

Rear corner XXXXX

Below XM

BackgroundX backlight

Polarizing MM XM light

In fig. 9A-9C show various examples of lighting that may be implemented in the lighting system 120 of FIG. 8 (some light sources 122 are omitted for the sake of clarity) to allow differentiation of particle species based on their interaction with light. In fig. 9A, light sources 122a and 122b provide back angle illumination suitable for imaging proteins as well as most types of light scattering particles. In fig. 9B, light source 122e provides illumination from below suitable for imaging reflective particles, such as glass slabs, reflecting light toward imager 110 (indicated by a horizontal arrow); Particles that scatter, but do not reflect, light (such as proteins) may not be visible to the sensor (diagonal arrow). In fig. 9C, light source 122f provides a uniform backlight illumination suitable for imaging light-occluding particles such as metal particles, dark plastic particles, and fibers. One skilled in the art will appreciate that other light sources and/or illumination examples and sequences are also possible.

In fig. 9D shows the sequential use of the lighting methods of FIG. 9A-9C to obtain time series data for scattering, reflecting and/or occluding particles. IN

- 16 044111 in this case, a system comprising uniform backlighting, backlighting, downlighting and one camera alternately illuminates each frame such that only one specific light source 122 (or combination of light sources 122) is active at any time. . When using a single imager (not shown), only one set of light sources is used for each acquired frame of time series data. Repeating this sequence provides a video image for each lighting configuration.

Sequencing video images using the lighting techniques described above in a specific sequence provides nearly simultaneous video images for each light source 122. Once the process is complete, this method provides three interleaved video images, one for each lighting method. In each video image, a particle in a single frame may correlate with a specified particle in two other video images obtained using alternating lighting methods (minor time differences between frames are not taken into account). Using general information obtained from the interaction of a single particle with various methods of illumination, it is possible to infer the material composition of the particle.

This method can be combined with other information for identifying objects in the image to improve certainty. For example, video images can be automatically divided to identify objects in each frame. Information such as size, shape, brightness, smoothness, etc. can be automatically determined for each object and each lighting method. This process can help differentiate different types of particles that have similar visibility characteristics when using each of the different lighting methods.

In fig. 10A-10C illustrate a method for reducing glare caused by unwanted reflection/refraction of light emitted by light sources 122 from a container 10. Illumination of the container 10 results in unwanted glare in images produced by imagers 110 whose optical axes coincide with the direction of travel of the light from the sources. 122 light reflected from the surface of the container. The glare can hide particles that would otherwise be detected and saturates areas of the sensor. The glare detected by the sensor can be reduced or eliminated by positioning the imaging device 110 or light sources 122 such that the optical axis of the imaging device is not aligned with or parallel to the light rays emitted by the light sources 122 and reflected from the container. For example, placing the light source or sources 122 outside the enclosed area defined by rotation of the imaging device about the longitudinal axis of the container 10 can reduce the amount of unwanted reflected and/or refracted light received by the sensor. In yet another embodiment, zone 100 may be defined as a plane orthogonal to the central axis of the cylindrical container, the thickness of which is equal to the height of the vertical walls of the containers. It will be apparent to one skilled in the art that containers having more complex shapes, such as concave side walls, may have other enclosed areas and different correction optics.

Oblique illumination of the side walls of the container from a point above or below zone 1000, or directly below the base of the container, also reduces the glare detected by the imaging device 110. Illuminating the container 10 from below (eg, via light source 122e (FIG. 8)) also allows for excellent contrast between light reflecting particles (eg, glass slabs) and light scattering particles (eg, protein particles).

In fig. 10D-10E show an alternative lighting arrangement for reducing or eliminating glare from container 10 in which one or more light sources 122 are located in the above-described enclosed area (eg, in the horizontal plane of container 10).

In fig. 10D-10E is a beam optics model showing the movement of beams from outside the sensor of the imaging apparatus 110, through the imaging optics of the imaging apparatus (as shown, including the telecentric lens), and back through the capacitance 10. A light source located along any of the beams is passing back from the sensor refracts or reflects light onto the sensor, thereby potentially hiding the container 10 and its contents. However, it should be noted that two portions 1001 are provided located in the horizontal plane of the container 10 in the vicinity of the outer wall of the container 10. Referring to FIG. 10E, by placing one or more light sources 122 on areas 1001, glare from the light sources can be reduced or substantially eliminated.

It should be noted that due to the use of a telecentric lens in the example shown, in the ray optics model it is necessary to consider only rays incident on the sensor along the normal. However, given the additional beams, a similar approach can be applied to other types of imaging optics. For example, in some embodiments, a representative series of rays from the sensor (e.g., including the original rays of the imaging system) may be reversed to determine areas lacking or substantially lacking

- 17 044111 rays passing back. Light sources can be placed in detected areas to avoid glare.

In fig. 11 shows a design for distinguishing elongated protein aggregates from cellulose and/or fibers (natural or synthetic) using polarized light. The illumination system 120 emits light toward a container 10 positioned between crossed polarizers 900 providing a black, particle-free image. Particles that change (eg, rotate) the polarization of the incident light appear white in the time series data detected by the imaging device 110.

In the event that the particles of interest have known fluorescence, fluorescence imaging can be used to identify the particle as shown in FIG. 12. In this case, illumination source 920 emits blue light to excite the particle of interest. A narrow bandpass (eg, green) filter 922 placed in front of the imager 110 allows only fluorescence from the excited particles to reach the detector. Illumination and filter wavelength data can be selected to match specific wavelengths of interest.

Finally, particles that do not scatter (refract) or reflect light, such as small particles of black opaque material, can be detected (and identified). To detect such opaque particles, the sample must be illuminated directly from behind. In this case, particles can be distinguished as dark elements against a bright background. The images of opaque particles can be inverted, if necessary, to produce images that have the same polarity as the images of scattering and reflective particles (ie, causing the particles to appear as light spots on a dark background, instead of black spots on a light background).

Specific visual inspection platforms for lamina detection.

As is known to those skilled in the art, glass flakes are thin, flexible particles or flakes of glass formed as a result of chemical reactions on the internal surfaces of glass containers. The claimed systems and techniques can be used and/or adapted to detect, identify and count glass flakes in order to minimize the likelihood of prescribing drugs containing glass flakes to prevent the prescription of drugs containing (excessive amounts of) glass flakes. The claimed systems and techniques can also be used and/or adapted to study the formation of glass platelets, depending on the composition of the drug in question; Glass platelets differ from proteins and other types of particles in that they reflect and scatter light. Without limiting ourselves to any particular theory, it is believed that certain conditions are more likely to favor or inhibit the formation of glass platelets. For example, glass vials produced by tubing and/or high temperatures are generally less resistant to platelet formation than molded glass vials. Drug solutions prepared at high pH (alkaline) and with certain buffers such as citrate and tartrate also promote platelet formation. The length of time that the product is exposed to the interior of the container, as well as the temperature of the drug product, also affect the likelihood of glass platelet formation. For more information, see, for example, the US Food and Drug Administration's Guidance to Drug Manufacturers: Glass Plate Formation in Some Injectable Drugs (March 25, 2011) (www.fda.gov/Drugs/DrugSafety /ucm248490.htm), incorporated herein by reference in its entirety.

To create a differentiation system based on this principle, the imaging device can be conventionally fitted to the bottle such that light is incident through the bottom of the container (orthogonal to the axis of the camera). This configuration allows receiving only a small signal from light-scattering particles (for example, proteins), and a strong signal from particles that reflect light (for example, glass plates). In other words, as the plates float through the vessel, they periodically sparkle in the image. This technique has high specificity for differentiating platelet particles from protein aggregates. In addition, the signal obtained using the specified imaging technique correlates with the concentration of platelets inside the vial. As a result, this technique not only can be used for non-destructive detection of platelets in commercial products, but also as a tool for identifying drug formulations that lead to increased/decreased presence of platelets.

In fig. 13A and 13B show maximum intensity projection (MIP) images of glass slabs (FIG. 13A) and protein (FIG. 13B) obtained by the disclosed visual inspection system. Known MIP images are used in computed tomography to provide visualization of three-dimensional space viewed along a single spatial axis, for example, the z-axis. A typical MIP image displays the maximum data value acquired along the optical

- 18 044111 beam parallel to the visualization axis. However, in this case, the mip images of FIG. 13A and 13B are data visualizations showing the evolution of a two-dimensional image over time, i.e. they are projections along a time axis rather than a spatial axis.

To generate the MIP images of FIG. 13A and 13B, the processor selects the maximum value of at least some pixels from the time series data, where each pixel represents the amount of light reflected (and/or transmitted) from a corresponding point in space in the vessel. By mapping the obtained values, a mip image similar to that shown in FIG. 13A and 13B, displaying the brightest previously recorded pixel value. The processor evaluates the mip image by counting the number of pixels in the mip image whose value exceeds a specified threshold. If the estimated value exceeds the previously recorded value indicating the number of plates in a similar vessel, the processor determines that this vessel is statistically likely to contain glass plates. The processor may also determine the degree of slab contamination by estimating the number, average size, and/or size distribution of glass slabs based on the MIP image.

The inventive systems can also be used to distinguish between glass slabs and other particles in a container, for example, based on differences in the volume of light reflected by the particles as a function of time and/or based on differences in the volume of light transmitted by the particles. Some non-lamellar particles may reflect light received from a light source illuminating the vessel from below (eg, source 122e in FIG. 8) onto the detector. Glass particles, metal particles, and foreign fibers, for example, may continually appear in the image when using a bottom-lit container configuration. These types of particles are consistently detected as they move through the container, unlike platelets, whose detection depends on their spatial orientation, and which are only visible for a few frames each time their location causes them to reflect light towards the imager. For images of time series data under illumination from below, particle tracking can be applied to track continuously visible but moving particles. These trajectories can then be removed from the MIP calculation used to estimate the number of platelets, or, in yet another embodiment, can be included in a method for obtaining global light information to determine the type of interaction of a single particle with other spatial orientations of illumination. For example, a metal particle that reflects light can be tracked using a bottom-lit configuration. However, the particle absorbs light when illuminated by backlighting (eg, by light source 122f in FIG. 8). By using both of these indicators, differentiation of a metal particle and a glass particle that reflects light when illuminated from below, but does not absorb background light, can be ensured.

Particle detection, tracking and characterization.

According to the above description, the visual inspection unit 100 of FIG. 1 can record a high quality, high resolution monochrome image stream (time series data) of bright particles imaged against a dark background. (In yet another embodiment, the particles may be displayed as dark dots on a white background.) Because a drug product may contain a wide range of fundamentally different particles, time series data can be analyzed using a number of different approaches to differentiate between objects. image and background. Often, the appearance of a particle in a single image (time series data frame) is not enough to accurately quantify particularly important metrics (e.g., particle number/size). For example, the apparent image of one particle in one frame of time series data may actually be an image of two or more particles colliding with or passing each other, which may lead to inaccurate particle counts and/or particle size estimates.

Temporal correlation of objects in an image across multiple frames of a video sequence can improve the accuracy of counting the number of particles and measuring their sizes. The process of matching objects across successive frames to produce a time-dependent trajectory for each particle is called particle tracking, registration, or particle assignment. There are particle tracking techniques intended for other applications (for example, experimental fluid dynamics). However, in these areas, well-defined spherical tracer particles are typically used. The application of this principle in the field of drugs and other liquids requires a much more complex solution. In addition, for some types of particles, time analysis (tracking) is not always practical. In such cases, a statistical approach can be used as an alternative to obtain measured particle characteristics.

In fig. 14 provides an overview of the high-level detection and identification process for 1300 particles, starting with the acquisition of 1310 time series data. The time series data (and/or reverse time series data) is preprocessed 1320, after which the preprocessed reverse time series data is used for two-dimensional identification and measurement

- 19 044111

1330 particles, which may include statistical analysis 1340 and/or particle tracking 1350 based on reverse time series data. As described above, reverse time series data is time series data whose frame order is presented in reverse chronological order. The generation of the 1360 report occurs after the identification and measurement of 1330 particles is completed.

Preprocessing of time series data.

Pre-processing 1320 includes removing 1321 static elements (background subtraction), suppressing/filtering 1322 noise in the image, and determining 1323 a brightness threshold. Removing 1321 static elements takes advantage of the fact that rotation of the container excites the liquid and the particles it contains. Their dynamic movement allows them to be distinguished from other objects in the image. Due to the fact that the imaging process is started after the container has completed its rotation, any moving object is considered a possible particle. Therefore, static elements are not important and can be removed from the image for the sake of clarity.

In one implementation, the minimum brightness projection allows the generation of an illustrative template for static elements in the image. Static elements include, for example, scratches, dirt and defects that may be present on the wall of the container. This image of static elements can then be subtracted from the entire sequence of video images to form a new sequence of video images containing only moving elements against a dark background. For example, in FIG. 15A and 15B show one frame of time series data before and after removing static elements. Glare, scratches, and other static elements obscure some areas of the container in FIG. 15A. Background subtraction removes many static elements to produce an image (FIG. 15B) containing more clearly visible moving particles.

The problem with this approach is that most glass defects, such as surface scratches, scatter a relatively large amount of light and appear bright white in the resulting image due to saturation of the detector pixels. Subtracting these elements may result in dead areas in the image. As particles pass behind or in front of these highlighted defects, the particles may be partially obscured or may even disappear completely. To solve this problem, an image of static features can be stored, analyzed and used to correlate defect locations with particle positions to minimize the impact of surface defects on sizing and particle counting data. (It should also be noted that pre-cleaning of the vessel before using the system is recommended to remove as much surface defects as possible.) The data can also be filtered 1322, for example, to remove high-frequency and/or low-frequency noise. For example, applying a spatial bandpass filter to (inverse) time series data allows data above the first spatial frequency or the second spatial frequency to be removed and/or suppressed.

After removing background elements, the time series data is thresholded 1323 by adjusting the brightness value of each pixel to one of a specified number of values. The black and white images shown in FIG. 16A and 16C, and normalized according to the eight-bit scale shown on the left (other possible scales include 16-bit and 32-bit scale). Each pixel has a brightness value from 0 to 255, with a value of zero representing no light detected and a value of 255 representing the highest amount of light detected. Reducing the specified brightness values (127 and below) to zero, and normalizing the specified brightness values (128 and above) to 255 produces the black-and-white images shown in FIG. 16B and 16D. One skilled in the art will appreciate that other threshold values (as well as multiple threshold values) are also possible.

Particle detection.

Effective detection of particles in an image is based on a number of image processing and segmentation techniques. The term segmentation refers to a computational process in which objects of interest in an image are simplified into discrete, easy-to-use elements. Segmentation methods for extracting objects from an image are widely used, for example, in the field of medical imaging, and these techniques are used to identify particles. In general, images received from a camera are preprocessed using luminance thresholding, background subtraction (static elements), filtering (eg, bandpass filtering), and/or other techniques to maximize contrast. Upon completion of the pre-processing, the processor 130 segments the image and classifies the specified areas accordingly. Suitable segmentation approaches include, but are not limited to, trust junction, watershed, level function, graph partitioning, compression-based approach, clustering, region growing, multi-scale approach, edge extraction, and histogram-based approaches. Once images are acquired, segmentation can provide additional information to correlate a specific feature in the acquired image with the appearance of the particle. For example, information about a specific segmented element, such as

- 20 044111 area, perimeter, brightness, clarity and other characteristics can then be used to determine the type of particle.

Particle tracking and time series reversal.

Of particular importance is the fact that none of the prior art particle identification tools fully takes into account the temporal behavior of the particles as they move through the vial. When measurements are taken from a single photographic image, particle counting and sizing may not be accurate. However, time series data provides a more complete picture of particle behavior, which can be revealed by tracking 1340 particles, allowing the generation of time-dependent dynamic tables for each individual particle, providing a much more reliable and accurate measurement of its basic properties. Particle tracking is a technique widely used in video microscopy, as well as in engineering fluid dynamics (in the latter field, this technique is often called particle tracking velocity measurement).

Although measuring velocity by tracking particle trajectories is known in the art, most particle tracking solutions assume that particle movement between successive frames of a video image is negligible and that the movement is less than the standard distance between particles in a single image. In such cases, it is sufficient to link particle positions by determining the nearest neighboring particles in two images. However, in many applications this model is not suitable. Due to high rotation speeds (e.g., approximately 300 rpm, 1600 rpm, and/or 1800 rpm) and potentially high particle concentrations, it is likely that particles will move between successive frames at distances significantly greater than normal. distance between particles. This problem can be solved by using some form of predictive tracking that involves searching for a particle in an area predicted based on the particle's previously noted movement. Predictive tracking involves evaluating physics equations to make a mathematical prediction of the particle's approximate future position at the next frame, as shown in FIG. 17. To improve performance, this predictive tracking phase may be supplemented with local fluid behavior data (if available), for example as described with reference to FIG. 21C.

Forming an accurate forecast of a particular trajectory may require elements of known data on which to base the trajectory construction. The problem is that at the beginning of the image sequence, when the particles are moving at their highest speed, there may be little or no known data on which to base the position prediction. However, over time, friction against the wall in the container leads to a slowdown and subsequent complete stop of the movement of the rotating fluid. By capturing time series data over a sufficient period of time, it is possible to obtain frames in which particles slow down significantly or even stop.

Reversing the timeline of a video image 1331, whereby the particles in the image are initially stationary and slowly accelerate as the video image is viewed, provides elements of known data for trajectory determination. At the beginning of the video image, when the particles are now moving very slowly, the closest match principle can be used to plot the initial phase of each of the trajectories. Then, at the appropriate point in time, the system can enter predictive mode. This reversal of the chronology of received data significantly improves productivity.

In fig. Figure 17 shows an overview of the predictive tracking process with history reversal. The task of particle tracking is to form a connection between the position of particle ai on frame i with its position a _i+1 on frame i+1, according to FIG. 17(a). This process does not cause difficulties if the distance of movement of particle a between frames is less than the distance d from the nearest neighboring particle b. In the case where the direction of particle movement is unknown or random, the simplest methodology is to provide a search area, usually a circle with radius r _s , with the value of r _s chosen so that it is greater than the expected range of particle movement, but less than the standard distance d between particles, according to Fig. 17(b). After reversing the video history according to FIG. 17(c), the particles in the image begin to move slowly. However, after a while, the particles in the image accelerate, and the closest match search method may begin to produce incorrect results. The first few frames of backward time series data partially define the trajectory, providing some data about the particle's velocity and acceleration. This information can be plugged into appropriate equations to provide a prediction of the approximate location of the particle in frame i+1, as shown in FIG. 17(d). The tracking method described above is significantly more efficient than simple nearest match tracking, especially in samples with high particle densities and/or high particle velocities.

Detecting the center of mass.

- 21 044111

In fig. 18A and 18B show the detection of the center of mass for particles on (reverse) time series data after determining the brightness threshold. First, processor 130 converts the grayscale image (FIG. 18A) into an image with a certain luminance threshold (FIG. 18B). Each particle in the image is represented as a two-dimensional projection, the shape and size of which depend on the shape, size and spatial orientation of the particle at the time the frame was taken. The processor then calculates the geometric center, or center of gravity, of each of the two-dimensional projections (eg, indicated by coordinates xi and yi) by any suitable method (eg, vertical method, geometric decomposition, etc.). Processor 130 may perform a frame-by-frame comparison of the location of a particular particle's center of gravity to determine the particle's trajectory.

Occlusion of particles.

Each of the visual inspection systems disclosed herein projects a three-dimensional volume (the container and its contents) onto a two-dimensional surface of the imaging sensor. When using a single 2D sensor, particle trajectories can intersect in a 3D volume. In this case, one particle can partially or completely block another, according to Fig. 19. In FIG. 19(1), a new particle is identified in the image sequence; tracking a particle through a sequence of images allows one to obtain a series of sequential stages, according to FIG. 19(2). A search area is used to search for potential matches on successive frames, as shown in FIG. 19(3). Periodically, several probable particles are located in the search area, according to FIG. 19(4), in which case the system selects the best match. One skilled in the art will appreciate that the best match can be selected through any combination of different approaches. For example, data representing a probable particle in one frame can be compared and/or correlated with data representing a particle in a previous frame. Comparison and/or correlation of parameters, including, but not limited to, size, shape, brightness and/or changes in appearance in the image, provide a match for a likely particle. The disclosed visual inspection systems are configured to distinguish between collisions, occlusions, and temporary particle disappearances, such as the occlusions of FIG. 19(5). After re-detecting the particle according to FIG. 19(6), the particle path can be constructed anew. The claimed systems also allow conflicts caused by the collision of two paths (and their search zones) to be resolved, ensuring the construction of correct trajectories, according to FIG. 19(7).

In fig. Figure 20 shows another case of particle obstruction in a two-dimensional image: Figure 20(a) is a conventional image of suspended particles. In fig. 20(b)-(e) show enlarged views of the selected area of FIG. 20(a), in which two particles approach each other from opposite directions. The following frames of (reverse) time series data show that occlusion causes two particles in the image to appear as one, unnaturally large particle. In the case of partial occlusion (Fig. 20(c)), in such an image the particles may appear as one unnaturally large particle. In the case of complete occlusion (Fig. 20(d)), the smaller particle may disappear completely from view, and the total number of particles counted may decrease by 1 particle. This situation can be extremely important in drug research due to the fact that an artificially high size measurement may be sufficient to exceed regulatory guidelines despite the fact that the drug in question contains only acceptable, invisible particles. By the time shown in FIG. 20(e), the particles have moved beyond each other and independent tracking can be resumed. The visual inspection system can automatically correct errors caused by occlusion by analyzing particle trajectories and subsequent time-dependent size profiles, resulting in lower false failure rates.

Counting lost particles.

As described above, particles may disappear from a portion of a video sequence for a number of reasons. Particles may pass through a blind spot and/or dead spot created by removing static elements as described above. Finally, some types of particles may exhibit optical behavior that involves them appearing and disappearing (glare) when viewed through imaging optics. In such cases, the processor may predict the movement of said lost particles as described below. When a particle reappears at its expected location within a specified time period, the processor can link trajectories and interpolate virtual particle data for intermediate frames. It should be noted that, from a regulatory perspective, it is important to provide appropriate labeling of virtual particle data to ensure that said virtual data can be distinguished from true, measured particle data.

In fig. 21A-21C show one technique for tracking and re-locating lost particles, i.e. particles temporarily disappearing from view during a video sequence. The disappearance can be caused by occlusion behind another (larger) particle, occlusion behind a surface defect, passing through a known blind spot, or simply a property of the particle's optical geometry (for example, some types of particles may only be visible at certain spatial orientations). Finding or re-detecting particles that disappear from view can improve the accuracy of particle detection and identification.

In fig. 21A shows predictive tracking for detecting a particle occluded by a defect on the surface of a container. A surface defect scatters a significant amount of light, saturating the corresponding area of the image. After removing static objects, the specified area becomes a dead zone in the image. Any particles passing through the designated area will temporarily disappear. Processor 130 may rediscover lost particles by generating virtual particles in a limited number of steps. If a particle reappears and is detected, the particle paths are combined.

Specifically, processor 130 uses predictive tracking to determine the speed of a particle before it disappears. The processor can also use predictive tracking and particle velocity to extrapolate the particle's expected location. If the particle reappears at the expected location, the virtual locations can be linked together to form a complete trajectory. If a particle does not reappear within a given period of time, it can be marked as irretrievably lost and tracking is stopped.

In fig. 21B shows a method for tracking a particle experiencing a significant acceleration or change in direction during a period when the particle is invisible. Instead of predicting a particle's trajectory, processor 130 retrospectively associates fragments of trajectories using data about the nature of local fluid behavior. In this case, processor 130 combined the trajectories by considering the characteristics of laminar fluid flow at a given speed and scale.

In fig. 21C shows the disappearance and reappearance of particles as they pass through known blind spots. In this example, the particle passes through a known blind spot at the far edge of the container. Programming processor 130 using information about the location of a blind spot in an image of the container allows processor 130 to reconstruct the trajectory.

Irregularly shaped particles.

Some particles are not spherical or small enough to be considered as a point, which is used in most particle tracking techniques. As such, many of the particles are irregularly shaped and may roll and rotate relative to the chamber as they move through the fluid, as shown in FIG. 22A-22C. In some cases, an irregularly shaped particle may appear in the image as two separate particles, each with its own trajectory, as shown in FIG. 22V. Such unpredictable movement of the established center of mass of a two-dimensional object can hide the true movement of the particle. This behavior seriously complicates the predictive tracking process. The visual inspection system of the present invention may have a function to take into account the apparently distorted movement of an irregularly shaped particle, for example by calculating an average trajectory, as shown in FIG. 22A and 22C, irregularly shaped particles.

Specific hydrodynamics of the container/product.

The movement of particles in the container upon completion of rotation is the result of a combination of fluid movement and gravitational influence. Fluid movement is a function of fluid viscosity, fill volume, container shape and size, and initial rotation speed. Particle tracking performance can be significantly improved by incorporating knowledge of the physical constraints of the fluid system into the trajectory generation process.

The hydrodynamics of fluids swirling in known containers can, under certain conditions, be surprisingly complex. The incorporation of fluid dynamics data (for containers commonly used in pharmaceuticals) into the trajectory generation process represents an area of significant innovation and development compared to the state of the art.

In fig. 23 shows some examples of fluid behavior in known containers, with the results of the computational model compared to actual particle trajectories obtained through a visual inspection platform. During the study, unexpected subtleties were discovered: for example, in Fig. 23(d) shows the movement of particles along a narrow vertical column in the center of the vial, caused by the weakening of the vortex formed during the rotation phase (Fig. 23(a)). As the fluid moves upward vertically in said central column, the fluid can pick up and carry upward heavy particles that should theoretically sink. This situation may, for example, lead to errors in the identification of bubbles that theoretically should float and foreign particles that float due to fluid movement specific to a given container.

The claimed visual inspection systems can use known data about the expected hydrodynamics of drugs to obtain more accurate results, which is not possible in other cases. Combining a physical model like that shown in FIG. 23, with particle tracking, represents a significant improvement on existing technology.

Error correction.

- 23 044111

Although the visual inspection systems of the present invention demonstrate reliability under most experimental conditions, the complexity of the task of tracking large numbers of particles moving in a small three-dimensional volume implies a constant risk of errors, mainly resulting in the formation of incorrect trajectories between successive frames upon collision. particles. This phenomenon is shown in Fig. 24A.

Understanding the physical limitations of a visual inspection system can be an advantage. In general, the primary mode of local fluid motion around each particle is laminar motion (rather than turbulent or random). Essentially this means that, given a fast enough camera, the natural trajectories of the particles in a given system should vary slightly without sudden abrupt changes in direction, particularly as the particles pass the center of the container shown in the image. Once the initial linking of trajectories is completed, the system can retrospectively analyze the trajectories to detect similar errors. If errors are detected, the system can compare nearby trajectories in order to possibly detect a more physically consistent solution. This process is shown in Fig. 24V.

Accurate particle counting.

The number of particles can be inferred based on counting the number of particles in a photographic image taken at one point in time (eg, according to FIG. 24A) after detection of the particles, each particle being assigned a serial number. This approach is simple, but when used, the calculation of the number of particles in a volume is often incomplete for a number of reasons. For example, one or more particles may be obscured by another particle or a surface defect. Particles may be located in known (or unknown) blind spots. In addition, extremely small or faint particles may periodically appear and disappear from view as they move between measurement limits.

One of the advantages of particle tracking disclosed herein is that the process addresses all of these issues. As a result, to ensure reliable particle tracking, particle counting can be improved by counting the number of individual particle paths (as per FIG. 24B) rather than the number of particles in a single image or statistical analysis of multiple images. Counting the number of particle trajectories instead of the number of particles in a single frame (or set of frames) represents a significant advance over known particle tracking techniques. The increase in accuracy varies depending on the number and size of particles present. In general, as the number of particles increases, the likelihood of occlusion increases and, accordingly, the accuracy improvement due to the timing capabilities of the claimed particle tracking process increases proportionally.

Accurate particle size calculation.

Known particle measurement systems measure particle size based on static images. Most often this process is carried out by measuring the length of the longest apparent axis of the particle, or Feret diameter, according to FIG. 25, in accordance with regulatory and/or industry standards, which may take the particle size to be the largest extent of the particle. According to this definition, a hair 1 mm long is in the same classification category as a spherical particle with a diameter of 1 mm. Considering the above, the maximum Feret diameter is a reasonable measurement based on a 2D image. However, measuring particle size from static images poses a number of significant challenges.

First, in a two-dimensional projection of a three-dimensional volume, several particles can easily overlap each other to form a single, much larger particle in the image. In an industry where regulatory authorities impose very strict upper limits on acceptable particle size, the above is a very significant problem, particularly in the manufacturing field where such an approach can lead to spurious failures, particularly when studying samples with dense particle populations.

Second, irregularly shaped particles can rotate unpredictably (relative to the chamber) as they move through the container. When using a single two-dimensional photographic image, it may not be possible to accurately ensure that the greatest extent of a single particle is orthogonal to the camera's viewing axis. Consequently, the system may systematically underestimate particle sizes, which can have highly undesirable consequences in a highly regulated industry. Studying the time-dependent maximum Feret diameter of a particle as it moves through a container through particle tracking provides a significantly more accurate measurement of the particle's greatest extent.

Third, as the particles move through the cylindrical container, the length axis of the particles generally corresponds to the direction of movement of the ambient fluid current, as shown in FIG. 25A and 25B. In general, when using a cylindrical container, this means that elongated particles may appear larger in the center of the image compared to the far side edges. Forming device

- 24 044111 images typically detect the maximum apparent particle size (Feret diameter) when moving the particle orthogonally to the optical axis of the imaging sensor. By tracking a single particle as it passes through a container, the correct maximum elongation of the particle can be accurately measured, something that is difficult to achieve using a static measurement procedure.

Finally, despite every attempt to minimize the effect of motion blur by strobing the illumination (as described above), it is still possible for some image motion blur to occur early in the imaging sequence when fluid and particles are moving at maximum speed. By using time-dependent particle size analysis, data artifacts caused by image blur (which typically increases the measured particle size) can be detected and suppressed.

In fig. 25C-25E illustrate the process of using time series data to track particle trajectories to provide more accurate particle size measurements. In fig. 25C shows a typical path of a 100 micron polymer microsphere through a vial after it has been rotated. The particles move most quickly relative to the camera due to the fact that in the image they cross the center of the container at the moment when their speed is orthogonal to the direction of inspection, according to Fig. 25D. For example, if the initial rotation speed is 300 rpm and the radial location of the particle rp is 5 mm, then the speed of the particle vp is approximately 9.4 m/s. At this speed, the camera's exposure time of only 10 μs doubles the apparent particle size due to image blur. In fig. Figure 25E shows how severely motion blur can affect images: on the left, particles move quickly (about 300 rpm) and appear elongated; on the right, the same particles are motionless and appear more round.

In fig. 25F is a plot of Feret diameter versus time for the particle shown in FIG. 25C. Due to the lensing effect of the cylindrical container, the apparent particle size near the edge of the container (point D on the right axis) is reduced. The most accurate estimate of the maximum particle size was made when the particle passed the center of the container at low speed (point B on the right axis). If the speed is too high (which usually occurs within the first few seconds after the container is rotated), the blurring of the image causes the apparent size of the particle to increase (point A on the right axis). Over time, under the influence of the friction force of the fluid, the particle completely stops moving (point C on the right axis). In this case, the mid-range peak values (point B on the right axis) represent the most accurate indicator of the maximum particle size.

Determination of particle characteristics.

In fig. 26A shows successive frames of time series data containing particles and their trajectories. The roughly planar paths represent the trajectories of 100-μm polymer microspheres mimicking protein aggregates. These particles, which have practically neutral buoyancy, move with the fluid and do not float or sink significantly. The vertical downward paths represent the trajectories of 100-micron glass beads that initially rotated with the fluid but then sank as the sequence progressed. The upward paths represent the trajectories of air bubbles and particles with positive buoyancy.

Particle tracking allows the measurement of a number of time-dependent properties that can provide important clues to the nature of the particles being studied. For example, air bubbles, generally considered harmless from a regulatory perspective, can cause errors in known optical inspection mechanisms, resulting in false positives and false failures. In this case, the time-dependent movement of the particle (air bubbles typically rise in a vertical direction as the fluid slows down) provides a very obvious characteristic that can be easily identified based on the trajectory obtained by tracking the particles. Similarly, neutrally buoyant particles may not float or sink to a significant extent, while dense particles sink and fall to the bottom of the container. Lighter particles can be caught up in the vortex created by the rotating fluid, while heavier particles can follow straight trajectories.

Overall, the particle tracking process provides a time-dependent dynamic table similar to that shown in FIG. 26B, and containing details regarding all parameters of interest, including location, speed of movement, direction of movement, acceleration, size (ie, two-dimensional area), size (maximum Feret diameter), elongation, sphericity, contrast and brightness. These parameters provide a set of characteristics that can be used to classify a particle (attribute it to a certain type). This approach, feasible through a particle tracking solution, works well for most particles of interest. The ability to categorize particles separately on a per-particle basis based on such a range of time-dependent measurements is a particular advantage of the present invention.

- 25 044111

Video compression.

The use of extremely high-resolution imaging sensors facilitates the imaging of extremely small particles in a relatively large container. To ensure accurate trajectory generation, it is also necessary to maximize the imaging speed. The combination of these requirements results in extremely large video file sizes, such as 1 GB, 2 GB, 5 GB, 10 GB, or larger. In some applications, it may be necessary to archive the original video image in addition to the analysis data. Even for moderately sized sample sets, large file sizes can result in prohibitive data storage costs.

To reduce the size of the (reverse) time series data files, video compression of the (reverse) time series data can be applied. To maintain particle data accuracy, it may be necessary to use lossless video compression. Research shows that more widely used (and more cost-effective) lossy compression systems (such as MPEG) can significantly distort and introduce noise into the image, producing a number of unwanted visual artifacts.

Although lossless compression is generally relatively inefficient compared to lossy compression, a number of steps can be taken to improve the efficiency of lossless compression. Most time series data frames show a small number of small, bright objects against a dark background. The dark background does not contain useful information. The background is not completely black, but is formed from extremely weak random noise. Replacing the specified background with a completely black background greatly simplifies the image, and makes standard lossless compression methods (such as zip or Huffyuv) work much more efficiently.

This process is known from the prior art. The innovation of the present invention lies in making a specific decision about what actually constitutes the background in a given frame. Other compression processes set a brightness threshold and assume that all pixels in the image whose brightness value is below the threshold are part of the background. In a broad sense, this approach is effective, but may result in a small reduction in the size of the image of the particles being retained, and may also result in the complete removal of extremely faint particles with a brightness of the same order of magnitude as the upper bound of the random background noise inherent in the image.

Although these known techniques are capable of working with (reverse) time series data, the compression used in the illustrated embodiments includes a unique step that provides background analysis to detect faint particles before using a destructive thresholding process. This process provides the best balance between maintaining data accuracy and minimizing data storage requirements.

Determination of filling volume and meniscus.

Automated implementation examples of the visual inspection platform allow accurate determination of sample loading volume, which is important in research applications where the filling volume cannot be guaranteed to be constant throughout the experimental period. This process is especially useful when working with particularly large data files, such as files generated by high-resolution image sensors that put pressure on data transmission and storage. For this reason, it may be desirable to limit the captured image so that it only covers the volume of the fluid, since any additional information is not of interest.

The systems shown may, for example, use automated edge detection or object recognition algorithms to detect the boundaries of a container in an image, as shown in FIG. 27-29 and the description below. Due to the fact that the meniscus and the base of the vial are unique objects, a number of possible lighting configurations and/or image processing techniques can be used to accurately identify their position in the image. The area of interest is determined by measuring the volume of the fill and determining the area of the image occupied by the fluid. In particular, according to FIG. 8, configurations using light sources 122f (backlight), 122e (bottom illumination), and a combination of sources 122a and 122b (backlight) can be used to determine the filling amount as described below.

In fig. 27A-27F show automatic detection of a region of interest within a container using rear corner illumination 122a and 122b of FIG. 8. In FIG. 27A shows a static image of a container in which the base of the container and the meniscus are clearly visible as sharp, bright objects. As an example, the processor may use edge extraction to identify the vertical walls of a container and the width w of a region of interest, as shown in FIG. 27V. To detect the meniscus and base of the vial, which may be less predictable in appearance, the process may utilize luminance thresholding and segmentation to provide a simplified image of the region of interest (shown in FIG. 27C). At this point, the processor may automatically identify containers that may be unsuitable for particle analysis, e.g.

- 26 044111 items whose surfaces are scratched and/or covered with dirt. The effectiveness of the system can be compromised by excessive turbidity, defects in the surface of the container, or excessively high particle concentrations (with the result that individual particles can no longer be identified in the image). If the processor determines that the container is suitable, the objects corresponding to the meniscus and base of the vial can then be extracted and simplified, according to FIG. 27D. The processor sets the vertical height h of the area of interest as the distance between the lower edge of the meniscus and the upper edge of the base of the vial, according to FIG. 27E. Finally, the processor may crop the original image stream using the width and height values of the region of interest, resulting in only the image region occupied by the visible fluid being captured, as shown in FIG. 27F.

In fig. 28A-28C show a similar meniscus detection process performed based on data obtained using a background lighting configuration (eg, light source 122f in FIG. 8). In fig. 28A shows a time series data frame displaying a conventional capacitance imaged using backlighting. The meniscus, walls and base are easily distinguishable and can be automatically detected by contour extraction according to FIG. 28V. However, defects such as large scratches may interfere with the accuracy of the meniscus detection process when using backlighting (FIG. 28B) or backlighting (eg, according to FIG. 29C, described below). In one embodiment, an image brightness threshold is used to identify the meniscus and the base of the vial. Due to the fact that these objects are relatively large, and also due to their shape, they scatter a relatively large amount of light towards the detector, and they can be easily identified due to the fact that they are significantly different from any other objects that may be present.

In fig. 29A-29D show detection of a meniscus in a cylindrical vessel with an approximately planar bottom. Automatic determination of the filling volume begins with the determination of the brightness threshold (Fig. 29A) in order to detect the meniscus, which allows the location of interest to be identified and is also a measure of the filling volume. Then, in FIG. 29B, angled lighting allows you to highlight surface defects such as scratches (shown in the drawing), dust, fingerprints, glass defects or condensation that can make it difficult to highlight contours. Illumination of the bottle from below (eg, via light source 122e of FIG. 8), according to FIG. 29C highlights the meniscus in a manner that is (relatively) insensitive to surface imperfections - in this example the meniscus is visible even though the surface is heavily scratched. Lighting from below also makes it possible to differentiate between empty vials and full vials, according to FIG. 29D, and accurately determine the height of the meniscus at any filling level between the specified extreme values. Illuminating the vial from below improves the detection of the meniscus because it eliminates errors caused by scratches and other surface defects (Fig. 27C). Arranging the light source 122e to illuminate the vessel at a shallow angle further reduces sensitivity to surface defects. When lighting syringes, which are difficult to illuminate from below due to the lack of a transparent base of the container, a similar effect can be achieved by lighting at a low angle.

Test techniques similar to the meniscus detection described above can also be used to screen for items that may adversely affect any subsequent attempts to identify and analyze fluid-borne particles. Such options may include identifying excessively turbid liquids, severely damaged containers (including a large number of scratches or debris on the surface), and liquids in which the concentration of particles is so high that individual particles can no longer be identified in the image.

Processors and memory.

One of ordinary skill in the art will appreciate that the processors disclosed herein may include any suitable device for processing and storing data and including input/output devices for running application programs and the like. Example processors may be embedded in integrated circuits, field programmable gate arrays, and/or any other suitable architecture. The disclosed processors may also be coupled to other computing devices, including other processors and/or server computers, via communications networks. The communication network may be part of a dial-up network, a wide area network (for example, the Internet), a worldwide group of computers, a local area network or a wide area network, as well as gateways that use appropriate protocols (TCP/IP, Bluetooth, etc.) to provide mutual connection. Other architectures of electronic devices and computer networks are also possible.

In fig. 30 is a diagram of the internal structure of an exemplary processor 50. The processor 50 includes a system bus 79, where the bus is a set of hardware lines used to transfer data between components of a computer or processing system. Bus 79 is essentially a common channel that connects various elements of a computer system (eg, processor, disk drive, memory, I/O ports, network ports, etc.) and allows information to be transferred between elements. Connected to system bus 79 is

- 27 044111 I/O device interface 82 for connecting various input and output devices (for example, keyboard, mouse, displays, printers, speaker, etc.) to the processor 50. Network interface 86 allows the computer to communicate with various other devices connected to the network. Memory 90 provides volatile and/or non-volatile storage of computer software instructions 92 and data 94 used to execute example implementations of the disclosed visual inspection systems and techniques. Disk drive 5 provides (additional) non-volatile storage of computer software instructions 92 and data 94 used to perform the visual inspection example implementation. The processor central unit 84 is also coupled to the system bus 79 and provides execution of computer commands.

In one embodiment, the processor operation 92 and data 94 is a computer program product (generally designated 92) comprising a computer-readable medium (e.g., a removable storage medium such as one or more DVD-ROM disks, CD-ROM disks, floppy disks, magnetic tapes, etc.) providing at least a portion of the software instructions for the disclosed visual inspection systems. The computer software product may be installed using any suitable software installation procedure known in the art. In yet another embodiment, at least a portion of the program instructions may also be downloaded via a cable, communications, and/or wireless connection. In other embodiments, the exemplary programs are a signal product 107 transmitted by a computer program, configured as a transmitted signal in a propagation medium (e.g., a radio wave, an infrared wave, an audio wave, or an electrical wave transmitted over a wide area network such as the Internet, or over another network or networks). Said media or signals provide at least a portion of program instructions for the disclosed processes or program 92.

In other embodiments, the transmitted signal is an analog carrier wave or a digital signal transmitted over a propagation medium. For example, the transmitted signal may be a digitized signal transmitted over a wide area network (eg, the Internet), a long distance network, or other network. In one embodiment, the transmitted signal is a signal transmitted in a propagation medium over a period of time, such as commands to a software application sent as data packets over a network over a period of several milliseconds, seconds, minutes, or more. In yet another embodiment, the computer-readable medium of the computer program product 92 represents a distribution medium that can be received and read by the processor 50, for example, by receiving the distribution medium and identifying a transmission signal contained in the distribution medium as described above about a signal product transmitted by a computer program.

In general, the term medium or temporary medium covers described transient signals, transmitted signals, propagation medium, storage medium, and the like.

Cooling the sensor.

In the above-described embodiments, electronic sensors are used to obtain images of the particles. Electronic sensors, such as CCD sensors, are subject to various types of random noise that interfere with the accuracy of the measured signal, especially when receiving a low signal strength. In some embodiments, the sensors may be cooled to reduce noise. Cooling may be accomplished using any suitable technology, including, for example, the use of thermoelectric refrigerators, heat exchangers (eg, cryocoolers), liquid nitrogen cooling, and a combination of the above technologies.

In various embodiments, noise reduction provides an advantage in particle detection, particularly associated with detection of protein aggregates. In standard applications, protein aggregates may be relatively large (e.g., up to several hundred microns in diameter), but the physical structure of these aggregate particles is often very loose, with low density (a significant portion of the particle may be porous and filled with the surrounding medium) and low refractive index contrast relative to the surrounding environment. Due to these physical properties, protein aggregates can scatter relatively small amounts of light compared to other particles such as glass fragments or fibers.

Much of the noise affecting modern electronic image sensors is thermal in nature. This noise affects the lower part of the dynamic range of the sensor. For example, in some embodiments, the lower X% (eg, 10%) of the dynamic range is occupied by noise and must be removed during the image luminance threshold determination process (eg, as described above). The minimum threshold value for particle detection must exceed the specified value ~X%, which allows low brightness data to be removed from the signal. This removal may prevent the accurate detection of dim particles such as protein aggregates. By reducing noise, for example by cooling the sensor, a lower threshold can be used, allowing for improved detection of low brightness signals.

- 28 044111

In fig. 31 illustrates the above-described problem associated with determining a threshold value. In panel A of FIG. 31 shows a trimmed segment of a typical data sequence obtained by the inventive devices and methods. According to the drawing, the images are eight-bit halftone images, i.e. each pixel can have a brightness value that varies linearly from 0 (black) to 255 (white). The image contains two particles, one of which is relatively bright and the other is extremely dim. In panel B of FIG. 31 is a luminance histogram showing background luminance data corresponding to a region of the image that does not contain particles.

The sensor exhibits a Gaussian background noise curve at the lower end of the luminance histogram due at least in part to thermal effects. The width of the specified curve determines the threshold value for particle detection. In general, particles must be significantly brighter than the background noise to avoid being removed when determining the brightness threshold.

In panel C of FIG. Figure 31 shows a brightness histogram for a bright particle. The particle image contains a significant number of pixels located to the right of the threshold value in the histogram, and hence the particle is easily detected once the threshold value is determined.

On the other hand, according to panel D of FIG. 31, the fainter particle contains a relatively small number of pixels above the threshold and is likely to be removed during the thresholding and segmentation process. However, by using cooling or other noise floor techniques that shift the threshold to the left, it is possible that a fainter particle may be detected.

Light-based counting and non-destructive sizing (LENS).

In some embodiments, when performing non-destructive sizing and counting of particles in a container, there are noticeable artifacts caused by the container itself. The liquid interface refracts light passing through the vial, causing noticeable distortion in the image or images of particles used during the sizing and counting procedure. As a result, particles of a given size can appear, for example, up to 4 times larger in the image, depending on their spatial location in the vial. It should be noted that when using a cylindrical container, the particle image is usually stretched only along the lateral axis, but not along the vertical axis of the bottle. (See FIG. 5E for these effects.)

As described above, in some embodiments, these distortion effects can be corrected (eg, reduced or even eliminated) through corrective optics techniques. However, in some embodiments, such optical correction may be incomplete or unavailable. In such cases, a direct correlation between the particle size and the corresponding detector image is not possible.

For example, in FIG. 32 shows a size-defined image histogram of a population of standard size particles (polymer microspheres) in a fluid obtained by a system in which capacitance-induced distortion is not corrected (corresponding to the situation shown in FIG. 5E). Significant variability in apparent particle size due to capacitance distortion effects is clearly visible.

This variability makes it difficult to differentiate between populations of particles of different sizes due to the fact that there may be significant overlap in the size value for each particle population at an obvious location on the detector. For example, in FIG. 33 shows histograms of a certain image size for two populations of standard size particles (in the drawing they have a diameter of 100 μm and 140 μm) in a fluid. The significant overlap between the two histograms for two populations of particles of different sizes is clearly visible.

In some embodiments, a processing method may be applied to obtain accurate dimensional information even in the presence of the above-described distortion effect. The machining process is calibrated using data obtained using known dimensional standards. For example, in FIG. Figure 34 shows experimentally obtained apparent size histograms for four different populations of standard size particles (polymer microspheres). Although four calibration curves are shown, any suitable number of curves may be used in various embodiments. In some embodiments, at least two, at least three, at least four, at least five, or at least six curves may be used. In some embodiments, the number of curves ranges from 2 to 100, or any subrange, such as 4 to 6. In some embodiments, a set of experimental calibration curves may be interpolated to generate additional curves (e.g., corresponding to size values between experimentally measured values).

In some embodiments, the calibration curves may correspond to populations of particles having actual sizes that differ from each other by any suitable amount, for example, at least 1 μm, at least 5 μm, at least 10 μm, at least 20 μm , or more, in the range from 1 µm to 1000 µm or in any sublimits.

After determining the calibration curves, a distribution curve of visible times can be obtained

- 29 044111 measures for a sample containing particles of unknown size (for example, based on a static image or images, or using any other suitable technique). The sample curve may be obtained under identical or similar experimental conditions (eg, using the same or similar container size and shape, identical or similar fluid properties, lighting conditions, imaging conditions, etc.). This sample curve is compared with calibration curves to obtain information indicating the particle sizes in the sample.

For example, in some embodiments, a weighted overlay of the calibration curves is compared to the sample curve. The weighting of the overlay is varied to ensure that the overlay is consistent with the sample curve, for example, using any suitable approximation techniques known in the art. Weighting the closest fit onto the sample curve then provides information about the actual particle sizes in the sample. For example, in some embodiments, the number of times each calibration curve appears in the best fit overlay corresponds to the number of particles of a specified size in the sample.

In fig. Figure 35 shows the alignment of the overlay of the calibration curves on the experimental sample curve. In this case, the sample is prepared in such a way that the particle diameter is known to be between 75 and 125 µm. In fig. 36 shows the best-fit particle size counts compared to particle size counts obtained by simply binning the apparent size random variable values, without processing, based on the corresponding image. The raw data contains a significant number of spurious counts that are outside the actual size range of 75 to 125 µm. On the other hand, the results obtained from the best fit calibration curves show a significantly reduced number of dummy counts.

It should be noted that while one possible approach to comparing sample data and calibration data has been described, other suitable methods may be used. For example, in some embodiments, the sample curve may be decomposed using the calibration curves as basis functions, similar to the Fourier decomposition of a waveform using sinusoidal basis functions. In general, any suitable convolution, decomposition, decomposition, or other technique may be used.

In some embodiments, light-based counting and non-destructive sizing (LENS) techniques may be used in combination with the particle tracking techniques described above. For example, the LENS technique usually works best in cases where the particle shape roughly matches the shape of the standard particle sizes used to obtain the calibration data. Additionally, the techniques generally work well when there are large numbers of particles (eg, greater than 10, greater than 50, greater than 100, or more), providing a larger data set for the algorithm to process.

However, in some applications the particle count may be small. In some applications, the study focuses on larger particles in the sample. Additionally, in some applications, the sample may contain particles that are shaped differently than the standard particle size. For example, the fibers have an elongated shape, different from the spherical shape used in many standards. LENS techniques may not work effectively under such conditions.

In general, any number of particles can be counted using the techniques described above. In some embodiments, an upper limit on the number of particles that can be counted is determined based on the occlusion of one particle by another in the sample. In general, as the number of particles in a container increases, the likelihood that two particles on a 2D detector will not appear as separate particles increases. The above principle is a function of the number of particles in the volume and the particle size. In general, larger particles occupy a larger area on the detector (and therefore more occlusion events occur for the number of particles per milliliter compared to smaller particles). For example, under certain conditions, up to about 500 particles of 50 μm diameter can be counted in a 10 ^cc vial filled with 8 ml of fluid, and increasing concentrations result in undercounts and oversize effects due to particle occlusion.

However, the above particle tracking techniques can be effective in counting and sizing relatively large particles. Accordingly, in some implementations, an approach that is a hybrid of the two approaches may be used. In fig. Figure 37 shows an example of the implementation of such a hybrid process. At step 3701, a sequence of images is captured, for example, through any of the approaches disclosed herein. At step 3702, the image sequence is processed (eg, filtered, luminance thresholded, segmented, etc.). At step 3703, the particle data obtained at step 3702 may be pre-screened to detect particles whose size exceeds a threshold value. These large particles may be removed from the data set and processed at step 3704 using tracking techniques. This process can provide high-quality, time-dependent measurements of the size of large particles. If there is a background of smaller particles (below a size threshold), said particles can be processed at step 3705 using LENS techniques. The data obtained using the two different approaches can then be combined at step 3706 to generate an overall report of the particles in the test container.

In various embodiments, the size threshold used to determine the technique used may be any suitable threshold or minimum value of about 1 μm or greater, for example, ranging from about 1 to 400 μm in particle width or diameter or any subranges, for example, from about 1 to about 50 μm, from about 50 to about 250 μm, or from about 75 to about 100 μm. In some embodiments, the particle data submitted for processing using each of the techniques may be selected based on criteria other than size, such as information related to the shape of the particle. In general, any suitable combination of criteria may be used.

3D imaging and particle detection techniques.

As described above, in some embodiments, the automated visual inspection unit 100 may include two or more imaging devices 110 to provide a three-dimensional image of the contents of the container 10.

For example, in FIG. 38A-C show a block 100 containing three imaging devices 110. As shown in the drawing, the imaging devices 110 are arranged in a circle around the container 10 at 120° intervals, but more or fewer sensors may be used in different embodiments. The angles between adjacent imaging sensors are not necessarily equal to each other, but in some embodiments, an equal angle configuration allows the image processing techniques described below to be simplified.

In some embodiments, all imaging devices 110 are substantially identical. The imaging devices 110 may be adjusted such that they are all located at the same physical height relative to the container 10, and the container 10 is located in the center of the viewing area of each of the imaging devices.

In some implementations, small errors in placement may occur even if the specified physical arrangement is carefully optimized. To solve this problem, the imaging devices 110 can be calibrated by imaging a known calibration device. Any sufficiently small lateral or vertical deviations from the fit can then be detected by re-sampling and shifting the resulting images accordingly. In some implementations, the images may be processed to correct for sensitivity variability or other differences in performance between the various sensors used in the imaging devices 110.

In fig. 38C shows one image tap for the block 100. According to the above detailed description, when using a telecentric imaging configuration, it can be ensured that only rays substantially parallel to the imaging axis reach the sensor surface of the imaging apparatus 110. According to FIG. 39, using geometric techniques of ray optics (or other suitable techniques), a pattern of rays within the container 10 can be formed, passing through the wall of the container and reaching the surface of the sensor.

Given known ray vectors, a point or region can be taken from the two-dimensional image and the indicated brightness can be sent back to the container 10. By examining the horizontal rows of the two-dimensional image one at a time, a two-dimensional horizontal lattice within the container volume can be mapped. Horizontal gratings associated with each of the three imaging devices 110 may be superimposed on each other to form an overall map. By repeating this process for additional horizontal rows of sensors, a vertical stack of two-dimensional arrays can be constructed to form a three-dimensional (3D) structure, for example corresponding to all or part of the volume of container 10.

Possible particles can be identified in the resulting three-dimensional structure by determining a brightness threshold similar to that disclosed in the above description. Determination of the threshold value can be performed on the original two-dimensional images obtained from the imaging devices 110, or can be performed on horizontal maps within the three-dimensional structure after overlay.

Probable particles can be identified using a three-dimensional structure with a certain brightness threshold, resulting in a direct measurement of the particle's three-dimensional position within the fluid volume of container 10. In typical applications, the three-dimensional position measurement is accurate for most of the fluid volume, but in some cases in which the imaging devices 110 contain telecentric lenses, blind spots caused by vessel curvature and the accompanying lens effect may occur (eg, according to FIG. 39, right panel).

When using three image taps set at an angle of 120° according to the drawing, the blind spots are closely correlated in pairs (see FIG. 39, right panel). Accurate determination of the three-dimensional position in three areas of the blind spots 3901 can be eliminated. However, in these areas, location data can be determined by examining 2D data obtained from the nearest image tap.

In various embodiments, the blind spot problem can be reduced or eliminated by increasing the number of sensor taps to provide overlapping imaging.

Although one example of using multiple imaging devices 110 to obtain three-dimensional information about the contents of container 10 has been described, it should be understood that other techniques may be used. For example, in embodiments that employ two imaging devices, stereoscopic imaging techniques may be used to determine three-dimensional information.

In some embodiments, for example, in which the sample of interest is static or moving slowly, three-dimensional information can be obtained using a rotating image arm, similar to medical tomography computing devices. The rotating image tap obtains a time series of two-dimensional images taken from various points, which can be used to construct three-dimensional information, for example, through any suitable method, such as methods known in the art in the field of medical imaging. When images are acquired at high speed relative to sample dynamics, 3D imaging can provide accurate 3D information for particle detection.

In some embodiments, the three-dimensional information obtained through the techniques described above may be suitable for detecting the position of a likely particle, but are not ideal for determining other characteristics of the particle, such as particle size or shape. Accordingly, in some embodiments, a hybrid approach may be used. For example, in some embodiments, the three-dimensional location of a particle is determined based on three-dimensional information (eg, a three-dimensional structure constructed as described above). Once the three-dimensional position of the particles has been established, size and shape measurements obtained from two-dimensional images obtained from some or all of the imaging devices 110 can be associated with the position.

In some embodiments, particle tracking may be accomplished based on 3D position data, for example, through 3D tracking techniques similar to the 2D tracking techniques described above.

In some embodiments, three-dimensional tracking has advantages, particularly when used with two-dimensional images obtained from each of the imaging devices 110.

When using three-dimensional tracking, the occlusion of one particle by another (eg, according to Fig. 5E) is reduced or eliminated. In some embodiments, occlusions are possible, for example when examining dense samples in blind spots where true 3D positioning is ineffective.

Similar to the two-dimensional tracking as described above, in some embodiments, predictive tracking techniques may be used in the context of a three-dimensional image using information associated with fluid dynamics in the tank 10.

In some implementations, once tracking of 3D particle positions is complete, information about particle characteristics (eg, size and shape) may be aggregated from 2D data obtained from multiple imaging devices 110 into multiple time-dependent data sets for each particle. In some embodiments, this technique may provide a more accurate measurement of characteristics (eg, size or shape) of individual particles compared to measurement by a single imaging sensor. For example, in some embodiments, this approach allows for improved detection and sizing of elongated particles due to the fact that the appearance of a particle in an image is no longer dependent solely on its spatial orientation relative to a single imaging device 110.

In some embodiments, this approach can be used to reduce the lensing effect caused by the curvature of the container 10. By using the three-dimensional position of the particle, the measured particle size in the two-dimensional images acquired by each of the imaging devices 110 can be adjusted to compensate for the lensing effect. for example, by changing the lateral (horizontal) component of the size measurement using a lens effect conversion factor. Said conversion factor can be determined based on the optical model of light movement through the capacitor 10 for each of the imaging devices 110 as described above.

- 32 044111

Spectral detection.

In fig. 45 shows a sensor 4500 (shown as a diffraction spectrometer) that can be used with a visual inspection unit 100 of the type disclosed herein.

For example, sensor 4500 may form a fourth image tap used with the example implementation of block 100 of FIG. 38 A.

Sensor 4500 may be used to detect a characteristic (eg, spectral signature) of one or more particles in container 10. For example, as shown in the drawing, container 10 is illuminated by broadband light sources 122. Sensor 4500 receives light from capacitance 10 through distortion-correcting optics 4501 (e.g., of any type described above) and telecentric lens 4501. Light from lens 4501 is directed to a diffraction grating 4503 that separates components of the light spectrum, which are then imaged at imaging sensor 4504. In some implementations, the diffraction grating 4503 operates such that the location of the incident light along one dimension of the sensor 4504 (eg, the vertical dimension) corresponds to a wavelength of the light. Another measurement on the imaging sensor 4504 corresponds to various spatial points within the container 10. That is, the sensor 4500 provides spectral data for a portion of the container, e.g., in the selected configuration, the portion is shown as a horizontal piece of the container 10.

As particles move through said central horizontal plane, their stereoscopic profile can be recorded. At the same time, as detailed above, standard image taps of block 100 can be used to track the position of a particle within a container (eg, in three dimensions). This data can be used to determine when a single particle enters a detection area detected by sensor 4500. When a particle enters an area, sensor 4500 records a characteristic (eg, spectral profile) of the particle. Unit 100 may generate data associated with said characteristic and may associate said data with data indicative of the nature of the particle in the tracking data.

In various embodiments, the characteristic data can be used for any suitable purpose, for example, to determine the type of particle. For example, spectral data about an individual particle can be combined with data about the size, shape, movement of the particle, or other data about the particle to determine the type of particle.

In some embodiments, sensor 4500 and illuminating light sources 122 may be modified to detect particle fluorescence or any other suitable characteristics. In general, any spectral characteristic of the particles may be detected, including color, absorption spectrum, emission spectrum, or transmission spectrum, or a combination of any of these characteristics.

Although in the above example the sensor 4500 is included in a block 100 containing three image fingers, in other embodiments any number of image fingers may be used, such as one, two, four, five or more fingers. In some embodiments in which a single image tap is used, the sensor 4500 may be matched to the image tap, for example, by using a beam splitter (not shown) to split the beam of light received from the capacitor 10 and transfer the components of the beam to the single image tap and to sensor 4500. In other implementations, such as those that employ multiple image bends, sensor 4500 may be oriented at any suitable angle relative to the imaging devices.

In situ measurement of sample properties.

In some embodiments, inspection unit 100 may include one or more detectors (not shown) that can be used to measure the refractive index of the fluid in container 10. For example, in some embodiments, a fluid-filled portion of container 10 may be directed a narrow off-axis collimated laser beam detected to measure beam displacement due to refraction in the container 10. If the material and geometry of the container 10 are known, this information can be used to determine the refractive index of the fluid. In various embodiments, any other suitable method for determining the refractive index may be used.

In some embodiments, the measured refractive index of the fluid may be used as an input parameter for any data processing schemes disclosed herein (eg, data processing to compensate for lens effects caused by the curvature of the container 10).

In some embodiments, inspection unit 100 may also include one or more detectors (not shown) that may be used to determine information indicative of the shape of container 10. For example, in some embodiments, through an air-filled (e.g., top) portion capacitor 10 can be directed to a narrow off-axis collimated laser beam, detected for the purpose of measuring the displacement of the beam compared to a reference. The specified deviation can be used to accurately measure the wall thickness of the container (for example, with an accuracy of 0.25 mm or less). In various embodiments, any other suitable technique for determining the shape of the container may be used.

In some embodiments, the detected geometric data can be used, for example, as described above, to determine the refractive index of the fluid in the container 10. In some implementations, the detected geometric data can be used as an input parameter for various data processing techniques disclosed in the present description (for example, processing data to compensate for lens effects caused by the curvature of the container 10), or for any other suitable purpose.

Image formation using the immersion method.

According to the present detailed description, in various embodiments, the refractive properties of the fluid in the container 10 may cause undesirable image distortion effects. In some embodiments, these effects can be reduced by filling all (or a portion) of the space between the container 10 and the imaging device 110 used to image the container with a medium having a refractive index closer to that of the fluid than to the refractive index of the fluid. air.

In some embodiments, refractive distortion can be further reduced by ensuring that the refractive index of the container 10 matches the refractive index of the fluid located within the container.

In some embodiments, these immersion imaging techniques may reduce or eliminate the need for correction optics and/or processing used to reduce distortion (eg, the lens effect as detailed above).

Controlling sample temperature.

In some embodiments, the inspection unit 100 may include one or more devices (not shown) for controlling the temperature of the sample within the container 10. For example, in some implementations, the temperature control device may be used to vary the temperature of the container between 0 and 40° C, from 0 to 100°C, or other suitable ranges. In some embodiments, the temperature control device may maintain the temperature at a set value, such as a value varying by less than 5, 2.5, 1, 0.1, 0.01 °C or less.

Temperature control may be of particular benefit in applications where temperature control is important to prevent sample degradation during the detection process. In some embodiments, temperature- and time-dependent stability studies of temperature-sensitive products can be conducted by controlled changes in sample temperature. For example, the platform can be used to measure the dissolution (or, in some cases, formation) of protein aggregates as the temperature of the drug is controlled to increase from, for example, 4°C (refrigeration temperature) to 20°C (room temperature), or up to 37 °C (human body temperature).

In various embodiments, temperature control may be achieved in any suitable manner. In some embodiments, the environment within the inspection unit may be hermetically sealed and thermally insulated, and temperature control may be accomplished by, for example, an air conditioning unit. In some embodiments, a heating coil and a thermoelectric cooler (eg, based on the Peltier effect) may be integrated into the sample holder for container 10. In embodiments in which multiple containers are located on a tray, temperature control of the tray may be accomplished through the tray by means of a heating/cooling process fluid (eg, by passing the process fluid through a heat exchanger). In general, one or more temperature sensors and/or thermostats may be used to provide a closed-loop temperature control system.

Multiple inspection techniques.

In some embodiments, inspection unit 100 may re-inspect a single sample by changing one or more operating parameters (eg, rotation speed) that may be selected based on the results of the first inspection. This process can be repeated many times in order to better adjust the operating parameter for a specific test sample.

For example, in some embodiments, the inspection may be repeated (eg, with a modified rotation speed) if the particle detection and sizing data obtained are outside the range of expected results (indicating an error in the original inspection).

Background reference mapping for automatic calibration.

According to the above detailed description, in various embodiments, it is desirable to determine the characteristics of distortion effects (eg, lens effects) caused by the refraction of light passing through the capacitance 10 to the image sensor 110. In some implementations, the inspection unit 100 itself may be used to map distortions caused by the capacitance 10. The resulting map may then be used to compensate for these effects during image processing.

For example, in some embodiments, one or more calibration indicators (eg, a grating) may be placed behind the container 10 as a background to the imaging device 110. Refractive distortion can be detected and mapped by detecting specified indicators in the acquired image (eg, using edge detection or other suitable object detection techniques) and comparing their appearance in the image with their known actual appearance.

In some embodiments, this approach can be used to correct for distortion caused by non-cylindrical containers, such as containers that are rotationally symmetrical on an axis but with varying peripheral lengths around the axis (such as containers shaped like modern soft drink bottles ).

Automatic vibration detection and suppression.

As described above, in some embodiments, vibrations may degrade the performance of the inspection unit 100. Vibration causes normally static objects (such as cosmetic defects on the surface of a container) to vibrate during image formation. This can reduce performance during the static removal phase due to the formation of small but significant vibrating halos that are not removed during static removal and can cause incorrect results in subsequent particle detection algorithms. In various embodiments, one or more of the following techniques may be used to reduce the effect of oscillation.

In some embodiments, vibrating ghosting features that form around distant static elements can be suppressed by increasing the size of the image region corresponding to the detected static elements (e.g., by one to several pixels thick), resulting in image regions containing subtle vibrating ghosting , are also removed before the particle analysis step begins. However, in some embodiments, a disadvantage of this approach is that it reduces the substantially available sensor area.

In some embodiments, a screening algorithm may be used to detect the presence of vibrating elements. For example, features can be detected by processing the image to detect vibrating features that do not move across the image. In some embodiments, these elements may also be identified based on their spatial proximity to detected static elements.

In some embodiments, vibration indicators of a container may be detected from acquired images, such as by using edge extraction to detect movement of the walls of the container, whereby the system may automatically detect and possibly alert the user to unacceptably high levels of vibration in the environment.

In some embodiments, vibration indicators of the container may be detected through physical sensors. For example, in some embodiments, the head of a tool that holds and manipulates a container during inspection may include motion detection devices (e.g., high-precision accelerometers) that provide feedback that the system can automatically alert users to when vibration levels exceed a preset threshold. meaning.

Examples

Below are exemplary performance characteristics of an example implementation of a visual inspection unit 100 of the type disclosed herein.

According to FIG. 40, block 100 was used with containers 10, each containing one polymer sphere of known size. Multiple detection processes (n=80) were performed for each container, and the percentage detection value was measured (indicated in the figure by APT bars). According to the drawing, the percentage detection value of the system was over 90% for particles with a diameter of 15 to 200 μm. Detection percentages when performing an identical task by a trained person are presented for comparison (indicated by person bars in the figure). It should be noted that human particle detection ability drops sharply when particles smaller than 200 microns are detected.

According to FIG. 41, in another experiment, block 100 was used with containers containing particles with diameters above and below the 125 µm cutoff of the visible spectrum. Unit 100 identified the particle and also classified the particles based on size, distinguishing between particles above and below the 125 µm cutoff of the visible spectrum. According to the drawing, the percentage detection value of the system was over 90% for particles with a diameter of 15 to 200 μm. Unit 100 also accurately categorized detected particles with an extremely high degree of accuracy.

- 35 044111

According to FIG. 42, a series of solutions for several standard particle sizes were provided, each series consisting of containers containing particles at a given concentration. The resulting containers were analyzed by unit 100 to obtain particle count data, and regression was used to determine R ² values for count linearity versus inverse dilution factor. According to the drawing, the R ² value was over 0.95 for particle sizes from 15 to 200 μm, indicating excellent linearity.

According to FIG. 43, an excited sample containing protein particles was analyzed by unit 100 to determine the number of particles grouped by particle size. The accuracy of counting the number of particles for each group over 10 cycles is shown. The size of the protein particles is unknown, making absolute comparisons of sizing accuracy impossible, but according to the drawing, the system is nevertheless highly accurate in counting and sizing proteins. The normalized measurement error was 3%, indicating excellent accuracy.

According to FIG. 44, block 100 was also tested to detect empty and protein-containing vials. The performance of unit 100 was compared to that of a qualified visual inspection operator when examining the same set of vials. Block 100 (denoted by APT bars in the figure) correctly identified all 40 vials containing protein particles and 80 empty vials over three runs. Self-match in classifying visible and invisible particles was 100%. The person's score was only about 85% in both categories.

Conclusion.

It will be apparent to one skilled in the art that the processes included in the automated system and method for non-destructive detection and identification of particles (processing of time series data obtained through visual inspection) can be implemented in the form of an article containing a machine-readable medium. For example, the computer-readable medium may comprise a readable storage device, such as a hard disk, CD-ROM, DVD-ROM, computer floppy disk, or solid-state memory components (ROM, RAM), on which computer-readable segments of program code are stored. The computer-readable medium may also comprise communication or transmission means, such as a bus or communication channel, which is optical, wired, or wireless, carrying program code segments in the form of digital or analog data signals.

In the present description, a block diagram is used. The use of a flowchart should not be construed as limiting the order in which operations can be performed. The material disclosed herein contains illustrations of other components located within or connected to other distinct components. It should be understood that such an illustrated architecture is provided by way of example only, and as such, many other architectures may be implemented that provide identical functionality. In a fundamental sense, any configuration of components that provides identical functionality is essentially associated with providing the desired functionality. Therefore, any two components combined to provide a specific function can be considered to be associated with each other to provide the desired functionality, regardless of the architecture or intermediate components. Similarly, any two components so coupled can be considered to be in operative connection or operative communication with each other to provide the desired function, and any two components so connected can be considered to be in operative connection. communication with each other to provide the desired function. Specific examples of operably communicating components include, but are not limited to, physically connected and/or physically interoperable components and/or wirelessly interoperable and/or wirelessly interoperable components and/or logically interoperable and/or logically interoperable components .

With respect to the use of substantially any plural and/or singular terms, those skilled in the art are free to convert plural to singular and/or singular to plural depending on the context and/or scope of application. Various singular and/or plural variations may be indicated herein for purposes of clarity.

One skilled in the art will understand that in general, the terms used in the present specification, particularly in the following claims (e.g., in the text of the claims) are essentially open terms (e.g., the term containing is to be understood as including without limitation, the term having must be understood as having at least, the term includes must be understood as includes without limitation, etc. It will also be obvious to a specialist that if there is an intention to indicate a certain number in the text of a claim, the specified intention is clearly included in the text of the claim, and in the absence such indication such an intention does not occur. For example, to facilitate understanding, the following claims may contain at least one introductory phrase and one or more as an introduction to the text of the claim.

If a certain number is indicated in the text of the claim, it will be obvious to the specialist

-

Claims (15)

that the specified text should be understood to cover at least the specified number (for example, simply indicating two elements without further specification usually means at least two elements, or two or more elements). In addition, in cases where a phrase similar to at least one of A, B and C, etc. is used, such construction generally relies on the meaning of the phrase being obvious to one skilled in the art (e.g., a system containing at least one of A, B and C includes, without limitation, systems containing only A, only B, only B, A and B, A and B, B and C, and/or A, B and C, etc.). In cases where a phrase similar to at least one of A, B or C, etc. is used, such construction generally relies on the meaning of the phrase being obvious to one skilled in the art (e.g., a system containing at least one of A, B or C includes, without limitation, systems containing only A, only B, only B, A and B, A and B, B and C, and/or A, B and C, etc.). It will also be apparent to one skilled in the art that virtually any disjunctive conjunction and/or phrase providing two or more alternative terms in the description, claims, and drawings allows for the possibility of including one of the terms, either of the terms, or both of the terms. For example, the phrase A or B takes into account the probability of A or B, or A and B. As used herein, the term optical element may refer to one or more refractive, reflective, diffractive, holographic, polarizing or filtering elements in any suitable combination. As used herein, terms such as light, optical, or other related terms refer not only to light visible to humans, but may also include, for example, light from the ultraviolet, visible, and infrared portions of the electromagnetic spectrum. The present description of the illustrative embodiments is for purposes of illustration and description. The description is not to be construed as all-inclusive or limited solely to the form disclosed, and modifications and variations of the disclosed materials are possible and may also be apparent from implementation of the disclosed embodiments. The scope of the invention is defined by the accompanying claims and equivalent claims. CLAIM 1. A device for non-destructively detecting an undissolved particle in a vessel at least partially filled with a fluid, the device comprising: sensor; an imaging device configured to form one or more images of a particle in a fluid, the imaging device comprising at least one optical imaging element positioned to form an image of the particle onto the sensor; an illumination source positioned to illuminate the contents of the image forming vessel by the imaging device, thereby substantially eliminating the presence of rays of light emitted by the illumination source and reflected or refracted from the surface of the vessel being imaged by the at least one optical element onto the sensor. 2. The apparatus of claim 1, wherein the illumination source is located in a region substantially free of return rays passing from the sensor, through the at least one imaging optical element, and then through the vessel. 3. The device according to claim 2, wherein at least one optical element comprises a telecentric lens. 4. The device according to claim 1, wherein the optical axis of the image forming device is arranged to pass through the vessel substantially orthogonal to and intersects with the longitudinal axis of the vessel, the longitudinal axis corresponding to the axis of symmetry of the vessel. 5. The apparatus of claim 1, wherein the imaging apparatus is configured to obtain time series data representing the trajectory of a particle in a fluid, and the apparatus further comprises: (a) a memory in operative communication with the imaging device and configured to store time series data; and (b) a processor in operative communication with the memory and configured to detect the particle by: (i) reversing the temporal order of the time series data to obtain reverse time series data; (ii) estimating particle trajectory from backward time series data; and (iii) determining the presence or type of particle based on its trajectory. 6. A method for non-destructively detecting an undissolved particle in a vessel at least partially filled with a fluid, the method comprising: - 37 044111 using an imaging device to obtain one or more images of a particle in a fluid, the imaging device comprising at least one optical imaging element positioned to form an image of the particle onto the sensor; and illuminating the container by means of a light source positioned to illuminate the contents of the container to form an image by the imaging device, thereby substantially eliminating the presence of rays of light emitted by the illumination source and reflected or refracted from the surface of the container being imaged by the at least one optical element to the sensor. 7. The method of claim 6, wherein the illumination source is located in a region substantially free of return rays passing from the sensor, through the at least one optical element, and then through the vessel. 8. The method of claim 7, wherein the at least one optical element comprises a telecentric lens. 9. The method of claim 6, wherein the optical axis of the imaging device extends through the vessel substantially orthogonally to and intersects the longitudinal axis of the vessel, the longitudinal axis corresponding to an axis of symmetry of the vessel. 10. The method according to claim 6, additionally including: using an imaging device to obtain time series data depicting the trajectory of a particle in a fluid; and detecting the particle by: (i) reversing the temporal order of the time series data to obtain reverse time series data; (ii) estimating particle trajectory from backward time series data; and (iii) determining the presence or type of particle based on its trajectory. 11. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform non-destructive detection of an undissolved particle in a container at least partially filled with liquid by: causing the imaging device to acquire one or more images of a particle in a fluid, the imaging device comprising at least one optical imaging element positioned to form an image of the particle onto the sensor; and causing illumination of the vessel by a light source positioned to illuminate the contents of the vessel to form an image by the imaging device, thereby substantially eliminating the presence of rays of light emitted by the illumination source and reflected or refracted from the surface of the vessel, which are imaged by the at least one optical element forms on the sensor. 12. The non-transitory computer-readable medium of claim 11, wherein the illumination source is located in a region substantially free of return rays passing from the sensor, through the at least one optical element, and then through the vessel. 13. The non-transitory computer-readable medium of claim 12, wherein at least one optical element comprises a telecentric lens. 14. The non-transitory computer-readable medium of claim 11, wherein the optical axis of the imaging device extends through the vessel substantially orthogonally to and intersects the longitudinal axis of the vessel, the longitudinal axis corresponding to an axis of symmetry of the vessel. 15. The non-transitory computer-readable medium of claim 11, further comprising instructions that, when executed by a processor, cause the imaging device to obtain time series data depicting the trajectory of a particle in the fluid and detect the particle by: (i) reversing the temporal order of the time series data to obtain reverse time series data; (ii) estimating particle trajectory from backward time series data; and (iii) determining the presence or type of particle based on its trajectory. -