EA038813B1 - Methods and apparatus for nondestructive detection of undissolved particles in a fluid - Google Patents

Description

Prior art

To determine the quality of a drug formulation, it is important to differentiate between different types of particles. For example, due to low differentiation specificity, objects such as glass plates can be mistaken for solid protein particles. The high specificity of the differentiation system is essential for making accurate formulation decisions. In the absence of data on the type or species of particles in a particular drug, the choice of the exact formulation of the drug can be difficult.

Unfortunately, current particle detection techniques are not suitable for detecting protein aggregates and other small and / or fragile particles. Particles less than about 100 microns typically cannot be manually detected by humans. Automated exploration techniques are usually destructive; in other words, these techniques involve removing the test fluid from its container, after which it is usually unusable for therapeutic use. In addition, known non-destructive examination systems use only one photographic image of the container to determine the presence or absence of particles, which often leads to inaccurate measurements of particle size and / or their number. Known research techniques can also involve breaking down more fragile particles such as protein aggregates. For example, high speed (eg, 2000 rpm or higher for a few seconds) centrifugation of a fluid tube can rupture protein aggregates in the fluid.

The essence of the invention

One embodiment of the technology disclosed herein relates to a device for non-destructive detection of a particle (i.e., undissolved particle) in a vessel at least partially filled with a fluid such as an aqueous fluid, an emulsion, a petroleum product, and an organic solvent. As used herein, the term detecting or detecting includes detecting, characterizing, differentiating, distinguishing or identifying the presence, number, location, nature, size, shape (e.g., elongation or roundness), color, fluorescence, contrast, light absorption coefficient, light reflectance or another characteristic, or a combination of two, three, four, five, six, seven, eight, nine, ten, eleven, twelve or more of the specified characteristics of a particle. In the illustrated embodiments, the apparatus comprises an imaging apparatus for acquiring time series data representing the trajectory of a particle in a fluid medium. The time series data is stored in a memory in operative communication with the imaging apparatus, and a processor in operative communication with the memory detects and / or identifies the particle. In particular, the processor reverses the temporal order of the time series data to produce inverse time series data, estimates the particle trajectory based on the inverse time series data, and determines the presence or type of the particle based on its trajectory. As used herein, reverse time series data includes time series data frames presented in reverse chronological order such that the last event is displayed first (and vice versa).

Other embodiments include a method and associated computer program product for non-destructive detection of undissolved particles in a vessel at least partially filled with fluid. The method includes reversing the time order of the time series data representing the trajectory of a particle in a fluid medium to obtain inverse time series data, for example, by means of a processor executing instructions encrypted in the nonvolatile memory of a computer program product. The method also includes estimating the trajectory of the particle based on the inverse time series data and then detecting and / or identifying the particle based on its trajectory.

Another embodiment is a device for non-destructive detection of undissolved particles in a vessel at least partially filled with a fluid medium, comprising:

(a) at least two imaging devices located to provide imaging of the particle from different angles, each imaging device configured to obtain one or more two-dimensional images of the particle in a fluid medium;

(b) a memory in operative communication with the imaging apparatus configured to store a time series;

(c) a processor in working communication with memory, configured to detect a particle by:

(i) combining 2D images obtained from at least three imaging devices to determine 3D data indicative of the location of the particle in the vessel;

(ii) detecting the particle based at least in part on the 3D data.

Also disclosed is a method for non-destructive detection of an undissolved particle in a vessel at least partially filled with a fluid medium, comprising:

1 038813 (a) using at least two imaging devices to image the particle from different angles such that each imaging device respectively obtains one or more two-dimensional images of the particle in the fluid;

(b) combining 2D images obtained from at least two imaging devices to determine 3D data indicative of the location of the particle in the vessel;

(c) detecting a particle based at least in part on the 3D data.

Still other embodiments of the present invention include an apparatus, method, and computer program product for nondestructively detecting (one or more) transparent or reflective objects (eg, glass slabs) in a vessel at least partially filled with a fluid. The imaging apparatus acquires data representing light reflected from multiple spatial locations in the vessel as a function of time, and stores the data in a memory in operative communication with the imaging apparatus. The processor, in working communication with the memory, detects objects (for example, slabs of glass), possibly in response to indications encrypted in the computer program product based on the data, by identifying the corresponding maximum amount of reflected light for each point from the set of points represented in the data. The processor then determines the presence or absence of objects (eg, slabs of glass) in the vessel based on the number of spatial arrangements in which the corresponding maximum amount of reflected light exceeds a predetermined value.

Another embodiment of the invention is a method of non-destructively counting and sizing undissolved particles in a vessel at least partially filled with a fluid. The method includes:

(a) obtaining at least one image of the particles in the vessel, obtained under predetermined imaging conditions;

(b) detecting particles and determining information indicating the apparent size of the detected particles in the image, based on the specified at least one image;

(c) determining information about the population of visible size particles, indicating the distribution of the visible size of the detected particles;

(d) determining the actual size particle population information indicative of the actual size distribution of the detected particles, based on (i) the apparent size particle population information, (ii) the calibration population data indicating the apparent size distribution of one or more standard particle populations size, the image of which was obtained under conditions corresponding to the specified conditions of image formation.

Another embodiment is a device for counting and determining the size of undissolved particles in a vessel at least partially filled with a fluid medium, containing at least one processor configured to:

(a) obtaining at least one image of the particles in the vessel, obtained under predetermined imaging conditions;

(b) detecting particles and determining information indicative of the apparent size of the detected particles in the image, based on the specified at least one image;

(c) determining information about the population of visible size particles, indicating the distribution of the visible size of the detected particles;

(d) determining the actual size particle population information indicative of the actual size distribution of the detected particles based on (i) the apparent size particle population information, (ii) the calibration population data indicating the apparent size distribution of one or more standard particle populations size, the image of which was obtained under conditions corresponding to the specified conditions of image formation.

Another embodiment of the invention is a computer program product for non-destructive counting and determination of the size of undissolved particles in a vessel at least partially filled with a fluid medium, containing non-volatile machine-readable instructions that, when executed by a processor, provide the following processor actions:

(a) obtaining at least one image of the particles in the vessel, obtained under predetermined imaging conditions;

(b) detecting particles and determining data indicative of the apparent size of the detected particles in the image, based on the specified at least one image;

(c) determining information about the population of visible size particles, indicating the distribution of the visible size of the detected particles;

(d) determining the actual size particle population information indicative of the actual size distribution of the detected particles, based on

- 2,038813 (i) information on the population of visible size particles, (ii) data on the calibration population indicating the apparent size distribution of one or more aggregates of standard size particles, which were imaged under conditions corresponding to the specified imaging conditions.

Another embodiment of the invention is a method for non-destructive detection of an undissolved particle in a vessel at least partially filled with a fluid, comprising:

(a) using at least one imaging device to form an image of the particle;

(b) image processing to determine location data indicative of the location of the particle in the vessel;

(c) detecting a particle based at least in part on the location data, wherein detecting the particle based at least in part on the location data comprises determining the presence of a particle at a portion of the vessel;

(d) using a sensor to determine the characteristics of the particle when the particle is located in the specified area of the vessel;

(e) generating particle characteristic data indicative of a specific characteristic;

(e) the relationship of the particle characteristic data to the particle identification data.

Another embodiment of the invention is a device for non-destructive detection of undissolved particles in a vessel at least partially filled with a fluid medium, comprising:

(a) at least one imaging device located to form an image of the particle;

(b) at least one sensor configured to determine the characteristics of the particle when the particle is located on the specified portion of the vessel;

(c) at least one processor in operative communication with each of the at least one imaging apparatus and sensor, configured:

processing the image to determine location data indicative of the location of the particle in the vessel;

detecting the particle based at least in part on the location data, determining the presence of the particle at the site of the vessel;

using the signal from the sensor to determine the characteristics of the particle when the particle is located in the specified area of the vessel;

generating particle characteristic data indicative of a specific characteristic; the relationship of the particle characteristic data with the particle identification data.

Another embodiment of the invention is a device for non-destructive detection of an undissolved particle in a vessel at least partially filled with a fluid medium, in which the vessel contains a transparent tubular wall of the vessel extending along the longitudinal axis, and the device comprises: an imaging device configured to obtain one or more images of a particle in a fluid medium, the imaging apparatus comprising at least one imaging optical element arranged to form an image of the particle onto the sensor; an illumination source located at least partially in a plane passing through the vessel and substantially orthogonal with respect to the longitudinal axis of the vessel, the illumination source being configured to substantially completely eliminate rays of light emitted by the source and reflected or refracted from the surface of the vessel wall, the image of which at least one optical element forms on the sensor.

Another embodiment of the invention is a method for non-destructive detection of an undissolved particle in a vessel at least partially filled with a fluid medium, in which the vessel comprises a transparent tubular vessel wall extending along a longitudinal axis, and the method includes: using an imaging device to obtain one or more imaging a particle in a fluid medium, the imaging apparatus comprising at least one imaging optical element arranged to form an imaging of the particle onto the sensor; and illuminating the vessel with an illumination source located at least partially in a plane passing through the vessel and substantially orthogonal to the longitudinal axis of the vessel, the illumination source being configured to substantially eliminate light rays emitted by the source and reflected or refracted from the wall surface vessels, the image of which is formed by at least one optical element on the sensor.

Unlike other particle detection systems and technologies, the systems and technologies of the present invention are non-destructive — there is no need to remove fluid from the vessel to detect, count, and identify particles in the vessel. As a result, the systems and technologies of the present invention can be used to investigate changes and interactions between particles, fluid, and vessel over extended periods of time, such as minutes, hours, days, months, or years. In addition, the systems and technologies of the present invention do not necessarily imply or result in the destruction of even more fragile particles, such as small protein aggregates, in the vessel. Present systems and technologies also make it possible to obtain time series data, i.e. data representing the trajectories of particles in a moving fluid. Due to the fact that the present systems use time-series data instead of single-frame photographic images of the vessel, the present systems allow for more accurate estimates of the number of particles in the vessel and the size of the particles. The present systems also provide additional information about each of the particles, such as particle morphology and particle composition, based on particle movement data. For example, sinking particles are usually denser than floating particles.

This summary is illustrative only and should not be considered in any way limiting. In addition to the above-described illustrative aspects, exemplary implementations and features, further aspects, exemplary implementations and features described with reference to the following drawings and detailed description will be apparent to those skilled in the art.

List of figures of drawings and other materials

The accompanying drawings, included in this description and forming part of it, show examples of the implementation of the disclosed technology; the drawings, in conjunction with the description, are intended to illustrate the principles of the disclosed technology.

FIG. 1A-1C show a visual inspection unit, a visual inspection imaging unit and a visual inspection platform, respectively; each of these elements can be used to detect and identify particles in a container at least partially filled with fluid.

FIG. 2A illustrates sample preparation, loading, and operation of the visual inspection systems of FIG. 1A-1C.

FIG. 2B shows processed images of particles and their trajectories in a moving fluid in a vessel, obtained through an illustrative visual inspection system.

FIG. 3A-3C show three types of shaking of a vessel containing a fluid and one or more particles in preparation for detection and identification of particles: rotation of the cylindrical vessel (FIG. 3A), inversion and rotation of the syringe (FIG. 3B), shaking of the syringe (FIG. . 3C).

FIG. 4 shows an optical beam diagram of a telecentric lens used to form an image of a cylindrical vessel.

FIG. 5A shows a meniscus of a fluid and a detectable volume in a cylindrical vessel containing a fluid.

FIG. 5B shows distortion and blind spots in a cylindrical container due to the shape of the container.

FIG. 5C and 5D show techniques for compensating for distortion and blind spots when imaging cylindrical vessels.

FIG. 5E shows distortion and blind spots in a cylindrical container due to the shape of the container for particles at various points within the container.

FIG. 5F shows theoretical models for cylindrical capacitance distortion; each model corresponds to the same container filled with a fluid with a different refractive index. The drawing also shows the corresponding experimental measurements, confirming the validity of the theoretical models.

FIG. 5G shows the use of a correcting optical element to correct distortion in a cylindrical container formed due to the shape of the container.

FIG. 5H is a detailed view of the correcting optical element of FIG. 5G.

FIG. 5I shows an apparatus for selecting one or more correcting optical elements.

FIG. 6A-6D show particle tracking systems with multiple imaging devices for obtaining time series data for moving particles from multiple angles (FIGS. 6A and 6B), with high frame rates from one angle (FIG. 6C), at different spatial resolutions from one angle. (Fig. 6D).

FIG. 7A and 7B illustrate triggering the imaging and illumination process for particle imaging by dual sensor imaging devices.

FIG. 8 is a schematic diagram of an adaptable multipurpose lighting configuration comprising light sources in front of, behind, below the test vessel.

FIG. 9A-9C illustrate lighting at different angles to distinguish different types of particles using the light sources of FIG. eight.

FIG. 9D is a lighting and timing sequence diagram using the configurations of FIG. 9A-9C to distinguish between different types of particles.

FIG. 10A-10C show the flare from a vessel partially filled with fluid (FIG. 10A) and the placement of the light sources outside of the area defined by rotating the imager about the longitudinal axis of the vessel (FIGS. 10B and 10C).

- 4 038813

FIG. 10D, 10E show an alternative lighting scheme for reducing or eliminating flare from the vessel.

FIG. 11 is a schematic diagram of a configuration suitable for imaging polarizing (eg, chiral) particles.

FIG. 12 is a schematic diagram of a configuration suitable for exciting and imaging fluorescent particles.

FIG. 13A and 13B are projection images of maximum brightness for glass (FIG. 13A) and protein (FIG. 13B) slabs obtained by the disclosed visual inspection system.

FIG. 14 shows block diagrams illustrating the general particle detection and identification process, as well as sub-processes for image preprocessing, particle tracking, and statistical analysis.

FIG. 15A and 15B show a time series data frame before (FIG. 15A) and after (FIG. 15B) background subtraction.

FIG. 16A shows a time series data frame showing the particle as an eight-bit halftone image (left).

FIG. 16B is an approximate view of the time series data frame shown in FIG. 16B.

FIG. 16C and 16D illustrate options with defined boundaries for the time series data frames shown in FIG. 16A and 16B, respectively.

FIG. 17A-17D show how two consecutive frames of time series data (FIG. 17A) can be used to perform predictive tracking (FIGS. 17B-17D).

FIG. 18A shows a half-tone frame of time series data showing several particles.

FIG. 18B depicts the edge-to-edge embodiment of FIG. 18A used to locate the geometric center of a particle.

FIG. 19 shows sequential frames of time series data illustrating particle collision / obstruction.

FIG. 20A is a time series data frame showing two adjacent particles in a highlighted area.

FIG. 20B-20E show sequential time series data frames showing particle occlusion clearly visible as particles pass through the highlighted region of FIG. 20A past each other.

FIG. 21A-21C show the apparent obstruction of a moving particle due to subtraction of background artifact such as a scratch or dirt particles on the vessel wall for straight paths (FIG. 21A), curved paths (FIG. 21B), parabolic paths (FIG. 21C).

FIG. 22A-22C show the determination of the center of mass of irregularly shaped particles using inverse time series data (FIGS. 22B and 22C), using the center of mass location data to determine the trajectory of the particle (FIG. 22A).

FIG. 23A-23D show fluid dynamics marked and modeled in cylindrical vessels. FIG. 23A shows changes in the shape of the meniscus. FIG. 23B and 23C show the formation of vortices in a fluid-filled vessel, and FIGS. 23D shows the trajectories of particles in an illustrative vortex.

FIG. 24A and 24B are approximate views of successive frames of reverse time series data in which particle collisions were not correctly identified (FIG. 24A), and the same plot after error correction (FIG. 24B) is shown.

FIG. 25A-25E show the time dependence of particle size measurement due to particle movement.

FIG. 25F is a plot of Feret diameter over time for the particle shown in FIG. 25C.

FIG. 26A shows frames of processed time series data at different intervals, with lines representing different particle paths.

FIG. 26B shows exemplary measurement data of several time-dependent particle properties corresponding to the particle paths of FIG. 26A.

FIG. 27A-27F show the detection of a region of interest by means of an angled backlight. FIG. 27A shows the original image (time series data frame) subjected to contouring (FIG.27B), halftone boundaries (FIG.27C), identifying the meniscus and base of the tube (FIG.27D), determining the region of interest (delimited by the dotted line in FIG.27E) ), cropping (FIG. 27F) to form an image of the fluid visible in the container.

FIG. 28A-28C show the determination of the fill volume of a backlit tube. FIG. 28A shows a raw image of a tube. FIG. 28B shows a region of interest (delimited by a dotted line) defined by defining boundaries and extracting outlines. Defects on the surface of the tube (shown in Fig. 28C) can make it difficult to determine the fill volume.

FIG. 29A-29D show the determination of the fill volume of a bottom highlighted tube. FIG. 29A and 29B show pseudo-color images of a partially filled vessel (FIG. 29A) and an empty vessel (FIG. 29B). FIG. 29C and 29D show automatic meniscus detection for partially filled, empty, partially filled vessels.

FIG. 30 shows a processor configured to process time series data.

FIG. 31 shows an example of determining the boundaries of halftones of an image including a bright particle and a dim particle.

FIG. 32 shows a histogram of apparent particle sizes for a population of standard particle sizes (100 μm).

FIG. 33 shows the apparent particle size data curves for two populations of particles, each population containing particles of the specified standard size (μm).

FIG. 34 shows the calibration curves of the apparent particle size data for four populations of particles, each population containing particles of the specified standard size (μm).

FIG. 35 shows the matching of the overlay of the calibration curves to the curve of the apparent particle size data in the sample.

FIG. 36 shows a comparison of the results of using two counting and particle sizing techniques: unprocessed binning and LENS.

FIG. 37 shows the process of counting and sizing particles using various techniques for sizing particles below and above the size threshold.

FIG. 3 8A-3 8C show particle tracking systems with multiple imaging devices for obtaining time series data for moving particles at multiple angles.

FIG. 39 shows the movement through the container of light rays received by each of the two imaging devices (left panel) and each of the three imaging devices (right panel) of the particle tracking systems of FIG. 38A-C.

FIG. 40 shows the results of particle detection by an automatic particle detection system (referred to as APT) compared to the results of human visual inspection.

FIG. 41 shows the results of particle detection and classification by an automatic particle detection system.

FIG. 42 is a table summarizing particle count linearity as a function of sample dilution for an automated particle detection system.

FIG. 43 illustrates the accuracy of an automatic particle detection system used to detect and count particles of protein aggregates.

FIG. 44 shows the results of particle detection of protein aggregates by an automatic particle detection system (referred to as APT) compared to the results of human visual inspection.

FIG. 45 shows a spectrometer used with a visual inspection unit.

Information confirming the possibility of carrying out the invention

FIG. 1A shows an illustrative embodiment of an automated visual inspection unit 100 capable of non-destructive detection and / or identification of particles in a transparent container 10 at least partially filled with a fluid such as a protein-containing drug, drugs, biotechnology products, beverages, and other translucent fluids. environments regulated by the United States Food and Drug Administration.

While detecting the presence or absence of a particle can be accomplished by inspecting portions of the container that have an irregular shape on the outside (e.g., the back), in standard embodiments, particle characteristics measurements such as counting and sizing can be performed. inspecting the particles through a substantially uniform vertical vessel wall is necessary to reduce distortion. This approach imposes restrictions on the minimum fill volume due to the fact that the apparent two-dimensional cross-section of the fluid in the container 10, accessible for inspection by the block 100, must have a suitable area to provide acceptable statistics. The required filling volume depends on the circumferential diameter of the container (smaller containers require less filling volume). In various embodiments, the internal volume of the container can be filled with fluid by at least 1%, at least 5%, at least 10%, at least 20%, at least 30%, at least 40%, at least 50%, at least 60%, at least 70%, at least 80%, at least 90%, or at least 100%.

In various embodiments, the particle detection techniques disclosed herein are optical in nature. Accordingly, in some embodiments, the walls of container 10 are sufficiently transparent at the wavelength of illumination to render the fluid within the container visible. For example, in some embodiments, container 10 can be made of clear borosilicate glass, but other suitable materials can be used. The turbidity of the fluid disposed within the vessel is also important and should be low enough to provide the desired degree of visualization. In some embodiments, the turbidity of the fluid is in the range of 0 to 100 NTU (nephelometric turbidity units), preferably in the range of 0 to 20 NTU, more preferably in the range of 0 to 10 NTU.

- 6 038813

Standard practices for measuring turbidity are given, for example, in EPA Toolkit, Turbidity Regulations, Chapter 3 (April 1999).

Exemplary systems are configured to detect and identify transparent and / or translucent particles refracting and / or scattering light (e.g. protein aggregates, glass shards or plates, and oil droplets), light-reflecting particles (e.g. metal particles) , and / or light-absorbing particles (eg, carbon black and plastic particles) based on differences in their optical characteristics. Some of the disclosed visual inspection units 100 are configured to detect all three types of particles using illumination sequences such as those disclosed in the description below. The disclosed visual inspection units 100 may also be configured to specifically detect, identify, and / or track proteins that may be present as tightly bound aggregates, loosely bound cotton-like structures with a high water content, (light reflecting) crystals, gelatinous structures, and / or amorphous aggregates.

The term protein, as well as the synonymous term polypeptide in its broadest sense, refers to a substance consisting of two or more subunits of α-amino acids, amino acid analogs or peptidomimetics. Subunits can be linked through peptide bonds. In yet another embodiment, the subunits can be linked through other linkages, for example, ester linkages, ether linkages, etc. As used herein, the term amino acid refers to natural and / or synthetic amino acids, including glycine and the D and L optical isomers, amino acid analogs, and peptidomimetics. A peptide composed of three or more amino acids is often called an oligopeptide if it has a short peptide chain. When there is a long peptide chain, the peptide is called a polypeptide or protein. The term peptide fragment as used herein also refers to a peptide chain.

The container 10 can be a rectangular or cylindrical vessel made of glass or plastic (eg, a cuvette, bottle, cartridge, test tube, or syringe); the container can also have a different shape and / or can be made of a different material, provided that the contents of the container can be visualized at the imaging wavelength. While certain implementations provide clear and unperturbed rendering of the contents of the container, in still other implementations the imaging periods may coincide with the periods during which the container is unperturbed and / or post-processing can be used to compensate for distortion of the recorded data. ...

Block 100 includes a collecting optics imaging apparatus 110 that projects images of the contents of the container onto the sensor. In this case, the collecting optics comprises a telecentric lens 114, and the sensor is a charge coupled device (CCD) 112. A memory 140 associated with the CCD 112 records and stores a stream of images representing the contents of the container, and a processor 130 associated with the memory 140, analyzes the captured image sequence as described below to detect and identify particles in vessel 10. It will be apparent to the skilled person that the 130 processor can be configured in conjunction with a suitably configured general purpose computer (for example, using an Intel® Core ™ i5 processor or Advanced Micro Devices Athlon ™), a field programmable gate array (such as an Altera® Stratix® or Xilinx® Spartan®-6 FPGA), or an ASIC. The memory 140 can be solid state memory (eg, flash memory), an optical disc (eg, CD or DVD), or magnetic media, and can be any selected size (eg, 1 GB, 10 GB, 100 GB or more).

An illumination system 120 comprising one or more light sources 122a and 122b located around the container 10 illuminates the container 10 and its contents during imaging. The visual inspection unit 100 may be incorporated into the inspection module 160, also containing a spindle 150, a shaker, an ultrasonic vibrator, or other shaker to rotate, shake, or otherwise excite the contents of the container prior to imaging, and to hold the container 10 during forming the image, according to FIG. 1 (b).

FIG. 1 (c) shows a medium / high speed visual inspection platform 170 comprising one or more inspection modules 160-1 through 160-5 (collectively referred to as inspection modules 160), an automatic machine 180, a vessel tray 172, in individual vessel recesses. of which the uninspected and / or inspected containers 10 are located. Upon receipt of appropriate instructions from the user or an automatic controller (not shown), the machine 180 moves the container 10 from the container tray 172 to the inspection module 190, which receives and records time series data for particles moving along containers 10. The machine 180 then returns the container 10 to the container tray 172.

In some embodiments, the top layer of the receptacle tray 172 and / or the edges of the receptacles are made of Delrin® polyformaldehyde resin or other similar material, and the inner edges of the receptacles are beveled to prevent scratching the receptacles 10 when they are inserted into the receptacles, and removing from them. The receptacle tray 172 may contain a base layer made of aluminum or other similar material that is not easily deformable or cracked. The walls of the receptacle recesses are typically thick to provide a secure hold of the receptacles when the tray 172 is transported (eg, by a person) to and from the viewing platform 170. Depending on the design of the receptacle tray 170, it can hold the receptacles 10 in predetermined positions within the micron range to facilitate the removal and insertion of the receptacles by an automatic machine 180 that can operate to micron precision.

The machine 180 is a lifting and transport system that retrieves the containers from the tray 172, moves each container 10 along a rail 182 extending from the area above the tray 172 to the area above the spindles 160, and places the container 10 on a specific spindle 160. Some machines may also be implemented allowing the container 10 to rotate prior to placing the container 10, thereby eliminating the need for a spindle 160. In another embodiment, the machine 180 may include a six-axis mechanical arm configured to rotate, oscillate, and / or shake (i.e. shaking the needle back and forth) of the container 10, which also obviates the need for the spindle 160. One skilled in the art will appreciate that other loading and shaking mechanisms and sequences may be used in the disclosed visual inspection systems and processes.

Platform 170 operates in accordance with FIG. 2 (a). In step 202, containers 10 to be inspected are cleaned (e.g., manually using suitable solvents), then placed on tray 172 in step 204. Machine 180 removes container 10 from tray 172 and places it on spindle 160. Next, in step 206, processor 130 determines the size and location of the meniscus and / or area of interest (eg, a portion of the container 10 filled with fluid) based on an image of the stationary container 10 obtained by the imaging apparatus 110. In yet another embodiment, the user can specify the location of the meniscus and / or region of interest if the fill volume, shape and volume of the container are known with sufficient accuracy. Upon detection of a region of interest by processor 130, spindle 160 rotates and stops rotation of container 10 at step 208, resulting in fluid movement and suspension of particles in the moving fluid in container 10. At step 210, imaging apparatus 110 records time series data in memory 140 as a series of still images (called frames) displaying photographic images of an area of interest taken at regular intervals.

Once the imager 110 has acquired sufficient time series data, the processor 130 subtracts background data, which may represent dirt and / or scratches on one or more surfaces of the container. The processor can also filter out noise from the time series data in a manner obvious to a person skilled in the art, and can also perform luminance threshold determination as described below. Processor 130 also reverses the temporal order of the time series data. That is, if each data frame of the time series has a mark 1, 2, ..., n-1, n, indicating the order of receipt of the specified frame, then the marks of the data frames of the reverse time series are ordered as follows: n, n-1,. .., 2, 1. If necessary, the processor 130 also selects the start and end points of the analyzed data as described below. (One of ordinary skill in the art will appreciate that the processor 130 can perform background subtraction, noise filtering, luminance threshold determination, time series data reversal, and / or start / end point determination in any order.) Processor 130 tracks particles moving in or with a fluid , in step 214, and then sizing, counting, and / or otherwise characterizing the particles based on their trajectories in step 214.

Each inspection module 160 can perform one general inspection, which allows parallel inspection of the containers 10; the number of modules 160 can be adjusted depending on the required bandwidth. In other implementations, each module 160 may be configured to perform different types of inspection. For example, each module 160 can inspect particles at different wavelengths of radiation: module 160-1 can search for particles that respond to light in the visible spectrum (i.e., radiation with a wavelength of about 390 nm to about 760 nm), module 160 -2 can investigate capacities using near infrared radiation (760-1400 nm), module 160-3 can investigate capacities using infrared radiation in the short wavelength range (1.4-3.0 μm), module 160-4 can examine particles at wavelengths ultraviolet spectrum (10-390 nm), and module 160-5 can examine particles at X-ray wavelengths (less than 10 nm). In yet another embodiment, one or more modules 160 may be used to search for polarization and / or fluorescence effects of particles.

In embodiments with different types of modules 160, the first module 160-1 may perform a preliminary inspection, and the subsequent inspection depends on the results of the preliminary inspection. For example, the first module 160-1 may perform visual inspection, the results of which indicate that a certain container contains polarization-sensitive particles. The processor 130 may then instruct the polarization-based measurement module 160-2 to inspect the container to confirm (or deny) the presence of polarization-sensitive particles. Visible time-series data obtained by module 160-1 may indicate the presence of multiple particles in a specific container 10, but not the type of particles, as a result of which the processor 130 can command an infrared inspection to the module 160-3.

Shaking the container to move the particles

As described above, mechanically shaking the container 10 results in the suspension of particles at the bottom of the container 10 or on the inner walls of the container in the fluid within the container. In certain embodiments, the user and / or the visual inspection system selects and performs a shaking sequence that induces the fluid in the container into a laminar flow pattern in which the fluid flows in parallel layers without eddy, turbulent motion, or layer disturbances. In hydrodynamics, laminar flow is a flow pattern characterized by high momentum dispersion and low momentum circulation; in other words, laminar flow is the opposite of an uneven, turbulent flow. Shaking also causes the particles to be suspended in the moving fluid. Subsequently, friction stops the movement of the fluid, after which the particles can stick to the walls of the container or settle at the bottom of the container.

Compared to turbulent flow, laminar flow provides smoother particle movement, making it easier to estimate particle trajectories. (Naturally, the processor can also be configured to estimate particle trajectories in some turbulent flow conditions, provided that the sensor frame rate is high enough to capture images of smooth sections of particle trajectories.) If necessary, the container can be shaken to provide a substantially laminar flow. For example, the spindle can rotate the container at a certain speed (or speed profile) for a certain period of time, determined based on measurements of the behavior of the fluid in containers of different sizes and shapes and / or different degrees of filling of the vessel and the viscosity of the fluid.

In one embodiment, a servo motor or stepper motor drives a spindle holding a cylindrical container, which causes the container to rotate about its central axis as shown in FIG. 3 (a). Rotating the container 10 at a sufficient speed causes even heavy particles (such as metal particles) to rise from the bottom of the container 10 and penetrate into the liquid. When used with many liquids and particles, the motor drives the spindle holding the container 10 at 300 rpm for about three seconds. (Higher rotational speeds may be necessary to excite heavy particles.) At the end of rotation within three seconds, the engine stops abruptly, and the fluid can flow freely in the newly stationary container. At this time, the imaging apparatus 110 begins to record a video image of the rotating fluid. Memory 140 stores video images for up to about 7-15 seconds, depending on the size of the container being examined (memory 140 stores video images for a shorter duration when examining a fluid in smaller containers due to the fact that the liquid in smaller containers slows down faster due to the greater effect of friction about wall).

In yet another embodiment, the spindle rotates the container 10 in a two-phase shake / imaging sequence. During the first phase, the spindle rotates the container 10 at 300 rpm for three seconds, which results in the weighing of less dense (and smaller) particles, such as proteins, in the moving fluid. The imaging apparatus 110 then records video images of the proteins in the moving fluid. Once sufficient time series data is acquired by the imaging device 110, the second phase begins: the spindle rotates the container 10 at a speed of about 1600-1800 rpm, which leads to the weighing of denser particles, such as metal particles, in the moving fluid, and the device 110 imaging acquires time series data representing the movement of denser particles in vessel 10. The high speed rotation in the second phase can be intense enough to temporarily dissolve or denature protein aggregates that may re-form after slowing or stopping fluid flow. The two-phase principle of operation allows the detection of dense particles, the excitation of which may not be achieved with low speed rotation, and proteins, which can be denatured during high speed rotation.

In the disclosed systems, other rotation sequences can be used depending on the following parameters (without limitation): fluid viscosity, degree of filling of the vessel with fluid, type of fluid, surface tension, container shape, container size, container material, container texture, dimensions particles, particle shapes, particle species and particle density. For example, the disclosed systems can rotate larger containers for longer periods of time before imaging the contents of the container. The exact shaking profile for a given fluid / container combination can be calculated, characterized and / or determined experimentally.

If the visual inspection module uses a predetermined shaking sequence for a sufficiently known fluid / container combination, the module can only trigger the acquisition process when the fluid (and suspended solids) are in laminar flow. In yet another implementation, the module can obtain additional time series data and the processor can automatically select the start and end frames based on the selected fluid / container combination and / or shake sequence.

In some implementations, the data acquisition process can be triggered based on the detected characteristic of the fluid flow in the container. For example, as detailed below, in some implementations, it is possible to detect a meniscus of a fluid in a container and track the movement of the meniscus to determine the point in time at which the vortex in the fluid weakens after rotation. In some of these cases, the acquisition process may begin when the detected meniscus movement returns to a substantially stable state.

Any of the above-described visual inspection systems can also be used to detect and / or identify intrinsic and foreign particles in a syringe 12 at least partially filled with a drug 32 or other fluid as shown in FIG. 3B. Syringes 12 are often stored with the needle pointing down. Accordingly, particles can be deposited in the syringe needle 34. To ensure the visualization of these particles, an automaton or a person turns over the syringe 12 - i.e. an automaton or a person rotates the syringe 12 by 180 ° about an axis perpendicular to its longitudinal axis, with the result that the needle 34 is directed upward. The particles deposited in the needle 34 fall in the vertical direction, which allows them to be visualized by the imaging apparatus 110. The machine or person can also rotate the syringe as it is inverted to fully excite the fluid.

Many syringes 12 have barrels with relatively small internal diameters (eg, about 5 mm), which greatly increases the wall friction effect. For many drugs 32, wall friction causes the entire pivotal movement of the fluid to stop in about 1 second. For particle analysis in practice, such a time window is extremely small. However, gently shaking the syringe 12 about an axis perpendicular to its longitudinal axis as shown in FIG. 3 (c), allows particles to move for more than one second. Side shaking, which can be done automatically or manually, excites the particles as a result of movement of the syringe 12 and the movement of an air bubble or bubbles 30 vibrating within the barrel of the syringe 12. The above-described modules, blocks and visual inspection platforms are configurable and can be used with with the specified alternative shaking method.

Once shaking is complete, the visual inspection system must remain stationary to complete the video recording phase. Due to the high resolution of commonly used imaging devices, the spatial resolution of the images is very precise (eg, about 10 μm or less) and can be at least equal in accuracy to the diffraction limit. In some configurations, a small (for example, 10 μm) movement of the sample corresponds to a whole pixel of movement in the acquired image. This movement negatively affects the efficiency of removing static elements (background subtraction), which in turn degrades the performance of the analysis tools and the accuracy of the output.

With this in mind, vibration elimination is a key design consideration. In certain embodiments, the base of the exemplary visual inspection system is mechanically isolated from the laboratory environment, for example, by vibration damping dampers, platforms, and / or spacers. In addition, within the unit, vibration sources such as computers and automated controllers can be mechanically isolated from the rest of the system. In yet another embodiment, the acquisition of data may be synchronized with the residual movement of the container relative to the imaging apparatus, or may be performed by a camera performing pixel shift or some other motion compensating measures. This residual movement can also be captured for post-processing to eliminate the negative effects of image movement.

Imaging device configurations

Exemplary visual inspection systems can use standard available imaging devices with any suitable sensor, including, but not limited to, charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) arrays. The choice of sensor is quite wide and depends to some extent on the requirements imposed by a particular field of application. For example, high frame rate sensors provide the ability to accurately map the trajectories of rapidly moving particles (for example, in low viscosity fluids). Sensitivity and noise performance are also important due to the fact that many of the protein particles in solution are transparent and weakly scatter light, making their images subtle.

- 10 038813

To improve the noise performance, the sensor can be cooled in a manner known in the art. For most applications, monochrome sensors provide the best performance with slightly higher resolution than color cameras and also have higher sensitivity. However, for a small subset of applications, color sensors may be preferable due to the fact that they capture the color of the particle, which can be extremely important in determining its source (for example, when examining the fibers of clothing). For example, in product quality testing (also called technical expertise), color sensors can be used to distinguish between different types of materials (eg fibers) that could potentially contaminate a drug in a manufacturing facility.

For a complete inspection of the container, the inspection area of the imaging device must cover the entire volume of the fluid. At the same time, the imaging apparatus must be capable of distinguishing between small particles. Visual inspection systems allow for large inspection areas and high resolution with high-resolution wide-format sensors such as the 8-megapixel Allied Vision Technologies (AVT) Prosilica GX3300 CCD sensor with 3296x2472 pixel resolution. Other suitable sensors include the 5MP ACT Pike F505-B and Basler Pilot piA2400-17gm cameras. After selecting the optics for a complete imaging of the liquid carrying body of a BD Hypak 1 ml syringe, the AVT Prosilica GX3300 CCD sensor acquires time series data with a spatial resolution of approximately 10 µm per pixel in both lateral dimensions. The combination of high speed and high resolution suggests that high bit rates and large file sizes may be required to record time series data. As a result of the above, the video compression techniques described below have been specially selected to reduce the storage requirements while maintaining the fidelity of the fine details of the particles captured in the image.

The collection optics that feed the image of the region of interest to the sensor should be selected so that they provide clear images of the entire volume with a minimum spot size equal to or less than the sensor pixel size to ensure the system operates at the highest resolution.

In addition, the collecting optics preferably have a depth of field large enough to cover the entire sample volume.

Telecentric lenses such as lens 114 of FIG. 4 are particularly well suited for visual inspection of fluid volumes due to the fact that they are specially designed to be insensitive to depth of field. One skilled in the art will understand that a telecentric lens is a multicomponent lens in which the main beams are collimated and parallel to the optical axis in the image and / or object space, resulting in a constant magnification regardless of the image and / or location of the object. In other words, for an object located within a certain range of distances from the telecentric lens imaging apparatus, the image of the object obtained by the imaging apparatus is sharp and has a constant magnification regardless of the distance of the object from the imaging apparatus. This allows images to be obtained in which the particles on the back of the container 10 look similar to the particles on the front of the container 10. The use of a telecentric lens can also reduce the detection of ambient light when a uniform dark background is used. Suitable telecentric lenses 114 include the Edmund Optics NT62-901 wide angle telecentric lens and the Edmund Optics NT56-675 TECHSPEC Silver Series 0.16x telecentric lens.

Inherent blind spots

One of the tasks of almost any visual inspection system is to ensure one hundred percent inspection of the entire volume of the container. In practice, however, the container may have fixed regions where particles cannot be detected, as shown in FIG. 5A. First, it can be difficult to include fluid around the meniscus in the analysis due to the meniscus itself scattering light in such a way that the detector in this area can be saturated, and any particles or other elements of interest can be hidden. Secondly, in the case of using a bottle, the base of the container is usually curved at the corners (this area is usually called the leg). The curved stem distorts and essentially hides any particles that are sufficiently close to the bottom of the bottle. Thirdly, when using a syringe, the rubber plug contains a central cone that slightly protrudes into the volume of the container. Although the tip of this cone is quite small, it has the potential to hide particles. The least noticeable blind spots are caused by the curvature of the vessel.

Cylindrical containers can also be responsible for the lens effect shown in FIG. 5B (denoted by curved beams 18), which adversely affects the operation of the telecentric lens. The curved walls of the container also form blind spots 14.

FIG. 5E shows an example of a lens effect caused by a cylindrical container 10. The camera / viewer is located at the bottom of the drawing. As described above, a telecentric lens can be used when imaging particles in container 10 to provide a uniform appearance of particles in the image, regardless of their position in the container, i. E. their distance from the camera. To this end, in some embodiments, the selected depth of field of the telecentric lens is greater than the diameter of the fluid volume. In some embodiments, the curvature of the capacitance violates this principle in the absence of a correcting optical element.

As shown, the shape and magnification of the illustrated particle in the container 10 depends on the position of the particle in the container. Particle 510, located in the center of the container in the foreground, is absolutely not distorted (top inset). Identical particle 502, located on the side in the background, is most distorted (bottom inset). It should be noted that when using a cylindrical container, distortion occurs only along the horizontal axis (as can be seen in the second inset).

To mitigate these effects, an optional correction optics such as a correction lens 116 according to FIG. 5C. Additional spatial correction optics 118 may provide additional compensation for capacitance shape distortion, as shown in FIG. 5D. In various implementations, in addition to or in place of the correction lens 116 and optics 118, any suitable correction optics may be used, for example, specially selected based on the curvature of the container 10 and / or the refractive index of the fluid.

For example, in some embodiments, a model of the lens effect caused by a cylindrical container 10 can be developed. The model can be based on a suitable set of parameters characterizing optical distortion, including, for example, the outer diameter of the container, the inner diameter of the container, the refractive index of the container, the refractive index of the fluid. medium, wavelength of light radiation. The model can be developed using any technique known in the art, including, for example, a ray tracing technique. FIG. 5F shows examples of theoretical models of the lensing effect for two different groups of capacitance parameters (top left, bottom left), as well as experimental data for the corresponding physical conditions (top right, bottom right). It can be seen from the drawing that the theoretical model exactly matches the experimental data.

As shown in FIG. 5G and 5H, a correcting optical element 503 (represented by a lens in the drawing) is used to correct the above-described lens effect. The design of the corrective optical element can be based on a theoretical optical model of the capacitance, on experimental data describing the optical properties of the capacitance, or on combinations of the two indicated types of data. According to the drawing, the correcting optical element 503 is made of a refractive material and has cylindrical front and rear surfaces. In some embodiments, the lens design can be determined using free parameters, including the radius of the front and back surfaces, the thickness of the lens, the refractive index of the lens, the location of the lens relative to the container.

In some embodiments, the front and back surfaces of the lens may have a different shape, for example, parabolic or any desired shape. In some implementations, the relaxation of the requirements for cylindrical shape of the surfaces allows an increase in the size of the parameter space for the design of the correction optic element 503, which allows for improved distortion correction.

In some embodiments, the correction optics 503 may include multiple elements to further enhance design space. In some embodiments, correction optics 503 can correct other types of optical distortion, aberration, or other effects. For example, when using multiple wavelength illumination, the correction optics 503 can be used to correct chromatic aberration.

In some embodiments, the correction optics 503 may be configured to correct for distortion caused by a particular type of container and / or fluid. Due to the fact that a single automated visual inspection unit 100 can be used with several types of containers, in some implementations it may be desirable to allow selective replacement of the correction optic element 503 to match the specific container 10 under investigation. For example, in FIG. 51, a post 504 is shown that houses a plurality of correcting optics 503. The post can be moved (manually or automatically) to position a selected element in an optical circuit for imaging apparatus 110. It is to be understood that although a rack is shown in the drawing, any other mechanism may be used in various implementations suitable for selecting one optical element from a set of multiple optical elements.

Alternative visual inspection systems may include adaptive optics to compensate for distortion caused by capacitance curvature. For example, telecentric lens 114 may be configured to image a container 10 reflected from a deformable fecal mirror, such as a microelectromechanical system (MEMS) mirror. Sensor 112 uses background data to infer the nature and extent of aberrations caused by surface curvature, surface defects, and other defects in capacitance 10. Sensor 112 sends this information back to the deformable mirror, which reacts by correcting its surface to compensate for aberrations. For example, a deformable mirror can bend or bend in one direction to compensate for the curvature of the container. Due to the fact that the deformable mirror reacts dynamically, it can be used to compensate for the aberrations inherent in each individual container 10.

In addition, the particle tracking can be configured to detect the disappearance of particles in known areas corresponding to the indicated blind spots, which allows the program to predict the possibility and timing of possible subsequent particle reappearance in the video image, as described below.

Additional techniques for dealing with blind spots (eg, by using multiple imaging devices) are described below.

Camera frame rate

Effective particle tracking using the closest match (absorbing) algorithm described below can be thought of as a function of three main factors: camera imaging rate (frame rate), particle density (in 2D), and standard particle velocity. For nearly effective closest match tracking, the camera must have a frame rate fast enough to meet the following criteria:

camera frame rate> maximum particle speed / minimum distance between particles.

In practice, when projecting a three-dimensional volume onto a two-dimensional image, it is possible that the particles appear as close as possible to each other (or even block each other), while in reality they are sufficiently spaced apart in the container. Taking into account this fact, it seems more appropriate to consider the average distance from a nearby particle, rather than the obvious minimum distance between particles. It should be noted that the distance from a nearby particle is the distance between neighboring particles on a single time series data frame, and the closest match distance refers to the distance between the positions marked for one particle on successive time series data frames. Recalculating the camera speed criterion in accordance with the distance of the closest match gives:

camera frame rate> maximum particle speed / minimum distance between particles.

In alternative visual inspection systems, predictive tracking techniques may be used instead of closest match (absorbing) particle tracking techniques. Predictive techniques use data on the known trajectory of a particle, together with data on spatial capacity constraints and expected fluid behavior, to estimate the most likely position of the particle in the next frame. If performed correctly, this approach can allow for more accurate tracking of particles moving at speed in intense images.

When trying to detect and measure extremely small particles in relatively large containers, it is preferable to maximize the spatial resolution of the image sensor. In general, the direct effect of this increase is to lower the maximum possible frame rate of the sensor.

Visual inspection using multiple imaging devices

When using a single camera, the quality of the inspection may be reduced due to the presence of known blind spots. In addition, superimposing a 3D image of a particle distribution on a 2D image can lead to confusion due to obstruction of particles (eg, as shown in FIG. 5E, in which a particle in the center of the container in the background is obstructed by a particle in the center of the foreground). In alternative visual inspection systems (eg, as shown in FIG. 6), this problem can in principle be solved by correlating the results obtained from two or more imaging systems. By correlating positional trajectory information from two or more cameras, detailed 3D trajectory maps can be generated that can be more reliable and less prone to errors due to occlusion (as described below) than 2D trajectory maps.

Increasing the spatial resolution of the imaging apparatus also limits the data acquisition rate (frame rate) for a given particle concentration and particle velocity. When examining unknown containers, there can be no guarantee that the particle concentration will be sufficiently low. At the same time, in order to ensure the suspension of heavy particles, such as glass or metal, in a fluid medium, a rather high rotation speed in the container may be required, which leads to a high speed of particle movement on the resulting video stream. One way to solve this contradiction is to use the innovative imaging hardware covered in the description below. By using the best sensors available on the market and scattering enough light into the container by particles, the data acquisition rate can be further increased by compaction of data from two or more sensors with constant reliable triggering through a dedicated trigger source.

In addition, exemplary visual inspection systems may be configured to provide spatial resolution above 10 microns by eliminating the need for a complete inspection of the container and viewing only a fraction of the volume. In general, this technique is suitable for the study of invisible particles, especially protein aggregates, due to the fact that smaller particles are usually present in large quantities and are more evenly distributed throughout the volume. In yet another embodiment, exemplary visual inspection systems can provide a complete view of the vessel and high spatial resolution by using multiple imaging devices at varying magnifications to acquire wide area and high resolution time series data in parallel.

Different degrees of magnification can be used simultaneously, for example as shown in FIG. 6A, in which one imager 1102 is used to inspect the entire container, and a second imager 1104 at a higher magnification (e.g., a long focal length microscope lens) is used to detail a smaller sub-volume and examine, for example, extremely small particles (e.g. particles with a diameter of about 10 μm, 5 μm, 1 μm or less). Other visual inspection systems may include multiple imaging devices 1102, 1104, and 1106 positioned around a container, illuminated by one or more rings of LEDs 1120 positioned above and below container 10 of FIG. 6B. Identical imaging devices 1102 placed at different points provide binocular vision. A long focal length microscope imager 1104 provides high resolution for a subvolume of container 10, and an alternative sensor (eg, infrared sensor, bolometer, etc.) imager 1106 provides additional time series data.

FIG. 6C and 6D show alternative imaging configurations using telecentric imaging properties. At the rear aperture of the telecentric lens, a 50/50 beamsplitter cube 1202 divides the projected image into two separate image taps. Each of the image taps may include a low speed, high resolution sensor 1222 operating in turn with a sensor 1222 in the other arm, as shown in FIG. 6C, in order to double the frame rate. In other words, using two sensors 1222 at the same time with a relative phase shift of a half-cycle doubles the time resolution. The image streams can then be combined to provide a single video image at double (relative to the sensor's nominal) frame rate.

In yet another embodiment, each tap may include a different sensor, as shown in FIG. 6D, for example, to compensate for the tradeoff of a sensor array for imaging: the higher the resolution of the camera, the lower the maximum possible frame rate of the camera (e.g. 10-50 or 15-25 frames per second at full resolution, 50-200 frames per second at low resolution, etc.). The most important sensor performance parameter for accurate particle tracking is its high time resolution (high frame rate). However, for accurate particle size determination, the most important sensor performance parameter is high spatial resolution (the largest possible number of pixels in the image). Currently, the main limiting factor for spatial resolution and data rate is the data bus. Prior art imaging devices can acquire time series data for a 4 cm high container with a spatial resolution of about 10 µm per pixel and a data rate of about 25 frames per second for a standard personal computer bus (eg, dual GigE bus or CameraLink).

FIG. 6D illustrates one way to achieve high frame rate and high resolution: a fluid image is acquired by a low speed, high resolution sensor 1222 and by a sensor 1224 with a lower spatial resolution, but at a higher frame rate. An external trigger can provide comparable synchronization between the two cameras. Because cameras capture copies of a single image, camera data can be directly correlated to provide improved particle analysis.

FIG. 7A and 7B illustrate timing and control of lights 120 and multiple cameras. FIG. 7A and FIG. 7B, the trigger controller 702 emits two trigger signals, designated ARM 1 and ARM 2 in FIG. 7A and 7B, the signals being derived by decimating the main pulse signal. The trigger signal ARM 1 drives the first camera (item 1102a in FIGS. 7A, 1222a in FIG. 7B) and the trigger signal ARM 2 drives the second camera (item 1102b in FIGS. 7A, 1222b in FIG. 7B) in turn. In other words, the first and second cameras receive sequential frame sequences under the action of the triggers. The trigger controller 702 may also drive the illumination source 120 with an illumination signal, upon receipt of which the illumination source 120 illuminates the container each time the first or second camera acquires an image. Other startup sequences are also possible; for example, trigger controller 702 can drive additional cameras and / or combinations of high and low resolution cameras to capture images at different frame rates.

It will be apparent to the skilled person that other configurations are equally possible. For example, the imaging sensors at each of the taps may be mutually equivalent, but the converging optics may be different. One arm can contain additional optics to enlarge the image in order to detail a specific area of the image, providing wide-field and enlarged views at the same time.

Lighting configurations

The claimed visual inspection systems use the interaction of various particles with light to detect and identify particles in containers containing liquid. The interaction of a particle with light is a complex function of a number of factors, including particle size, shape, refractive index, reflectance, and opacity. Protein-containing particles can mainly scatter light by refraction, and plate-like glass particles can mainly reflect light. Certain particles, such as collagen fibers, can change the physical properties of light, such as rotation and polarization. High-precision detection and differentiation can be achieved by adapting the detector, particle and light geometry to maximize the contrast between different types of particles.

FIG. 8-12 show different lighting configurations adapted or switchable / actuated to different lighting types for particular types of particles, container and / or fluid. For example, light sources can illuminate particles to maximize the amount of light reflected or refracted by them towards the detector, leaving the background dark to maximize the contrast between the particle images and the background. In addition, sources can emit radiation at any suitable wavelength or in any wavelength range. For example, sources can emit broadband white light (390-760 nm), a narrow-band beam (eg 632 nm), or even ultraviolet or X-ray radiation. Suitable ranges include: 10-3000 nm, 100-390 nm (ultraviolet), 390 760 nm (visible), 760-1400 nm (near infrared), 1400-3000 nm (mid infrared). X-rays are also possible (<10 nm). In its entirety, the scope of illumination options disclosed in the present invention allows the claimed visual inspection systems to detect and identify the full range of particles that may be present in medicinal products.

Due to the fact that some particles are extremely weak in scattering light, it is often preferable to irradiate the sample with as much light as possible. The upper limit of possible sample irradiation primarily depends on the photosensitivity of the test product. Weighted choice of wavelength may also be necessary, especially for biological products; the specific choice depends on the irradiated product. Monochrome red light with a wavelength of about 630 nm represents a sweet spot, and this wavelength is widely available in inexpensive light sources.

LED arrays such as CCS Lighting's range of LDL2 LED arrays are effective at illuminating particles found in pharmaceutical products; however, collimated laser beams can also be used. In some cases, light optics can structure or shape a radiation beam collimated inside the volume of the fluid medium (and not outside the container). In the event that heating the light source using alternative light sources is undesirable, light can be delivered to the inspection site using optical waveguides or optical fibers 124 of FIG. eight.

The illumination wavelength can be selected based on the light absorption and / or reflectivity of the fluid and / or the analyzed particles; this process is especially important in the study of photosensitive pharmaceutical products. Red light (630 nm) provides an acceptable balance between low light absorption of protein and low light absorption of water. Lighting gating, synchronized with time series data, helps preserve the integrity of light-sensitive pharmaceutical products by minimizing exposure to incident light. Gating has two additional benefits: LEDs operate more economically in this mode, and gating reduces the effect of motion blur, which, if uncorrected, can negatively affect particle size measurements, as described below.

FIG. 8 illustrates an exemplary variable lighting system 120 comprising multiple light sources 122a-122f (collectively referred to as light sources 122), which may be LEDs, lasers, fluorescent or incandescent lamps, flash lamps, or any other suitable light source or combination of suitable sources. Sveta. Sources 122

- 15,038813 lights can emit visible, infrared and / or ultraviolet radiation. The light sources can be narrow-band or wide-band as desired and can be filtered by means of appropriate optical filters or polarizers. FIG. 8, for example, polarizer 126 polarizes light emitted from light source 122f providing background illumination of the container. In addition to the background light 122f, the lighting system 120 includes four light sources 122a-122d located at the corners of a rectangular prism around the vessel 10. Another source 122e illuminates the vessel 10 from below via an optical fiber 124 coupled to a collimator 126 directed towards the bottom of the vessel 10. In some cases, the fiber 124 and collimator 126 may be located within the hollow shaft 128 of the spindle used to rotate the vessel.

Several light sources 122 shown in FIG. 8 can be used to determine the optical properties of a particular particle to provide differentiation based on the interaction of the given particle with light. It will be clear to the skilled person that different particles interact with light in different ways. Common types of interactions include scattering, reflection, occlusion or polarization change of light according to table. 1, in which the X indicates the visibility of a given type of particle when using a given type of lighting; examples are shown in FIG. 9A9D and FIG. 11 (described below). The M symbol indicates the possible visibility of particles of a given type using a given type of illumination, but particles can also be detected / differentiated by image segmentation during subsequent processing and by means of object identification techniques.

Table 1. Interaction of various types of particles with light

Particle type

Protein Plates Opaque Cellulose Air Particles

Prevailing interaction

Method Diffusion Reflection Occlusion Change Diffuse illumination polarization

Back angle XXXXX

Bottom XM

BackgroundX illumination

Polarizing MM XM light

FIG. 9A-9C show various examples of lighting that may be implemented in the lighting system 120 of FIG. 8 (some light sources 122 have been omitted for clarity) to differentiate particle species based on their interaction with light. FIG. 9A, light sources 122a and 122b provide back corner illumination suitable for imaging proteins as well as most types of light scattering particles. FIG. 9B, the light source 122e provides illumination from below, suitable for rendering light reflecting particles, such as glass plates, reflecting light towards the imager 110 (indicated by the horizontal arrow); particles that scatter but do not reflect light (eg proteins) may not be visible to the sensor (diagonal arrow). FIG. 9C, the light source 122f provides a uniform backlighting from the back, suitable for rendering particles occluding light, such as metal particles, dark plastic particles and fibers. It will be apparent to those skilled in the art that other light sources and / or examples and lighting sequences are also possible.

FIG. 9D illustrates the sequential use of the lighting techniques of FIG. 9A-9C to obtain time series data for scattering, reflective and / or occluding particles. In this case, a system comprising uniform backlight, rear corner lighting, bottom lighting, and a single camera alternately illuminates each frame such that only one specific light source 122 (or combination of lights 122) is active at any one time. When using one imaging device (not shown), only one set of light sources is used for each acquired time series data frame. Repetition of the specified 16,038813 sequence allows for video imaging for each lighting configuration.

Acquiring a sequence of video images using the above-described lighting techniques in sequence provides a nearly simultaneous video image for each light source 122. Upon completion of the process, this method provides three intermittent video images, one for each illumination method. In each video image, a particle in a single frame can correlate with a specified particle in the other two video images obtained using alternating lighting methods (a slight time difference between frames is not taken into account). Using the general information obtained on the basis of the interaction of a single particle with different methods of illumination, it is possible to draw a conclusion about the material composition of the particle.

This method can be combined with other information of the selection of objects in the image in order to increase the certainty. For example, video images can be automatically split to identify objects at each frame. Information such as size, shape, brightness, smoothness, etc. can be automatically determined for each object and for each lighting method. This process can help differentiate between different types of particles that have similar visibility when using each of the different lighting methods.

FIG. 10A-10C show a method for reducing glare caused by unwanted reflection / refraction of light emitted by light sources 122 from container 10. Illumination of container 10 results in unwanted glare in images obtained by imaging devices 110, the optical axes of which coincide with the direction of movement of light from the sources. 122 light reflected from the surface of the container. Flare can obscure otherwise detectable particles and saturate areas of the sensor. The flare detected by the sensor can be reduced or eliminated by positioning the imaging apparatus 110 or light sources 122 such that the optical axis of the imaging apparatus does not align or parallel with the light rays emitted by the light sources 122 and reflected from the container. For example, placing the light source or sources 122 outside of an enclosed area defined by rotating the imaging apparatus about the longitudinal axis of the container 10 can reduce the amount of unwanted reflected and / or refracted light received by the sensor. In another embodiment, the area 100 may be defined as a plane orthogonal to the central axis of the cylindrical container, the thickness of which is equal to the height of the vertical walls of the containers. It will be apparent to the skilled person that containers having more complex shapes, such as concave side walls, may have different closed areas and different correction optics.

Oblique illumination of the sidewalls of the container from a point above or below zone 1000, or directly below the base of the container, also reduces the flare detected by the imaging apparatus 110. Illumination of the container 10 from below (eg, by light source 122e (FIG. 8)) also allows excellent contrast between light reflecting particles (eg glass slabs) and light scattering particles (eg protein particles).

FIG. 10D, 10E illustrate an alternative illumination scheme for reducing or eliminating glare from container 10, in which one or more light sources 122 are located in the above-described closed area (for example, in the horizontal plane of the container 10).

FIG. 10D, 10E show a model of beam optics showing the movement of beams outside of the sensor of the imaging apparatus 110, through the imaging optics of the imaging apparatus (as shown, including the telecentric lens) and back through the container 10. A light source located along any of the rays passing through back from the sensor, refracts or reflects light onto the sensor, thereby potentially hiding the container 10 and its contents. However, it should be noted that two portions 1001 are provided located in the horizontal plane of the container 10 in the vicinity of the outer wall of the container 10. As shown in FIG. 10E, by placing one or more light sources 122 on portions 1001, glare from light sources can be reduced or substantially eliminated.

It should be noted that due to the use of a telecentric lens in the shown example in the model of ray optics, it is necessary to consider only the rays incident on the sensor along the normal. However, given the additional beams, a similar approach can be applied to other types of imaging optics. For example, in some embodiments, a representative set of beams from the sensor (eg, including the original beams of the imaging system) can be traversed back to identify areas that are devoid of, or substantially devoid of, backward beams. Lights can be placed on the detected areas to avoid glare.

FIG. 11 shows a design for distinguishing elongated protein aggregates from cellulose and / or fibers (natural or synthetic) by means of polarized light. Illumination system 120 emits light towards container 10 positioned between crossed polarizers 900 to provide a black image in the absence of particles. Particles that change (for example, rotate) the polarization of the incident light, on the time series data detected by the device

- 17 038813 VOM 110 imaging, appear white.

In case the particles of interest have known fluorescence, fluorescence imaging can be used to identify the particle according to FIG. 12. In this case, illumination source 920 emits blue light to excite the particle of interest. A narrow band (eg, green) filter 922 placed in front of the imager 110 allows only fluorescence from the excited particles to reach the detector. The given illumination and filter wavelengths can be selected to match the specific wavelengths of interest.

Finally, non-scattering (refractive) or non-reflective particles, such as small particles of black opaque material, can be detected (and identified). To detect such opaque particles, the sample must be illuminated directly from behind. In this case, the particles can be distinguished as dark elements against a bright background. The images of opaque particles can be inverted, if necessary, to obtain images that have the same polarity as the images of the scattering and reflective particles (i.e., resulting in particles as light spots on a dark background instead of black spots on a light background).

Specific visual inspection platforms for plate detection

As known to those skilled in the art, flakes of glass are thin, flexible particles or flakes of glass formed by chemical reactions on the inner surfaces of glass containers. The claimed systems and techniques can be used and / or adapted for the detection, identification and counting of glass plates in order to minimize the likelihood of prescribing drugs containing glass plates to prevent the prescription of drugs containing (excess) glass plates. The claimed systems and techniques can also be used and / or adapted to study the formation of glass plates, depending on the composition of the drug in question; glass plates differ from proteins and other types of particles in that they reflect and scatter light. Without being limited to any particular theory, we note that presumably certain conditions are more conducive to or prevent the formation of glass plates. For example, glass vials made by tubing and / or high temperatures are generally less lamellar resistant than molded glass vials. Drug solutions prepared at high pH (alkaline) levels and with some buffers such as citrate and tartrate also promote platelet formation. The length of time that the product is exposed to the inner surface of the container and the temperature of the drug will also affect the likelihood of glass flake formation. For more information, see, for example, FDA Drugs Manufacturers Guidelines: Glass Plate Formation in Certain Injectables (March 25, 2011) (www.fda.gov/Drugs/DrugSafety /ucm248490.htm), incorporated by reference in its entirety into this application.

To create a differentiation system based on this principle, the imaging device can be standardly fitted to the bottle in such a way that light falls through the bottom of the container (orthogonal to the camera axis). This configuration allows only a small signal from light scattering particles (eg proteins) and a strong signal from reflecting particles (eg glass plates) to be received. In other words, as the plates float through the vessel, they periodically sparkle in the image. This method has a high specificity of differentiation of particle-plates from protein aggregates. In addition, the signal obtained using this imaging technique correlates with the concentration of the plates inside the vial. As a result, this technique can not only be used for non-destructive detection of plaques in commercial products, but also as a tool for determining drug formulations leading to increased / decreased availability of plaques.

FIG. 13A and 13B show maximum luminance projection (MIP) images of glass (FIG. 13A) and protein (FIG. 13B) slabs obtained by the disclosed visual inspection system. Known MIP images are used in computed tomography to provide visualization of three-dimensional space viewed along one spatial axis, for example, the z-axis. A normal MIP image displays the maximum data value taken along an optical beam parallel to the imaging axis. However, in this case, the MIP of FIG. 13A and 13B are visualizations of data depicting the evolution of a 2D image over time, i. E. they are projections along the time axis, not the space axis.

To generate the MIPs of FIG. 13A and 13B, the processor selects the maximum value of at least some of the pixels from the time series data, where each pixel represents the amount of light reflected (and / or transmitted) from a corresponding point in space in the vessel. By mapping the obtained values, a MIP image similar to that shown in FIG. 13A and 13B, displaying the brightest previously recorded pixel value. Processor evaluates MIP

- 18 038813 image by counting the number of pixels in the MIP that exceed a specified threshold. If the estimated value exceeds the previously recorded value representing the number of plates in a similar vessel, the processor determines that the content of glass plates is statistically probable in this vessel. The processor can also determine the degree of fouling by the slabs by estimating the number, average size and / or size distribution of the slabs of glass based on the MIP image.

The claimed systems can also be used to distinguish between glass plates and other particles in a vessel, for example, based on differences in the volume of light reflected by the particles as a function of time, and / or based on differences in the volume of light transmitted by the particles. Some non-platelike particles can reflect light received from a light source illuminating the vessel from below (eg, source 122e in FIG. 8) onto a detector. Glass particles, metal particles and foreign fibers, for example, can constantly appear in the image when using the bottom illumination configuration. These types of particles are invariably detected as they move through the container, in contrast to the plates, the detection of which depends on their spatial orientation, and which are distinguishable only for a few frames each time their location causes them to reflect light towards the imaging device. For images of time series data under illumination from below, particle tracking can be applied to track constantly visible but moving particles. These paths can then be removed from the MIP calculation used to estimate the number of plates, or, in another implementation, can be included in a method of obtaining general information of light to determine the type of interaction of a single particle with other spatial orientations of illumination. For example, a metal particle reflecting light can be tracked using a bottom lighting configuration. However, the specified particle absorbs light when illuminated by a backlight from the back (for example, by the light source 122f in FIG. 8). By using both of these indicators, differentiation between a metal particle and a glass particle that reflects light when illuminated from below, but does not absorb background light, can be ensured.

Detection, tracking and characterization of particles

As described above, the visual inspection unit 100 of FIG. 1 can record a high-quality, high-resolution monochrome image stream (time series data) of bright particles imaged against a dark background. (In yet another embodiment, the particles can be displayed as dark dots on a white background.) Because a drug can contain a wide variety of fundamentally different particles, time series data can be analyzed using a number of different approaches to differentiate objects in the image. and background. Often, the appearance of a particle in one image (time series data frame) is not enough to accurately quantify highly important parameters (for example, the number / size of particles). For example, a visible image of one particle in one frame of time series data may actually be an image of two or more particles colliding with or passing each other, which can lead to inaccurate particle counts and / or particle size estimates.

The temporal correlation of objects in the image on several frames of the video sequence allows to increase the accuracy of counting the number of particles and measuring their sizes. The process of matching objects on successive frames to form a time-dependent trajectory for each particle is called tracking, registering, or assigning particles. There are particle tracking techniques for other applications (eg experimental fluid dynamics). However, these areas usually use well-defined spherical tracer particles. The application of this principle in the field of drugs and other liquids requires a much more complex solution. In addition, for some types of particles, temporal analysis (tracking) is not always practical. In such cases, as an alternative, a statistical approach can be applied, which allows obtaining data from measurements of particle characteristics.

FIG. 14 provides an overview of the high-level 1300 particle detection and identification process, starting with 1310 time series data. The time series data (and / or inverse time series data) is preprocessed 1320, after which the preprocessed inverse time series data is used for two-dimensional identification and measurement of 1330 particles, which may include statistical analysis 1340 and / or tracking 1350 particles based on the inverse data. time series. As described above, reverse time series data is time series data whose frame order is presented in reverse chronological order. The 1360 report is generated after the identification and measurement of 1330 particles is completed.

Time series data preprocessing

Preprocessing 1320 includes removing 1321 static elements (background subtraction), suppression / filtering 1322 noise in the image, determining 1323 a brightness threshold. When removing 1321 static elements, the fact that the rotation of the container excites the liquid and the particles contained in it is used. Their dynamic movement allows them to be distinguished from other objects in the image. Due to the fact that the imaging process is started after the rotation of the container is completed, any moving object is mistaken for a possible particle. Consequently, static elements are not important and can be removed from the image for greater clarity.

In one implementation, the minimum luminance projection provides an illustrative pattern for static elements in an image. Static elements include, for example, scratches, dirt and defects that may be present on the vessel wall. This static element image can then be subtracted from the entire video sequence to form a new video sequence containing only moving elements on a dark background. For example, in FIG. 15A and 15B show one frame of time series data before and after removal of static elements. Flare, scratches, and other static elements obscure some portions of the container in FIG. 15A. Subtraction of the background removes many static elements to obtain an image (FIG. 15B) containing more clearly visible moving particles.

The problem with this approach is that most glass defects, such as scratches on the surface, scatter a relatively significant amount of light, and appear bright white in the resulting image due to saturation of the detector pixels. Subtracting these elements can lead to the formation of dead areas in the image. When particles pass behind or in front of said illuminated defects, the particles can be partially obstructed or even completely disappear. To solve this problem, the image of static elements can be stored, analyzed and used to correlate the location of defects with the position of the particles in order to minimize the effect of surface defects on the data of sizing and counting particles. (It should also be noted that pre-cleaning the vessel before using the system is recommended to remove surface defects as much as possible.) Data can also be filtered 1322, for example, to remove high frequency and / or low frequency noise. For example, superimposing a spatial bandpass filter on (inverse) time series data can remove and / or suppress data exceeding the first spatial frequency or the second spatial frequency.

After removing the background elements, a threshold value 1323 of the time series data is determined by converting the luminance value of each pixel to one of a predetermined number of values. The black and white images shown in FIG. 16A and 16C, and normalized according to the eight-bit scale shown on the left (other possible scales include 16-bit and 32-bit). Each pixel has a luminance value between 0 and 255, where zero represents no detected light and 255 represents the highest amount of light detected. Bringing the indicated brightness values (127 and below) to zero and reducing the indicated brightness values (128 and above) to 255 allows obtaining black-and-white images shown in FIG. 16B and 16D. It will be apparent to those skilled in the art that other thresholds (as well as multiple thresholds) are also possible.

Particle detection

Effective particle detection in an image is based on a number of image processing and segmentation techniques. The term segmentation is understood as a computational process during which objects of interest in the image are simplified to obtain discrete, easy-to-use elements. Segmentation techniques for separating objects from an image are widely used, for example, in the field of medical imaging, and these techniques are used to identify particles. In general, images from the camera are preprocessed using luminance threshold detection, background (static) subtraction, filtering (eg, bandpass filtering), and / or other techniques to maximize contrast. Upon completion of preprocessing, the processor 130 segments the image and classifies the areas accordingly. Suitable segmentation approaches include, but are not limited to, confidence join, watershed, level function, graph splitting, squeeze-based approach, clustering, region augmentation, multi-scale approach, contouring, histogram-based approaches. Once images are acquired, segmentation can provide additional information to correlate a specific element in the resulting image with a particle view. For example, information about a particular segmented element, such as area, perimeter, brightness, clarity, and other characteristics, can then be used to determine the type of particle.

Particle tracking and time series reversal

Of particular importance is the fact that none of the prior art particle identification tools fully takes into account the temporal behavior of the particles as they move through the vial. When measurements are taken from a single photographic image, particle counting and sizing may not be accurate. However, time series data provide a more complete picture of particle behavior, which can be uncovered by tracking 1340 particles, allowing time-dependent dynamic tabs to be generated for each individual particle, which provides a much more reliable and accurate measurement of its basic properties. Particle tracking is a technique widely used in video microscopy, as well as in technical hydrodynamics (in the latter field, this technique is often referred to as velocity measurement by tracking the trajectories of particles).

Although the measurement of velocity by tracking particle trajectories is known in the art, most particle tracking solutions assume that the movement of particles between successive frames of the video image is negligible, and the specified movement is less than the standard distance between particles in a single image. ... In such cases, it is sufficient to link the positions of the particles by determining the nearest neighboring particles in two images. However, in many applications this model is not suitable. Due to high rotational speeds (for example, about 300 rpm, 1600 rpm and / or 1800 rpm) and potentially high particle concentrations, it is likely that particles move between successive frames at a distance significantly greater than the standard distance between the particles. This problem can be solved by using a certain type of predictive tracking, including searching for a particle in the area predicted based on the previously noted movement of the particle. Predictive tracking includes evaluating physical equations to mathematically predict the approximate future position of a particle in the next frame, as shown in FIG. 17. To improve performance, this predictive tracking phase may be supplemented with local fluid behavior data (if any), for example as described with reference to FIG. 21C.

Accurately predicting a specific trajectory may require elements of known data on which to build the trajectory. The problem is that at the beginning of the sequence of images, when the particles are moving at the fastest speed, there may be little or no known data on which the position prediction is based. However, over time, friction against the wall in the container leads to a deceleration and subsequent complete stop of the movement of the rotating fluid. If you capture the time series data for a sufficient period of time, you can get frames in which the particles slow down significantly or even stop.

Reversing the history of the video image 1331, whereby the particles in the image are initially stationary and slowly accelerate as the video image is viewed, allows for the provision of elements of known data to determine the trajectory. At the beginning of the video, when the particles are now moving very slowly, the principle of nearest coincidence can be used to plot the initial phase of each of the trajectories. Then, at the appropriate time, the system can go into predictive mode. This reversal of the history of acquired data significantly improves performance.

FIG. 17 shows an overview of the historical reversal predictive tracking process. The task of tracking particles is to form a connection between the position of the particle ai on frame i with its position ai ₊₁ on frame i + 1, according to FIG. 17 (a). The specified process does not cause difficulties if the distance of movement of the particle a between frames is less than the distance d from the nearest neighboring particle b to it. In the event that the direction of movement of the particle is unknown or random, the simplest methodology is to provide a search area, usually a circle with a radius r _s , and the value of r _{s is} chosen such that it is greater than the expected range of movement of the particle, but less than the standard distance d between particles, according to FIG. 17 (b). After reversing the video history, as shown in FIG. 17 (c), the particles in the image begin to move slowly. However, after a while, the particles in the image are accelerated, and the closest match method may start to give incorrect results. The first few frames of the inverse time series data partially define the trajectory, providing some data on the particle's velocity and acceleration. This information can be substituted into the appropriate equations to predict the approximate location of the particle in frame i + 1, as shown in FIG. 17 (d). The above tracking method is significantly more efficient than simple closest match tracking, especially in samples with high density and / or high particle velocity.

Center of mass detection

FIG. 18A and 18B show the detection of the center of mass for particles on (inverse) time series data after the brightness threshold has been determined. First, the processor 130 converts a grayscale image (FIG. 18A) into an image with a certain threshold of brightness (FIG. 18B). Each particle in the image is presented in the form of a two-dimensional projection, the shape and size of which depend on the shape, size and spatial orientation of the particle at the moment of taking the frame. The processor then calculates the geometric center, or center of gravity, of each of the two-dimensional projections (eg, indicated by xi and yi coordinates) by any suitable method (eg, vertical, geometric decomposition, etc.). Processor 130 may compare frame by frame the location of the center of gravity of a particular particle to determine the trajectory of the particle.

- 21 038813

Particle occlusion

Each of the visual inspection systems disclosed herein projects a three-dimensional volume (container and contents) onto a two-dimensional surface of an imaging sensor. When using a single 2D sensor, particle paths can intersect in a 3D volume. In this case, one particle can partially or completely obscure the other, according to FIG. 19. FIG. 19 (1), a new particle is identified in the sequence of images; Tracking a particle in a sequence of images allows a number of successive steps to be obtained, as shown in FIG. 19 (2). To search for potential matches on successive frames, the search area of FIG. 19 (3). Several probable particles are periodically found in the search area, as shown in FIG. 19 (4), in which case the system selects the best match. It will be apparent to those skilled in the art that the best match can be selected through any of a combination of different approaches. For example, data displaying a probable particle in one frame can be compared and / or correlated with data displaying a particle in a previous frame. Comparison and / or correlation of parameters, including but not limited to size, shape, brightness, and / or changes in appearance in the image, allows for a match for a likely particle. The disclosed visual inspection systems are configured to discriminate between collisions, obstructions, and temporary disappearances of particles, such as obstruction according to FIG. 19 (5). After re-detecting the particle according to FIG. 19 (6) the particle path can be rebuilt. The claimed systems also make it possible to resolve conflicts caused by the collision of two paths (and their search zones) while ensuring the construction of correct trajectories according to FIG. 19 (7).

FIG. 20 shows another case of particle obstruction in a two-dimensional image: FIG. 20 (a) is a conventional image of suspended particles. FIG. 20 (b) - (e) are enlarged views of the highlighted area of FIG. 20 (a), in which two particles approach each other from opposite sides. In the following data frames of the (reverse) time series, it can be seen that, as a result of occlusion, two particles in the image appear to be one, unnaturally large particle. In the case of partial occlusion (Fig. 20 (c)), in such an image, the particles may appear as one unnaturally large particle. In the case of complete occlusion (Fig. 20 (d)), the finer particle may completely disappear from the field of view, and the total number of particles counted may decrease by 1 particle. This situation can be extremely important in the study of medicinal products due to the fact that an artificially overestimated measurement of the size may be sufficient to exceed the normative regulation, despite the fact that the drug in question contains only permissible, invisible particles. At the point in time shown in FIG. 20 (e), particles have moved outside each other and independent tracking can be resumed. The visual inspection system can automatically correct occlusion errors by analyzing particle trajectories and subsequent time-dependent size profiles, resulting in a lower false rejection rate.

Counting Lost Particles

As described above, particles can disappear from a portion of a video sequence for a number of reasons. Particles can pass through the blind spot and / or dead spot resulting from the removal of static elements, as described above. Finally, some types of particles can exhibit optical appearance and disappearance (brilliance) behavior when viewed through imaging optics. In such cases, the processor can predict the movement of said lost particles as described below. When the particle reappears at the expected location within a given time interval, the processor can associate trajectories and interpolate virtual data about the particle for intermediate frames. It should be noted that from a regulatory point of view, it is important to provide appropriate labeling of the virtual data about the particle in order to be able to distinguish the indicated virtual data from the true, measured data about the particle.

FIG. 21A-21C show one technique for tracking and re-detecting lost particles, i. E. particles temporarily disappearing from the field of view during the sequence of video images. The disappearance can be caused by obstruction behind another (larger) particle, obstruction behind a surface defect, passing through a known blind spot, or simply by a property of the optical geometry of the particle (for example, some types of particles can be visible only in certain spatial orientations). Finding or re-detecting particles that disappear from the field of view improves the accuracy of particle detection and identification.

FIG. 21A shows predictive tracking for detecting a particle obstructed by a defect on the surface of a container. A surface defect scatters a significant amount of light, saturating the corresponding area of the image. After removing static objects, the specified area becomes a dead zone in the image. Any particles passing through the specified area temporarily disappear. The processor 130 can re-detect the lost particles by generating virtual particles in a limited number of steps. In case of reappearance and detection of a particle,

- 22 038813 the paths of the particles are combined.

In particular, processor 130 uses predictive tracking to determine the speed of a particle before it disappears. The processor can also use predictive tracking and particle velocity to extrapolate the expected location of the particle. If the particle reappears at the expected location, the virtual locations can be linked to form a complete trajectory. If a particle does not appear again within a specified period of time, it can be marked as irretrievably lost, and its tracking is stopped.

FIG. 21B shows a method for tracking a particle undergoing significant acceleration or direction change during a period when the particle is invisible. Instead of predicting the trajectory of the particle, the processor 130 retrospectively associates the fragments of the trajectories using data about the nature of the local fluid behavior. In this case, the processor 130 has combined the trajectories by considering the characteristics of the laminar fluid flow at a given speed and scale.

FIG. 21C shows the disappearance and reappearance of particles as they pass through known blind spots. In this example, the particle passes through the known blind spot at the far end of the container. Programming processor 130 using blind spot location information in the container image allows processor 130 to reconstruct the trajectory.

Irregular particles

Some particles are not spherical or not small enough to be viewed as a point, which is used in most particle tracking techniques. As such, many particles are irregular in shape and can roll and rotate relative to the chamber as they travel through the fluid as shown in FIG. 22A-22C. In some cases, a particle of irregular shape may appear in the image as two separate particles, each of which has its own trajectory, as shown in FIG. 22B. Such an unpredictable movement of the established center of mass of a 2D object can obscure the true movement of the particle. This behavior seriously complicates the predictive tracking process. The visual inspection system of the present invention may have a function to account for apparently distorted movement of an irregularly shaped particle, for example, by calculating an average trajectory as shown in FIG. 22A and 22C, irregularly shaped particles.

Specific vessel / product hydrodynamics

The movement of particles in the container at the end of rotation is the result of a combination of fluid movement and gravity. Fluid movement is a function of fluid viscosity, fill volume, container shape and size, and initial rotational speed. Particle tracking performance can be significantly improved by incorporating the physical constraints of the fluid system into the trajectory construction process.

The hydrodynamics of fluids rotating in known containers can be surprisingly complex under certain conditions. The incorporation of hydrodynamic data (in relation to containers widely used in pharmacology) in the trajectory construction process is an area of significant innovation and development over the prior art.

FIG. 23 shows some examples of fluid behavior in prior art containers, with the results of the computational model being compared to actual particle paths obtained by the visual inspection platform. In the course of the study, unexpected subtleties were discovered: for example, in FIG. 23 (d) shows the movement of particles along a narrow vertical column in the center of the vial caused by the weakening of the vortex generated during the rotation phase (FIG. 23 (a)). As the vertical movement of the fluid in the specified central column upward, the fluid can pick up and carry upward heavy particles, which in theory should sink. This situation can, for example, lead to errors in the identification of bubbles, which theoretically should float, and foreign particles, which float due to the movement of the fluid, specific to the given container.

The claimed systems of visual inspection can use the known data on the expected hydrodynamics of drugs to obtain more accurate results, which is not possible in other cases. Combining a physical model like that shown in FIG. 23 with particle tracking represents a significant improvement over existing technology.

Correction of errors

Although the visual inspection systems of the present invention demonstrate robustness under most experimental conditions, the difficulty of tracking a large number of particles moving in a small 3D volume implies a constant risk of errors, mainly resulting in incorrect trajectories between successive frames when collision of particles. This phenomenon is shown in FIG. 24A.

Understanding the physical limitations of a visual inspection system can be beneficial. In general, the main type of local fluid movement around each particle is laminar (rather than turbulent or random) movement. Essentially, this means that given a fast enough camera, the natural trajectories of particles in a given system should vary slightly without unexpected abrupt changes in direction, especially as the particles pass the center of the container shown in the image. After the initial linking of trajectories is complete, the system can retrospectively analyze the trajectories to detect such errors. If errors are found, the system can compare the trajectories passing nearby in order to possibly find a more physically logical solution. This process is shown in FIG. 24V.

Accurate particle counting

The number of particles can be inferred from a count of the number of particles in a photographic image taken at one point in time (eg, as shown in FIG. 24A) after detecting the particles, with each particle being assigned a sequential number. This approach is simple, but when used, the counting of the number of particles in the volume is often incomplete for a number of reasons. For example, one or more particles may be obscured by another particle or surface defect. Particles can be located in known (or unknown) blind spots. In addition, extremely small or dim particles can periodically appear and disappear from the field of view as they move between the measurement limits.

One of the advantages of particle tracking disclosed herein is that the process allows for all of these problems. As a result, particle counting can be improved to provide reliable particle tracking by counting the number of individual particle paths (as shown in FIG. 24B) rather than the number of particles in a single image or statistical analysis of multiple images. Counting the number of particle trajectories instead of the number of particles in one frame (or set of frames) is a significant improvement over the known particle tracking techniques. The increase in accuracy will vary depending on the number and size of particles present. In general, as the number of particles increases, the likelihood of obstruction increases, and, accordingly, the increase in accuracy due to the timing of the claimed particle tracking process increases proportionally.

Accurate particle size counting

Known particle measuring systems measure particle size based on static images. Most often, this process is carried out by measuring the length of the longest apparent axis of the particle, or the Feret diameter, as shown in FIG. 25, in accordance with regulatory and / or industry standards, which can be taken as the largest particle size. According to this definition, hairs with a length of 1 mm are in the same classification category with a spherical particle with a diameter of 1 mm. Considering the above, the maximum Feret diameter is a reasonable measurement based on the 2D image. However, measurement of particle size from static images entails a number of significant problems.

First, in a two-dimensional projection of a three-dimensional volume, several particles can easily overlap each other to form one, much larger particle in the image. In an industrial field in which regulatory structures impose very strict upper bounds on the particle size tolerance, the above is an extremely significant problem, especially in the manufacturing field, where such an approach can lead to false rejections, especially when studying samples with dense populations of particles.

Second, irregularly shaped particles can rotate unpredictably (relative to the camera) as they move through the container. When using a single two-dimensional photographic image, it may not be possible to accurately ensure that the greatest extent of a single particle is orthogonal to the camera's viewing axis. Consequently, the system can systematically underestimate the particle size, which can have extremely undesirable consequences in a highly regulated industry. Studying the time-dependent maximum Feret diameter of a particle as it travels through the container by tracking the particle provides a much more accurate measurement of the longest particle size.

Third, as the particles move through the cylindrical container, the particle length axis usually corresponds to the direction of movement of the surrounding fluid current, as shown in FIG. 25A and 25B. In general, when using a cylindrical container, this means that elongated particles may appear larger in the center of the image compared to the far side edges. The imaging apparatus typically detects the maximum apparent particle size (Feret diameter) when the particle moves orthogonally to the optical axis of the imaging sensor. By tracking an individual particle as it travels through the vessel, the correct maximum particle elongation can be accurately measured, which is difficult to achieve with a static measurement procedure.

Finally, despite all attempts to minimize the blurring effect by strobing lighting (as described above), it is still possible for some blurring to occur at the start of the imaging sequence when fluid and particles move at maximum speed. By using time-dependent particle size analysis, data artifacts caused by image blur (which typically increase the measured particle size) can be detected and suppressed.

FIG. 25C-25E show the process of using time series data to track particle trajectories to provide more accurate particle size measurements. FIG. 25C shows the typical path of a 100 micron polymer microsphere through a vial after rotation. The particles move most rapidly relative to the camera due to the fact that in the image they cross the center of the container at the moment when their speed is orthogonal to the viewing direction, according to FIG. 25D. For example, if the initial rotation speed is 300 rpm, and the radial location of the particle rp is 5 mm, then the particle velocity vp is about 9.4 m / s. At this speed, a camera exposure time of as little as 10 μs doubles the apparent particle size due to image blur. FIG. 25E shows how severe image blur can affect images: on the left, particles move quickly (about 300 rpm) and appear elongated; on the right, the same particles are stationary and appear more round.

FIG. 25F is a plot of Feret diameter over time for the particle shown in FIG. 25C. Due to the lens effect of the cylindrical container, the apparent particle size near the edge of the container (point D on the right axis) is reduced. The most accurate estimate of the maximum particle size was carried out when the particle passed the center of the container at a low speed (point B on the right axis). At an excessively high speed (which usually occurs within the first few seconds after rotating the container), blurring of the image leads to an increase in the apparent size of the particle (point A on the right axis). Over time, under the action of the frictional force of the fluid, the particle stops moving completely (point C on the right axis). In this case, the mid-range peaks (point B on the right axis) represent the most accurate indication of the maximum particle size.

Particle characterization

FIG. 26A shows sequential frames of time series data containing particles and their paths. The roughly planar pathways represent the trajectories of 100 micron polymer microspheres that mimic protein aggregates. These particles, which have practically neutral buoyancy, move with the fluid and do not significantly float or sink. The vertical downward paths represent the trajectories of 100-micron glass beads that initially rotated with the fluid, but then sank as the sequence progressed. The upward paths represent the trajectories of air bubbles and particles with positive buoyancy.

Particle tracking allows the measurement of a number of time-dependent properties that can provide important clues to the nature of the particles being investigated. For example, air bubbles, generally considered to be normatively harmless, can cause errors in known optical inspection mechanisms, leading to false positives and false failures. In this case, the time-dependent movement of the particle (air bubbles usually rise in the vertical direction as the fluid decelerates) provides an extremely obvious characteristic that can be easily identified based on the trajectory obtained by tracking the particles. Likewise, particles with neutral buoyancy may not float or sink to a significant extent, while dense particles sink and fall to the bottom of the tank. Lighter particles can be picked up by the vortex generated by the rotating fluid, and heavier particles can have rectilinear trajectories.

In general, the particle tracking process provides a time-dependent spreadsheet similar to that shown in FIG. 26B containing details of all parameters of interest, including location, travel speed, direction of travel, acceleration, size (i.e., two-dimensional area), size (maximum Feret diameter), elongation, sphericity, contrast, and brightness. These parameters provide a set of features that can be used to classify a particle (assign it to a specific species). This approach, accomplished by a particle tracking solution, works well for most particles of interest. The ability to categorize particles separately for each particle based on a similar range of time-dependent measurements is a particular advantage of the present invention.

Video compression

The use of extremely high resolution imaging sensors facilitates the imaging of extremely small particles in a relatively large vessel. To ensure accurate trajectory construction, it is also necessary to maximize the imaging speed. A combination of these requirements results in extremely large video files such as 1 GB, 2 GB, 5 GB, 10 GB or more. In some applications it may be necessary to archive the original video in addition to the analysis data. Even for medium sample sets, large file sizes can lead to prohibitive storage costs.

To reduce the size of the data files of the (reverse) time series, video compression of the (reverse) time series data can be applied. Lossless video compression may be required to maintain the accuracy of the particle data. Research has shown that the more widely used (and more economical) lossy compression systems (e.g.

MPEG) can significantly distort and interfere with the image with the formation of a number of unwanted visual artifacts.

Although lossless compression is generally relatively inefficient compared to lossy compression, a number of steps can be taken to improve the efficiency of lossless compression. Most time series data frames show a small number of small, bright objects against a dark background. A dark background does not contain useful information. The background is not completely black, but is formed from extremely weak random noise. Replacing the specified background with a completely black background greatly simplifies the image and makes standard lossless compression methods (such as zip or Huffyuv) work much more efficiently.

This process is known from the prior art. The innovation of the present invention lies in making a concrete decision about what is actually the background in a given frame. Other compression processes set a luminance threshold and assume that all pixels in the image below the luminance value are part of the background. In a broad sense, this approach is effective, but it can lead to a small reduction in the size of the image of the stored particles, and can also lead to the complete removal of extremely dim particles with a brightness of the same order of magnitude as the upper limit of the random background noise inherent in the image.

While these prior art techniques are capable of working with (inverse) time series data, the compression used in these embodiments includes a unique step in which background analysis is provided to detect faint particles prior to using a destructive threshold determination process. This process provides the best balance between maintaining data accuracy and minimizing data storage requirements.

Determination of filling volume and meniscus

Automated examples of the implementation of the visual inspection platform allow you to accurately determine the volume of the sample filling, which is important in research applications where there is no guarantee of a constant filling volume throughout the entire period of experimental work. This process is especially useful when dealing with particularly large data files, such as files generated by high-resolution image sensors, which put pressure on data transmission and storage operations. For this reason, it may be desirable to constrain the captured image so that it only covers the volume of the fluid, since any additional information is not of interest.

The systems shown can use, for example, automated contouring or object recognition algorithms to detect container boundaries in an image, as shown in FIG. 27-29 and the description below. Because the meniscus and base of the bottle are unique objects, a number of possible lighting configurations and / or image processing techniques can be used to accurately identify their position in the image. The region of interest is determined by measuring the fill volume and determining the region of the image occupied by the fluid. In particular, referring to FIG. 8, configurations in which light sources 122f (backlight), 122e (bottom light), and a combination of sources 122a and 122b (backlight) are used can be used to determine the fill volume as described below.

FIG. 27A-27F show automatic detection of a region of interest within a container using illumination 122a and 122b from the rear corner of FIG. 8. In FIG. 27A shows a static view of a container in which the base of the container and the meniscus are clearly visible as crisp, bright objects. As an example, the processor may use contouring to identify the vertical walls of the container and the width w of the region of interest, as shown in FIG. 27B. To detect the meniscus and base of the vial, which may be less predictable, the process can use luma threshold and segmentation to provide a simplified image of the region of interest (shown in FIG. 27C). At this point, the processor can automatically identify containers that may not be suitable for particle analysis, such as containers whose surfaces are scratched and / or covered with dirt. System performance can be compromised by excessive turbidity, vessel surface defects, or excessive particle concentration (as a result of which individual particles can no longer be highlighted in the image). If the processor determines that the container is suitable, the objects corresponding to the meniscus and base of the vial can then be highlighted and simplified as shown in FIG. 27D. The processor defines the vertical height h of the region of interest as the distance between the lower edge of the meniscus and the upper edge of the base of the bottle, as shown in FIG. 27E. Finally, the processor can trim the original image stream using the width and height values of the region of interest, so that only the visible fluid region of the image is captured as shown in FIG. 27F.

FIG. 28A-28C show a similar meniscus detection process performed based on data obtained using a background illumination configuration (eg, light source 122f in FIG. 8). FIG. 28A shows a time series data frame depicting a typical capacitance, which is imaged using backlighting. The meniscus, walls, and base are easily distinguishable and can be automatically detected by the outlining according to FIG. 28B. However, defects such as large scratches can interfere with the accuracy of the meniscus detection process when using backlighting (FIG. 28B) or backlighting (for example, referring to FIG. 29C, described below). In one embodiment, an image luminance threshold determination is used to identify the meniscus and base of the vial. Due to the fact that these objects are relatively large, and also due to their shape, they scatter a relatively large amount of light towards the detector, and they can be easily identified due to the fact that they are significantly different from any other objects possibly present.

FIG. 29A-29D show the detection of a meniscus in a cylindrical vessel with an approximately planar bottom. The automatic detection of the fill volume begins by determining the brightness threshold (Fig. 29A) in order to detect the meniscus, which allows the location of interest to be identified, and is also a measure of the fill volume. Then, in FIG. 29B, angled illumination illuminates surface imperfections such as scratches (shown in the drawing), dust, fingerprints, glass imperfections, or condensation that can make it difficult to distinguish outlines. Illumination of the bottle from below (eg, by the light source 122e of FIG. 8) as shown in FIG. 29C, illuminates the meniscus in a manner that is (relatively) insensitive to surface imperfections - in this example, the meniscus is visible even though the surface is severely scratched. The illumination from below also makes it possible to differentiate between empty vials and full vials, as shown in FIG. 29D, and accurately determine the meniscus height at any fill level between the specified extremes. Bottom illumination of the vial enhances meniscus detection by eliminating errors caused by scratches and other surface defects (FIG. 27C). Arranging the light source 122e so that it illuminates the vessel at a slight angle further reduces sensitivity to surface defects. When illuminating syringes that are difficult to illuminate from below due to the lack of a transparent container base, a similar effect can be achieved by lighting at a low angle.

Testing techniques similar to the above-described meniscus detection can also be used to screen for objects that could adversely affect any subsequent attempts to identify and analyze particles suspended in the fluid. Such options may include identifying excessively cloudy liquids, severely damaged containers (including a large number of scratches or debris on the surface), and liquids in which the concentration of particles is so high that individual particles can no longer be distinguished in the image.

Processors and memory

One of ordinary skill in the art will appreciate that the processors disclosed herein may comprise any suitable device capable of processing and storing data, and including input / output devices capable of running application programs and the like. Examples of processors may be embedded in integrated circuits, field programmable gate arrays, and / or any other suitable architecture. The disclosed processors may also be associated with other computing devices, including other processor and / or server computers, through communications networks. A communication network can be part of a dial-up network, a wide area network (for example, the Internet), a worldwide group of computers, a local area network or a wide area network, as well as gateways using appropriate protocols (TCP / IP, Bluetooth, etc.) to provide mutual connection. Other architectures of electronic devices and computer networks are also possible.

FIG. 30 is a diagram of the internal structure of an exemplary processor 50. Processor 50 includes a system bus 79, the bus being a set of hardware lines used to transfer data between components of a computer or processing system. Bus 79 is essentially a common channel connecting various elements of a computer system (eg, processor, disk drive, memory, I / O ports, network ports, etc.), allowing information to be transferred between elements. Associated with the system bus 79 is an I / O device interface 82 for connecting various input and output devices (e.g., keyboards, mice, displays, printers, speakers, etc.) to the processor 50. The network interface 86 allows the computer to communicate with various others. devices connected to the network. Memory 90 provides volatile and / or nonvolatile storage of computer software instructions 92 and data 94 used to perform an example implementation of the disclosed systems and visual inspection techniques. The disk drive 5 provides (optional) non-volatile storage of computer software instructions 92 and data 94 used to perform an example visual inspection implementation. The central processor unit 84 is also connected to the system bus 79 and provides the execution of computer instructions.

In one embodiment, processor operation 92 and data 94 is a computer program product (generally designated 92) containing a computer-readable medium (e.g., removable storage media such as one or more DVD-ROMs, CD-ROMs, floppy disks , magnetic tapes, etc.), providing at least part of the program instructions

- 27 038813 for deployed visual inspection systems. The computer software product can be installed by any suitable software installation procedure known in the art. In yet another embodiment, at least a portion of the software instructions can also be downloaded via cable, communication means, and / or wireless connection. In other implementations, the example programs are a signal product 107 transmitted by a computer program implemented as a transmitted signal in a propagation medium (e.g., radio wave, infrared wave, sound wave, or electric wave transmitted over a wide area network such as the Internet, or other network or networks). The specified medium or signals provide at least part of the program instructions for the disclosed processes or programs 92.

In other implementations, the transmitted signal is an analog carrier wave or a digital signal transmitted in a propagation medium. For example, the transmitted signal can be a digitized signal transmitted over a wide area network (eg, the Internet), a long-distance network, or other network. In one implementation, the transmitted signal is a signal transmitted in a propagation medium over a period of time, such as commands for a software application sent as data packets over a network over a period of several milliseconds, seconds, minutes, or more. In yet another embodiment, the computer-readable medium of the computer program product 92 is a distribution medium that can be received and read by the processor 50, for example, by receiving the distribution medium and identifying a transmitted signal included in the distribution environment as described above for the signal product transmitted by the computer program.

In general, the term medium or temporary medium encompasses the described transient signals, transmission signals, propagation medium, storage medium, and the like.

Sensor cooling

In the above embodiments, electronic sensors are used to obtain particle images. Electronic sensors, such as CCD sensors, are susceptible to various types of random noise that interfere with the accuracy of the measured signal, especially when receiving a low signal strength. In some implementations, the sensors may be cooled to reduce noise. Cooling can be carried out using any suitable technology, including, for example, the use of thermoelectric refrigerators, heat exchangers (eg, cryo-coolers), liquid nitrogen cooling, and a combination of the above technologies.

In various embodiments, noise reduction provides an advantage in particle detection, particularly associated with the detection of protein aggregates. In standard applications, protein aggregates can be relatively large (for example, up to several hundred microns in diameter), but the physical structure of these aggregate particles is often quite loose, with low density (a significant part of the particle can be porous and filled with the environment) and low contrast. refractive index relative to the environment. Due to these physical properties, protein aggregates can scatter relatively small amounts of light compared to other particles such as glass fragments or fibers.

Much of the noise affecting modern electronic image sensors is thermal in nature. The specified noise affects the lower part of the sensor's dynamic range. For example, in some implementations, the bottom X% (eg, 10%) of the dynamic range is occupied by noise and must be removed during the image luminance threshold determination process (eg, as described above). The minimum threshold for particle detection must be greater than the specified value of ~ X% to remove low brightness data from the signal. This removal can prevent accurate detection of dull particles such as protein aggregates. When reducing noise, for example by cooling the sensor, a lower threshold can be used to provide improved detection of low brightness signals.

FIG. 31 illustrates the above-described threshold determination problem. Panel A of FIG. 31 shows a cropped segment of a typical data sequence obtained by the claimed devices and methods. According to the drawing, the images are eight-bit grayscale images, i.e. each pixel can have a luminance value that varies linearly from 0 (black) to 255 (white). The image contains two particles, one of which is relatively bright and the other extremely dim. Panel B of FIG. 31 is a luminance histogram showing background luminance data corresponding to an area in an image that does not contain particles.

The sensor exhibits a Gaussian background noise curve at the lower end of the luminance histogram, at least in part due to thermal effects. The width of the specified curve determines the threshold value for particle detection. In general, particles should be significantly brighter than background noise to avoid removing them when determining the brightness threshold.

Panel C, FIG. 31, a brightness histogram for a bright particle is shown.

- 28 038813

The particle image contains a significant number of pixels located to the right of the threshold value in the histogram, and therefore the particle is easily detected after the threshold value is determined.

On the other hand, according to panel D of FIG. 31, the fainter particle contains relatively few pixels above the threshold and is likely to be removed during the thresholding and segmentation process. However, when using cooling or other techniques to reduce the noise floor, leading to a shift of the threshold value to the left, it is possible that a dimmer particle can be detected.

Light based counting and non-destructive sizing (LENS)

In some implementations, when performing non-destructive sizing and counting of particles in the container, there are observable artifacts caused by the container itself. The liquid interface refracts light passing through the vial, causing noticeable distortion in the image or particle images used during the sizing and counting procedure. As a result, particles of a given size can appear, for example, up to 4 times larger in the image, depending on their spatial location in the vial. It should be noted that when using a cylindrical container, the particle image is usually stretched only along the lateral axis, but not along the vertical axis of the bottle. (See Fig. 5E for these effects.)

As described above, in some implementations, these distortion effects can be corrected (eg, reduced or even eliminated) through correction optics techniques. However, in some implementations, such optical correction may be incomplete or unavailable. In such cases, a direct correlation between the particle size and the corresponding image on the detector is impossible.

For example, in FIG. 32 shows a histogram of a determined image size for a population of standard size particles (polymer microspheres) (in the drawing they have a diameter of 100 μm) in a fluid obtained by a system in which the distortion caused by capacitance is not corrected (corresponds to the situation shown in Fig. 5E). Significant variability in apparent particle sizes due to the effects of capacitance distortion is clearly visible.

This variability makes it difficult to differentiate populations of particles of different sizes due to the fact that at an obvious location on the detector there can be a significant overlap of size values for each population of particles. For example, in FIG. 33 shows histograms of defined image size for two populations of standard size particles (in the figure they have diameters of 100 μm and 140 μm) in a fluid. There is clearly a significant overlap of the two histograms for two populations of particles of different sizes.

In some implementations, a processing method may be applied to obtain accurate sizing information even in the presence of the above-described distortion effect. The machining process is calibrated using data obtained using known dimensional standards. For example, in FIG. 34 shows experimentally obtained apparent size histograms for four different populations of standard size particles (polymer microspheres). Although four calibration curves are shown, any suitable number of curves may be used in various implementations. In some embodiments, at least two, at least three, at least four, at least five, or at least six curves may be used. In some implementations, the number of curves ranges from 2 to 100, or any sub-range, such as 4 to 6. In some implementations, a set of experimental calibration curves may be interpolated to generate additional curves (for example, corresponding to size values between experimentally measured values ).

In some embodiments, the calibration curves may correspond to populations of particles having actual sizes that differ from each other by any suitable amount, for example, at least 1 micron, at least 5 microns, at least 10 microns, at least 20 microns, or more, in the range from 1 micron to 1000 microns, or any sub-limits.

Once the calibration curves have been determined, an apparent size distribution curve can be obtained for a sample containing particles of unknown size (eg, from a static image or images, or using any other suitable technique). The sample curve can be obtained under identical or similar experimental conditions (eg, using the same or similar container in size and shape, identical or similar fluid properties, lighting conditions, imaging conditions, etc.). The specified sample curve is compared to the calibration curves to provide information indicative of the particle size of the sample.

For example, in some embodiments, a weighted overlay of calibration curves is compared to a sample curve. The weighting of the overlap is varied to ensure that the overlap is consistent with the sample curve, for example using any suitable approximation known in the art. Weighing the closest match onto the sample curve then provides information about the actual particle size in the sample. For example, in some implementations, the number of times each calibration curve appears in the best fit overlay corresponds to

- 29 038813 the number of particles of the specified size in the sample.

FIG. 35 shows the matching of the overlay of the calibration curves to the curve of the experimental sample. In this case, the sample is prepared in such a way that the particle diameter is known to be in the range of 75 to 125 μm. FIG. 36 shows best fit particle size counts compared to particle size counts obtained by simply dividing the apparent size random variable into intervals, without processing, based on the corresponding image. The raw data contains a significant number of bogus counts outside the actual size range of 75 to 125 µm. On the other hand, the results obtained based on the best fit of the calibration curves show a significantly reduced number of bogus counts.

It should be noted that while one possible approach to comparing sample data and calibration data has been described, other suitable methods may be used. For example, in some implementations, the sample curve can be decomposed using the calibration curves as basis functions, similar to the Fourier decomposition of the waveform using sinusoidal basis functions. In general, any suitable convolution, sweep, decomposition, or other technique can be used.

In some implementations, light-based counting and non-destructive sizing (LENS) techniques can be used in combination with the above-described particle tracking techniques. For example, LENS reception generally works best in cases where the particle shape is approximately the same as the standard particle size used to obtain the calibration data. In addition, the techniques usually work well when there are a large number of particles (eg, greater than 10, greater than 50, greater than 100, or more) providing a larger dataset for the algorithm to process.

However, in some applications, the amount of particles may be small. In some applications, research focuses on the larger particles in the sample. In addition, in some applications, the sample may contain particles having a shape other than the standard particle size. For example, fibers have an elongated shape other than the spherical shape used in many standards. Under these conditions, LENS techniques may not work effectively.

In general, any number of particles can be counted using the above techniques. In some embodiments, the upper bound on the number of particles that can be counted is determined based on the occlusion of one particle by another in the sample. In general, as the number of particles in the container increases, the likelihood that the two particles will not appear as separate particles on the 2D detector increases. The above principle is a function of the number of particles in the volume and the size of the particles. Usually large particles take up a large area on the detector (and therefore more occlusion occurs for particles per milliliter compared to smaller particles). For example, under certain conditions, in a flask of 10 ^cm3 filled with 8 ml of fluid can be estimated to about 500 particles of a diameter of 50 microns and an increased concentration leads to incomplete counting and effects due to shielding oversized particles.

However, the above particle tracking techniques can be effective in counting and sizing relatively large particles. Accordingly, in some implementations, an approach that is a hybrid of the two approaches can be used. FIG. 37 shows an example of the implementation of such a hybrid process. At block 3701, the sequence of images is captured, for example, using any of the approaches disclosed herein. At 3702, the image sequence is processed (eg, filtered, luma threshold is determined, segmented, etc.). At 3703, the particle data obtained at 3702 may be pre-screened to detect particles that are larger than a threshold. These large particles can be removed from the dataset and processed at block 3704 using tracking techniques. This process can provide high-quality, time-dependent measurements of the size of large particles. If there is a background of smaller particles (below the size threshold), these particles can be processed in step 3705 using LENS techniques. The data obtained using two different approaches can then be combined at block 3706 to generate a general report of the particles in the test vessel.

In various embodiments, the size cutoff used to determine the technique used can be any suitable cutoff or minimum value of about 1 micron or greater, such as about 1 to 400 microns across the width or diameter of the particle, or any sub-range. for example, from about 1 to about 50 microns, from about 50 to about 250 microns, or from about 75 to about 100 microns. In some embodiments, the particle data sent for processing using each of the techniques may be selected based on criteria other than size, such as information related to the shape of the particle. In general, any suitable combination of criteria can be used.

- 30 038813

3D Imaging and Particle Detection Techniques

As described above, in some embodiments, the automated visual inspection unit 100 may comprise two or more imaging devices 110 to provide a three-dimensional image of the contents of the container 10.

For example, in FIG. 38A-C show a block 100 containing three imaging devices 110. As shown, the imaging devices 110 are arranged in a circle around the container 10 at 120 ° intervals, but more or fewer sensors may be used in various implementations. The angles between adjacent imaging sensors are not necessarily equal to each other, but in some implementations, the equal angle configuration simplifies the imaging techniques described below.

In some implementations, all imaging devices 110 are substantially identical. The imaging devices 110 can be adjusted so that they are all located at the same physical height with respect to the container 10, and the container 10 is located in the center of the viewing area of each of the imaging devices.

In some implementations, small placement errors may occur even if the specified physical relationship is carefully optimized. To solve this problem, the imaging devices 110 can be calibrated by imaging a known calibration tool. Any sufficiently small lateral or vertical deviations from fit can then be detected by re-sampling and shifting the resulting images accordingly. In some implementations, images may be processed to correct for sensitivity variability or other performance differences between different sensors used in imaging devices 110.

FIG. 38C shows one fade for block 100. As described in detail above, by using a telecentric imaging configuration, it can be ensured that only beams substantially parallel to the imaging axis reach the sensor surface of the imaging apparatus 110. As shown in FIG. 39, using ray optics geometric techniques (or other suitable techniques), a model of rays within the container 10 can be generated, passing through the container wall and reaching the sensor surface.

Given known beam vectors, a point or region can be taken from the 2D image and the specified brightness can be sent back to the vessel 10. By examining the horizontal rows of the 2D image one at a time, the 2D horizontal grating within the vessel volume can be mapped. Horizontal lattices associated with each of the three imaging devices 110 may be superimposed on each other to form a common map. By repeating this process for additional horizontal rows of sensors, a vertical stack of two-dimensional lattices can be built to form a three-dimensional (3D) structure, for example, corresponding to the entire volume of the container 10 or part of it.

Potential particles can be identified in the resulting three-dimensional structure by defining a brightness threshold similar to that disclosed in the above description. The threshold determination may be performed on the original 2D images obtained from the imaging devices 110, or may be performed on horizontal maps within the 3D structure after being superimposed.

Probable particles can be identified using a three-dimensional structure with a defined brightness threshold, resulting in a direct measurement of the three-dimensional position of the particle in the fluid volume of the container 10. In standard applications, the three-dimensional position measurement is accurate for most of the fluid volume, but in some cases where imaging devices 110 include telecentric lenses, blind spots caused by vessel curvature and an accompanying lensing effect may occur (eg, as shown in FIG. 39, right panel).

Using three image taps set at 120 ° as shown in the figure, the blind spots closely correlate in pairs (see FIG. 39, right panel). Accurate determination of the three-dimensional position in the three areas of blind spots 3901 can be excluded. However, in these areas, the position data can be established by examining the 2D data obtained from the nearest image point.

In various embodiments, the blind spot problem can be reduced or eliminated by increasing the number of sensor taps to provide overlapping imaging.

While one example of the use of multiple imaging devices 110 has been described to obtain three-dimensional information about the contents of the container 10, it should be understood that other techniques could be used. For example, in embodiments that use two imaging devices, stereoscopic imaging techniques may be used to determine three-dimensional information.

- 31 038813

In some implementations, for example, where the test sample is static or moving slowly, three-dimensional information can be obtained using a rotary image pickup, similar to medical tomography computers. The rotary image capture acquires a time series of 2D images taken from different points that can be used to construct 3D information, for example, by any suitable method, such as those known in the medical imaging art. In case the images are acquired at a high speed relative to the dynamics of the sample, the 3D image can provide accurate 3D information for particle detection.

In some embodiments, the three-dimensional information obtained by the techniques described above may be suitable for detecting the position of a candidate particle, but are not ideal for determining other characteristics of the particle, such as particle size or shape. Accordingly, in some implementations, a hybrid approach can be used. For example, in some embodiments, the 3D location of a particle is established based on 3D information (eg, a 3D structure constructed as described above). Once the three-dimensional position of the particles has been established, dimensional and shape measurements from some or all of the imaging devices 110 can be associated with the location.

In some embodiments, particle tracking may be performed based on 3D position data, for example, using 3D tracking techniques similar to the above-described 2D tracking techniques.

In some embodiments, 3D tracking is beneficial, especially when used with 2D images obtained from each of the imaging devices 110.

By using 3D tracking, occlusion of one particle by another (eg, as shown in FIG. 5E) is reduced or eliminated. In some embodiments, obstructions are possible, for example, when examining dense samples in blind spots, in which true three-dimensional positioning is ineffective.

Similar to 2D tracking as described above, in some embodiments, predictive tracking reception can be used in the context of a 3D image using information related to fluid dynamics in vessel 10.

In some embodiments, upon completion of tracking the positions of the three-dimensional particles, information about particle characteristics (e.g., size and shape) may be grouped from the two-dimensional data obtained from multiple imaging devices 110 into multiple time-dependent datasets for each particle. In some embodiments, the technique can provide a more accurate measurement of characteristics (eg, size or shape) of individual particles compared to measurement with a single imaging sensor. For example, in some embodiments, this approach allows for clearer detection and sizing of elongated particles due to the fact that the appearance of a particle in an image is no longer solely dependent on its spatial orientation relative to a single imaging device 110.

In some embodiments, this approach can be used to mitigate the lens effect caused by the curvature of the capacitance 10. By using the three-dimensional position of the particle, the measured particle size in the two-dimensional images obtained by each of the imaging devices 110 can be adjusted to compensate for the lens effect, for example , by changing the side (horizontal) component of the dimension measurement using a lens effect scaling factor. The specified conversion factor may be determined based on the optical model of the movement of light through the container 10 for each of the imaging devices 110 according to the above description.

Spectral detection

FIG. 45 shows a sensor 4500 (shown as a diffraction spectrometer) that may be used with a visual inspection unit 100 of the type disclosed herein. For example, sensor 4500 may form a fourth image finger used with the exemplary embodiment of block 100 of FIG. 38A.

The sensor 4500 can be used to detect a characteristic (eg, spectral characteristic) of one or more particles in the vessel 10. For example, as shown, the vessel 10 is illuminated by broadband light sources 122. Sensor 4500 receives light from container 10 through distortion-correcting optics 4501 (eg, of any kind described above) and telecentric lens 4501. Light from lens 4501 is directed to a diffraction grating 4503 that separates light spectrum components, which are then imaged by imaging sensor 4504. In some embodiments, the diffraction grating 4503 operates such that the location of the incident light along one measurement of the sensor 4504 (eg, a vertical measurement) corresponds to the wavelength of the light. Another measurement at imaging sensor 4504

- 32 038813 corresponds to different spatial points within the vessel 10. That is, the sensor 4500 provides spectral data for a section of the vessel, for example, in the selected configuration, the section is shown as a horizontal piece of the vessel 10.

As the particles move through the specified central horizontal plane, their stereoscopic profile can be recorded. At the same time, according to the detailed description above, the standard taps of the block 100 can be used to track the position of the particle within the container (for example, in three dimensions). This data can be used to determine the moment of penetration of a single particle into the detection area recorded by the sensor 4500. When a particle enters the area, the sensor 4500 records the characteristic (eg, spectral profile) of the particle. Block 100 can generate data associated with the specified characteristic, and can associate the specified data with data indicative of the nature of the particle in the tracking data.

In various implementations, the characteristic data can be used for any suitable purpose, for example, to determine the type of particle. For example, spectral data for a single particle can be combined with data about size, shape, particle movement, or other data about a particle to determine the type of particle.

In some embodiments, the sensor 4500 and illuminating light sources 122 may be modified to detect particle fluorescence or any other suitable characteristic. In general, any spectral characteristic of the particles can be detected, including color, absorption spectrum, emission spectrum, or transmission spectrum, or a combination of any of these characteristics.

Although in the above example, the sensor 4500 is included in the block 100 containing three image taps, in other implementations, any number of image taps may be used, for example, one, two, four, five or more taps. In some implementations that use a single image arm, the sensor 4500 can be tailored to the image arm, for example, by using a beam splitter (not shown) to split the light beam received from the receptacle 10 and transmit the beam components to a single image arm, and to the sensor 4500. In other implementations, for example, in which multiple image taps are used, the sensor 4500 may be oriented at any suitable angle with respect to the imaging devices.

On-site measurement of sample properties

In some embodiments, inspection unit 100 may include one or more detectors (not shown) that may be used to measure the refractive index of a fluid in vessel 10. For example, in some embodiments, a narrow off-axis collimated laser beam, detected to measure the displacement of the beam due to refraction in vessel 10. If the material and geometry of vessel 10 are known, this information can be used to determine the refractive index of the fluid. In various embodiments, any other suitable technique for determining the refractive index may be used.

In some embodiments, the measured refractive index of the fluid can be used as an input to any of the data processing schemes disclosed herein (eg, data processing to compensate for lens effects caused by the curvature of capacitance 10).

In some embodiments, the inspection unit 100 may also comprise one or more detectors (not shown) that can be used to determine information indicative of the shape of the container 10. For example, in some embodiments, through an air-filled (e.g., top) portion of the container 10 a narrow, non-axial collimated laser beam can be directed to be detected to measure the displacement of the beam relative to the reference. The specified deviation can be used to accurately measure the wall thickness of the vessel (for example, with an accuracy of 0.25 mm or less). In various embodiments, any other suitable technique for determining the shape of the container may be used.

In some implementations, the detected geometric data can be used, for example, as described above, to determine the refractive index of the fluid in the vessel 10. In some implementations, the detected geometric data can be used as an input to various data processing techniques disclosed herein. description (for example, data processing to compensate for lens effects caused by the curvature of the capacitance 10), or for any other suitable purpose.

Immersion imaging

As described herein, in various embodiments, the refractive properties of the fluid in the container 10 may cause undesirable image distortion effects. In some embodiments, these effects can be mitigated by filling all (or part) of the space between the container 10 and the imaging device 110 used to imaging the container with a medium with a refractive index closer to that of the fluid than to the refractive index of air.

In some embodiments, refractive distortion can be further reduced by ensuring that the refractive index of the container 10 matches the refractive index of the fluid located within the container.

In some implementations, these immersion imaging techniques can reduce or eliminate the need for correction optics and / or processing used to reduce distortion (eg, lensing as detailed above).

Sample temperature control

In some embodiments, the inspection unit 100 may include one or more devices (not shown) to control the temperature of the sample within the vessel 10. For example, in some embodiments, the temperature control device may be used to vary the temperature of the vessel from 0 ° C to 40 ° C. C, from 0 ° C to 100 ° C, or in other suitable ranges. In some embodiments, the temperature control device can maintain the temperature at a predetermined value, for example, a value varying by less than 5 ° C, 2.5 ° C, 1 ° C, 0.1 ° C, 0.01 ° C, or less.

Temperature control can be particularly advantageous in applications where temperature control is important to prevent sample degradation during the detection process. In some embodiments, temperature- and time-dependent stability studies of temperature-sensitive products can be conducted by controlling the temperature of the sample. For example, the platform can be used to measure the dissolution (or, in some cases, formation) of protein aggregates as the temperature of the drug rises in a controlled manner from, for example, 4 ° C (refrigeration temperature) to 20 ° C (room temperature) or 37 ° C (human body temperature).

In various embodiments, temperature control may be provided in any suitable manner. In some embodiments, the environment inside the inspection unit may be hermetically sealed and thermally insulated, and temperature control may be accomplished by, for example, an air conditioning unit. In some embodiments, a heating coil and a thermoelectric cooler (eg, based on the Peltier effect) may be incorporated into the sample holder for container 10. In embodiments in which multiple containers are disposed on the tray, temperature control of the tray may be through the tray by means of a heating / cooling working fluid (eg, by passing the working fluid through a heat exchanger). In general, one or more temperature sensors and / or thermostats can be used to provide a closed loop temperature control system.

Multiple examination techniques

In some embodiments, the inspection unit 100 may re-inspect a single sample with one or more operating parameters (eg, rotational speed) that may be selected based on the results of the first inspection. This process can be repeated many times in order to better adjust the operating parameter for a particular test sample.

For example, in some implementations, the inspection may be repeated (eg, with a changed rotational speed) if the received particle detection and particle size measurements are outside the range of expected results (indicating an error in the original review).

Background reference mapping for automatic calibration

As described in detail above, in various implementations, it is desirable to characterize distortion effects (eg, lens effects) caused by refraction of light passing through the container 10 to the imaging device 110. In some embodiments, the inspection unit 100 itself can be used to map the distortions caused by the capacitance 10. The resulting map can then be used to compensate for these effects during image processing.

For example, in some implementations, one or more calibration indicators (eg, a grating) may be placed behind the container 10 as a background for the imaging apparatus 110. Refractive distortion can be detected and mapped by detecting specified pointers in the resulting image (for example, using edge enhancement or other suitable object detection techniques) and comparing their appearance in the image with their known actual appearance.

In some implementations, this approach can be used to correct for distortion caused by non-cylindrical containers, for example, containers with axial rotational symmetry, but with different circumferential lengths around the axis (such as containers that resemble the shape of modern soft drink bottles).

- 34 038813

Automatic vibration detection and suppression

As described above, in some embodiments, fluctuations can degrade the performance of the inspection unit 100. As a result of vibration, normally static objects (such as cosmetic defects on the surface of a container) vibrate during imaging. This can reduce the quality of work at the stage of removing static objects due to the formation of small but significant vibrating halos that are not removed during removal of static elements and can lead to incorrect results when implementing subsequent particle detection algorithms. In various embodiments, one or more of the techniques described below may be used to reduce the effect of oscillation.

In some embodiments, vibrating halos formed around distant static elements can be suppressed by increasing the size of the image area corresponding to the detected static features (for example, by one to several pixels thick), resulting in areas of the image containing thin vibrating halos. also removed prior to starting the particle analysis step. However, in some implementations, the disadvantage of this approach is that it reduces the substantially available sensor area.

In some embodiments, a screening algorithm may be used to detect the presence of vibrating elements. For example, elements can be detected by image processing to detect vibrating elements that do not move across the image. In some embodiments, these elements can also be identified based on their spatial proximity to the detected static elements.

In some embodiments, vessel vibration indications may be detected based on captured images, for example, by using contouring to detect movement of vessel walls, whereby the system can automatically detect and possibly alert the user to unacceptably high levels of vibration in the environment.

In some embodiments, capacitance vibration indications may be detected by physical sensors. For example, in some implementations, the head of a tool holding and manipulating a container during inspection may include motion detection devices (e.g., high-precision accelerometers) that provide feedback so the system can automatically alert users of the presence of vibration levels above a predetermined threshold. ...

Examples of

The following are exemplary performance characteristics of an example implementation of the visual inspection unit 100 of the type disclosed herein.

As shown in FIG. 40, block 100 was used with containers 10, each containing one polymer sphere of known size. Multiple detection processes (n = 80) were performed for each container, and the percentage of detection was measured (indicated by the APT columns in the drawing). According to the drawing, the percentage of detection in the system was over 90% for particles with a diameter of 15 to 200 microns. Detection percentages for an identical task by a trained person are presented for comparison (denoted by human columns in the figure). It should be noted that a person's ability to detect particles drops dramatically when particles smaller than 200 microns are detected.

As shown in FIG. 41, in another experiment, block 100 was used with containers containing particles with diameters above and below the 125 µm limit of the visible spectrum. Block 100 identified the particle and also classified the particles based on size, distinguishing between particles above and below the 125 µm limit of the visible spectrum. According to the drawing, the percentage of detection in the system was over 90% for particles with a diameter of 15 to 200 microns. Block 100 also accurately categorized the detected particles with an extremely high degree of accuracy.

As shown in FIG. 42, a number of solutions were provided for several standard particle sizes, each row consisted of containers containing particles at a predetermined concentration. The resulting capacities were analyzed by block 100 to obtain particle count data and ^{regression was used to determine the R 2} values of linearity versus inverse dilution factor. According to the drawing, the R ² value was over 0.95 for particle sizes from 15 to 200 µm, indicating excellent linearity.

As shown in FIG. 43, an excited sample containing protein particles was analyzed by block 100 to determine the number of particles grouped by particle size. The accuracy of counting the number of particles for each group is shown for 10 cycles. The particle size of the protein is unknown, making an absolute comparison of sizing accuracy impossible, but nevertheless, according to the drawing, the system is highly accurate in counting and sizing proteins. The normalized measurement error was 3%, which indicates excellent accuracy.

As shown in FIG. 44, block 100 was also tested to detect empty vials containing protein particles. The performance of unit 100 was compared to that of a qualified visual inspection operator when viewing the same set of vials. Block 100

- 35 038813 (indicated by the APT columns in the drawing) correctly identified all 40 vials of protein particles and 80 empty vials for three cycles. Self-coincidence in the classification of visible and invisible particles was 100%. The human result was only about 85% in both categories.

Conclusion

One skilled in the art will appreciate that the processes included in the automated system and method for non-destructive particle detection and identification (processing time series data obtained by visual inspection) can be implemented as an article containing a computer-readable medium. For example, said computer-readable medium may comprise a readable storage device such as a hard disk, CD-ROM, DVD-ROM, computer floppy disk, or solid state memory (ROM, RAM) components that store computer-readable program code segments. The computer-readable medium may also comprise communication or transmission means, such as a bus or communication channel, which is optical, wired or wireless, carrying segments of program code in the form of digital or analog data transmission signals.

In the present description, a block diagram is used. The use of a flowchart should not be construed as limiting the order of operations performed. The material disclosed herein contains illustrations of other components located within or connected to other distinct components. It should be understood that such an illustrated architecture is intended solely as an example, and as such, many other architectures may be implemented providing identical functionality. In a fundamental sense, any configuration of components that provide identical functionality is essentially concerned with providing the desired functionality. Therefore, any two components combined to provide a specific function can be viewed as related to each other to provide the desired functionality, regardless of architecture or intermediate components. Likewise, any two components in a similar manner can be considered to be in operable connection or in operative communication with each other to provide the desired function, and any two components configured to such communication can be considered to be operable. communication with each other to ensure the desired function. Specific examples of operable components include, but are not limited to, physically connectable and / or physically interacting components and / or wirelessly and / or wirelessly communicating components and / or logically interacting and / or configured with the possibility of logical interaction of the components.

With regard to the use of essentially any terms in the plural and / or singular, one skilled in the art is free to translate the plural to the singular and / or the singular to the plural depending on the context and / or field of application. Various variants of the singular and / or plural may be indicated in the present description for purposes of clarity.

The specialist should understand that, in general, the terms used in the present description, especially in the following claims (for example, in the text of the claims) are essentially open terms (for example, the term containing should be understood as containing without limitation, the term having should be understood as having at least the term includes, it should be understood as including without limitation, etc. It will also be obvious to a person skilled in the art that if you intend to indicate a certain number in the text of a claim, the indicated intention is included in the text of the claim in plain text, and in the absence of For example, for ease of understanding, the following claims may contain at least one and one or more introductory phrases as an introduction to the text of the claim.

If a certain number is indicated in the text of a claim, it will be obvious to a person skilled in the art that the said text must be understood as covering at least the indicated number (for example, a simple indication of two elements without additional indications usually means at least two elements, or two or more elements ).

In addition, in cases of using a phrase similar to at least one of A, B and C, etc., such a construction generally relies on the fact that a specialist will understand the meaning of the phrase (for example, a system containing at least one from A, B and C without limitation includes systems containing only A, only B, only C, A and B, A and C, B and C and / or A, B and C, etc.) In cases where the phrase is used analogous to at least one of A, B or C, etc., such a construction generally relies on the fact that the specialist will understand the meaning of the phrase (for example, a system containing at least one of A, B or C without limitation includes systems containing only A, only B, only C, A and B, A and C, B and C and / or A, B and C, etc.)

It will also be apparent to those skilled in the art that virtually any dividing union and / or phrase providing two or more alternative terms in the description, claims, and drawings is subject to the likelihood of including one of the terms, either of the terms, or both. For example, phrase A or B takes into account the likelihood of A or B or A and B.

- 36 038813

In this application, the term optical element can refer to one or more refractive, reflective, diffractive, holographic, polarizing or filtering elements in any suitable combination. In this application, terms such as light, optical or other related terms not only refer to light perceptible to humans, but can also include, for example, light from the ultraviolet, visible and infrared regions of the electromagnetic spectrum.

The present description of illustrative examples of implementation is made for purposes of illustration and description. The description should not be construed as comprehensive or limited solely by the disclosed form, and modifications and variations of the disclosed materials are possible, which may also be obvious when the disclosed embodiments are implemented. The scope of the invention is defined by the accompanying claims and the equivalents of the claims.

Claims (14)

CLAIM 1. A method for non-destructive detection of an undissolved particle in a vessel at least partially filled with a fluid medium, comprising: (a) using at least one imaging device to form an image of the particle; (b) image processing to determine location data indicative of the location of the particle in the vessel; (c) detecting a particle based at least in part on the location data, wherein detecting the particle based at least in part on the location data comprises determining the presence of a particle at a portion of the vessel; (d) using a sensor to determine the characteristics of the particle when the particle is located in the specified area of the vessel; (e) generating particle characteristic data indicative of a specific characteristic; (f) correlation of particle characteristic data with particle identification data. 2. The method according to claim 1, wherein: at least one imaging apparatus comprises at least two imaging apparatus arranged to form images of a particle from different angles in order to obtain each of them, respectively, one or more two-dimensional images of a particle in a fluid medium; image processing for determining location data includes combining two-dimensional images obtained from at least two imaging devices to determine three-dimensional data indicative of the location of the particle in the vessel; and detecting a particle includes detecting a particle based at least in part on three-dimensional data. 3. The method of claim 2, wherein the particle characteristic comprises a spectral characteristic. 4. The method of claim 3, wherein the sensor comprises a spectrometer disposed to detect one or more spectral characteristics of light incident from a portion of the vessel. 5. The method of claim 4, wherein the portion of the vessel comprises a substantially planar layer of fluid in the vessel. 6. The method of claim 5, wherein the sensor comprises a telecentric imaging device positioned to image a portion of the vessel onto a spectrometer. 7. The method of claim 4, wherein the one or more spectral characteristics include color, absorption spectrum, or transmission spectrum, or a combination of these characteristics. 8. A device for non-destructive detection of undissolved particles in a vessel at least partially filled with a fluid medium, comprising: (a) at least one imaging device located so as to form an image of the particle; (b) at least one sensor configured to detect a characteristic of a particle when the particle is located at a portion of the vessel; (c) at least one processor in operative communication with each of the at least one imaging apparatus and sensor, configured: processing an image from the at least one imaging device to determine location data indicative of the location of the particle in the vessel; detecting the particle based at least in part on the location data and identifying the presence of the particle at the site of the vessel; using the signal from the sensor to determine the characteristics of the particle when the particle is located in the specified area of the vessel; creating particle characteristic data indicative of a specific characteristic; the relationship of the particle characteristic data with the particle identification data. - 37 038813 9. The device according to claim 8, in which: at least one imaging device comprises at least two imaging devices arranged to form images of a particle from different angles in order to obtain each of them, respectively, one or more two-dimensional images of a particle in a fluid medium; the processor is configured to combine two-dimensional images obtained from at least two imaging devices to determine three-dimensional data indicative of the location of the particle in the vessel; and the processor is configured to detect the particle based at least in part on the 3D data. 10. The apparatus of claim 9, wherein the particle characteristic comprises a spectral characteristic. 11. The apparatus of claim 10, wherein the sensor comprises a spectrometer disposed to detect one or more spectral characteristics of light incident from a portion of the vessel. 12. The apparatus of claim 11, wherein the portion of the vessel comprises a substantially planar layer of fluid in the vessel. 13. The apparatus of claim 12, wherein the sensor comprises a telecentric imaging device arranged to form an image of a portion of the vessel onto a spectrometer. 14. The apparatus of claim 11, wherein the one or more spectral characteristics include color, absorption spectrum, or transmission spectrum, or a combination of these characteristics.