EP4673728A1

EP4673728A1 - Fixed-position imaging systems for automated visual inspection

Info

Publication number: EP4673728A1
Application number: EP24715952.8A
Authority: EP
Inventors: Al Patrick GOODWIN; Thomas Clark PEARSON; Osvaldo PEREZ-VARELA
Original assignee: Amgen Inc
Current assignee: Amgen Inc
Priority date: 2023-02-27
Filing date: 2024-02-26
Publication date: 2026-01-07
Also published as: MX2025009712A; JP2026510261A; IL321819A; WO2024182262A1; KR20250155521A; CN120752517A; AU2024230801A1

Abstract

An automated visual inspection (AVI) system of the present disclosure may position a vessel fixture such that a central vessel axis is optically aligned with a central image axis of a fixed-position imaging system. Once the vessel fixture is positioned such that the central vessel axis of the vessel fixture is aligned with the central image axis of the fixed-position imaging system, the fixed-position imaging system may simultaneously capture a first image of a first vessel and a second image of a second vessel. The fixed-position imaging system may capture the first image of the first vessel from the same perspective as the fixed-position imaging system simultaneously captures the second image of the second vessel. The AVI system may analyze the first image and the second image to inspect the first subset of vessels and the second subset of vessels for a same set of one or more features

Description

FIXED-POSITION IMAGING SYSTEMS FOR AUTOMATED VISUAL INSPECTION

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] Priority is claimed to United States Provisional Patent Application No. 63/448,552, filed February 27, 2023, the entire contents of which are hereby incorporated by reference herein.

FIELD OF DISCLOSURE

[0002] The present application relates generally to the inspection of vessels, and more specifically to imaging systems for automated visual inspection.

BACKGROUND

[0003] In certain contexts, such as quality control procedures for manufactured drug products, it is necessary to examine vessels (e.g., containers, vials, syringes, cartridges, etc., and/or their contents) for the presence of various defects (e.g., cracks, defective seals, air gap measurements, plunger depth measurements, low fill, high fill, foreign particles, fibers, etc.). The acceptability of a given vessel or sample, under the applicable quality standards, may depend on metrics such as a condition of the vessel, the presence of undesired particles within the vessel, etc. If a vessel has unacceptable metrics, the vessel and contents may be rejected and discarded.

[0004] To handle the quantities of vessels typically associated with commercial production of pharmaceuticals, product inspection tasks have increasingly become automated. Known automated visual inspection (AVI) systems (e.g., AVI system 100 of FIG. 1) have struggled to overcome various barriers to achieving good product fidelity void of system complexities. The AVI system 100 receives a plurality of vessels 102 via a conveyor 101. A subset of vessels 102 (e.g., four vessels) are transferred to a vessel fixture 104 of a turntable 103. The turntable 103 aligns a central vessel fixture axis 109 of the vessel fixture 104 with an imaging system central axis 108 of the imaging system with mechanical movement 105.

[0005] In addition to mechanical movement to align the central vessel fixture axis 108 with the imaging system central axis 108, AVI system 100 requires that a complex camera with mechanical movement mechanisms 105 sequentially align a central image axis 108b, of the camera with mechanical movement mechanisms 105, with a central container axis 109b to sequentially capture an image 102b of each vessel 102a. Accordingly, the camera with mechanical movement mechanisms 105, along with complex optics 106 and connection cables 107, are subject to associated physical stress. AVI system 100 often requires high maintenance due to millions of associated mechanical movement cycles of the camera with mechanical movement mechanisms 105.

[0006] Imaging systems are need that reduce mechanical movement of associated cameras, optics, and associated camera connections.

SUMMARY

[0007] Embodiments described herein relate to fixed-position imaging systems and automated visual inspection (AVI) systems that incorporate fixed-position imaging.

[0008] As described herein, a system for capturing an image of a plurality of vessels includes a first fixed-position imaging system configured to capture first images, a second fixed-position imaging system configured to capture second images, and a conveying mechanism configured to transport vessels past the first and second fixed-position imaging systems. The first fixed- position imaging system, the second fixed-position imaging system, and the conveying mechanism are configured such that the first fixed-position imaging system captures a first image of a first subset of the vessels and the second fixed-position imaging system simultaneously captures a second image of a second subset of the vessels from a same perspective view relative to the vessels.

[0009] A method for imaging a plurality of vessels includes conveying, using a conveying mechanism, vessels in front of a first fixed-position imaging system and a second fixed-position imaging system. The method also includes capturing, using the first fixed-position imaging system, a first image of a first subset of the vessels. The method further includes capturing, using the second fixed-position imaging system, a second image of a second subset of the vessels, wherein capturing the first image and capturing the second image occur simultaneously and from a same perspective view relative to the vessels.

[0010] A non-transitory computer-readable medium having computer-readable instructions stored thereon that, when executed by one or more processors, causes the one or more processors to control a conveying mechanism to convey vessels in front of a first fixed-position imaging system and a second fixed-position imaging system. Further execution of the computer-readable instructions by the one or more processors, causes the one or more processors to simultaneously (I) control the first fixed- position imaging system to capture a first image of a first subset of the vessels, and (II) control the second fixed-position imaging system to capture a second image of a second subset of the vessels, from a same perspective view relative to the vessels.

[0011] Novel fixed-position imaging systems and automated visual inspection (AVI) systems are provided that incorporate fixed-position imaging systems. Novel methods for operating the AVI systems are also provided.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The skilled artisan will understand that the figures described herein are included for purposes of illustration and do not limit the present disclosure. The drawings are not necessarily to scale, and emphasis is instead placed upon illustrating the principles of the present disclosure. It is to be understood that, in some instances, various aspects of the described implementations may be shown exaggerated or enlarged to facilitate an understanding of the described implementations. In the drawings, like reference characters throughout the various drawings generally refer to functionally similar and/or structurally similar components.

[0013] FIG. 1 depicts a known automated visual inspection (AVI) system with a mechanically positioned imaging system.

[0014] FIG. 2 depicts a top plan view of an example automated visual inspection (AVI) system with a fixed-position imaging system.

[0015] FIG. 3A depicts a top plan view of an example automated visual inspection (AVI) system with a fixed-position imaging system.

[0016] FIG. 3B depicts a top side perspective view of an example automated visual inspection (AVI) system of FIG. 3A.

[0017] FIG. 3C depicts a top rear perspective view of an example automated visual inspection (AVI) system of FIG. 3A.

[0018] FIG. 4 depicts a top plan view of an example automated visual inspection (AVI) system with a fixed-position imaging system.

[0019] FIG. 5 depicts a high level block diagram of an example automated visual inspection (AVI) system with a fixed-position imaging system.

[0020] FIG. 6 depicts an example method of implementing an automated visual inspection (AVI) system with a fixed-position imaging system.

[0021] Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions and/or relative positioning of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of various embodiments of the present invention. Also, common but well-understood elements that are useful or necessary in a commercial feasible embodiment are often not depicted in order to facilitate a less obstructed view of these various embodiments. It will further be appreciated that certain actions and/or steps may be described or depicted in a particular order of occurrence while those skilled in the art will understand that such specificity with respect to sequence is not actually required. It will further be appreciated that certain actions and/or steps may be described or depicted in a particular order of occurrence while those skilled in the art will understand that such specificity with respect to sequence is not actually required. It will also be understood that the terms and expressions used herein have the ordinary technical meaning as is accorded to such terms and expressions by persons skilled in the technical field as set forth above except where different specific meanings have otherwise been set forth herein.

DETAILED DESCRIPTION

[0022] The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways, and the described concepts are not limited to any particular manner of implementation. Examples of implementations are provided for illustrative purposes.

[0023] The fixed-position imaging systems of the present disclosure may reduce, or eliminate entirely, complex mechanical movement of associated cameras, optics, camera connections, etc. Rather than sequentially capturing an image 102b of each vessel 102a as in known AVI system 100, the systems described herein may, for example, simultaneously control a first fixed- position imaging system to capture a first image of a first subset of vessels and control a second fixed-position imaging system to capture a second image of a second subset of vessels from a same perspective view relative to the vessels. The fixed-position imaging systems of the present disclosure may enable vessel inspection in milliseconds where known systems 100 include complex mechanical movement to perform the same activity at a much slower speed. Fixed-position imaging systems may eliminate a high maintenance requirement and downtime due to various failure modes that is due to millions of complex mechanical imager movement cycles in known systems 100.

[0024] As used in the context of the present disclosure, “simultaneously” shall mean “substantially at the same time.” For, example, a fixed-position imaging system may capture a first image and a second image within a predetermined time (e.g., 50 milliseconds, 100 milliseconds, etc.). In any case, the fixed-position imaging system will capture the first image and the second image without physically moving the fixed-position imaging system or the vessel fixture in between capturing the first image and the second image.

[0025] FIG. 2 is a top plan view of an automated visual inspection (AVI) system 200 with a fixed-position imaging system 205. The AVI system 200 includes a conveying mechanism 203 configured to position a vessel fixture 204 such that a central vessel fixture axis 209 is aligned with a central image axis 208 of the fixed-position imaging system 205. Once the vessel fixture 204 is positioned such that the central vessel fixture axis 209 is aligned with the central image axis 208, the fixed-position imaging system 205 may simultaneously capture a first image 202b 1 of a first subset of vessels 202a1 and a second image 202b2 of a second set of vessel 202a2. The fixed-position imaging system 205 may capture the first image 202b 1 from a same perspective with respect to a first central image axis 208a as the fixed-position imaging system 205 simultaneously captures the second image 202b2 with respect to a second central image axis 208b. In any event, the fixed-position imaging system 205 simultaneously captures the first image 202b 1 and the second image 202b2 with no mechanical movement in between.

[0026] As illustrated in FIG. 2, the first subset of vessels 202a1 and the second subset of vessels 202a2 may each include two vessels 202. While both the first subset of vessels 202a1 and the second subset of vessels 202a1 are shown in FIG. 2 to include two vessels 202, either the first subset of vessels 202a1 or the second subset of vessels 202a1 may include one or more vessels 202. Thereby, the AVI system 200 may simultaneously capture images 202b1, 202b2 in less time than is required by AVI system 100 to mechanically align imaging system 105 and capture a single image 102b of a single vessel 102a.

[0027] The AVI system 200 may further include an illumination source 210. The illumination source 210 may be configured as a fixed-position backlight. In any event, the AVI system 200 may energize the illumination source 210 prior to capturing the first and second images 202b1, 202b2. While the first and second images 202b1, 202b2 include only a portion of the respective vessels 202 (/. e. , a syringe flange, a part of a syringe barrel, a plunger, an air gap, and a part of a product within the syringe), the first and second images 202b1, 202b2 may include a profile view of the entirety of the respective containers 202a1, 202a2. [0028] The AVI system 200 may be used to inspect vessels 202 in, for example, a final packing area of a related manufacturing facility. The AVI system 200 may be used to inspect vessels 202 in inspection areas within a manufacturing facility in addition to, or in lieu of a final packing area. AVI system 200 may be configured to, for example, inspect a given number of vessels 202 in less than twenty milliseconds compared to AVI system 100 requiring three to four seconds (/'.e., AVI system 200 may be one hundred times faster than AVI system 100). In any event, AVI system 200 may eliminate millions of mechanical cycles of AVI system 100 that require mechanical movement of sensitive optics, and camera cables bending millions of times. While the AVI system 200 is illustrated in FIG. 2 with respect to a prefilled syringe, the AVI system 200 may apply to inspection in the assembly, labeling, and packaging phase of an associated manufacturing process. Similarly, the AVI system 200 may assess attributes on a combination product (e.g. a handheld autoinjector, an autoinjector, etc.) or associated packaging (e.g., label presence, label position, etc.).

[0029] The AVI system 300a-c of FIGs. 3A-3C, may be similar to the AVI system 200. The AVI system 300a-c may include a conveying mechanism 303 configured to position a vessel fixture 304 with a central vessel fixture axis 309 aligned with a central image axis 308 of a fixed-position imaging system 305.

[0030] The fixed-position imaging system 305 includes a first fixed-position imaging system 305a having a first telecentric lens 306a and a first camera connection 307a. The first fixed-position imaging system 305a may be, for example, oriented in a fixed- position such that a first central optical axis 308a is aligned with a first central vessel fixture axis 309a of a first subset of vessels 302a1 using a first mirror 340a. While the first mirror 340a is illustrated in FIGs. 3A-3B as having a planer reflective surface oriented at a forty-five degree angle with respect to the first central optical axis 308a, the first mirror 340a may be oriented at an angle with respect to the first central optical axis 308a such that a first subset of vessels 302a1 is included within a field of view of the first fixed-position imaging system 305a. The first subset of vessels 302a1 may include a single vessel (e.g., vessel 102a of FIG. 1), a portion of two vessels (e.g., a portion of vessels 202a1, 202a2 of FIG. 2), or a predetermined number of vessels based upon, for example, a desired vessel inspection speed and/or a predetermined image resolution. As an alternative to providing a plan view of the first subset of vessels 302a1, the first mirror 340a may be rotationally oriented with respect to the first central optical axis 308a such that a perspective view of the first subset of vessels 302a1 is included within a field of view of the first fixed-position imaging system 305a (e.g., oriented to view a syringe flange from a predetermined perspective, oriented to view a vial seal from a predetermined perspective, etc.). The first telecentric lens 306a may include any number and types of optical elements that may be configured to, for example, align a central imager axis of an image sensor of the first fixed-position imaging system 305a with the first central optical axis 308a. Thus, the central imager axis of the image sensor may be oriented in any fixed-position relative the first central optical axis 308a.

[0031] The fixed-position imaging system 305 includes a second fixed-position imaging system 305b having a second telecentric lens 306b and a second camera connection 307b. The second fixed-position imaging system 305b may be, for example, oriented in a fixed-position such that a second central optical axis 308b is aligned with a second subset of vessels 302a2 using a second mirror 340b. The second subset of vessels 302a2 may include a single vessel (e.g., vessel 102a of FIG. 1), two vessels (e.g., vessels 202a1 or vessels 202a2 of FIG. 2), or a predetermined number of vessels based upon, for example, a desired vessel inspection speed and/or a predetermined image resolution. The second subset of vessels 302a2 may include more or less vessels than the first subset of vessels 302a1. As an alternative to providing a plan view of the second subset of vessels 302a2, the second mirror 340b may be rotationally oriented with respect to the second central optical axis 308b such that a perspective view of the second subset of vessels 302a2 is included within a field of view of the second fixed-position imaging system 305b (e.g., oriented to view a syringe flange from a predetermined perspective, oriented to view a vial seal from a predetermined perspective, etc.). The second telecentric lens 306b may include any number and types of optical elements that may be configured to, for example, align a central imager axis of an image sensor of the second fixed-position imaging system 305b with the first central optical axis 308b. Thus, the central imager axis of the image sensor may be oriented in any fixed- position relative the first central optical axis 308b.

[0032] In any event, once the AVI system 300a-c aligns the central vessel fixture axis 309 with the central image axis 308, the fixed-position imaging system 305 may simultaneously control the first fixed-position imaging system 305a to capture a first image 302b 1 of a first subset of vessels 302a1 and control the second fixed-position imaging system 305b to capture a second image 302b2 of a second subset of vessels 302a2. The first fixed-position imaging system 305a may capture the first image 302b 1 relative a first central image axis 308a of the first subset of vessels 302a1 from the same perspective as the second fixed- position imaging system 305b simultaneously captures the second image 302b2 relative a second central image axis 308b of the second set of vessel 302a2. The fixed-position imaging system 305 simultaneously captures the first image 302b 1 and the second image 320b2 without mechanical movement in between as is required to sequentially capture images 102b using, for example, the AVI system 100 of the prior art.

[0033] The fixed-position imaging system 305a may further include an illumination source 310 (e.g., a fixed-position back light, a back light mounted on a turntable, a back light incorporated into a vessel fixture, etc.). The fixed-position imaging system 305 may include a fixed-position illumination source 310 attached in a fixed-position via, for example, a bracket 311.

[0034] While the AVI system 300a is illustrated in FIGs. 3A with respect to a prefilled vial, the AVI system 200 may apply to inspection in the assembly, labeling, and packaging phase of an associated manufacturing process. Similarly, the AVI system 300a may assess attributes on a combination product (e.g. a handheld autoinjector, an autoinjector, etc.) or associated packaging (e.g., label presence, label position, etc.).

[0035] The AVI system 400 may be similar to the AVI system 300a-c of FIGs. 3A-3C or the AVI system 200 of FIG. 2. The AVI system 400 may include a conveying mechanism configured to position a vessel fixture 404 with a central vessel fixture axis 409 aligned with a central image axis 408 of a fixed-position imaging system 405.

[0036] The fixed-position imaging system 405 includes a first fixed-position imaging system 405a having a first telecentric lens 406a and a first camera connection 407a. The first fixed-position imaging system 405a may be, for example, oriented in a fixed- position such that a first central optical axis 408a is aligned with a first central vessel axis 409a of a first subset of vessels 402a1 such that a first subset of vessels 402a1 is included within a field of view of the first fixed-position imaging system 405a. The first subset of vessels 402a1 may include a single vessel (e.g., vessel 102a of FIG. 1), a portion of two vessels (e.g., a portion of vessels 202a1, 202a2 of FIG. 2), a profile view of an entire vessel (e.g., a profile view of vessels 302a1, 302a2 of FIG. 3A) or a predetermined number of vessels based upon, for example, a desired vessel inspection speed and/or a predetermined image resolution. As an alternative to providing a plan view of the first subset of vessels 402a1, the first fixed-position imaging system 405a may be rotationally oriented with respect to the first central optical axis 408a such that a perspective view of the first subset of vessels 402a1 is included within a field of view of the first fixed-position imaging system 405a (e.g., oriented to view a vial seal from a predetermined perspective, oriented to view a syringe flange from a predetermined perspective, etc.). The first telecentric lens 406a may include any number and types of optical elements that may be configured to, for example, align a central imager axis of an image sensor of the first fixed-position imaging system 405a with the first central optical axis 408a. Thus, the central imager axis of the image sensor may be oriented in any fixed-position relative the first central optical axis 408a.

[0037] The fixed-position imaging system 405 includes a second fixed-position imaging system 405b having a second telecentric lens 406b and a second camera connection 407b. The second fixed-position imaging system 405b may be, for example, oriented in a fixed-position such that a second central optical axis 408b is aligned with a second subset of vessels 402a2. The second subset of vessels 402a2 may include a single vessel (e.g., vessel 102a of FIG. 1), two vessels (e.g., vessels 202a1 or vessels 202a2 of FIG. 2), a profile view of an entire vessel (e.g., a profile view of vessels 302a1, 302a2 of FIG. 3A), or a predetermined number of vessels based upon, for example, a desired vessel inspection speed and/or a predetermined image resolution. The second subset of vessels 402a2 may include more or less vessels than the first subset of vessels 402a1. As an alternative to providing a plan view of the second subset of vessels 402a2, the second fixed-position imaging system 405b may be rotationally oriented with respect to the second central optical axis 408b such that a perspective view of the second subset of vessels 402a2 is included within a field of view of the second fixed-position imaging system 405b (e.g., oriented to view a syringe flange from a predetermined perspective, oriented to view a vial seal from a predetermined perspective, etc.). The second telecentric lens 406b may include any number and types of optical elements that may be configured to, for example, align a central imager axis of an image sensor of the second fixed-position imaging system 405b with the first central optical axis 408b. Thus, the central imager axis of the image sensor may be oriented in any fixed-position relative the first central optical axis 408b. [0038] In any event, once the AVI system 400 aligns the central vessel fixture axis 409 with the central image axis 408, the fixed-position imaging system 405 may simultaneously control the first fixed-position imaging system 405a to capture a first image 402b 1 of a first subset of vessels 402a1 and control the second fixed-position imaging system 405b to capture a second image 402b2 of a second subset of vessels 402a2. The first fixed-position imaging system 405a may capture the first image 402b 1 relative a first central image axis 408a of the first subset of vessels 402a1 from the same perspective as the second fixed- position imaging system 405b simultaneously captures the second image 402b2 relative a second central image axis 408b of the second set of vessel 402a2. The fixed-position imaging system 405 simultaneously captures the first image 402b 1 and the second image 402b2 without mechanical movement in between as is required to sequentially capture images 102b using, for example, the AVI system 100 of the prior art.

[0039] The fixed-position imaging system 405 may further include an illumination source 410 (e.g., a fixed-position back light, a back light mounted on a turntable, a back light incorporated into a vessel fixture, etc.). The fixed-position imaging system 405 may include a fixed-position illumination source 410 attached in a fixed-position via, for example, a bracket 411.

[0040] While the AVI system 400 is illustrated in FIG. 4 with respect to a prefilled vial, the AVI system 400 may apply to inspection in the assembly, labeling, and packaging phase of an associated manufacturing process. Similarly, the AVI system 200 may assess attributes on a combination product (e.g. a handheld autoinjector, an autoinjector, etc.) or associated packaging (e.g., label presence, label position, etc.).

[0041] FIG. 5 is a simplified block diagram of an example AVI system 500 that may implement various techniques relating to the training (and possibly validation and/or qualification) and/or use of one or more neural networks or non-machine learning (ML) systems. The AVI system 500 could also be used to test/qualify non-ML AVI systems. In addition to, or as an alternative to, ML systems, the AVI system 500 may include “computer vision” algorithms that do not use ML, but instead use fixed rules (e.g., empty vial, low fill, high fill, etc.).

[0042] An AVI system 500 may include one or more AVI neural network(s). Once trained and qualified, the AVI system 500 may be used in production to detect defects associated with vessels and/or contents of those vessels. In a pharmaceutical context, for example, the AVI system 500 may be used to detect defects associated with syringes, cartridges, vials or other vessel types (e.g., bruised crimps/seals, cracks, scratches, stains, missing components, etc., of the vessels), and/or to detect defects associated with liquid or lyophilized drug products within the vessels (e.g., the presence of fibers, metallic particles, and/or other foreign particles, variations in color of the product, etc.). As used herein, “defect detection” may refer to the classification of vessel images as exhibiting or not exhibiting defects (or particular defect categories), and/or may refer to the detection of particular objects or features (e.g., particles or cracks) that are relevant to whether a vessel and/or its contents should be considered defective, depending on the embodiment. [0043] AVI system 500 includes a visual inspection system (VIS) 505 communicatively coupled to a computer system 520. VIS 505 includes hardware (e.g., an illumination source 510, telecentric optics 506, etc.), as well as firmware and/or software, that is configured to capture digital images of a sample (e.g., a vessel holding a fluid or lyophilized substance). VIS 505 may include any of the fixed-position imaging systems 205, 305a-c, 405 described herein respectively with reference to FIGs. 2-4, for example, or may be some other suitable VIS.

[0044] For ease of explanation, AVI system 500 is described herein as training and validating one or more AVI neural networks using vessel images from VIS 505, and then using the trained/validated neural network(s) to perform AVI/defect detection. It is understood, however, that this need not be the case. For example, the AVI system 500 may perform training and/or validation using vessel images generated by a number of different visual inspection systems instead of, or in addition to, VIS 505. Moreover, the training/validation may be performed by another system, and AVI system 500 may then use the trained neural network(s) (e.g., during commercial production). In some embodiments, some or all of the vessel images used for training and/or validation are generated using one or more offline (e.g., lab-based) “mimic stations” that closely replicate important aspects of commercial line equipment stations (e.g., optics, lighting, etc.), thereby expanding the training and/or validation library without causing excessive downtime of the commercial line equipment.

[0045] VIS 505 may image each of a number of vessels simultaneously. To this end, VIS 505 may include, or operate in conjunction with, holding means such as a conveyance mechanism, a turntable, a cartesian robot, carousel, starwheel and/or any other holding means that can successively move each vessel into an appropriate position for imaging, and then moves the vessel away once imaging of the vessel is complete. While not shown in FIG. 5, VIS 505 may include a communication interface and processors to enable communication with computer system 520. In other embodiments (e.g., lab-based setups), the VIS 505 includes simpler holding means (e.g., a stage with a hole covered by a glass plate).

[0046] Computer system 520 may generally be configured to control/automate the operation of VIS 505, and to receive and process images captured/generated by VIS 505, as discussed further below. Computer system 520 may be a general-purpose computer that is specifically programmed to perform the operations discussed herein, or may be a special-purpose computing device. As seen in FIG. 5, computer system 520 includes a user interface 521, a processing unit 522, and a memory unit 523. In some embodiments, however, computer system 520 includes two or more computers that are either co-located or remote from each other. In these distributed embodiments, the operations described herein relating to processing unit 522 and memory unit 523 may be divided among multiple processing units and/or memory units, respectively.

[0047] Processing unit 522 includes one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in memory unit 523 to execute some or all of the functions of computer system 520 as described herein. Processing unit 522 may include one or more graphics processing units (GPUs) and/or one or more central processing units (CPUs), for example. Alternatively, or in addition, some of the processors in processing unit 522 may be other types of processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), and some of the functionality of computer system 520 as described herein may instead be implemented in hardware.

[0048] Memory unit 523 may include one or more volatile and/or non-volatile memories. Any suitable memory type or types may be included in memory unit 523, such as read-only memory (ROM), random access memory (RAM), flash memory, a solid- state drive (SSD), a hard disk drive (HDD), and so on. Collectively, memory unit 523 may store one or more software applications, the data received/used by those applications, and the data output/generated by those applications.

[0049] Memory unit 523 stores the software instructions of various modules that, when executed by processing unit 522, performs various functions for the purpose of training, validating, and/or qualifying one or more AVI neural networks. Specifically, in the example embodiment of FIG. 5, memory unit 523 includes an image analysis module 525 and a visual inspection system (VIS) control module 526. In other embodiments, memory unit 523 may omit one or more of modules 525, 526 and/or include one or more additional modules. In addition, or alternatively, one, some, or all of modules 525, 526 may be implemented by a different computer system (e.g., a remote server coupled to computer system 520 via one or more wired and/or wireless communication networks). Moreover, the functionality of any one of modules 525 and 526 may be divided among different software applications and/or computer systems. As just one example, in an embodiment where computer system 520 accesses a web service to train and use one or more AVI neural networks, the software instructions of image analysis module 525 may be stored at a remote server.

[0050] Image analysis module 525 comprises software that uses images stored in an image library 530 to train one or more AVI neural networks. Image library 530 may be stored in memory unit 523, or in another local or remote memory (e.g., a memory coupled to a remote library server, etc.). In addition to training, the image analysis module 525 may implement/run the trained AVI neural network(s), e.g., by applying images newly acquired by VIS 505 (or another visual inspection system) to the neural network(s), possibly after certain pre-processing is performed on the images as discussed below. In various embodiments, the AVI neural network(s) trained and/or run by the image analysis module 525 may classify entire images (e.g., defect vs. no defect, or presence or absence of a particular type of defect such as a crimp bruise or crimp defect generally, etc.), detect objects in images (e.g., detect the position of foreign objects that are not bubbles within vessel images), or some combination thereof (e.g., one neural network classifying images, and another performing object detection). As used herein, unless the context clearly indicates a more specific use, “object detection” broadly refers to techniques that identify the particular location of an object (e.g., a particle, a fiber, etc.) within an image, and/or that identify the particular location of a feature of a larger object (e.g., a bruised crimp or seal, a crack or chip on a syringe or cartridge barrel, etc.), and can include, for example, techniques that perform segmentation of the vessel image or image portion (e.g., pixel-by-pixel classification), or techniques that identify objects and place bounding boxes (or other boundary shapes) around those objects.

[0051] In embodiments where the AVI neural network(s) detect vessel defects, the defects may relate to any suitable vessel feature(s). Referring to the example vessels of FIGs. 2-4, for instance, a particular AVI neural network implemented by the image analysis module 525 may detect whether a vessel has a crack or stain, whether a flange is misshapen, whether a needle shield is not properly positioned, whether a plunger or piston has any defects, whether a luer lock has any defects, whether a crimp is properly positioned and/or has any defects (e.g., bruising), whether a flip cap is properly positioned and/or has any defects, and so on.

[0052] The image analysis module 525 may run the trained AVI neural network(s) for purposes of validation, qualification, and/or inspection during commercial production. In one embodiment, for example, the image analysis module 525 is used only to train and validate the AVI neural network(s), and the trained neural network(s) is/are then transported to another computer system for qualification and inspection during commercial production (e.g., using another module similar to module 525). In some embodiments where image analysis module 525 trains/runs multiple neural networks, the image analysis module 525 includes separate software for each neural network.

[0053] AVI neural network training may be performed on images from, for example, six vials after augmenting the associated training images by adjusting brightness, vertical mirroring, adding noise, and skewing the images, as well as skewing the bounding boxes (/'.e., the training set may be multiplied fivefold). Generally, deep learning may be used to detect defects in the images. Use of previously trained AVI neural network(s) further reduces time required to set up an automated inspection recipe for new products. AVI neural networks of the present disclosure may be implemented for high-mix, low-volume production scenario such as clinical operations or small batches of product, then using modern deep learning techniques (e.g., image analysis module 525 of FIG. 5). [0054] In some embodiments, VIS control module 526 controls/automates operation of VIS 505 such that vessel images can be generated with little or no human interaction. VIS control module 526 may cause a given fixed-position imaging system to capture a vessel image by sending a command or other electronic signal (e.g., generating a pulse on a control line, etc.) to that imager. VIS 505 may send the captured vessel images to computer system 520, which may store the images in memory unit 523 for local processing. In alternative embodiments, VIS 505 may be locally controlled, in which case VIS control module 526 may have less functionality than is described herein (e.g., only handling the retrieval of images from VIS 505), or may be omitted entirely from memory unit 523.

[0055] FIG. 6 is a method 600 of operating an automated visual inspection (AVI) system which may be implemented by a processor (e.g., processing unit 522 of FIG. 5) executing, for example, at least a portion of the visual inspection system (VIS) control module 526 and/or the image analysis module 525. The AVI system may be similar to, for example, any one of the AVI systems 200 of FIG. 2, 300a-c of FIGs. 3A-3B, 400 of FIG. 4, or 500 of FIG. 5. In particular, processing unit 522 may execute the VIS control module 526 to cause the processing unit 522 to, for example, align a central vessel fixture axis 209, 309, 409 of a vessel fixture 204, 304, 404 with a central image axis 208, 308, 408 of a fixed-position imaging system 205, 305, 405 (block 640). The processing unit 522 may execute the VIS control module 526 to cause the processing unit 522 to, for example, energize an illumination source 210, 310, 410, 510 (block 641).

[0056] The processing unit 522 may further execute the VIS control module 526 to cause the processing unit 522 to, for example, simultaneously capture a first image 202b1, 302b1, 402b1 of a first subset of vessels 202a1, 302a1, 402a1 and a second image 202b2, 302b2, 402b2 of a second subset of vessels 202a2, 302a2, 402a2 (block 642). The processing unit 522 may execute the image analysis module 525 to cause the processing unit 522 to, for example, analyze the first image 202b1, 302b1, 402b1 and the second image 202b2, 302b2, 402b2 to inspect the first subset of vessels 202a1, 302a1, 402a1 and the second subset of vessels 202a2, 302a2, 402a2 for a same set of one or more features (block 643).

[0057] A method for imaging a plurality of vessels includes conveying, using a conveying mechanism, vessels in front of a first fixed-position imaging system and a second fixed-position imaging system. The method also includes capturing, using the first fixed-position imaging system, a first image of a first subset of the vessels. The method further includes capturing, using the second fixed-position imaging system, a second image of a second subset of the vessels, from a same perspective view relative to the vessels, and simultaneously with the first image.

[0058] The fixed-position imaging systems of the present disclosure may reduce complexity of the AVI system. The fixed- position imaging systems may also reduce lifecycle maintenance on the AVI system. The fixed-position imaging systems may further improve quality inspection by reduction of vibration and reduction of lens errors.

[0059] Camera connections and signal management present design challenges in the moving AVI system 100. The fixed- position imaging systems of the present disclosure do not include moving camera connections.

[0060] The fixed-position imaging systems may have a reduction in design cost compared to the imaging system 105. The fixed-position imaging systems may have station speed increase compared to AVI system 100. The fixed-position imaging systems may be a hundred times faster than the mechanical AVI system 100 of FIG. 1.

[0061] Although the systems, methods, devices, and components thereof, have been described in terms of exemplary embodiments, they are not limited thereto. The detailed description is to be construed as exemplary only and does not describe every possible embodiment of the invention because describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent that would still fall within the scope of the claims defining the invention.

[0062] Those skilled in the art will recognize that a wide variety of modifications, alterations, and combinations can be made with respect to the above described embodiments without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.

Claims

What is claimed is:

1. A system for capturing an image of a plurality of vessels, the system comprising: a first fixed-position imaging system configured to capture first images; a second fixed-position imaging system configured to capture second images; and a conveying mechanism configured to transport vessels past the first and second fixed-position imaging systems, wherein the first fixed-position imaging system, the second fixed-position imaging system, and the conveying mechanism are configured such that the first fixed-position imaging system captures a first image of a first subset of the vessels, and the second fixed-position imaging system simultaneously captures a second image of a second subset of the vessels, from a same perspective view relative to the vessels.

2. The system as in claim 1, further comprising: one or more processors configured to analyze the first image and the second image to inspect the first subset of vessels and the second subset of vessels for a same set of one or more features.

3. The system as in either of claims 1 or 2, wherein each vessel of the transported vessels is selected from a group including: a vial, a syringe, or a cartridge.

4. The system as in any one of claims 1-3, wherein the first fixed-position imaging system includes a first imaging device having a first telecentric lens and the second fixed-position imaging system includes a second imaging device having a second telecentric lens.

5. The system as in any one of claims 1-4, wherein the first fixed-position imaging system includes a first mirror configured to align a first central image axis with the first subset of vessels, and wherein the second fixed-position imaging system includes a second mirror configured to align a second central image axis with the second subset of vessels.

6. The system as in any one of claims 1-5, wherein the first fixed-position imaging system includes a first mirror oriented at a forty-five degree angle with respect to the first central image axis, and wherein the second fixed-position imaging system includes a second mirror oriented at a forty-five degree angle with respect to the second central image axis.

7. The system as in any one of claims 1-6, further comprising: a backlight oriented to emit light toward the first subset of vessels, the first fixed-position imaging system, the second subset of vessels, and the second fixed-position imaging system.

8. A method for imaging a plurality of vessels, the method comprising: conveying, using a conveying mechanism, vessels in front of a first fixed-position imaging system and a second fixed- position imaging system; capturing, using the first fixed-position imaging system, a first image of a first subset of the vessels; and capturing, using the second fixed-position imaging system, a second image of a second subset of the vessels, wherein capturing the first image and capturing the second image occur simultaneously and from a same perspective view relative to the vessels.

9. The method of claim 8, further comprising: one or more processors configured to analyze the first image and the second image to inspect the first subset of vessels and the second subset of vessels for a same set of one or more features.

10. The method as in either of claims 8 or 9, wherein the first fixed-position imaging system includes a first imaging device having a first telecentric lens and the second fixed-position imaging system includes a second imaging device having a second telecentric lens.

11. The method as in any one of claims 8-10, wherein the first fixed-position imaging system includes a first mirror configured to align a first central image axis with the first subset of vessels, and wherein the second fixed-position imaging system includes a second mirror configured to align a second central image axis with the second subset of vessels.

12. The method as in any one of claims 8-11, wherein the first fixed-position imaging system includes a first mirror oriented at a forty-five degree angle with respect to the first central image axis, and wherein the second fixed-position imaging system includes a second mirror oriented at a forty-five degree angle with respect to the second central image axis.

13. The method as in any one of claims 8-12, further comprising: emitting, using a backlight, light toward the first subset of vessels, the first fixed-position imaging system, the second subset of vessels, and the second fixed-position imaging system.

14. The method as in any one of claims 8-13, wherein the first subset of vessels includes two or more vessels, wherein the second subset of vessels includes two or more vessels.

15. A non-transitory computer-readable medium having computer-readable instructions stored thereon that, when executed by one or more processors, causes the one or more processors to: control a conveying mechanism to convey vessels in front of a first fixed-position imaging system and a second fixed- position imaging system; and simultaneously (I) control the first fixed-position imaging system to capture a first image of a first subset of the vessels, and (II) control the second fixed-position imaging system to capture a second image of a second subset of the vessels, from a same perspective view relative to the vessels.

16. The non-transitory computer-readable medium as in claim 15, wherein further execution of the computer- readable instructions by the one or more processors, further causes the one or more processors to: analyze the first image and the second image to inspect the first subset of vessels and the second subset of vessels for a same set of one or more features.

17. The non-transitory computer-readable medium as in either of claims 15 or 16, wherein the first fixed-position imaging system includes a first imaging device having a first telecentric lens and the second fixed-position imaging system includes a second imaging device having a second telecentric lens.

18. The non-transitory computer-readable medium as in any one of claims 15-17, wherein the first fixed-position imaging system includes a first mirror configured to align a first central image axis with the first subset of vessels, and wherein the second fixed-position imaging system includes a second mirror configured to align a second central image axis with the second subset of vessels.

19. The non-transitory computer-readable medium as in any one of claims 15-18, wherein the first fixed-position imaging system includes a first mirror oriented at a forty-five degree angle with respect to the first central image axis, and wherein the second fixed-position imaging system includes a second mirror oriented at a forty-five degree angle with respect to the second central image axis.

20. The non-transitory computer-readable medium as in any one of claims 15-19, wherein further execution of the computer-readable instructions by the one or more processors, further causes the one or more processors to: control a backlight that is oriented to emit light toward the first subset of vessels, the first fixed-position imaging system, the second subset of vessels, and the second fixed-position imaging system.