WO2025240656A1

WO2025240656A1 - Automated visual inspection image processing using gradient imaging techniques

Info

Publication number: WO2025240656A1
Application number: PCT/US2025/029422
Authority: WO
Inventors: Graham F. MILNE; Thomas Clark PEARSON; Al Patrick GOODWIN; Jason KENNON; Christian Omar PEREZ PEREZ
Original assignee: Amgen Inc
Current assignee: Amgen Inc
Priority date: 2024-05-15
Filing date: 2025-05-14
Publication date: 2025-11-20
Anticipated expiration: 2026-11-15

Abstract

A module may receive or generate a mask that defines a shape of a feature depicted in a first image. The mask includes first boundary pixels that define an outer perimeter of the feature. The module may identify a transfer area in a second image to which the feature will be transferred and overlay the mask on the transfer area to align the first boundary pixels with second boundary pixels of the second image that define an outer perimeter of the transfer area. The module may define respective gradient values for pixels of the first image within the first boundary pixels of the mask and generate a set of replacement values for pixels of the transfer area based on the second boundary pixels and the respective gradient values. The module may generate the synthetic image by replacing pixel values of the transfer area with the set of replacement values

Description

AUTOMATED VISUAL INSPECTION IMAGE PROCESSING USING GRADIENT IMAGING TECHNIQUES

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63/648,107, filed May 15, 2024.

FIELD OF DISCLOSURE

[0002] The present application relates generally to automated visual inspection systems for pharmaceutical or other applications, and more specifically to techniques that augment image libraries using gradient imaging techniques to generate synthetic images for use in developing, training, and/or validating such systems.

BACKGROUND

[0003] In various contexts, quality control procedures require the careful examination of samples for defects, with any samples exhibiting defects being rejected, discarded, and/or further analyzed In a pharmaceutical manufacturing context, for example, containers (e.g., syringes or vials) and/or their contents (e g., fluid or lyophilized drug products) must be rigorously inspected for defects prior to sale or distribution. Numerous other industries likewise rely on visual inspection in order to ensure product quality, or for other purposes. Increasingly, the defect inspection task has become automated (i.e., “automated visual inspection” or “AVI”) in order to remove human error, lower costs, and/or reduce inspection times (e.g., to handle large quantities of drugs or other items in commercial production). For example, “computer vision” or “machine vision" software has been used in pharmaceutical contexts. [0004] Recently, deep learning techniques have emerged as a promising tool for AVI. Generally, however, these techniques require far more images than traditional AVI systems to develop, train, and fully test the models (e.g., neural networks). Moreover, robust model performance generally depends on a carefully designed image set. For example, the image set should exhibit sufficiently diverse conditions (e.g., by showing defects in different locations, and having a range of different shapes and sizes, etc.). Further, even a large and diverse training image library can result in poor AVI performance if the image set causes the deep learning model to make decisions for the wrong reasons (e.g., based on irrelevant image features). This can be particularly problematic in contexts or scenarios where depicted defects are small or indistinct relative to other (non-defect) image features

[0005] For both deep learning and more traditional (e g , machine vision) AVI systems, development and qualification processes that use sample image libraries should ensure that false negatives or “false accepts” (i.e., a defect is missed), as well as false positives or “false rejects” (i.e , a defect is incorrectly identified), are within tolerable thresholds. For example, zero or near-zero false negatives may be required in certain contexts (e.g., pharmaceutical contexts where patient safety is a concern). While false positives can be less critical, they can be very costly in economic terms, and can be more difficult to address than false negatives when developing an AVI system. These and other factors can make the development of an image library a highly iterative process that is very complex, labor-intensive, and costly. Further still, any product line changes (e.g., new drugs, new containers, new fill levels for drugs within the containers, etc.), or changes to the inspection process itself (e.g., different types of camera lenses, changes in camera positioning or illumination, etc.), can require not only retraining and/or requalifying the model, but also (in some cases) a partial or total rebuild of the image library.

[0006] Furthermore, because of the cost or practical difficulties (e g., having to shut down assembly lines, introduce unsafe broken elements into working machines, etc.) associated with generating images for the image library, various techniques for generating synthetic images for some or all of the sample image libraries have been deployed Such techniques can include full generation of images from trained neural networks or augmentations of original or real images with defects or features present in other images. However, the established, Al-based image generation methods can produce low fidelity or anomalous images inadequate for AVI recipe development or deep learning model training. Furthermore, the augmentation techniques can also produce inadequate training images when used in transferring features onto original images, or portions thereof, that contain high contrast or non-uniform backgrounds.

SUMMARY

[0007] In some aspects, the techniques described herein relate to a method of generating a synthetic image, the method including: receiving or generating a mask that defines a shape of a feature depicted in a first image, wherein the mask includes first boundary pixels that define an outer perimeter of the feature; identifying a transfer area in a second image to which the feature will be transferred; overlaying the mask on the transfer area to align the first boundary pixels with second boundary pixels of the second image that define an outer perimeter of the transfer area; defining respective gradient values for pixels of the first image within the first boundary pixels of the mask; generating a set of replacement values for pixels of the transfer area based on the second boundary pixels and the respective gradient values for the pixels of the first image within the first boundary pixels of the mask; and generating the synthetic image by replacing pixel values of the transfer area with the set of replacement values

[0008] In some aspects, the techniques described herein relate to a system including: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the system to: receive or generate a mask that defines a shape of a feature depicted in a first image, wherein the mask includes first boundary pixels that define an outer perimeter of the feature; identify a transfer area in a second image to which the feature will be transferred; overlay the mask on the transfer area to align the first boundary pixels with second boundary pixels of the second image that define an outer perimeter of the transfer area; define respective gradient values for pixels of the first image within the first boundary pixels of the mask; generate a set of replacement values for pixels of the transfer area based on the second boundary pixels and the respective gradient values for the pixels of the first image within the first boundary pixels of the mask; and generate a synthetic image by replacing pixel values of the transfer area with the set of replacement values.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The skilled artisan will understand that the figures described herein are included for purposes of illustration and do not limit the present disclosure The drawings are not necessarily to scale, and emphasis is instead placed upon illustrating the principles of the present disclosure It is to be understood that, in some instances, various aspects of the described implementations may be shown exaggerated or enlarged to facilitate an understanding of the described implementations.

[0010] FIG. 1 is a simplified block diagram of an example system that can implement various techniques described herein relating to the development and/or assessment of an automated visual inspection (AVI) image library.

[0011] FIG. 2 depicts an example visual inspection system that may be used in a system such as the system of FIG. 1.

[0012] FIGs. 3A through 3C depict various example container types that may be inspected using a visual inspection system such as the visual inspection system of FIG. 2.

[0013] FIG. 4A depicts a schematic representation of a feature image used to generate a synthetic image for inclusion in an AVI the image library of a system such as the system of FIG 1

[0014] FIG. 4B depicts a schematic representation of an original image used to generate a synthetic image for inclusion in an AVI the image library of a system such as the system of FIG 1 [0015] FIG. 4C depicts a schematic representation of a synthetic image formed using a portion of the feature image of FIG. 4A and the original image of FIG. 4B for inclusion in an AVI the image library of a system such as the system of FIG. 1.

[0016] FIG. 5A depicts a mask of a feature image overlaid on an original image in the process of generating a synthetic image as described herein.

[0017] FIG. 5B depicts a synthetic image generated from the combination of the original image and mask of FIG. 5A

[0018] FIG. 6A depicts a feature image used in the process of forming a synthetic image as described herein

[0019] FIG. 6B depicts a synthetic image contrasting application of a feature of the feature image of FIG 6A using the gradient imaging techniques as described and prior techniques.

[0020] FIG. 6C depict a close up of the synthetic image of FIG. 6B on the feature applied using the gradient imaging techniques as described herein.

[0021] FIG. 7A depicts an oversized mask of a feature of a feature image.

[0022] FIG. 7B depicts a synthetic image formed from the oversized mask of FIG. 7A using the gradient imaging techniques as described herein.

[0023] FIG. 8A depicts a more precise mask on the feature of FIG. 7A on as compared with the oversized mask of the feature

[0024] FIG. 8B depicts a synthetic image formed from the more precise mask of FIG. 8A using the gradient imaging techniques as described herein.

[0025] FIG. 8C depicts a synthetic image formed from the more precise mask of FIG. 8A using another variation of the gradient imaging techniques as described herein.

[0026] FIG 8D depicts a schematic of a mask with first boundary pixels within one pixel a distance of the edges of a feature

[0027] FIG 8E depicts a schematic of a mask with first boundary pixels within a two pixel distance of the edges of a feature

[0028] FIG 8F depicts a schematic of a mask with first boundary pixels within a three pixel distance of the edges of a feature

[0029] FIG. 9A depicts a feature image with a label applied to a feature present in the feature image.

[0030] FIG. 9B depicts the feature image of FIG. 9A with a mask representative of the labeled feature

[0031] FIG. 9C depicts a synthetic image formed using the mask from the feature image of FIG. 9B

[0032] FIG. 10A depicts an original image used to generate a synthetic image using the gradient imaging techniques as described herein.

[0033] FIG. 10B depicts the synthetic image generated using the original image of FIG. 10A.

[0034] FIGs. 11 A and 11 B depict synthetic images generated using the gradient imaging techniques as described herein.

[0035] FIG. 12 is a flow diagram of an example method for generating a synthetic image by transferring a feature from a first image onto a second image.

[0036] FIG. 13 is a flow diagram of an example method for automating generation of masks and transferring features to generate a library of synthetic images.

DETAILED DESCRIPTION

[0037] The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways, and the described concepts are not limited to any particular manner of implementation. Examples of implementations are provided for illustrative purposes.

[0038] As the terms are used herein, “synthetic image” and “augmented image” (used interchangeably) generally refers to an image that has been digitally altered to depict something different than what the image originally depicted, and is to be distinguished from the output produced by other types of image processing (e.g , adjusting contrast, changing resolution, cropping, filtering, etc.) that do not change the nature of the thing depicted. Conversely, a "real image,” as referred to herein, refers to an image that is not a synthetic/augmented image, regardless of whether other type(s) of image processing have previously been applied to the image. An “original image,” as referred to herein, is an image that may be digitally modified (e.g., by, using the techniques described herein) to generate a synthetic/augmented image, and may be a real image or a synthetic image (e.g., an image that was previously augmented, prior to an additional round of augmentation). References herein to depicted “features” (e.g., depicted “defects”) include references to objects (e.g., a syringe, cartridge, vial, container, cartons, etc.) depicted within an underlying image and/or characteristics of the object or other thing imaged (e.g., a crack or meniscus of a syringe as shown in an image of the syringe, or a scratch or dent on an automobile body component as shown in an image of the component, etc.), and are to be distinguished from features of the image itself that are unrelated to the nature of the thing imaged (e.g., missing or damaged portions of an image, such as faded or defaced portions of an image, etc.). The features may also include portions of images that depict broken glass shards, metal scrap, broken containers, etc.

[0039] FIG. 1 is a simplified block diagram of a non-limiting example of a system 100 that can implement various techniques described herein relating to the development and/or assessment of an automated visual inspection (AVI) training and/or validation image library. For example, the image library may be used to train one or more neural networks to perform AVI tasks. Once trained and qualified, the AVI neural network(s) may be used for quality control at the time of manufacture (and/or in other contexts) to detect defects. In a pharmaceutical context, for example, the AVI neural network(s) may be used to detect defects associated with syringes, vials, cartridges, or other container types (e.g., cracks, scratches, stains, missing components, particles, foreign material, damaged seals and damaged stoppers, etc , of the containers), and/or to detect defects associated with fluid or lyophilized drug products within the containers (e g , the presence of fibers and/or other foreign particles as well as all the previously mentioned defect types) As another example, in an automotive context, the AVI neural network(s) may be used to detect defects in the bodywork of automobiles or other vehicles (e.g., cracks, scratches, dents, stains, etc.), during production and/or at other times (e.g., to help determine a fair resale value, to check the condition of a returned rental vehicle, etc.). Numerous other uses are also possible Because the disclosed techniques can substantially lower the cost and time associated with building an image library, AVI neural networks may be used to detect visible defects in virtually any quality control application (e.g., checking the condition of appliances, home siding, textiles, glassware, etc., prior to sale). It is understood that, while the examples provided herein relate primarily to the pharmaceutical context, the techniques described herein need not be limited to such applications. Moreover, in some implementations, the synthetic images are used for a purpose other than training an AVI neural network. For example, the images may instead be used to develop and/or qualify a system that uses computer vision without deep learning.

[0040] System 100 includes a visual inspection system 102 that is configured to produce training and/or validation images. Specifically, visual inspection system 102 includes hardware (e.g., a conveyance mechanism, light source(s), camera(s), etc.), as well as firmware and/or software, that is configured to capture digital images of a sample (e.g., a container holding a fluid or lyophilized substance). One example of visual inspection system 102 is described below with reference to FIG. 2, although any suitable visual inspection system may be used. In some embodiments, the visual inspection system 102 is an offline (e.g., lab-based) “mimic station” that closely replicates important aspects of a commercial line equipment station (e.g., optics, lighting, etc.), thereby allowing development of the training and/or validation library without causing excessive downtime of the commercial line equipment.

[0041] Visual inspection system 102 may image each of a number of samples (e.g., containers) sequentially. To this end, visual inspection system 102 may include, or operate in conjunction with, a Cartesian robot, conveyor belt, carousel, starwheel, and/or other conveying means that successively move each sample into an appropriate position for imaging, and then move the sample away once imaging of the sample is complete While not shown in FIG. 1, visual inspection system 102 may include a communication interface and processors to enable communication with computer system 104.

[0042] Computer system 104 may generally be configured to control/automate the operation of visual inspection system 102, and to receive and process images captured/generated by visual inspection system 102, as discussed further below Computer system 104 may be a general-purpose computer that is specifically programmed to perform the operations discussed herein, or a specialpurpose computing device. As seen in FIG. 1, computer system 104 includes a processing unit 110 and a memory unit 114. In some embodiments, however, computer system 104 includes two or more computers that are either co-located or remote from each other. In these distributed embodiments, the operations described herein relating to processing unit 110 and memory unit 114, or relating to any of the modules implemented when processing unit 110 executes instructions stored in memory unit 114, may be divided among multiple processing units and/or multiple memory units.

[0043] Processing unit 110 includes one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in memory unit 114 to execute some or all of the functions of computer system 104 as described herein. Processing unit 110 may include one or more graphics processing units (GPUs) and/or one or more central processing units (CPUs), for example. Alternatively, or in addition, one or more processors in processing unit 110 may be other types of processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.), and some of the functionality of computer system 104 as described herein may instead be implemented in hardware.

[0044] Memory unit 114 may include one or more volatile and/or non-volatile memories Any suitable memory type or types may be included in memory unit 114, such as read-only memory (ROM) and/or random access memory (RAM), flash memory, a solid-state drive (SSD), a hard disk drive (HDD), and so on Collectively, memory unit 114 may store one or more software applications, the data received/used by those applications, and the data output/generated by those applications

[0045] In particular, memory unit 114 stores the software instructions of various modules that, when executed by processing unit 110, perform various functions for the purpose of training, validating, and/or qualifying one or more AVI neural networks, and/or other types of AVI software (e.g., computer vision software). Specifically, in the example embodiment of FIG. 1, memory unit 114 includes an AVI neural network module 120, a visual inspection system (VIS) control module 122, a library expansion module 124, and an Im age/library assessment module 126. In other embodiments, memory unit 114 may omit one or more of modules 120, 122, 124 and 126, and/or include one or more additional modules. As noted above, computer system 104 may be a distributed system, in which case one, some, or all of modules 120, 122, 124 and 126 may be implemented in whole or in part by a different computing device or system (e.g., by a remote server coupled to computer system 104 via one or more wired and/or wireless communication networks). Moreover, the functionality of any one of modules 120, 122, 124 and 126 may be divided among different software applications. As just one example, in an embodiment where computer system 104 accesses a web service to train and use one or more AVI neural networks, some or all of the software instructions of AVI neural network module 120 may be stored and executed at a remote server. [0046] AVI neural network module 120 comprises software that uses images stored in a training image library 140 to train one or more AVI neural networks. Training image library 140 may be stored in memory unit 114, and/or in another local or remote memory (e.g., a memory coupled to a remote library server, etc.) In some embodiments, in addition to training, AVI neural network module 120 may implement/run the trained AVI neural network(s), e.g., by applying images newly acquired by visual inspection system 102 (or another visual inspection system) to the neural network(s) for validation, qualification, or possibly even run-time operation. In various embodiments, the AVI neural network(s) trained by AVI neural network module 120 to classify entire images (e.g., defect vs. no defect, or presence or absence of a particular type of defect, etc.), classify images on a per-pixel basis (i.e., image segmentation), detect objects in images (e g., detect the presence and position of particular defect types such as scratches, cracks, foreign objects, stains, missing components, particles, foreign material, damaged seals and damaged stoppers, etc.), or some combination thereof (e.g., one neural network classifying images, and another performing object detection). The AVI neural network module 120 may include deep learning software such as Halcon® from MVTec, Vidi® from Cognex®, Rekognition® from Amazon®, TensorFlow, PyTorch, and/or any other suitable off-the-shelf or customized deep learning software. The software of AVI neural network module 120 may be built on top of one or more pre-trained networks, such as ResNet50 or VGGNet, for example, and/or one or more custom networks. It should also be appreciated that the AVI neural network module 120 may be used in conjunction with traditional image processing tools to provide a comprehensive system for detecting defects. Accordingly, the disclosed techniques for generating synthetic images may also be applied to develop and/or validate traditional image processing tools as well.

[0047] In some embodiments, VIS control module 122 controls/automates operation of visual inspection system 102 such that sample images (e.g , container images) can be generated with little or no human interaction. VIS control module 122 may cause a given camera to capture a sample image by sending a command or other electronic signal (e.g., generating a pulse on a control line, etc.) to that camera. Visual inspection system 102 may send the captured container images to computer system 104, which may store the images in memory unit 114 for local processing In alternative embodiments, visual inspection system 102 may be locally controlled, in which case VIS control module 122 may have less functionality than is described herein (e.g., only handling the retrieval of images from visual inspection system 102), or may be omitted entirely from memory unit 114.

[0048] Library expansion module 124 (also referred to herein as simply “module 124") processes sample images generated by visual inspection system 102 (and/or other visual inspection systems) to generate additional, synthetic/augmented images for inclusion in training image library 140. Library expansion module 124 may implement one or more image augmentation techniques, including any one or more of the image augmentation techniques disclosed herein As discussed below, some of those image augmentation techniques may make use of a feature image library 142 to generate synthetic images Feature image library 142 may be stored in memory unit 114, and/or in another local or remote memory (e g , a memory coupled to a remote library server, etc ), and contains images of various types of defects (e.g., cracks, scratches, chips, stains, foreign objects, particles, damaged seals and damaged stoppers, etc.), and/or images of variations of each defect type (e g., cracks with different sizes and/or patterns, foreign objects having different shapes and sizes, etc.). Alternatively, or in addition, feature image library 142 may include images of various other types of features (e.g., different meniscuses), which may or may not exhibit defects. The images in feature image library 142 may be cropped portions of full sample images, for example, such that a substantial portion of each image includes the feature (e.g., defect). The images in feature image library 142 may also include pharmaceutical containers (e.g., a syringe, cartridge, vial, container, etc.) and/or broken pieces of containers or production process equipment (e.g., small steel, acrylic, glass, etc pieces).

[0049] Generally, the feature image library 142 may include images of virtually any type(s) of feature associated with the samples being imaged. In a pharmaceutical context, for example, the feature image library 142 may include defects associated with containers (e.g., syringes, cartridges, vials, etc.), container contents (e.g., liquid or lyophilized drug products), and/or interactions between the containers and their contents (e g., leaks, etc.). As non-limiting examples, the defect images may include images of syringe defects such as: a crack, chip, scratch, and/or scuff in the barrel, shoulder, neck, or flange; a broken or malformed flange; an air line in glass of the barrel, shoulder, or neck wall; a discontinuity in glass of the barrel, shoulder, or neck; a stain on the inside or outside (or within) the barrel, shoulder, or neck wall; adhered glass on the barrel, shoulder, or neck; a knot in the barrel, shoulder, or neck wall; a foreign particle embedded within glass of the barrel, shoulder, or neck wall; a foreign, misaligned, missing, or extra plunger; a stain on the plunger, malformed ribs of the plunger; an incomplete or detached coating on the plunger; a plunger in a disallowed position; a missing, bent, malformed, or damaged needle shield; a needle protruding from the needle shield; etc. Examples of defects associated with the interaction between syringes and the syringe contents may include a leak of liquid through the plunger, liquid in the ribs of the plunger, a leak of liquid from the needle shield, and so on. Various components of an example syringe are shown in FIG. 3A, discussed below.

[0050] Non-limiting examples of defects associated with cartridges may include: a crack, chip, scratch, and/or scuff in the barrel or flange; a broken or malformed flange; a discontinuity in the barrel; a stain on the inside or outside (or within) the barrel; materials adhered to the barrel; a knot in the barrel wall; a foreign, misaligned, missing, or extra piston; a stain on the piston; malformed ribs of the piston; a piston in a disallowed position; a flow mark in the barrel wall; a void in plastic of the flange, barrel, or luer lock; an incomplete mold of the cartridge; a missing, cut, misaligned, loose, or damaged cap on the luer lock; foreign particles and foreign fibers, etc. Examples of defects associated with the interaction between cartridges and the cartridge contents may include a leak of liquid through the piston, liquid in the ribs of the piston, and so on. Various components of an example cartridge are shown in FIG. 3B, discussed below.

[0051] Non-limiting examples of defects associated with vials may include: a crack, chip, scratch, and/or scuff in the body; an air line in glass of the body; a discontinuity in glass of the body; a stain on the inside or outside (or within) the body; foreign particles and foreign fibers; adhered glass on the body; a knot in the body wall; a flow mark in the body wall; a missing, misaligned, loose, protruding or damaged crimp; a missing, misaligned, loose, or damaged flip cap; etc. Examples of defects associated with the interaction between vial and the vial contents may include a leak of liquid through the crimp or the cap, and so on. Various components of an example vial are shown in FIG. 3C, discussed below.

[0052] Non-limiting examples of defects associated with container contents (e.g , contents of syringes, cartridges, vials, or other container types) may include: a foreign particle suspended within liquid contents; a foreign particle resting on the plunger dome, piston dome, or vial floor; a discolored liquid or cake; a cracked, dispersed, or otherwise atypically distri buted/formed cake; a turbid liquid; a high or low fill level; etc “Foreign” particles may be, for example, fibers, bits of rubber, metal, stone, or plastic, hair, glass, and so on In some embodiments, bubbles are considered to be innocuous and are not considered to be defects

[0053] Non-limiting examples of other types of features that may be depicted in images of feature image library 142 may include: meniscuses of different shapes and/or at different positions; plungers of different types and/or at different positions; bubbles of different sizes and/or shapes, and/or at different locations within a container; different air gap sizes in a container; different sizes, shapes, and/or positions of irregularities in glass or another translucent material; etc.

[0054] It should be appreciated that while the term “defect” is used throughout to refer to a feature that is transferred onto an original image, in some embodiments, the feature is not a defect. For example, when performing image segmentation to, for example, detect a position of a stopper, the presence of bubbles may affect the segmentation algorithm. Accordingly, to improve the ability of the segmentation algorithms to be robust with respect to bubbles, a bubble feature may be transferred onto original images. Thus, the use of “defect" encompasses other features (such as bubbles or other features) that, while not conventionally understood to be defects, nonetheless affect the performance of one or more visual inspection analyses. Additionally, as will be described below, the presence of a container (or other physical object, such as a screw, bolt, container shard, etc.) in an inappropriate location may be considered a “defect” for certain automated processes. Thus, the term “defect” may also encompass features representative of physical objects when training a visual inspection model that monitors these automated processes.

[0055] In operation, the computer system 104 stores the sample images collected by visual inspection system 102 (possibly after cropping and/or other image pre-processing by computer system 104), as well as synthetic images generated by library expansion module 124, and possibly real and/or synthetic images from one or more other sources, in training image library 140. AVI neural network module 120 then uses at least some of the sample images in training image library 140 to train the AVI neural network(s), and uses other images in library 140 (or in another library not shown in FIG 1) to validate the trained AVI neural network(s). As the terms are used herein, “training,” “validating,” or “qualifying” a neural network encompasses directly executing the software that runs the neural network, and also encompasses initiating the running of the neural network (e.g., by commanding or requesting a remote server to train the neural network or run the trained neural network). In some embodiments, for example, computer system 104 may “train" a neural network by accessing a remote server that includes AVI neural network module 120 (e g., by accessing a web service supported by the remote server).

[0056] The operation of each of modules 120 through 126 is discussed in further detail below, with reference to elements of various other figures.

[0057] FIG. 2 depicts a non-limiting example of a visual inspection system 200 that may be used as the visual inspection system 102 of FIG. 1, in a pharmaceutical application. Visual inspection system 200 includes a camera 202, a lens 204, forward-angled light sources 206a and 206b, rear-angled light sources 208a and 208b, a backlight source 210, and an agitation mechanism 212 Camera 202 captures one or more images of a container 214 (e.g., a syringe, vial, cartridge, or any other suitable type of container) while container 214 is held by agitation mechanism 212 and illuminated by light sources 206, 208, and/or 210 (e.g., with VIS control module 122 activating different light sources for different images, sequentially or simultaneously). The visual inspection system 200 may include additional or fewer light sources (e.g., by omitting backlight source 210). Container 214 may hold a liquid or lyophilized pharmaceutical product, for example

[0058] Camera 202 may be a high-performance industrial camera or smart camera, and lens 204 may be a high-fidelity telecentric lens, for example In one embodiment, camera 202 includes a charge-coupled device (CCD) sensor. For example, camera 202 may be a Basler® pilot piA2400-17gm monochrome area scan CCD industrial camera, with a resolution of 2448 x 2050 pixels or a Baumer VLXT-240M CMOS sensor camera, with a resolution of 5312 x 4600 pixels As used herein, the term “camera” may refer to any suitable type of imaging device (e g , a camera that captures the portion of the frequency spectrum visible to the human eye, or an infrared camera, etc )

[0059] The different light sources 206, 208 and 210 may be used to collect images for detecting defects in different categories. For example, forward-angled light sources 206a and 206b may be used to detect reflective particles or other reflective defects, rearangled light sources 208a and 208b may be used for particles generally, and backlight source 210 may be used to detect opaque particles, and/or to detect incorrect dimensions and/or other defects of containers (e.g., container 214). Light sources 206 and 208 may include CCS® LDL2-74X30RD bar LEDs, and backlight source 210 may be a CCS® TH-83X75RD backlight, for example.

[0060] Agitation mechanism 212 may include a carousel system, a robotic system with grippers, a chuck, and/or other means for holding and rotating (e.g., spinning) containers such as container 214. For example, agitation mechanism 212 may include any high speed motor, such as Animatics® SM23165D SmartMotor, with a spring-loaded chuck securely mounting each container (e.g., syringe) to the motor.

[0061] While the visual inspection system 200 may be suitable for producing container images to train and/or validate one or more AVI neural networks, the ability to detect defects across a broad range of categories may require multiple perspectives. Thus, in some implementations, visual inspection system 102 of FIG 1 may instead be a multi-camera system. In still other implementations, visual inspection system 102 of FIG 1 may include a line-scan camera, and rotate the sample (e.g., container) to capture each image. Moreover, automated handling/conveyance of samples may be desirable in order to quickly obtain a much larger set of training images. Visual inspection system 102 may include any other suitable visual inspection system for any type of product. In an automotive context, for example, visual inspection system 200 may include a conveyor belt with illumination sources and multiple cameras mounted above and/or around a particular conveyor belt station The visual inspection system 200 may also include a line clearance camera configured to detect if a pharmaceutical container (e. g . , a syringe, cartridge, vial, container, etc.) has fallen off the process conveyance system or if pieces of the containers have been broken by the machine, etc.

[0062] While the visual inspection system 200 may be suitable for producing container images to train and/or validate one or more AVI neural networks, the ability to detect defects across a broad range of categories may involve multiple cameras with different perspectives. Moreover, automated handling/conveyance of containers may be desirable in order to obtain a much larger set of container images, and therefore train the AVI neural network(s) to more accurately detect defects. As such, other example visual inspection systems used in defect detecting inspection operations may facilitate such features.

[0063] For example, an example visual inspection system for detecting defects may include three cameras mounted on a platform, in a generally radial configuration around (and directed in towards) a container (e.g., a syringe, vial, cartridge, or any other suitable type of container). Each of the cameras may be similar to camera 202 and may include a telecentric lens similar to lens 204, for example, and the container may hold a liquid or lyophilized pharmaceutical product. An agitation mechanism may hold and agitate the container. The agitation mechanism may be similar to the agitation mechanism 212, for example. Opposite each of the cameras is a respective one of rear light sources, which include both rear-angled light sources (e.g., each similar to the combination of light sources 208a and 208b) and a backlight source (e.g., similar to backlight source 210). In these embodiments, the cameras are aligned such that the optical axis of each camera falls within the same horizontal plane, and passes through the container.

[0064] Another example visual inspection system for detecting defects may include three cameras (possibly mounted on a platform similar to the platform described above), in a generally radial configuration around (and directed in towards) a container (e.g., a syringe, vial, cartridge, or any other suitable type of container) holding a liquid or lyophilized product. In this embodiment, each of the cameras are coupled to a right-angle telecentric lens, in order to reduce the overall footprint while maintaining telecentric performance For example, each of the cameras may be a Basler® Ace CMOS camera coupled to an OptoEngineering® TCCR23048-C right-angle telecentric lens An agitation mechanism (similar to the agitation mechanism 212) may hold and agitate the container. Opposite each of the cameras is a respective one of the rear light sources that includes both rear-angled light sources (e.g., similar to light sources 208a and 208b) and a backlight source (e.g., similar to backlight source 210). The cameras may be aligned such that the optical axis of each camera falls within the same horizontal plane, and passes through the container. In these embodiments, the visual inspection system also includes forward-angled light sources (e.g., each similar to the combination of light sources 206a and 206b).

[0065] The triangular camera configurations can increase the space available for multiple imaging stations, and potentially provide other advantages. For example, such an arrangement may make it possible to capture the same defect more than once, either at different angles (e.g., for container defects) or with three shots/images simultaneously (e.g., for particle defects), which in turn could increase detection accuracy. As another example, such an arrangement may facilitate conveyance of containers into and out of the imaging region

[0066] FIGs. 3A through 3C depict various non-limiting example container types that, in certain pharmaceutical contexts, may be used as the samples imaged by visual inspection system 102 of FIG. 1 or visual inspection system 200 of FIG. 2. Referring first to FIG. 3A, an example syringe 300 includes a hollow barrel 302, a flange 304, a plunger 306 that provides a movable fluid seal within the interior of barrel 302, and a needle shield 308 to cover the syringe needle (not shown in FIG. 3A). Barrel 302 and flange 304 may be formed of glass and/or plastic, and plunger 306 may be formed of rubber and/or plastic, for example. The needle shield 308 is separated by a shoulder 310 of syringe 300 by a gap 312. Syringe 300 contains a liquid (e.g., drug product) 314 within barrel 302 and above plunger 306. The top of liquid 314 forms a meniscus 316, above which is an air gap 318 [0067] Referring next to FIG. 3B, an example cartridge 320 includes a hollow barrel 322, a flange 324, a piston 326 that provides a movable fluid seal within the interior of barrel 322, and a luer lock 328. Barrel 322, flange 324, and/or luer lock 328 may be formed of glass and/or plastic and piston 326 may be formed of rubber and/or plastic, for example. Cartridge 320 contains a liquid (e.g., drug product) 330 within barrel 322 and above piston 326. The top of liquid 330 forms a meniscus 332, above which is an air gap 334. [0068] Referring next to FIG. 3C, an example vial 340 includes a hollow body 342 and neck 344, with the transition between the two forming a shoulder 346. At the bottom of vial 340, body 342 transitions to a heel 348. A crimp 350 includes a stopper (not visible in FIG. 3C) that provides a fluid seal at the top of vial 340, and a flip cap 352 covers crimp 350. Body 342, neck 344, shoulder 346, and heel 348 may be formed of glass and/or plastic, crimp 350 may be formed of metal, and flip cap 352 may be formed of plastic, for example. Vial 340 may include a liquid (e.g., drug product) 354 within body 342. The top of liquid 354 may form a meniscus 356 (e.g., a very slightly curved meniscus, if body 342 has a relatively large diameter), above which is an air gap 358. In other embodiments, instead of liquid 354, there is instead a solid material within vial 340. For example, vial 340 may include a lyophilized (freeze dried) drug product, also referred to as “cake."

[0069] Various image augmentation techniques to generate synthetic images may be implemented by library expansion module 124 (as executed by processing unit 110) For example, an arithmetic transposition algorithm, digital “inpainting” techniques such as the PatchMatch algorithm, deep learning-based inpainting such as a partial convolution model (e.g., a model that performs convolutions across the entire image to add an aspect of pixel noise and variation to the synthetic (inpainted) image and slightly distinguish the synthetic image from the original beyond the inpainted region), etc.

[0070] However, these image augmentation techniques have certain drawbacks that make them less suitable for use in transposing features onto original images, particularly when transposing onto original images or portions thereof, that contain non- uniform backgrounds For example, image augmentation techniques (e.g., simple cut and paste methods) can result in artifacts in the synthetic image, including around the boundary pixels of the transposed feature such that the feature is not integrated seamlessly into the background. For example, the feature may have characteristics (e.g., brightness, intensity, etc.) that do not match the characteristics of the area where the feature is transposed, and these differences can result in artifacts around the boundary pixels of the transposed feature. Artifacts are particularly common when a feature is transposed to an area that is non-uniform or that has high contrast. A non-uniform or high contrast area includes one where there is a rapid change in brightness (e.g. about 50 units or greater), for example an area where pixels transition from high intensity to low intensity over a short distance (e.g., about 5 pixels or fewer). Even if these artifacts cannot be detected by a human observer, they may impact training of a neural network. If a synthetic image includes artifacts, a neural network may be improperly trained to associate artifacts with features of defects and thereby focus on the “wrong” or irrelevant features of a synthetic image.

[0071] The gradient transposition techniques described herein address the problem of artifacts in synthetic images with transposed features. In situations where the background is non-uniform or high contrast, gradient transposition techniques as described herein can be used to generate more suitable synthetic images. For example, the gradient transposition techniques as described herein result in the transposed feature in the synthetic image maintaining the characteristics (e.g., brightness, intensity, etc.) of the background of the original image, reducing artifacts. This effect can be enhanced when a narrowly tailored (or narrowly defined or tightly defined) mask is used to define the feature before transposing. Using the gradient transposition techniques as described herein, artifacts are reduced.

[0072] Gradient transposition techniques include the library expansion module 124 loading a feature image 400 (shown schematically in FIG. 4A) from the feature image library 142. The feature image 400 includes a feature 402 that is to be transposed onto an original image 408 (FIG 4B) to generate a synthetic image 418 (FIG. 4C). The feature 402 in the feature image 400 may include a particular type of defect (e.g., scratch, crack, stain, foreign object, malformed plunger, cracked cake, bubble(s), missing components, particles, damaged seals and damaged stoppers, etc.) or an object (e.g., a syringe, cartridge, vial, container, etc.). The feature 402 occupies a specific area or region of the feature image 400 and a mask 404 defines a shape of the feature 402 within the area or region. The mask 404 includes first boundary pixels 406 that define an outer perimeter of the feature 402. The mask 404 can be dynamically generated by the library expansion module 124 as described in more detail below, be embedded metadata of the feature image 400, and/or otherwise associated with the feature image 400 in the feature image library 142.

[0073] As shown in FIG. 4B, the original image 408 loaded by the library expansion module 124 from the feature image library 142 includes a transfer area 410 identified by the library expansion module 124. The transfer area 410 has a shape that matches the mask 404 and second boundary pixels 412 that define an outer perimeter of the transfer area 410. In some embodiments, the library expansion module 124 defines the transfer area 410 using the mask 404 of the feature image 400, such as by overlaying the mask 404 onto the original image 408 In some embodiments, the mask 404 is randomly overlaid within a surrogate feature 411 of the original image 408. The surrogate feature 411 may correspond to an area of the original image 408 on which the feature 402 would likely be present in the original image 408. For example, where the feature 402 is a crack in a glass container, the surrogate feature 411 can be areas of the original image 408 corresponding to glass features of a container rather than non-glass elements like the background, container, lid, etc. Furthermore, the second boundary pixels 412 divide the original image 408 into interior pixels 414 of the transfer area 410 and exterior pixels 416 of the remainder of the original image 408. The original image 408 can be a container image (e.g., a syringe, cartridge, or vial similar to one of the containers shown in FIGs. 3A through 3C) or system image (e.g., an image showing a portion of a packing or manufacturing machine for a container or object). The container image and the system image may be a real image captured by visual inspection system 102 of FIG 1 or visual inspection system 200 of FIG 2, for example Depending on the implementation, the real image may have been processed in other ways (e g , cropped, filtered, etc ) prior to being loaded by the library expansion module 124

[0074] As shown in FIG. 4C, a variant 420 of the feature 402 replaces the transfer area 410 within the synthetic image 418 while the exterior pixels 416 from the original image 408 are preserved. The variant 420 represents the transposed feature 402 and is defined by the second boundary pixels 412 and modified pixels 422 within the second boundary pixels 412. That is, the library expansion module 124 may form the variant 420 by generating a set of replacement values 423 corresponding to the modified pixels 422 and replacing the pixel values of the interior pixels 414 with the set of replacement values 423. In some embodiments, the set of replacement values 423 can also include a set of replacement values corresponding to the second boundary pixels 412.

[0075] The library expansion module 124 (as executed by processing unit 110) is configured to generate the set of replacement values 423 using respective gradient values for pixels of the feature image 400 within the first boundary pixels 406 of the mask 404 (e.g., gradient values for pixels that represent the feature 402). These respective gradient values may incorporate differences in intensity between each pixel and neighboring pixels thereof (e.g., accounting for changes in light intensity in every direction). However, it is possible that the respective gradient values are determined based on changes with respect to only some neighboring pixels (e.g., only horizontal or vertical light intensity change). In some embodiments, the set of replacement values 423 can also be generated using gradient values for the interior pixels 414 such as by blending gradient values (the gradient values obtained from the feature image 400 with gradient values for the interior pixels 414) using a weighted average or the like. It should be appreciated that term “gradient” as used herein may include a scalar value derived from the gradient vector gradient that is the first derivative in n- dimensions. In particular, this scalar value may include a magnitude value determined by taking the dot product of the gradient vector. [0076] In some embodiments, the respective gradient values for pixels of the feature image 400 within the first boundary pixels 406 of the mask 404 are determined by the library expansion module 124 using Poisson imaging techniques, but other methods for determining the gradient may also be utilized. The Poisson method operates with two images A and B, where image B is smaller than image A and image B is to be merged into image A such that image B appears seamless and the texture of image B is preserved within the newly formed image.

[0077] In the particular context of FIGs 4A-4C, the library expansion module 124 performs the Poisson method by extracting the values for the first boundary pixels 406 and the pixels of the feature image 400 within the mask 404 into an operational format (e.g., a vector, matrix, array, etc.). To transpose the feature 402, the library expansion module 124 first identifies the second boundary pixels 412 defining where the feature 402 is to be transposed (which is used as the dQ in the Poisson method) The Poisson method then defines the area represented by the interior pixels 414 as Q. The texture of the feature image 400 (e.g., image B) is preserved by numerically computing the Laplacian (or gradient) of the feature image 400 (AB), then solving for the pixel intensity values given the boundary of pixels comprising area dQ. In the equations below, the original image 408 is image A and H represents the replacement values 423 for the pixels within the area Q that preserve the gradient of the feature image 400 while keeping the pixel values of area dQ unchanged For each pixel, an equation for computing the gradient is given by:

These equations, applied to every pixel in the area O, results in a large sparse matrix with n equations and n unknown values of H. The resulting matrix is sparse in nature such that numerical approximations of the solution generally converge to an intensity value to use as the replacement value 423 for the pixel The library expansion module 124 can then use these replacement values 423 for the pixels to transpose the feature 402 from the feature image 400 onto the original image 408 to generate the synthetic image 418. [0078] It should be appreciated that by transposing the feature 402 using the gradient of pixel values within the mask 404, the resulting representation of the feature in the synthetic image 418 generally maintains characteristics of the original image 408 (e.g., brightness, intensity, etc ) As a result, the feature 402 as represented in the synthetic image 418 blends in with the exterior pixels 416, thereby reducing artifacts from the image transposition process As a result, when the synthetic image 418 is used to train a visual inspection process, the training process does not associate features of the transposition artifacts as being the features indicative of the presence of the transposed defect. Therefore, the resulting classifiers are more accurate than some conventional approaches that transpose pixel intensity values from feature images.

[0079] Additional examples of the application of the gradient imaging techniques described herein are shown in FIGs. 5A through 6C. FIG. 5A shows an original image 500 of a defect free pharmaceutical container 502 overlaid with a mask 504 for a crack defect 506. The mask 504 may be generated using any of the techniques described herein. In some embodiments, the computer system 104 is configured to present a user interface via which a user is able to overlay the mask 504 at a desired location on the original image 500. For example, the user may be able to drag the representation of the mask 504 to move the mask 504 to the desired location. Additionally, the computer system 104 may be configured to automatically suggest locations for overlaying the mask 504. For example, the computer system 104 may perform a segmentation analysis of the original image 500 to detect a predetermined portion of the original image 500 at which the feature can manifest. In the scenario of FIG. 5A where the feature is a crack, the predetermined portion may correspond to a medicament chamber. In some embodiments, the computer system 104 may then randomly locate the mask 504 within the segment of the original image 500 associated with the medicament chamber In other embodiments, the computer system 104 may locate the mask 504 within the segment of the original image 500 in a scripted or strategic manner, such as by applying a spatial distribution across a 2D grid of a subset of the original image 500 when sequentially generating a plurality of synthetic images. Ensuring spatial distribution of defects in the training image library may help to eliminate spatial bias. Moreover, in the scenario of FIG. 5A, the medicament chamber portion of the original image 500 includes the image data representative of the air- to-water interface associated with the high contrast regions that in conventional image transposition techniques produce undesirable artifacts. The techniques described herein also provide improved feature transposition in other high contrast areas such as around the plunger on syringes and the rubber stopper on vials. Specifically, the gradient imaging techniques as described herein are especially suited to transpose features onto areas of original images where different materials converge (e.g. the material of the stopper relative to the contents of the vial or the material of a stopper and plunger relative to the contents of the vial and syringe). Thus, the scripted or strategic approach to generate synthetic images for inclusion in the training image library 140 improves the ability of AVI systems to classify features that overlap the air-to-water interface. It should be appreciated that in these embodiments, the computer system 104 may still enable the user to manually adjust the automatically defined position of the mask 504. Additionally, the user interface may enable the user to perform one or more modifications of a feature generated by the mask 504 prior to transposing the feature. For example, the user interface may enable the user to enlarge, shrink, rotate, invert, reflect, or perform other operations that modify the shape of the feature associated with the mask 504. Furthermore, in some embodiments, the computer system 104 may rotate, scale, stretch, and/or skew the feature based on the specific location within the original image selected for the mask 504 (e.g., a relative location within the portion of the original image 500). For example, when the portion is a curved cylindrical medicament chamber, the computer system 104 may rotate, scale, stretch, and/or skew the feature when being transferred to reflect the curvature of the medicament chamber In this example, a crack feature may be scaled horizontally depending on a horizontal (e g , radial) coordinate of the target location on the medicament chamber shown in the original image 500

[0080] Regardless of the particular technique, the library expansion module 124 may then implement the feature transposition technique described with respect to FIGs. 4A-4C to transpose the defect onto the original image 500. As a result, the library expansion module 124 produces a synthetic image 508 that includes the transposed crack defect 506 overlaid seamlessly onto the defect free pharmaceutical container 502 as shown in FIG. 5B. It should be appreciated that the gradient imaging techniques described herein also enable defects to be transferred between different types of objects or containers and different locations of objects (e.g., transfer from a location by meniscus to location further from meniscus) while still minimizing transposition artifacts. For example, the library expansion module 124 may use the feature image 600 in FIG. 6A to transfer a crack 602 in a syringe 604 to a vial 606 to form synthetic image 608 (FIG. 6B). As shown in FIG. 6B and FIG 6C the crack 602 is seamlessly transferred without introducing artifacts into the synthetic image. On the other hand, as shown in FIG. 6B, using traditional cut and paste methods produce synthetic images 608 that include a discordant background on the crack (transposed feature 602B). As a result, the transposed feature 602B is poorly integrated into the remainder of the vial 606 in FIG 6B.

[0081] As described above, without seamless transposition of the feature using the disclosed techniques, (for example, as shown in FIG. 6C), an AVI neural network might focus on the “wrong” characteristics (e.g., pixel-level artifacts) when determining that a synthetic image is defective. While the material (e.g., glass or plastic) of a container may appear to the naked eye as a homogenous surface, characteristics of the illumination and container material (e.g., container curvature) in fact cause pixel-to-pixel variations, and each surrogate area on a given container image differs in at least some respects from every other potential surrogate area. Moreover, differences between the conditions/materials (e.g., illumination and container material/shape) used when capturing the defect images, as compared to the conditions/materials used when capturing the “good” container images, can lead to even larger variations. The crack (transposed feature 602B) in FIG. 6B, transferred using traditional cut and paste methods, is potential example of this issue. Because the boundaries of the transposed feature 602B can clearly be seen, an AVI neural network trained using synthetic images with these kinds of transferred defects may simply look for a similar boundary when inspecting containers, for example, which might result in a large number of false negatives and/or other inaccuracies

[0082] Because the disclosed techniques rely on a gradient approach, how closely the mask aligns with the transposed features can have an impact on how well the transposed feature blends into the original image when generating the synthetic image. To this end, if the mask includes pixels in the interior region that do not correspond to the feature, those pixels are still included in the gradient calculations described herein. This effect may manifest as a blurry region proximate to the transposed feature in the synthetic image. This effect is especially pronounced in cases where the original image has a non-consistent background intensity (e.g., the mask is overlaid onto a region in the original image that includes areas with high-intensity backgrounds and low-intensity backgrounds) such that the contrasting values are included in the gradient calculation.

[0083] FIGs. 7A and 7B illustrate this artifact. Starting with FIG. 7A, depicted is a feature 700 with an oversized mask 702 (e.g., the mask 702 includes extraneous portions of the background and is not tightly defined or narrowly tailored close to the feature 700). When the oversized mask 702 is used in conjunction with the gradient imaging techniques described herein to transpose the feature 700 onto an original image with a non-consistent background, blurring artifacts 704 as shown in FIG. 7B can occur on the transposed feature 706 at locations where the background noticeably shifts in intensity. However, when the oversized mask 702 is replaced by a narrowly tailored or narrowly defined or tightly defined mask, the blurring artifacts are mitigated

[0084] A narrowly tailored or tightly defined mask may include a mask that is manually or automatically drawn adjacent to or within one, two, or three pixels of an edge of the actual feature or defect depicted in the feature image, such that no or very little background of the feature image is included and none of the feature or defect is excluded Where an edge of a feature occurs over multiple pixels (e g , four to five pixels), the edge can be defined as the midpoint of a transition in intensity A mask can be drawn to capture the maximum gradient for an edge, then add at least one pixel after. In the context of the feature image 400 as described above, the first boundary pixels 406 would comprise the outer perimeter of pixels one to three pixels’ distance from an edge of the feature or defect depicted. As such, where the pixel distance is one pixel, the first boundary pixels 406 are adjacent to or within a one pixel distance from the edge of the feature or defect depicted. And where the pixel distance is two pixels, the first boundary pixels 406 are within a two pixel distance from the edge of the feature or defect depicted. Where the pixel distance is three pixels, the first boundary pixels 406 are adjacent to or within a three pixel distance from the edge of the feature or defect depicted.

[0085] In some embodiments, the pixel distance between the first boundary pixels and the edge of the actual feature or defect for the mask may be non-uniform and depend on features of the original image to which the feature or defect is being added. For example, the pixel distance may be narrowly tailored or tightly defined {e.g., narrow or tight fitti n g_one or two pixels) for portions of the feature that will be transferred over high contrast areas of the original image and a wider distance {e.g., value greater than two pixels) for portions of the feature that will be transferred over uniform contrast areas of the original image.

[0086] FIGs. 8A-8C depict the transposition process for the same feature using a more tailored or defined mask 800 FIG. 8A depicts the mask 800 being narrowly defined around the perimeter of the feature 700 prior to performing the feature transposition. As can be seen in FIG. 8B, when the disclosed gradient-based transposition techniques are performed using the mask 800, the transposed feature 706 does not include the blurring artifacts 704 visible in FIG. 7B.

[0087] FIGs. 8D-8F show schematics of narrowly tailored masks with different pixel distances between the first boundary pixels 806 and the edge of a feature 802. In each figure, the mask is shown by a dotted line, the feature is showed by dotted hatching, and the boundary pixels are shown in diagonal hatching. FIG. 8D shows the most narrowly tailored mask, with a one pixel distance, where the first boundary pixels 806 are within a one pixel distance from the edge of the feature 802. FIG. 8E shows a mask with a two pixel distance, where the first boundary pixels 806 are within a two pixel distance from the edge of the feature 802. FIG. 8F shows a mask with a three pixel distance, where the first boundary pixels 806 are within a three pixel distance from the edge of the feature 802. [0088] Furthermore, a defect such as a crack transferred by the gradient imaging techniques disclosed herein can appear dark against a bright background of a glass container depending on the lighting. However, when a real crack passes into a meniscus area of a liquid, some of the crack can appear light colored. Thus, inverting the pixels of a defect can produce more realistic results when placed over a dark area, such as a meniscus. Furthermore, the techniques described herein are also advantageous for transferring features onto images with similar regions of contrasting brightness or onto anywhere that there is a sharp edge dividing two regions of contrasting brightness For example, the techniques described herein improve feature transfer onto areas that cross over a syringe plunger or vial stopper or that synthetically place a foreign object in an original image of an assembly line station (see, e.g., FIGS. 10A-11 B). On the other hand, it should be appreciated that if the defect is darker than the original surrounding area, the solution for some gray levels can become negative by the inversion. As image data cannot be associated with negative intensity values, the resulting negative values are instead set to zero. Thus, by inverting the portions of the transposed feature that would fall in the low- intensity regions of the original image prior to performing the gradient techniques disclosed herein, the resulting synthetic image looks even more realistic (e.g., further mitigates transposition artifacts). A low-intensity region can include a region that has an intensity that is about 50 units lower than a higher intensity area.

[0089] FIG. 8C depicts this further improvement. In particular, in FIG. 8C, prior to performing the gradient transposition techniques described herein, the portion of the feature that would be overlaid onto the low-intensity region (the portion 802) is inverted. As a result, the synthetic image more realistically depicts the feature 700

[0090] For at least the above reason related to mitigating transposition artifacts, it is beneficial to narrowly tailor the shape of the feature mask to the shape of the feature However, manually defining masks is tedious, time-consuming, vulnerable to human error and is impractical when attempting to transpose a significant number of features Instead, automated thresholding techniques or deep learning segmentation techniques may be implemented by the library expansion module 124 to expedite the mask definition process. For example, the “Segment Anything Method” (Meta Al, 2023, available at https://segment-anything com/) has been found to define narrowly-tailored masks without requiring any additional training, like other available deep learning methods. In some embodiments, the automated thresholding techniques are implemented by performing outlier detection on a portion of the feature image To this end, a feature often manifests in the image as a region with different intensity values than the background portion. Accordingly, the outlier detection techniques may be configured to automatically detect outliers from a reference value for the portion of the feature image. These outlier pixels may then be used to define a shape of the mask. In another example, deep learning models can be trained on features of interest to define masks. In another example, computer vision segmentations, such as Ostu's method or standard deviation methods can be used to automatically define masks.

[0091] FIGS. 9A-9C depict the improvement in mask accuracy when implementing the Segment Anything Method (SAM) process to define the mask. Starting with FIG 9A, depicted is a feature image 900 that includes a label 902 applied to a feature 904. The label 902 defines the general location of the feature 904 in the feature image 900, such as a bounding box, that is a common output of visual inspection systems. However, these bounding boxes do not provide the closely defined mask useful for the gradient imaging techniques as described herein. FIG. 9B depicts the output of a trained model employing the SAM process when the image data within the label 902 is input into the model. In particular, FIG. 9B depicts a mask 906 that is closely aligned with or narrowly tailored to the shape of the feature 904. The mask 906 may then be saved in the feature image library 142 for use by the library expansion module 124 in generating synthetic images. Fig. 9C depicts a synthetic image 908 that was generated by transposing the feature 904 using the mask 906 As one can see in FIG. 9C, the feature 904 has been transposed with minimal artifacts, resulting in a synthetic image that is well-suited for training an AVI model.

[0092] It should be appreciated that other techniques for automating generation of the mask 906 (and the masks 404, 504, 800, etc.) are possible For example, in some embodiments, the computer system 104 may automatically generate the masks described here using a mask generating machine learning model. The mask generating machine learning model may be configured to receive a feature image (e.g., feature images 400, 600, 900, etc.) as an input, identify the feature (e g., features 402, crack 602, feature 700, feature 904, etc.) within the feature image, and output data that defines a narrowly tailored mask around the feature. The mask generating machine learning model may comprise a set of trained parameter values (weights, biases, etc.) arranged in a suitable architecture (e.g., convolutional neural network, transformer, recurrent neural network, etc.) and trained or tuned to generate the mask defining output data based on training feature images with known defined feature masks. For example, the training feature images may include sets of images where one or more users have manually defined narrowly tailored masks as described herein Furthermore, the training feature images may be limited to a set of images for which the known defined masks produced satisfactory synthetic images (e.g., where a user inspected and approved the synthetic image generated using the known defined mask). For example, synthetic images that have minimal or no visible artifacts (e.g., blurring artifacts 704) may be marked as satisfactory.

[0093] Additionally or alternatively, in some embodiments, the parameter values of the mask generating machine learning model may be trained and/or tuned using reinforcement learning techniques. In these embodiments, the parameter values of the model may be updated based on user assessment or feedback on synthetic images (e.g., user approval similar to synthetic images generated using masks defined from the training feature images) produced using masks defined by the model. It should be appreciated that the reinforcement learning techniques may be used in addition or as an alternative to training/tuning using the training feature images [0094] In some embodiments, the computer system 104 may generate any of the masks described herein (e g , the masks 404, 504, 800, 906 etc ) using an interactive user process For example, the computer system 104 or another similar computing system may present one or more feature images on a display device and a user may identify the features and define the masks around those features within the feature image through user input with the computer system 104 or other similar computing system. In particular the user may “paint’ the mask over and/or around the feature so that the mask is tightly defined or narrowly tailored around the feature as described herein. In some embodiments, the computer system 104 or similar computing system may automatically generate a suggested mask using one or more of the automated mask generating techniques described above (e g., using the SAM process and/or the mask generating machine learning model) and the user may modify and/or approve the suggested mask for storage in the training image library 140 and future use in generating synthetic images as described herein In some embodiments, the user may modify the suggested mask by dilating or eroding portions of the mask around the feature so that the suggested mask is tightly defined or narrowly tailored around the feature as described herein.

[0095] According to aspects, the disclosed gradient transfer techniques can be applied to transpose defects other than those that appear on containers. For example, the disclosed gradient transfer techniques can be used to train AVI models that detect errors, anomalies or non-conformities in monitored manufacturing processes. Because the gradient imaging techniques as described herein perform especially well on highly contrasting or non-uniform backgrounds, the disclosed gradient imaging techniques may be used to generate synthetic images for training of deep learning anomaly detection models associated with line clearance tasks in packaging and/or other machinery. In this use case, a line clearance camera is configured to detect if a pharmaceutical container (e.g., a syringe, cartridge, vial, container, etc.) has fallen off the process conveyance system, if glass or other material has broken off the conveyance system or a container conveyed thereby, whether a component of the conveyance system (e.g , a fastener) has become dislodged and/or fallen off, etc. Often, the conveyance system is compliant with Good Manufacturing Practice (GMP) standards and it is not safe or advisable to place stray objects physically in the machinery to generate training images. Thus, the gradient imaging techniques can be used to generate the training images and enable verification that the deep learning models properly identify obstructions in the conveyance system.

[0096] Figures 10A, 10B, 11A, and 11 B depict examples of generating training data for these systems. Starting with FIG. 10A, FIG. 10A shows an original image 1000 of a portion of a packaging or manufacturing machine. As illustrated, the containers being conveyed through the packaging or manufacturing machine are in expected locations. Turning to FIG. 10B, the library expansion module 124 utilized the disclosed gradient imaging techniques to generate a synthetic image 1002 that overlays a feature 1004 (e g., a syringe) on the original image 1000. In particular, the library expansion module 124 may overlay the feature 1004 at a position at which the syringe should not be located such that the synthetic image 1002 may be utilized to train a visual inspection model (such as a visual inspection model configured to monitor line clearance) to detect this condition. Similarly, FIGs. 11 A and 11 B show additional example synthetic images 1100 and 1102, respectively, that include the feature 1004 overlaid on the original image at a position indicative of anomalous condition.

[0097] The library expansion module 124 can repeat the gradient imaging techniques for any number of different “good" images and any number of “defect” images, in any desired combination (e.g., applying each of L defect images to each of M good container images in each of A/ locations, to generate L x Mx A/ synthetic images based on Mgood container images in the training image library 140). Thus, for example, 10 defect images, 1,000 good container images, and 10 defect locations per defect type can result in 100,000 defect images. The locations/positions on which defects are transposed for any particular good container image may be predetermined, or may be randomly determined (e.g., by library expansion module 124). In this way, a small number of defects can be used to generate a large number of synthetic images with defects, even if the number of “good” images is limited

[0098] In some implementations, the defects and/or other features depicted in images of feature image library 142 can be morphed in one or more ways prior to library expansion module 124 adding those features to an original image (see e g , the differently rotated syringe features 1004 in FIGs. 11 A and 11 B). In this manner, library expansion module 124 can effectively increase the size and variability of feature image library 142, and thus increase the size and variability of training image library 140. For example, library expansion module 124 may morph defects and/or other features by applying rotations, scaling/stretching (in one or two dimensions), skewing, and/or other transformations. Additionally or alternatively, depicted features may be modified in more complex and/or subtle ways For example, library expansion module 124 may fit a defect (e.g , a crack) to different arcs, or to more complex crack structures (e.g., to each of a number of different branching patterns). Furthermore, in some cases, the library expansion module 124 can generate additional synthetic images using alternative techniques. In such cases, the combination of synthetic images formed using different imaging techniques can form a more robust set of training images for the AVI neural network module 120 Specifically, the use of different imaging techniques can increase the chance that the trained model detects features of the actual defect rather than artifacts left behind in the images from the transferring processes employed.

[0099] In some implementations, processing power constraints may limit the size of the images to be augmented (e.g., images of roughly 512x512 pixels or smaller), which can in turn make it necessary to crop images prior to augmentation, and then re-insert the augmented image crop. This takes extra time, and can have other undesired consequences (e.g., for the deep learning-based inpainting techniques, failing to achieve the benefits of adding slight noise/variation to the entire image rather than just the smaller/cropped portion). In some implementations, library expansion module 124 addresses this by using a ResNet feature extractor rather than a VGG feature extractor. Feature extractors such as these are used to calculate the losses that are used to tune the weights of the inpainting model during training. The library expansion module 124 may use any suitable version of a ResNet feature extractor (e.g., ResNet50, ResNet101, ResNet152, etc.), depending on the image dimensions and the desired training speed. [00100] Moreover, in some implementations, library expansion module 124 may apply post-processing to synthetic images in order to reduce undesired artifacts. For example, library expansion module 124 may add noise to each synthetic image, perform filtering/smoothing on each synthetic image, and/or perform Fast Fourier Transform (FFT) frequency spectrum analysis and manipulation on each synthetic image. Such techniques may help to mitigate any artifacts, and generally make the images more realistic As another example, library expansion module 124 may pass each synthetic image through a refiner, where the refiner was trained by pairing the refiner with a discriminator. During training, both the refiner and the discriminator are fed synthetic and real images (e g., by library expansion module 124). The goal of the discriminator is to discriminate between a real and synthetic image, while the goal of the refiner is to refine the synthetic image to a point where the discriminator can no longer distinguish the synthetic image from a real image. The refiner and discriminator are thus adversaries of each other, and work in a manner similar to a generative adversarial network (GAN). After multiple cycles of training, the refiner can become very adept at refining images, and therefore library expansion module 124 can use the trained refiner to remove artifacts from synthetic images that are to be added to the training image library 140. Any of the techniques described above can also be used to process/refine synthetic images that were generated without deep learning techniques, such as synthetic images generated using the gradient imaging techniques discussed above.

[00101] To ensure proper training of the AVI model, it is prudent to include quality control measures at one or more stages. This can be particularly important in the pharmaceutical context, where it is necessary to protect patient safety by ensuring a safe and reliable drug product. In some implementations, both “pre-processing” and “post-processing” quality checks are performed (e g., by image/library assessment module 126) Generally, these pre- and post-processing quality checks may leverage various image processing techniques to analyze and/or compare information on a per-pixel basis

[00102] In some implementations, the computer system 104 can determine one or more image quality metrics (e g , to determine a similarity between a given synthetic image and other images, or to measure diversity of an image set, etc ) These quality metrics can be useful when training the AVI neural network module 120. For example, knowing variability can be useful for two reasons. First, variability in certain attributes (e g., plunger position) can overwhelm the signal from the actual defect and thus lead to misclassifications, as the neural network might weigh the variable attribute more heavily than the defect itself. Second, for the purpose of image augmentation, it can be useful to know the range of variability in given attributes, in order to constrain those attributes to that range when creating population-representative synthetic images.

[00103] FIG 12 shows an example of a method 1200 for generating a synthetic image (such as the synthetic images 418, 608, 908, 1002, 1100, and 1102) by transferring a feature (such as the features 402, 602, 700, and 904) of a first image (e.g., a feature image such as feature image 400, 600, or 900) onto a second image (e.g., an original image such as original images 408, and 1000) may be executed by library expansion module 124 of FIG. 1 (e.g., when processing unit 110 executes instructions of library expansion module 124 stored in memory unit 114), for example. In some embodiments, the second image is an image of a container (such as the containers 300, 320, 340, 502, and 606) and the feature is a defect associated with the container or contents of the container. When the container is a syringe, the feature may be a defect associated with a barrel of the syringe, a plunger of the syringe, a needle shield of the syringe, or a fluid within the syringe. When the container is a vial, the feature may be a defect associated with a wall of the vial, a cap of the vial, a crimp of the vial, or a fluid or lyophilized cake within the vial. In other embodiments, the feature is a container or object and the second image is an image of a portion of a packing or manufacturing machine for the container or object (such as machinery included in the conveying system described with respect to FIGS. 10A.B and 11A,B).

[00104] At block 1210, the library expansion module 124 receives or generates a mask (e.g., the masks 404, 504, 800, 906) that defines a shape of a feature depicted in the first image. The mask includes first boundary pixels (e.g., the first boundary pixels 406) that define an outer perimeter of the feature. The method 1200 may include inputting the first image into a trained segmentation neural network to automatically generate the mask. The method 1200 may also include identifying outlier pixel values in an image area of the first image and defining the mask based on the outlier pixel values The first boundary pixels of the mask may be adjacent to or within one pixel, two, or three pixels distance of an edge of the feature as depicted in the first image. The first boundary pixels of the mask may also be a non-uniform distance from an edge of the feature as depicted in the first image. In some embodiments, the method 1200 may include inputting the first image into a mask generation machine learning model to automatically generate the mask. The mask generation machine learning model may be trained using at least one of training feature images having known defined masks and reinforcement learning with respect to training synthetic images.

[00105] At block 1220, the library expansion module 124 identifies a transfer area (e g., the transfer area 410 or an area associated with a feature of the container 502) in the second image to which the feature will be transferred.

[00106] At block 1230, the library expansion module 124 overlays the mask on the transfer area to align the first boundary pixels with second boundary pixels (e.g., second boundary pixels 412) of the second image that define an outer perimeter of the transfer area. In some embodiments, identifying the transfer area in the second image includes identifying a surrogate feature of the second image on which the feature is to be overlaid and defining the transfer area to be randomly located within the surrogate feature. Additionally, in some embodiments, the library expansion module 124 may repeat the method 1200 to generate a plurality of synthetic images by transferring the feature to a plurality of different transfer areas within the second image. In these embodiments, the plurality of different transfer areas are evenly or randomly distributed across a portion of the second image. The portion of the second image may be a pre-determined area corresponding to a portion at which the feature can manifest. Furthermore, the library expansion module 124 may rotate, scale, stretch, and/or skew the feature placed in the plurality of different transfer areas based on a relative location of the different transfer areas within the portion of the image

[00107] At block 1240, the library expansion module 124 defines respective gradient values for pixels of the first image within the first boundary pixels of the mask. In some embodiments, the respective gradient values defined for the pixels of the first image within the first boundary pixels of the mask incorporate differences in intensity between each pixel and neighboring pixels thereof. Defining the respective gradient values may include determining that the transfer area includes a sub-portion associated with low-intensity values, inverting pixels of the first image within the mask that are overlaid on pixels of the sub-portion of the transfer area, and defining the respective gradient values based on the inverted pixels. Determining that the transfer area includes a sub-portion that is associated with low-intensity values may include comparing values for the second boundary pixels across different sections of the transfer area. In some embodiments, defining respective gradient values for pixels of the first image within the first boundary pixels of the mask uses a Poisson imaging technique.

[00108] At block 1250, the library expansion module 124, generates a set of replacement values for pixels of the transfer area based on the second boundary pixels and the respective gradient values for the pixels of the first image within the first boundary pixels of the mask. Furthermore, in some embodiments, generating the set of replacement values based on the second boundary pixels and the respective gradient values includes generating interior replacement values that preserve the respective gradient values in relation to a set of boundary values, the set of boundary values being values of the second boundary pixels and combining the interior replacement values and the set of boundary values into the set of replacement values.

[00109] At block 1260, the library expansion module 124 generates the synthetic image (e.g., the synthetic image 418) by replacing pixel values of the transfer area with the set of replacement values. The synthetic images may be stored in a training image library (such as the training image library 140). [00110] The library expansion module 124 may repeat the method 1200 for each of a plurality of features corresponding to different features in a feature library (such as the feature image library 142). Similarly, the library expansion module 124 may repeat the method 1200 for each of a plurality of second images.

[00111] The method 1200 may also include additional steps in relation to training an AVI neural network (such a neural network associated withs the AVI neural network module 120) or non-neural network-based machine vision model. The method 1200 may also include training a neural network for automated visual inspection using the plurality of synthetic images and the plurality of second images (and/or any other images included in the training image library, e.g., synthetic images generated using alternative techniques). After the neural network is trained, the method 1200 may include inspecting a plurality of images for depicted features using the trained neural network.

[00112] It is understood that the blocks of the method 1200 need not occur strictly in the order shown.

[00113] FIG 13 shows an example of a method 1300 for automating generation of masks and transferring features to generate a library of synthetic images. In method 1300, a library of synthetic images (such as the synthetic images 418, 608, 908, 1002, 1100, and 1102) may be generated by transferring features (such as the features 402, 602, 700, and 904) of a first image from a first image library (e.g., a feature image such as feature image 400, 600, or 900) onto an second image from a second image library (e.g., an original image such as original images 408, and 1000). This method 1300 may be executed by library expansion module 124 of FIG. 1 (e.g., when processing unit 110 executes instructions of library expansion module 124 stored in memory unit 114), for example.

[00114] At block 1310, the library expansion module 124 prepares a first image library, where each image of the first image library includes a feature.

[00115] At block 1320, the library expansion module 124 prepares a library of feature masks (a mask library), each mask defining a shape of a feature depicted in an image from the first image library and including a first boundary of pixels that define an outer perimeter of the feature The feature masks can be prepared using a segmentation algorithm, such as a SAM or other deep learning model. A segmentation model can be trained specifically on features of the features in the first library. Additionally or alternatively, the feature masks can be prepared using computer vision methods of segmenting, such as Otsu’s method or standard deviation methods. After automatic segmentation, the masks can optionally be inspected to inspect the masks to confirm that the masks cover the feature but do not include extraneous portions of background

[00116] At block 1330, the library expansion module 124 prepares a second image library of images onto which the features are to be transferred. The second image library can include images without features.

[00117] At block 1340, the library expansion module 124 defines transfer areas in images from the second library at random locations.

[00118] At block 1350, the library expansion module 124 iterates through the images from the second image library and the first image library to transfer at least one feature from an image from the first image library to a random location within the transfer area of each image from the second image library. For each pair of images from the first image library and the second image library, the library expansion module 124 overlays the mask for the image from the first image library on the transfer area to align the first boundary pixels of the mask with second boundary pixels (e.g., second boundary pixels 412) of the image from the second image library that define an outer perimeter of the transfer area. In some embodiments, identifying the transfer area in the image from the second image library includes identifying a surrogate feature of the image from the second image library on which the feature is to be overlaid and defining the transfer area to be randomly located within the surrogate feature Additionally, in some embodiments, the library expansion module 124 may iterate through different pairs of images from the first image library and second image library generate a plurality of synthetic images. In this way a plurality of synthetic images can be generated for each image of the second image library, each synthetic image including a different feature from the first image library.

[00119] At block 1360, the library expansion module 124 transforms the transferred feature by mirroring, rotating, skewing, or adjusting brightness to generate a synthetic image. The library expansion model can then repeat the step at block 1350 to continue iterating through images from the first library and second library and transferring features to a plurality of different transfer areas within each image of the second image library to generate a plurality of synthetic images. In this way, a plurality of synthetic images can be generated for each pair of images from the first image library and the second image library, each synthetic image including a feature at a different transfer area on the image from the second image library.

[00120] At block 1370, the library expansion module 124 repeats the step at block 1360 to transform transferred features in different ways until a desired number of synthetic images are produced. By repeating the steps at blocks 1350 and 1360, a small number of features (e g., a small first image library) can be used to generate a large number of synthetic images with features, even if the number of “good” images without features is limited (e.g., a small second image library).

[00121] Provided herein as Embodiment 1 is a method of generating a synthetic image, the method comprising: receiving or generating a mask that defines a shape of a feature depicted in a first image, wherein the mask includes first boundary pixels that define an outer perimeter of the feature; identifying a transfer area in a second image to which the feature will be transferred; overlaying the mask on the transfer area to align the first boundary pixels with second boundary pixels of the second image that define an outer perimeter of the transfer area; defining respective gradient values for pixels of the first image within the first boundary pixels of the mask; generating a set of replacement values for pixels of the transfer area based on the second boundary pixels and the respective gradient values for the pixels of the first image within the first boundary pixels of the mask; and generating the synthetic image by replacing pixel values of the transfer area with the set of replacement values

[00122] Provided herein as Embodiment 2 is the method of Embodiment 1 , wherein the respective gradient values defined for the pixels of the first image within the first boundary pixels of the mask incorporate differences in intensity between each pixel and neighboring pixels thereof.

[00123] Provided herein as Embodiment 3 is the method of any one of Embodiments 1-2, wherein generating the set of replacement values based on the second boundary pixels and the respective gradient values includes: generating interior replacement values that preserve the respective gradient values in relation to a set of boundary values, the set of boundary values being values of the second boundary pixels; and combining the interior replacement values and the set of boundary values into the set of replacement values.

[00124] Provided herein as Embodiment 4 is the method of any one of Embodiments 1-3, wherein defining the respective gradient values comprises: determining that the transfer area includes a sub-portion associated with low-intensity values; inverting pixels of the first image within the mask that are overlaid on pixels of the sub-portion of the transfer area; and defining the respective gradient values based on the inverted pixels.

[00125] Provided herein as Embodiment 5 is the method of Embodiment 4, wherein determining that the transfer area includes a sub-portion that is associated with low-intensity values comprises: comparing values for the second boundary pixels across different sections of the transfer area.

[00126] Provided herein as Embodiment 6 is the method of any one of Embodiments 1-5, wherein defining respective gradient values for pixels of the first image within the first boundary pixels of the mask uses a Poisson imaging technique.

[00127] Provided herein as Embodiment 7 is the method of any one of Embodiments 1-6, wherein the first boundary pixels of the mask are narrowly tailored around the perimeter the feature. [00128] Provided herein as Embodiment 8 is the method of any one of Embodiments 1-7, wherein the first boundary pixels of the mask are adjacent to or within a three pixel distance of an edge of the feature as depicted in the first image. Provided in other embodiments of Embodiment 8 is the method of any one of Embodiments 1-7, wherein the first boundary pixels of the mask are adjacent to or within a two pixel distance of an edge of the feature as depicted in the first image. Provided in other embodiments of Embodiment 8 is the method of any one of Embodiments 1-7, wherein the first boundary pixels of the mask are adjacent to or within a one pixel distance of an edge of the feature as depicted in the first image

[00129] Provided herein as Embodiment 9 is the method of any one of Embodiments 1-8, further comprising inputting the first image into a trained segmentation neural network to automatically generate the mask.

[00130] Provided herein as Embodiment 10 is the method of any of Embodiments 1-9, further comprising identifying outlier pixel values in an image area of the first image and defining the mask based on the outlier pixel values.

[00131] Provided herein as Embodiment 11 is the method of any one of Embodiments 1-10, further comprising inputting the first image into a mask generation machine learning model to automatically generate the mask.

[00132] Provided herein as Embodiment 12 is the method of Embodiment 11, wherein the mask generation machine learning model is trained using at least one of training feature images having known defined masks and reinforcement learning with respect to training synthetic images.

[00133] Provided herein as Embodiment 13 is the method of any one of Embodiments 1-12, wherein the first boundary pixels of the mask are a non-uniform distance from an edge of the feature as depicted in the first image.

[00134] Provided herein as Embodiment 14 is the method of any one of Embodiments 1-13, wherein: the second image is an image of a container; and the feature is a defect associated with the container or contents of the container

[00135] Provided herein as Embodiment 15 is the method of Embodiment 14, wherein: the container is a syringe; and the feature is a defect associated with a barrel of the syringe, a plunger of the syringe, a needle shield of the syringe, or a fluid within the syringe [00136] Provided herein as Embodiment 16 is the method of Embodiment 14, wherein: the container is a vial; and the feature is a defect associated with a wall of the vial, a cap of the vial, a crimp of the vial, or a fluid or lyophilized cake within the vial.

[00137] Provided herein as Embodiment 17 is the method of any one of Embodiments 1-16, wherein: the feature is a container or object; and the second image is an image of a portion of a packing or manufacturing machine for the container or object.

[00138] Provided herein as Embodiment 18 is the method of any one of Embodiments 1-17, wherein identifying the transfer area in the second image comprises: identifying a surrogate feature of the second image on which the feature is to be overlaid; and defining the transfer area to be randomly located within the surrogate feature.

[00139] Provided herein as Embodiment 19 is the method of any one of Embodiments 1-18, further comprising: repeating the method for each of a plurality of features corresponding to different features in a feature library to generate a plurality synthetic images for each of the plurality of features.

[00140] Provided herein as Embodiment 20 is the method of any one of Embodiments 1-19, further comprising: generating a plurality of synthetic images by repeating the method for each of a plurality of second images.

[00141] Provided herein as Embodiment 21 is the method of Embodiment 20, further comprising: training a neural network for automated visual inspection using the plurality of synthetic images and the plurality of second images.

[00142] Provided herein as Embodiment 22 is the method of Embodiment 21, further comprising: inspecting a plurality of images for depicted features using the trained neural network [00143] Provided herein as Embodiment 23 is the method of any one of Embodiments 1-22, further comprising: repeating the method to generate a plurality of synthetic images by transferring the feature to a plurality of different transfer areas within the second image.

[00144] Provided herein as Embodiment 24 is the method of Embodiment 23, wherein the plurality of different transfer areas are randomly distributed across a portion of the second image.

[00145] Provided herein as Embodiment 25 is the method of Embodiment 24, wherein the portion of the second image is a predetermined area corresponding to a portion at which the feature can manifest.

[00146] Provided herein as Embodiment 26 is the method of any one of Embodiments 23-25, further comprising: rotating, scaling, stretching, and/or, skewing the feature placed in the plurality of different transfer areas based on a relative location of the different transfer areas within the portion of the second image.

[00147] Provided herein as Embodiment 27 is a system comprising: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the system to: receive or generate a mask that defines a shape of a feature depicted in a first image, wherein the mask includes first boundary pixels that define an outer perimeter of the feature; identify a transfer area in a second image to which the feature will be transferred; overlay the mask on the transfer area to align the first boundary pixels with second boundary pixels of the second image that define an outer perimeter of the transfer area; define respective gradient values for pixels of the first image within the first boundary pixels of the mask; generate a set of replacement values for pixels of the transfer area based on the second boundary pixels and the respective gradient values for the pixels of the first image within the first boundary pixels of the mask; and generate a synthetic image by replacing pixel values of the transfer area with the set of replacement values

[00148] Provided herein as Embodiment 28 is the system of Embodiment 27, wherein the respective gradient values defined for the pixels of the first image within the first boundary pixels of the mask incorporate differences in intensity between each pixel and neighboring pixels thereof.

[00149] Provided herein as Embodiment 29 is the system of any one of Embodiments 27-28, wherein the set of replacement values are generated based on the second boundary pixels and the respective gradient values by: generating interior replacement values that preserve the respective gradient values in relation to a set of boundary values, the set of boundary values being values of the second boundary pixels; and combining the interior replacement values and the set of boundary values into the set of replacement values.

[00150] Provided herein as Embodiment 30 is the system of any one of Embodiments 27-29, wherein the respective gradient values are defined by: determining that the transfer area includes a sub-portion associated with low-intensity values; inverting pixels of the first image within the mask that are overlaid on pixels of the sub-portion of the transfer area; and defining the respective gradient values based on the inverted pixels.

[00151] Provided herein as Embodiment 31 is the system of Embodiment 30 wherein determining that the transfer area includes a sub-portion that is associated with low-intensity values comprises: comparing values for the second boundary pixels across different sections of the transfer area.

[00152] Provided herein as Embodiment 32 is the system of any one of Embodiments 27-31, wherein respective gradient values for pixels of the first image within the first boundary pixels of the mask are defined using a Poisson imaging techniques.

[00153] Provided herein as Embodiment 33 is the system of any one of Embodiments 27-32, wherein the first boundary pixels of the mask are narrowly tailored around the perimeter of the feature. [00154] Provided herein as Embodiment 34 is the system of any of Embodiments 27-33, wherein the first boundary pixels of the mask are adjacent to or within a three pixel distance of an edge of the feature as depicted in the first image. Provided in other embodiments, is the system of any one of Embodiments 27-33, wherein the first boundary pixels of the mask are adjacent to or within a two pixel distance of an edge of the feature as depicted in the first image. Provided in other embodiments, is the system of any one of Embodiments 27-33, wherein the first boundary pixels of the mask are adjacent to or within a one pixel distance of an edge of the feature as depicted in the first image

[00155] Provided herein as Embodiment 35 is the system of any one of Embodiments 27-34, wherein the instructions, when executed by the one or more processors, further cause the system to input the first image into a trained segmentation neural network to automatically generate the mask.

[00156] Provided herein as Embodiment 36 is the system of any of Embodiments 27-35, wherein the instructions, when executed by the one or more processors, further cause the system to identify outlier pixel values in an image area of the first image and defining the mask based on the outlier pixel values.

[00157] Provided herein as Embodiment 37 is the system of any of Embodiments 27-39, wherein the instructions, when executed by the one or more processors, further cause the system to the first image into a mask generation machine learning model to automatically generate the mask.

[00158] Provided herein as Embodiment 38 is the system of Embodiment 37, wherein the mask generation machine learning model is trained using at least one of training feature images having known defined masks and reinforcement learning with respect to training synthetic images.

[00159] Provided herein as Embodiment 39 is the system of any of Embodiments 27-38, wherein the first boundary pixels of the mask are a non-uniform distance from an edge of the feature as depicted in the first image

[00160] Provided herein as Embodiment 40 is the system of any one of Embodiments 27-39, wherein: the second image is an image of a container; and the feature is a defect associated with the container or contents of the container.

[00161] Provided herein as Embodiment 41 is the system of Embodiment 40, wherein: the container is a syringe; and the feature is a defect associated with a barrel of the syringe, a plunger of the syringe, a needle shield of the syringe, or a fluid within the syringe. [00162] Provided herein as Embodiment 42 is the system of Embodiment 40, wherein: the container is a vial; and the feature is a defect associated with a wall of the vial, a cap of the vial, a crimp of the vial, or a fluid or lyophilized cake within the vial.

[00163] Provided herein as Embodiment 43 is the system of any one of Embodiment 27-41, wherein: the feature is a container or object; and the second image is an image of a portion of a packing or manufacturing machine for the container or object.

[00164] Provided herein as Embodiment 44 is the system of any one of Embodiments 27-43, wherein the transfer area is identified in in the second image by: identifying a surrogate feature of the second image on which the feature is to be overlaid; and defining the transfer area to be randomly located within the surrogate feature.

[00165] Provided herein as Embodiment 45 is the system of any one of Embodiments 27-44, wherein the instructions, when executed by the one or more processors, further cause the system to generate a plurality of synthetic images by repeatedly transferring the feature to a plurality of different transfer areas within the second image.

[00166] Provided herein as Embodiment 46 is the system of Embodiment 45, wherein the plurality of different transfer areas are randomly distributed across a portion of the second image.

[00167] Provided herein as Embodiment 47 is the system of Embodiment 46, wherein the portion of the second image is a predetermined area corresponding to a portion at which the feature can manifest. [00168] Provided herein as Embodiment 48 is the system of any one of Embodiments 45-47, wherein the instructions, when executed by the one or more processors, further cause the system to rotate, scale, stretch, and/or skew the feature placed in the plurality of different transfer areas based on a relative location of the different transfer areas within the portion of the second image. [00169] Any of the embodiments disclosed herein may be properly combined with any other embodiment disclosed herein. The combination of any one the embodiments disclosed herein with any other embodiments is expressly contemplated.

[00170] Although the systems, methods, devices, and components thereof, have been described in terms of exemplary embodiments, they are not limited thereto. The detailed description is to be construed as exemplary only and does not describe every possible embodiment of the invention because describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent that would still fall within the scope of the claims defining the invention.

[00171] Those skilled in the art will recognize that a wide variety of modifications, alterations, and combinations can be made with respect to the above described embodiments without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.

Claims

What is claimed is:

1 . A method of generating a synthetic image, the method comprising: receiving or generating a mask that defines a shape of a feature depicted in a first image, wherein the mask includes first boundary pixels that define an outer perimeter of the feature; identifying a transfer area in a second image to which the feature will be transferred; overlaying the mask on the transfer area to align the first boundary pixels with second boundary pixels of the second image that define an outer perimeter of the transfer area; defining respective gradient values for pixels of the first image within the first boundary pixels of the mask; generating a set of replacement values for pixels of the transfer area based on the second boundary pixels and the respective gradient values for the pixels of the first image within the first boundary pixels of the mask; and generating the synthetic image by replacing pixel values of the transfer area with the set of replacement values.

2. The method of claim 1 , wherein the respective gradient values defined for the pixels of the first image within the first boundary pixels of the mask incorporate differences in intensity between each pixel and neighboring pixels thereof.

3. The method of any one of claims 1-2, wherein generating the set of replacement values based on the second boundary pixels and the respective gradient values includes: generating interior replacement values that preserve the respective gradient values in relation to a set of boundary values, the set of boundary values being values of the second boundary pixels; and combining the interior replacement values and the set of boundary values into the set of replacement values

4. The method of any one of claims 1-3, wherein defining the respective gradient values comprises: determining that the transfer area includes a sub-portion associated with low-intensity values; inverting pixels of the first image within the mask that are overlaid on pixels of the sub-portion of the transfer area; and defining the respective gradient values based on the inverted pixels.

5. The method of claim 4, wherein determining that the transfer area includes a sub-portion that is associated with low-intensity values comprises: comparing values for the second boundary pixels across different sections of the transfer area.

6. The method of any one of claims 1-5, wherein defining respective gradient values for pixels of the first image within the first boundary pixels of the mask uses a Poisson imaging technique

7. The method of any one of claims 1-6, wherein the first boundary pixels of the mask are narrowly tailored around the perimeter the feature.

8. The method of any one of claims 1-7, wherein the first boundary pixels of the mask are adjacent to or within a three pixel distance of an edge of the feature as depicted in the first image.

9. The method of any one of claims 1-8, further comprising inputting the first image into a trained segmentation neural network to automatically generate the mask.

10 The method of any of claims 1-9, further comprising identifying outlier pixel values in an image area of the first image and defining the mask based on the outlier pixel values.

11 The method of any one of claims 1-10, further comprising inputting the first image into a mask generation machine learning model to automatically generate the mask.

12 The method of claim 11, wherein the mask generation machine learning model is trained using at least one of training feature images having known defined masks and reinforcement learning with respect to training synthetic images

13 The method of any one of claims 1-12, wherein the first boundary pixels of the mask are a non-uniform distance from an edge of the feature as depicted in the first image.

14 The method of any one of claims 1-13, wherein: the second image is an image of a container; and the feature is a defect associated with the container or contents of the container

15 The method of claim 14, wherein: the container is a syringe; and the feature is a defect associated with a barrel of the syringe, a plunger of the syringe, a needle shield of the syringe, or a fluid within the syringe.

16 The method of claim 14, wherein: the container is a vial; and the feature is a defect associated with a wall of the vial, a cap of the vial, a crimp of the vial, or a fluid or lyophilized cake within the vial.

17 The method of any one of claims 1-16, wherein: the feature is a container or object; and the second image is an image of a portion of a packing or manufacturing machine for the container or object.

18 The method of any one of claims 1-17, wherein identifying the transfer area in the second image comprises: identifying a surrogate feature of the second image on which the feature is to be overlaid; and defining the transfer area to be randomly located within the surrogate feature.

19 The method of any one of claims 1-18, further comprising: repeating the method for each of a plurality of features corresponding to different features in a feature library to generate a plurality synthetic images for each of the plurality of features.

20 The method of any one of claims 1-19, further comprising: generating a plurality of synthetic images by repeating the method for each of a plurality of second images.

21 The method of claim 20, further comprising: training a neural network for automated visual inspection using the plurality of synthetic images and the plurality of second images.

22 The method of claim 21, further comprising: inspecting a plurality of images for depicted features using the trained neural network.

23 The method of any one of claims 1-22, further comprising: repeating the method to generate a plurality of synthetic images by transferring the feature to a plurality of different transfer areas within the second image.

24 The method of claim 23, wherein the plurality of different transfer areas are randomly distributed across a portion of the second image

25 The method of claim 24, wherein the portion of the second image is a pre-determined area corresponding to a portion at which the feature can manifest.

26 The method of any one of claims 23-25, further comprising: rotating, scaling, stretching, and/or skewing the feature placed in the plurality of different transfer areas based on a relative location of the different transfer areas within the portion of the second image.

27 A system comprising: one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the system to: receive or generate a mask that defines a shape of a feature depicted in a first image, wherein the mask includes first boundary pixels that define an outer perimeter of the feature; identify a transfer area in a second image to which the feature will be transferred; overlay the mask on the transfer area to align the first boundary pixels with second boundary pixels of the second image that define an outer perimeter of the transfer area; define respective gradient values for pixels of the first image within the first boundary pixels of the mask; generate a set of replacement values for pixels of the transfer area based on the second boundary pixels and the respective gradient values for the pixels of the first image within the first boundary pixels of the mask; and generate a synthetic image by replacing pixel values of the transfer area with the set of replacement values.

28 The system of claim 27, wherein the respective gradient values defined for the pixels of the first image within the first boundary pixels of the mask incorporate differences in intensity between each pixel and neighboring pixels thereof.

29 The system of any one of claims 27-28, wherein the set of replacement values are generated based on the second boundary pixels and the respective gradient values by: generating interior replacement values that preserve the respective gradient values in relation to a set of boundary values, the set of boundary values being values of the second boundary pixels; and combining the interior replacement values and the set of boundary values into the set of replacement values.

30 The system of any one of claims 27-29, wherein the respective gradient values are defined by: determining that the transfer area includes a sub-portion associated with low-intensity values; inverting pixels of the first image within the mask that are overlaid on pixels of the sub-portion of the transfer area; and defining the respective gradient values based on the inverted pixels.

31 The system of claim 30 wherein determining that the transfer area includes a sub-portion that is associated with low-intensity values comprises: comparing values for the second boundary pixels across different sections of the transfer area

32 The system of any one of claims 27-31, wherein respective gradient values for pixels of the first image within the first boundary pixels of the mask are defined using a Poisson imaging techniques.

33 The system of any one of claims 27-32, wherein the first boundary pixels of the mask are narrowly tailored around the perimeter of the feature.

34 The system of any of claims 27-33, wherein the first boundary pixels of the mask are adjacent to or within a three pixel distance of an edge of the feature as depicted in the first image.

35 The system of any one of claims 27-34, wherein the instructions, when executed by the one or more processors, further cause the system to input the first image into a trained segmentation neural network to automatically generate the mask.

36 The system of any of claims 27-35, wherein the instructions, when executed by the one or more processors, further cause the system to identify outlier pixel values in an image area of the first image and defining the mask based on the outlier pixel values.

37 The system of any of claims 27-36, wherein the instructions, when executed by the one or more processors, further cause the system to the first image into a mask generation machine learning model to automatically generate the mask. 38 The system of claim 37, wherein the mask generation machine learning model is trained using at least one of training feature images having known defined masks and reinforcement learning with respect to training synthetic images

39 The system of any of claims 27-38, wherein the first boundary pixels of the mask are a non-uniform distance from an edge of the feature as depicted in the first image.

40 The system of any one of claims 27-39, wherein: the second image is an image of a container; and the feature is a defect associated with the container or contents of the container.

41 The system of claim 40, wherein: the container is a syringe; and the feature is a defect associated with a barrel of the syringe, a plunger of the syringe, a needle shield of the syringe, or a fluid within the syringe.

42 The system of claim 41, wherein: the container is a vial; and the feature is a defect associated with a wall of the vial, a cap of the vial, a crimp of the vial, or a fluid or lyophilized cake within the vial

43 The system of any one of claims 27-41, wherein: the feature is a container or object; and the second image is an image of a portion of a packing or manufacturing machine for the container or object.

44 The system of any one of claims 27-43, wherein the transfer area is identified in in the second image by: identifying a surrogate feature of the second image on which the feature is to be overlaid; and defining the transfer area to be randomly located within the surrogate feature.

45 The system of any one of claims 27-44, wherein the instructions, when executed by the one or more processors, further cause the system to generate a plurality of synthetic images by repeatedly transferring the feature to a plurality of different transfer areas within the second image

46 The system of claim 45, wherein the plurality of different transfer areas are randomly distributed across a portion of the second image.

47 The system of claim 46, wherein the portion of the second image is a pre-determined area corresponding to a portion at which the feature can manifest. 48 The system of any one of claims 45-47, wherein the instructions, when executed by the one or more processors, further cause the system to rotate, scale, stretch, and/or skew the feature placed in the plurality of different transfer areas based on a relative location of the different transfer areas within the portion of the second image.