EA046107B1 - ASSEMBLY LINE WITH BUILT-IN ELECTRONIC VISUAL INSPECTION - Google Patents

Description

This international application under the PCT claims the priority of US Provisional Application No. 62/368,438, filed July 29, 2016, which is incorporated herein by reference in its entirety.

State of the art

Assembling products that contain many elements, such as drug packaging, is a challenging task. Assembly can be carried out in one or more stages, with elements being placed into the preparation at each stage. At each step, errors can be introduced due to the correct element not being placed in the preparation, too much of the correct element being placed in the preparation, and/or the wrong element being placed in the preparation. Drugs that end up being shipped incorrectly result in lost revenue, increased consumer claims, and wasted time in resolving consumer complaints. In the case of drug product packaging, one unintended result of improper packaging is that prescribing physicians or patients may be reluctant to use the drug contained in an improperly assembled package. This may be particularly relevant for drugs that are administered parenterally, for example, subcutaneously, intramuscularly, intravenously, intraocularly or by inhalation. Even if an improperly assembled package is returned to the manufacturer by the prescribing physician or patient, a regulatory agency such as the US Food and Drug Administration will not allow the drug to be repackaged, resulting in a Notification of Event (NOE). Such NOEs entail investigations, additional costs and potentially result in reduced competitiveness.

Therefore, it is advisable to develop new technologies for drug assembly that remove these and other limitations of the prior art and improve it by reducing errors and increasing the efficiency of drug assembly.

The essence of the invention

It should be understood that both the following general description and the following detailed description are illustrative and explanatory only and not restrictive. Disclosed are methods and systems for obtaining a first image of a tray, determining the presence or absence of one or more first images in the first image, determining the rotation of each of the one or more first images in the first image, and performing an action based on the presence or absence and rotation of the one or more first images. in the first image.

Additional benefits are set forth in part in the following description or may be ascertained through practice. The advantages can be understood and achieved by the elements and combinations specifically set forth in the appended claims.

Brief description of drawings

The accompanying drawings, which are included in and constitute a part of this specification, illustrate embodiments and, together with the description, serve to explain the principles of the methods and systems:

fig. 1 is an example of a system;

fig. 2 is an example of an object image;

fig. 3A is an example of an object image;

fig. 3B is an example of an object image;

fig. 4A is an example of an object image;

fig. 4B is an example of an object image;

fig. 5A is an example of an object image;

fig. 5B is an example of an object image;

fig. 6A is an example of an object image;

fig. 6B is an example of an object image;

fig. 7A is an example of an object image;

fig. 7B is an example of an object image;

fig. 8A is an example of an object image;

fig. 8B is an example of an object image;

fig. 9 is an example embodiment of an example system;

fig. 10 is a block diagram illustrating an example of a method; and figs. 11 is an example of a typical operating environment.

Detailed description

Before methods and systems are disclosed and described, it should be understood that the methods and systems are not limited to specific methods, specific components, or specific implementations. It should also be understood that the terminology used herein is intended to describe specific embodiments only and is not intended to be limiting.

As used in the specification and the accompanying claims, the use of the singular number does not exclude the plural unless the context clearly suggests otherwise. Ranges may be stated herein as: from about one particular value and/or to about another particular value. When formulating such a range

- 1 046107 zone, another embodiment includes values from one specific value and/or to another specific value. Likewise, when formulating values as approximations by using an antecedent, it is approximately understood that the particular value constitutes another embodiment. It should also be understood that the limit values of each range are important both in relation to the other limit value and independently of the other limit value.

Optional or in some cases means that the event or circumstance described thereafter may or may not occur, and that the description includes cases in which the event or circumstance referred to occurs and cases in which it does not occur.

Throughout the specification and claims of this invention, the word contain and variations of the word, such as containing and contains, means including, but not limited to, and is not intended to exclude, for example, other components, integers, or steps. Illustrative means an example and is not intended to convey an indication of a preferred or ideal embodiment. Such as is not used in a restrictive sense, but for explanatory purposes.

Components that can be used to implement the described methods and systems are disclosed. These and other components are disclosed herein, and it should be understood that when combinations, subsets, interactions, groups, etc. are disclosed. of these components, although a specific example of each of the various individual and joint combinations and permutations thereof may not be explicitly disclosed, each is specifically contemplated and described herein for all methods and systems. This applies to all aspects of this invention, including, but not limited to, steps in the described methods. Thus, while there are a number of additional steps that may be performed, it should be understood that each of these additional steps may be performed with any particular embodiment or combination of embodiments of the described methods.

The present methods and systems can be more easily understood in view of the following detailed description of preferred embodiments and the examples included herein, as well as the drawings and the preceding and following descriptions thereof.

Those skilled in the art will appreciate that methods and systems may take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Additionally, the methods and systems may take the form of a computer program product or computer-readable storage medium comprising computer-readable program instructions (eg, computer software) implemented on the storage medium. In particular, the present methods and systems may take the form of computer software implemented over the Internet. Any suitable machine-readable storage medium may be used, including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.

Embodiments of methods and systems are described below with reference to illustrations in the form of block diagrams and block diagrams of methods, systems, devices and computer program products. It should be understood that each block of the block diagram and block diagram illustrations and combinations of blocks in the block diagram and block diagram illustrations, respectively, can be implemented by computer program instructions. These computer program instructions may be loaded into a mainframe computer, a special purpose computer, or other programmable processing device to create a computer, whereby the instructions, which are executed on the computer or other programmable processing device, create a means for implementing the functions specified in the block or blocks structural diagram.

These computer program instructions may also be stored in computer readable memory, which may direct the computer or other programmable data processing device to operate in a particular manner, causing the instructions stored in the computer readable memory to produce an item, including computer readable instructions, to implement the function specified in a block or blocks of a block diagram. Computer program instructions may also be loaded into a computer or other programmable processing device to initiate a sequence of steps of operation performed on the computer or other programmable device to create a computerized process, whereby the instructions that are executed on the computer or other programmable device provide the steps for implementing functions specified in a block or blocks of a block diagram.

In this regard, blocks of illustrations in the form of block diagrams and block diagrams support combinations of means for performing specified functions, combinations of steps for performing specified functions, and means of program instructions for performing specified functions. It should also be understood that each block of the block diagram and block diagram illustrations and combinations of blocks in the block diagram and block diagram illustrations may be implemented using specialized hardware computing systems that perform specified functions or steps, or combinations of specialized hardware and computer instructions.

- 2 046107

The present invention relates to improvements in computer functionality related to drug manufacturing and assembly.

Fig. 1 is a block diagram illustrating various aspects of an example system 100 in which the present methods and systems may operate. Those skilled in the art will appreciate that this document provides a functional description, and that the corresponding functions may be performed by software, hardware, or a combination of software and hardware.

In one aspect, system 100 may include a conveyor belt 101. Conveyor belt 101 may include one or more slats 102. One or more slats 102 may be made of rubber or similar material for attachment to the conveyor belt 101. One or more slats 102 may be raised or otherwise extended above the surface of the conveyor belt 101. One or more bars 102 may include a front bar and a back bar based on the direction 103 of movement. The front bar and the back bar may be positioned relative to an object placed on the belt such that the front bar is in front of the object with respect to the direction of movement 103 and the back bar is behind the object with respect to the direction of movement 103 . In this regard, the front bar for the first object may also be the back bar for the second object, which is in front of the first object, and so on. One or more objects 104 may be placed on the conveyor belt 101. In one aspect, the one or more objects 104 may contain a drug in one or more states of assembly. For example, one or more objects 104 may comprise a tray. The tray may be configured to receive one or more items. One or more elements may relate to treatment. For example, one or more elements may contain one or more syringes, automatic medical syringes, one or more syringe needles, one or more medication containers, one or more pamphlets or sets of written instructions, combinations thereof, and the like.

In one aspect, the set of written instructions provides information on how to use or administer the medication. In yet another aspect, the written instructions are a drug label approved by a regulatory authority, such as the US Food and Drug Administration.

In one aspect, the medicament is a solid dosage form. In yet another aspect, the medicament is a liquid dosage form. In yet another aspect, the medicament is a gel dosage form.

In one aspect, the medicament is formulated for oral administration. In yet another aspect, the medicament is formulated for parenteral administration. In yet another aspect, the medicament is formulated for subcutaneous administration. In yet another aspect, the medicament is formulated for intramuscular administration. In yet another aspect, the medicament is formulated for intravenous administration. In yet another aspect, the medicament is formulated for administration by inhalation. In yet another aspect, the medicament is formulated for intraocular administration.

In one aspect, the medicament contains a low molecular weight active ingredient. In yet another aspect, the medicament comprises a biological agent. In yet another aspect, the medicament contains a peptide or polypeptide active ingredient.

In one aspect, the medicament contains the active ingredient vascular endothelial growth factor (VEGF) derivative. In yet another aspect, the medicament contains aflibercept, which is described in one or more of US patent No. 7,070,959; 7,303,746; 7,303,747; 7,306,799; 7,374,757; 7,374,758; 7,531,173; 7,608,261; 7,972,598; 8,029,791; 8,092,803; 8,343,737; 8,647,842, each of which is incorporated herein by reference in its entirety.

The conveyor belt 101 may extend over a drive roller, which may be driven by a stepper motor 105. The use of the stepper motor 105 allows for precise positioning of one or more objects 104 relative to the camera 106, camera 107, and camera 108. The length of each of the one or more objects 104 can be represented as the exact number of motor steps. The conveyor belt 101 can be accurately advanced forward or backward to initiate movement of each of one or more objects 104 in the field of view 109, field of view 110, and field of view 111 associated with camera 106, camera 107, and camera 108, respectively. The programmable logic controller (PLC) 112 (PLC 112 may be a computing device, PLC, or other controller/processor) may be configured to cause the stepper motor 105 to execute any number of steps in any direction to initiate movement of one or more objects 104 in the review field 109, the review field 110 and the review field 111.

In one aspect, camera 106, camera 107, and/or camera 108 may be configured to scan, decode, read, sense, render, capture, and/or interpret visual codes. In some aspects, camera 106, camera 107, and/or camera 108 may be configured to process laser, line, or matrix images. For example, in one aspect, camera 106, camera 107, and/or camera 108 may include an imager for scanning, reading, and decoding one-dimensional or two-dimensional bar codes. Chamber 106, chamber 107

- 3 046107 and/or the camera 108 may include any imager, bar code scanner, or visual code scanner capable of extracting information from the visual codes in accordance with the described embodiments. In some aspects, camera 106, camera 107, and/or camera 108 may be configured to process scanned bar codes, images, and other data. Camera 106, camera 107, and/or camera 108 may include one or more depth cameras for capturing, processing, sensing, observing, modeling, detecting, and interacting with three-dimensional environments. In some aspects, camera 106, camera 107, and/or camera 108 can sense and detect depths and colors of objects in field of view 109, field of view 110, and field of view 111, respectively. Camera 106, camera 107, and/or camera 108 may also provide other camera and DVR functionality, such as taking photos, recording videos, streaming images or other data, storing data in image buffers, etc. This functionality may or may not include depth information. With respect to hardware and/or software processes in accordance with the described embodiments, camera 106, camera 107, and/or camera 108 may determine the dimensions, orientations, and visual properties of one or more objects 104. Camera 106, camera 107, and/or camera 108 may contain or implement any camera known to those skilled in the art that is capable of performing the processes disclosed herein. Camera 106, camera 107, and/or camera 108 may include appropriate hardware and software components (e.g., circuitry, software instructions, etc.) for transmitting signals and information to and from pass/fail controller 113 to carry out processes in accordance with with the described embodiments. Pass/fail controller 113 may comprise a computing device, PLC, or other controller/processor. Camera 106, camera 107, and/or camera 108 may provide an image and/or one or more notifications to monitor 114, monitor 115, and monitor 116, respectively.

The positioning of one or more objects 104 in the field of view 109, field of view 110, and field of view 111 may occur upon startup of the system 100 and may be adjusted during use of the system 100. One or more of camera 106, camera 107, and/or camera 108 may be used to ensuring proper positioning of the conveyor belt 101. For example, the camera 107 may be configured to generate an image of an area in the field of view 110. Camera 107 may determine the location of one or more slats 102 in the image. In one aspect, camera 107 may determine the location of the front bar. Camera 107 may compare a specific location of one or more slats 102 in the image with a reference location. If the determined location is equal to the reference location, then adjustment of the conveyor belt 101 is not necessary. If the determined location is not equal to the reference location, camera 107 may determine an offset based on the difference between the determined location and the reference location. The offset may be specified in units of distance, such as millimeters, centimeters, inches, etc., and/or the offset may be specified as a number of steps. The camera 107 may transmit a signal to the PLC 112 to move the conveyor belt 101 forward or backward by displacement by operating the stepper motor 105.

During operation, the system 100 may be configured to evaluate the current assembly state of one or more objects 104 and perform one or more actions based on the current assembly state. As each of the one or more objects 104 is moved forward by the conveyor belt 101, each of the one or more objects 104 will be placed in the field of view 109, field of view 110, and field of view 111 of camera 106, camera 107, and/or camera 108, respectively. Even though FIG. 1 illustrates only three cameras, in particular it is contemplated that less than three or more than three cameras may be used. It is further contemplated that the conveyor belt 101 may be configured to accommodate more than three illustrated objects 104 regardless of the number of cameras. As one or more objects 104 move forward along the conveyor belt 101, one or more elements may be assembled into one or more objects 104 by a human operator or robot.

When each of one or more objects 104 is in the field of view of one of the cameras, the camera may generate an image of the object 104 in the field of view associated with the camera. For example, camera 106 may generate an image of an area in field of view 109, camera 107 may generate an image of an area in field of view 110, and camera 108 may generate an image of an area in field of view 111. Camera 106, camera 107, and/or camera 108 can each analyze their respective images. Image analysis may include determining the presence or absence of one or more images. The one or more images may include a text image, a numeric image, a character image, and combinations thereof. For example, the text image can contain any sequence of letters, such as FILTER NEEDLE. A numeric pattern can contain any sequence of numbers, such as 6941518. A character pattern can contain any sequence of characters, such as »op*. In one aspect, camera 106, camera 107, and/or camera 108 may use optical character recognition (OCR) to “read” one or more images. In yet another aspect, camera 106, camera 107, and/or camera 108 may be configured to

- 4 046107 using OCR, and can be configured to recognize exclusively one or more images in the form of a specific image.

In one aspect, one or more images may be implemented on one or more elements assembled into one or more objects 104. In one aspect, at least a portion of the one or more elements may comprise one or more corresponding images. Thus, in the event that camera 106, camera 107, and/or camera 108 detects the presence of one or more images, the presence of one or more images indicates the presence of an element associated with a particular image. For example, if camera 106 detects the presence of a FILTER NEEDLE in the acquired image of an area in view field 109, then it may be concluded that an element associated with the FILTER NEEDLE image is present in object 104 in view field 109. Camera 106, camera 107, and/or camera 108 may be configured to determine the presence or absence of multiple images in a single image. For example, camera 106 may detect the presence of a FILTER NEEDLE and a FILTER NEEDLE in an acquired image of an area in the field of view 109 . The presence of both images may indicate that the element associated with the two occurrences of the FILTER NEEDLE image is present in the object 104 in the field of view 109.

Each of the elements, which may be collected into one or more objects 104, may be associated with one or more patterns that signify the presence or absence of a specific amount of the element. For example, an element can only be realized with the appearance of a specific image. If camera 106, camera 107, and/or camera 108 determine that a particular image appears only once, then it can be concluded that only one of the elements is present. However, if camera 106, camera 107, and/or camera 108 determine that a particular image appears two or more times, then it may be concluded that more than one element is present. In yet another example, an element may be implemented with two occurrences of a particular image. If camera 106, camera 107, and/or camera 108 determine that a particular image appears only twice, then it can be concluded that only one element is present. However, if camera 106, camera 107, and/or camera 108 determine that a particular image appears one or three or more times, then it can be concluded that more than one element is present. In yet another example, an element may be implemented with a range of specific images. For example, an element may be implemented with one or two appearances of a particular image. If camera 106, camera 107, and/or camera 108 determine that a particular image appears once or twice, then it can be concluded that only one element is present. However, if camera 10 6, camera 107, and/or camera 108 determine that a particular image appears three or more times, then it can be concluded that more than one element is present.

Camera 106, camera 107, and/or camera 108 may each be configured to analyze an entire image or one or more specific areas of an image. Fig. 2 illustrates an example image 200 of an object 104. The object 104 may include a tray 201 configured to store one or more items. One or more elements may be assembled into tray 201 such that at least a portion of one or more elements are present in one or more specific areas. Tray 201 may contain one or more areas, such as area 202, area 203, and area 204. Each of area 202, area 203, and area 204 may be associated with an area in which one or more patterns must be present if the item is present in the tray 201. For example, area 202 may be associated with the location of a vial cap when assembled into tray 201, area 203 may be associated with the location of one or more syringes and/or one or more needles when assembled into tray 201, and area 204 may be associated with location of one or more booklets when assembled into tray 201. Each of camera 106, camera 107, and/or camera 108 may be configured to analyze one or more specified areas of image 200. For example, camera 106 may be configured to analyze area 202 and area 203, camera 107 may be specified to analyze region 203, and camera 108 may be specified to analyze region 203 and region 204. Any combination of specified regions is contemplated. In addition, each of camera 10 6, camera 107, and/or camera 108 may be configured to determine the presence or absence of one or more specified patterns in specified areas. For example, camera 106 may be configured to determine the presence or absence of a vial cap in area 202 and the presence or absence of a first pattern (including the number of occurrences of the first pattern) in region 203, camera 107 may be configured to determine the presence or absence of a second pattern (including the number of occurrences second pattern) in area 203, and camera 108 may be configured to determine the presence or absence of a third pattern (including the number of occurrences of the third pattern) in region 203 and the presence or absence of a fourth pattern (including the number of occurrences of the fourth pattern) in region 204. Any combination is provided. given images and given areas.

According to FIG. 1, each of the one or more objects 104 may be configured to contain a specific amount of each of the one or more elements. The presence of a specific amount of each element means that one or more objects 104 are assembled correctly. The presence of any other than a specific amount of each element indicates that one or more objects 104 are not assembled correctly. Each of chamber 106, chamber 107, and/or chamber 108 may

- 5 046107 be made with the ability to conduct an independent assessment of the object 104 in the corresponding field of view. If the camera determines that a particular number of elements are present that the camera is configured to detect, the camera may provide a PASS signal to the pass/fail controller 113 . If the camera determines that a specific number of elements that the camera is configured to detect are missing, the camera may output a FAIL signal to the pass/fail controller 113 . If each of camera 106, camera 107, and/or camera 108 provides a PASS signal to the pass/fail controller 113, then the pass/fail controller 113 may provide a signal to the PLC 112 to cause the stepper motor 105 to advance the conveyor belt 101 to moving forward one or more objects 104 to be placed in the next camera's field of view. The pass/fail controller 113 may further transmit a notification to each of the monitors 114-116 to display a PASS notification. If one or more of camera 106, camera 107, and/or camera 108 outputs a FAIL signal to pass/fail controller 113, pass/fail controller 113 will not output a signal to PLC 112 to initiate forward movement of stepper motor 105. The pass/fail controller 113 may further notify monitors 114116 associated with the FAIL signal camera(s) to display the FAIL notification. An operator (eg, human or robot) located at the monitors 114-116 displaying the FAIL notification may take corrective action to correct the FAIL condition. For example, if a FAIL signal was issued as a result of a missing element, the operator can replace the missing element, after which the camera that made the preliminary FAIL determination can regenerate and reanalyze the image to determine that the element is currently present and issue the signal SUCCESSFUL PASS into controller 113 pass/fail. In yet another example, if a FAIL signal was generated as a result of one or more extraneous elements, the operator can remove one or more extraneous elements, after which the camera that made the preliminary FAIL determination can regenerate and reanalyze the image to determine that the required number of elements is currently present, and output a SUCCESS signal to the pass/fail controller 113.

In yet another aspect, image analysis by camera 106, camera 107, and/or camera 108 may include not only determining the presence or absence of one or more images, but also determining the rotation of two or more images. In one aspect, two or more images may be implemented on one or more elements assembled into one or more objects 104 along a particular axis. In one aspect, at least a portion of the one or more elements may comprise two or more corresponding images along a particular axis. Thus, in the case where camera 106, camera 107, and/or camera 108 detects the presence of two or more images along a particular axis, the presence of two or more images along a particular axis means the presence of an element associated with a particular image along a particular axis. For example, if camera 106 detects the presence of a FILTER NEEDLE and a FILTER NEEDLE along the same axis (e.g., 30°, 60°, 90°, 120°, 180°, etc.) in the acquired image of an area in the field of view 109, then it can be concluded that an element associated with the image FILTER NEEDLE and FILTER NEEDLE along the same axis is present in the object 104 in the viewing field 109. Camera 106, camera 107, and/or camera 108 may be configured to detect the rotation of multiple images in a single image. For example, camera 106 may detect the presence of a FILTER NEEDLE and a FILTER NEEDLE along a first axis and the presence of a SYRINGE NEEDLE and a SYRINGE NEEDLE along a second axis in the acquired image of an area in the field of view 109 . The presence of both patterns along two different axes may mean that the element associated with the two occurrences of the FILTER NEEDLE pattern along the first axis is present in object 104, and the element associated with the two occurrences of the SYRINGE NEEDLE pattern along the second axis is also present in object 104. As a second example, camera 106 may detect the presence of a FILTER NEEDLE and a FILTER NEEDLE along a first axis and the presence of a FILTER NEEDLE along a second axis in an acquired image of a region in the field of view 109 . The presence of both patterns along two different axes may mean that two occurrences of the element associated with the FILTER NEEDLE pattern are present in object 104.

Each of the elements that may be assembled into one or more objects 104 may be associated with one or more patterns that are implemented along a particular axis indicating the presence or absence of a specific amount of the element. For example, an element may be implemented with two occurrences of a particular image along a particular axis. If camera 106, camera 107, and/or camera 108 determine that a particular image appears only twice along a particular axis, then it can be concluded that only one element is present. However, if camera 106, camera 107 and/or camera 108

- 6 046107 determine that a particular image appears along more than one axis, it can be concluded that more than one element is present.

Fig. 3A and FIG. 3B illustrate an example view 300 and 303 of a tray 201 that includes an element 301 and an element 302. For example, element 301 may be a bottle and element 302 may be a filter needle. Any one of camera 106, camera 107, and/or camera 108 that generates image 300 may determine that a vial cap is present in area 202. The presence of one vial cap indicates that element 301 is present. Camera 106, camera 107, and/or camera 108 that generates image 300 may determine that two occurrences of the TEXT A image are present in the area 203. In one aspect, two occurrences of the TEXT A image may indicate that one or more instances of the element 302 are present, with camera 106, camera 107, and /or camera 108 may generate a PASS or FAIL signal as appropriate. In yet another aspect, depending on the configuration of the image on element 302 (e.g., one instance of element 302 may have either one occurrence of TEXT A or two occurrences of TEXT A), camera 106, camera 107, and/or camera 108 may determine whether TEXT A appears and TEXT A on the same axis. If TEXT A and TEXT A happen to be on the same axis, then camera 106, camera 107, and/or camera 108 may determine that a single instance of element 302 is present, with camera 106, camera 107, and/or camera 108 as appropriate can generate a PASS or FAIL signal. If TEXT A and TEXT A happen to be on different axes, then camera 106, camera 107, and/or camera 108 may determine that more than one instance of element 302 is present, and camera 106, camera 107, and/or camera 108, as appropriate, may generate a PASS or FAIL signal. In one aspect, axis detection can be used to confirm that a certain amount of element 302 is present and generate a PASS or FAIL signal based on the expected number of instances of element 302 relative to the determined number of instances of element 302.

Fig. 4A illustrates an example image 400 of tray 201 that includes element 301 and two instances of element 302. Camera 106, camera 107, and/or camera 108 that generates image 400 may determine that three occurrences of the image (TEXT A) are present in area 203. In one aspect, three occurrences of the TEXT A image may indicate that one or more instances of element 302 are present, and camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In yet another aspect, depending on the configuration of the image on element 302 (e.g., a single instance of element 302 may have a one-time occurrence of TEXT A, a two-time occurrence of TEXT A, or a three-time occurrence of TEXT A), camera 106, camera 107, and/or camera 108 may determine whether there are three occurrences of TEXT A on the same axis. As shown in FIG. 4A, two occurrences of TEXT A appear on the same axis, and one occurrence of TEXT A appears on a different axis. In this regard, camera 106, camera 107, and/or camera 108 may determine that more than one instance of element 302 is present, and camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. WALKTHROUGH. In one aspect, axis detection can be used to confirm that a certain amount of element 302 is present and generate a PASS or FAIL signal based on the expected number of instances of element 302 relative to the determined number of instances of element 302.

Fig. 4B illustrates an example image 401 of a tray 201 that includes an element 301, one instance of an element 302, and one instance of an element 402. In one aspect, the camera 106, camera 107, and/or camera 108 that generates the image 400 can determine what is present in the area 203 two appearances of the first image (TEXT A) and one appearance of the second image (TEXT B) are present. In one aspect, two occurrences of the TEXT A image may indicate that one or more instances of element 302 are present, and camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In yet another aspect, depending on the configuration of the image on element 302 (e.g., a single instance of element 302 may have one occurrence of TEXT A, two occurrences of TEXT A, or three occurrences of TEXT A), camera 106, camera 107, and/or camera 108 may determine whether whether two occurrences of TEXT A appear on the same axis. As shown in FIG. 4B, two occurrences of TEXT A appear on the same axis. In this regard, camera 106, camera 107, and/or camera 108 may determine that more than one instance of element 302 is present, and camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. . However, one occurrence of the TEXT B image may indicate that an item has been placed in tray 201 that should not be in tray 201 at this point in the assembly process. In this regard, camera 106, camera 107, and/or camera 108 may generate a FAIL signal based on the presence of an image that

- 7 046107 should not be present.

In yet another aspect, camera 106, camera 107, and/or camera 108 that generates image 400 may determine that the TEXT B image is present and may ignore the presence of the TEXT A image (or as required by any other image). In one aspect, one occurrence of the TEXT B image may indicate that one instance of element 302 is present, and camera 106, camera 107, and/or camera 108 may generate a PASS signal.

Fig. 5A illustrates an example image 500 of a tray 201 that includes an element 301, an element 302, and a single instance of an element 501. The camera 106, camera 107, and/or camera 108 that generates the image 500 may be configured to ignore the vial cap in the area 202 and ignore presence of image TEXT A in area 203. However, camera 106, camera 107, and/or camera 108 that generates image 400 may determine that two occurrences of another image (TEXT B) are present in area 203. In one aspect, two occurrences of the TEXT B image may indicate that either one or more than one instance of element 501 is present, whereby camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In yet another aspect, depending on the configuration of the image on element 501 (e.g., a single instance of element 501 may have a one-time occurrence of TEXT B, a two-time occurrence of TEXT B, or a three-time occurrence of TEXT B), camera 106, camera 107, and/or camera 108 may determine whether whether two occurrences of TEXT B appear on the same axis. As shown in FIG. 5A, two occurrences of TEXT B appear on the same axis. In this regard, camera 106, camera 107, and/or camera 108 may determine that one instance of element 501 is present, and camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In one aspect, axis detection may be used to confirm that a quantity of element 501 is present and generate a PASS or FAIL signal based on the expected number of instances of element 501 relative to the determined number of instances of element 501. FIG. 5B illustrates an example image 503 of tray 201 that includes element 301, element 302, and a single instance of element 501. FIG. 5B is similar to FIG. 5A except that FIG. 5B illustrates that the TEXT B image appears twice along the same axis, but at a different angle than the axis in FIG. 5A.

Fig. 6A illustrates an example image 600 of a tray 201 that includes an element 301 and two instances of an element 501. The camera 106, camera 107, and/or camera 108 that generates the image 600 may determine that four occurrences of the TEXT B image are present in the area 203. In one In aspects, four occurrences of the TEXT B image may indicate that one or more instances of element 501 are present, wherein camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In yet another aspect, depending on the configuration of the image on item 501 (e.g., a single instance of item 501 may have a one-time TEXT B occurrence, a two-time TEXT B occurrence, a three-time TEXT B occurrence, or a four-time TEXT B occurrence) camera 106, camera 107, and/or camera 108 may determine the axes on which the four occurrences of TEXT B occur. As shown in FIG. 5A, two occurrences of TEXT B appear on the first axis, and the other two occurrences of TEXT B appear on the second axis. In this regard, since the two groups of TEXT B appear on different axes, camera 106, camera 107, and/or camera 108 may determine that more than one instance of element 501 is present, with camera 106, camera 107, and/or camera 108 in may generate a PASS or FAIL signal as appropriate. In one aspect, axis detection can be used to confirm that a certain amount of element 501 is present and generate a PASS or FAIL signal based on the expected number of instances of element 501 relative to the determined number of instances of element 501.

Fig. 6B illustrates an example image 601 of tray 201 that includes element 301, element 302, and two instances of element 501. Camera 106, camera 107, and/or camera 108 that generates image 601 may determine that three occurrences of the TEXT B image are present in area 203 In one aspect, three occurrences of the TEXT B image may indicate that one or more instances of element 501 are present, and camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In yet another aspect, depending on the configuration of the image on item 501 (e.g., a single instance of item 501 may have a one-time TEXT B occurrence, a two-time TEXT B occurrence, a three-time TEXT B occurrence, or a four-time TEXT B occurrence) camera 106, camera 107, and/or camera 108 may determine the axes on which three occurrences of TEXT B occur. As shown in FIG. 6B, two occurrences of TEXT B appear on the first axis and one occurrence of TEXT B appears on the second axis. In this regard, since the two groups of TEXT B appear on different axes, camera 106, camera 107 and/or camera 108 can determine that more than

- 8 046107 one instance of element 501, wherein camera 106, camera 107 and/or camera 108 may generate a PASS or FAIL signal as appropriate. In one aspect, axis detection can be used to confirm that a certain amount of element 501 is present and generate a PASS or FAIL signal based on the expected number of instances of element 501 relative to the determined number of instances of element 501.

Fig. 7A illustrates an example image 700 of tray 201 that includes element 301, element 302, element 501, a single instance of element 701, and a single instance of element 702. The camera 106, camera 107, and/or camera 108 that generates the image 700 may be configured to ignoring the vial cap in area 202 and ignoring the presence of the TEXT A and TEXT B images in area 203. However, camera 106, camera 107, and/or camera 108 that generates image 700 may determine that two occurrences of the other are present in area 203 image (TEXT D). In one aspect, two occurrences of the TEXT D image may indicate that either one or more than one instance of element 701 is present, whereby camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In yet another aspect, depending on the configuration of the image on element 701 (e.g., a single instance of element 701 may have a one-time occurrence of TEXT D, a two-time occurrence of TEXT D, or a three-time occurrence of TEXT D), camera 106, camera 107, and/or camera 108 may determine whether whether two occurrences of TEXT D appear on the same axis. As shown in FIG. 7A, two occurrences of TEXT D appear on the same axis. In this regard, camera 106, camera 107, and/or camera 108 may determine that one instance of element 701 is present, and camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In one aspect, axis detection can be used to confirm that a certain amount of element 701 is present and generate a PASS or FAIL signal based on the expected number of instances of element 701 relative to the determined number of instances of element 701. In the same image 7 00 camera 106, camera 107, and/or camera 108 may determine that two occurrences of another image (TEXT C) are present in area 204. In one aspect, two occurrences of the TEXT C image may indicate that either one or more than one instance of element 702 is present, whereby camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In yet another aspect, depending on the configuration of the image on element 702 (e.g., a single instance of element 702 may have a one-time occurrence of TEXT C, a two-time occurrence of TEXT C, or a three-time occurrence of TEXT C), camera 106, camera 107, and/or camera 108 may determine whether whether two occurrences of TEXT C appear on the same axis. As shown in FIG. 7A, two occurrences of TEXT C appear on the same axis. In this regard, camera 106, camera 107, and/or camera 108 may determine that one instance of element 702 is present, and camera 106, camera 107, and/or camera 108 may generate a PASS or FAIL signal as appropriate. In one aspect, axis detection may be used to confirm that a quantity of element 702 is present and generate a PASS or FAIL signal based on the expected number of instances of element 702 relative to the determined number of instances of element 702. FIG. 7B illustrates an example image 703 of tray 201 that includes element 301, element 302, element 501, a single instance of element 701, and a single instance of element 702. FIG. 7B is similar to FIG. 7A except that FIG. 7B illustrates that the TEXT D image appears twice along the same axis, but at a different angle than the axis in FIG. 7A, and similarly, the TEXT C image appears twice along the same axis, but at a different angle than the axis in FIG. 7A.

Fig. 8A illustrates an example image 800 of a tray 201 that includes an element 301, an element 302, an element 501, two instances of an element 701, and a single instance of an element 702. The camera 106, camera 107, and/or camera 108 that generates the image 800 may determine that region 203, three occurrences of the TEXT D pattern are present. In one aspect, three occurrences of the TEXT D pattern may indicate that one or more instances of element 701 are present, whereby camera 106, camera 107, and/or camera 108 may, as appropriate, generate a signal SUCCESSFUL PASSAGE or FAILURE PASSAGE. In yet another aspect, depending on the configuration of the image on element 701 (e.g., a single instance of element 701 may have one occurrence of TEXT D, two occurrences of TEXT D, three times TEXT D, or four occurrences of TEXT D) camera 106, camera 107, and/or camera 108 may determine the axes on which the three occurrences of TEXT D occur. As shown in FIG. 8A, two occurrences of TEXT D appear on the first axis and one occurrence of TEXT D appears on the second axis. In this regard, since the two groups of TEXT D appear on different axes, camera 106, camera 107 and/or camera 108 may determine that more than one instance of element 701 is present, while

- 9 046107 pa 106, camera 107 and/or camera 108 may generate a PASS or FAIL signal as appropriate. In one aspect, axis detection may be used to confirm that a certain amount of element 701 is present and generate a PASS or FAIL signal based on the expected number of instances of element 701 relative to the determined number of instances of element 701. FIG. 8B is similar to FIG. 8A except that FIG. 8B illustrates that the TEXT D pattern appears twice along the first axis and once along the second axis, but the first and second axes are at different angles than the axes in FIG. 8A.

According to FIG. 1, each of camera 106, camera 107, and camera 108 can independently determine both the presence and absence of one or more patterns in the image and determine the rotation of each of the one or more patterns in the image of object 104. Each of camera 106, camera 107, and camera 108 may perform an action based on the presence or absence and rotation of one or more patterns in the image. If the camera determines that the correct amount of element is present in the image of the object 104 based on the presence of the image and the rotation of the image, the action may include transmitting a PASS signal to the pass/fail controller 113 . If the camera determines that an incorrect amount of element is present in the image of the object 104 based on the presence of the image and the rotation of the image, the action may include transmitting a FAIL signal to the pass/fail controller 113 . If each of camera 10 6, camera 107, and/or camera 108 outputs a PASS signal to the pass/fail controller 113, then the pass/fail controller 113 may provide a signal to the PLC 112 to initiate forward movement by the stepper motor 105 of the conveyor belt 101 to advance one or more objects 104 placed in the field of view of the next camera. The pass/fail controller 113 may also notify each of the monitors 114-116 to display a PASS notification. If one or more of camera 10 6, camera 107, and/or camera 108 outputs a FAIL signal to pass/fail controller 113, pass/fail controller 113 will not output a signal to PLC 112 to initiate forward movement by stepper motor 105. The pass/fail controller 113 may also transmit a notification to each of the monitors 114-116 associated with the camera(s) outputting the FAIL signal to display the FAIL notification. An operator (eg, human or robot) located at the monitors 114-116 displaying the FAIL notification may take corrective action to correct the FAIL condition.

In yet another aspect, one or more of camera 106, camera 107, and camera 108 may count the number of one or more objects 104. For example, when one or more objects 104 pass by one of camera 106, camera 107, and camera 108, the camera may increase the number of one or more objects 104 displayed by the camera. In yet another aspect, a number of empty spaces may be scattered among one or more objects 104. For example, in some scenarios, one or more of camera 106, camera 107, and camera 108 may not contain object 104 in their respective field of view. Conveyor belt 101 may have an image implemented thereon (eg, no tray image) in a position where object 104 would otherwise be placed. Camera 106, camera 107, and camera 108 may identify the image and provide a SUCCESS signal to assist in forward movement. conveyor belt 101.

Fig. 9 illustrates an example embodiment of system 100 illustrating the positioning of camera 10 6, camera 107, and camera 108 relative to conveyor belt 101. FIG. 9 also illustrates the positioning of monitors 114-116. A stepper motor 105 is shown at one end of the conveyor belt 101. One or more of the PLC 112 and/or pass/fail controller 113 may be housed within a housing 901. One or more of the dispensers 902 may be configured to store one or more items with access during assembly into one or more objects 104. System 100 may include one or more emergency stop (E-Stop) buttons 903. E-Stop buttons 903 may be activated at any time to temporarily stop operation of the system 100 for any reason. The E-Stop buttons 903 may return to their original position and the system 100 may be restarted (eg, by an operator or technician who has decided it is safe to do so). System 100 may include one or more OptoSwitches 904. OptoSwitches 904 may be actuated by placing a finger into the concave structure of OptoSwitch 904. This action breaks the optical signal path, causing a switching condition. OptoSwitches 904 can be used to accept visual control during Manual Activation mode and start/restart belt movement during Autonomous (or Automatic) mode.

System 100 may include a key switch mechanism 905. The 905 key switch mechanism can be used to change between Standalone Mode and Manual Mode

- 10 046107 noah activation. During normal operation, regardless of mode, the first operator terminal may include an operator loading trays onto the conveyor belt 101. In one aspect, these trays may be pre-loaded with a pre-filled bottle with a cap. In manual activation mode on the second operator terminal, the operator can load the filter needle tip into the tray. After this operation, camera 106 monitors the tray for the presence of the necessary elements. At the third operator terminal, an injection needle tip can be added to the tray. Camera 107 then monitors the tray for the presence of the required elements. At the fourth operator terminal, the operator loads an empty syringe into a tray placed in a blister pack. The fifth operator then loads the Insert (PI) into the tray. After loading, PI camera 108 monitors the tray to detect loading completion. Once the tray passes this last terminal, the completely filled tray exits the conveyor belt 101 for packaging into boxes.

In automatic mode, the trays move along the conveyor belt 101 automatically. The system 100 may maintain a wait time (eg, 1-5 seconds) before the conveyor belt 101 moves to the next position. Displacement occurs only when all three control cameras (eg, camera 106, camera 107, and camera 108) allow the passage of the tray that is being monitored by the corresponding camera. A problem at any monitoring terminal may result in a red light (Fail) state on the conveyor belt 101, whereby the operator may correct the problem or remove the tray from the conveyor belt 101 (each camera may allow the conveyor belt 101 to advance when in its field of view no tray). The forward progress of the conveyor belt 101 may depend on all the cameras detecting the configuration of the successfully passing tray. A display screen (eg, monitors 114-116) at each camera terminal may display the video stream from the corresponding camera with Pass, Fail, or Idle states overlaid depending on the inspection results. The camera's network connection status can be reset from monitors 114-116 if required during operation.

In one aspect, illustrated in FIG. 10, a method 1000 is described that includes obtaining a first image of a tray at 1010. The method 1000 may include determining the presence or absence of one or more first images in the first image at 1020. The one or more first images may comprise text images, numeric images, symbolic images and their combinations. Method 1000 may include determining a rotation of each of the one or more first images in the first image by 1030. Method 1000 may include performing an action based on the presence or absence of and rotating the one or more first images in the first image by 1040. In one of In aspects, each step of method 1000 may be performed by a computing device, a camera (with processing functionality), or a combination thereof. In some aspect, multiple computing devices and/or cameras may be used to implement method 1000. For example, multiple cameras may be used, wherein a first camera may perform steps 1010, 1020, and step 1030 while a second camera performs step 1040. Still in In one aspect, the method 1000 may be repeated in each of multiple chambers and/or computing devices as the tray moves along the assembly line. For example, steps 1010, 1020, 1030, and 1040 may be performed by a first camera for a particular image(s), then steps 1010, 1020, 1030, and 1040 may be performed again by a second camera for another specific image(s). Additionally, one or more of the substeps described herein may be performed by a dedicated camera and/or computing device.

Determining the presence or absence of one or more first images in the first image may include determining the presence of one or two of the one or more first images, and wherein determining the rotation of each of the one or more first images in the first image may include determining whether that one or two of one or more of the first images are on the first axis. Performing an action based on the presence or absence and rotation of one or more first images in the first image may include generating an inspection success signal and advancing a belt containing a tray disposed thereon. Determining the presence or absence of one or more first images in the first image may include determining the presence of three or more of the one or more first images. Performing an action based on the presence or absence and rotation of the one or more first images in the first image may include generating an inspection failure signal and notifying the operator that the first element associated with the one or more first images must be removed from the tray. Determining the presence or absence of one or more first images in the first image may include determining the presence of two of the one or more first images, and wherein determining the rotation of each of the one or more first images in the first image may include determining that two from one or more of the first images are not on the same axis. Performing an action based on the presence or absence and rotation of one or more first images in the first image may include generating a control failure signal and notifying the operator that the first element associated with the one or more first images

- 11 046107 high images, must be removed from the tray.

The method 1000 may further include obtaining a second image of the tray, determining the presence or absence of one or more second images in the second image, determining the rotation of each of the one or more second images in the second image, and performing an action based on the presence or absence and rotation of the one or more second images in the second image. The one or more second patterns may comprise text patterns, numeric patterns, character patterns, and combinations thereof. Determining the presence or absence of one or more second images in the second image may include determining the presence of one or two of the one or more second images, and wherein determining the rotation of each of the one or more second images in the second image may include determining whether that one or two of the one or more second images are on the second axis. Performing an action based on the presence or absence and rotation of one or more second images in the second image may include generating an inspection success signal and advancing a belt containing a tray disposed thereon. Determining the presence or absence of one or more second images in the second image may include determining the presence of three or more of the one or more second images. Performing an action based on the presence or absence and rotation of the one or more second images in the second image may include generating an inspection failure signal and notifying the operator that the second element associated with the one or more second images must be removed from the tray. Determining the presence or absence of one or more second images in the second image may include determining the presence of two of the one or more second images, and wherein determining the rotation of each of the one or more second images in the second image may include determining that the two of one or more second images are not on the same axis. Performing an action based on the presence or absence and rotation of the one or more second images in the second image may include generating an inspection failure signal and notifying the operator that the second element associated with the one or more second images must be removed from the tray.

The method 1000 may further include determining the location of the bar in the first image, comparing the determined location of the bar in the first image to a reference location, determining that the determined location is different from the reference location, determining an offset based on the difference between the determined location and the reference location, and transmitting a signal to the belt controller to adjust the distance for advancing the belt containing the tray disposed thereon by the amount of the offset. The offset can be one of a negative value, a positive value, or a zero value. In one aspect, determining an offset based on the difference between a determined location and a reference location and transmitting a signal to a belt controller to adjust the advance distance of a belt containing a tray disposed thereon by the amount of the offset may be performed by one or more of the cameras. For example, a separate camera can be specified to determine the offset. The offset determination can be made after each movement of the tape.

Method 1000 may further include repeatedly obtaining a first image of the tray, determining the presence or absence of one or more first images in the first image, determining the rotation of each of the one or more first images in the first image, and performing an action based on the presence or absence and rotation of one or more first images. more than the first images in the first image for each of the plurality of trays.

Method 1000 may further include counting quantities in a plurality of trays, with a number of empty tray locations scattered among the plurality of trays. The method 1000 may further include counting the number of empty tray locations. Determining the presence or absence of one or more first patterns in the first image may include determining the absence pattern of the trays. Performing an action based on the presence or absence of rotation of one or more second patterns in the first image may include advancing a tray containing the tray-free pattern thereon.

In one exemplary aspect, the methods and systems may be implemented on computer 1101, as illustrated in FIG. 11 and is described below. As an example, camera 106, camera 107, camera 108, PLC 112, and/or pass/fail controller 113 (or a component thereof) in FIG. 1 may be a computer 1101, as illustrated in FIG. 11. Likewise, the described methods and systems may use one or more computers to perform one or more functions at one or more locations. Fig. 2 is a block diagram illustrating an example operating environment 1100 for implementing the described methods. This example operating environment 1100 is an example of an operating environment only and is not intended to be any limitation as to the scope of use or functionality of the operating environment architecture. Neither the operating environment 1100 nor a requirement related to any of the components illustrated in the example operating environment 1100, or combinations thereof, should be construed as

- 12 046107 having any interdependence.

The present methods and systems may operate with a variety of other general purpose or special purpose computing system environments or configurations. Examples of known computing systems, environments and/or configurations that may be suitable for use with the above systems and methods include, but are not limited to, personal computers, server computers, portable devices, and multiprocessor systems. Other examples include set-top boxes, programmable consumer electronics, networked personal computers, programmable logic controllers (PLCs), minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, etc.

Processing in the disclosed methods and systems may be performed by software components. The disclosed systems and methods may be described in the general context of computer-executable instructions, such as program modules, executable by one or more computers or other devices. Typically, software modules contain computer code, routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific types of abstract data. The disclosed methods may also be implemented in grid-based and distributed computing environments in which tasks are performed by remote processing devices coupled over a data network. In a distributed computing environment, program modules may be located in local and/or remote computer storage environments, including storage devices.

In addition, those skilled in the art will appreciate that the systems and methods disclosed herein may be implemented by a general purpose computing device in the form of a computer 1101. The computer 1101 may include one or more components, such as one or more processors 1103, a system a memory 1112 and a bus 1113 that connects various components of the computer 1101, including one or more processors 1103, to system memory 1112. In the case of multiple processors 1103, the system may utilize parallel computing.

Bus 1113 may comprise one or more of several possible types of bus structures, such as a memory bus, a memory controller, a peripheral bus, an accelerated graphics port and a processor, or a local bus, using any of various bus architectures. Bus 1113 and all buses defined herein may also be implemented over a wired or wireless network connection.

The computer 1101 typically includes various machine-readable media. Examples of computer-readable media may include any accessible media accessible to computer 1101, including, for example, but not limited to, both volatile and non-volatile media, removable and non-removable media. System memory 1112 may comprise computer-readable media in the form of volatile memory, such as random access memory (RAM), and/or non-volatile memory, such as read-only memory (ROM). System memory 1112 typically may contain data, such as image analysis data 1107, and/or software modules, such as operating system 1105 and image analysis software 1106, that are accessible to and/or running on one or more processors 1103 .

In yet another aspect, computer 1101 may also include other removable/non-removable, volatile/non-volatile computer readable media. Mass storage device 1104 may provide nonvolatile storage of computer code, machine-readable instructions, data structures, program modules, and other data for computer 1101. For example, mass storage device 1104 may be a hard disk drive, magnetic disk, removable optical disk, magnetic cassette tapes, or other magnetic storage devices, flash memory cards, CD-ROMs, digital versatile discs (DVDs) or other optical storage devices, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), etc. .P.

Any number of software modules may be stored in mass storage device 1104 as needed, including, by way of example, operating system 1105 and image analysis software 1106. One or more of the operating system 1105 and image analysis software 1106 (or some combination thereof) may comprise elements of programming and image analysis software 1106. The image analysis data 1107 may also be stored in mass storage device 1104. The image analysis data 1107 may be stored in any one or more of the known databases. Examples of such databases include DB2, Microsoft Access, Microsoft SQL Server, Oracle, mySQL, PostgreSQL, etc. The databases may be centralized or distributed across multiple locations within the network 1115.

In yet another aspect, a user may enter commands and information into the computer 1101 through an input device (not shown). Examples of such input devices include, but are not limited to, a keyboard, a pointing device (e.g., a computer mouse, a remote control), a microphone, a joystick, a scanner, touch devices such as a touch screen, etc.

- 13 046107 tile input, such as gloves and other body coverings, motion sensors, etc. These and other input devices may be coupled to one or more processors 1103 via a human machine interface 1102 that is coupled to bus 1113, but may be coupled via other interface and bus structures, such as, but not limited to, a parallel port, a game port, an IEEE 1394 port ( also known as the Firewire port), serial port, 1108 network adapter, and/or universal serial bus (USB).

In yet another aspect, display device 1111 may be coupled to bus 1113 via an interface such as display adapter 1109. It is contemplated that computer 1101 may have more than one display adapter 1109, and computer 1101 may have more than one display device 1111.

For example, display device 1111 may be a monitor, LCD (liquid crystal display), light-emitting diode (LED) display, TV, smart lens, smart glass, and/or projector. In addition to the display device 1111, other output peripherals may include components such as speakers (not shown) and a printer (not shown), which may connect to the computer 1101 via an I/O Interface 1110. Any step and/or result of the methods may be output in any form to an output device. Such output may be any form of visual representation, including, but not limited to, text, graphics, animation, audio, tactile, etc. The display 1111 and the computer 1101 may be part of the same device or separate devices.

In one aspect, computer 1101 may be coupled to system 100 via I/O Interface 1110. The computer 1101 may be configured to monitor and store data. The computer 1101 may be configured to store images that are captured by cameras coupled to the system 100, store data that relates to pass/fail statistics generated during system-generated inspections, etc. Computer 1101 may also be used as a programming interface for one or more smart devices (eg, smart cameras) and/or embedded logic controllers that require custom firmware to operate. Computer 1101 may be used to generate, troubleshoot, download, and store iterations of the given software or firmware.

Computer 1101 may operate in a networked environment using logical connections to one or more remote computing devices 1114aDc. For example, remote computing device 1114a.Kc may be a personal computer, a data center (e.g., a workstation), a portable computer (e.g., a laptop, a mobile phone, a tablet device), a smart device (e.g., a smartphone, a smart watch, a tracker physical activity, smart clothing, smart accessory), security device and/or monitoring device, server, router, network computer, peer device, edge device or other shared network node, and so on. Logical connections between computer 1101 and remote computing device 1114a,b.c may be made through a network 1115, such as a local area network (LAN) and/or a wide area network (WAN). Such network connections may be made through the network adapter 1108. The network adapter 1108 may be implemented in both wired and wireless environments. Such networking environments are traditional and common in homes, offices, enterprise computer networks, intranets, and the Internet. In one aspect, the network adapter 1108 may be configured to provide power to one or more connected devices (eg, a camera). For example, the network adapter 1108 may comply with a Power over Ethernet (PoE) standard or the like.

For clarity, application programs and other executable software components, such as the operating system 1105, are illustrated herein as discrete units, although it should be appreciated that such programs and components may reside at different times in different storage components of the computing device 1101 and are executing. one or more processors 1103 of the computer 1101. An implementation of the image analysis software 1106 may be stored in or transmitted through some form of computer-readable medium. Any of the disclosed methods may be performed using machine-readable instructions implemented on machine-readable media. Machine-readable media can be any accessible media that can be accessed by a computer. By way of non-limiting example, computer readable media may include computer storage media and transmission media. Computer storage media may include volatile and nonvolatile, removable and non-removable media, implemented in any method or technology for storing information such as machine-readable instructions, data structures, program modules, or other data. Examples of computer storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk drive or other magnetic storage devices or any other environment that can be used

- 14 046107 used to store the necessary information and which can be accessed by a computer.

The methods and systems may use artificial intelligence (AI) techniques such as machine learning and iterative learning. Examples of such methods include, but are not limited to, expert systems, case-based reasoning, Bayesian networks, behavior-based AI, neural networks, fuzzy logic-based systems, evolutionary computing (e.g., genetic algorithms), swarm intelligence (e.g., ant algorithms) and hybrid artificial intelligence systems (for example, expert inference rules generated using a neural network, or production rules from statistical learning).

The disclosed methods and systems have been implemented, tested, and results verified using a standard manual operator-controlled assembly line process. The following table shows that the disclosed methods and systems are superior to the process of a standard manual, operator-controlled assembly line:

Although methods and systems have been described in connection with preferred embodiments and specific examples, the scope of the invention is not intended to be limited to the specific embodiments set forth, as the embodiments herein are in all respects illustrative and not restrictive.

Unless otherwise expressly provided, any method set forth herein is not in any way intended to be construed as requiring its steps to be performed in a particular order. In this regard, in cases where the claim for a method does not essentially contain the order in which its steps must be performed, or in other cases the claims and descriptions do not specifically indicate that the steps must be limited to a specific order, in no way in no way is it supposed that any order is implied in any relation. This is true for any possible explicit basis for interpretation, including: questions of logic regarding the arrangement of steps or the sequence of work; common meaning derived from grammatical organization or punctuation; the number or type of embodiments set forth in the specification.

It will be apparent to those skilled in the art that various variations are possible within

-

Claims (21)

scope or essence of the invention. Upon review of the specification and implementation of the disclosures therein, other embodiments will become apparent to those skilled in the art. It is intended that the description and examples be construed as illustrative only, the scope and spirit of the invention being set forth in the following claims. CLAIM 1. A method for monitoring one or more objects, containing the steps of: electronically obtaining a first image of a first object while the first object is stationary and located at the first monitoring location; determining the number and/or orientation of one or more first elements in the first image; comparing the determined number and/or orientation of one or more first elements in the first image with the expected number and/or orientation of one or more first elements in the first image; and providing, by means of the electronic controller, an electronic instruction allowing the first object to move from the first control location when the determined number and/or orientation of the one or more first elements in the first image corresponds to the expected number and/or orientation of the one or more first elements in the first image, wherein , in the absence of an electronic instruction, the first object remains stationary until the electronic controller provides another electronic instruction based on 1) an additional determination of the number and/or orientation of one or more first elements in the first image and 2) a comparison of the additionally determined amount and/or or the orientation of one or more first elements in the first image with the expected number and/or orientation of the one or more elements in the first image. 2. The method of claim 1, wherein allowing the first object to move away from the first monitoring location includes sending an electronic instruction to the moving device instructing the moving device to move the first object away from the first monitoring location. 3. The method of claim 2, wherein the moving device includes a conveyor belt. 4. The method of claim 1, wherein the one or more first elements comprise one or more first patterns, including text patterns, numeric patterns and/or character patterns. 5. The method of claim 4, wherein obtaining a first image of the first object includes obtaining a first image from a first display device, and wherein determining the number and/or orientation of one or more first elements in the first image includes determining the presence of two or more the first images in the corresponding field of view of the first display device. 6. The method of claim 5, wherein determining the number and/or orientation of one or more first elements in the first image includes determining whether the two or more first images are aligned along the same axis in a corresponding field of view of the first display device and wherein the provision of electronic instruction occurs when the two or more first images are aligned along the same axis. 7. The method of claim 1, further including the step of prohibiting movement of the first object from the first control location when the determined number and/or orientation of the one or more first elements in the first image differs from the expected number and/or orientation of the first one or more elements in the first image. 8. The method of claim 7, further comprising the step of generating a notification that the determined number and/or orientation of one or more first elements of the first image differs from the expected number and/or orientation of the one or more first elements in the first image. 9. The method of claim 8, further comprising the step of, after generating the notification, comparing the newly determined number and/or orientation of the one or more first elements in the first image with the expected number and/or orientation of the one or more first elements in the first image, and allowing the first object to move from the first control location when the newly determined number and/or orientation of the one or more first elements in the first image matches the expected number and/or orientation of the one or more first elements in the first image. 10. The method of claim 8, wherein generating the notification causes the machine to take corrective action with respect to the first object. 11. The method of claim 10, wherein the corrective action includes adding one or more first elements to the first object or removing one or more first elements from the first object. - 16 046107 12. The method according to claim 1, further comprising the steps of: electronically obtaining a second image of a second object while the second object is located at a second monitoring location remote from the first monitoring location; determining the number and/or orientation of one or more second elements in the second image; comparing the determined number and/or orientation of one or more second elements in the second image with the expected number and/or orientation of one or more second elements in the second image; and allowing the second object to move from the second control location only when (1) the determined number and/or orientation of one or more first elements in the first image matches the expected number and/or orientation of the one or more first elements in the first image, and (2 ) the determined number and/or orientation of the one or more second elements in the second image corresponds to the expected number and/or orientation of the one or more second elements in the second image. 13. The method of claim 12, wherein the one or more first elements are identical to the one or more second elements. 14. The method of claim 1, wherein the one or more first elements include one or more of the following: a syringe, a syringe needle, an automatic medical syringe. 15. An image-based electronic inspection method, comprising the steps of: each of the plurality of imagers obtains a first image; determining, for each of the plurality of imagers, the number and/or orientation of one or more first images in the first image; comparing, for each of the plurality of imagers, the determined number and orientation of one or more first images in the first image with the expected number and/or orientation of the one or more first images; generating, for each of the plurality of imagers, a pass signal or a failure signal based on the comparison; and advancing the tape containing the elements having one or more first images only upon receiving a pass signal for each of the plurality of imagers, wherein the tape is prohibited from moving when a failure signal is received, and the tape remains stationary until a success signal is received from each of the plurality of imagers. 16. The method of claim 15, wherein the one or more first images comprise text patterns, numeric patterns, character patterns, or combinations thereof. 17. The method of claim 15, wherein the one or more first patterns are located on one or more of a syringe, a syringe needle, or an automatic medical syringe. 18. Image-based electronic monitoring system comprising: moving platform; an image former configured to: obtain an image of the field of view; analyze the image to determine the number and/or orientation of one or more elements in the field of view; and generate a pass signal or a fail signal based on the number and/or orientation of one or more elements in the field of view; and a processor coupled to the image sensor, configured to: receiving a signal of successful passage of control or a signal of unsuccessful passage of control; advancing the moving platform only after receiving a successful control signal from the image former; and prohibiting the moving platform from moving until another signal is received. 19. The system of claim 18, wherein the one or more elements includes one or more of the following: a syringe, a syringe needle, an automatic medical syringe. 20. The system of claim 18, wherein the one or more elements comprise one or more first images. 21. The system of claim 18, wherein the processor is further configured to notify an operator when the movable platform is prohibited from moving. -