EP4168941A1 - Systems, methods, and media for manufacturing processes - Google Patents

Systems, methods, and media for manufacturing processes

Info

Publication number
EP4168941A1
EP4168941A1 EP21825727.7A EP21825727A EP4168941A1 EP 4168941 A1 EP4168941 A1 EP 4168941A1 EP 21825727 A EP21825727 A EP 21825727A EP 4168941 A1 EP4168941 A1 EP 4168941A1
Authority
EP
European Patent Office
Prior art keywords
component
quality metric
final quality
station
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP21825727.7A
Other languages
German (de)
French (fr)
Other versions
EP4168941A4 (en
Inventor
Andrew Sundstrom
Eun-Sol Kim
Damas Limoge
Vadim Pinskiy
Matthew C. PUTMAN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanotronics Imaging Inc
Original Assignee
Nanotronics Imaging Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US17/091,393 external-priority patent/US20210132593A1/en
Priority claimed from US17/195,746 external-priority patent/US12125236B2/en
Application filed by Nanotronics Imaging Inc filed Critical Nanotronics Imaging Inc
Publication of EP4168941A1 publication Critical patent/EP4168941A1/en
Publication of EP4168941A4 publication Critical patent/EP4168941A4/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00Programme-control systems
    • G05B19/02Programme-control systems electric
    • G05B19/04Programme control other than numerical control, i.e. in sequence controllers or logic controllers
    • G05B19/042Programme control other than numerical control, i.e. in sequence controllers or logic controllers using digital processors
    • G05B19/0426Programming the control sequence
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B19/00Programme-control systems
    • G05B19/02Programme-control systems electric
    • G05B19/418Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS] or computer integrated manufacturing [CIM]
    • G05B19/41875Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS] or computer integrated manufacturing [CIM] characterised by quality surveillance of production
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • G06N20/20Ensemble learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/044Recurrent networks, e.g. Hopfield networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/70Determining position or orientation of objects or cameras
    • G06T7/73Determining position or orientation of objects or cameras using feature-based methods
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/30Nc systems
    • G05B2219/32Operator till task planning
    • G05B2219/32187Correlation between controlling parameters for influence on quality parameters
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/30Nc systems
    • G05B2219/32Operator till task planning
    • G05B2219/32188Teaching relation between controlling parameters and quality parameters
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/30Nc systems
    • G05B2219/32Operator till task planning
    • G05B2219/32193Ann, neural base quality management
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/30Nc systems
    • G05B2219/32Operator till task planning
    • G05B2219/32194Quality prediction
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/30Nc systems
    • G05B2219/37Measurements
    • G05B2219/37559Camera, vision of tool, compute tool center, detect tool wear
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/084Backpropagation, e.g. using gradient descent
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02PCLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/02Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Definitions

  • a manufacturing system includes one or more stations, a monitoring platform, and a control module. Each station is configured to perform at least one step in a multi-step manufacturing process for a component. The monitoring platform configured to monitor progression of the component throughout the multi-step manufacturing process.
  • the control module configured to dynamically adjust processing parameters of a step of the multi-step manufacturing process to achieve a desired final quality metric for the component, the control module configured to perform operations.
  • the operations include receiving image data of tooling of a first station of the one or more stations.
  • the operations further include identifying a set of keypoints from the image data.
  • the keypoints correspond to position information of the tooling during processing at the first station.
  • the operations further include determining, by a machine learning model, a final quality metric for the component, based on the keypoints.
  • the operations further include, based on the determining, assigning the component to a class of components based on a comparison between the final quality metric generated by the machine learning model and a canonical final quality metric for the component.
  • a computer-implemented method for controlling a multi-step manufacturing process involves one or more stations of a manufacturing system. Each station is configured to perform at least one step in a multi-step manufacturing process for a component.
  • a computing system associated with the manufacturing system receives image data of tooling of a first station of the one or more stations. The computing system identifies a set of keypoints from the image data, the set of keypoints corresponding to position information of the tooling during processing at the first station.
  • a machine learning model associated with the computing system determines a final quality metric for the component, based on the set of keypoints. Based on the determining, the computing system assigns the component to a class of components based on a comparison between the final quality metric generated by the machine learning model and a canonical final quality metric for the component.
  • a manufacturing system includes one or more stations, a monitoring platform, and a control module.
  • Each station is configured to perform at least one step in a multi-step manufacturing process for a component.
  • the monitoring platform configured to monitor progression of the component throughout the multi-step manufacturing process.
  • T control module configured to dynamically adjust processing parameters of a step of the multi-step manufacturing process to achieve a desired final quality metric for the component, the control module configured to perform operations.
  • the operations include receiving image data of tooling of a first station of the one or more stations.
  • the operations further include identifying a set of keypoints from the image data. The keypoints correspond to position information of the tooling during processing at the first station.
  • the operations further include determining, by a machine learning model, a final quality metric for the component, based on the keypoints.
  • the operations further include determining that the final quality metric is not within a threshold tolerance from the final quality metric.
  • the operations further include based on the determining, assigning the component to a class of components based on a comparison between the final quality metric generated by the machine learning model and a canonical final quality metric for the component.
  • the operations further include, based on the assigning, determining that the class assigned to the component is not an acceptable class.
  • the operations further include based on the determining, inferring positional information corresponding to the component at the first processing station.
  • the operations further include based on the determining, generating an updated instruction set to be performed by at least one of the first processing station or a downstream station.
  • the operations further include predicting, by a machine learning model, a final quality metric for the component based on the updated instruction set.
  • the operations further include, based on the predicted final quality metric, providing the updated instruction set to at least one of the first processing station or the downstream station.
  • Figure 1 is a block diagram illustrating a manufacturing environment, according to example embodiments.
  • FIG. 2 is a block diagram illustrating control module, according to exemplary embodiments.
  • Figure 3 is a block diagram illustrating an exemplary architecture of LSTM model, according to example embodiments.
  • Figure 4 is a block diagram visually illustrating the overall process flow of a feedback segment for tooling module, according to example embodiments.
  • Figure 5 is a block diagram illustrating architecture of GRU model, according to example embodiments.
  • Figure 6A is a flow diagram illustrating a method of correcting a multi-step manufacturing process, according to example embodiments.
  • Figure 6B is a flow diagram illustrating a method of correcting a multi-step manufacturing process, according to example embodiments.
  • Figure 7 is a flow diagram illustrating a method of correcting a multi-step manufacturing process, according to example embodiments.
  • Figure 8A illustrates a system bus computing system architecture, according to example embodiments.
  • Figure 8B illustrates a computer system having a chipset architecture, according to example embodiments.
  • Manufacturing processes may be complex and include raw materials being processed by different process stations (or “stations”) until a final component is produced.
  • each process station receives an input for processing and may output an intermediate output that may be passed along to a subsequent (downstream) process station for additional processing.
  • a final process station may receive an input for processing and may output the final component or, more generally, the final output.
  • each station may include one or more tools/equipment that may perform a set of process steps.
  • Exemplary process stations may include, but are not limited to, conveyor belts, injection molding presses, cutting machines, die stamping machines, extruders, computer numerical control (CNC) mills, grinders, assembly stations, three-dimensional printers, quality control stations, validation stations, and the like.
  • CNC computer numerical control
  • each process station may be governed by one or more process controllers.
  • each process station may include one or more process controllers that may be programmed to control the operation of the process station.
  • an operator, or control algorithms may provide the station controller with station controller setpoints that may represent the desired value, or range of values, for each control value.
  • values used for feedback or feed forward in a manufacturing process may be referred to as control values.
  • Exemplary control values may include, but are not limited to: speed, temperature, pressure, vacuum, rotation, current, voltage, power, viscosity, materials/resources used at the station, throughput rate, outage time, noxious fumes, and the like.
  • One or more techniques provided herein improves upon conventional processes by providing a system through which image and/or video data may be used to predict or forecast a final quality metric of a component.
  • the system may include a monitoring system configured to capture or record video and/or image data of the tooling of each processing node or station of manufacturing system. Based on the positioning of the tooling during a process step, the present system may be configured to predict or forecast a final quality metric of a component.
  • the present system may generate and provide updated processing instructions to at least one of the current processing nodes k or stations k or to processing nodes (k + 1) and/or stations ( k + 1), in an attempt to correct for any errors in processing, such that a desired final quality metric may be achieved.
  • the system may include a monitoring system configured to capture or record video and/or image data of a component at each processing node or station of the manufacturing system. Based on visual information about the component at an end of a processing node, the present system may be configured to predict or forecast a final quality metric of a component. If the predicted final quality metric falls outside of a range of acceptable values, the present system may generate and provide updated processing instructions to at least one of processing nodes k or stations k or to processing nodes ( k + 1) and/or stations ( k + 1), in an attempt to correct for any errors in processing, such that a desired final quality metric may be achieved.
  • a monitoring system configured to capture or record video and/or image data of a component at each processing node or station of the manufacturing system. Based on visual information about the component at an end of a processing node, the present system may be configured to predict or forecast a final quality metric of a component. If the predicted final quality metric falls outside of a range of acceptable values, the present system may generate and provide updated
  • the present system is able to predict or forecast a final quality metric of a component, at any stage of the manufacturing process, without having to actually test the component.
  • Such system is particularly useful for final quality metrics that would otherwise require destructive testing, or for which the final quality metric cannot be evaluated until processing is complete.
  • FIG. 1 is a block diagram illustrating a manufacturing environment 100, according to example embodiments.
  • Manufacturing environment 100 may include a manufacturing system 102, a monitoring platform 104, and a control module 106.
  • Manufacturing system 102 may be broadly representative of a multi-step manufacturing system.
  • manufacturing system 102 may be representative of an assembly line system, where each processing station may be representative of a human worker.
  • manufacturing system 102 may be representative of a manufacturing system for use in additive manufacturing (e.g., 3D printing system).
  • additive manufacturing e.g., 3D printing system
  • manufacturing system 102 may be representative of a manufacturing system for use in subtractive manufacturing (e.g., CNC machining.
  • manufacturing system 102 may be representative of a manufacturing system for use in a combination of additive manufacturing and subtractive manufacturing. More generally, in some embodiments, manufacturing system 102 may be representative of a manufacturing system for use in a general manufacturing process.
  • Manufacturing system 102 may include one or more stations 108i-108 n (generally, “station 108”). Each station 108 may be representative of a step and/or station in a multi-step manufacturing process. For example, each station 108 may be representative of a layer deposition operation in a 3D printing process (e.g., station 108i may correspond to layer 1, station IO82 may correspond to layer 2, etc.). In another example, each station 108 may correspond to a specific processing station. In another example, each station 108 may correspond to a specific human operator performing a specific task in an assembly line manufacturing process.
  • Each station 108 may include a process controller 114 and control logic 116. Each process controller 114i-114 n may be programmed to control the operation of each respective station 108.
  • control module 106 may provide each process controller 114 with station controller setpoints that may represent the desired value, or range of values, for each control value.
  • Control logic 116 may refer to the attributes/parameters associated with a station’s 108 process steps. In operation, control logic 116 for each station 108 may be dynamically updated throughout the manufacturing process by control module 106, depending on a current trajectory of a final quality metric.
  • Monitoring platform 104 may be configured to monitor each station 108 of manufacturing system 102.
  • monitoring platform 104 may be a component of manufacturing system 102.
  • monitoring platform 104 may be a component of a 3D printing system.
  • monitoring platform 104 may be independent of manufacturing system 102.
  • monitoring platform 104 may be retrofit onto an existing manufacturing system 102.
  • monitoring platform 104 may be representative of an imaging device configured to capture an image of a component or tooling (e.g., a worker or a process tool) at each step of a multi-step process.
  • monitoring platform 104 may be configured to capture an image of the component at each station 108 and/or an image of a component developing the component at each station 108 (e.g., tooling, human, etc.) ⁇
  • monitoring platform 104 may be configured to capture information associated with production of a component (e.g., an image, a voltage reading, a speed reading, etc.) and/or tool (e.g., hand position, tooling position, etc.), and provide that information, as input, to control module 106 for evaluation.
  • Control module 106 may be in communication with manufacturing system 102 and monitoring platform 104 via one or more communication channels.
  • the one or more communication channels may be representative of individual connections via the Internet, such as cellular or Wi-Fi networks.
  • the one or more communication channels may connect terminals, services, and mobile devices using direct connections, such as radio frequency identification (RFID), near-field communication (NFC), BluetoothTM, low-energy BluetoothTM (BLE), Wi-FiTM, ZigBeeTM, ambient backscatter communication (ABC) protocols, USB, WAN, or LAN.
  • RFID radio frequency identification
  • NFC near-field communication
  • BLE low-energy BluetoothTM
  • Wi-FiTM Wireless Fidelity
  • ZigBeeTM ambient backscatter communication
  • Control module 106 may be configured to control each process controller of manufacturing system 102. For example, based on information captured by monitoring platform 104, control module 106 may be configured to adjust process controls associated with a specific station 108. In some embodiments, control module 106 may be configured to adjust process controls of a specific station 108 based on a projected final quality metric.
  • Control module 106 may include a tooling module 202 and a component module 204.
  • Tooling module 202 may be configured to project a final quality metric of a specimen at a given stage of production based on image data obtained by monitoring platform 104.
  • control module 106 may receive input from monitoring platform 104.
  • such input may take the form of an image or video of the tooling performing a subprocess at a given step of the multi-step manufacturing process.
  • the image or video data may include image or video data of a human’s hands while performing a specific subprocess of the multi-step manufacturing process.
  • the image or video data may include image or video data of a three-dimensional printer performing depositing a specific layer of a multi-layer manufacturing process.
  • control module 106 may project a final quality metric of the component.
  • control module 106 may determine one or more actions to take in subsequent manufacturing steps in order to reach a desired or threshold final quality metric. For example, if the projected final quality metric falls outside of a range of acceptable values, control module 106 may take one or more actions to rectify the manufacturing process.
  • control module 106 may interface with station controllers in at least one of a current station 108 (k) or subsequent stations 108 (k + 1) to adjust their respective control and/or station parameters.
  • control module 106 may provide human manufacturers with updated instructions to be performed at each processing station ( k + 1) of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
  • Component module 204 may be configured to project a final quality metric of a specimen at a given stage of production based on image data obtained by monitoring platform 104.
  • component module 204 may receive input from monitoring platform 104.
  • such input may take the form of an image or video of the component at a given step of the multi-step manufacturing process.
  • component module 204 may receive inferred component data at a given step of the multi-step manufacturing process from tooling module 202.
  • tooling module 202 may infer information about the component at a given step of the multi-step manufacturing process based on the tooling image or video data and provide that inferred information to component module 204 as input.
  • component module 204 may project a final quality metric of the component. Depending on the projected final quality metric of the component, component module 204 may determine one or more actions to take in subsequent manufacturing steps in order to reach a desired or threshold final quality metric. For example, if the projected final quality metric falls outside of a range of acceptable values, component module 204 may identify one or more actions to rectify the manufacturing process.
  • control module 106 may interface with station controllers in at least one of the current station 108 k or subsequent stations 108 (k + 1) to adjust their respective control and/or station parameters.
  • control module 106 may provide human manufacturers with updated instructions to be performed in at least one of a current processing station k or other processing stations (k + 1) of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
  • Each of tooling module 202 and component module 204 may include one or more software modules.
  • the one or more software modules may be collections of code or instructions stored on a media (e.g., memory of computing systems associated with control module 106) that represent a series of machine instructions (e.g., program code) that implements one or more algorithmic steps. Such machine instructions may be the actual computer code the processor interprets to implement the instructions or, alternatively, may be a higher level of coding of the instructions that is interpreted to obtain the actual computer code.
  • the one or more software modules may also include one or more hardware components. One or more aspects of an example algorithm may be performed by the hardware components (e.g., circuitry) itself, rather as a result of the instructions. Further, in some embodiments, each of tooling module 202 and component module 204 may be configured to transmit one or more signals among the components. In such embodiments, such signals may not be limited to machine instructions executed by a computing device.
  • tooling module 202 and component module 204 may communicate via one or more local networks.
  • Network may be of any suitable type, including individual connections via the Internet, such as cellular or Wi-Fi networks.
  • network may connect terminals, services, and mobile devices using direct connections, such as radio frequency identification (RFID), near-field communication (NFC), BluetoothTM, low-energy BluetoothTM (BLE), Wi-FiTM, ZigBeeTM, ambient backscatter communication (ABC) protocols, USB, WAN, or LAN.
  • RFID radio frequency identification
  • NFC near-field communication
  • BLE low-energy BluetoothTM
  • Wi-FiTM Wireless Fidelity
  • ZigBeeTM ambient backscatter communication
  • USB wide area network
  • WAN wide area network
  • Tooling module 202 may include acquisition system 206, extractor module 208, and prediction module 210.
  • a multi -node or multi-station assembly environment such as manufacturing system 102, may be represented, broadly as . where may represent the states of the component at all i nodes and where may represent that set of actions to be performed on the component by the tooling at all i nodes.
  • tooling module 202 may be configured to optimize the error in the assembly process with an estimated quality metric, may be within a threshold distance
  • tooling module 202 may estimate the state, which may be a numerical representation of a state of the component at all N nodes, and the actions, , which may represent the instructions or control values at each node.
  • Acquisition system 206 may be configured to receive image data of the assembly process at each node, N.
  • acquisition system 206 may receive the image data of the assembly process of monitoring platform 104.
  • acquisition system 206 may receive V number of images, where V may represent the number of cameras of monitoring platform 104 that may record the assembly procedure at each node. Accordingly, each image of the V number of images may capture a different perspective of the component during processing.
  • acquisition system 206 may be configured to extract a subset of images of frames. For example, acquisition system 206 may be configured to extract L number of images from the received image data.
  • the extracted images may be referred to as landmark frames. Landmark frames may be those image frames that are in high motion.
  • the extracted images may include those images or frames that include certain landmarks, /, of the component, where , and may represent the entire manufacturing process for the component.
  • Both minimal-motion (e.g., “landmark”) frames and maximal-motion (e.g., “high motion”) frames for a given operator may contain useful information for a classifier that is trying to correlate fmger-hand-arm data and flight performance data in a robust way across a number of operators.
  • an optical flow algorithm may be used to measure an amount of motion in any given frame. Acquisition system 206 may select those frames that contain the most motion.
  • Extractor module 208 may be configured to extract keypoints from the L number of images. For example, extractor module 208 may be configured to extract K number of keypoints, i.e., (x,y) pairs, per landmark /;. In other words, extractor module 208 may be configured to output K number of keypoints for a given input, As output, extractor module 208 may generate a single vector This vector may include landmark representations of K number of (x, y) pairs which may be represented by
  • extractor module 208 may implement two separate algorithms: (1) a bounding box estimation; and (2) keypoint detection.
  • each landmark frame may be processed with a threshold image segmentation to generate a mask image for each tooling component.
  • extractor module 208 may generate a mask image for each of the user’s hands.
  • extractor module 208 may implement blob detection to locate components of the tooling. Using a human as an example, extractor module 208 may assume that the image always contains both the left and right hands of the user. When a frame fails to include both hands, extractor module 208 may assign the value with an arbitrary constant value, c.
  • extractor module 208 may identify keypoints of the tooling based on the estimated bounding boxes. For example, with given input, extractor module 208 may estimate K number of points, p t , along with its confidence value, fy. In some embodiments, extractor module 208 may estimate, not only the points that are visible on the frame, but also points that may be occluded from the frame due to one or more of articulation, viewpoints, objects, or tool interactions. Because the objective may be to predict the quality measurement using the tracked keypoints, the non-estimated occluded points may be unique and important features representing the assembly process.
  • an occlusion threshold value may be derived from observation of c L value on occluded points in small randomly chosen subset of the landmark frames.
  • extractor module 208 may filter out the estimation that are c 0 ⁇ t 0 .
  • extractor module 208 may assign them an arbitrary constant value, c.
  • the output of extractor module 208 may include L * K(x, y) pairs for each component.
  • extractor module 208 may be configured to assign a default basal level of confidence for occluded components and, therefore, estimate those keypoints so that a full set of keypoints may be available to measure discrepancies between observed trial tooling component positions and the canonical tooling component positions.
  • tooling module 202 may draw inferences about the state of the component at a given point in time (e.g., station i) and then output recommended modified-from-canonical subsequent actions to take to correct for the measured discrepancies.
  • Prediction module 210 may be configured to predict a final quality metric, V C .
  • prediction module 210 may be configured to predict a final quality metric, V c . where given tooling tracking information for the L number of points-in-time, P, where P G may be gathered from each processing station 108.
  • prediction module 210 may implement a long short-term memory (LSTM) model to output the final quality metric.
  • the LSTM model may allow prediction module 210 to overcome a vanishing gradient problem that is common in conventional recurrent neural networks. The vanishing gradient problem is the case where a model fails to relate the early weights of the neural network when the magnitude of the gradients are small for later layers. LSTM model eliminates this issue.
  • tooling module 202 may further include a classification module 215.
  • Classification module 215 may be configured to classify a specimen into one or more classes based on the predicted final quality metric, V c .
  • classification module 215 may receive a canonical or desired final quality metric, V c . for the specimen.
  • Classification module 215 may be configured to compare the predicted final quality metric, V c , to the canonical or desired final quality metric, V c , to generate a delta, D n .
  • classification module 215 may sort the specimen into one or more classes.
  • classification module 215 may sort the specimen into one of four classes: fail (class 1), fair (class 2), excellent (class 3), and over-qualify (class 4).
  • Classification module 215 may sort the specimen based on associated class definitions.
  • the associated definitions may be:
  • control module 106 may use the classification assigned to the component to determine whether the final quality metric is acceptable. For example, an operator or client may specify that components classified in Class 1 are not acceptable, while components classified in any of Classes 2-4 are acceptable. In another example, an operator or client may specify that components classified in Class 1 or Class 2 are not acceptable, while components classified in Class 3 or Class 4 are acceptable.
  • FIG. 3 is a block diagram illustrating an exemplary architecture of LSTM model, according to example embodiments.
  • LSTM model 300 may include three layers 302I-3023 (generally layer 302).
  • each layer 302 may include one or more cells 304.
  • each cell’s 304 input may be p i where i E [0 , L — 1],
  • Each cell in the LSTM may be defined as: where equation 1 decides whether to keep information from the previous cell or not; equation (2) decides which values to update; equation (3) updates the cell; and equation (4) decides which part to output.
  • Equation (5) may filter the output parts so that LSTM model 300 only outputs what it is programmed to output.
  • LSTM model 300 may include three layers 302 with 30 hidden sizes.
  • LSTM model 300 may be a sequence-to-one LSTM model.
  • LI mean absolute error
  • MAE mean absolute error
  • loss function may be minimized using an Adam optimizer.
  • MAE may be used because the goal may be to minimize or reduce the magnitude of errors, regardless of the direction of the error.
  • prediction module 210 may be trained using a canonical set of instructions, a, and multiple components, M.
  • the input for prediction module 210 may be structured by first being pre- processed through acquisition system 206 and extractor module 208.
  • each component involved in training may be validated through a validation algorithm that checks their composition (e.g., shape).
  • the validation algorithm may compute a similarity index on the composition at the end of each node by comparing the composition with canonical compositions. As a result, the components for training are roughly similar to the canonical composition.
  • the corresponding physical components may be tested in a controlled environment to measure their quality metric.
  • prediction module 210 may be trained, for example, with a portion of M being training data and another portion of M being validation data. Once trained, prediction module 210 may be able to predict a quality measurement of a component at a given processing step, based on image data of the tooling.
  • component module 204 may include stochastic gradient descent (SGD) module 212, gated recurrent unit (GRU) model 214, and simulation module 216.
  • SGD stochastic gradient descent
  • GRU gated recurrent unit
  • a partial construct of a component may be defined as a k . where step k introduces an irreversible error in the manufacturing process, and steps k + 1, . . . , N are not yet defined.
  • Component module 204 may be configured to identify an optimal corrective sequence of remaining actions where r and t may correspond to specific operations to be performed on the component, at least one of the current processing station or each subsequent processing station (k + 1 to N) of manufacturing system 102.
  • any component a may be defined as a sequence of all operations performed at each processing station 1 ... N of manufacturing system.
  • the virtual representation system may represent the component in Euclidean space (e.g., R 3 ) as a set of connected surface and a set of connected points uniformly distributed along the outer contour of each surface.
  • the virtual representation functions generating these representations may be referred to as respectively.
  • component module 204 may be configured to correct a k toward a particular, canonical component,
  • Simulation module 216 may be configured to simulate or generate a surface model for a given component q
  • simulation module 216 may receive tooling information from tooling module 202. Based on the keypoints generated by tooling module 202, simulation module 216 may be configured to generate a surface model representing a state of the component a L at a specific process step i. In some embodiments, the surface model may be represented as S(a j ). In some embodiments, simulation module 216 may further be configured to generate or estimate a quality metric of the component fy. From the surface model, simulation module 216 may be configured to generate a points model, P(d j ), representing specific coordinates of the component, a t . For example, from surface model S(a i ).
  • simulation module 216 may create points model, P(a;), by placing a number of points uniformly spaced around a bounding contour of each surface in S(a j ).
  • S(a;) may be used to simulate performance of the artifact fy.
  • SGD module 212 may receive the points model, P(d j ), from simulation module 216. SGD module 212 may determine whether an irreversible error, k, has occurred by comparing the points model, P(a;), at step i to a canonical points model, P(a * ) of a canonical component, a * . An irreversible error may be defined to be a measurably significant structural deviation from a canonical component at step k. SGD module 212 may be configured to detect an irreversible error by taking a Hausdorff distance. For example, SGD module 212 may match a processing step of current component a to canonical component a * based on respective Euclidean point sets. Mathematically, SGD module 212 may be configured to compute the Hausdorff distance between For example, where d(x,y ) may be the Euclidean distance between x and y, and the undirected Hausdorff distance may be:
  • An irreversible error may be present when the Hausdorff distance between the current component a; and the canonical component a * exceeds some threshold tolerance.
  • SGD module 212 may determine that an error occurs at step k when: where t H is some suitably defined tolerance threshold.
  • SGD module 212 may be configured to construct a set of updated actions given a set of actions up to a point of error an irreversible error k.
  • this set of updated actions may be referred to as Xtaii ⁇
  • the sequences of steps or actions that preceded and included the error step k may be referred to as x head .
  • x tail and x head may define a component
  • SGD module 212 may solve for x tail using a stochastic gradient descent method.
  • GRU model 214 may be configured to predict a final quality metric for the component, . where may represent a vector concatenation operator.
  • the final quality metric generated by GRU model 214 may be compared against a canonical final quality metric to determine if x taii is proper. For example, assuming that the combination of x ta u and xhead yields a final quality metric that falls outside of a range of acceptable values, GRU model 214 may instruct SGD module 212 to generate an updated sequence of updated actions for further evaluation.
  • FIG. 5 is a block diagram illustrating architecture of GRU model 214, according to example embodiments.
  • GRU model 214 may include N GRU cells 502 I -502 N (generally, GRU cell 502), with each GRU cell 502 corresponding to a respective processing station 108.
  • Each GRU cell 502 may include an (r u t L ) input pair and a hidden state output hi of a predetermined size. Together, these input pairs ( r u t L ) may define a given component a n
  • each GRU cell 502 may be defined by:
  • the hidden state at time may be the input at time t, and represent the reset, update, and new gates at time t, respectively.
  • GRU model 214 may be trained to generate weights corresponding thereto. For example, GRU model 214 may be trained iteratively to bias GRU model 214 toward solving a specific subproblem. In some embodiments, during a first iteration, GRU model 214 may generate a plurality (e.g., several hundred to several thousand) best predictions (one for each possible error at step k ) to complete a given component a k at steps k + I, . , ., N, along with the corresponding predicted quality metrics corresponding to each prediction.
  • a plurality e.g., several hundred to several thousand
  • all of the predicted completions may be rendered in a virtual representation system, their stepwise Hausdorff distances computed, and their rendered surfaces simulated, to obtain distance measures between the generated predictions and the canonical.
  • the loss values between the canonical and predicted quality metric measures may be computed and fed back into the GRU model 214, whose weights may be adjusted via backpropagation, producing a second iteration. This process may continue until a desired error threshold is obtained.
  • GRU model 214 may be trained on a prepared or synthetic data set.
  • simulator module 216 or another simulation system e.g., a CAD model
  • a simulation system may generate, for example, a set of prepared components (e.g., 100,000 prepared components) to represent many unbiased randomized trials of actions that may cover the full admissible range of all parameter values.
  • the set prepared components may also include biased randomized trials of action, in addition to the unbiased randomized trials. This set may be referred to as P(t/(0,1), 100000). Each p E P(U( 0,1), 100000) may obtain its defining parameter values by sampling from the uniform distribution 1/(0, 1).
  • a set of 50,000 prepared components may be generated to represent many trials of processing a component to replicate the canonical component a * . This may be referred to as the set Each 50000) may obtain its defining parameter values by a statistical sampling process.
  • the i th action may centrally tend to the corresponding canonically- defined fold and may vary according to an appropriately parameterized Gaussian distribution.
  • Figure 4 is a block diagram visually illustrating the overall process flow of a feedback segment 400 for tooling module 202, according to example embodiments.
  • feedback segment 400 may include a first portion 402, a second portion 404, and a third portion 406.
  • tooling module 202 may perform an acquisition process, such as that carried out, at least partially, by acquisition system 206.
  • manufacturing system 102 may receive processing instructions for processing a component.
  • manufacturing system 102 may receive processing instructions on a per station 108 basis.
  • processing instructions may include control values that define attributes (e.g., temperature, pressure, etc.) of a station 108 for manufacturing.
  • processing instructions may include videos or images that visually illustrate to a human operator how to perform a specific processing step at a processing node of the manufacturing process.
  • acquisition system 206 may receive image data from monitoring platform 104.
  • acquisition system 206 may receive V number of images, where V may represent the number of cameras of monitoring platform 104 that may record the assembly procedure at each node.
  • acquisition system 206 may be configured to extract a subset of images of frames.
  • acquisition system 206 may be configured to extract L number of images from the received image data.
  • the extracted images may be referred to as landmark frames.
  • the extracted images may include those images or frames that include certain landmarks and may represent the entire manufacturing process for the component.
  • Second portion 404 may correspond to operations performed by extractor module 208.
  • extractor module 208 may receive at least the extracted images from acquisition system 206.
  • Extractor module 208 may be configured to extract keypoints from the L number of images.
  • extractor module 208 may be configured to extract K number of keypoints, i.e., (x,y) pairs, per landlark /;.
  • extractor module 208 may perform bounding box estimation. For example, given /, each landmark frame may be processed with a threshold image segmentation to generate a mask image for each tooling component. For example, as shown, in an embodiment in which the tooling is a human, extractor module 208 may generate a mask image for each of the user’s hands.
  • extractor module 208 may perform keypoint detection, given the bounding box estimation. For example, with given input, extractor module 208 may estimate K number of points, p;, along with its confidence value, c L . In some embodiments, extractor module 208 may estimate, not only the points that are visible on the frame, but also points that may be occluded from the frame due to one or more of articulation, viewpoints, objects, or tool interactions.
  • Third portion 406 may correspond to operations performed by prediction module 210.
  • prediction module 210 may receive keypoint information from extractor module 208 and may be configured to predict a final quality metric, V c .
  • prediction module 210 may implement a long short-term memory (LSTM) model to output the final quality metric.
  • LSTM long short-term memory
  • Figure 6A is a flow diagram illustrating a method 600 of correcting a multi-step manufacturing process, according to example embodiments.
  • Method 600 may begin at step 602.
  • an instruction set may be provided to manufacturing system 102.
  • the instruction set may be representative of a set of instructions for a manufacturing process to be carried out by manufacturing system 102.
  • the instruction set may be provided to each station 108.
  • each canonical instruction set provided to each respective station 108 may define the processing parameters for a specific manufacturing step.
  • each canonical instruction set may be a video of discrete steps to be performed by a human actor at a specific processing node or station 108.
  • control module 106 may receive image data of the tooling (e.g., station 108) from monitoring platform 104.
  • acquisition system 206 may receive image data of the assembly process at a respective processing node.
  • acquisition system 206 may receive V number of images, where V may represent the number of cameras of monitoring platform 104 that may record the assembly procedure at a specific procession station 108. Accordingly, each image of the V number of images may capture a different perspective of the tooling during processing.
  • control module 106 may extract a subset of images from the obtained image data.
  • acquisition system 206 may be configured to extract a subset of images of frames.
  • acquisition system 206 may be configured to extract L number of images, i.e., landmark frames, from the received image data.
  • the extracted images may be referred to as landmark frames.
  • Landmark frames may be those image frames that are in high motion.
  • the extracted images may include those images or frames that include certain landmarks , of the component, where and may represent the entire manufacturing process for the component.
  • control module 106 may extract one or more keypoints of the tooling from the landmark frames.
  • extractor module 208 may extract keypoints from the L number of images. Extractor module 208 may identify or extract K number of keypoints for a given input, As output, extractor module 208 may generate a single vector This vector may include landmark representations of K number of (x, y) pairs which may be represented by where
  • control module 106 may predict a final quality metric for the component based on at least the identified keypoints.
  • prediction module 210 may implement a long short-term memory (LSTM) model to output the final quality metric.
  • LSTM long short-term memory
  • control module 106 may compare the final quality metric to a desired quality metric. If, at step 612, control module 106 determines that the final quality metric is a threshold tolerance of the desired quality metric, the manufacturing process may proceed to the next processing station or node (e.g., step 616), in accordance with the original instruction set. If, however, at step 612, control module 106 determines that the final quality metric is not within a threshold tolerance of the desired quality metric, then, at step 614 control module 106 may adjust the processing parameters of at least one of the processing station k or one or more processing stations k + 1, where k may represent the current processing station and k+1 may represent other processing stations downstream of the current processing station.
  • control module 106 may interface with station controllers in a current station 108 (k) and/or subsequent stations 108 ( k + 1) to adjust their respective control and/or station parameters.
  • control module 106 may provide human manufacturers with updated instructions to be performed in at least one of a processing station k or one or more processing station ( k + l)of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
  • Figure 6B is a flow diagram illustrating a method 650 of correcting a multi-step manufacturing process, according to example embodiments.
  • Method 650 may begin at step 652.
  • an instruction set may be provided to manufacturing system 102.
  • the instruction set may be representative of a set of instructions for a manufacturing process to be carried out by manufacturing system 102.
  • the instruction set may be provided to each station 108.
  • each canonical instruction set provided to each respective station 108 may define the processing parameters for a specific manufacturing step.
  • each canonical instruction set may be a video of discrete steps to be performed by a human actor at a specific processing node or station 108.
  • control module 106 may receive image data of the tooling (e.g., station 108) from monitoring platform 104.
  • acquisition system 206 may receive image data of the assembly process at a respective processing node.
  • acquisition system 206 may receive V number of images, where V may represent the number of cameras of monitoring platform 104 that may record the assembly procedure at a specific procession station 108. Accordingly, each image of the V number of images may capture a different perspective of the tooling during processing.
  • control module 106 may extract a subset of images from the obtained image data.
  • acquisition system 206 may be configured to extract a subset of images of frames.
  • acquisition system 206 may be configured to extract L number of images, i.e., landmark frames, from the received image data.
  • the extracted images may be referred to as landmark frames.
  • Landmark frames may be those image frames that are in high motion and may contain the most meaningful features that clearly captures the progress of the manufacturing process.
  • the extracted images may include those images or frames that include certain landmarks, of the component, where , and may represent the entire manufacturing process for the component.
  • control module 106 may extract one or more keypoints of the tooling from the landmark frames.
  • extractor module 208 may extract keypoints from the L number of images. Extractor module 208 may identify or extract K number of keypoints for a given input, As output, extractor module 208 may generate a single vector This vector may include landmark representations of K number of (x, y) pairs which may be represented by where
  • control module 106 may predict a final quality metric for the component based on at least the identified keypoints.
  • prediction module 210 may implement a long short-term memory (LSTM) model to output the estimate of the final quality metric.
  • LSTM long short-term memory
  • control module 106 may classify the component based on the final quality metric.
  • classification module 215 may classify the component into one or more classes based on the predicted final quality metric, V c .
  • tooling module 202 may receive a canonical or desired final quality metric, V c . for the component.
  • Tooling module 202 may be configured to compare the predicted final quality metric, V c . to the canonical or desired final quality metric, V c , to generate a delta, D ⁇ .
  • classification module 215 may sort the component into one or more classes.
  • method 650 may further include operations 664-668.
  • control module 106 may determine if the class assigned to the component is an acceptable class. If, at step 614, control module 106 determines that component has been assigned to an acceptable class, the manufacturing process may proceed to the next processing station or node (e.g., step 616), in accordance with the original instruction set. If, however, at step 614, control module 106 determines that component is assigned to an unacceptable class, then, at step 618 control module 106 may adjust the processing parameters of at least one of a processing station k or one or more processing stations (k + 1). In some embodiments, control module 106 may interface with station controllers in at least one of current station 108 or subsequent stations 108 to adjust their respective control and/or station parameters.
  • control module 106 may provide human manufacturers with updated instructions to be performed in at least one of a processing station k or one or more processing stations ( k + 1) of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
  • FIG. 7 is a flow diagram illustrating a method 700 of correcting a multi-step manufacturing process, according to example embodiments.
  • Method 700 may begin at step 702.
  • an instruction set may be provided to manufacturing system 102.
  • the instruction set may be representative of a set of instructions for a manufacturing process to be carried out by manufacturing system 102.
  • the instruction set may be provided to each station 108.
  • each canonical instruction set provided to each respective station 108 may define the processing parameters for a specific manufacturing step.
  • each canonical instruction set may be a video of discrete steps to be performed by a human actor at a specific processing node or station 108.
  • control module 106 may identify information corresponding to a component at a respective processing node.
  • simulation module 216 may receive tooling information from tooling module 202. Based on the keypoints generated by tooling module 202, simulation module 216 may be configured to generate a surface model representing a state of the component a L at a specific process step i. In some embodiments, the surface model may be represented as In some embodiments, simulation module 216 may further be configured to generate or estimate a quality metric of the component a L . From the surface model, simulation module 216 may be configured to generate a points model, P representing specific coordinates of the component,
  • control module 106 may determine whether an irreversible error has occurred.
  • SGD module 212 may receive points model, from simulation module 216. SGD module 212 may determine whether an irreversible error, k, has occurred by comparing the points model, at step i to a canonical points model, P of a canonical component, SGD module 212 may be configured to detect an irreversible error by taking a Hausdorff distance between the points model and the corresponding canonical points model. An irreversible error may be present when the Hausdorff distance between the current component and the canonical component , at the respective processing station or node, exceeds some threshold tolerance.
  • control module 106 determines that an irreversible error has not occurred, then the manufacturing process may proceed to the next processing station or node (step 705), in accordance with the original instruction set. If, however, at step 706, control module 106 determines that an irreversible error has occurred, then method 700 proceed to step 708.
  • control module 106 may generate an updated set of actions to correct the irreversible error.
  • SGD module 212 may construct a set of updated actions given a set of actions up to a point of error an irreversible error k. In some embodiments, this set of updated actions may be referred to as x ta u ⁇
  • the sequences of steps or actions that preceded and included the error step k may be referred to as X head - Together, x tail and x head may define a component fi j .
  • SGD module 212 may solve for x tail using a stochastic gradient descent method.
  • x head - SGD module 212 may solve for x tail using a stochastic gradient descent method.
  • control module 106 may generate a predicted final quality metric for the component, based on the set of actions generated by SGD module 212.
  • GRU model 214 may be configured to predict a final quality metric for the component, ci L . based on x tail Q x head where 0 may represent a vector concatenation operator.
  • control module 106 may determine if the predicted final quality metric is within a threshold tolerance of the canonical final quality metric. For example, the final quality metric generated by GRU model 214 may be compared against a canonical final quality metric to determine if x taii is proper. If, at step 712, control module 106 determines that the predicted quality metric is within the threshold tolerance, at step 714, control module 106 may adjust the processing parameters of at least one of processing station k or one or more processing stations (k + 1). In some embodiments, control module 106 may interface with station controllers in a station 108 (k) or subsequent stations 108 (k + 1) to adjust their respective control and/or station parameters.
  • control module 106 may provide human manufacturers with updated instructions to be performed in at least one of a processing station k or one or more processing stations ( k + 1) of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
  • control module 106 determines that the predicted quality metric is not within the threshold tolerance
  • GRU model 214 may prompt SGD module 212 to generate a new x tail F° r example, method 700 may revert to step 708 for SGD module 212 to construct a new set of updated actions.
  • FIG. 8A illustrates a system bus computing system architecture 800, according to example embodiments.
  • One or more components of system 800 may be in electrical communication with each other using a bus 805.
  • System 800 may include a processor (e.g., one or more CPUs, GPUs or other types of processors) 810 and a system bus 805 that couples various system components including the system memory 815, such as read only memory (ROM) 820 and random access memory (RAM) 825, to processor 810.
  • System 800 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 810.
  • System 800 can copy data from memory 815 and/or storage device 830 to cache 812 for quick access by processor 810.
  • cache 812 may provide a performance boost that avoids processor 810 delays while waiting for data.
  • These and other modules can control or be configured to control processor 810 to perform various actions.
  • Other system memory 815 may be available for use as well.
  • Memory 815 may include multiple different types of memory with different performance characteristics.
  • Processor 810 may be representative of a single processor or multiple processors.
  • Processor 810 can include one or more of a general purpose processor or a hardware module or software module, such as service 1 832, service 2 834, and service 3 836 stored in storage device 830, configured to control processor 810, as well as a special-purpose processor where software instructions are incorporated into the actual processor design.
  • Processor 810 may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc.
  • a multi-core processor may be symmetric or asymmetric.
  • an input device 845 which can be any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth.
  • An output device 835 can also be one or more of a number of output mechanisms known to those of skill in the art.
  • multimodal systems can enable a user to provide multiple types of input to communicate with computing device 800.
  • Communications interface 840 can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.
  • Storage device 830 may be a non-volatile memory and can be a hard disk or other types of computer readable media that can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs) 825, read only memory (ROM) 820, and hybrids thereof.
  • RAMs random access memories
  • ROM read only memory
  • Storage device 830 can include services 832, 834, and 836 for controlling the processor 810. Other hardware or software modules are contemplated. Storage device 830 can be connected to system bus 805. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 810, bus 805, display 835, and so forth, to carry out the function.
  • FIG. 8B illustrates a computer system 850 having a chipset architecture, according to example embodiments.
  • Computer system 850 may be an example of computer hardware, software, and firmware that can be used to implement the disclosed technology.
  • System 850 can include one or more processors 855, representative of any number of physically and/or logically distinct resources capable of executing software, firmware, and hardware configured to perform identified computations.
  • processors 855 can communicate with a chipset 860 that can control input to and output from one or more processors 855.
  • chipset 860 outputs information to output 865, such as a display, and can read and write information to storage device 870, which can include magnetic media, and solid state media, for example.
  • Chipset 860 can also read data from and write data to RAM 875.
  • a bridge 880 for interfacing with a variety of user interface components 885 can be provided for interfacing with chipset 860.
  • Such user interface components 885 can include a keyboard, a microphone, touch detection and processing circuitry, a pointing device, such as a mouse, and so on.
  • inputs to system 850 can come from any of a variety of sources, machine generated and/or human generated.
  • Chipset 860 can also interface with one or more communication interfaces 890 that can have different physical interfaces.
  • Such communication interfaces can include interfaces for wired and wireless local area networks, for broadband wireless networks, as well as personal area networks.
  • Some applications of the methods for generating, displaying, and using the GUI disclosed herein can include receiving ordered datasets over the physical interface or be generated by the machine itself by one or more processors 855 analyzing data stored in storage 870 or 875. Further, the machine can receive inputs from a user through user interface components 885 and execute appropriate functions, such as browsing functions by interpreting these inputs using one or more processors 855.
  • example systems 800 and 850 can have more than one processor 810 or be part of a group or cluster of computing devices networked together to provide greater processing capability.
  • Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory (ROM) devices within a computer, such as CD-ROM disks readably by a CD-ROM drive, flash memory, ROM chips, or any type of solid-state non-volatile memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid state random-access memory) on which alterable information is stored.
  • ROM read-only memory
  • writable storage media e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid state random-access memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Manufacturing & Machinery (AREA)
  • Quality & Reliability (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • General Factory Administration (AREA)
  • Image Analysis (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

A manufacturing system is disclosed herein. The manufacturing system includes one or more stations, a monitoring platform, and a control module. Each station of the one or more stations is configured to perform at least one step in a multi-step manufacturing process for a component. The monitoring platform is configured to monitor progression of the component throughout the multi-step manufacturing process. The control module is configured to dynamically adjust processing parameters of each step of the multi-step manufacturing process to achieve a desired final quality metric for the component.

Description

SYSTEMS, METHODS, AND MEDIA FOR MANUFACTURING PROCESSES Cross-Reference to Related Applications [0001] This application claims priority to U.S. Provisional Application Serial No.63/040,792, filed June 18, 2020, which is hereby incorporated by reference in its entirety. This application is a continuation-in-part of U.S. Provisional Application Serial No.17/195,746, filed March 9, 2021, which claims priority to U.S. Provisional Application Serial No.62/986,987 filed March 9, 2020, which is hereby incorporated by reference in its entirety, and is a continuation-in-part of U.S. Application Serial No. 17/091,393, filed November 6, 2020, which claims priority to U.S. Provisional Application Serial No.62/931,448, filed November 6, 2019, U.S. Provisional Application Serial No.62/932,063, filed November 7, 2019, and U.S. Provisional Application Serial No.62/931,453, filed November 6, 2019, which are hereby incorporated by reference in their entireties. Field of Disclosure [0002] The present disclosure generally relates to a system, method, and media for manufacturing processes. Background [0003] Since the dawn of the industrial revolution in the 18th century, automation has governed the production of goods. Although today’s factories have fully embraced automation as a core principle – with robots performing many repeatable tasks in high-production environments – many assembly tasks continue to be performed by humans. These tasks are difficult to automate due to cost, risk of critical failure, or logistics of deploying a robotic system for a low-quantity production run. These production lines are overseen by standard process control and people management, such that an assembler is taught to perform a certain quality metric over time, or they are replaced by another operator. This process has remained largely unchanged since the advent of the assembly. Summary [0004] In some embodiments, a manufacturing system is disclosed herein. The manufacturing system includes one or more stations, a monitoring platform, and a control module. Each station is configured to perform at least one step in a multi-step manufacturing process for a component. The monitoring platform configured to monitor progression of the component throughout the multi-step manufacturing process. The control module configured to dynamically adjust processing parameters of a step of the multi-step manufacturing process to achieve a desired final quality metric for the component, the control module configured to perform operations. The operations include receiving image data of tooling of a first station of the one or more stations. The operations further include identifying a set of keypoints from the image data. The keypoints correspond to position information of the tooling during processing at the first station. The operations further include determining, by a machine learning model, a final quality metric for the component, based on the keypoints. The operations further include, based on the determining, assigning the component to a class of components based on a comparison between the final quality metric generated by the machine learning model and a canonical final quality metric for the component.
[0005] In some embodiments, a computer-implemented method for controlling a multi-step manufacturing process is disclosed herein. The multi-step manufacturing process involves one or more stations of a manufacturing system. Each station is configured to perform at least one step in a multi-step manufacturing process for a component. A computing system associated with the manufacturing system receives image data of tooling of a first station of the one or more stations. The computing system identifies a set of keypoints from the image data, the set of keypoints corresponding to position information of the tooling during processing at the first station. A machine learning model associated with the computing system determines a final quality metric for the component, based on the set of keypoints. Based on the determining, the computing system assigns the component to a class of components based on a comparison between the final quality metric generated by the machine learning model and a canonical final quality metric for the component.
[0006] In some embodiments, a manufacturing system is disclosed herein. The manufacturing system includes one or more stations, a monitoring platform, and a control module. Each station is configured to perform at least one step in a multi-step manufacturing process for a component. The monitoring platform configured to monitor progression of the component throughout the multi-step manufacturing process. T control module configured to dynamically adjust processing parameters of a step of the multi-step manufacturing process to achieve a desired final quality metric for the component, the control module configured to perform operations. The operations include receiving image data of tooling of a first station of the one or more stations. The operations further include identifying a set of keypoints from the image data. The keypoints correspond to position information of the tooling during processing at the first station. The operations further include determining, by a machine learning model, a final quality metric for the component, based on the keypoints. The operations further include determining that the final quality metric is not within a threshold tolerance from the final quality metric. The operations further include based on the determining, assigning the component to a class of components based on a comparison between the final quality metric generated by the machine learning model and a canonical final quality metric for the component. The operations further include, based on the assigning, determining that the class assigned to the component is not an acceptable class. The operations further include based on the determining, inferring positional information corresponding to the component at the first processing station. The operations further include based on the determining, generating an updated instruction set to be performed by at least one of the first processing station or a downstream station. The operations further include predicting, by a machine learning model, a final quality metric for the component based on the updated instruction set. The operations further include, based on the predicted final quality metric, providing the updated instruction set to at least one of the first processing station or the downstream station.
Brief Description of the Drawings
[0007] So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure may admit to other equally effective embodiments.
[0008] Figure 1 is a block diagram illustrating a manufacturing environment, according to example embodiments.
[0009] Figure 2 is a block diagram illustrating control module, according to exemplary embodiments.
[0010] Figure 3 is a block diagram illustrating an exemplary architecture of LSTM model, according to example embodiments.
[0011] Figure 4 is a block diagram visually illustrating the overall process flow of a feedback segment for tooling module, according to example embodiments.
[0012] Figure 5 is a block diagram illustrating architecture of GRU model, according to example embodiments.
[0013] Figure 6A is a flow diagram illustrating a method of correcting a multi-step manufacturing process, according to example embodiments. [0014] Figure 6B is a flow diagram illustrating a method of correcting a multi-step manufacturing process, according to example embodiments.
[0015] Figure 7 is a flow diagram illustrating a method of correcting a multi-step manufacturing process, according to example embodiments.
[0016] Figure 8A illustrates a system bus computing system architecture, according to example embodiments.
[0017] Figure 8B illustrates a computer system having a chipset architecture, according to example embodiments.
[0018] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation.
Detailed Description
[0019] Manufacturing processes may be complex and include raw materials being processed by different process stations (or “stations”) until a final component is produced. In some embodiments, each process station receives an input for processing and may output an intermediate output that may be passed along to a subsequent (downstream) process station for additional processing. In some embodiments, a final process station may receive an input for processing and may output the final component or, more generally, the final output.
[0020] In some embodiments, each station may include one or more tools/equipment that may perform a set of process steps. Exemplary process stations may include, but are not limited to, conveyor belts, injection molding presses, cutting machines, die stamping machines, extruders, computer numerical control (CNC) mills, grinders, assembly stations, three-dimensional printers, quality control stations, validation stations, and the like.
[0021] In some embodiments, operations of each process station may be governed by one or more process controllers. In some embodiments, each process station may include one or more process controllers that may be programmed to control the operation of the process station. In some embodiments, an operator, or control algorithms, may provide the station controller with station controller setpoints that may represent the desired value, or range of values, for each control value. In some embodiments, values used for feedback or feed forward in a manufacturing process may be referred to as control values. Exemplary control values may include, but are not limited to: speed, temperature, pressure, vacuum, rotation, current, voltage, power, viscosity, materials/resources used at the station, throughput rate, outage time, noxious fumes, and the like.
[0022] One or more techniques provided herein improves upon conventional processes by providing a system through which image and/or video data may be used to predict or forecast a final quality metric of a component. In some embodiments, the system may include a monitoring system configured to capture or record video and/or image data of the tooling of each processing node or station of manufacturing system. Based on the positioning of the tooling during a process step, the present system may be configured to predict or forecast a final quality metric of a component. If the predicted final quality metric falls outside of a range of acceptable values, the present system may generate and provide updated processing instructions to at least one of the current processing nodes k or stations k or to processing nodes (k + 1) and/or stations ( k + 1), in an attempt to correct for any errors in processing, such that a desired final quality metric may be achieved.
[0023] In some embodiments, the system may include a monitoring system configured to capture or record video and/or image data of a component at each processing node or station of the manufacturing system. Based on visual information about the component at an end of a processing node, the present system may be configured to predict or forecast a final quality metric of a component. If the predicted final quality metric falls outside of a range of acceptable values, the present system may generate and provide updated processing instructions to at least one of processing nodes k or stations k or to processing nodes ( k + 1) and/or stations ( k + 1), in an attempt to correct for any errors in processing, such that a desired final quality metric may be achieved.
[0024] In this manner, the present system is able to predict or forecast a final quality metric of a component, at any stage of the manufacturing process, without having to actually test the component. Such system is particularly useful for final quality metrics that would otherwise require destructive testing, or for which the final quality metric cannot be evaluated until processing is complete.
[0025] Figure 1 is a block diagram illustrating a manufacturing environment 100, according to example embodiments. Manufacturing environment 100 may include a manufacturing system 102, a monitoring platform 104, and a control module 106. Manufacturing system 102 may be broadly representative of a multi-step manufacturing system. In some embodiments, manufacturing system 102 may be representative of an assembly line system, where each processing station may be representative of a human worker. In some embodiments, manufacturing system 102 may be representative of a manufacturing system for use in additive manufacturing (e.g., 3D printing system). In some embodiments, manufacturing system 102 may be representative of a manufacturing system for use in subtractive manufacturing (e.g., CNC machining. In some embodiments, manufacturing system 102 may be representative of a manufacturing system for use in a combination of additive manufacturing and subtractive manufacturing. More generally, in some embodiments, manufacturing system 102 may be representative of a manufacturing system for use in a general manufacturing process.
[0026] Manufacturing system 102 may include one or more stations 108i-108n (generally, “station 108”). Each station 108 may be representative of a step and/or station in a multi-step manufacturing process. For example, each station 108 may be representative of a layer deposition operation in a 3D printing process (e.g., station 108i may correspond to layer 1, station IO82 may correspond to layer 2, etc.). In another example, each station 108 may correspond to a specific processing station. In another example, each station 108 may correspond to a specific human operator performing a specific task in an assembly line manufacturing process.
[0027] Each station 108 may include a process controller 114 and control logic 116. Each process controller 114i-114n may be programmed to control the operation of each respective station 108. In some embodiments, control module 106 may provide each process controller 114 with station controller setpoints that may represent the desired value, or range of values, for each control value. Control logic 116 may refer to the attributes/parameters associated with a station’s 108 process steps. In operation, control logic 116 for each station 108 may be dynamically updated throughout the manufacturing process by control module 106, depending on a current trajectory of a final quality metric.
[0028] Monitoring platform 104 may be configured to monitor each station 108 of manufacturing system 102. In some embodiments, monitoring platform 104 may be a component of manufacturing system 102. For example, monitoring platform 104 may be a component of a 3D printing system. In some embodiments, monitoring platform 104 may be independent of manufacturing system 102. For example, monitoring platform 104 may be retrofit onto an existing manufacturing system 102. In some embodiments, monitoring platform 104 may be representative of an imaging device configured to capture an image of a component or tooling (e.g., a worker or a process tool) at each step of a multi-step process. For example, monitoring platform 104 may be configured to capture an image of the component at each station 108 and/or an image of a component developing the component at each station 108 (e.g., tooling, human, etc.)· Generally, monitoring platform 104 may be configured to capture information associated with production of a component (e.g., an image, a voltage reading, a speed reading, etc.) and/or tool (e.g., hand position, tooling position, etc.), and provide that information, as input, to control module 106 for evaluation.
[0029] Control module 106 may be in communication with manufacturing system 102 and monitoring platform 104 via one or more communication channels. In some embodiments, the one or more communication channels may be representative of individual connections via the Internet, such as cellular or Wi-Fi networks. In some embodiments, the one or more communication channels may connect terminals, services, and mobile devices using direct connections, such as radio frequency identification (RFID), near-field communication (NFC), Bluetooth™, low-energy Bluetooth™ (BLE), Wi-Fi™, ZigBee™, ambient backscatter communication (ABC) protocols, USB, WAN, or LAN.
[0030] Control module 106 may be configured to control each process controller of manufacturing system 102. For example, based on information captured by monitoring platform 104, control module 106 may be configured to adjust process controls associated with a specific station 108. In some embodiments, control module 106 may be configured to adjust process controls of a specific station 108 based on a projected final quality metric.
[0031] Figure 2 is a block diagram illustrating control module 106, according to exemplary embodiments. Control module 106 may include a tooling module 202 and a component module 204.
[0032] Tooling module 202 may be configured to project a final quality metric of a specimen at a given stage of production based on image data obtained by monitoring platform 104. In operation, control module 106 may receive input from monitoring platform 104. In some embodiments, such input may take the form of an image or video of the tooling performing a subprocess at a given step of the multi-step manufacturing process. For example, the image or video data may include image or video data of a human’s hands while performing a specific subprocess of the multi-step manufacturing process. In another example, the image or video data may include image or video data of a three-dimensional printer performing depositing a specific layer of a multi-layer manufacturing process. Based on the input, control module 106 may project a final quality metric of the component. Depending on the projected final quality metric of the component, control module 106 may determine one or more actions to take in subsequent manufacturing steps in order to reach a desired or threshold final quality metric. For example, if the projected final quality metric falls outside of a range of acceptable values, control module 106 may take one or more actions to rectify the manufacturing process. In some embodiments, control module 106 may interface with station controllers in at least one of a current station 108 (k) or subsequent stations 108 (k + 1) to adjust their respective control and/or station parameters. In some embodiments, control module 106 may provide human manufacturers with updated instructions to be performed at each processing station ( k + 1) of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
[0033] Component module 204 may be configured to project a final quality metric of a specimen at a given stage of production based on image data obtained by monitoring platform 104. In some embodiments, component module 204 may receive input from monitoring platform 104. In some embodiments, such input may take the form of an image or video of the component at a given step of the multi-step manufacturing process. In some embodiments, component module 204 may receive inferred component data at a given step of the multi-step manufacturing process from tooling module 202. For example, tooling module 202 may infer information about the component at a given step of the multi-step manufacturing process based on the tooling image or video data and provide that inferred information to component module 204 as input. Based on the input, component module 204 may project a final quality metric of the component. Depending on the projected final quality metric of the component, component module 204 may determine one or more actions to take in subsequent manufacturing steps in order to reach a desired or threshold final quality metric. For example, if the projected final quality metric falls outside of a range of acceptable values, component module 204 may identify one or more actions to rectify the manufacturing process. In some embodiments, control module 106 may interface with station controllers in at least one of the current station 108 k or subsequent stations 108 (k + 1) to adjust their respective control and/or station parameters. In some embodiments, control module 106 may provide human manufacturers with updated instructions to be performed in at least one of a current processing station k or other processing stations (k + 1) of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
[0034] Each of tooling module 202 and component module 204 may include one or more software modules. The one or more software modules may be collections of code or instructions stored on a media (e.g., memory of computing systems associated with control module 106) that represent a series of machine instructions (e.g., program code) that implements one or more algorithmic steps. Such machine instructions may be the actual computer code the processor interprets to implement the instructions or, alternatively, may be a higher level of coding of the instructions that is interpreted to obtain the actual computer code. The one or more software modules may also include one or more hardware components. One or more aspects of an example algorithm may be performed by the hardware components (e.g., circuitry) itself, rather as a result of the instructions. Further, in some embodiments, each of tooling module 202 and component module 204 may be configured to transmit one or more signals among the components. In such embodiments, such signals may not be limited to machine instructions executed by a computing device.
[0035] In some embodiments, tooling module 202 and component module 204 may communicate via one or more local networks. Network may be of any suitable type, including individual connections via the Internet, such as cellular or Wi-Fi networks. In some embodiments, network may connect terminals, services, and mobile devices using direct connections, such as radio frequency identification (RFID), near-field communication (NFC), Bluetooth™, low-energy Bluetooth™ (BLE), Wi-Fi™, ZigBee™, ambient backscatter communication (ABC) protocols, USB, WAN, or LAN. Because the information transmitted may be personal or confidential, security concerns may dictate one or more of these types of connection be encrypted or otherwise secured. In some embodiments, however, the information being transmitted may be less personal, and therefore, the network connections may be selected for convenience over security.
[0036] Tooling module 202 may include acquisition system 206, extractor module 208, and prediction module 210. Generally, a multi -node or multi-station assembly environment, such as manufacturing system 102, may be represented, broadly as . where may represent the states of the component at all i nodes and where may represent that set of actions to be performed on the component by the tooling at all i nodes. Given a norm or canonical quality measurement, Vc, tooling module 202 may be configured to optimize the error in the assembly process with an estimated quality metric, may be within a threshold distance In some embodiments, tooling module 202 may estimate the state, which may be a numerical representation of a state of the component at all N nodes, and the actions, , which may represent the instructions or control values at each node.
[0037] Acquisition system 206 may be configured to receive image data of the assembly process at each node, N. In some embodiments, acquisition system 206 may receive the image data of the assembly process of monitoring platform 104. In some embodiments, for each node N, acquisition system 206 may receive V number of images, where V may represent the number of cameras of monitoring platform 104 that may record the assembly procedure at each node. Accordingly, each image of the V number of images may capture a different perspective of the component during processing. Following receipt of the image data, acquisition system 206 may be configured to extract a subset of images of frames. For example, acquisition system 206 may be configured to extract L number of images from the received image data. The extracted images may be referred to as landmark frames. Landmark frames may be those image frames that are in high motion. The extracted images may include those images or frames that include certain landmarks, /, of the component, where , and may represent the entire manufacturing process for the component.
[0038] Both minimal-motion (e.g., “landmark”) frames and maximal-motion (e.g., “high motion”) frames for a given operator may contain useful information for a classifier that is trying to correlate fmger-hand-arm data and flight performance data in a robust way across a number of operators. In some embodiments, an optical flow algorithm may be used to measure an amount of motion in any given frame. Acquisition system 206 may select those frames that contain the most motion.
[0039] Extractor module 208 may be configured to extract keypoints from the L number of images. For example, extractor module 208 may be configured to extract K number of keypoints, i.e., (x,y) pairs, per landmark /;. In other words, extractor module 208 may be configured to output K number of keypoints for a given input, As output, extractor module 208 may generate a single vector This vector may include landmark representations of K number of (x, y) pairs which may be represented by
[0040] In some embodiments, to generate the P, extractor module 208 may implement two separate algorithms: (1) a bounding box estimation; and (2) keypoint detection.
[0041] With respect to the bounding box estimation, given /, each landmark frame may be processed with a threshold image segmentation to generate a mask image for each tooling component. For example, in an embodiment in which the tooling is a human, extractor module 208 may generate a mask image for each of the user’s hands. In some embodiments, extractor module 208 may implement blob detection to locate components of the tooling. Using a human as an example, extractor module 208 may assume that the image always contains both the left and right hands of the user. When a frame fails to include both hands, extractor module 208 may assign the value with an arbitrary constant value, c.
[0042] With respect to keypoint detection, extractor module 208 may identify keypoints of the tooling based on the estimated bounding boxes. For example, with given input, extractor module 208 may estimate K number of points, pt, along with its confidence value, fy. In some embodiments, extractor module 208 may estimate, not only the points that are visible on the frame, but also points that may be occluded from the frame due to one or more of articulation, viewpoints, objects, or tool interactions. Because the objective may be to predict the quality measurement using the tracked keypoints, the non-estimated occluded points may be unique and important features representing the assembly process. Therefore, an occlusion threshold value, t0, may be derived from observation of cL value on occluded points in small randomly chosen subset of the landmark frames. Using t0, extractor module 208 may filter out the estimation that are c0 < t0. For those filtered points, extractor module 208 may assign them an arbitrary constant value, c. Regardless of the visibility of the tooling or keypoints on the frame, the output of extractor module 208 may include L * K(x, y) pairs for each component. [0043] In other words, extractor module 208 may be configured to assign a default basal level of confidence for occluded components and, therefore, estimate those keypoints so that a full set of keypoints may be available to measure discrepancies between observed trial tooling component positions and the canonical tooling component positions. In some embodiments, tooling module 202 may draw inferences about the state of the component at a given point in time (e.g., station i) and then output recommended modified-from-canonical subsequent actions to take to correct for the measured discrepancies.
[0044] Prediction module 210 may be configured to predict a final quality metric, VC. For example, prediction module 210 may be configured to predict a final quality metric, Vc. where given tooling tracking information for the L number of points-in-time, P, where P G may be gathered from each processing station 108. In some embodiments, prediction module 210 may implement a long short-term memory (LSTM) model to output the final quality metric. The LSTM model may allow prediction module 210 to overcome a vanishing gradient problem that is common in conventional recurrent neural networks. The vanishing gradient problem is the case where a model fails to relate the early weights of the neural network when the magnitude of the gradients are small for later layers. LSTM model eliminates this issue. [0045] In some embodiments, tooling module 202 may further include a classification module 215. Classification module 215 may be configured to classify a specimen into one or more classes based on the predicted final quality metric, Vc. For example, in operation, classification module 215 may receive a canonical or desired final quality metric, Vc. for the specimen. Classification module 215 may be configured to compare the predicted final quality metric, Vc, to the canonical or desired final quality metric, Vc, to generate a delta, Dn. Depending on the value of Dn, classification module 215 may sort the specimen into one or more classes. For example, assume that classification module 215 may sort the specimen into one of four classes: fail (class 1), fair (class 2), excellent (class 3), and over-qualify (class 4). Classification module 215 may sort the specimen based on associated class definitions. For example, the associated definitions may be:
[0046] As those skilled in the art recognize, the associated definitions may change based on operator or client preference.
[0047] In some embodiments, control module 106 may use the classification assigned to the component to determine whether the final quality metric is acceptable. For example, an operator or client may specify that components classified in Class 1 are not acceptable, while components classified in any of Classes 2-4 are acceptable. In another example, an operator or client may specify that components classified in Class 1 or Class 2 are not acceptable, while components classified in Class 3 or Class 4 are acceptable.
[0048] Figure 3 is a block diagram illustrating an exemplary architecture of LSTM model, according to example embodiments. As shown, LSTM model 300 may include three layers 302I-3023 (generally layer 302). As shown, each layer 302 may include one or more cells 304. In some embodiment, each cell’s 304 input may be p i where i E [0 , L — 1], Each cell in the LSTM may be defined as: where equation 1 decides whether to keep information from the previous cell or not; equation (2) decides which values to update; equation (3) updates the cell; and equation (4) decides which part to output. Equation (5) may filter the output parts so that LSTM model 300 only outputs what it is programmed to output.
[0049] In some embodiments, LSTM model 300 may include three layers 302 with 30 hidden sizes. In some embodiments, LSTM model 300 may be a sequence-to-one LSTM model. For training, the LI, mean absolute error (MAE), loss function: may be minimized using an Adam optimizer. In some embodiments, MAE may be used because the goal may be to minimize or reduce the magnitude of errors, regardless of the direction of the error.
[0050] Referring back to Figure 2, prediction module 210 may be trained using a canonical set of instructions, a, and multiple components, M. For example, using 10 node videos of M number of data, the input for prediction module 210 may be structured by first being pre- processed through acquisition system 206 and extractor module 208. In some embodiments, each component involved in training may be validated through a validation algorithm that checks their composition (e.g., shape). In some embodiments, the validation algorithm may compute a similarity index on the composition at the end of each node by comparing the composition with canonical compositions. As a result, the components for training are roughly similar to the canonical composition.
[0051] In some embodiments, for output, the corresponding physical components may be tested in a controlled environment to measure their quality metric. Using the prepared input data with corresponding output data, prediction module 210 may be trained, for example, with a portion of M being training data and another portion of M being validation data. Once trained, prediction module 210 may be able to predict a quality measurement of a component at a given processing step, based on image data of the tooling.
[0052] Now referring to component module 204, component module 204 may include stochastic gradient descent (SGD) module 212, gated recurrent unit (GRU) model 214, and simulation module 216. For purposes of this discussion, a partial construct of a component may be defined as ak. where step k introduces an irreversible error in the manufacturing process, and steps k + 1, . . . , N are not yet defined. Component module 204 may be configured to identify an optimal corrective sequence of remaining actions where r and t may correspond to specific operations to be performed on the component, at least one of the current processing station or each subsequent processing station (k + 1 to N) of manufacturing system 102. More generally, any component a may be defined as a sequence of all operations performed at each processing station 1 ... N of manufacturing system. Mathematically, At each manufacturing step i = 1, . . . , 10. the virtual representation system may represent the component in Euclidean space (e.g., R3) as a set of connected surface and a set of connected points uniformly distributed along the outer contour of each surface. In some embodiments, the virtual representation functions generating these representations may be referred to as respectively. In some embodiments, component module 204 may be configured to correct ak toward a particular, canonical component,
[0053] Simulation module 216 may be configured to simulate or generate a surface model for a given component q For example, simulation module 216 may receive tooling information from tooling module 202. Based on the keypoints generated by tooling module 202, simulation module 216 may be configured to generate a surface model representing a state of the component aL at a specific process step i. In some embodiments, the surface model may be represented as S(aj). In some embodiments, simulation module 216 may further be configured to generate or estimate a quality metric of the component fy. From the surface model, simulation module 216 may be configured to generate a points model, P(dj), representing specific coordinates of the component, a t. For example, from surface model S(ai). simulation module 216 may create points model, P(a;), by placing a number of points uniformly spaced around a bounding contour of each surface in S(aj). In some embodiments, S(a;) may be used to simulate performance of the artifact fy.
[0054] SGD module 212 may receive the points model, P(dj), from simulation module 216. SGD module 212 may determine whether an irreversible error, k, has occurred by comparing the points model, P(a;), at step i to a canonical points model, P(a*) of a canonical component, a*. An irreversible error may be defined to be a measurably significant structural deviation from a canonical component at step k. SGD module 212 may be configured to detect an irreversible error by taking a Hausdorff distance. For example, SGD module 212 may match a processing step of current component a to canonical component a* based on respective Euclidean point sets. Mathematically, SGD module 212 may be configured to compute the Hausdorff distance between For example, where d(x,y ) may be the Euclidean distance between x and y, and the undirected Hausdorff distance may be:
[0055] An irreversible error may be present when the Hausdorff distance between the current component a; and the canonical component a* exceeds some threshold tolerance. For example, SGD module 212 may determine that an error occurs at step k when: where tH is some suitably defined tolerance threshold.
[0056] Assuming an error is present, SGD module 212 may be configured to construct a set of updated actions given a set of actions up to a point of error an irreversible error k. In some embodiments, this set of updated actions may be referred to as Xtaii · The sequences of steps or actions that preceded and included the error step k may be referred to as xhead. Together, xtail and xhead may define a component Based on xhead , SGD module 212 may solve for xtail using a stochastic gradient descent method.
[0057] GRU model 214 may be configured to predict a final quality metric for the component, . where may represent a vector concatenation operator. The final quality metric generated by GRU model 214 may be compared against a canonical final quality metric to determine if xtaii is proper. For example, assuming that the combination of xtau and xhead yields a final quality metric that falls outside of a range of acceptable values, GRU model 214 may instruct SGD module 212 to generate an updated sequence of updated actions for further evaluation.
[0058] Figure 5 is a block diagram illustrating architecture of GRU model 214, according to example embodiments. As shown, GRU model 214 may include N GRU cells 502I-502N (generally, GRU cell 502), with each GRU cell 502 corresponding to a respective processing station 108. Each GRU cell 502 may include an (ru tL) input pair and a hidden state output hi of a predetermined size. Together, these input pairs ( ru tL) may define a given component an In some embodiments, each GRU cell 502 may be defined by:
where may be the hidden state at time may be the input at time t, and represent the reset, update, and new gates at time t, respectively.
[0059] GRU model 214 may be trained to generate weights corresponding thereto. For example, GRU model 214 may be trained iteratively to bias GRU model 214 toward solving a specific subproblem. In some embodiments, during a first iteration, GRU model 214 may generate a plurality (e.g., several hundred to several thousand) best predictions (one for each possible error at step k ) to complete a given component ak at steps k + I, . , ., N, along with the corresponding predicted quality metrics corresponding to each prediction.
[0060] In some embodiments, all of the predicted completions may be rendered in a virtual representation system, their stepwise Hausdorff distances computed, and their rendered surfaces simulated, to obtain distance measures between the generated predictions and the canonical. In some embodiments, the loss values between the canonical and predicted quality metric measures may be computed and fed back into the GRU model 214, whose weights may be adjusted via backpropagation, producing a second iteration. This process may continue until a desired error threshold is obtained.
[0061] In some embodiments, GRU model 214 may be trained on a prepared or synthetic data set. For example, simulator module 216 or another simulation system (e.g., a CAD model) may be used to generate a training set for training of GRU model 214. For example, a simulation system may generate, for example, a set of prepared components (e.g., 100,000 prepared components) to represent many unbiased randomized trials of actions that may cover the full admissible range of all parameter values. In some embodiments, the set prepared components may also include biased randomized trials of action, in addition to the unbiased randomized trials. This set may be referred to as P(t/(0,1), 100000). Each p E P(U( 0,1), 100000) may obtain its defining parameter values by sampling from the uniform distribution 1/(0, 1).
[0062] In some embodiments, for example, a set of 50,000 prepared components may be generated to represent many trials of processing a component to replicate the canonical component a*. This may be referred to as the set Each 50000) may obtain its defining parameter values by a statistical sampling process. In some embodiments, the ith action may centrally tend to the corresponding canonically- defined fold and may vary according to an appropriately parameterized Gaussian distribution. The prepared component data set may be defined as P =
[0063] Figure 4 is a block diagram visually illustrating the overall process flow of a feedback segment 400 for tooling module 202, according to example embodiments.
[0064] As shown, feedback segment 400 may include a first portion 402, a second portion 404, and a third portion 406. During first portion 402, tooling module 202 may perform an acquisition process, such as that carried out, at least partially, by acquisition system 206. [0065] As shown, at block 408, manufacturing system 102 may receive processing instructions for processing a component. In some embodiments, manufacturing system 102 may receive processing instructions on a per station 108 basis. For example, each station 108 of manufacturing system 102 may receive independent processing instructions. In some embodiments, processing instructions may include control values that define attributes (e.g., temperature, pressure, etc.) of a station 108 for manufacturing. In some embodiments, processing instructions may include videos or images that visually illustrate to a human operator how to perform a specific processing step at a processing node of the manufacturing process.
[0066] At block 410, acquisition system 206 may receive image data from monitoring platform 104. In some embodiments, for each node N, acquisition system 206 may receive V number of images, where V may represent the number of cameras of monitoring platform 104 that may record the assembly procedure at each node. Following receipt of the image data, at block 412, acquisition system 206 may be configured to extract a subset of images of frames. For example, acquisition system 206 may be configured to extract L number of images from the received image data. The extracted images may be referred to as landmark frames. The extracted images may include those images or frames that include certain landmarks and may represent the entire manufacturing process for the component.
[0067] Second portion 404 may correspond to operations performed by extractor module 208. As shown, extractor module 208 may receive at least the extracted images from acquisition system 206. Extractor module 208 may be configured to extract keypoints from the L number of images. For example, extractor module 208 may be configured to extract K number of keypoints, i.e., (x,y) pairs, per landlark /;.
[0068] At block 414, extractor module 208 may perform bounding box estimation. For example, given /, each landmark frame may be processed with a threshold image segmentation to generate a mask image for each tooling component. For example, as shown, in an embodiment in which the tooling is a human, extractor module 208 may generate a mask image for each of the user’s hands.
[0069] At block 416, extractor module 208 may perform keypoint detection, given the bounding box estimation. For example, with given input, extractor module 208 may estimate K number of points, p;, along with its confidence value, cL. In some embodiments, extractor module 208 may estimate, not only the points that are visible on the frame, but also points that may be occluded from the frame due to one or more of articulation, viewpoints, objects, or tool interactions.
[0070] Third portion 406 may correspond to operations performed by prediction module 210. As shown, at block 418, prediction module 210 may receive keypoint information from extractor module 208 and may be configured to predict a final quality metric, Vc. In some embodiments, prediction module 210 may implement a long short-term memory (LSTM) model to output the final quality metric.
[0071] Figure 6A is a flow diagram illustrating a method 600 of correcting a multi-step manufacturing process, according to example embodiments. Method 600 may begin at step 602.
[0072] At step 602, an instruction set may be provided to manufacturing system 102. The instruction set may be representative of a set of instructions for a manufacturing process to be carried out by manufacturing system 102. In some embodiments, the instruction set may be provided to each station 108. For example, each canonical instruction set provided to each respective station 108 may define the processing parameters for a specific manufacturing step. In another example, each canonical instruction set may be a video of discrete steps to be performed by a human actor at a specific processing node or station 108.
[0073] At step 604, control module 106 may receive image data of the tooling (e.g., station 108) from monitoring platform 104. For example, acquisition system 206 may receive image data of the assembly process at a respective processing node. In some embodiments, acquisition system 206 may receive V number of images, where V may represent the number of cameras of monitoring platform 104 that may record the assembly procedure at a specific procession station 108. Accordingly, each image of the V number of images may capture a different perspective of the tooling during processing.
[0074] At step 606, control module 106 may extract a subset of images from the obtained image data. For example, following receipt of the image data, acquisition system 206 may be configured to extract a subset of images of frames. For example, acquisition system 206 may be configured to extract L number of images, i.e., landmark frames, from the received image data. The extracted images may be referred to as landmark frames. Landmark frames may be those image frames that are in high motion. The extracted images may include those images or frames that include certain landmarks , of the component, where and may represent the entire manufacturing process for the component.
[0075] At step 608, control module 106 may extract one or more keypoints of the tooling from the landmark frames. For example, extractor module 208 may extract keypoints from the L number of images. Extractor module 208 may identify or extract K number of keypoints for a given input, As output, extractor module 208 may generate a single vector This vector may include landmark representations of K number of (x, y) pairs which may be represented by where
[0076] At step 610, control module 106 may predict a final quality metric for the component based on at least the identified keypoints. In some embodiments, prediction module 210 may implement a long short-term memory (LSTM) model to output the final quality metric.
[0077] At step 612, control module 106 may compare the final quality metric to a desired quality metric. If, at step 612, control module 106 determines that the final quality metric is a threshold tolerance of the desired quality metric, the manufacturing process may proceed to the next processing station or node (e.g., step 616), in accordance with the original instruction set. If, however, at step 612, control module 106 determines that the final quality metric is not within a threshold tolerance of the desired quality metric, then, at step 614 control module 106 may adjust the processing parameters of at least one of the processing station k or one or more processing stations k + 1, where k may represent the current processing station and k+1 may represent other processing stations downstream of the current processing station. In some embodiments, control module 106 may interface with station controllers in a current station 108 (k) and/or subsequent stations 108 ( k + 1) to adjust their respective control and/or station parameters. In some embodiments, control module 106 may provide human manufacturers with updated instructions to be performed in at least one of a processing station k or one or more processing station ( k + l)of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
[0078] Figure 6B is a flow diagram illustrating a method 650 of correcting a multi-step manufacturing process, according to example embodiments. Method 650 may begin at step 652.
[0079] At step 652, an instruction set may be provided to manufacturing system 102. The instruction set may be representative of a set of instructions for a manufacturing process to be carried out by manufacturing system 102. In some embodiments, the instruction set may be provided to each station 108. For example, each canonical instruction set provided to each respective station 108 may define the processing parameters for a specific manufacturing step. In another example, each canonical instruction set may be a video of discrete steps to be performed by a human actor at a specific processing node or station 108.
[0080] At step 654, control module 106 may receive image data of the tooling (e.g., station 108) from monitoring platform 104. For example, acquisition system 206 may receive image data of the assembly process at a respective processing node. In some embodiments, acquisition system 206 may receive V number of images, where V may represent the number of cameras of monitoring platform 104 that may record the assembly procedure at a specific procession station 108. Accordingly, each image of the V number of images may capture a different perspective of the tooling during processing.
[0081] At step 656, control module 106 may extract a subset of images from the obtained image data. For example, following receipt of the image data, acquisition system 206 may be configured to extract a subset of images of frames. For example, acquisition system 206 may be configured to extract L number of images, i.e., landmark frames, from the received image data. The extracted images may be referred to as landmark frames. Landmark frames may be those image frames that are in high motion and may contain the most meaningful features that clearly captures the progress of the manufacturing process. The extracted images may include those images or frames that include certain landmarks, of the component, where , and may represent the entire manufacturing process for the component.
[0082] At step 658, control module 106 may extract one or more keypoints of the tooling from the landmark frames. For example, extractor module 208 may extract keypoints from the L number of images. Extractor module 208 may identify or extract K number of keypoints for a given input, As output, extractor module 208 may generate a single vector This vector may include landmark representations of K number of (x, y) pairs which may be represented by where
[0083] At step 660, control module 106 may predict a final quality metric for the component based on at least the identified keypoints. In some embodiments, prediction module 210 may implement a long short-term memory (LSTM) model to output the estimate of the final quality metric.
[0084] At step 662, control module 106 may classify the component based on the final quality metric. For example, classification module 215 may classify the component into one or more classes based on the predicted final quality metric, Vc. To classify the component, tooling module 202 may receive a canonical or desired final quality metric, Vc. for the component. Tooling module 202 may be configured to compare the predicted final quality metric, Vc. to the canonical or desired final quality metric, Vc, to generate a delta, D^. Depending on the value of D^, classification module 215 may sort the component into one or more classes.
[0085] In some embodiments, method 650 may further include operations 664-668.
[0086] At step 614, control module 106 may determine if the class assigned to the component is an acceptable class. If, at step 614, control module 106 determines that component has been assigned to an acceptable class, the manufacturing process may proceed to the next processing station or node (e.g., step 616), in accordance with the original instruction set. If, however, at step 614, control module 106 determines that component is assigned to an unacceptable class, then, at step 618 control module 106 may adjust the processing parameters of at least one of a processing station k or one or more processing stations (k + 1). In some embodiments, control module 106 may interface with station controllers in at least one of current station 108 or subsequent stations 108 to adjust their respective control and/or station parameters. In some embodiments, control module 106 may provide human manufacturers with updated instructions to be performed in at least one of a processing station k or one or more processing stations ( k + 1) of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
[0087] Figure 7 is a flow diagram illustrating a method 700 of correcting a multi-step manufacturing process, according to example embodiments. Method 700 may begin at step 702. [0088] At step 702, an instruction set may be provided to manufacturing system 102. The instruction set may be representative of a set of instructions for a manufacturing process to be carried out by manufacturing system 102. In some embodiments, the instruction set may be provided to each station 108. For example, each canonical instruction set provided to each respective station 108 may define the processing parameters for a specific manufacturing step. In another example, each canonical instruction set may be a video of discrete steps to be performed by a human actor at a specific processing node or station 108.
[0089] At step 704, control module 106 may identify information corresponding to a component at a respective processing node. In some embodiments, simulation module 216 may receive tooling information from tooling module 202. Based on the keypoints generated by tooling module 202, simulation module 216 may be configured to generate a surface model representing a state of the component aL at a specific process step i. In some embodiments, the surface model may be represented as In some embodiments, simulation module 216 may further be configured to generate or estimate a quality metric of the component aL. From the surface model, simulation module 216 may be configured to generate a points model, P representing specific coordinates of the component,
[0090] At step 706, control module 106 may determine whether an irreversible error has occurred. For example, SGD module 212 may receive points model, from simulation module 216. SGD module 212 may determine whether an irreversible error, k, has occurred by comparing the points model, at step i to a canonical points model, P of a canonical component, SGD module 212 may be configured to detect an irreversible error by taking a Hausdorff distance between the points model and the corresponding canonical points model. An irreversible error may be present when the Hausdorff distance between the current component and the canonical component , at the respective processing station or node, exceeds some threshold tolerance.
[0091] If at step 706, control module 106 determines that an irreversible error has not occurred, then the manufacturing process may proceed to the next processing station or node (step 705), in accordance with the original instruction set. If, however, at step 706, control module 106 determines that an irreversible error has occurred, then method 700 proceed to step 708.
[0092] At step 708, control module 106 may generate an updated set of actions to correct the irreversible error. SGD module 212 may construct a set of updated actions given a set of actions up to a point of error an irreversible error k. In some embodiments, this set of updated actions may be referred to as xtau· The sequences of steps or actions that preceded and included the error step k may be referred to as Xhead- Together, xtail and xhead may define a component fij. Based on xhead · SGD module 212 may solve for xtail using a stochastic gradient descent method. The sequences of steps or actions that preceded and included the error step k may be referred to as Xhead- Together, xtail and Xhead may define a component at. Based on xhead - SGD module 212 may solve for xtail using a stochastic gradient descent method.
[0093] At step 710, control module 106 may generate a predicted final quality metric for the component, based on the set of actions generated by SGD module 212. For example, GRU model 214 may be configured to predict a final quality metric for the component, ciL. based on xtail Q x head where 0 may represent a vector concatenation operator.
[0094] At step 712, control module 106 may determine if the predicted final quality metric is within a threshold tolerance of the canonical final quality metric. For example, the final quality metric generated by GRU model 214 may be compared against a canonical final quality metric to determine if xtaii is proper. If, at step 712, control module 106 determines that the predicted quality metric is within the threshold tolerance, at step 714, control module 106 may adjust the processing parameters of at least one of processing station k or one or more processing stations (k + 1). In some embodiments, control module 106 may interface with station controllers in a station 108 (k) or subsequent stations 108 (k + 1) to adjust their respective control and/or station parameters. In some embodiments, control module 106 may provide human manufacturers with updated instructions to be performed in at least one of a processing station k or one or more processing stations ( k + 1) of a production line. These adjustments may aid in correcting the manufacturing process, such that the final quality metric may be within the range of acceptable quality metrics.
[0095] If, however, at step 712, control module 106 determines that the predicted quality metric is not within the threshold tolerance, GRU model 214 may prompt SGD module 212 to generate a new xtailr example, method 700 may revert to step 708 for SGD module 212 to construct a new set of updated actions.
[0096] Figure 8A illustrates a system bus computing system architecture 800, according to example embodiments. One or more components of system 800 may be in electrical communication with each other using a bus 805. System 800 may include a processor (e.g., one or more CPUs, GPUs or other types of processors) 810 and a system bus 805 that couples various system components including the system memory 815, such as read only memory (ROM) 820 and random access memory (RAM) 825, to processor 810. System 800 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 810. System 800 can copy data from memory 815 and/or storage device 830 to cache 812 for quick access by processor 810. In this way, cache 812 may provide a performance boost that avoids processor 810 delays while waiting for data. These and other modules can control or be configured to control processor 810 to perform various actions. Other system memory 815 may be available for use as well. Memory 815 may include multiple different types of memory with different performance characteristics. Processor 810 may be representative of a single processor or multiple processors. Processor 810 can include one or more of a general purpose processor or a hardware module or software module, such as service 1 832, service 2 834, and service 3 836 stored in storage device 830, configured to control processor 810, as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor 810 may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.
[0097] To enable user interaction with the computing device 800, an input device 845 which can be any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device 835 can also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input to communicate with computing device 800. Communications interface 840 can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed. [0098] Storage device 830 may be a non-volatile memory and can be a hard disk or other types of computer readable media that can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs) 825, read only memory (ROM) 820, and hybrids thereof.
[0099] Storage device 830 can include services 832, 834, and 836 for controlling the processor 810. Other hardware or software modules are contemplated. Storage device 830 can be connected to system bus 805. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 810, bus 805, display 835, and so forth, to carry out the function.
[0100] Figure 8B illustrates a computer system 850 having a chipset architecture, according to example embodiments. Computer system 850 may be an example of computer hardware, software, and firmware that can be used to implement the disclosed technology. System 850 can include one or more processors 855, representative of any number of physically and/or logically distinct resources capable of executing software, firmware, and hardware configured to perform identified computations. One or more processors 855 can communicate with a chipset 860 that can control input to and output from one or more processors 855. In this example, chipset 860 outputs information to output 865, such as a display, and can read and write information to storage device 870, which can include magnetic media, and solid state media, for example. Chipset 860 can also read data from and write data to RAM 875. A bridge 880 for interfacing with a variety of user interface components 885 can be provided for interfacing with chipset 860. Such user interface components 885 can include a keyboard, a microphone, touch detection and processing circuitry, a pointing device, such as a mouse, and so on. In general, inputs to system 850 can come from any of a variety of sources, machine generated and/or human generated.
[0101] Chipset 860 can also interface with one or more communication interfaces 890 that can have different physical interfaces. Such communication interfaces can include interfaces for wired and wireless local area networks, for broadband wireless networks, as well as personal area networks. Some applications of the methods for generating, displaying, and using the GUI disclosed herein can include receiving ordered datasets over the physical interface or be generated by the machine itself by one or more processors 855 analyzing data stored in storage 870 or 875. Further, the machine can receive inputs from a user through user interface components 885 and execute appropriate functions, such as browsing functions by interpreting these inputs using one or more processors 855.
[0102] It can be appreciated that example systems 800 and 850 can have more than one processor 810 or be part of a group or cluster of computing devices networked together to provide greater processing capability.
[0103] While the foregoing is directed to embodiments described herein, other and further embodiments may be devised without departing from the basic scope thereof. For example, aspects of the present disclosure may be implemented in hardware or software or a combination of hardware and software. One embodiment described herein may be implemented as a program product for use with a computer system. The program(s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory (ROM) devices within a computer, such as CD-ROM disks readably by a CD-ROM drive, flash memory, ROM chips, or any type of solid-state non-volatile memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid state random-access memory) on which alterable information is stored. Such computer-readable storage media, when carrying computer-readable instructions that direct the functions of the disclosed embodiments, are embodiments of the present disclosure. [0104] It will be appreciated to those skilled in the art that the preceding examples are exemplary and not limiting. It is intended that all permutations, enhancements, equivalents, and improvements thereto are apparent to those skilled in the art upon a reading of the specification and a study of the drawings are included within the true spirit and scope of the present disclosure. It is therefore intended that the following appended claims include all such modifications, permutations, and equivalents as fall within the true spirit and scope of these teachings.

Claims

1. A manufacturing system, comprising: one or more stations, each station configured to perform at least one step in a multi-step manufacturing process for a component; a monitoring platform configured to monitor progression of the component throughout the multi-step manufacturing process; and a control module configured to dynamically adjust processing parameters of a step of the multi-step manufacturing process to achieve a desired final quality metric for the component, the control module configured to perform operations, comprising: receiving image data of tooling of a first station of the one or more stations; determining, by a first machine learning model, a final quality metric for the component, based on the image data; determining that the final quality metric is outside an acceptable range of the desired final quality metric; based on the determining, generating, by the first machine learning model, an updated instruction set to be performed by at least one of the first station or a downstream station to achieve the desired final quality metric; predicting, by a second machine learning model, an updated final quality metric for the component based on the updated instruction set; and based on the updated final quality metric, providing the updated instruction set to the first station and the downstream station.
2. The manufacturing system of claim 1, wherein the second machine learning model is trained with a training set comprising a synthetic set of data.
3. The manufacturing system of claim 1, wherein the final quality metric cannot be measured until processing of the component is complete.
4. The manufacturing system of claim 1, further comprising: determining, based on positional information of the component, that an irreversible error is not present.
5. The manufacturing system of claim 1, further comprising: identifying a set of keypoints from the image data, the set of keypoints corresponding to position information of the tooling during processing at the first station.
6. The manufacturing system of claim 5, wherein the set of keypoints are used by the first machine learning model to determine the final quality metric.
7. The manufacturing system of claim 5, further comprising: comparing the set of keypoints corresponding to coordinates of the component to a canonical set of keypoints corresponding to a canonical component.
8. A method for dynamically adjusting processing parameters of a step of a multi-step manufacturing process to achieve a desired final quality metric for a component undergoing the multi-step manufacturing process, comprising: receiving, by a computing system, image data of tooling of a first station of one or more stations involved in the multi-step manufacturing process; determining, by a first machine learning model of the computing system, a final quality metric for the component, based on the image data; determining, by a computing system, that the final quality metric is outside an acceptable range of the desired final quality metric; based on the determining, generating, by the first machine learning model, an updated instruction set to be performed by at least one of the first station or a downstream station to achieve the desired final quality metric; predicting, by a second machine learning model of the computing system, an updated final quality metric for the component based on the updated instruction set; and based on the updated final quality metric, providing, by a computing system, the updated instruction set to the first station and the downstream station.
9. The method of claim 8, wherein the second machine learning model is trained with a training set comprising a synthetic set of data.
10. The method of claim 8, wherein the final quality metric cannot be measured until processing of the component is complete.
11. The method of claim 8, further comprising: determining, based on positional information of the component, that an irreversible error is not present.
12. The method of claim 8, further comprising: identifying a set of keypoints from the image data, the set of keypoints corresponding to position information of the tooling during processing at the first station.
13. The method of claim 12, wherein the set of keypoints are used by the first machine learning model to determine the final quality metric.
14. The method of claim 12, further comprising: comparing the set of keypoints corresponding to coordinates of the component to a canonical set of keypoints corresponding to a canonical component.
15. A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by a computing system, causes a processor to perform operations for dynamically adjusting processing parameters of a step of a multi-step manufacturing process to achieve a desired final quality metric for a component undergoing the multi-step manufacturing process, comprising: receiving, by a computing system, image data of tooling of a first station of one or more stations involved in the multi-step manufacturing process; determining, by a first machine learning model of the computing system, a final quality metric for the component, based on the image data; determining, by a computing system, that the final quality metric is outside an acceptable range of the desired final quality metric; based on the determining, generating, by the first machine learning model, an updated instruction set to be performed by at least one of the first station or a downstream station to achieve the desired final quality metric; predicting, by a second machine learning model of the computing system, an updated final quality metric for the component based on the updated instruction set; and based on the updated final quality metric, providing, by a computing system, the updated instruction set to the first station and the downstream station.
16. The non-transitory computer readable medium of claim 15, wherein the second machine learning model is trained with a training set comprising a synthetic set of data.
17. The non-transitory computer readable medium of claim 15, wherein the final quality metric cannot be measured until processing of the component is complete.
18. The non-transitory computer readable medium of claim 15, further comprising: determining, based on positional information of the component, that an irreversible error is not present.
19. The non-transitory computer readable medium of claim 15, further comprising: identifying a set of keypoints from the image data, the set of keypoints corresponding to position information of the tooling during processing at the first station.
20. The non-transitory computer readable medium of claim 19, wherein the set of keypoints are used by the first machine learning model to determine the final quality metric.
EP21825727.7A 2020-06-18 2021-06-18 Systems, methods, and media for manufacturing processes Pending EP4168941A4 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202063040792P 2020-06-18 2020-06-18
US17/091,393 US20210132593A1 (en) 2019-11-06 2020-11-06 Systems, Methods, and Media for Manufacturing Processes
US17/195,746 US12125236B2 (en) 2021-03-09 Systems, methods, and media for manufacturing processes
PCT/US2021/038085 WO2021257988A1 (en) 2020-06-18 2021-06-18 Systems, methods, and media for manufacturing processes

Publications (2)

Publication Number Publication Date
EP4168941A1 true EP4168941A1 (en) 2023-04-26
EP4168941A4 EP4168941A4 (en) 2024-10-02

Family

ID=79171624

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21825727.7A Pending EP4168941A4 (en) 2020-06-18 2021-06-18 Systems, methods, and media for manufacturing processes

Country Status (5)

Country Link
EP (1) EP4168941A4 (en)
JP (1) JP7517740B2 (en)
KR (1) KR20230025698A (en)
CN (1) CN115943428A (en)
WO (1) WO2021257988A1 (en)

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7130710B2 (en) * 2002-08-07 2006-10-31 Kimberly-Clark Worldwide, Inc. System and method for tracking and exploiting per station information from a multiple repeat manufacturing device
JP2004178388A (en) 2002-11-28 2004-06-24 Toppan Printing Co Ltd Quality control system in post-processing
US7526463B2 (en) * 2005-05-13 2009-04-28 Rockwell Automation Technologies, Inc. Neural network using spatially dependent data for controlling a web-based process
JP5012660B2 (en) 2008-05-22 2012-08-29 住友金属工業株式会社 Product quality prediction and control method
JP2016157357A (en) 2015-02-26 2016-09-01 株式会社日立製作所 Operator quality control method and operator quality management device
US10727057B2 (en) * 2018-03-20 2020-07-28 Tokyo Electron Limited Platform and method of operating for integrated end-to-end self-aligned multi-patterning process
WO2019182913A1 (en) * 2018-03-20 2019-09-26 Tokyo Electron Limited Self-aware and correcting heterogenous platform incorporating integrated semiconductor processing modules and method for using same
US10518480B2 (en) * 2018-04-02 2019-12-31 Nanotronics Imaging, Inc. Systems, methods, and media for artificial intelligence feedback control in additive manufacturing
US10481579B1 (en) * 2019-02-28 2019-11-19 Nanotronics Imaging, Inc. Dynamic training for assembly lines
KR20220005434A (en) 2019-04-19 2022-01-13 나노트로닉스 이미징, 인코포레이티드 Correct assembly errors for assembly lines

Also Published As

Publication number Publication date
JP7517740B2 (en) 2024-07-17
WO2021257988A1 (en) 2021-12-23
CN115943428A (en) 2023-04-07
JP2023530944A (en) 2023-07-20
KR20230025698A (en) 2023-02-22
EP4168941A4 (en) 2024-10-02

Similar Documents

Publication Publication Date Title
US20210132593A1 (en) Systems, Methods, and Media for Manufacturing Processes
US20210311440A1 (en) Systems, Methods, and Media for Manufacturing Processes
US11117328B2 (en) Systems, methods, and media for manufacturing processes
US20210263495A1 (en) Systems, Methods, and Media for Manufacturing Processes
US20210138735A1 (en) Systems, Methods, and Media for Manufacturing Processes
EP4118512A1 (en) Systems, methods, and media for manufacturing processes
US12125236B2 (en) Systems, methods, and media for manufacturing processes
US20210192779A1 (en) Systems, Methods, and Media for Manufacturing Processes
JP7517740B2 (en) Systems, methods, and media for manufacturing processes
TWI854583B (en) Systems and methods for manufacturing processes
WO2021050508A1 (en) Systems, methods, and media for manufacturing processes
Chung Process Monitoring and Control of Advanced Manufacturing based on Physics-Assisted Machine Learning

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20221028

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06N0003080000

Ipc: G05B0019418000

A4 Supplementary search report drawn up and despatched

Effective date: 20240903

RIC1 Information provided on ipc code assigned before grant

Ipc: G06N 3/084 20230101ALN20240828BHEP

Ipc: G06N 3/044 20230101ALI20240828BHEP

Ipc: G05B 19/042 20060101ALI20240828BHEP

Ipc: G05B 19/418 20060101AFI20240828BHEP