JP2023040079A - Assembly error correction for assembly lines - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a computational model that utilizes machine learning for detecting errors during a manual assembly process, and for determining a sequence of steps to complete the manual assembly process in order to mitigate the detected errors.

SOLUTION: A method comprises evaluating a target object at a step of an assembly process where an error is detected in the light of a nominal object so as to obtain a comparison, and, based on this comparison, obtaining a sequence of steps for completion of the assembly process of the target object. The assembly instructions for creating the target object are adjusted based on this sequence of steps.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of U.S. patent application Ser. Application No. 16/587,366, filed February 28, 2019, is U.S. Application No. 16/289,422 entitled "DYNAMIC TRAINING FOR ASSEMBLY LINES," now U.S. Patent No. 10,481, This is a continuation application of '579. Further, this application is subject to U.S. Provisional Application Serial No. 62/836,192 entitled "A COMPUTATION MODEL FOR DECISION-MAKING AND ASSEMBLY OPTIMIZATION IN MANUFACTURING," filed on April 19, 2019, November 6, 2019. U.S. Provisional Application No. 62/931,448, entitled "A COMPUTATION MODEL FOR DECISION-MAKING AND ASSEMBLY OPTIMIZATION IN MANUFACTURING," filed on Nov. 7, 2019, and "DEEP LEARNING QUALITY PREDICTOR," filed November 7, 2019; No. 62/932,063 entitled FOR MANUAL ASSEMBLY VIDEO ANALYSIS. The entire contents of the above applications and patents are incorporated herein by reference.

The subject technology provides improvements to assembly line workflow, and in particular, a system for adaptively updating assembly line operator instructions based on feedback and feedforward error propagation predictions made using machine learning models. and methods. As discussed in further detail below, some aspects of the technology automatically generate instructional videos provided at one or more worker stations based on inferences made regarding manufacturing or assembly deviations. Systems and methods for matching are included.

Traditional assembly line workflows require human (operator) oversight and specialized techniques to detect manufacturing errors and to determine how errors can be eliminated through corrections in downstream processes. technology is required. Note that assembly and manufacturing, and assembly line and production line are used interchangeably herein. Due to the reliance on humans to detect assembly errors, there is a high probability that errors will be overlooked (or not reported) and propagated directly downstream in the assembly process. Furthermore, many assemblers are only trained to perform a narrow set of tasks, and thus are themselves trained to best correct errors that occur upstream in the assembly workflow. may not be aware of how to modify their workflow.

In traditional manufacturing workflows, the correction of human errors in part of the manual inspection process is often handled by taking corrective action on the human node. If there is an ongoing problem with the person, the person is often replaced by another person who is vulnerable to many of the same limitations as all humans are. It is difficult to repeat an action consistently over the years without committing an error, and taking corrective action is beyond the rights of most builders. Even given these rights, there are contradictions and they can only be told by the experience the human has in the application of the single process node. Moreover, there is no mechanism for learning from any mistakes or even from any positive corrective actions.

In addition, electronic monitoring of assembly lines has limitations and robust mechanisms for providing opportunistic adjustments to downstream steps in the assembly line to compensate for errors made in upstream steps. not included. In addition, assess how changes in worker motion and/or changes in assembly patterns affect the resulting manufactured product, and take corrective action to improve the performance and/or characteristics of the manufactured product. A new mechanism is needed to provide

In some aspects, the disclosed technology relates to a method for optimizing a workflow on an assembly line, the method comprising, at a step of assembling the target object, detecting an error in assembling the target object; Evaluate the target and nominal targets at that step in the process, thereby obtaining a comparison, and, based on that comparison, the sequence of steps necessary to minimize the deviation between the target and nominal targets. and determining. In some aspects, the method may further include adjusting building instructions for the target object based on the sequence of steps.

In another aspect, the disclosed technology encompasses a system for optimizing a workflow on an assembly line, the system comprising a plurality of image capture devices, each image capture device for a target assembly process. with a plurality of image capture devices placed at different positions to capture the movements of the worker during and an assembly instruction module coupled to the plurality of imaging devices. a building instruction module receiving motion data from a plurality of image capture devices, the motion data corresponding to a worker performing a set of steps for building a target object; and determining an error in the assembly of the target object based on the motion data in one of the steps of . In some embodiments, the assembly instruction module evaluates a target object and a nominal object in said step of said set of steps, performs operations to thereby obtain a comparison, and based on said comparison , further configured to determine a sequence of steps required to minimize the deviation between the target target and the nominal target, and adjust the assembly instructions provided to the operator based on the sequence of steps. can do. Modified assembly instructions may be in the form of, but are not limited to, generated or edited video of motion data, text-based instructions from natural language processing (NLP) of identified deviations, or instructions to the operator. may include other feedback mechanisms.

In yet another aspect, the disclosed technology relates to a non-transitory computer-readable medium containing instructions stored thereon, the instructions, when executed by one or more processors, to a target object assembly process. In one step, detecting an error in the assembly of the target object; evaluating the target object and the nominal object in said step of the assembly process to thereby obtain a comparison; A processor is configured to execute instructions including performing the steps of determining a sequence of steps necessary to minimize the deviation between objects. In some implementations, the instructions may be further configured to cause the processor to perform operations for adjusting the building instructions for the target object based on the sequence of steps described above.

Particular features of the subject technology are set forth in the appended claims. However, the accompanying drawings, which are included to provide a further understanding, illustrate the disclosed aspects and, together with the description, serve to explain the principles of the subject technology.

1 conceptually illustrates a flowchart of an exemplary production line deployment, in accordance with certain aspects of the disclosed technology; FIG.

FIG. 3 illustrates an example process for implementing assembly error correction at a given worker station, in accordance with certain aspects of the disclosed technology;

1 illustrates an example electronic system that can be used to implement some aspects of the subject technology; FIG.

The detailed description presented below is intended to be a description of various configurations of the subject technology, and is not intended to represent only the configurations in which the subject technology can be practiced. The accompanying drawings are incorporated into this specification and constitute a part of the detailed description. The detailed description includes specific details to provide a more thorough understanding of the subject technology. However, it will be clear and obvious that the subject technology is not limited to the specific details shown herein and may be practiced without these details. In some instances, structures and components are shown in block diagram form in order to avoid obscuring concepts of the subject technology.

Aspects of the disclosed technology overcome the above drawbacks of conventional assembly line process flows by providing methods for tracking, training, and incrementally improving production line assembly and resulting manufactured products. Deal with limits. Improvements are realized by providing dynamic visual or other feedback and instructions to individual builders, and in some embodiments worker feedback is based on errors, which are caused by: These may include, but are not limited to, assembly errors, ineffective processes and/or motions, and inferior product detected at one or more points in the production line.

By implementing the disclosed techniques, for example, based on near-real-time error detection, by rapidly revising and changing the reference/indicative information provided at individual stations (or all stations), manual implementation With respect to the speed of error correction of the method, its speed can be significantly improved. Although some embodiments described herein contemplate using reference/indicative information in the form of video, other formats are also contemplated. For example, assembly/manufacturing instructions may be provided to the assembler as audio, visual, textual and/or tactile cues or as other forms of reference. As an example, the audible instructional information may include verbal instructions or other audible indicators. Visual building instruction information may include video or animated formats such as the use of Augmented Reality (A/R) or Virtual Reality (V/R) systems. In some aspects, the visual assembly instructions may be provided as animations that provide examples of how the workpiece (or tool) should be handled by workers at a given station of the assembly line. Further, in some aspects, the assembly/manufacturing instructions may include machine instructions that can be received and implemented by, for example, a robotic or mechanical assembler. As used herein, the term worker can mean a human, robot or machine that uses motion to assemble a manufactured product. Further, the term worker encompasses human-assisted manufacturing implementations, such as in instances where human workers work with or are assisted by robots or machines. there is

In instances where assembly/manufacturing instructions are provided as reference/instruction videos, such videos are sometimes referred to as standard operating protocols (SOPs). Systems of the disclosed technology can be effectively deployed due to minimal hardware requirements, such as the use of video cameras and displays for individual workers, while machine learning training, updating and error propagation can be implemented on centralized computing resources, such as in computing clusters or cloud environments, for example.

In some aspects, video instructional information may be provided to one or more workers as part of an augmented reality display. instructions, or deviations from standard assembly/manufacturing methods, can be communicated to workers using augmented reality, and displays can be enhanced video, animated graphics, and/or recorded scenarios. provided as a mixture of video data representing As an example, an augmented reality display may provide instructions or guidance provided as an animated or graphical overlay to a real-time feed of the workpiece being assembled and/or the tools being used in the assembly/manufacturing process. can be done.

In some implementations, systems of the disclosed technology include one or more video or motion capture devices located at various worker stations on a production line. The capture device is configured to record the worker's motion/interaction with parts, devices, materials or other tools (“components”) at that particular station. In some aspects, worker motion can be captured using video recording, but other motion capture formats, such as 3-D points representing worker motion and/or worker interaction with a tool or manufactured product. Cloud use is also contemplated. In addition, it is also possible to create a station-by-station reference video by recording the motion of one or more technicians to a particular station and their interactions with the components at that station. This video can be created from a single example of the expert's actions or from multiple examples. Motion paths for each expert can be derived, and in embodiments where several experts or several examples are used, performing a calculation on the set (e.g., average) of the derived motion paths, A reference video can be created for a particular station. A reference video may be in the form of a digital or animated representation of the motion path to be performed at that particular station. Note that expert can mean someone with specialized skills or knowledge of the particular assembly steps for which guidance is provided.

In some embodiments, video or motion capture devices placed at various worker stations on a production line can be used to calculate assembly errors for workpieces/components/ Tool attributes (eg quality, tensile strength, number of defects) can also be captured.

By capturing a worker's interaction at each station, worker error can be determined by comparing the captured interaction with a baseline (ground truth) model representing the ideal/skilled worker interaction/workflow. can be detected. assembly errors that can be corrected at various locations in the assembly chain, i.e. using operator deviations from an idealized interaction model, for example by revising operator instructions/guidance provided at different stations can be calculated. Additionally, it is possible to capture the quality of the components at an individual station and compare against the baseline component for that station. Also, the deviation of a component from a baseline component can be used to assign a quality grade to a component at a particular station, or make repairs by revising operator instructions/guidance provided at various stations. / Assembly error can also be calculated.

Assembly modifications can be implemented in a variety of ways depending on the desired implementation. In some embodiments, operator variations/errors are used, for example, to classify parts into quality grades (e.g., A, B, C, etc.) and subsequently direct these parts to the appropriate production line. can be implemented. In another aspect, detected assembly errors can be used to revise processes at a given station, thereby improving quality and reducing variability. That is, detected assembly errors can be used to automatically provide instructions or guidance to the same station so that, for example, errors introduced at that station can be corrected (eg, in-station rework). . NLP can be used to process instructions or guidance to workers. For example, NLP can be used to convert verbal instructions into textual form, or vice versa.

For example, assembly error detection can be used to drive updates/changes to operator instructions or video provided to a given station where an error is known to occur. As an example, once an error/deviation is identified as being due to a first worker working at a first station, the The provided assembly instructions can be revised, thereby reducing the error spread associated with the manufactured articles leaving the first station.

In another aspect, detected assembly errors can be used to revise subsequent station assembly, thereby overcoming station dispersion. That is, error detection can be used to automatically trigger downstream propagation of new/updated building instructions based on errors caused by upstream workers. For example, error distribution for motions performed by a first worker can be used to adjust assembly instructions provided to a second worker associated with a second station downstream from the first station. can.

Yet another aspect is that the detected error distribution can be propagated forward across all stations so that full or partial rework can be performed over the course of the rest of the downstream assembly chain. can be guaranteed. That is, by adjusting the assembly instructions provided to one or more downstream workers, errors generated across one or more stations can be corrected/reduced. In one example, via work subsequently performed by a second worker at a second station and by a third worker at a third station, i.e., to the second and third stations By adjusting the assembly instructions, the error distribution of the manufactured article caused by the first operator at the first station can be corrected.

In another example, the error variance accumulated over multiple stations can be reduced by one or more subsequent stations. For example, by adjusting assembly instructions provided to a third station and a fourth station (e.g., a third worker and a fourth worker, respectively), Error distribution in the article of manufacture can be subsequently repaired.

By treating individual workers/worker stations in the assembly flow as network nodes, machine learning models are used to optimize the assembly process by minimizing errors through reduction of assembly variance at individual nodes (stations) can be optimized. By minimizing individual node (worker) variance and implementing real-time updates to mitigate forward error propagation, the system of the disclosed technology can significantly reduce manufacturing variance to the final product. can. Additionally, by accurately quantifying and tracking error contributions from specific segments in the assembly workflow, products can be graded and sorted by product quality or amount of deviation. Thus, for example, depending on product quality, products of a particular quality class can be directed to different manufacturing processes or to different customers.

Error detection can be performed using machine learning/artificial intelligence (AI) models and/or corrections necessary to optimize station assembly changes can be performed. By way of example, the machine learning model may include, but is not limited to, final product ratings, final product change statistics, desired end product characteristics (e.g., assembly time, amount of materials used, physical properties, number of defects, etc.). , station-specific component ratings, station-specific component variations, and desired station component characteristics. Additionally, the deployed machine learning model is provided by an expert or "master designer" so that formal knowledge can be represented in the idealized model used to perform the error detection and error quantification calculations. It can be trained or initialized based on the input it receives.

As will be appreciated by those skilled in the art, machine learning-based classification techniques can be modified according to desired implementation without departing from the disclosed technology. For example, machine learning classification schemes include Hidden Markov Models, Recurrent Neural Networks, Convolutional Neural Networks (CNN), Reinforcement Learning, Deep Learning, Bayesian Symbolic Models, Generative Adversarial Networks (GAN), Support Vector Machines, Image Registration Methods, Applied One or more of the possible rule-based system and/or any other suitable artificial intelligence algorithms may be utilized alone or in combination. Where regression algorithms are used, they can include, but are not limited to, Stochastic Gradient Descent Regressor and/or Passive Aggressive Regressor, and the like.

Machine learning classification models include clustering algorithms (e.g. Mini-Batch K-Means clustering algorithm), recommendation algorithms (e.g. Miniwise Hashing algorithm or Euclidean Locality-Sensitive Hashing (LSH) algorithm), and/or local outlier factor methods. can also be based on the anomaly detection algorithm of Further, the machine learning model may be subjected to dimensionality reduction such as one or more of a Mini-batch Dictionary Learning algorithm, an Incremental Principal Component Analysis (PCA) algorithm, a Latent Dirichlet Allocation algorithm and/or a Mini-batch K-means algorithm, etc. It is also possible to use techniques.

In some implementations, multiple different types of machine learning trained/artificial intelligence models can be deployed. As an example, general forms of machine learning can be used to dynamically adjust assembly line processes to optimize manufactured products. As those skilled in the art will appreciate, the selected machine learning/artificial intelligence model does not simply contain a set of assembly/manufacturing instructions, but feedback on the entire assembly line process and the resulting A method for providing its effect on manufactured products and also for providing dynamic adjustments to downstream worker stations in an assembly line, thereby compensating for actions occurring at upstream worker stations. This type of artificial intelligence-based feedback and feedforward model is referred to herein as Artificial Intelligence Process Control (AIPC). In some embodiments, machine learning utilizes targeted gated recurrent unit (GRU) model-based learning and Hausdorff distance minimization to effectively explore a space of possible recovery paths, thereby assembling It can be based on deep learning models in simulated environments that find optimal paths to correct errors during the process. In still other embodiments, for example, a machine learning algorithm can be based on the Long Short-Term Memory model to analyze the video input of the assembly process and predict the final quality output. Additionally, machine learning models, in the form of NLP algorithms, are used to maximize worker compliance and comprehension of coordinated instructions, such as text-to-speech or speech-to-text. You can also adjust the feedback to the station.

In some embodiments, computational models that utilize machine learning can be used to correct errors during the manual assembly process.

A target object can be assembled through a series of steps defined by a procedure. During the process, irreversible errors can occur at certain steps k that would require revision of all remaining work in order to obtain the final configuration of the nominal object. In some embodiments, the technique for correcting the error includes comparing the defective target object at step k to a nominal target at the same step k, or comparing the defective target object at step k to its A step of comparing against a nominal target of the final configuration may be included. These comparisons can be used to determine the sequence of steps required to minimize the deviation between the final configuration of the defective target object and the final configuration of the nominal object. In some embodiments, the target object quality metric can also be used to guide the correction of defective target objects.

A general numerical approach can solve the problem by using the Hausdorff distance algorithm, whereby a sequence of k steps to assemble the faulty target object transforms the nominal object into its final configuration. It can be determined how similar the sequence of steps to assembly is. Some methods for computationally minimizing the Hausdorff distance between the above sequence of k steps in the faulty assembled object and the sequence of steps in the final assembled nominal object are instantaneous Optimizing the Markov Decision Process (MDP) via reward formulation, multiple reward formulation, or delayed reward formulation. However, the search spaces associated with these formulations require a significant amount of computational resources.

Alternatively, machine learning frameworks can be developed with delayed reward policy agents using reinforcement learning. A reinforcement learning framework is designed to let policy agents determine the appropriate steps required to correct faulty target target errors, resulting in a final configuration that has a performance metric that matches the nominal target performance metric. can do. Rewards given to policy agents are delayed and paid to policy agents only when the final step is executed.

In some embodiments, the design for the optimal/desired manufactured product can be selected and a skilled worker deployed to individual workers to assemble the manufactured product according to the selected design. Individual steps performed at the station can be implemented. Optimal can be based on the desired performance and/or characteristics of the resulting product that minimize errors in the resulting manufactured product, or some other criteria (e.g., if the manufactured product is a paper airplane). , the optimal paper airplane would be the one that achieves the desired flight goal). Using multiple imaging devices to capture worker motion and worker interaction with the manufactured product they are assembling, thereby generating video, images and/or 3D point cloud data can be done. The captured data includes the worker's hand coordinates in relation to the manufactured product as it is being assembled, the relation of one hand to the other, and the fingers (and some Detailed information such as knuckle (knuckle) relationships in embodiments may be provided. Data collected from skilled workers can be used as ground truth for assembling the optimal/desired manufactured product. This ground truth from a single example is, by itself, sufficient for use in creating initial machine learning models, or additional data can be collected. For example, deploying many workers to assemble an optimal manufactured product to understand how variations in worker motion or errors can affect the resulting manufactured product. Or multiple steps can be performed. This can be done for each worker station on the assembly line. Both the resulting final products and their respective assembly processes are compared against each other and against the ground truth so that errors and/or changes in operator motion are reflected in the characteristics of the manufactured product and/or Or it can determine how performance can be affected (eg, operator speed can result in a plane of lesser quality). Data collected on a worker basis during the actual assembly process (i.e., a process in which a human, robot, or machine is performing motions at one or more stations) is herein referred to as "actual It will be called "training data". The actual training data can be supplemented with simulated data to obtain a richer data set and to provide additional variations to achieve optimal manufactured product. can. Note that the terms "optimal" and "desired" will be used interchangeably herein.

In some embodiments, the different AI/machine learning/deep learning models discussed herein are used below to achieve Artificial Intelligence Process Control (AIPC) to optimize assembly of articles of manufacture. It can unfold in the specific order described. An exemplary process by which an AIPC deep learning model may be implemented is discussed in greater detail in connection with FIG. 1 (eg, in connection with AIPC deep learning model 112) and in connection with FIG. An example of a hardware system and/or device that can be used to implement an AIPC deep learning model is provided in FIG. 3, and a corresponding description is also provided below.

First, CNNs can be used in assembly line processes to classify the characteristics of workers' hands and different configurations of manufactured articles at individual worker stations.

Second, both Reinforcement Learning (RL) and RL agents can be used and rewarded to achieve desired results from CNN classification and for defined desired results. . The RL agent may or may not monitor.

Third, Generative Adversarial Networks (GAN) can be used to choose between competing RL agents. A GAN can include minimal human oversight, relying only on humans to select RL agents that enter the GAN as nodes.

Fourth, RNNs can take winning RLs as input nodes to create feedback and feedforward systems, thus making learning continuous and monitoring learning. It doesn't have to be.

Implementations of these four AI/machine learning models are discussed in more detail below.

In some embodiments, the actual training data classifies relevant data in the assembly process, e.g., fingers/hands used in individual assembly steps per worker station, of the product being assembled, The part touched by the operator's finger at any point in time and space, and the shape or configuration of the manufactured product being assembled at any point in time and space, can be input into the CNN to classify.

In still other embodiments, hand motions are not tracked, but data representing various changes in the assembly pattern of the manufactured product may be collected (e.g., if the manufactured product is a folded paper airplane, folding Data can be collected based on changing the order, implementing folding changes, and/or introducing potential errors, and if the manufactured product is a clothing article, e.g. Data may be collected based on implementing changes and/or introducing potential errors). This data can be simulated and/or gathered from actual training data. The resulting manufactured products and their respective assembly processes can be compared to determine how errors or changes in assembly patterns affect the characteristics and/or performance of the manufactured products.

In some embodiments, captured data (eg, video and hand tracking of the assembly process, etc.) is used to predict final output quality. This quality prediction allows the captured data to be used during the manufacturing process to classify products into quality bins without the need to manually inspect product quality and to take corrective action downstream. be able to.

In some embodiments, the system can focus on manual assembly of a target object, where the assembly process involves a worker performing different tasks on the target object for each set of instructions. Contains several discrete steps. The system uses a machine learning framework (across all worker actions) that uses deep learning models to establish correlations between the time series of worker hand positions and the final quality of the target object. can be built. In some embodiments, the model may consist of two neural networks, a first neural network used to extract the position data of the worker's hand in the 3D environment, and a second neural network A neural network is used to remove unnecessary elements from the hand position data and correlate it with the final quality of performance of the target object.

In some embodiments, the first neural network uses the video acquisition system to generate images during the assembly process in different node videos corresponding to individual discrete steps that the worker performs in connection with the assembly of the target object. , a video of the worker's hand can be recorded. For example, a worker may carry out an assembly process using several cameras positioned at different locations and configured to simultaneously record the assembly process. These cameras can be used to capture video multiple times at predefined positions of the worker's hand. These videos can then be processed to extract a number of images or landmark frames representing the overall target object assembly process. These landmark frames can be used to extract hand tracking information that facilitates defining the positions or key points of the worker's hands and fingers during the assembly process.

In some embodiments, a bounding box prediction algorithm and a hand keypoint detector algorithm can be applied to extract hand tracking information. Specifically, the bounding box prediction algorithm may involve using threshold image segmentation to process the landmark frames from the assembly process, thereby obtaining a mask image for the worker's hand. The hand can be positioned over the mask using blob detection. Bounding box prediction uses the mask image to form a box around each of the worker's hands such that the box contains the highest point of the hand position shape at least up to the hand list point. The bounding boxes and their corresponding landmark frames are then fed to the hand keypoint detector algorithm.

A hand keypoint detector algorithm can include a machine learning model that can detect specific keypoints on a worker's hand. The hand keypoint detector algorithm can predict not only keypoints that are visible in the landmark frame, but also keypoints that are occluded from the frame due to joint, viewpoint, object, and hand interactions. A particular occlusion point in one frame cannot be occluded in another frame because different hand positions produce different occlusion points in different frames. A hand keypoint detector predicts the location of occluded keypoints with a certain level of certainty. However, prediction of occluded keypoint locations may result in the same keypoint locations being recorded for different hand positions. During the steps of the manual assembly process, the hand keypoints defining the worker's hand are then provided to a second neural network.

In some embodiments, a second neural network is used to predict the quality of the final state of the assembled object. In some embodiments, the neural network can be based on the Long Short-Term Memory (LSTM) model. The LSTM has several cells that are arranged in sequence and together represent the overall assembly process of the final object. The input to the LSTM cell may be hand keypoint data corresponding to a worker's work in the particular step of the assembly process represented by the LSTM cell. Each cell in the LSTM decides whether to store information from the preceding cell, chooses a value to update, performs an update to the cell, chooses a value to output, and then chooses Filter values so that cells only output values. A LSTM may be a sequence for one model trained using the Adam optimizer or other adaptive learning rate optimization algorithms. The neural network uses the LSTM framework to correlate input data extracted from the manual assembly process, thereby determining quality measurements of the final product.

In some embodiments, the video and hand-tracking information representing the building process for the target object, i.e. the input data used to train the model, are generated from multiple models using a single set of building instructions. It can be collected from multiple workers performing an assembly process for assembling the target object. A target object assembled by a worker can be used in a controlled environment to collect corresponding quality measures for the performance of the assembled object, i.e. the output data required to train the model.

In some aspects, the training data used to generate the machine learning model is from simulated data, from actual training data, and/or from expert ground truth recordings. There may be, and they may be combined or separate. In some embodiments, simulated data results can be used to build machine learning models, such as (but not limited to) Reinforcement Learning (RL) agents. In other embodiments, the actual training data may be used to build machine learning models, such as (but not limited to) Reinforcement Learning (RL) agents. RL agents are rewarded for achieving good/desired results and penalized for bad results.

In some instances, many RL agents, some based on real training data and some based on simulated data, can be deployed to work in tandem, It can also be configured to maximize the cumulative award, e.g. build the manufactured product with the smallest deviation from the ideal model/example. Exemplary outcomes that may reward the RL agent include completing a complete manufactured product in as few steps as possible, reducing the amount of materials or time required to achieve the manufactured product. RL agents based on simulated data and RL agents based on actual training data can be used to determine optimal motion patterns and/or optimal assembly patterns that result in optimal/desired manufactured articles.

These two groups of RL agents (e.g., RL agents created based on real training data, and RL agents created based on simulated data) are both optimal/desired manufactured products. You can work together and even compete here because you are paid for the act of creating In some embodiments, data obtained from simulated-based RL agents that yielded optimal assembly patterns for optimal manufactured products are used to reduce the probability space for the actual training dataset. be able to. For example, a simulated RL agent can be used to determine the optimal assembly pattern, then collect actual training data only for the optimal assembly pattern, not for the non-optimal assembly pattern. be able to. By focusing only on the collection of actual training data, or only on the optimal assembly pattern, less training data can be collected and/or a larger training data can be collected to collect more actual training data. Capacity can be achieved, but only for optimal assembly patterns.

Relying solely on reinforcement learning to optimize assembly lines is limited because rewards can sometimes be conflicting. For example, in product assembly, for the least amount of spurious motion (e.g., folding and immediately unfolding, or adding a seam and immediately removing the seam), several RL agents , while other RL agents can be rewarded for speed. An RL agent that is paid for speed can determine that more spurious movements result in faster assembly times because fewer modifications are required downstream in the assembly process. Making such implementation trade-off decisions is not something that can be readily determined by a human being. Even with experience and a large number of examples, humans lack the computational power to understand how subtle it is to perceive the end result of different workers working in different ways. still lacking.

To resolve these conflicting RL agent optimizations, GANs can be deployed to act as decision makers. Conflicts may arise between RL agents based on actual training data, between RL agents based on simulated data, and/or between RL agents based on actual training data and RL agents based on simulated data. may be

In some embodiments, the GAN can test each of the RL agents and store the results to create more robust neural networks. GANs operate by taking RL agents and using models that yield winners and losers in zero-sum games. There are "generators" and "discriminators" in GANs. The generator would in this case store reward data from opposing RL agents, and the discriminator would evaluate which of these was most relevant to the task of creating the desired manufactured product. will do. A GAN uses a deep network of nodes (or neurons) to determine how to weight nodes. Since each RL agent believes that each RL agent has already made the optimal decision, it is impossible to determine which of the conflicting RL agents actually makes the most relevant choice. It is the role of the GAN and the discriminator adjusts the weights accordingly. When a zero-sum game is initiated between opposing RL agents, groups of winners are generated among the opposing RL agents, and only those winners are used to optimize workflow on the assembly line. will be used for machine learning models that are Although a large amount of data can be generated to determine the winning RL agent, the results are more than those used to create and find those winners used as input nodes. is also much poorer.

Once the RL agents that survived the GAN war and were appropriately paid are determined, they can be input into another AI system called the Recurrent Neural Network (RNN) in some embodiments. RNN has many similarities to CNN in that it is a Deep Learning Neural Network and the final result is optimized through various forms of weighting of the input data. One difference is that unlike CNNs, which are linear processes from input to output, RNNs are loops that feed back the resulting output, as well as internal nodes, as new training information. RNNs are both feedback and feedforward systems such as GRUs.

In some embodiments, a machine learning framework can be built utilizing targeted GRU model-based learning. The GRU model can be chosen instead of reinforcement learning because of its predictive power and relatively short training time. The GRU is used by the RNN to distinguish between observations that should be stored in memory or update state and observations that should be forgotten or reset state.

In some embodiments, a GRU model can consist of a number of GRU cells corresponding to the number of assembly steps required to build the target object. Each GRU cell representing one of the number of assembly steps can have several input parameters and hidden state outputs. The GRU cell, which represents the final step in the assembly process, will output the target object. The output of the model is the deviation of the target target from the nominal target. This deviation can be calculated using the stepped Hausdorff distance from the target object to the nominal object and the performance metric of the final configuration of the nominal object. Individual GRU cells are defined by reset, update and new gates. The GRU neural network is iteratively trained, thereby biasing the GRU neural network toward solving a particular sub-problem, and identifying a set of weights for the GRU. For example, in each iteration several predictions (one prediction for each possible error in a particular step) are generated to complete the assembly process in subsequent steps. Additionally, a corresponding predicted distance measure for the modified assembly process can be generated. These predicted assembly process completions can be depicted in a virtual representation system and their stepwise Hausdorff distances calculated to obtain a "ground truth" distance measure. The difference between the "ground truth" and the predicted distance measure can be computed and fed back into the model, whose network weights are adjusted via backpropagation to produce the next iteration. This process can continue until the set of weights for the GRU is identified. In some embodiments, a stochastic gradient descent method can be used to correct the defective target and derive the steps necessary to obtain a satisfactory final configuration. can.

In some embodiments, a simulation, such as a parametric computer-aided design and drafting (CAD) model of the target object in process, can be generated to develop and validate the machine learning model. A CAD system can use a local coordinate frame corresponding to the current state of the target object being processed, and input parameters representing individual assembly steps. Using the local coordinate frame of the target object under processing and the input parameters, the CAD system can determine dimensional information for each assembly step. A CAD system can then generate a three-dimensional CAD model that represents the configuration of the outputs at the individual steps. The CAD system can continue the process until all steps in the assembly process have been performed and output a CAD model of the final configuration of the assembled object. By providing various input parameters to the CAD system, CAD models of different configurations can be generated. To obtain a set of CAD models having a specified range of input criteria, such as length or width, to provide a statistical sample of this input criteria to a CAD system to generate the set of CAD models. can be done.

CAD models may be of varying levels of detail and intelligence, but trained models and systems are specifically designed to work with lower detail CAD systems, thereby making them a non-computationally expensive method. can generate a vast number of examples in , and provide sufficient surface morphology detail for trained models and profiling. In some embodiments, the referenced CAD system can be paired with Finite Element Analysis (FEA) or basic surface modeling tools to generate a structural analysis of the surface. This data can be used as an additional quality score for model training and analysis.

In some embodiments, the CAD system can be incorporated into model training so that additional surface models can be generated as required for examples from the model or as needed for additional exploration data. This approach is coupled with physical observation and allows pre-trained models to be developed without the need for a vast amount of physical samples in space.

In some embodiments, a CAD model of the final configuration of the assembled object may be used for simulation to generate performance metrics. By using a CAD model of the final configuration of the assembled object, the simulation can utilize numerical and computational methods to generate performance metrics.

The real-world application of Artificial Intelligence Process Control (AIPC) is to provide feedback (e.g., by automatically correcting video instructions) to workers on the assembly line who have already completed their tasks, as well as to provide feedback downstream on the assembly line. side, providing instructions to workers who have not yet completed their tasks (again, eg, by automatically modifying video instructions) (“feedforward”). This feedback-feedforward system, or AIPC, can be achieved using the AI techniques described herein, in some embodiments in the specific order described herein. Thus, workers on the assembly line can make choices that optimize the resulting manufactured product without the need for additional human oversight.

In some embodiments, this simply involves compressing the system into an RNN and observing any movement during the process of creating one or more manufactured products, in two ways, success or failure. Individual movements work as training. If the RNN's output node is not optimal, the network can feed back to the actual individual on the assembly line to make a different choice, and in the path through many nodes and layers of the RNN, the weights can be redone and the output will be labeled as either success or failure. As the process iterates, the weights improve their accuracy on their own. Additionally, the network can learn what is working and what is not working, even if the individual performing the assembly is not working. This increases the training set. It also allows adjustments to be made at different stages of the assembly process. In some instances, the best way to produce an article of manufacture with specific properties at any given moment is not to return to the starting point, but to adjust the instructions as the process progresses. can be found. In this case, the RNN is constantly optimizing for the best manufactured product and also provides feedback to individual workers at worker stations on the production line who have already performed their tasks and also learn to feed forward information to workers at worker stations who have not yet performed their tasks.

FIG. 1 conceptually illustrates a flowchart of an exemplary process 100 for implementing production line deployment, in accordance with some aspects of the disclosed technology. The process of FIG. 1 begins at step 102 to initiate production deployment. An exemplary production line workflow typically includes multiple worker stations (nodes) at which workpieces (products) are assembled or manufactured. The various nodes can be organized in a sequential fashion such that work on each subsequent node begins only after work on the preceding node is completed.

At step 104, one or more reference videos are generated and/or updated. The videos described above can be used to provide manufacturing/assembly instructions to specific nodes (also referred to herein as worker stations). That is, each node in the workflow can have a reference video that delivers guidance on how to complete the step in the manufacturing workflow that corresponds to that particular node.

At step 106, each of the videos generated at step 104 is deployed to respective stations/nodes. As an example, a given workflow may contain 10 nodes, each with their own different/unique reference video. In other implementations, the number of videos may be less than the total number of nodes. Depending on the implementation, the reference videos deployed to various stations/nodes may be unique or may provide similar guidance/instructions. As discussed in more detail below, the content of the reference video may be dynamic and constantly updated/augmented.

At step 108, continuous recordings of motion are captured at individual stations/nodes. Motion data obtained from motion recordings can describe worker interactions with workpieces/components/tools at those nodes/stations in the workflow. That is, the motion data captured at an individual node can represent one or more worker actions corresponding to a particular portion of product assembly or manufacturing, and is provided by the reference video associated with that node. Able to respond to instructions. In some instances, motion capture may include capturing video data, ie, recording all or part of the worker's actions at the station. In other embodiments, motion capture can include, for example, recording a 3D point cloud, where motion is recorded for one or more specific points in the field of view of the image capture device. Both worker actions as well as component attributes (eg, component quality, tensile strength, number of defects) can be captured at individual nodes/stations in the workflow.

At step 110, process method analysis deviations may be calculated, analyzing the motion data captured for one or more of the stations at step 108 to obtain, for example, an idealized motion profile for the corresponding station. Any deviation from the comparative model that includes (or represents) can be identified. As illustrated in FIG. 1, step 110 comprises, for example, identifying/classifying motion deviations from the comparison model and making inferences as to the extent to which the assembly or manufacturing process may have been affected. An AIPC deep learning model can be utilized (step 112). The comparison can be performed on a per station level and/or on an overall process level. The analysis may also consider the attributes of the components at individual stations, or the extent to which the deviations of the components from the baseline and the motion deviations of the stations affect the quality of the components.

The AIPC deep learning model called in step 112 is based on collection of various types of training data, which can include examples of ideal or quality-controlled assembly/manufacturing interactions for a given station/node, for example. be able to. AIPC deep learning models are also trained by feedback from customers regarding specific products manufactured using process 100 (step 111), and for specific products manufactured using process 100 (step 113). Data provided by domain/industry information 115 can be used to augment (or adjust) feedback from quality control inspections. It should be appreciated that AIPC deep learning models can be implemented using a variety of computing systems including distributed hardware and/or software modules. As an example, an AIPC deep learning model is deployed on an assembly line and multiple images combined into one or more systems configured to implement various AI/machine learning models and/or classifiers. It can be implemented using a distributed system that includes a capture device and a display device.

Once deviations from the comparative model are detected/identified at step 110, the AIPC deep learning model 112 can be used to generate automatic adjustments at step 114. FIG. As discussed above, video adjustments can be aimed at improving manufacturing/assembly quality at one or more stations of the workflow. where errors are known (or predicted) to occur, e.g. Video adjustments can be applied to a given node/station. In other implementations, video adjustments can be applied downstream from the station where the error occurred, eg, to correct the error before the manufacturing workflow is complete. In yet other implementations, once a workflow is completed, the entire workflow can be analyzed and adjustments can be made to one or more stations in the workflow.

In some embodiments, adjustments are made in real-time immediately after an error is detected. In other embodiments, adjustments are made at regular intervals or after the workflow is completed.

In some aspects, the automatic adjustments determined at step 114 may be summarized at step 117 and/or provided as a product quality report. One or Multiple quality reports can be generated.

FIG. 2 illustrates an exemplary process 200 for performing error detection analysis that may be used to facilitate assembly error correction, in accordance with some aspects of the present technique.

Beginning at step 210, the process of improving manufacturing/assembly using idealized video guidance can be implemented. At step 215, video tracking of one or more assembly stations is performed. Video tracking can include recordings of human workers at a given station/node. In some embodiments, video tracking can further include capturing component attributes at a given station/node.

At steps 220-224, processing is performed to analyze the video recorded from the assembly station. For example, in some embodiments background extraction can be performed to isolate motion/components in the recorded video. In some aspects, once background extraction is complete, the processed video includes only motion/video data related to the assembly worker (step 224), and the included components are the corresponding assembly steps (step 224). step 220). At step 220, additional processing may be performed to isolate parts/components. As illustrated by the diagram of process 200, step 220 performs additional processing operations including anomaly detection (step 221), surface change detection (222) and part classification and/or quality scoring (step 223). can contain. Any video processing step is performed using various signal and/or image processing techniques including, but not limited to, the use of one or more AI/machine learning algorithms and/or classifiers, can perform anomaly detection (221), surface change detection (222), and/or scoring/classification (step 223), for example.

After processing steps 220-224 are completed, process 200 may proceed to step 226 where a motion comparison is performed. Motion comparison (step 226) compares process assembly station video data including one or more station workers at one or more stations/nodes with corresponding idealized video/motion data. can include Motion comparisons performed across multiple stations/nodes can be used to infer/predict the resulting change in part/component quality.

At step 228, distribution/quality classification for various parts/components may be performed. As an example, parts/components can be classified into different quality tiers and/or identified for removal or repair depending on their associated classifications/differences.

After the classifications/differences have been determined, process 200 can proceed to step 230 where an analysis of the overall process/workflow is performed, e.g. conducted based on By analyzing the entire workflow, automatic adjustments to the video can be made to address detected deviations/defects, as discussed above.

FIG. 3 illustrates an exemplary processing device that can be used to implement systems of the disclosed technology. Processing device 300 includes a master central processing unit (CPU) 362, an interface 368 and a bus 315 (eg, PCI bus). Operating under the control of appropriate software or firmware, CPU 362 is responsible for performing various error detection monitoring and process adjustment steps of the disclosed techniques. CPU 362 preferably performs all these functions under the control of software, including an operating system and any suitable application software. CPU 362 may include one or more processors 363, such as processors from the Motorola family of microprocessors or from the MIPS family of microprocessors. In an alternative embodiment, processor 363 is hardware specifically designed to control the operation of AIPC system 310 . Memory 361 (such as non-volatile RAM and/or ROM) also forms part of CPU 462 in certain embodiments. However, there are many different ways memory can be coupled to the system.

In some aspects, processing device 310 may include or be coupled to image processing system 370 . The image processing system 370 can include various image capture devices such as video cameras that can monitor worker movement and generate motion data. As an example, image processing system 370 may be configured to capture video data and/or output/generate a 3D point cloud.

Interface 368 is typically provided as an interface card (sometimes called a "line card"). Generally, they control the transmission and reception of data packets over networks and sometimes support other peripherals used with routers. Interfaces that may be provided are, among others, an Ethernet interface, a frame relay interface, a cable interface, a DSL interface, a token ring interface, and so on. In addition, various very high speed interfaces can be provided such as high speed token ring interfaces, radio interfaces, Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces, and so on. Generally, these interfaces may include ports suitable for communicating with any suitable medium. In some cases, they may also include independent processors, and in some instances may include volatile RAM. Independent processors can control such communication-intensive tasks as packet switching, media control and management. By providing separate processors for communication-intensive tasks, these interfaces can allow the master microprocessor 362 to efficiently perform routing calculations, network diagnostics, security functions, and the like.

The system shown in FIG. 3 is one particular processing device of the present invention, but by no means the only network device architecture in which the present invention can be implemented. For example, architectures with a single processor that handles communications as well as routing calculations, etc., are often used. Additionally, other types of interfaces and media may also be used.

Regardless of the configuration of the network device, the network device will include one or more memories or memory modules (including memory 361) configured to store program instructions for general network operations and the Mechanisms for roaming, route optimization and routing functions provided by the standard can be used. Program instructions may, for example, control operation of an operating system and/or one or more applications. One or more memories can also be configured to store tables such as mobility bindings, registration and association tables, and so on.

The logical operations of various embodiments are represented as (1) a sequence of computer-implemented steps, operations or procedures running on programmable circuits within a general-purpose computer; (2) a sequence of computer-implemented steps, operations running on application-specific programmable circuits. or procedures; and/or (3) implemented as a program engine within interconnected machine modules or programmable circuits. System 300 can practice all or part of the described methods, may be part of the described systems, and/or can act according to the instructions of Such logical operations may be implemented as modules configured to control processor 363, thereby performing specific functions according to programming of that module.

It is understood that any specific order or hierarchy of steps in the processes disclosed is an illustration of exemplary approaches. Based on design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged, or that only some of the steps illustrated may be performed. Some of the steps can be performed simultaneously. For example, multitasking and parallel processing may be advantageous in certain situations. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, nor should the described program components and systems , in general, may be integrated together within a single software product, or may be packaged into multiple software products.

The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may also be applied to other aspects. Accordingly, it is not intended that the claims be limited to the embodiments shown herein, and that the claims be construed to the full extent consistent with the language of the claims, singular References to elements of form shall imply "one or more" rather than implying "one and only", unless expressly stated to the contrary. is intended.

A phrase such as "an aspect" does not imply that such aspect is essential to the subject technology or that such aspect applies to all configurations of the subject technology. A disclosure relating to an aspect may apply to all configurations, or to one or more configurations. A phrase such as an aspect can refer to one or more aspects, and one or more aspects can refer to an aspect. A phrase such as "configuration" does not imply that such configuration is essential to the subject technology or that such configuration applies to all configurations of the subject technology. A disclosure relating to a configuration may apply to all configurations, or to one or more configurations. A phrase such as a configuration can mean one or more configurations, and one or more configurations can mean a configuration.

The word "exemplary" is used herein to mean "serving as an example or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as over, preferred, or advantageous over other aspects or designs.
Statement of Disclosure

Statement 1: A method for optimizing a workflow in an assembly line, the method comprising, at a step of the target assembly process, detecting errors in the target assembly and the target at that step of the assembly process. evaluating the target and the nominal target thereby obtaining a comparison; and based on the comparison determining the sequence of steps required to minimize the deviation between the target target and the nominal target; and adjusting the building instructions for the target object based on the sequence of steps.

Statement 2: The method of statement 1, wherein the target object is evaluated against the nominal object in the above step of the assembly process.

Statement 3: The method of any of statements 1-2, wherein the target object is evaluated against the final configuration of the nominal object.

Statement 4: The method of any of statements 1-3, wherein the sequence of steps is determined using a machine learning model configured to minimize deviation.

Statement 5: The method of any of statements 1 through 4, wherein the deviation is the sequence of steps for completing the target object assembly process and another sequence of steps for completing the nominal object assembly process is determined based on the similarity between

Statement 6: The method of any of statements 1-5, wherein the deviation is minimized using the Markov Decision Process (MDP) via reward formulation.

Statement 7: The method of any of statements 1-6, wherein stochastic gradient descent is used to derive the sequence of steps for completing the target object assembly process.

Statement 8: A system for optimizing workflow on an assembly line, the system comprising a plurality of image capture devices, each of which captures worker movement during a targeted assembly process. a plurality of image capture devices positioned at different positions to capture and an assembly instruction module configured to automatically modify the guidance and instructions provided to the operator, the plurality of image captures an assembly instruction module coupled to the device, the assembly instruction module receiving motion data from the plurality of image capture devices by the assembly instruction module, the motion data representing the movement of the target object by the operator; determining errors in building a target object based on steps corresponding to performing a set of steps for assembly and motion data in a step of said set of steps; and said set of steps. evaluating the target target and the nominal target in the above step of to obtain a comparison, and based on the comparison, the sequence of steps necessary to minimize the deviation between the target target and the nominal target and adjusting the assembly instructions provided to the operator based on the sequence of steps.

Statement 9: The system of Statement 8, wherein the motion data includes digital recordings of the worker's hand movements while assembling the target object.

Statement 10: The system of any of statements 8-9, wherein the assembly instruction module is further configured to apply stochastic gradient descent to derive the sequence of steps.

Statement 11: The system of any of statements 8-10, wherein the assembly instruction module is further configured to determine the sequence of steps using a machine learning model, the machine learning model minimizing deviations configured as

Statement 12: The system of any of statements 8 through 11, wherein the deviation is between said sequence of steps for completing assembly of the target object and another sequence of steps for completing assembly of the nominal object. determined based on the similarity between

Statement 13: The system of any of statements 8-12, wherein the deviation is minimized using the Markov Decision Process (MDP) via reward formulation.

Statement 14: The system of any of statements 8 through 13, wherein the assembly instruction module derives from the motion data a set of images representative of the assembly of the target object, evaluates the set of images, It is further configured to identify execution of the set of steps by the operator to assemble the target object.

Statement 15: A non-transitory computer-readable medium containing instructions stored thereon which, when executed by one or more processors, cause the target object to Detecting errors in assembly, Evaluating target and nominal targets in the above steps of the assembly process, thereby obtaining a comparison, Based on the comparison, minimizing the deviation between the target and nominal targets. The processor is configured to execute instructions including determining a sequence of steps required to achieve the desired sequence of steps, and adjusting building instructions for the target object based on the sequence of steps.

Statement 16: The non-transitory computer-readable medium of statement 15, wherein the instructions are further configured to cause the processor to derive the above sequence of steps using stochastic gradient descent.

Statement 17: The non-transitory computer-readable medium of any of statements 15-16, wherein the target object is evaluated against the nominal object in the above step of the assembly process.

Statement 18: The non-transitory computer-readable medium of any of statements 15-17, wherein the target object is evaluated against the final configuration of the nominal object.

Statement 19: The non-transitory computer-readable medium of any of statements 15-18, wherein the instructions determine the sequence of steps using a machine learning model configured to minimize deviation by the processor is further configured to be

Statement 20: The non-transitory computer-readable medium of any of statements 15-19, wherein the deviation is minimized using a Markov Decision Process (MDP) via reward formulation.

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of U.S. patent application Ser. Application No. 16/587,366, filed February 28, 2019, is U.S. Application No. 16/289,422 entitled "DYNAMIC TRAINING FOR ASSEMBLY LINES," now U.S. Patent No. 10,481, This is a continuation application of '579. Further, this application is subject to U.S. Provisional Application Serial No. 62/836,192 entitled "A COMPUTATION MODEL FOR DECISION-MAKING AND ASSEMBLY OPTIMIZATION IN MANUFACTURING," filed on April 19, 2019, November 6, 2019. U.S. Provisional Application No. 62/931,448, entitled "A COMPUTATION MODEL FOR DECISION-MAKING AND ASSEMBLY OPTIMIZATION IN MANUFACTURING," filed on Nov. 7, 2019, and "DEEP LEARNING QUALITY PREDICTOR," filed November 7, 2019; No. 62/932,063 entitled FOR MANUAL ASSEMBLY VIDEO ANALYSIS. The entire contents of the above applications and patents are incorporated herein by reference.

The subject technology provides improvements to assembly line workflow and, in particular, a system for adaptively updating assembly line operator instructions based on feedback and feedforward error propagation predictions made using machine learning models. and methods. As discussed in more detail below, some aspects of the technology automatically generate instructional videos provided at one or more worker stations based on inferences made regarding manufacturing or assembly deviations. Systems and methods for matching are included.

Traditional assembly line workflows require human (operator) oversight and specialized techniques to detect manufacturing errors and to determine how errors can be eliminated through corrections in downstream processes. Technology is required. Note that assembly and manufacturing, and assembly line and production line are used interchangeably herein. Due to the reliance on humans to detect assembly errors, there is a high probability that errors will be overlooked (or not reported) and propagated directly downstream in the assembly process. Moreover, many assemblers are only trained to perform a narrow set of tasks, and thus are themselves trained to best correct errors that occur upstream in the assembly workflow. may not be aware of how to modify their workflow.

In traditional manufacturing workflows, the correction of human errors in part of the manual inspection process is often handled by taking corrective action on the human node. If there is an ongoing problem with the person, the person is often replaced by another person who is vulnerable to many of the same limitations as all humans are. It is difficult to repeat an action consistently over the years without committing an error, and taking corrective action is beyond the right of most builders. Even given these rights, there are contradictions and they can only be told by the experience the human has in the application of the single process node. Moreover, there is no mechanism for learning from any mistakes or even from any positive corrective actions.

In addition, electronic monitoring of assembly lines has limitations and robust mechanisms for providing opportunistic adjustments to downstream steps in the assembly line to compensate for errors made in upstream steps. not included. In addition, assess how changes in worker motion and/or changes in assembly patterns affect the resulting manufactured product, and take corrective action to improve the performance and/or characteristics of the manufactured product. A new mechanism is needed to provide

In some aspects, the disclosed technology relates to a method for optimizing a workflow on an assembly line, the method comprising, at a step of assembling the target object, detecting an error in assembling the target object; Evaluate the target and nominal targets at that step in the process, thereby obtaining a comparison, and, based on that comparison, the sequence of steps necessary to minimize the deviation between the target and nominal targets. and determining. In some aspects, the method may further include adjusting building instructions for the target object based on the sequence of steps.

In another aspect, the disclosed technology encompasses a system for optimizing a workflow on an assembly line, the system comprising a plurality of image capture devices, each image capture device for a target assembly process. with a plurality of image capture devices placed at different positions to capture the movements of the worker during and an assembly instruction module coupled to the plurality of imaging devices. a building instruction module receiving motion data from a plurality of image capture devices, the motion data corresponding to a worker performing a set of steps for building a target object; and determining an error in the assembly of the target object based on the motion data in one of the steps of . In some embodiments, the assembly instruction module evaluates a target object and a nominal object in said step of said set of steps, performs operations to thereby obtain a comparison, and based on said comparison , further configured to determine a sequence of steps required to minimize the deviation between the target target and the nominal target, and adjust the assembly instructions provided to the operator based on the sequence of steps. can do. Modified assembly instructions may be in the form of, but are not limited to, generated or edited video of motion data, text-based instructions from natural language processing (NLP) of identified deviations, or instructions to the operator. may include other feedback mechanisms.

In yet another aspect, the disclosed technology relates to a non-transitory computer-readable medium containing instructions stored thereon, the instructions, when executed by one or more processors, to a target object assembly process. In one step, detecting an error in the assembly of the target object; evaluating the target object and the nominal object in said step of the assembly process to thereby obtain a comparison; A processor is configured to execute instructions including performing the steps of determining a sequence of steps necessary to minimize the deviation between objects. In some implementations, the instructions may be further configured to cause the processor to perform operations for adjusting the building instructions for the target object based on the sequence of steps described above.

Particular features of the subject technology are set forth in the appended claims. However, the accompanying drawings, which are included to provide a further understanding, illustrate the disclosed aspects and, together with the description, serve to explain the principles of the subject technology.

1 conceptually illustrates a flowchart of an exemplary production line deployment, in accordance with certain aspects of the disclosed technology; FIG.

FIG. 3 illustrates an example process for implementing assembly error correction at a given worker station, in accordance with certain aspects of the disclosed technology;

1 illustrates an example electronic system that can be used to implement some aspects of the subject technology; FIG.

The detailed description presented below is intended to be a description of various configurations of the subject technology, and is not intended to represent only the configurations in which the subject technology can be practiced. The accompanying drawings are incorporated into this specification and constitute a part of the detailed description. The detailed description includes specific details to provide a more thorough understanding of the subject technology. However, it will be clear and obvious that the subject technology is not limited to the specific details shown herein, and that it may be practiced without these details. In some instances, structures and components are shown in block diagram form in order to avoid obscuring concepts of the subject technology.

Aspects of the disclosed technology overcome the above drawbacks of conventional assembly line process flows by providing methods for tracking, training, and incrementally improving production line assembly and resulting manufactured products. Deal with limits. Improvements are realized by providing dynamic visual or other feedback and instructions to individual builders, and in some embodiments worker feedback is based on errors, which are caused by: These may include, but are not limited to, assembly errors, ineffective processes and/or motions, and inferior product detected at one or more points in the production line.

By implementing the disclosed techniques, for example, based on near-real-time error detection, by rapidly revising and changing the reference/indicative information provided at individual stations (or all stations), manual implementation With respect to the speed of error correction of the method, its speed can be significantly improved. Although some embodiments described herein contemplate using reference/indicative information in the form of video, other formats are also contemplated. For example, assembly/manufacturing instructions may be provided to the assembler as audio, visual, textual and/or tactile cues or as other forms of reference. As an example, the audible instructional information may include verbal instructions or other audible indicators. Visual building instruction information may include video or animated formats such as the use of Augmented Reality (A/R) or Virtual Reality (V/R) systems. In some aspects, the visual assembly instructions may be provided as animations that provide examples of how the workpiece (or tool) should be handled by workers at a given station of the assembly line. Further, in some aspects, the assembly/manufacturing instructions may include machine instructions that can be received and implemented by, for example, a robotic or mechanical assembler. As used herein, the term worker can mean a human, robot or machine that uses motion to assemble a manufactured product. Further, the term worker encompasses human-assisted manufacturing implementations, such as in instances where human workers work with or are assisted by robots or machines. there is

In instances where assembly/manufacturing instructions are provided as reference/instruction videos, such videos are sometimes referred to as standard operating protocols (SOPs). Systems of the disclosed technology can be effectively deployed due to minimal hardware requirements, such as the use of video cameras and displays for individual workers, while machine learning training, updating and error propagation can be implemented on centralized computing resources, such as in computing clusters or cloud environments.

In some aspects, video instructional information may be provided to one or more workers as part of an augmented reality display. instructions, or deviations from standard assembly/manufacturing methods, can be communicated to workers using augmented reality, and displays can be enhanced video, animated graphics, and/or recorded scenarios. provided as a mixture of video data representing As an example, an augmented reality display may provide instructions or guidance provided as an animated or graphical overlay to a real-time feed of the workpiece being assembled and/or the tools being used in the assembly/manufacturing process. can be done.

In some implementations, systems of the disclosed technology include one or more video or motion capture devices located at various worker stations on a production line. A capture device is configured to record the worker's motion/interaction with parts, devices, materials or other tools (“components”) at that particular station. In some aspects, worker motion can be captured using video recording, but other motion capture formats, such as 3-D points representing worker motion and/or worker interaction with a tool or manufactured product. Cloud use is also contemplated. In addition, it is also possible to create a station-by-station reference video by recording the motion of one or more technicians to a particular station and their interactions with the components at that station. This video can be created from a single example of the expert's actions or from multiple examples. Motion paths for each expert can be derived, and in embodiments where several experts or several examples are used, performing a calculation on the set (e.g., average) of the derived motion paths, A reference video can be created for a particular station. A reference video may be in the form of a digital or animated representation of the motion path to be performed at that particular station. Note that a skilled person can mean someone with specialized skills or knowledge of the particular assembly steps for which guidance is provided.

In some embodiments, video or motion capture devices placed at various worker stations on a production line can be used to calculate assembly errors for workpieces/components/ Tool attributes (eg quality, tensile strength, number of defects) can also be captured.

By capturing a worker's interaction at each station, worker error can be determined by comparing the captured interaction with a baseline (ground truth) model representing the ideal/skilled worker interaction/workflow. can be detected. assembly errors that can be corrected at various locations in the assembly chain, i.e. using operator deviations from an idealized interaction model, for example by revising operator instructions/guidance provided to different stations can be calculated. Additionally, it is possible to capture the quality of the components at an individual station and compare against the baseline component for that station. Also, the deviation of a component from a baseline component can be used to assign a quality grade to a component at a particular station, or make repairs by revising operator instructions/guidance provided at various stations. / It is also possible to calculate the assembly error.

Assembly modifications can be implemented in a variety of ways depending on the desired implementation. In some embodiments, operator variations/errors are used to classify parts, for example, into quality grades (e.g., A, B, C, etc.) and subsequently direct these parts to the appropriate production line. can be implemented. In another aspect, detected assembly errors can be used to revise processes at a given station, thereby improving quality and reducing variability. That is, detected assembly errors can be used to automatically provide instructions or guidance to the same station, thereby, for example, correcting errors introduced at that station (eg, in-station rework). . NLP can be used to process instructions or guidance to workers. For example, NLP can be used to convert verbal instructions to text form, or vice versa.

For example, assembly error detection can be used to drive updates/changes to operator instructions or video provided to a given station where an error is known to occur. As an example, once an error/deviation is identified as being due to a first worker working at a first station, the The provided assembly instructions can be revised, thereby reducing the error spread associated with the manufactured articles leaving the first station.

In another aspect, detected assembly errors can be used to revise subsequent station assembly, thereby overcoming station dispersion. That is, error detection can be used to automatically trigger downstream propagation of new/updated building instructions based on errors caused by upstream workers. For example, error distribution for motions performed by a first worker may be used to adjust assembly instructions provided to a second worker associated with a second station downstream from the first station. can.

Yet another aspect is that the detected error distribution can be propagated forward across all stations so that full or partial rework can be performed over the course of the rest of the downstream assembly chain. can be guaranteed. That is, by adjusting the assembly instructions provided to one or more downstream workers, errors generated across one or more stations can be corrected/reduced. In one example, via work subsequently performed by a second worker at a second station and by a third worker at a third station, i.e., to the second and third stations By adjusting the assembly instructions, the error distribution of the manufactured article caused by the first operator at the first station can be corrected.

In another example, the error variance accumulated over multiple stations can be reduced by one or more subsequent stations. For example, by adjusting assembly instructions provided to a third station and a fourth station (e.g., a third worker and a fourth worker, respectively), Error distribution in the article of manufacture can be subsequently repaired.

By treating individual workers/worker stations in the assembly flow as network nodes, machine learning models are used to optimize the assembly process by minimizing errors through reduction of assembly variance at individual nodes (stations) can be optimized. By minimizing individual node (worker) variance and implementing real-time updates to mitigate forward error propagation, the system of the disclosed technology can significantly reduce manufacturing variance to the final product. can. Additionally, by accurately quantifying and tracking error contributions from specific segments in the assembly workflow, products can be graded and sorted by product quality or amount of deviation. Thus, for example, depending on product quality, products of a particular quality class can be directed to different manufacturing processes or to different customers.

Error detection can be performed using machine learning/artificial intelligence (AI) models and/or corrections necessary to optimize station assembly changes can be performed. By way of example, the machine learning model may include, but is not limited to, final product ratings, final product change statistics, desired end product characteristics (e.g., assembly time, amount of materials used, physical properties, number of defects, etc.). , station-specific component ratings, station-specific component variations, and desired station component characteristics. Additionally, the deployed machine learning model is provided by an expert or "master designer" so that formal knowledge can be represented in the idealized model used to perform the error detection and error quantification calculations. It can be trained or initialized based on the input it receives.

As will be appreciated by those skilled in the art, machine learning-based classification techniques can be modified according to desired implementation without departing from the disclosed technology. For example, machine learning classification schemes include Hidden Markov Models, Recurrent Neural Networks, Convolutional Neural Networks (CNN), Reinforcement Learning, Deep Learning, Bayesian Symbolic Models, Generative Adversarial Networks (GAN), Support Vector Machines, Image Registration Methods, Applied One or more of the possible rule-based system and/or any other suitable artificial intelligence algorithms may be utilized alone or in combination. Where regression algorithms are used, they can include, but are not limited to, Stochastic Gradient Descent Regressor and/or Passive Aggressive Regressor, and the like.

Machine learning classification models include clustering algorithms (such as the Mini-Batch K-Means clustering algorithm), recommendation algorithms (such as the Miniwise Hashing algorithm or the Euclidean Locality-Sensitive Hashing (LSH) algorithm), and/or local outlier factor methods. can also be based on the anomaly detection algorithm of Further, the machine learning model may be subjected to dimensionality reduction such as one or more of a Mini-batch Dictionary Learning algorithm, an Incremental Principal Component Analysis (PCA) algorithm, a Latent Dirichlet Allocation algorithm and/or a Mini-batch K-means algorithm, etc. It is also possible to use the technique.

In some implementations, multiple different types of machine learning trained/artificial intelligence models can be deployed. As an example, general forms of machine learning can be used to dynamically adjust assembly line processes to optimize manufactured products. As those skilled in the art will appreciate, the selected machine learning/artificial intelligence model does not simply contain a set of assembly/manufacturing instructions, but feedback on the entire assembly line process and the resulting A method for providing its effect on manufactured products and also for providing dynamic adjustments to downstream worker stations in an assembly line, thereby compensating for actions occurring at upstream worker stations. This type of artificial intelligence-based feedback and feedforward model is referred to herein as Artificial Intelligence Process Control (AIPC). In some embodiments, machine learning utilizes targeted gated recurrent unit (GRU) model-based learning and Hausdorff distance minimization to effectively explore a space of possible recovery paths, thereby assembling It can be based on deep learning models in simulated environments that find optimal paths to correct errors during the process. In still other embodiments, for example, a machine learning algorithm can be based on the Long Short-Term Memory model, which can analyze the video input of the assembly process and predict the final quality metric (output). . Additionally, machine learning models, in the form of NLP algorithms, are used to maximize worker compliance and comprehension of coordinated instructions, such as text-to-speech or speech-to-text. You can also adjust the feedback to the station.

In some embodiments, computational models that utilize machine learning can be used to correct errors during the manual assembly process.

A target object can be assembled through a series of steps defined by a procedure. During the process, irreversible errors can occur at certain steps k that would require revision of all remaining work in order to obtain the final configuration of the nominal object. In some embodiments, the technique for correcting the error includes comparing the defective target object at step k to a nominal target at the same step k, or comparing the defective target object at step k to its A step of comparing against a nominal target of the final configuration may be included. These comparisons can be used to determine the sequence of steps required to minimize the deviation between the final configuration of the defective target object and the final configuration of the nominal object. In some embodiments, the target object quality metric can also be used to guide the correction of defective target objects.

A general numerical approach can solve the problem by using the Hausdorff distance algorithm, whereby a sequence of k steps to assemble the faulty target object transforms the nominal object into its final configuration. It can be determined how similar the sequence of steps to assembly is. Some methods for computationally minimizing the Hausdorff distance between the above sequence of k steps in the faulty assembled object and the sequence of steps in the final assembled nominal object are instantaneous Optimizing the Markov Decision Process (MDP) via reward formulation, multiple reward formulation, or delayed reward formulation. However, the search spaces associated with these formulations require a significant amount of computational resources.

Alternatively, machine learning frameworks can be developed with delayed reward policy agents using reinforcement learning. A reinforcement learning framework is designed to let the policy agent determine the appropriate steps required to correct the faulty target target error, resulting in a final configuration with a performance metric that matches the nominal target performance metric. can do. Rewards given to policy agents are delayed and paid to policy agents only when the final step is executed.

In some embodiments, the design for the optimal/desired manufactured product can be selected and a skilled worker deployed to individual workers to assemble the manufactured product according to the selected design. Individual steps performed at the station can be implemented. Optimal can be based on the desired performance and/or characteristics of the resulting product that minimize errors in the resulting manufactured product, or some other criteria (e.g., if the manufactured product is a paper airplane). , the optimal paper plane would be the one that achieves the desired flight goal). Using multiple imaging devices to capture worker motion and worker interaction with the manufactured product they are assembling, thereby generating video, images and/or 3D point cloud data can be done. The captured data includes the worker's hand coordinates in relation to the manufactured product as it is being assembled, the relation of one hand to the other, and the fingers (and some Detailed information such as knuckle (knuckle) relationships in embodiments may be provided. Data collected from skilled workers can be used as ground truth for assembling the optimal/desired manufactured product. This ground truth from a single example is, by itself, sufficient for use in creating initial machine learning models, or additional data can be collected. For example, deploying many workers to assemble an optimal manufactured product to understand how variations in worker motion or errors can affect the resulting manufactured product. Or multiple steps can be performed. This can be done for each worker station on the assembly line. Both the resulting final products and their respective assembly processes are compared against each other and against the ground truth so that errors and/or changes in operator motion are reflected in the characteristics of the manufactured product and/or Or it can determine how performance can be affected (eg, operator speed can result in a plane of lesser quality). Data collected on a worker basis during the actual assembly process (i.e., a process in which a human, robot, or machine is performing motions at one or more stations) is herein referred to as "actual It will be called "training data". The actual training data can be supplemented with simulated data to obtain a richer data set and to provide additional variations to achieve optimal manufactured product. can. Note that the terms "optimal" and "desired" will be used interchangeably herein.

In some embodiments, the different AI/machine learning/deep learning models discussed herein are used below to achieve Artificial Intelligence Process Control (AIPC) to optimize the assembly of articles of manufacture. It can unfold in the specific order described. An exemplary process by which an AIPC deep learning model may be implemented is discussed in greater detail in connection with FIG. 1 (eg, in connection with AIPC deep learning model 112) and in connection with FIG. An example of a hardware system and/or device that can be used to implement an AIPC deep learning model is provided in FIG. 3, and a corresponding description is also provided below.

First, CNNs can be used in assembly line processes to classify characteristics of workers' hands and different configurations of manufactured articles at individual worker stations.

Second, both Reinforcement Learning (RL) and RL agents can be used and rewarded to achieve desired results from CNN classification and for defined desired results. . The RL agent may or may not monitor.

Third, Generative Adversarial Networks (GAN) can be used to choose between competing RL agents. A GAN can include minimal human oversight, relying only on humans to select RL agents that enter the GAN as nodes.

Fourth, RNNs can take winning RLs as input nodes to create feedback and feedforward systems, thus making learning continuous and monitoring learning. It doesn't have to be.

Implementations of these four AI/machine learning models are discussed in more detail below.

In some embodiments, the actual training data classifies relevant data in the assembly process, e.g., fingers/hands used in individual assembly steps per worker station, of the product being assembled, The part touched by the operator's finger at any point in time and space, and the shape or configuration of the manufactured product being assembled at any point in time and space, can be input into the CNN to classify.

In still other embodiments, hand motions are not tracked, but data representing various changes in the assembly pattern of the manufactured product may be collected (e.g., if the manufactured product is a folded paper airplane, folding Data can be collected based on changing the order, implementing folding changes, and/or introducing potential errors, and if the manufactured product is a clothing article, e.g. Data may be collected based on implementing changes and/or introducing potential errors). This data can be simulated and/or gathered from actual training data. The resulting manufactured products and their respective assembly processes can be compared to determine how errors or changes in assembly patterns affect the characteristics and/or performance of the manufactured products.

In some embodiments, captured data (eg, video and hand tracking of the assembly process, etc.) is used to predict final output quality. This quality prediction allows the captured data to be used during the manufacturing process to classify products into quality bins without the need to manually inspect product quality and to take corrective action downstream. be able to.

In some embodiments, the system can focus on manual assembly of a target object, where the assembly process involves a worker performing different tasks on the target object for each set of instructions. Contains several discrete steps. The system uses a machine learning framework (across all worker actions) that uses deep learning models to establish correlations between the time series of worker hand positions and the final quality of the target object. can be built. In some embodiments, the model may consist of two neural networks, a first neural network used to extract the position data of the worker's hand in the 3D environment, and a second neural network A neural network is used to remove unnecessary elements from the hand position data and correlate it with the final quality of performance of the target object.

In some embodiments, the first neural network uses the video acquisition system to generate images during the assembly process in different node videos corresponding to individual discrete steps that the worker performs in connection with the assembly of the target object. , a video of the worker's hand can be recorded. For example, a worker may carry out an assembly process using several cameras positioned at different locations and configured to simultaneously record the assembly process. These cameras can be used to capture video multiple times at predefined positions of the worker's hand. These videos can then be processed to extract a number of images or landmark frames representing the overall target object assembly process. These landmark frames can be used to extract hand tracking information that facilitates defining the positions or key points of the worker's hands and fingers during the assembly process.

In some embodiments, a bounding box prediction algorithm and a hand keypoint detector algorithm can be applied to extract hand tracking information. Specifically, the bounding box prediction algorithm may involve using threshold image segmentation to process the landmark frames from the assembly process, thereby obtaining a mask image for the worker's hand. The hand can be positioned over the mask using blob detection. Bounding box prediction uses the mask image to form a box around each of the worker's hands such that the box contains the highest point of the hand position shape at least up to the hand list point. The bounding boxes and their corresponding landmark frames are then fed to the hand keypoint detector algorithm.

A hand keypoint detector algorithm can include a machine learning model that can detect specific keypoints on a worker's hand. The hand keypoint detector algorithm predicts not only the keypoints that are visible in the landmark frame, but also the keypoints that are hidden (occluded) from the frame due to joint, viewpoint, object, and hand interactions. can be done. A particular occlusion (occlusion) point in one frame cannot be occluded (occluded) in another frame because different hand positions produce different occlusion (occlusion) points in different frames. A hand keypoint detector predicts the location of obscured (occluded) keypoints with a certain level of certainty. However, prediction of hidden (occluded) keypoint locations can result in the same keypoint locations being recorded for different hand positions. During the steps of the manual assembly process, the hand keypoints defining the worker's hand are then provided to a second neural network.

In some embodiments, a second neural network is used to predict the final state quality of the assembled object. In some embodiments, the neural network can be based on the Long Short-Term Memory (LSTM) model. The LSTM has several cells that are arranged in sequence and together represent the overall assembly process of the final object. The input to the LSTM cell may be hand keypoint data corresponding to a worker's work in the particular step of the assembly process represented by the LSTM cell. Each cell in the LSTM decides whether to store information from the preceding cell, chooses a value to update, performs an update to the cell, chooses a value to output, and then chooses Filter the values so that the cell outputs only the values. An LSTM may be a sequence for one model trained using the Adam optimizer or other adaptive learning rate optimization algorithms. The neural network uses the LSTM framework to correlate input data extracted from the manual assembly process, thereby determining quality measurements of the final product.

In some embodiments, the video and hand-tracking information representing the building process for the target object, i.e. the input data used to train the model, are generated from multiple models using a single set of building instructions. It can be collected from multiple workers performing an assembly process for assembling the target object. A target object assembled by a worker can be used in a controlled environment to collect corresponding quality measures for the performance of the assembled object, i.e. the output data required to train the model.

In some aspects, the training data used to generate the machine learning model is from simulated data, from actual training data, and/or from expert ground truth recordings. There may be, and they may be combined or separate. In some embodiments, the simulated data results can be used to build machine learning models, such as (but not limited to) Reinforcement Learning (RL) agents. In other embodiments, the actual training data may be used to build machine learning models, such as (but not limited to) Reinforcement Learning (RL) agents. RL agents are rewarded for achieving good/desired results and penalized for bad results.

In some instances, many RL agents, some based on real training data and some based on simulated data, can be deployed to work in tandem, It can also be configured to maximize the cumulative award, e.g. build the manufactured product with the smallest deviation from the ideal model/example. Exemplary outcomes that may reward the RL agent include completing a complete manufactured product in as few steps as possible, reducing the amount of materials or time required to achieve the manufactured product. RL agents based on simulated data and RL agents based on actual training data can be used to determine optimal motion patterns and/or optimal assembly patterns that result in optimal/desired manufactured articles.

These two groups of RL agents (e.g., RL agents created based on real training data, and RL agents created based on simulated data) are both optimal/desired manufactured products. You can work together and even compete here because you are paid for the act of creating In some embodiments, data obtained from simulated-based RL agents that yielded optimal assembly patterns for optimal manufactured products is used to reduce the probability space for the actual training dataset. be able to. For example, a simulated RL agent can be used to determine the optimal assembly pattern, then collect actual training data only for the optimal assembly pattern, not for the non-optimal assembly pattern. be able to. By focusing only on the collection of actual training data, or only on the optimal assembly pattern, less training data can be collected and/or a larger training data to collect more actual training data. Capacity can be achieved, but only for optimal assembly patterns.

Relying solely on reinforcement learning to optimize assembly lines is limited because rewards can sometimes be conflicting. For example, in product assembly, for the least amount of spurious motion (e.g., folding and immediately unfolding, or adding a seam and immediately removing the seam), several RL agents , while other RL agents can be rewarded for speed. An RL agent that is paid for speed can determine that more spurious movements result in faster assembly times because fewer modifications are required downstream in the assembly process. Making such implementation trade-off decisions is not something that can be readily determined by a human being. Even with experience and a large number of examples, humans lack the computational power to understand how subtle it is to perceive the end result of different workers working in different ways. still lacking.

To resolve these conflicting RL agent optimizations, GANs can be deployed to act as decision makers. Conflicts may arise between RL agents based on actual training data, between RL agents based on simulated data, and/or between RL agents based on actual training data and RL agents based on simulated data. may be

In some embodiments, the GAN can test each of the RL agents and store the results to create more robust neural networks. GANs operate by taking RL agents and using models that yield winners and losers in zero-sum games. There are "generators" and "discriminators" in GANs. The generator would in this case store reward data from opposing RL agents, and the discriminator would evaluate which of these was most relevant to the task of creating the desired manufactured product. will do. A GAN uses a deep network of nodes (or neurons) to determine how to weight nodes. Since each RL agent believes that each RL agent has already made the optimal decision, it is impossible to determine which of the conflicting RL agents actually makes the most relevant choice. It is the role of the GAN and the discriminator adjusts the weights accordingly. When a zero-sum game is initiated between opposing RL agents, groups of winners are generated among the opposing RL agents, and only those winners are used to optimize workflow on the assembly line. will be used for machine learning models that are Although a large amount of data can be generated to determine the winning RL agent, the results are more than those used to create and find those winners used as input nodes. is also much poorer.

Once the RL agents that survived the GAN war and were appropriately paid are determined, they can be input into another AI system called the Recurrent Neural Network (RNN) in some embodiments. RNN has many similarities to CNN in that it is a Deep Learning Neural Network and the final result is optimized through various forms of weighting of the input data. One difference is that unlike CNNs, which are linear processes from input to output, RNNs are loops that feed back the resulting output, as well as internal nodes, as new training information. RNNs are both feedback and feedforward systems such as GRUs.

In some embodiments, a machine learning framework can be built utilizing targeted GRU model-based learning. The GRU model can be chosen instead of reinforcement learning because of its predictive power and relatively short training time. The GRU is used by the RNN to distinguish between observations that should be stored in memory or update state and observations that should be forgotten or reset state.

In some embodiments, a GRU model can consist of a number of GRU cells corresponding to the number of assembly steps required to build the target object. Each GRU cell representing one of the number of assembly steps can have several input parameters and hidden state outputs. The GRU cell, which represents the final step in the assembly process, will output the target object. The output of the model is the deviation of the target object from the nominal object. This deviation can be calculated using the stepped Hausdorff distance from the target object to the nominal object and the performance metric of the final configuration of the nominal object. Individual GRU cells are defined by reset, update and new gates. The GRU neural network is iteratively trained, thereby biasing the GRU neural network toward solving a particular sub-problem, and identifying a set of weights for the GRU. For example, in each iteration several predictions (one prediction for each possible error in a particular step) are generated to complete the assembly process in subsequent steps. Additionally, a corresponding predicted distance measure for the modified assembly process can be generated. These predicted assembly process completions can be depicted in a virtual representation system and their stepwise Hausdorff distances calculated to obtain a "ground truth" distance measure. The difference between the "ground truth" and the predicted distance measure can be computed and fed back into the model, whose network weights are adjusted via backpropagation to produce the next iteration. This process can continue until the set of weights for the GRU is identified. In some embodiments, a stochastic gradient descent method can be used to correct the defective target object and derive the steps necessary to obtain a satisfactory final configuration. can.

In some embodiments, a simulation, such as a parametric computer-aided design and drafting (CAD) model of the target object in process, can be generated to develop and validate the machine learning model. A CAD system can use a local coordinate frame corresponding to the current state of the target object being processed, and input parameters representing individual assembly steps. Using the local coordinate frame of the target object under processing and the input parameters, the CAD system can determine dimensional information for each assembly step. A CAD system can then generate a three-dimensional CAD model that represents the configuration of the outputs at the individual steps. The CAD system can continue the process until all steps in the assembly process have been performed and output a CAD model of the final configuration of the assembled object. By providing various input parameters to the CAD system, CAD models of different configurations can be generated. To obtain a set of CAD models having a specified range of input criteria, such as length or width, to provide a statistical sample of this input criteria to a CAD system to generate the set of CAD models. can be done.

CAD models may be of varying levels of detail and intelligence, but trained models and systems are specifically designed to work with lower detail CAD systems, thereby making them a non-computationally expensive method. can generate a vast number of examples in , and provide sufficient surface morphology detail for trained models and profiling. In some embodiments, the referenced CAD system can be paired with Finite Element Analysis (FEA) or basic surface modeling tools to generate a structural analysis of the surface. This data can be used as an additional quality score for model training and analysis.

In some embodiments, the CAD system can be incorporated into model training so that additional surface models can be generated as required for examples from the model or as needed for additional exploration data. This approach is coupled with physical observation and allows pre-trained models to be developed without the need for a vast amount of physical samples in space.

In some embodiments, a CAD model of the final configuration of the assembled object may be used for simulation to generate performance metrics. By using a CAD model of the final configuration of the assembled object, the simulation can utilize numerical and computational methods to generate performance metrics.

The real-world application of Artificial Intelligence Process Control (AIPC) is to provide feedback (e.g., by automatically correcting video instructions) to workers on the assembly line who have already completed their tasks, as well as to provide feedback downstream on the assembly line. side, providing instructions to workers who have not yet completed their tasks (again, eg, by automatically modifying video instructions) (“feedforward”). This feedback-feedforward system, or AIPC, can be achieved using the AI techniques described herein, in some embodiments in the specific order described herein. Thus, workers on the assembly line can make choices that optimize the resulting manufactured product without the need for additional human oversight.

In some embodiments, this simply involves compressing the system into an RNN and observing any movement during the process of creating one or more manufactured products, in two ways, success or failure. Individual movements work as training. If the RNN's output node is not optimal, the network can feed back to the actual individual on the assembly line to make a different choice, and in the path through many nodes and layers of the RNN, the weights can be redone and the output will be labeled as either success or failure. As the process iterates, the weights improve their accuracy on their own. Additionally, the network can learn what is working and what is not working, even if the individual performing the assembly is not working. This increases the training set. It also allows adjustments to be made at different stages of the assembly process. In some instances, the best way to produce an article of manufacture with specific properties at any given moment is not to return to the starting point, but to adjust the instructions as the process progresses. can be found. In this case, the RNN is constantly optimizing for the best manufactured product and also provides feedback to individual workers at worker stations on the production line who have already performed their tasks and also learn to feed forward information to workers at worker stations who have not yet performed their tasks.

FIG. 1 conceptually illustrates a flowchart of an exemplary process 100 for implementing production line deployment, in accordance with some aspects of the disclosed technology. The process of FIG. 1 begins at step 102 to initiate production deployment. An exemplary production line workflow typically includes multiple worker stations (nodes) at which workpieces (products) are assembled or manufactured. The various nodes can be organized in a sequential fashion so that work on each subsequent node begins only after the work on the preceding node is completed.

At step 104, one or more reference videos are generated and/or updated. The videos described above can be used to provide manufacturing/assembly instructions to specific nodes (also referred to herein as worker stations). That is, each node in the workflow can have a reference video that delivers guidance on how to complete the step in the manufacturing workflow that corresponds to that particular node.

At step 106, each of the videos generated at step 104 is deployed to respective stations/nodes. As an example, a given workflow may contain 10 nodes, each with their own different/unique reference video. In other implementations, the number of videos may be less than the total number of nodes. Depending on the implementation, the reference videos deployed to various stations/nodes may be unique or may provide similar guidance/instructions. As discussed in more detail below, the content of the reference video may be dynamic and constantly updated/augmented.

At step 108, continuous recordings of motion are captured at individual stations/nodes. Motion data obtained from motion recordings can describe worker interactions with workpieces/components/tools at those nodes/stations in the workflow. That is, the motion data captured at an individual node can represent one or more worker actions corresponding to a particular portion of product assembly or manufacturing, and is provided by the reference video associated with that node. Able to respond to instructions. In some instances, motion capture may include capturing video data, ie, recording all or part of the worker's actions at the station. In other embodiments, motion capture can include, for example, recording a 3D point cloud, where motion is recorded for one or more specific points in the field of view of the image capture device. Both worker actions as well as component attributes (eg, component quality, tensile strength, number of defects) can be captured at individual nodes/stations in the workflow.

At step 110, process method analysis deviations can be calculated, analyzing the motion data captured for one or more of the stations at step 108 to obtain, for example, an idealized motion profile of the corresponding station. Any deviation from the comparative model that includes (or represents) can be identified. As illustrated in FIG. 1, step 110 comprises, for example, identifying/classifying motion deviations from the comparison model and making inferences as to the degree to which the assembly or manufacturing process may have been affected. An AIPC deep learning model can be utilized (step 112). The comparison can be performed on a per station level and/or on an overall process level. The analysis can also consider the attributes of the components at individual stations, or the extent to which the deviations of the components from the baseline and the motion deviations of the stations affect the quality of the components.

The AIPC deep learning model called in step 112 is based on collection of various types of training data, which can include examples of ideal or quality-controlled assembly/manufacturing interactions for a given station/node, for example. be able to. AIPC deep learning models are also trained by feedback from customers regarding specific products manufactured using process 100 (step 111), and for specific products manufactured using process 100 (step 113). Data provided by domain/industry information 115 can be used to augment (or adjust) feedback from quality control inspections. It should be appreciated that AIPC deep learning models can be implemented using a variety of computing systems including distributed hardware and/or software modules. As an example, an AIPC deep learning model is deployed on an assembly line and multiple images combined into one or more systems configured to implement various AI/machine learning models and/or classifiers. It can be implemented using a distributed system that includes a capture device and a display device.

Once deviations from the comparative model are detected/identified at step 110, the AIPC deep learning model 112 can be used to generate automatic adjustments at step 114. FIG. As discussed above, video adjustments can be aimed at improving manufacturing/assembly quality at one or more stations of the workflow. where errors are known (or predicted) to occur, e.g. Video adjustments can be applied to a given node/station. In other implementations, video adjustments can be applied downstream from the station where the error occurred, eg, to correct the error before the manufacturing workflow is complete. In yet other implementations, once a workflow is completed, the entire workflow can be analyzed and adjustments can be made to one or more stations in the workflow.

In some embodiments, adjustments are made in real-time immediately after an error is detected. In other embodiments, adjustments are made at regular intervals or after the workflow is completed.

In some aspects, the automatic adjustments determined at step 114 may be summarized at step 117 and/or provided as a product quality report. One or Multiple quality reports can be generated.

FIG. 2 illustrates an exemplary process 200 for performing error detection analysis that may be used to facilitate assembly error correction, in accordance with some aspects of the present technique.

Beginning at step 210, the process of improving manufacturing/assembly using idealized video guidance can be implemented. At step 215, video tracking of one or more assembly stations is performed. Video tracking can include recordings of human workers at a given station/node. In some embodiments, video tracking can further include capturing component attributes at a given station/node.

At steps 220-224, processing is performed to analyze the video recorded from the assembly station. For example, in some embodiments background extraction can be performed to isolate motion/components in the recorded video. In some aspects, once background extraction is complete, the processed video includes only motion/video data related to the assembly worker (step 224), and the included components are the corresponding assembly steps (step 224). step 220). At step 220, additional processing may be performed to isolate parts/components. As illustrated by the diagram of process 200, step 220 performs additional processing operations including anomaly detection (step 221), surface change detection (222) and part classification and/or quality scoring (step 223). can contain. Any video processing step is performed using various signal and/or image processing techniques including, but not limited to, the use of one or more AI/machine learning algorithms and/or classifiers, can perform anomaly detection (221), surface change detection (222), and/or scoring/classification (step 223), for example.

After processing steps 220-224 are completed, process 200 may proceed to step 226 where a motion comparison is performed. Motion comparison (step 226) compares process assembly station video data including one (one) or more station workers at one or more stations/nodes with corresponding idealized video/motion data. can include Motion comparisons performed across multiple stations/nodes can be used to infer/predict the resulting change in part/component quality.

At step 228, distribution/quality classification for various parts/components may be performed. As an example, parts/components can be classified into different quality tiers and/or identified for removal or repair depending on their associated classifications/differences.

After the classifications/differences have been determined, process 200 can proceed to step 230 where an analysis of the overall process/workflow is performed, e.g. conducted based on By analyzing the entire workflow, automatic adjustments to the video can be performed to address detected deviations/defects, as discussed above.

FIG. 3 illustrates an exemplary processing device that can be used to implement systems of the disclosed technology. Processing device 300 includes a master central processing unit (CPU) 362, an interface 368 and a bus 315 (eg, PCI bus). Operating under the control of appropriate software or firmware, CPU 362 is responsible for performing various error detection monitoring and process adjustment steps of the disclosed techniques. CPU 362 preferably performs all these functions under the control of software, including an operating system and any suitable application software. CPU 362 may include one or more processors 363, such as processors from the Motorola family of microprocessors or from the MIPS family of microprocessors. In an alternative embodiment, processor 363 is hardware specifically designed to control the operation of AIPC system 310 . Memory 361 (such as non-volatile RAM and/or ROM) also forms part of CPU 462 in certain embodiments. However, there are many different ways memory can be coupled to the system.

In some aspects, processing device 310 may include or be coupled to image processing system 370 . The image processing system 370 can include various image capture devices such as video cameras that can monitor worker movement and generate motion data. As an example, image processing system 370 may be configured to capture video data and/or output/generate a 3D point cloud.

Interface 368 is typically provided as an interface card (sometimes called a "line card"). Generally, they control the transmission and reception of data packets over networks and sometimes support other peripherals used with routers. Interfaces that may be provided are, among others, an Ethernet interface, a frame relay interface, a cable interface, a DSL interface, a token ring interface, and so on. In addition, various very high speed interfaces can be provided such as high speed token ring interfaces, radio interfaces, Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces, and so on. Generally, these interfaces may include ports suitable for communicating with any suitable medium. In some cases, they may also include independent processors, and in some instances may include volatile RAM. Independent processors can control such communication-intensive tasks as packet switching, media control and management. By providing separate processors for communication-intensive tasks, these interfaces can allow the master microprocessor 362 to efficiently perform routing calculations, network diagnostics, security functions, and the like.

The system shown in FIG. 3 is one particular processing device of the present invention, but by no means the only network device architecture in which the present invention can be implemented. For example, architectures with a single processor that handles communications as well as routing calculations, etc., are often used. Additionally, other types of interfaces and media may also be used.

Regardless of the configuration of the network device, the network device will include one or more memories or memory modules (including memory 361) configured to store program instructions for general network operations and the Mechanisms for roaming, route optimization and routing functions provided by the standard can be used. Program instructions may, for example, control operation of an operating system and/or one or more applications. One or more memories can also be configured to store tables such as mobility bindings, registration and association tables, and so on.

The logical operations of various embodiments are represented as (1) a sequence of computer-implemented steps, operations or procedures running on programmable circuits within a general-purpose computer; (2) a sequence of computer-implemented steps, operations running on application-specific programmable circuits. or procedures; and/or (3) implemented as a program engine within interconnected machine modules or programmable circuits. System 300 can practice all or part of the described methods, may be part of the described systems, and/or can act according to the instructions of Such logical operations may be implemented as modules configured to control processor 363, thereby performing specific functions according to programming of that module.

It is understood that any specific order or hierarchy of steps in the processes disclosed is an illustration of exemplary approaches. Based on design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged, or that only some of the steps illustrated may be performed. Some of the steps can be performed simultaneously. For example, multitasking and parallel processing may be advantageous in certain situations. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, nor should the described program components and systems , in general, may be integrated together within a single software product, or may be packaged into multiple software products.

The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may also be applied to other aspects. Accordingly, it is not intended that the claims be limited to the embodiments shown herein, and that the claims be construed to the full extent consistent with the language of the claims, singular References to elements of form shall imply "one or more" rather than implying "one and only", unless expressly stated to the contrary. is intended.

A phrase such as "an aspect" does not imply that such aspect is essential to the subject technology or that such aspect applies to all configurations of the subject technology. A disclosure relating to an aspect may apply to all configurations, or to one or more configurations. A phrase such as an aspect can refer to one or more aspects, and one or more aspects can refer to an aspect. A phrase such as "configuration" does not imply that such configuration is essential to the subject technology or that such configuration applies to all configurations of the subject technology. A disclosure relating to a configuration may apply to all configurations, or to one or more configurations. A phrase such as a configuration can mean one or more configurations, and one or more configurations can mean a configuration.

The word "exemplary" is used herein to mean "serving as an example or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as over, preferred, or advantageous over other aspects or designs.
disclosure statement

Statement 1: A method for optimizing a workflow in an assembly line, the method comprising, at a step of a target assembly process, detecting an error in the assembly of a target object and a target at that step of the assembly process. evaluating the target and the nominal target thereby obtaining a comparison; and based on the comparison determining the sequence of steps required to minimize the deviation between the target target and the nominal target; and adjusting the building instructions for the target object based on the sequence of steps.

Statement 2: The method of Statement 1, wherein the target object is evaluated against the nominal object in the above steps of the assembly process.

Statement 3: The method of any of statements 1-2, wherein the target object is evaluated against the final configuration of the nominal object.

Statement 4: The method of any of statements 1-3, wherein the sequence of steps is determined using a machine learning model configured to minimize deviation.

Statement 5: The method of any of Statements 1 through 4, wherein the deviation is the sequence of steps for completing the target object assembly process and another sequence of steps for completing the nominal object assembly process is determined based on the similarity between

Statement 6: The method of any of statements 1 through 5, wherein the deviation is minimized using the Markov Decision Process (MDP) via reward formulation.

Statement 7: The method of any of Statements 1 to 6, wherein stochastic gradient descent is used to derive the sequence of steps for completing the target object assembly process.

Statement 8: A system for optimizing workflow on an assembly line, the system comprising a plurality of image capture devices, each of which captures worker movement during a targeted assembly process. a plurality of image capture devices positioned at different locations to capture and an assembly instruction module configured to automatically modify guidance and instructions provided to an operator, wherein the plurality of image captures an assembly instruction module coupled to the device, the assembly instruction module receiving motion data from the plurality of image capture devices by the assembly instruction module, the motion data representing the movement of the target object by the operator; determining errors in building a target object based on steps corresponding to performing a set of steps for assembly and motion data in a step of said set of steps; and said set of steps. Evaluating the target target and the nominal target in the above step of thereby obtaining a comparison, and based on the comparison, the sequence of steps necessary to minimize the deviation between the target target and the nominal target and adjusting the assembly instructions provided to the operator based on the sequence of steps.

Statement 9: The system of Statement 8, wherein the motion data includes digital recordings of the worker's hand movements while assembling the target object.

Statement 10: The system of any of Statements 8-9, wherein the assembly instruction module is further configured to apply stochastic gradient descent to derive the sequence of steps.

Statement 11: The system of any of Statements 8 through 10, wherein the assembly instruction module is further configured to determine the sequence of steps using a machine learning model, the machine learning model minimizing deviations configured as

Statement 12: The system of any of Statements 8 through 11, wherein the deviation is between the above sequence of steps for completing assembly of the target object and another sequence of steps for completing assembly of the nominal object. determined based on the similarity between

Statement 13: The system of any of statements 8 through 12, wherein the deviation is minimized using the Markov Decision Process (MDP) via reward formulation.

Statement 14: The system of any of statements 8 through 13, wherein the assembly instruction module derives from the motion data a set of images representative of the assembly of the target object, evaluates the set of images, It is further configured to identify execution of the set of steps by the operator to assemble the target object.

Statement 15: A non-transitory computer-readable medium containing instructions stored thereon which, when executed by one or more processors, cause the target object to Detecting errors in assembly, Evaluating target and nominal targets in the above steps of the assembly process, thereby obtaining a comparison, Based on the comparison, minimizing the deviation between the target and nominal targets. The processor is configured to execute instructions including determining a sequence of steps required to achieve the desired sequence of steps, and adjusting building instructions for the target object based on the sequence of steps.

Statement 16: The non-transitory computer-readable medium of statement 15, wherein the instructions are further configured to cause the processor to derive the above sequence of steps using stochastic gradient descent.

Statement 17: The non-transitory computer-readable medium of any of statements 15-16, wherein the target object is evaluated against the nominal object in the above step of the assembly process.

Statement 18: The non-transitory computer-readable medium of any of statements 15-17, wherein the target object is evaluated against the final configuration of the nominal object.

Statement 19: The non-transitory computer-readable medium of any of statements 15-18, wherein the instructions determine the sequence of steps using a machine learning model configured to minimize deviation by the processor is further configured to be

Statement 20: The non-transitory computer-readable medium of any of Statements 15-19, wherein the deviation is minimized using the Markov Decision Process (MDP) via reward formulation.

Claims (15)

A method for optimizing workflow on an assembly line, comprising:
detecting, by a computer system, in a step in a process of assembling a target object, an error in assembling the target object corresponding to an error due to the motion of an assembler assembling the target object during said step;
Analyzing, by a computer system, the target object against the nominal object in the step of the assembly process in response to detecting the error to obtain a comparison;
determining, by a computer system, whether a deviation between the target target and the nominal target should be minimized based on the comparison;
based on the determination, generating, by a computer system, the sequence of steps necessary to minimize the deviation between the target target and the nominal target;
adjusting, by a computer system, building instructions for said target object based on said sequence of steps, thereby obtaining adjusted building instructions;
method including. 2. The method of claim 1, wherein a step in the assembling process evaluates the target object against the nominal object. 3. The method of any one of claims 1 and 2, wherein the target object is evaluated against the final configuration of the nominal object. 4. The method of any one of claims 1-3, wherein the sequence of steps is determined using a machine learning model configured to minimize the deviation. 5. The method of claim 4, wherein the machine learning model is a gated recurrent unit (GRU) model. 5. The method of claim 4, wherein the machine learning model is based on a Long Short-Term Memory (LSTM) model. 5. The method of claim 4, wherein the machine learning model is trained using tracking information representing the assembly process for the target object, the tracking information collected from a set of workers. 8. The method of any one of claims 1 to 7, further comprising capturing motion data corresponding to the assembling process by a computer system, the motion data comprising digital recordings of hand movements while assembling the target object. described method. extracting from the motion data, by a computer system, a set of images representing the assembly of the target object;
evaluating said set of images with a computer system to thereby identify performance of said sequence of steps by said operator to assemble said target object;
9. The method of claim 8, further comprising: 10. The method of any one of claims 1-9, wherein the errors in assembly of the target object are detected using one or more machine learning models. 11. The method of any one of claims 1-10, wherein the adjusted building instructions are configured to provide an operator with instructions for minimizing the deviation of the target object. The deviation is determined based on the similarity between the sequence of steps for completing the assembly process of the target object and another sequence of steps for completing the assembly process of the nominal object. A method according to any one of claims 1 to 11. 13. The method of any one of claims 1-12, wherein the deviation is minimized using the Markov Decision Process (MDP) via reward formulation. 14. Any of claims 1-13, wherein the adjusted building instructions are transformed using one or more natural language processing (NLP) algorithms to provide the adjusted building instructions to an operator. 1. The method according to item 1. 15. The method of any one of claims 1 to 14, wherein the sequence of steps for completing the assembly process of the target object is derived using stochastic gradient descent.

Images