JP2022522159A - Assembly error correction for assembly lines - Google Patents

Abstract

An aspect of the technique disclosed is a machine for detecting errors during the manual assembly process and for determining a series of steps to complete the manual assembly process in order to mitigate the detected errors. Provides a computational model that utilizes learning. In some embodiments, the disclosed technique evaluates a target object at a step in the assembly process where an error is detected against a nominal object, thereby obtaining a comparison. Based on this comparison, a series of steps is obtained to complete the target assembly process. Based on this series of steps, the assembly instructions for creating the target object are adjusted.

Description

Mutual reference to related applications This application is a partial continuation of US Patent Application No. 16 / 587,366, entitled "DYNAMIC TRAINING FOR ASSEMBLY LINES," filed on September 30, 2019. Application No. 16 / 587,366 is a US application No. 16 / 289,422, entitled "DYNAMIC TRAINING FOR ASSEMBLY LINES," filed on February 28, 2019, and is now US Pat. No. 10,481, It is a continuation application of No. 579. In addition, this application is filed on April 19, 2019, US Provisional Application No. 62 / 863,192, entitled "A COMPUTATION MODEL FOR DECISION-MAKING AND ASSEMBLY OPTIMIZETION IN MANUFACTURING,", November 6, 2019. US provisional application No. 62 / 931,448, filed on the same day, "A COMPUTATION-MODEL FOR DECISION-MAKING AND ASSEMBLY OPTIMIZETION IN MANUFACTURING," and "DEEP PRODUCT" filed on November 7, 2019. It claims the benefit of US Provisional Application No. 62 / 923,063, entitled "FOR MANUAL ASSEMBLY VIDEO ANALYSIS,". The entire contents of the above applications and patents are incorporated herein by reference.

Thematic technology provides improvements to the assembly line workflow and, in particular, a system for adaptively updating assembly line worker instructions based on feedback and feedforward error propagation predictions made using machine learning models. And methods are included. As discussed in more detail below, some aspects of the technique automatically produce guidance videos provided at one or more worker stations based on speculations made regarding manufacturing deviations or assembly deviations. It includes systems and methods for adaptation.

In traditional assembly line workflows, human (worker) monitoring and professionalism is used to detect manufacturing errors and to determine how the errors can be eliminated through corrections in downstream processes. Technology is needed. It should be noted that assembly and manufacturing, as well as assembly and production lines, are used interchangeably herein. Since the detection of assembly errors depends on humans, there is a high possibility that the errors will be overlooked (or not reported) and will be propagated downstream in the assembly process as is. In addition, many assemblers are only trained to perform a narrow set of tasks, and therefore the workers themselves to best correct errors that occur upstream in the assembly workflow. You may not be aware of how to modify your workflow.

In traditional manufacturing workflows, repairing human errors as part of the manual inspection process is often handled by taking corrective action on that human node. If there are ongoing problems with that person, that person is often replaced by others who are vulnerable to many of the same limits, as is the case with all humans. It is difficult to constantly repeat an action for years without making an error, and taking corrective action is outside the rights of most assemblers. Even if these rights are given, there is a contradiction, and in the application of that single process node, it will only be told by the experience that the person has. Moreover, there is no mechanism for learning from any mistakes, or even from any positive corrections.

In addition, the electronic monitoring of the assembly line is limited, and a rugged mechanism is provided to provide flexible adjustments to the downstream steps in the assembly line to compensate for errors that occur in the upstream steps. Does not include. In addition, modifications to assess how changes in worker motion and / or changes in assembly patterns affect the resulting manufactured product and to improve the performance and / or characteristics of the manufactured product. Need a new mechanism to provide.

In some embodiments, the disclosed technique relates to a method for optimizing a workflow in an assembly line, wherein the method is to detect an error in the assembly of the target object in one step of the assembly of the target object, and to assemble. A step that evaluates the target and the nominal object at that step of the process and thereby obtains a comparison, and a series of steps necessary to minimize the deviation between the target and the nominal object based on the comparison. Includes steps to determine. In some embodiments, the method may further include a step for adjusting assembly instructions for the target object based on the series of steps described above.

In another aspect, the disclosed technology comprises a system for optimizing a workflow in an assembly line, where the system is a plurality of image capture devices, each of which is a target assembly process. During, with multiple image capture devices placed in different positions to capture the movement of the worker, and an assembly instruction module configured to automatically modify the guidance and instructions provided to the worker. It includes an assembly instruction module that is coupled to the plurality of imaging devices. The assembly instruction module is a step of receiving motion data from a plurality of image capture devices, and the motion data corresponds to an operator's execution of a set of steps for assembling a target object, and the above-mentioned set. Based on the motion data in one of the steps, the operation can be configured to include a step of determining an error in the assembly of the target object. In some embodiments, the assembly instruction module evaluates the target object and the nominal object in the above set of steps, thereby performing an operation to obtain a comparison, and based on the comparison. Further configured to determine the sequence of steps required to minimize the deviation between the target and the nominal object, and to adjust the assembly instructions provided to the operator based on the sequence of steps described above. can do. The forms of the modified assembly instructions are not limited to them, but are the generated or edited video of the motion data, the text-based instructions from the natural language processing (NLP) of the identified deviations, or to the operator. Other feedback mechanisms can be included.

In yet another aspect, the disclosed technology relates to a non-temporary computer-readable medium containing instructions stored on it, which, when executed by one or more processors, is a target assembly process. In one step, the step of detecting an error in the assembly of the target object, the step of evaluating the target object and the nominal object in the above steps of the assembly process and thereby obtaining a comparison, and the target object and the nominal object based on the comparison. The processor is configured to execute an instruction that involves performing a step that determines the sequence of steps required to minimize the deviation between objects. In some embodiments, the instructions can be further configured to allow the processor to perform operations to coordinate assembly instructions for the target object based on the series of steps described above.

Specific features of the subject technology are set forth in the appended claims. However, the accompanying drawings included to provide further understanding illustrate the aspects to be disclosed and, in combination with the explanation, serve to explain the principles of the subject art.

It is a figure which conceptually shows the flowchart of the exemplary production line development by some aspect of the disclosed technique.

FIG. 6 illustrates an example of a process for performing assembly error correction at a given worker station, according to some aspects of the disclosed technique.

It is a figure which shows the example of the electronic system which can be used to realize some aspects of the subject art.

The detailed description given below is intended to be an explanation of the various configurations of the subject technology and is not intended to represent only the configurations in which the subject technology can be practiced. The accompanying drawings are incorporated herein and form part of a detailed description. The detailed description includes specific details to provide a more complete understanding of the subject technology. However, it will be clear and clear that the subject matter is not limited to the particular details presented herein and can be practiced without these details. In some examples, the structures and components are shown in the form of block diagrams to avoid obscuring the concept of the subject technology.

Aspects of the disclosed techniques are described above of conventional assembly line process flows by providing methods for tracking, training, and progressively improving production line assembly and the resulting manufactured products. Address the limits. Improvements are achieved by providing dynamic visual feedback or other feedback, and instructions to the individual assembler, and in some embodiments, the worker feedback is error-based and the error is. It can include, but is not limited to, assembly errors, useless processes and / or motions, and inferior products detected at one or more points on the production line.

Manual embodiments by implementing the disclosed techniques, for example by promptly revising and changing the reference / instruction information provided by individual stations (or all stations) based on quasi-real-time error detection. The speed can be significantly improved relative to the speed of error correction in the method. Although some embodiments described herein consider the use of reference / instruction information in the form of video, other formats are also contemplated. For example, assembly / manufacturing instructions can be provided to the assembly operator as auditory, visual, text and / or tactile cues, or as other forms of reference. As an example, auditory instruction information can include verbal instructions or other auditory indicators. The visual assembly instruction information can include a video or animation format such as the use of an augmented reality (A / R) or virtual reality (V / R) system. In some embodiments, the visual assembly instructions can be provided as a video providing an example of how the operator handles the workpiece (or tool) at a given station on the assembly line. Further, in some embodiments, the assembly / manufacturing instructions can include, for example, machine instructions that can be received and implemented by the robot assembly operator or the machine assembly operator. As used herein, the term worker can mean a human, robot or machine that uses the motion of assembling a manufactured product. Further, the term worker includes human-assisted manufacturing embodiments, as in the case of an example in which a human worker works with a robot or machine appliance, or a human worker is assisted by a robot or machine tool. There is.

In an example where assembly / manufacturing instructions are provided as reference / instruction videos, such videos are sometimes referred to as standard working protocols (SOPs). The system of disclosed technology can be effectively deployed for minimal hardware requirements, such as the use of video cameras and displays for individual workers, while machine learning training, updates. And error propagation can be performed on centralized computational resources, such as in computational clusters or cloud environments.

In some embodiments, the video instruction information can be provided to one or more workers as part of an augmented reality display. That is, deviations from instructions or standard assembly / manufacturing methods can be communicated to the operator using augmented reality, and the display is enhanced video, animated graphics, and / or recorded scenarios. It is provided as a mixture of video data representing. As an example, augmented reality displays provide instructions or guidance provided as animations or graphic overlays on the real-time feed of the workpiece being assembled and / or the tools used in the assembly / manufacturing process. Can be done.

In some embodiments, the disclosed technology system comprises one or more video capture devices or motion capture devices located at various worker stations on the production line. The capture device is configured to record the operator's motion / part, device, material or interaction with other tools (“components”) at that particular station. In some embodiments, worker motion can be captured using video recording, but a 3-D point that represents the worker's interaction with other motion capture formats, such as worker motion and / or tools or manufactured products. The use of the cloud is also planned. In addition, it is possible to create a station-by-station reference video by recording the motion of one or more experts to a particular station and the interaction of that expert with the components at that station. This video can be produced from a single example of the expert's actions, or from multiple examples. A motion path for each expert can be derived, and in embodiments where some experts or some examples are used, calculations are performed on the set of extracted motion paths (eg, average) to perform calculations. You can create a reference video for a particular station. The reference video may be in the form of a digital or animated representation of the motion path to be performed at that particular station. It should be noted that an expert can mean someone with special skills or knowledge about a particular assembly step for which guidance is provided.

In some embodiments, video or motion capture devices located at various worker stations on the production line can be used to calculate assembly errors, workpieces / components / at each station. Tool attributes (eg quality, tensile strength, number of defects) can also be captured.

Worker error by comparing the captured interaction with a baseline (ground truss) model that represents ideal / skilled worker interaction / workflow by capturing that interaction at each worker's station. Can be detected. That is, assembly errors that can be repaired at various locations in the assembly chain, for example by revising the operator instructions / guidance provided to different stations, using worker deviations from the idealized interaction model. Can be calculated. In addition, it is possible to capture the quality of components at an individual station and compare them to the baseline components for that station. Workers who can also use component deviations from baseline components to assign quality grades to components at a particular station or revise worker instructions / guidance provided to various stations. / It is also possible to calculate assembly errors.

The assembly modification can be carried out in various ways depending on the desired embodiment. In some embodiments, worker change / error is used, for example, to classify parts into quality grades (eg, A, B, C, etc.) and subsequently by directing these parts to the appropriate production line. Can be carried out. In another aspect, the detected assembly error can be used to revise the process at a given station, thereby improving quality and reducing changes. That is, the detected assembly errors can be used to automatically provide instructions or guidance to the same station, thereby correcting, for example, errors caused at that station (eg, in-station rework). .. NLP can be used to process instructions or guidance to workers. Oral instructions can be converted to text form, or text instructions can be converted to verbal form, for example using NLP.

For example, assembly error detection can be used to drive updates / changes to worker instructions or videos provided to a given station where errors are known to occur. As an example, if it is identified that the error / deviation is due to a first worker working at the first station, then to that first worker, for example, via the display device of the first station. The assembly instructions provided can be revised to reduce the error distribution associated with the manufactured article leaving the first station.

In another aspect, the detected assembly error can be used to revise the subsequent station assembly, thereby overcoming station distribution. That is, error detection can be used to automatically trigger downstream propagation of new / updated assembly guidance based on errors caused by upstream workers. For example, error distribution for motion performed by the first operator can be used to coordinate the assembly instructions provided to the second operator associated with the second station downstream from the first station. can.

In yet another embodiment, the error variance detected across all stations can be propagated forward, thereby performing a full or partial rework over the progress of the entire remaining downstream assembly chain. Can be guaranteed. That is, by coordinating the assembly instructions provided to one (one) or multiple downstream workers, errors generated across one or more stations can be repaired / reduced. In one example, it is provided by a second worker at the second station and through work subsequently performed by a third worker at the third station, i.e. to the second and third stations. By adjusting the assembly instructions, it is possible to repair the error dispersion of the manufactured article due to the first operator of the first station.

In another example, the error distribution accumulated across multiple stations can be reduced by one or more subsequent stations. Accumulated across the first and second stations, for example by coordinating the assembly instructions provided to the third and fourth stations (eg, third and fourth workers, respectively). The error dispersion of the manufactured article can be subsequently repaired.

The assembly process by treating individual workers / worker stations in the assembly flow as network nodes and using machine learning models to minimize errors through reduction of assembly distribution at individual nodes (stations). Can be optimized. By minimizing individual node (worker) distribution and also performing real-time updates to reduce forward error propagation, the disclosed technology system can significantly reduce manufacturing distribution to the final product. can. In addition, products can be graded and categorized by product quality or deviation by accurately quantifying and tracking error contributions from specific segments in the assembly workflow. Thus, for example, depending on the product quality, products of a particular quality category can be guided to different manufacturing processes or to different customers.

Error detection can be performed using machine learning / artificial intelligence (AI) models, and / or the necessary modifications can be made to optimize station assembly changes. As an example, machine learning models are, but are not limited to, final product ratings, final product change statistics, desired final product characteristics (eg assembly time, amount of materials used, physical properties, number of defects, etc.). You can train using multiple sources of training data, including station-specific component ratings, station-specific component variations, and desired station component characteristics. In addition, the deployed machine learning model is provided by experts or "master designers" so that formal knowledge can be expressed in the idealized model used to perform error detection and error quantification calculations. It can be trained or initialized based on the input given.

As will be appreciated by those skilled in the art, machine learning-based classification techniques can be modified according to desired embodiments without deviation from the disclosed techniques. For example, machine learning classification schemes include hidden Markov models, recurrent neural networks, convolutional neural networks (CNN), reinforcement learning, deep learning, Bayesian symbol models, hostile generation networks (GAN), support vector machines, image registration methods, and applications. Possible rule-based systems and / or one or more of any other suitable artificial intelligence algorithms can be utilized alone or in combination. When regression algorithms are used, they can include, but are not limited to, Stochastic Gradient Descent Regressor and / or Passive Aggressive Regressor, and the like.

Machine learning classification models include clustering algorithms (eg Mini-Batch K-mean clustering algorithm), recommended algorithms (eg Miniwise Hashing algorithm or Euclidean Locality-Sensitive Hashing (LSH) algorithm), and / or local outlier factor methods. It is also possible to use the anomaly detection algorithm of. In addition, the machine learning model may be one or more of a Mini-batch Dictionary Learning algorithm, an Incremental Principal Component Analysis (PCA) algorithm, a Latent Dirichlet Allocation algorithm and / or a Mini-batch K-average algorithm, and the like. It is also possible to use the method.

In some embodiments, different types of machine learning training / artificial intelligence models can be deployed. As an example, a common form of machine learning can be used to dynamically adjust the assembly line process to optimize the manufactured product. As will be appreciated by those skilled in the art, the selected mechanized learning / artificial intelligence model does not simply contain a set of assembly / manufacturing instructions, but provides feedback and results on the entire assembly line process. It is a method for providing its impact on manufactured products and also providing dynamic coordination for downstream worker stations in the assembly line, thereby compensating for actions that occur at upstream worker stations. This type of artificial intelligence-based feedback and feedforward model is referred to herein as the Artificial Intelligence Process Control (AIPC). In some embodiments, machine learning utilizes targeted recurrent unit (GRU) model-based learning and Hausdorff distance minimization to effectively explore and thus assemble the space of possible recovery paths. It can be based on a deep learning model in a simulated environment that finds the optimal path to correct errors during the process. In yet another embodiment, for example, a machine learning algorithm can be based on a Long Short-Term Memory model, which can analyze the video input of the assembly process and predict the final quality output. In addition, the machine learning model is used in the form of an NLP algorithm, such as converting text to voice or converting voice to text to maximize worker compliance and comprehension of coordinated instructions. You can also adjust the feedback to the station.

In some embodiments, computational models that utilize machine learning can be used to correct errors during the manual assembly process.

The target object can be assembled through a series of steps defined by the processing procedure. During the process, irreversible errors can occur at certain steps k where all remaining work would need to be revised to obtain the final configuration of the nominal object. In some embodiments, the method for correcting an error is to compare the defective target object in step k with respect to the nominal object in the same step k, or the defective target object in step k. It can include a step of comparison against the nominal object of the final configuration. These comparisons can be used to determine the set of steps required to minimize the deviation between the final configuration of the defective target object and the final configuration of the nominal target. In some embodiments, the quality metric of the target can also be used to guide the correction of the defective target.

A common numerical method can solve the problem by using the Hausdorff distance algorithm, which allows a series of k steps to assemble a defective target object into its final configuration. You can determine how similar it is to the sequence of steps to assemble. Several methods for computationally minimizing the Hausdorff distance between the above series of k steps in a defective assembled object and the series of steps in the final assembled nominal object are instantaneous. Optimize the Markov Distance Process (MDP) through reward formulating, multiple reward formulating, or delayed reward formulating. However, the search space associated with these formulas requires a significant amount of computational resources.

As an alternative, machine learning frameworks can also be evolved with delayed reward policy agents using reinforcement learning. The reinforcement learning framework is designed to let the policy agent determine the appropriate steps needed to correct an error in a defective target, resulting in a final configuration with performance metrics that match the performance metrics of the nominal target. can do. The rewards given to the policy agent are delayed and the policy agent is paid only when the final step is performed.

In some embodiments, the design for the optimum / desired manufactured product can be selected, and the skilled worker can be deployed to assemble the manufactured product according to the selected design. Individual steps performed at the station can be performed. Optimal can be based on the desired performance and / or characteristics of the resulting product, minimizing errors in the resulting manufactured product, or any other criterion (eg, the manufactured product is a paper plane). If so, the best paper plane would be one that achieves the desired flight target). Using multiple imaging devices to capture worker motions and worker interactions with the manufactured product that the worker is assembling, thereby generating video, images and / or 3D point cloud data. Can be done. The captured data is the worker's hand coordinates in relation to the manufactured product as assembled, the relationship between one hand and the other, and the fingers (and some) to the manufactured product as assembled. In the embodiment, detailed information such as the relationship between the knuckles) can be provided. The data collected from skilled workers can be used as a ground truth for assembling optimal / desired manufactured products. This ground truth from a single example, it is sufficient for use in the creation of early machine learning models, or it is possible to collect additional data. One to deploy many workers to assemble the best manufactured product, for example, to understand how changes or errors in worker motion can affect the resulting manufactured product. Alternatively, multiple steps can be performed. This can be done for each worker station on the assembly line. Both the resulting final product and their respective assembly processes are compared to each other and to the ground truth, whereby errors and / or changes in worker motion are characteristic of the manufactured product and / or. Or it can be determined how it can affect performance (eg worker speed can result in a poorer quality airplane). Data collected on a worker basis during the actual assembly process (ie, the process in which a human, robot or machine is performing motion at one or more stations) is referred to herein as "actual. It will be called "training data". The actual training data can be supplemented with simulated data, which can provide a richer dataset and also provide additional variants to achieve optimal manufacturing products. can. It should be noted that the terms "optimal" and "desired" will be used interchangeably herein.

In some embodiments, the different AI / machine learning / deep learning models discussed herein are described below in order to achieve the Artificial Intelligence Process Control (AIPC) and optimize the assembly of the manufactured article. It can be deployed in the specific order described. Illustrative processes that can realize AIPC deep learning models are discussed in more detail in connection with FIG. 1 (eg, in connection with AIPC deep learning model 112) and in connection with FIG. Examples of hardware systems and / or devices that can be used to implement AIPC deep learning models are provided in FIG. 3, and corresponding descriptions are also provided below.

First, CNNs can be used in the assembly line process to classify the characteristics of the worker's hands and the characteristics of manufactured articles of different configurations at individual worker stations.

Second, both can use and reward reinforcement learning (RL) and RL agents to achieve the desired results from the CNN classification and for the defined desired results. .. The RL agent may or may not be monitored.

Third, Generative Adversarial Networks (GAN) can be used to choose between opposing RL agents. The GAN can include minimal human monitoring that relies solely on humans to select the RL agent to enter into the GAN as a node.

Fourth, RNNs can take winning RLs (winning RLs) as input nodes to create feedback and feedforward systems, thus making learning continuous and also monitoring learning. It does not have to be.

Embodiments of these four AI / machine learning models are discussed in more detail below.

In some embodiments, the actual training data categorizes the relevant data in the assembly process, eg, fingers / hands used in individual assembly steps per worker station, of the assembled products. It can be entered into the CNN to classify the parts touched by the operator's finger at any time point and space, and the shape or composition of the assembled product at any time point and space.

In yet another embodiment, hand motion is not tracked, but it is possible to collect data representing various changes in the assembly pattern of the manufactured product (eg, if the manufactured product is a folded paper plane, folding). Data can be collected based on changing the order, achieving folding changes, and / or introducing potential errors, for example if the manufactured product is a clothing article, suture order, suture. Data can be collected based on achieving change and / or introducing potential errors). This data can be simulated and / or collected from actual training data. The resulting manufactured products and their respective assembly processes can be compared to determine how errors or changes in assembly patterns affect the characteristics and / or performance of the manufactured products.

In some embodiments, the captured data (eg, video and hand tracking of the assembly process, etc.) is used to predict the quality of the final output. This quality prediction allows the captured data to be used to classify products into quality bins without the need to manually inspect the quality of the product during the manufacturing process and to take corrective action downstream. be able to.

In some embodiments, the system can focus on manual assembly of the target, in which the assembly process is such that the worker performs different tasks on the target for each set of instructions. Includes several discrete steps. The system uses a machine learning framework (overall worker actions) that uses a deep learning model that establishes a correlation between the time series of worker hand positions and the final quality of the target. Can be built. In some embodiments, the model may consist of two neural networks, the first neural network is used to extract the position data of the worker's hand in a 3D environment, and the second. Neural networks are used to remove unwanted elements from hand position data and correlate them with the final quality of the target's performance.

In some embodiments, the first neural network uses a video acquisition system during the assembly process in different node videos corresponding to the individual discrete steps performed by the operator in connection with the assembly of the target object. , Can record video of worker's hand. For example, the operator can carry out the assembly process using several cameras that are located at different locations and are configured to record the assembly process at the same time. These cameras can be used to capture video multiple times in a pre-defined position on the operator's hand. These videos can then be processed to extract some images or landmark frames that represent the entire assembly process of the target. These landmark frames can be used to extract hand tracking information that facilitates the demarcation of worker hand and finger positions or keypoints during the assembly process.

In some embodiments, a bounding box prediction algorithm and a hand keypoint detector algorithm can be applied to extract hand tracking information. In particular, the bounding box prediction algorithm can include using threshold image segmentation to process landmark frames from the assembly process, thereby obtaining a mask image for the worker's hand. The hand can be positioned on the mask using blob detection. The bounding box prediction uses a mask image to form a box around each of the worker's hands so that the box contains the highest point in the shape of the hand position at least up to the hand wrist point. The bounding box and their corresponding landmark frames are then fed into the hand keypoint detector algorithm.

The hand keypoint detector algorithm can include a machine learning model that can detect specific keypoints in the worker's hand. The hand keypoint detector algorithm can predict not only the keypoints that can be seen in the landmark frame, but also the keypoints that are blocked from the frame due to the interaction of joints, viewpoints, objects, and hands. A particular block point in one frame cannot be blocked in another frame because different hand positions generate different block points in different frames. The hand keypoint detector predicts the location of a keypoint that will be blocked with a certain level of certainty. However, predictions of blocked keypoint locations can result in the same keypoint location being recorded for different hand positions. During the steps of the manual assembly process, the hand key points defining the worker's hand are then provided to the second neural network.

In some embodiments, a second neural network is used to predict the quality of the final state of the assembled object. In some embodiments, the neural network can be based on a Long Short-Term Memory (LSTM) model. The LSTMs are arranged in order and together have several cells that represent the entire assembly process of the final object. The input to the LSTM cell may be hand keypoint data corresponding to the worker's work at a particular step of the assembly process represented by the LSTM cell. Each cell in the LSTM determines whether information from the preceding cell should be stored, chooses the value to update, performs the update to the cell, chooses the value to output, and then chooses. Filter the values so that the cell outputs only the values. The LSTM may be a sequence for one model trained using an Adam optimizer or other adaptive learning rate optimization algorithm. The neural network uses the LSTM framework to correlate the input data extracted from the manual assembly process, thereby determining the quality measurement of the final product.

In some embodiments, the video and hand tracking information representing the assembly process for the target object, i.e., the input data used to train the model, is a plurality of assembly instructions using a single set of assembly instructions. It can be collected from multiple workers performing the assembly process to assemble the target object. The target object assembled by the operator can be used in a controlled environment to collect the corresponding quality measurements for the performance of the assembled object, i.e. the output data needed to train the model.

In some embodiments, the training data used to generate the machine learning model is from simulated data, from actual training data, and / or from expert ground truth records. They may be in combination or individually. In some embodiments, simulated data results can be used to build machine learning models, such as (but not limited to) Reinforcement Learning (RL) agents. In other embodiments, actual training data can be used to build machine learning models, such as (but not limited to) Reinforcement Learning (RL) agents. RL agents are rewarded for achieving good / desired results and are punished for bad results.

In some examples, many RL agents, some based on actual training data and some based on simulated data, can be deployed to work in columns, It can also be configured to maximize cumulative awards, eg assembling manufactured products with the smallest deviation from an ideal model / example. Illustrative results that can be rewarded to an RL agent include completing a complete manufactured product in as few steps as possible, reducing the amount or time of material required to achieve the manufactured product. RL agents based on simulated data and RL agents based on actual training data can be used to determine optimal motion patterns and / or optimal assembly patterns that result in the optimal / desired manufactured article.

These two groups of RL agents (eg, RL agents created based on actual training data and RL agents created based on simulated data) are both optimal / desired manufactured products. You can work together and even compete here because you are paid for the act of making. In some embodiments, the data obtained from a simulated-based RL agent that yields the optimal assembly pattern for the optimal manufactured product is used to reduce the potential space for the actual training dataset. be able to. For example, a simulated RL agent can be used to determine the optimal assembly pattern and then collect the actual training data only for the optimal assembly pattern, not for the non-optimal assembly pattern. be able to. By focusing only on the collection of actual training data or only on the optimal assembly pattern, less training data can be collected and / or larger for collecting more actual training data. Capacity can be achieved, but only for optimal assembly patterns.

Rewards can sometimes conflict, so there is a limit to optimizing an assembly line by relying solely on reinforcement learning. For example, in product assembly, some RL agents have the least false movements (eg, fold and immediately undo the fold, or add a seam and immediately remove the seam). Can be rewarded, while other RL agents can be rewarded for speed. An RL agent that is paid for speed can determine that more false movements result in faster assembly time because less modifications are required downstream in the assembly process. Making such an embodiment trade-off decision is not something that humans can easily determine. Even with experience and a large number of examples, humans have the computational power to understand how subtle it is to grasp the end result by different workers working differently. Still missing.

To resolve these conflicting RL agent optimizations, GANs can be deployed and act as determiners. Conflicts are between RL agents based on actual training data, between RL agents based on simulated data, and / or between RL agents based on actual training data and RL agents based on simulated data. May be.

In some embodiments, the GAN can test each of the RL agents and store the results in order to create even more robust neural networks. GAN works by taking an RL agent and also using a model that results in winners and losers in zero-sum games. There are "generators" and "discriminators" in GAN. The generator will in this case store reward data from conflicting RL agents, and the discriminator will assess which of these is most relevant to the task of producing the desired manufactured product. Will be done. GAN uses a deep network of nodes (or neurons) to determine how to weight the nodes. Since individual RL agents believe that individual RL agents have already made optimal decisions, it is not possible to determine which of the conflicting RL agents actually makes the most relevant choice. It is the role of the GAN, and the discriminator adjusts the weights accordingly. When a zero-sum game is started between opposing RL agents, a group of winners is created between the opposing RL agents, and only these winners are used to optimize the workflow in the assembly line. Will be used for machine learning models. A large amount of data may be generated to determine the winning RL agent, but the result is more than the one used to create and find these winners used as input nodes. Is also much poorer.

Once the RL agents that have survived the GAN war and have been properly paid are determined, in some embodiments they can be entered into another AI system called a Recurrent Neural Network (RNN). RNNs have many similarities to CNNs in that they are Deep Learning Neural Networks and the final result is optimized through weighting of various forms of input data. One difference is that unlike a CNN, which is a linear process from input to output, an RNN is a loop that feeds back the resulting output, as well as internal nodes, as new training information. RNNs are both feedback systems such as GRUs and feedforward systems.

In some embodiments, the machine learning framework can be built utilizing targeted GRU model-based learning. The GRU model can be selected as an alternative to reinforcement learning because of its predictive ability and relatively short training time. GRUs are used by RNNs to distinguish between observations that should be stored or updated in memory and observations that should be forgotten or reset.

In some embodiments, the GRU model can consist of several GRU cells corresponding to the number of assembly steps required to build the target object. Each GRU cell representing one of the number of assembly steps can have several input parameters and hidden state outputs. The GRU cell, which represents the final step in the assembly process, will output the target object. The output of the model is the deviation of the target object from the nominal object. This deviation can be calculated using the step Hausdorff distance from the target to the nominal target and the performance metrics of the final configuration of the nominal target. Individual GRU cells are defined by resets, updates and new gates. The GRU neural network is iteratively trained, thereby biasing the GRU neural network towards the solution of a particular subtitle and also identifying a set of weights for the GRU. For example, at each iteration, several predictions (one prediction for each possible error in a particular step) are generated to complete the assembly process in subsequent steps. In addition, the corresponding predicted distance measures for the modified assembly process can be generated. These predicted assembly process completions can be depicted in a virtual representation system, and in those stepped Hausdorff distances calculated to obtain a "ground torus" distance measure. The difference between the predicted distance measures can be calculated and fed back to the model as "ground truth", and its network weights are adjusted via backpropagation to generate the next iteration. This process can continue until the above set of weights for GRU is identified. In some embodiments, a stochastic gradient descent method may be used to correct the defective target and to elicit the steps necessary to obtain a satisfactory final configuration. can.

In some embodiments, simulations such as parameter computer-aided design and drafting (CAD) models of the target being processed can be generated to develop and validate machine learning models. The CAD system can use local coordinate frames corresponding to the current state of the target object being processed, as well as input parameters representing individual assembly steps. By using the local coordinate frame and input parameters of the target object being processed, the CAD system can determine the dimensional information for each assembly step. The CAD system can then generate a three-dimensional CAD model that represents the composition of the output at each step. The CAD system can continue this process until every step in the assembly process has been performed, and can output a CAD model of the final configuration of the assembled object. By providing various input parameters to the CAD system, it is possible to generate CAD models with different configurations. To obtain a set of CAD models with a specific range of input criteria such as length or width, provide a statistical sample of this input criteria to the CAD system to generate the set of CAD models described above. Can be done.

CAD models may be variable levels of detail and intellectual elements, but trained models and systems are specifically designed to work with lower detail CAD systems, thereby making it a non-computational and expensive method. A huge number of examples can be generated in, and sufficient surface morphology details can be provided for trained models and profiling. In some embodiments, the referenced CAD system can be paired with a Finite Element Analysis (FEA) or basic surface modeling tool, whereby a structural analysis of the surface can be generated. This data can be used as an additional quality score for model training and analysis.

In some embodiments, the CAD system can be incorporated into model training so that additional surface models can be generated as required by the example from the model or as needed for additional inquiry data. This technique is paired with physical observation and allows the development of pre-trained models without the need for a huge amount of physical samples in space.

In some embodiments, a CAD model of the final configuration of the assembled object can be used in the simulation to generate performance metrics. By using the CAD model of the final configuration of the assembled object, the simulation can generate performance metrics using numerical and computational methods.

The Artificial Intelligence Process Control (AIPC) real-world application provides feedback (eg, by automatically modifying video instructions) to workers who have already completed their tasks on the assembly line, as well as downstream on the assembly line. Includes providing instructions (also by automatically modifying video instructions, for example) to workers who have not yet completed their tasks on the side (“feed forward”). This feedback-feedforward system, ie AIPC, can be achieved using the AI techniques described herein using, in some embodiments, in the particular order described herein. Therefore, the operator on the assembly line can make the choice to optimize the resulting manufactured product without the need for additional human monitoring.

In some embodiments, this simply involves compressing the system into an RNN, and observing any movement in two ways, success or failure, during the process of producing one or more manufactured products. Individual movements serve as training. If the output node of the RNN is not optimal, the network can feed back to the actual individual in the assembly line to make different choices, and in the path through many nodes and layers of the RNN, the weights are weighted. It can be redone, and the output will be labeled as either successful or unsuccessful. As the process iterates, the weights improve their accuracy on their own. In addition, the network can learn what is working and what is not working, even if the individual performing the assembly is not working. This increases the training set. It also allows the implementation of adjustments at different stages of the assembly process. In some cases, the best way to produce a manufactured article with specific properties at any given moment is to adjust the instructions as the process progresses, rather than returning to the starting point. Can be found. In this case, the RNN is constantly optimizing for the best manufactured product and also provides feedback to individual workers who have already performed their tasks at the worker station on the production line and also on the production line. Learn to feed forward information to workers who have not yet performed their tasks at the worker station.

FIG. 1 is a conceptual illustration of a flow chart of an exemplary process 100 for realizing a production line deployment according to some aspects of the disclosed technology. The process of FIG. 1 is started in step 102 and production deployment is started. An exemplary production line workflow typically involves multiple worker stations (nodes) from which workpieces (products) are assembled or manufactured. Various nodes can be sequentially organized so that the work of each succeeding node is started only after the work of the preceding node is completed.

In step 104, one or more reference videos are generated and / or updated. The video described above can be used to provide manufacturing / assembly instructions to a particular node (also referred to herein as a worker station). That is, each node in the workflow can be equipped with a reference video that provides guidance on how to complete the steps in the manufacturing workflow for that particular node.

In step 106, each of the videos generated in step 104 is expanded to its respective station / node. As an example, a given workflow can include 10 nodes, each with its own different / unique reference video. In other embodiments, the number of videos may be less than the total number of nodes. Depending on the embodiment, the reference video deployed to the various stations / nodes may be unique or may provide similar guidance / instructions. As discussed in more detail below, the content of the reference video may be dynamic and can be constantly updated / augmented.

At step 108, continuous recording of motion is captured at each station / node. Motion data obtained from motion recordings can describe worker interactions with workpieces / components / tools at those nodes / stations in the workflow. That is, the motion data captured on an individual node can represent one or more worker actions corresponding to a particular part of product assembly or manufacturing, and is provided by a reference video associated with that node. Can respond to instructions. In some examples, motion capture can include capturing video data, i.e. recording all or part of the worker's actions at the station. In other embodiments, motion capture can include, for example, recording a 3D point cloud, where motion is recorded for one or more specific points in the field of view of the image capture device. Both worker behavior and component attributes (eg component quality, tensile strength, number of defects) can be captured at individual nodes / stations in the workflow.

At step 110, process method analysis deviations can be calculated to analyze the motion data captured for one or more of the stations in step 108, eg, an idealized motion profile for the corresponding station. Any deviation from a comparative model that contains (or represents) can be identified. As illustrated in FIG. 1, step 110 is configured to identify / classify motion deviations from, for example, comparative models and to make inferences about the extent to which the assembly or manufacturing process could be affected. It is possible to utilize an AIPC deep learning model that can be used (step 112). Comparisons can be made at the station level and / or at the integrated process level. The analysis can also take into account the component's attributes at individual stations, or the component's deviation from the baseline, and the extent to which the station's motion deviation affects the component's quality.

The AIPC deep learning model referred to in step 112 is based on the collection of various types of training data, which can include, for example, examples of ideal or quality controlled assembly / manufacturing interactions for a given station / node. be able to. The AIPC deep learning model is also based on customer feedback on a particular product manufactured using process 100 (step 111) and on a particular product manufactured using process 100 (step 113). Feedback from quality control inspections can be augmented (or adjusted) with the data provided by Domain / Industry Information 115. It should be understood that AIPC deep learning models can be implemented using various computing systems including distributed hardware modules and / or software modules. As an example, an AIPC deep learning model is a plurality of images deployed on an assembly line and coupled to one or more systems configured to implement various AI / machine learning models and / or classifiers. It can be achieved using a distributed system that includes capture and display devices.

Once the deviation from the comparative model is detected / identified in step 110, the AIPC deep learning model 112 can be used to generate an automatic adjustment in step 114. As discussed above, video tuning can be aimed at improving manufacturing / assembly quality at one or more stations in the workflow. For example, where an error is known (or expected) to change, for example, to change the instructions or guidance provided to the operator in a way that reduces or repairs the error when it occurs. Video adjustments can be applied to a given node / station. In another embodiment, video adjustments can be applied downstream from the station where the error occurred, for example to correct the error before the manufacturing workflow is complete. In yet another embodiment, once the workflow is complete, the entire workflow can be analyzed and adjustments can be made to one or more stations in the workflow.

In some embodiments, the adjustment is performed in real time immediately after the error is detected. In other embodiments, adjustments are made at regular intervals or after the workflow is complete.

In some embodiments, the automatic adjustment determined in step 114 can be summarized in step 117 and / or provided as a product quality report. One or one that describes various aspects of the quality of the workpiece based on the deviations identified from the idealized model of the assembly / manufacturing process, for example using the adjustments provided by the analysis of motion deviations (step 110). You can create multiple quality reports.

FIG. 2 illustrates an exemplary process 200 for performing error detection analysis that can be used to facilitate assembly error correction according to some aspects of the technique.

Starting at step 210, an idealized video guide can be used to implement the process of improving manufacturing / assembly. At step 215, video tracking of one or more assembly stations is performed. Video tracking can include recordings of human workers at a given station / node. In some embodiments, video tracking can further include capture of component attributes at a given station / node.

In steps 220 to 224, a process of analyzing the video recorded from the assembly station is performed. For example, in some embodiments, background extraction can be performed to isolate movements / components in the recorded video. In some embodiments, once background extraction is complete, the processed video contains only the motion / video data relevant to the assembler (step 224), and the contained components are the corresponding assembly steps (step 224). Used in step 220). At step 220, additional processing can be performed to isolate the component / component. As illustrated by the diagram of process 200, step 220 performs additional processing operations including anomaly detection (step 221), surface change detection (222) and component classification and / or quality scoring (step 223). Can include. Any video processing step can be performed using a variety of signal and / or image processing techniques, including, but not limited to, the use of one or more AI / machine learning algorithms and / or classifiers. It should be appreciated that, for example, anomaly detection (221), surface change detection (222), and / or scoring / classification (step 223) can be performed.

After the processing steps 220-224 are completed, the process 200 can proceed to step 226 where the motion comparison is performed. Motion comparison (step 226) is a comparison of process-assembled station video data involving one (one) or multiple station workers at one or more stations / nodes with the corresponding idealized video / motion data. Can be included. Motion comparisons performed across multiple stations / nodes can be used to infer / predict the resulting changes in component / component quality.

At step 228, distribution / quality classification can be performed for various parts / components. As an example, parts / components can be classified into different quality layers and / or identified for removal or repair depending on their related classification / difference.

After the classification / difference is determined, process 200 can proceed to step 230, where the analysis of the entire process / workflow is the station / node classification / difference determined, for example, in steps 226 and 228. It is carried out based on. By analyzing the entire workflow, automatic adjustments to the video can be performed, thereby addressing the detected deviations / defects as discussed above.

FIG. 3 illustrates an exemplary processing device that can be used to implement a system of disclosed technology. The processing device 300 includes a master central processing unit (CPU) 362, an interface 368 and a bus 315 (eg, a PCI bus). When operated under the control of appropriate software or firmware, the CPU 362 is responsible for performing process adjustment steps for various error detection and monitoring and disclosed technologies. It is preferred that the CPU 362 accomplish all these functions under the control of the software, including the operating system and any suitable application software. The CPU 362 can include one or more processors 363, such as processors from the Motorola family of microprocessors or from the MIPS family of microprocessors. In an alternative embodiment, processor 363 is hardware specifically designed to control the operation of AIPC system 310. In certain embodiments, memory 361 (such as non-volatile RAM and / or ROM) also forms part of the CPU 462. However, there are many different ways in which memory can be combined into a system.

In some embodiments, the processing device 310 may include image processing system 370 or may be coupled with image processing system 370. The image processing system 370 can include various image capture devices such as video cameras that can monitor the movement of the operator and also generate motion data. As an example, the image processing system 370 can be configured to capture video data and / or output / generate a 3D point cloud.

Interface 368 is typically provided as an interface card (sometimes referred to as a "line card"). In general, they control the sending and receiving of data packets over the network and, in some cases, support other peripherals used with routers. Interfaces that may be provided are, among other things, Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, Token Ring interfaces, and the like. Further, it is also possible to provide various extremely high-speed interfaces such as a high-speed token ring interface, a wireless interface, an Ethernet interface, a Gigabit Ethernet interface, an ATM interface, an HSSI interface, a POS interface, an FDDI interface, and the like. In general, these interfaces can include suitable ports for communication with the appropriate medium. In some cases they can also include independent processors, and in some examples they can also include volatile RAM. The independent processor can control such communication aggregation tasks as packet switching, medium control and management. By providing a separate processor for the communication aggregation task, these interfaces can allow the master microprocessor 362 to effectively perform routing calculations, network diagnostics, security functions, and so on.

The system shown in FIG. 3 is one particular processing device of the invention, but is by no means the only network device architecture that can implement the invention. Architectures with a single processor dealing with, for example, communications and routing computations, etc. are often used. In addition, other types of interfaces and media can be used.

Regardless of the configuration of the network device, the network device is one or more memories or memory modules (including memory 361) configured to store program instructions for general purpose network operations, and described herein. Mechanisms for roaming, routing and routing functions are available. Program instructions can, for example, control the operation of the operating system and / or one or more applications. It is also possible to configure one or more memories to store tables such as mobility binding, registration and association tables, and so on.

The logical operations of the various embodiments are: (1) a series of computer realization steps, operations or procedures running on a programmable circuit in a general purpose computer, (2) a series of computer realization steps, operations running on a special purpose programmable circuit. Or implemented as a procedure and / or (3) interconnect mechanical module or program engine in a programmable circuit. System 300 may practice all or part of the described method, may be part of the described system, and / or in the non-temporary computer-readable storage medium described. Can operate according to the instructions of. Such logical operations can be implemented as a module configured to control the processor 363, thereby performing a particular function according to the programming of that module.

It should be understood that any particular sequence or hierarchy of steps in the disclosed process is an illustration of an exemplary approach. It should be understood that a particular order or hierarchy of steps in the process can be rearranged, or only some of the steps illustrated can be performed, based on design preferences. Some of the steps can be performed at the same time. For example, in certain situations, multiple tasking and parallel processing may be advantageous. Moreover, the separation of the various system components in the embodiments described above should not be understood as such separation is required for all embodiments, and the program components and systems described are not included. It should be understood that, in general, it can be integrated into a single software product or packaged into multiple software products.

The above description is provided to allow all skill in the art to practice the various aspects described herein. Various modifications to these embodiments will be readily apparent to those of skill in the art, and the general principles defined herein can be applied to other embodiments as well. Therefore, the claims are not intended to be limited to the embodiments set forth herein, and the claims are consistent with the language claims and are singular. References to elements of form do not mean "one and only", but "one or more" unless explicitly stated to be so. Is intended.

A phrase such as "aspect" does not imply that such an aspect is essential to the subject art or that such an aspect applies to all configurations of the subject art. The disclosures associated with the embodiments can be applied to all configurations, or to one or more configurations. A phrase such as one aspect can mean one or more aspects, and one or more aspects can mean one aspect. Words such as "construction" do not imply that such a configuration is essential to the subject matter or that such a configuration applies to all configurations of the subject matter. Disclosures related to configurations can be applied to all configurations, or to one or more configurations. A phrase such as one configuration can mean one or more configurations, and one or more configurations can mean one configuration.

The term "exemplary" is used herein to mean "act as an example or an example." All aspects or designs described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other aspects or designs.
Disclosure statement

Statement 1: A method for optimizing the workflow in the assembly line, where the method is to detect an error in the assembly of the target object at a step in the assembly process of the target object and the goal in that step of the assembly process. A step of evaluating an object and a nominal object, thereby obtaining a comparison, and a step based on the comparison, determining a series of steps necessary to minimize the deviation between the target object and the nominal object, and the above. Includes steps to adjust assembly instructions for the target object based on a series of steps.

Statement 2: In the method of Statement 1, the target object is evaluated against the nominal object in the above steps of the assembly process.

Statement 3: In any of the methods of Statements 1 and 2, the target object is evaluated against the final configuration of the nominal object.

Statement 4: Any of the methods of Statements 1 to 3, the series of steps is determined using a machine learning model configured to minimize deviations.

Statement 5: In any of the methods of Statements 1 to 4, the deviation is a series of steps described above for completing the assembly process of the target object and another series of steps for completing the assembly process of the nominal object. Determined based on the similarity between and.

Statement 6: In any of the methods of Statements 1-5, the deviation is minimized using Markov Decision Process (MDP) via reward formulation.

Statement 7: Any of the methods of Statements 1-6, the stochastic gradient descent method is used to derive the above series of steps to complete the assembly process of the target object.

Statement 8: A system for optimizing the workflow in the assembly line, where the system is a plurality of image capture devices, each of which captures the movement of the operator during the assembly process of the target. Multiple image capture devices located at different locations for capture and an assembly instruction module configured to automatically modify the guidance and instructions provided to the operator, the multiple image captures described above. The assembly instruction module includes an assembly instruction module coupled to the device, which is a step of receiving motion data from a plurality of image capture devices by the assembly instruction module, and the motion data is a target target by an operator. A step corresponding to the execution of a set of steps for assembling, a step of determining an error in assembling the target object based on motion data in a certain step of the above set of steps, and the above set of steps. A series of steps required to evaluate the target object and the nominal object in the above steps, thereby obtaining a comparison, and to minimize the deviation between the target object and the nominal object based on the comparison. It is configured to carry out an operation including a step of determining an image and a step of adjusting an assembly instruction provided to an operator based on the above series of steps.

Statement 9: In the system of Statement 8, the motion data includes a digital recording of the worker's hand movements while assembling the target object.

Statement 10: In any system of Statements 8-9, the assembly instruction module is further configured to apply stochastic gradient descent to derive the series of steps described above.

Statement 11: In any system of Statements 8-10, the assembly instruction module is further configured to use a machine learning model to determine the sequence of steps described above, the machine learning model minimizing deviations. It is configured as follows.

Statement 12: In any system of statements 8-11, the deviation is the sequence of steps described above for completing the assembly of the target object and another sequence of steps for completing the assembly of the nominal object. Determined based on similarities between.

Statement 13: In any system of statements 8-12, deviations are minimized using Markov Decision Process (MDP) via reward formulation.

Statement 14: In any of the systems of Statements 8 to 13, the assembly instruction module extracts a set of images representing the assembly of the target object from the motion data, and evaluates the set of images. It is further configured to identify the execution of the above set of steps by the operator to assemble the target object.

Statement 15: A non-temporary computer-readable medium containing instructions stored on it that, when executed by one or more processors, are the targets of the target at some step in the process of assembling the target. Detecting errors in assembly, evaluating the target and nominal objects in the above steps of the assembly process, and obtaining a comparison thereby, minimizing the deviation between the target and the nominal object based on the comparison. The processor is configured to execute instructions, including determining a set of steps required for this, and adjusting assembly instructions for the target object based on the set of steps described above.

Statement 16: A non-temporary computer-readable medium of Statement 15, the instructions are further configured such that the processor derives the sequence of steps using stochastic gradient descent.

Statement 17: A non-temporary computer-readable medium of any of statements 15-16, the target object being evaluated against the nominal object in the above steps of the assembly process.

Statement 18: A non-temporary computer-readable medium of any of statements 15-17, the target object being evaluated against the final configuration of the nominal object.

Statement 19: A non-temporary computer-readable medium of any of statements 15-18, the instruction determines the sequence of steps described above using a machine learning model configured to minimize deviations. Further configured to be.

Statement 20: A non-temporary computer-readable medium of any of statements 15-19, the deviation of which is minimized using Markov Decision Process (MDP) via reward formulation.

Claims (15)

A method for optimizing the workflow in the assembly line,
In a step of the target target assembly process, a step of detecting the target target assembly error and a step of detecting the target target assembly error.
A step of evaluating the target object and the nominal object in the step of the assembly process and thereby obtaining a comparison.
Based on the comparison, a step of determining a series of steps required to minimize the deviation between the target object and the nominal object, and
A method comprising the step of adjusting an assembly instruction for the target object based on the series of steps and thereby obtaining the adjusted assembly instruction. The method of claim 1, wherein the target object is evaluated against the nominal object in the step of the assembly process. The method of any one of claims 1 and 2, wherein the target object is evaluated for the final configuration of the nominal object. The method according to any one of claims 1 to 3, wherein the series of steps is determined using a machine learning model configured to minimize the deviation. The method of claim 4, wherein the machine learning model is a gated recurrent unit (GRU) model. The method of claim 4, wherein the machine learning model is based on a Long Short-Term Memory (LSTM) model. 4. The method of claim 4, wherein the machine learning model is trained with tracking information representing the assembly process for the target object, and the tracking information is collected from a set of workers. The method of any of claims 1-7, further comprising capturing the motion data corresponding to the assembly process, comprising digitally recording the movement of the hand while the motion data is assembling the target object. A step of extracting a set of images representing the assembly of the target object from the motion data, and
8. The method of claim 8, further comprising a step of evaluating the set of images and thereby identifying the operator's execution of the series of steps to assemble the target object. The method of any of claims 1-9, wherein the error in the assembly of the target object is detected using one or more machine learning models. The method of any of claims 1-10, wherein the adjusted assembly instructions are configured to provide the operator with instructions for minimizing the deviation of the target object. The deviation is determined based on the similarity between the series of steps to complete the assembly process of the target object and another series of steps to complete the assembly process of the nominal object. , The method according to any one of claims 1 to 11. The method of any of claims 1-12, wherein the deviation is minimized using Markov Decision Process (MDP) via reward formulation. One of claims 1 to 13, wherein the adjusted assembly instructions are transformed using one or more natural language processing (NLP) algorithms to provide the operator with the adjusted assembly instructions. The method described in. The method of any of claims 1-14, wherein the stochastic gradient descent method is used to derive the series of steps to complete the assembly process of the target object.

Images