Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
  • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
  • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/29—Graphical models, e.g. Bayesian networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/004—Artificial life, i.e. computing arrangements simulating life
  • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/0464—Convolutional networks [CNN, ConvNet]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/09—Supervised learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/092—Reinforcement learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/0985—Hyperparameter optimisation; Meta-learning; Learning-to-learn
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
  • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
  • G06V10/776—Validation; Performance evaluation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
  • G06V10/778—Active pattern-learning, e.g. online learning of image or video features
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
  • G06V10/778—Active pattern-learning, e.g. online learning of image or video features
  • G06V10/7784—Active pattern-learning, e.g. online learning of image or video features based on feedback from supervisors
  • G06V10/7788—Active pattern-learning, e.g. online learning of image or video features based on feedback from supervisors the supervisor being a human, e.g. interactive learning with a human teacher
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/84—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N7/00—Computing arrangements based on specific mathematical models
  • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks

Definitions

• the present disclosure relates to artificial intelligence in general, and to automatic improvement of artificial intelligence classification models, in particular.
• AI Artificial Intelligence
• the intelligence processes may comprise learning the acquisition of information and rules for using the information, reasoning using rules to reach approximate or definite conclusions and self-correction.
• Particular applications of AI comprise expert systems, speech recognition, machine vision, autonomous driving, intelligent routing in content delivery networks, military simulations, and the like.
• AI has become very popular in inspection systems, in particular systems aimed for identification and classification of items or defects in items, products, and the like.
• AI techniques have become an essential part of the technology industry, helping to solve many challenging problems in the manufacturing process.
• One exemplary embodiment of the disclosed subject matter is a method comprising: obtaining an instruction to improve a classification model, wherein the classification model is utilized for a supervised image classification task; in response to said obtaining, iteratively improving a model-performance measurement of the classification model, wherein in each iteration the model-performance measurement of the classification model is improved by at least a predetermined target goal, wherein said iteratively improving comprises: determining a list of hypotheses, wherein each hypothesis of the list of hypotheses relates to a strategy to potentially improve the classification model, wherein for each hypothesis a score is determined, wherein the score of a hypothesis indicates a likelihood that an application of the hypothesis improves the model-performance measurement of the classification model; generating, based on the list of hypotheses and scores thereof, a hypotheses graph for improving the classification model, wherein each node of the hypotheses graph comprises a hypothesis of the list of hypotheses; selecting a selected hypothesis from the hypotheses graph, wherein said selecting is based on
• an AI system comprising: a classification model improvement module configured to obtain an instruction to improve a classification model.
• the classification model is utilized for a supervised image classification task.
• the classification model improvement module is configured to iteratively improve a model-performance measurement of the classification model. In each iteration the model-performance measurement of the classification model is improved by at least a predetermined target goal.
• the classification model improvement module comprises a planning module configured to: determine a list of hypotheses, wherein each hypothesis of the list of hypotheses relates to a strategy to potentially improve the classification model, wherein for each hypothesis a score is determined, wherein the score of a hypothesis indicates a likelihood that an application of the hypothesis improves the model-performance measurement of the classification model; generate, based on the list of hypotheses and scores thereof, a hypotheses graph for improving the classification model, wherein each node of the hypotheses graph comprises a hypothesis of the list of hypotheses; select a selected hypothesis from the hypotheses graph, wherein said selecting is based on a traversal of the hypotheses graph.
• the classification model improvement module further comprises an execution module configured to execute the selected hypothesis, update the classification model and improve the model-performance measurement by at least the predetermined target goal.
• Yet another exemplary embodiment of the disclosed subject matter is a computerized apparatus having a processor, the processor being adapted to perform the steps of: obtaining an instruction to improve a classification model, wherein the classification model is utilized for a supervised image classification task; in response to said obtaining, iteratively improving a model -performance measurement of the classification model, wherein in each iteration the model-performance measurement of the classification model is improved by at least a predetermined target goal, wherein said iteratively improving comprises: determining a list of hypotheses, wherein each hypothesis of the list of hypotheses relates to a strategy to potentially improve the classification model, wherein for each hypothesis a score is determined, wherein the score of a hypothesis indicates a likelihood that an application of the hypothesis improves the model-performance measurement of the classification model; generating, based on the list of hypotheses and scores thereof, a hypotheses graph for improving the classification model, wherein each node of the hypotheses graph comprises a hypothesis of the list of hypotheses; selecting a selected
• Y et another exemplary embodiment of the disclosed subject matter is a computer program product comprising a non-transitory computer readable storage medium retaining program instructions, which program instructions when read by a processor, cause the processor to perform a method comprising: obtaining an instruction to improve a classification model, wherein the classification model is utilized for a supervised image classification task; in response to said obtaining, iteratively improving a model- performance measurement of the classification model, wherein in each iteration the model-performance measurement of the classification model is improved by at least a predetermined target goal, wherein said iteratively improving comprises: determining a list of hypotheses, wherein each hypothesis of the list of hypotheses relates to a strategy to potentially improve the classification model, wherein for each hypothesis a score is determined, wherein the score of a hypothesis indicates a likelihood that an application of the hypothesis improves the model-performance measurement of the classification model; generating, based on the list of hypotheses and scores thereof, a hypotheses graph for improving the classification model,
• Figure 1 shows a flowchart diagram of a method, in accordance with some exemplary embodiments of the disclosed subject matter
• Figure 2 shows a flowchart diagram of a method, in accordance with some exemplary embodiments of the disclosed subject matter
• Figure 3 shows a block diagram of an apparatus, in accordance with some exemplary embodiments of the disclosed subject matter
• Figure 4 shows a schematic illustration of an exemplary architecture, in accordance with some exemplary embodiments of the disclosed subject matter;
• Figure 5A shows an illustration of an exemplary hypotheses goal graph, in accordance with some exemplary embodiments of the disclosed subject matter;
• Figure 5B shows an illustration of an execution of an exemplary hypotheses goal graph, in accordance with some exemplary embodiments of the disclosed subject matter
• Figure 6 shows an illustration of a hypotheses goal graph, in accordance with some exemplary embodiments of the disclosed subject matter.
• the AI-based software may be configured to monitor a production process or a product of manufacturing machine, such as in a factory, in a production plant, or the like.
• the AI-based software may be configured to determine, such as based on visual input, if the machine functions properly.
• the AI-based software may be configured to utilize a classification model to classify products image, classify items within the images, identify defects within items or products, or the like.
• AI-based software may be used for Automated Optical Inspection (AOI), such as in Flat Panel Displays (FPD) manufacture, Printed Circuit Board (PCB) manufacture, or the like.
• AOI Automated Optical Inspection
• FPD Flat Panel Displays
• PCB Printed Circuit Board
• the AOI may be performed using Al-based software, but without providing full information to the developer of the Al-based software.
• the same developer may provide the Al- based software to different competing manufacturers, and they may wish to limit the information they disclose to their shared vendor to limit potential exposure.
• the Al-based software may operate in a detached environment, such as in a customers’ factory, without the ability to send or share information with other AI software, the developer of the Al-based software, research and development center of the Al-based software, or the like. Additionally or alternatively, the Al-based software may operate in a connected environment, but may be subject to restrictions over volume of data to transfer and share with third parties, including the developer of the Al-based software. [0019] Another technical problem dealt with by the disclosed subject matter is to automatically enhance the accuracy of Al-based software, without relying on data scientists and human experts. In some exemplary embodiments, Al-based software may be used for supervised image classification in various systems. A user may control the Al-based software.
• the Al-based software may be configured to collect images. Each image may be labeled, such as by human users, for training. Experts may provide hyper parameters for the model utilized by the Al-based software, manually remove irrelevant features from input data, inspect errors, perform feature engineering, or the like. While AI-based software presents a range of new functionalities, AI-based software may be limited by the quality of the data provided thereto in training, and to the specific configurations of the model used thereby. [0020] One technical solution is to perform a self-motivated learning that iteratively improves an accuracy of the classification model. In some exemplary embodiments, the self-motivated learning may be performed automatically, occasionally, periodically, such as every predetermined timeframe, or the like.
• the self- motivated learning may be initiated by an instruction from a human user, from the Al based software, from a server associated with the AI-based software, or the like. It may however be appreciated that the self-motivated learning may be performed automatically and may be guided without the involvement of data scientists and human experts.
• the AI-based software may utilize pre-defined interfaces and provide the machine learning and AI mechanism for training models, improving models, tracking performance of models, or the like.
• the AI- based software may automatically interact with user application via two-way callbacks, enabling the AI-based software to invoke requests for and receive additional training data, improved information, or the like.
• improving the accuracy of the classification model may be measured by improving an accuracy measurement or any other model performance metric of the classification model, such as an F-score (e.g., FI score), an accuracy score, a precision score, a recall score, confusion matrix, area-under-curve, Receiver Operating Characteristics (ROC), accuracy differences between training set and validation set, time of GPU and CPU resources, computer memory needed for prediction, or the like.
• a predetermined target goal for improving the accuracy of the classification model may be set automatically based on the classification task, may be set manually be a user, such as by the operator of the AI software, or the like.
• the predetermined target goal may be a 0.5% improvement, e.g., improving an accuracy measurement of the classification model by at least 0.5% in each iteration of the learning.
• the predetermined target goal may be about 0.1%, 0.2%, 1%, or the like.
• a list of hypotheses for improving the classification model may be determined in each iteration.
• Each hypothesis may relate to a strategy to potentially improve the classification model.
• the hypothesis may relate to a choice of algorithm, such as which neural network to be used by the classification model, or the configuration of the algorithm, such as the neural network topology (e.g., number of layers, size of layers, connectivity between layers), hyper parameters, combination between several algorithmic concepts, or the like.
• a score may be determined for each hypothesis. The score may indicate a likelihood that an application of the hypothesis improves the accuracy measurement of the classification model.
• the score may be determined using a prediction model that implements machine learning techniques.
• the prediction model may implement supervised learning techniques and improve over time in view of actual observed results. Additionally or alternatively, each hypothesis may be associated with a cost function that indicates the cost of applying the hypothesis on the classification model, such as the time required for applying the hypothesis, the computational resources required for applying the hypothesis, additional resources required for applying the hypothesis (e.g., additional manual labeling for an additional training set), or the like.
• a hypotheses graph for improving the classification model may be generated based on the list of hypotheses and scores thereof.
• Each node of the hypotheses graph may comprise a hypothesis of the list of hypotheses.
• the hypotheses graph may be ordered based on the scores of the hypotheses, imposing a traversal order thereof. As an example, the graph may be ordered so that hypotheses with higher scores may be traversed first.
• a selected hypothesis may be selected from the hypotheses graph, based on a traversal thereof.
• the selected hypothesis may be executed to update the classification model and improve the accuracy measurement thereof by at least the predetermined target goal.
• several hypotheses may be executed in parallel and best candidate may be selected therefrom. The parallel execution may be performed using a same Graphical Processing Unit (GPU), a Central Processing Unit (CPU), a distributed execution environment, or the like.
• capabilities of the disclosed solution may be embedded into the internal algorithmic process of manufacturing a product and Quality Assurance (QA) thereof.
• QA Quality Assurance
• an AOI classification decision sub-system may be configured to detect and categorize defects using the automatically generated classification models.
• defect detection may be performed early in the manufacturing process, instead of inspecting for defects at the end of the scanning process, such as by generating and validating defect predictions.
• the AOI classification decision sub-system may operate and detect defects without providing images, videos, raw data, or the like, to an external party.
• the disclosed solution may be utilized in AOI during FPD manufacture.
• the FPD industry develops intensively and involves rapid and classified innovations that may be important not to be exposed.
• the disclosed solution may be employed in different stages in the FPD manufacture, such as in inspection, metrology, yield management solutions, test and repair solutions, or the like.
• the disclosed subject matter enables FPD manufacturers to detect critical defects, while locating sources of errors both in the ramp-up phase and during production, and analyze data from the production line in order to quickly diagnose and repair process-related defects.
• the disclosed solution may be utilized in AOI of PCBs, touch sensors, or other electronic devices during their manufacturing process, in automatic repair systems of such devices, in direct imaging systems for such devices, or the like.
• the disclosed subject matter may implement reinforcement learning regarding the hypotheses execution to improve the self-motivated learning.
• reinforcement learning may be performed to provide for a balance between exploration of hypotheses and exploitation of hypotheses that had proven to be useful in the past.
• the learning task may be classified and a reinforcement learning may be utilized to determine which hypotheses to be applied, an order of application between hypotheses, or the like.
• the disclosed subject matter may implement reinforcement learning to predict a useful hypotheses graph based on a sequence of previously generated hypotheses graphs of the same learning task, based on the classification of the learning task, based on predicted scores of the different hypotheses, or the like.
• the hypotheses to be used by the disclosed subject matter may include several different categories.
• a hypotheses generator may be utilized to generate a hypothesis for a given model. Different hypotheses generators may be provided.
• some hypotheses generators may be "best practice" generators, configured to generate hypotheses that are considered to be useful as a rule of thumb.
• some hypotheses generators may be manually programmed by Subject Matter Experts (SMEs), such as experts that have intimate knowledge of the manufacture process and machines used therein. The SME may program scripts that identify properties of the learning task and generate, in an automated manner, hypotheses that could improve the performance of the trained model.
• SMEs Subject Matter Experts
• the SME hypotheses generators may be manually programmed, but may be invoked automatically and may not require the involvement of the SME herself in the self-motivated learning process. Additionally or alternatively, some hypotheses generators may be random hypotheses generators configured to provide random hypotheses, so as to enable exploration of potentially unappealing hypotheses.
• One technical effect of utilizing the disclosed subject matter is providing an autonomous solution for self-motivated learning, disconnected from external computing and databases.
• the disclosed subject matter enables inferencing, data-cleansing, novelty- detection, and the like, without requiring additional data or exposing internal data.
• the disclosed solution utilizes the experience and the access to all relevant data that the developer of the AI machine, without exposing data of the factory or the production plant utilizing the AI machine, to the developer or to any other external party.
• Another technical effect of utilizing the disclosed subject matter is reducing the Time To Market (TTM) required from a product being conceived until its being available for sale.
• TTM Time To Market
• TTM is important in industries where products are outmoded quickly, especially in the world of micro-electronics, such as in FPD, PCB, or the like.
• the disclosed subject matter improves the robustness and stability of AI-based software, and enables automatic model performance tracking.
• Yet another technical effect of utilizing the disclosed subject matter is gradual improvement of a model, in an iterative manner.
• the disclosed subject matter may employ different hypotheses in different combinations and orders to provide an improved model.
• many small gradual improvements may be feasible using limited resources, while attempting to achieve a large improvement (that may be the accumulation of the small gradual improvements) may be hard or even unfeasible.
• the disclosed subject matter may provide for one or more technical improvements over any pre-existing technique and any technique that has previously become routine or conventional in the art. Additional technical problem, solution and effects may be apparent to a person of ordinary skill in the art in view of the present disclosure.
• Step 110 an instruction to improve a classification model may be obtained.
• the classification model may be utilized for a supervised image classification task, such as for AOI classification, In-Process-Quality- Control (IPCQ) tasks, QI task, or the like.
• the supervised image classification task may be a phase in an AI-based inspection software aimed for identification and classification of items or defects in items, products, or the like.
• the AI-based inspection software may be configured to define the AI interfaces, provides the machine learning and AI mechanism for training classification models, improving classification models, tracking, automatically interacting with a user application via two-way callbacks, or the like.
• the instruction may be obtained from a system utilizing the AI-based inspection software, a server related to, and not from an external application.
• the instruction may be provided as a self-motivated improving step of the AI-based inspection software.
• a model-performance measurement of the classification model such as but not limited to accuracy measurement may be improved by at least a predetermined target goal.
• the predetermined target goal may be a default target goal for improving model-performance measurements.
• the predetermined target goal may be predetermined by the AI inspection system or a user thereof, based on properties of the supervised image classification task, based on properties of the products being classified, based on properties of the manufacturing process of the products, or the like. Additionally or alternatively, the predetermined target goal may be determined based on the target model- performance measurement that the classification model is requested to achieve.
• Step 120 may be performed iteratively, such as a predetermined number of iteration, until a general target goal for improving the classification model is achieved, until no improvement is achieved, or the like. Additionally or alternatively, when the target goal is not reached in several successive iterations (such as above a predetermined number of iterations), it may be determined that no additional iteration should be performed as there may be low likelihood that an additional iteration would provide an improvement.
• Each iteration of Step 120 may comprise Steps 130-160.
• a list of hypotheses may be determined.
• each hypothesis of the list of hypotheses may relate to a strategy to potentially improve the classification model.
• the strategy may relate to a modification of a model type of the classification model, a training parameter of the classification model, a modeling parameter of the classification model, a modeling of the supervised image classification task, a training set utilized to train the classification model, a validation set utilized to validate the classification model, a test set utilized to test the classification model, or the like.
• a hypothesis may be related to the type of learning algorithm of the classification model, such as supervised learning, unsupervised learning, reinforcement learning, self-learning, feature learning, sparse dictionary learning, anomaly detection, association rules, or the like.
• the hypotheses list may comprise at least one hypothesis that is not related to the model type of the classification model, such as a hypothesis related to training parameters of the classification model, a hypothesis related to parameters of the classification task, or the like.
• the list of hypotheses or a portion thereof may be related to the supervised image classification task. Such hypotheses may be related to the number of classes of the supervised image classification task, the number of images in each class of the supervised image classification task, the image size in the supervised image classification task, a noisy dataset-element label estimation measurement of the supervised image classification task, or the like.
• the list of hypotheses may comprise three types of hypotheses: best practice hypotheses, Reinforcement Learning-based hypotheses, and SME hypotheses.
• a best practice hypothesis may be a hypothesis that is approved to be generally efficient in improving a classification model.
• a Reinforcement Learning-based hypothesis may be a hypothesis selected randomly for additional exploration or by exploitation of re- past Reinforcement Learning -based hypothesis.
• An SME hypothesis may be related to an SME of the supervised image classification task, such as a hypothesis that is known to be efficient in improving classification models of the specific supervised image classification task.
• the SME hypothesis may be programmed by a human, may be automatically obtained by the AI inspection system, or the like.
• a score may be determined for each hypothesis.
• the score of a hypothesis may indicate a likelihood that an application of the hypothesis improves the model-performance measurement of the classification model.
• the score may be determined based on previous learning or training, may be predicted based on the hypothesis parameters, or the like. Additionally or alternatively, the score may be externally set by a generator of the hypothesis, may be obtained from a database of hypothesis, or the like.
• determining the list of hypotheses may be performed by applying a reinforcement learning algorithm that utilizes a previous set of hypotheses associated with the supervised classification task.
• the reinforcement learning algorithm may be configured to improve the selection of hypotheses to the list of hypotheses, based on the previous set of hypotheses and evaluated scores thereof.
• a hypotheses graph for improving the classification model may be generated.
• the hypotheses graph may be generated based on the list of hypotheses and scores thereof.
• Each node of the hypotheses graph may comprise a hypothesis of the list of hypotheses.
• the hypotheses graph may be a directed graph with a preorder set in accordance with the scores of the hypotheses.
• the hypotheses graph may comprise parent nodes and child nodes. Each parent node may be connected by directed edges to one or more child nodes. Each parent node may represent an abstract Algorithm or a hypothesis. Each of the child nodes of a parent node may represent a different concretization of the abstract hypothesis represented by the parent node. In some exemplary embodiments, some child node may have child nodes thereof, connected thereto by directed edges. The child nodes of such child nodes, may represent an additional concretization to the concrete hypothesis represented by the child node. In some exemplary embodiments, the list of hypotheses may comprise the abstract hypothesis and the concretizations thereof. Additionally or alternatively, the list of hypotheses may comprise abstract hypotheses and concretization thereof may be obtained during the generation of the graph.
• hypotheses graph may be generated by applying a reinforcement learning algorithm that utilizes a sequence of previously generated hypotheses graphs.
• a selected hypothesis may be selected from the hypotheses graph.
• the selection may be performed based on a traversal of the hypotheses graph.
• the traversal of the hypotheses graph may be in accordance with the preorder thereof. Hypotheses with higher scores may be traversed first.
• the selection may comprise selecting a hypothesis that, when applied, improves the model-performance measurement of the classification task by a highest measurement compared to improvement gained by other hypotheses in the hypotheses graph.
• the selection may be performed after traversing the whole graph, a predetermined portion thereof, or the like.
• the selection may comprise selecting a first hypothesis encountered during the traversal of the hypotheses graph that an application thereof improves the model-performance measurement of the classification task by at least the predetermined target goal.
• an abstract computation relating to the abstract hypothesis of the parent node may be performed.
• the abstract computation may comprise determining how an application of the hypothesis improves the model-performance measurement of the classification model.
• a concrete computation may be performed.
• the concrete computation may comprise determining how much an application of the specific concretization of the abstract hypothesis affects the model- performance measurement of the classification model.
• the concrete computation may comprise executing the concrete hypothesis and determining the improvement in the model-performance measurement resulted from applying the hypothesis.
• the scores of the hypotheses may be updated based on the abstract computation and the concrete computation.
• the selected hypothesis may be selected based on the abstract computation and the concrete computation. It may be noted that in some cases the score of the abstract hypotheses and the scores of the concretizations thereof may not necessarily be correlated. A concrete hypothesis may be selected over other concrete hypotheses despite the abstract hypothesis associated therewith having a lower score. Additionally or alternatively, a best abstract hypothesis may be selected, and a best concretization thereof may be applied.
• Step 160 the selected hypothesis may be executed and the classification model and the model-performance measurement may be updated.
• Step 150 in case that the model-performance measurement is not improved by at least the predetermined target goal, Step 150 may be repeated.
• a hypotheses graph may be generated using reinforcement learning.
• the graph generation process may be an algorithmic leamable process performed based on previously generated hypotheses graphs.
• a reinforcement learning algorithm may be utilized to generate the hypotheses graph based on a sequence of previously generated hypotheses graphs.
• the reinforcement learning algorithm may be sequence-based memory algorithm, such as a Recursive NN, an LSTM, or the like, that can remember and recognize a sequence of inputs (such graphs of hypotheses and scores thereof) and generate as an output a new sequence may be utilized.
• the sequence input may comprise a sequence of previously generated hypotheses graphs.
• Each hypotheses graph may be labeled with a score metric indicating whether the hypotheses graph improved the associated classification model, and which hypothesis was applied.
• the reinforcement learning algorithm may be configured to generate a hypotheses graph with a best score metric based on the sequence input. [0050] Additionally or alternatively, the reinforcement learning may be applied while determining the list of hypotheses. A reinforcement learning algorithm that utilizes a previous set of hypotheses associated with the supervised classification task may be applied. [0051] On Step 220, the hypotheses graph may be executed. In some exemplary embodiments, the hypotheses graph may be traversed. An application of the hypothesis in each node of the graph may be fully or partially simulated on the classification model in order to determine the potential improvement of the model-performance measurement of the classification model in response to applying the hypothesis thereon. A hypothesis may be selected similar to Step 150 of Figure 1.
• the selected hypothesis may be executed and a trained model may be evaluated.
• An evaluated metric representing the improvement of the accuracy of the classification model as a result of applying the associated hypothesis from the hypotheses graph.
• An evaluated metric may indicate which hypothesis was applied.
• the generated hypotheses graph and the evaluated metrics may be saved.
• the generated hypotheses graph and the evaluated metrics may be saved in an internal database of goal graphs associated with the server performing the self-motivated learning for the classification model.
• the evaluated trained models may be saved in an associated trained model database for future learning.
• the saved data may be utilized to improve the reinforcement learning.
• the reinforcement learning algorithm may be configured to improve its solution based on past decisions thereof.
• the reinforcement learning algorithm may be configured to utilize the evaluated model and metric of each hypothesis to apply a positive or a reward-based reinforcement.
• positive reinforcement when an event, such as an improvement of the accuracy, occurs due to a particular behavior, such as a particular hypothesis or feature thereof, increases the strength and the frequency of the behavior, e.g., the hypothesis of the feature thereof the caused the improvement.
• An Apparatus 300 may be configured to support an active machinery for performing self- motivated learning, inferencing, data-cleansing, novelty-detection, or the like, in accordance with the disclosed subject matter.
• Apparatus 300 may comprise one or more Processor(s) 302.
• Processor 302 may be a GPU, a TensorCore, a CPU, a microprocessor, an electronic circuit, an Integrated Circuit (IC), or the like.
• Processor 302 may be utilized to perform computations required by Apparatus 300 or any of its subcomponents.
• Apparatus 300 may comprise an Input/Output (I/O) module 305.
• I/O Module 305 may be utilized to provide an output to and receive input from a user, such as, for example obtaining instructions, training data, labeling functions, outputting hypotheses graphs, or the like.
• Apparatus 300 may comprise Memory 307.
• Memory 307 may be a hard disk drive, a Flash disk, a Random Access Memory (RAM), a memory chip, or the like.
• Memory 307 may retain program code operative to cause Processor 302 to perform acts associated with any of the subcomponents of Apparatus 300.
• Apparatus 300 may be configured to monitor a classification model and autonomously improving the accuracy thereof, without relying on outside computing and databases.
• I/O Module 305 may be utilized to obtain an instruction to improve a classification model utilized for a supervised image classification task.
• the instruction may be obtained from a user, from a manufacturing machine server, or the like.
• Apparatus 300 may be configured to iteratively improving an model- performance measurement of the classification model, such as an accuracy measurement of the classification model, or the like.
• Hypotheses List Generator 310 may be configured to generate lists of hypotheses. Each hypothesis of the list of hypotheses relates to a strategy to potentially improve the classification model. Each hypothesis may be associated with a score indicating a likelihood that an application of the hypothesis improves the accuracy measurement of the classification model. The score may be computed by Hypotheses List Generator 310 or a component thereof. [0061] In some exemplary embodiments, Hypotheses List Generator 310 may be configured to utilize a hypothesis generator such as 312-316 to generate the list of hypotheses. Each generated hypothesis may relate to a modification of the classification model, such as a model type thereof, a parameter related to the training data thereof, a modeling parameter, a parameter related to the supervised image classification task, or the like.
• Hypotheses List Generator 310 may be configured to apply Best Practice Hypothesis Generator 312 to generate one or more hypotheses of the hypotheses list.
• Best Practice Hypothesis Generator 312 may be configured to generate predefined best practice hypotheses that are configured to improve the classification model.
• the predefined best practice hypotheses may be generally proved to improve classification models, and may not directly relate to the specific classification task of the classification model.
• predefined best practice hypothesis may be retraining the classification model with additional training data.
• Hypotheses List Generator 310 may be configured to apply SME Hypothesis Generator 316 to generate one or more hypotheses of the hypotheses list.
• SME Hypothesis Generator 316 may be a computer program product designed and developed by a SME to generate hypotheses based on one or more parameters of the supervised image classification task.
• SME Hypothesis Generator 316 may be developed by a user of the classification model, may be developed based on an input from the user, such as based on parameters of the supervised image classification task, based on parameters of the product being inspected, or the like.
• SME Hypothesis Generator 316 may be configured to generate hypotheses based on a number of classes of the supervised image classification task, a number of images in each class of the supervised image classification task, an image size in the supervised image classification task, a noisy label estimation measurement of the supervised image classification task, or the like.
• Hypotheses List Generator 310 may be configured to apply a Random Hypothesis Generator 314 to generate one or more hypotheses of the hypotheses list.
• Random Hypothesis Generator 314 may be configured to generate random hypotheses.
• the random hypotheses may be an exploration mechanism for improving the classification model.
• Random Hypothesis Generator 314 may be configured to randomly select or generate hypotheses, with reference to an estimated probability of the score thereof.
• random hypotheses generated by Random Hypothesis Generator 314 may be determined as SME hypotheses based on execution results thereof by Execution Module 350. Such hypotheses may be utilized by SME Hypothesis Generator 316 in successive iterations.
• Hypotheses Graph Generator 320 may be configured to generate graphs of hypotheses based on the list of hypotheses generated by Hypotheses List Generator 310. Hypotheses Graph Generator 320 may be configured to generate each graph based on the scores of the hypotheses in the associated list of hypotheses.
• Hypotheses Graph Generator 320 may be configured to utilize Reinforcement Learning Module 340 to generate a hypotheses graph.
• Reinforcement Learning Module 340 may be configured to learn and determine what is the predicted method to automatically generate a graph of probable hypotheses to be tested for the classification model.
• Reinforcement Learning Module 340 may be configured to train Hypotheses Graph Generator 320 with previously generated hypotheses graphs to generate a new hypotheses graph.
• Reinforcement Learning Module 340 may be configured to utilize a sequence of previously generated hypotheses graphs to generate the hypotheses graph as a label for such sequence.
• Reinforcement Learning Module 340 may be configured to train Hypotheses Graph Generator 320 to generate a graph to be executed given an input classification task. Additionally or alternatively, Reinforcement Learning Module 340 may be utilized by Hypotheses List Generator 310 to determine a list of hypotheses based on a previous set of hypotheses associated with the supervised classification task.
• graphs generated by Hypotheses Graph Generator 320 and evaluated metrics associated therewith may be stored in Hypotheses Graphs Database 390.
• Data stored in Hypotheses Graphs Database 390 may be utilized in next iterations of improvements, in generating graphs for other classification models, updating an existing model, or the like.
• Hypothesis Selection Module 330 may be configured to select a selected hypothesis from the hypotheses graph generated by Hypotheses Graph Generator 320. Hypothesis Selection Module 330 may be configured to traverse the hypotheses graph and select a hypothesis that, when applied, improves the accuracy measurement of the classification task by a highest measurement compared to improvement gained by other hypotheses in the hypotheses graph. Additionally or alternatively, Hypothesis Selection Module 330 may be configured to traverse the hypotheses graph and select the first hypothesis encountered during the traversal of the hypotheses graph that an application thereof improves the accuracy measurement of the classification task by at least the predetermined target goal.
• Execution Module 350 may be configured to execute the selected hypothesis to update the classification model and improving the accuracy measurement by at least the predetermined target goal.
• Execution Module 350 may be configured to utilize additional data in order to execute the hypothesis, such as additional training data obtained from Training Data 380, additional SME data from human experts, retagging images by users, modifying values of hyper-parameters, modifying the learning algorithm, or the like.
• Reinforcement Learning Module 340 may be configured to update the reinforcement learning algorithm utilized to generate the hypotheses graphs based on the results of executing the selected hypothesis. Reinforcement Learning Module 340 may be configured to improve the reinforcement learning algorithm based on the selected hypothesis, the improved classification model, the improvement in the accuracy, or the like.
• Apparatus 300 may comprise Prediction Tracker 360 that may be configured to monitor and regulate the prediction quantitative performance.
• Prediction Tracker 360 may be configured to analyze the instruction to keep track of the prediction performance of a classification model, to track and detect a probable mismatch between the data distribution of the incoming images to be classified as regarding to the data distribution of the images in the training phase, to track the reinforcement learning, to regulate the performance of the reinforcement learning, to validate that the predetermined target goal for improving the accuracy is achieved, or the like.
• FIG. 4 showing a schematic illustration of an exemplary architecture in which the disclosed subject matter may be utilized, in accordance with some exemplary embodiments of the disclosed subject matter.
• AI System 400 may be configured to automatically perform self-motivated improvement of a classification model.
• the classification model may be utilized for a supervised image classification task.
• AI System 400 may be associated with an inspection system aimed for classification of items, identification of defects in products, inspecting quality of products, or the like.
• the classification model may be utilized to classify images of the items or the products, detecting anomalies therein, detecting defects in the products, or the like.
• the self-motivation improvement mechanism may be driven by previously-set goals, such as goals set by the manufacture utilizing AI System 400, goals defined based on the desired quality of the products, goals defined based on the required final accuracy of the classification model, or the like.
• AI System 400 may be configured to employ a Brain Module 420 for performing the improvement of the classification model.
• a User AI Interactive Module 410 may be configured to issue an instruction to Brain 420 via a Model Learn Application Programing Interface (API) 424, to improve the classification model.
• Brain 420 may be configured to iteratively improve an accuracy measurement of the classification model by at least a predetermined target goal in each iteration.
• Brain 420 may be configured to obtain additional data for the improvement from external resources via Data API 422.
• the additional data may comprise training data, values of hyper-parameters, labeling functions, an improvement target goal, or the like.
• a Classification Module 430 of Brain 420 may be configured to analyze the learning instruction and classify the learning task of the classification model.
• Classification Module 430 may be configured to analyze the type of the learning task, an AI technique the classification model employs to perform the classification task, parameters of the classification model, parameters of the classification task, or the like.
• Classification Module 430 may classify the learning task such as based on the number of labeling/classes, number of instances in each class in the training set, a size of input, an aspect ratio of an image input, noisy label estimation measurement, and parameters related thereto, properties of instances of different classes, or the like.
• a Planning Module 440 of Brain 420 may be configured to generate an execution-plan for a hypotheses graph for improving the classification model based on the classification determined by Classification Module 430.
• Each node of the hypotheses graph may comprise a hypothesis relating to a strategy to potentially improve the classification model.
• Planning Module 440 may be configured to implement reinforcement learning regarding the hypotheses execution to improve the self-motivated learning.
• Planning Module 440 may be configured to perform reinforcement learning to determine which hypotheses to be applied, an order of application between hypotheses, or the like.
• Planning Module 440 may utilize reinforcement learning to predict a useful hypotheses graph based on a sequence of previously generated hypotheses graphs of the same learning task, based on the classification of the learning task, based on predicted scores of the different hypotheses, or the like.
• Planning Module 440 may be configured to employ other machine learning techniques to generate the execution-plan of the hypotheses graph, such as using Automated Machine Learning (Auto ML), Deep Neural Network (DNN), Convolutional Neural Network (CNN), Transfer Learning, Clustering, Weak Learning, or the like.
• Automated Machine Learning Auto ML
• DNN Deep Neural Network
• CNN Convolutional Neural Network
• Transfer Learning Clustering
• Weak Learning or the like.
• An Execution Module 450 of Brain 420 may be configured to execute the hypothesis graph. Executing the hypothesis graph may comprise traversing the graph to iteratively select a hypothesis that is likely to improve the accuracy measurement of the classification model by at least the predetermined target goal and executing it, until the predetermined target goal is achieved. In each iteration, Execution Module 450 may be configured to select a hypothesis that, when applied, improves the accuracy measurement of the classification task by a highest measurement compared to improvement gained by other hypotheses in the hypotheses graph or portion thereof.
• Execution Module 450 may be configured to select a first hypothesis encountered during the traversal of the hypotheses graph that an application thereof improves the accuracy measurement of the classification task by at least the predetermined target goal.
• each hypothesis that is executed may be executed on a standalone copy of the model, so as to obtain an updated model based on the application of the hypothesis.
• performance of different alternative updated models, each of which is the outcome of executing a different hypothesis may be compared to select the preferred hypothesis to be applied.
• Execution Module 450 may initially execute the hypotheses in a small scale, to determine their applicability. After a hypothesis is selected, the hypothesis may be executed in full to obtain the updated model.
• Execution Module 450 may be configured to employ a Task Execution Server 490 via a Task Execution API 486 to externally execute the hypotheses graph.
• Task Execution Server 490 may have large computational resources, such as by applying multiple GPUs, may be a distributed computational environment, a cloud computing service, or the like.
• an improved classification model with an accuracy measurement improved by at least the predetermined target goal may be generated.
• the generated hypothesis graph may not gain the desirable target and Planning Module 440 may be instructed to regenerate a new execution-plan hypotheses graph.
• a Memorizing Module 460 of Brain 420 may be configured to store the generated hypotheses graph and evaluated metrics associated therewith.
• the generated hypotheses graph and the evaluated metrics may be saved in an internal database of hypotheses graphs associated AI System 400, in external databases, in databases associated with other AI systems, or the like.
• the generated hypotheses graph and the evaluated metrics may be utilized in future learning, such as to improve the reinforcement learning solution of Planning Module 440.
• a Reporting Module 470 of Brain 420 may be configured to report the result of executing the hypothesis graph to AI System 400, such as via Brain-to-App API 482.
• User AI Interactive Module 410 may be configured to assess the report and determine whether or not to invoke an additional improvement iteration.
• the report may indicate that the target goal was reached, and as a result, another iteration may be performed to further improve the model. Additionally or alternatively, the report may indicate that the target goal was not reached, and as a result, it may be determined that no additional iteration should be performed as there may be low likelihood (e.g., below a threshold) that an additional iteration would provide an improvement. Additionally or alternatively, in case the report indicates that the target goal was not reached, an additional iteration may be attempted, during which a different hypotheses graph may be generated, taking into account the failure of the previous hypotheses graph in providing the desired improvement.
• Brain 420 may be configured to communicate with other modules, similar to Brain 420, such as a brain module of other AI systems similar to Brain 420, other brain modules with the same type of Brain 420, or the like, via Brain-to-Brain API 484.
• Brain-to-Brain API 484 may be utilized to provide data for other AI systems for collaboration, distribution, or the like.
• a Model Tracking API 426 may be utilized to obtain performance information regarding a classification model.
• the accuracy measurement of the classification model may be monitored and tracked.
• Information regarding the performance such as accuracy measurements, number of applications of the classification model, instances on which the classification model was applied, or the like, may be provided to Brain 420 via Model Tracking API 426.
• the model tracking may be performed with respect to the classification model, to one or more updated classification models, or the like.
• such information may be utilized by Classification Module 430, by Planning Module 440, Memorizing Module 460, or the like.
• Hypotheses Graph 500 may be an exemplary hypotheses goal graph generated in response to an instruction to improve a classification model utilized for a supervised image classification task.
• the classification model may be utilized for quality control of FPD products.
• Hypotheses Graph 500 may be utilized to improve an accuracy measurement of the classification model, such as an FI -score, by at least a predetermined target goal, such as 0.5%.
• nodes of Hypotheses Graph 500 may comprise hypotheses. Each hypothesis in Hypotheses Graph 500 may relate to a strategy to potentially improve the classification model, such as strategies related to retraining the classification model, modifying the model, or the like.
• a leaf node may comprise a concrete hypothesis. Additionally or alternatively, an internal node may comprise an abstract hypothesis, the children of which may comprise concretizations of the abstract hypothesis.
• a Root Node 510 may represent a degenerated hypothesis that represents the general notion of improving the classification model. Additionally or alternatively, Root Node 510 may not comprise any hypothesis.
• Goal Graph 500 may comprise internal nodes, each of which is associated with one or more child nodes.
• Each internal, non-leaf, node may represent an abstract hypothesis indicating an abstract strategy for improving the classification model accuracy.
• Each child node may represent a different concretization of the abstract hypothesis of the associated internal node.
• the concretization may be related to parameters of the abstract hypothesis, such as assigning different values thereto. Additionally or alternatively, the concretization may be related to the manner of applying the hypothesis, sources of additional data, utilized algorithms, or the like.
• the order between children nodes may impose a traversal order on the nodes, which may impose an order of application and execution of the different hypotheses.
• Hypothesis HI in Node 520 may indicate a strategy of retraining the classification model with a clear dataset.
• Hypothesis Hla in Node 522 may be a concretization of Hypothesis HI.
• Hypothesis Hla may indicate a strategy of retraining the classification model with a dataset comprising 25% of the original dataset utilized to train the classification model.
• Hypothesis Hlb in Node 524 may be a different concretization of Hypothesis HI.
• Hypothesis Hlb may indicate a strategy of retraining the classification model with a dataset comprising N% of the original dataset utilized to train the classification model. N may be determined during the execution of Goal Graph 500.
• Hypothesis H2 in Node 530 may indicate a strategy of retraining the classification model with more samples.
• Hypothesis H2a in Node 532 may be a concretization of Hypothesis H2.
• Hypothesis H2a may indicate a strategy of retraining the classification model that is configured to classify instances to an additional two classes, a first and a second classes with N1 labeled samples from a first class and N2 samples from a second class.
• N1 and N2 may be determined during execution of H2a. Additionally or alternatively, N 1 and N2 may be predetermined .
• Hypothesis H3 in Node 540 may indicate a strategy of retraining the classification model with an adaptive learning rate.
• Hypothesis H3a in Node 542, Hypothesis H3b in Node 544 and Hypothesis H3c in Node 546 may be concretizations of Hypothesis H3.
• Hypothesis H3a may indicate retraining the classification model with an adaptive learning rate using a first training algorithm for the adaptive learn rate.
• Hypothesis H3b may indicate retraining the classification model with an adaptive learning rate using a second training algorithm for the adaptive learn rate.
• Hypothesis H3c may indicate retraining the classification model with an adaptive learning rate using a third training algorithm for the adaptive learn rate.
• Hypothesis H4 in Node 550 may indicate a strategy of retraining the classification model to classify additional number of classes, defined using additional labeling functions.
• the additional labeling functions may be obtained via a callback API.
• the additional labeling functions may be provided by a human user, may be obtained from a repository, or the like.
• a hypotheses graph such as Hypotheses Graph 500, may be generated in response to an instruction to improve a classification model utilized for a supervised image classification task (505).
• the classification model may be utilized for quality control of FPD products.
• Hypotheses Graph 500 may be utilized to improve an accuracy measurement of the classification model, such as an FI -score, by at least a predetermined target goal, such as 0.5%.
• Goal Graph 500 may be automatically generated in order to improve the accuracy measurement of a given classification model by 0.5%.
• Goal Graph 500 may comprise different hypotheses, H1-H4, that are likely to improve the FI score of the classification model by at least 0.5%.
• each hypothesis may be assigned a score and a cost.
• the score of a hypothesis may indicate a likelihood that an application of the hypothesis improves the accuracy measurement of the classification model by at least the target goal.
• only hypotheses with a score above a predetermined threshold, such as about 90%, 95%, or the like, may be included in the hypotheses graph.
• the cost of a hypothesis may indicate the cost of applying the hypothesis on the classification model, such as the time required to apply and execute the hypothesis, the computational resources, or the like.
• the hypotheses goal graph may be ordered in accordance with the score, the cost, a combination thereof, or the like.
• the ordering of the hypotheses graph may impose a traversal order thereof, such that hypotheses with higher scores may be traversed first.
• the order of the traversal may be HI, Hla, Hlb, H2, H2a, H3, H3a, H3b, H3c and H4.
• a hypothesis selector may be configured to execute Goal Graph 500 in order to select a hypothesis that improves the classification model by at least 0.5% (515).
• the hypothesis selector may be configured to select a selected hypothesis based on a traversal of Hypotheses Graph 500.
• abstract computations relating to the abstract hypotheses may be performed when traversing parent nodes, and concrete computations may be performed when traversing child nodes.
• the abstract and the concrete computations may be utilized to determine updated scores of the hypotheses and selecting a hypothesis to be applied on the classification model.
• the selected hypothesis may be executed, the classification model may be updated and the accuracy measurement thereof may be improved by at least the predetermined target goal.
• Hypothesis HI may be an abstract hypothesis indicating a strategy of retraining the classification model with a clear dataset. Additional concrete computations may be performed while traversing Nodes 522 and 524.
• An algorithm for selecting 25% of the dataset as a cleaner set for the retraining may be applied (523). The algorithm may be configured to select the top 25% image-label accurate datum to be in the cleaner dataset.
• An outlier removal algorithm may be applied in order to enhance the algorithm of Hypothesis Hla (526).
• An algorithm for selecting a predetermined percent of the dataset as a cleaner label- wise set for the retraining may be applied (525). The predetermined percent to be utilized in Hypothesis HI may be determined (527).
• the determination may be performed based on the cost, based on the predetermined target goal, based on other parameters of the classification model, or the like.
• An outlier removal algorithm may be applied in order to enhance the training dataset (528).
• the training dataset for the retraining based on Hypothesis Hla or Hlb may be determined (529).
• Hypothesis H2 may be an abstract hypothesis indicating a strategy of retraining the classification model with more labeled samples.
• Hypothesis H2a may indicate a strategy of retraining the classification model that is configured to classify instances between first and second classes with N1 samples from a first class and N2 samples from a second class.
• Tagging of the samples may be verified and clear samples may be returned (535).
• a call back to request additional samples from different classes may be issued (537). The call back may be issued to an AI system, to a user of the goal graph, to an external source, or the like
• a decision of how many samples of each class (N1 and N2) to utilize may be performed (539).
• Hypothesis H3 may be an abstract hypothesis indicating a strategy of retraining the classification model with an adaptive learning rate.
• concrete computations may be performed to evaluate Hypothesis H3a, Hypothesis H3b and Hypothesis H3c (543,545).
• a training algorithm for Hypothesis H3a, Hypothesis H3b and Hypothesis H3c (traversed on 543), may be selected as a first, a second or a third training algorithm for the adaptive leam rate, respectively.
• a respective training dataset may be obtained (545).
• Hypothesis H4 may be an abstract hypothesis indicating a strategy of retraining the classification model to classify additional number of classes, defined using additional labeling functions.
• Labeling functions for the additional classes to be used in Hypothesis H4 may be obtained (533).
• the additional labeling functions may be provided by a human user, may be obtained from a repository, or the like (553).
• the additional labeling functions may be obtained via a callback API (555). The call back request may be addressed to a user, to an external database, or the like.
• the number of labeling functions to request may be determined (557).
• the training dataset for the retraining based on Hypothesis H4 may be obtained (559).
• the traversal of Goal Graph 500 may be terminated when a hypothesis is selected.
• the hypothesis with highest improvement over the accuracy measurement of the classification task by compared to improvement gained by other hypotheses in Goal Graph 500 may be selected.
• Goal Graph 500 may be entirely traversed.
• a first hypothesis encountered during the traversal of the Goal Graph 500 that an application thereof improves the accuracy measurement of the classification task by at least the predetermined target goal may be selected.
• Hypotheses Graph 600 may be an exemplary hypotheses goal graph generated in response to an instruction to improve a classification model utilized for a supervised image classification task.
• Nodes 630-680 of Hypotheses Graph 500 may comprise hypotheses relating to strategies to potentially improve the classification model, such as strategies related to a model type of the classification model, strategies related to a training parameter of the classification model, strategies related to a modeling parameter of the classification model, strategies related to a modeling of the supervised image classification task, strategies related to a training set utilizes to train the classification model, strategies related to a validation set utilized to validate the classification model, strategies related to a test set utilized to test the classification model, or the like.
• Node 630 comprises an abstract Hypothesis H5 for improving the classification model.
• Hypothesis H5 may be related to a strategy of training of the classification model.
• Hypothesis H5 suggests training with noisy labels assumption.
• the training may be performed while K% of the labels of the samples are assumed to be wrong.
• Parameter K may be determined during the execution of Hypotheses Graph 600, by a user, based on the classification task requirement, or the like.
• parameter K may be 10, 20, 30, or the like.
• Node 640 comprises an abstract Hypothesis H6 for improving the classification model.
• Hypothesis H6 may be related to a strategy related to a training parameter of the classification model, such as what type of training.
• Hypothesis H6 suggests utilizing mixed supervised and unsupervised training. A determination whether training with tagged images and a large set of untagged images can provide a better classification model, may be performed. It may be appreciated that several concretizations may be performed on abstract Hypothesis H6 in order to evaluate the likelihood of improving the accuracy measurement thereby, such as the amount of untagged images to be utilized in the training, what percent of unsupervised learning to be performed, or the like.
• Node 650 comprises an abstract Hypothesis H7 for improving the classification model.
• Hypothesis H7 may be related to a strategy related to a model type of the classification model.
• Hypothesis H7 suggests utilizing an ensemble of classifiers to perform the classification task.
• the tested hypothesis may be whether multiple -judges voting on each sample can provide more stable results against outliers, novelty, anomaly, or the like.
• Node 660 comprises an abstract Hypothesis H8 for improving the classification model.
• Hypothesis H8 may be related to a strategy related to a modeling parameter of the classification model.
• Hypothesis H8 suggests clustering some classes in the supervised image classification task into several sub-classes.
• the tested hypothesis may be whether sub-patterns inside the classification categories that the user have selected can be found.
• Node 670 comprises an abstract Hypothesis H9 for improving the classification model.
• Hypothesis H9 may be related to a strategy related to a modeling of the supervised image classification task.
• Hypothesis H9 suggests utilizing a lower number of classes.
• the tested hypothesis may be whether unifying set of classes can get better classification results. In this hypothesis, multiple solutions can be proposed to the user to decide thereupon.
• Hypothesis H9 may transform the original learning task into another related learning task, with a slight difference in its formal definition.
• Node 680 comprises an abstract Hypothesis H10 for improving the classification model.
• Hypothesis H10 may be related to a strategy related to a modeling of the supervised image classification task. Hypothesis H10 suggests utilizing a greater number of classes.
• the tested hypothesis may be whether unifying more classes can provide better confusion matrix results.
• the present invention may be a system, a method, or a computer program product.
• the computer program product may include a computer readable storage medium having instructions thereon for causing a processor to carry out aspects of the present invention.
• the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device.
• the computer readable storage medium may be, for example, but is not limited to, electronic, magnetic optical storage, or the like, such as but not limited to a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), or the like.
• the instructions may be downloadable to the storage medium from a server, a remote computer, a remote storage, or the like.
• Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.
• the program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
• the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
• LAN local area network
• WAN wide area network
• Internet Service Provider an Internet Service Provider
• the computer readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions specified in the block of the diagrams.
• each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
• the functions noted in the block may occur out of the order noted in the figures.
• two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
• each block of the diagrams and combinations of blocks can be implemented by special purpose hardware -based systems.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• Evolutionary Computation (AREA)
• General Physics & Mathematics (AREA)
• Artificial Intelligence (AREA)
• Software Systems (AREA)
• Computing Systems (AREA)
• Data Mining & Analysis (AREA)
• Computer Vision & Pattern Recognition (AREA)
• General Health & Medical Sciences (AREA)
• Health & Medical Sciences (AREA)
• Databases & Information Systems (AREA)
• General Engineering & Computer Science (AREA)
• Medical Informatics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Multimedia (AREA)
• Mathematical Physics (AREA)
• Molecular Biology (AREA)
• Biomedical Technology (AREA)
• Biophysics (AREA)
• Computational Linguistics (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Bioinformatics & Computational Biology (AREA)
• Evolutionary Biology (AREA)
• Probability & Statistics with Applications (AREA)
• Algebra (AREA)
• Computational Mathematics (AREA)
• Mathematical Analysis (AREA)
• Mathematical Optimization (AREA)
• Pure & Applied Mathematics (AREA)
• Image Analysis (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=76210950

Family Applications (1)

Country Status (6)

Families Citing this family (7)

Citations (2)

Family Cites Families (11)

• 2020
  • 2020-11-08 KR KR1020227022704A patent/KR102684032B1/ko active Active
  • 2020-11-08 CN CN202080084010.1A patent/CN114787831B/zh active Active
  • 2020-11-08 JP JP2022533364A patent/JP7477608B2/ja active Active
  • 2020-11-08 WO PCT/IL2020/051157 patent/WO2021111431A1/en not_active Ceased
  • 2020-11-16 TW TW109139955A patent/TWI834011B/zh active
  • 2020-11-30 US US17/107,851 patent/US11640559B2/en active Active

Patent Citations (2)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events