WO2024063807A1

WO2024063807A1 - Machine learning classifiers using data augmentation

Info

Publication number: WO2024063807A1
Application number: PCT/US2023/016105
Authority: WO
Inventors: Jiangsheng Yu
Original assignee: Futurewei Technologies, Inc.
Priority date: 2023-03-23
Filing date: 2023-03-23
Publication date: 2024-03-28

Abstract

A method for training a machine learning (ML) model includes performing data augmentation on a data object from a training data set to generate a plurality of augmented data, sets, the training data set being associated with a. classifier of the machine learning model, generating a plurality of testing accuracies of the classifier based on the plurality of augmented data, sets and determining at least one testing accuracy of the plurality of testing accuracies is below a pre-configured threshold accuracy, the at least, one testing accuracy corresponding to an augmented data set of the plurality of augmented data sets, and supplementing the training data set with the augmented data set to generate a revised training data set.

Description

CONFIGURING MACHINE LEARNING CLASSIFIERS USING DATA AUGMENTATION TECHNICAL FIELD [0001] The present disclosure is related to machine learning networks and, more specifically, to configuring machine learning classifiers using data augmentation. BACKGROUND [0002] With successful applications of machine learning networks, the requirements for network size and data volume are rapidly increasing. Consequently, efficient training of those networks, especially in a distributed training environment, is particularly important. However, training the machine learning network classifier and ensuring the classifier is robust can be resource- intensive and challenging. SUMMARY [0003] Various examples are now described to introduce a selection of concepts in a simplified form, which are further described below in the detailed description. The Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. [0004] According to a first aspect of the present disclosure, there is provided a computer-implemented method for training a machine learning (ML) model. The method includes performing data augmentation on a data object from a training data set to generate a plurality of augmented data sets. The training data set is associated with a classifier of the machine learning model. A plurality of testing accuracies of the classifier is generated based on the plurality of augmented data sets. At least one testing accuracy of the plurality of testing accuracies is determined to be below a pre-configured threshold accuracy. The at least one testing accuracy corresponds to an augmented data set of the plurality of augmented data sets. The training data set is supplemented with the augmented data set to generate a revised training data set. [0005] In a first implementation form of the method according to the first aspect as such, the performing the data augmentation further includes detecting that the data object comprises image data, and selecting a plurality of data transformations from available data transformations associated with the image data. [0006] In a second implementation form of the method according to the first aspect as such or any implementation form of the first aspect, the performing the data augmentation further includes applying the plurality of data transformations to the image data to generate the plurality of augmented data sets. Each data transformation of the plurality of data transformations corresponds to a respective augmented data set of the plurality of augmented data sets. [0007] In a third implementation form of the method according to the first aspect as such or any implementation form of the first aspect, the plurality of data transformations include at least two of: image translation, image reflection, image rotation, image enlargement, image filtering, image color enhancement, image affine transformation, or image noise enhancement. [0008] In a fourth implementation form of the method according to the first aspect as such or any implementation form of the first aspect, the determining the plurality of testing accuracies further includes selecting an augmented data object from the augmented data set of the plurality of augmented data sets and applying the classifier to the augmented data object to determine a data type of the augmented data object. [0009] In a fifth implementation form of the method according to the first aspect as such or any implementation form of the first aspect, the at least one testing accuracy corresponding to the augmented data set of the plurality of augmented data sets is determined based on a match between the data type of the augmented data object and a data type of the data object. [0010] In a sixth implementation form of the method according to the first aspect as such or any implementation form of the first aspect, the classifier is re-trained using the revised training data set, with the supplementing the training data set. The re-training of the classifier is performed in a processing loop. The processing loop is exited when each testing accuracy of the plurality of testing accuracies is equal to or above the pre-configured threshold accuracy. [0011] In a seventh implementation form of the method according to the first aspect as such or any implementation form of the first aspect, a probability array with a plurality of characteristics of the classifier is generated. Each characteristic of the plurality of characteristics indicates a probability that a corresponding testing accuracy of the plurality of testing accuracies is above the pre-configured threshold accuracy. [0012] In an eighth implementation form of the method according to the first aspect as such or any implementation form of the first aspect, a comparison is performed between the plurality of characteristics of the classifier and a second plurality of characteristics in a second probability array associated with a second classifier. A determination to replace the classifier with the second classifier is made based on the comparison. [0013] In a ninth implementation form of the method according to the first aspect as such or any implementation form of the first aspect, a preliminary step of selecting the data object from the training data set is performed. [0014] In a tenth implementation form of the method according to the first aspect as such or any implementation form of the first aspect, the classifier uses the revised training data set. [0015] According to a second aspect of the present disclosure, there is provided an apparatus for training a machine learning (ML) model. The apparatus includes a memory storing instructions and at least one processor in communication with the memory. The at least one processor is configured, upon execution of the instructions, to perform the following steps listed herein. Data augmentation is performed on a data object from a training data set to generate a plurality of augmented data sets. The training data set is associated with a classifier of the machine learning model. A plurality of testing accuracies of the classifier is generated based on the plurality of augmented data sets. At least one testing accuracy of the plurality of testing accuracies is determined to be below a pre-configured threshold accuracy. The at least one testing accuracy corresponds to an augmented data set of the plurality of augmented data sets. The training data set is supplemented with augmented data from the augmented data set to generate a revised training data set. [0016] In a first implementation form of the apparatus according to the second aspect as such, the performing the data augmentation further includes detecting that the data object comprises image data and selecting a plurality of data transformations from available data transformations associated with the image data. [0017] In a second implementation form of the apparatus according to the second aspect as such or any implementation form of the second aspect, the performing of the data augmentation further includes applying the plurality of data transformations to the image data to generate the plurality of augmented data sets. Each data transformation of the plurality of data transformations corresponds to a respective augmented data set of the plurality of augmented data sets. [0018] In a third implementation form of the apparatus according to the second aspect as such or any implementation form of the second aspect, the plurality of data transformations includes at least two of: image translation, image reflection, image rotation, image enlargement, image filtering, image color enhancement, image affine transformation, or image noise enhancement. [0019] In a fourth implementation form of the apparatus according to the second aspect as such or any implementation form of the second aspect, the determining the plurality of testing accuracies further includes selecting an augmented data object from the augmented data set of the plurality of augmented data sets and applying the classifier to the augmented data object to determine a data type of the augmented data object. [0020] In a fifth implementation form of the apparatus according to the second aspect as such or any implementation form of the second aspect, the at least one processor further executes the instructions to perform the steps of determining the at least one testing accuracy corresponding to the augmented data set of the plurality of augmented data sets based on a match between the data type of the augmented data object and a data type of the data object. [0021] In a sixth implementation form of the apparatus according to the second aspect as such or any implementation form of the second aspect, the at least one processor further executes the instructions to perform the steps of re- training the classifier using the revised training data set, with the supplementing the training data set and the re-training of the classifier being performed in a processing loop. The steps further include exiting the processing loop when each testing accuracy of the plurality of testing accuracies is equal to or is above the pre-configured threshold accuracy. [0022] In a seventh implementation form of the apparatus according to the second aspect as such or any implementation form of the second aspect, the at least one processor further executes the instructions to perform the steps of generating a probability array with a plurality of characteristics of the classifier. Each characteristic of the plurality of characteristics indicates a probability that a corresponding testing accuracy of the plurality of testing accuracies is above the pre-configured threshold accuracy. [0023] In an eighth implementation form of the apparatus according to the second aspect as such or any implementation form of the second aspect, the at least one processor further executes the instructions to perform the steps of performing a comparison between the plurality of characteristics of the classifier and a second plurality of characteristics in a second probability array associated with a second classifier. The steps further include determining whether to replace the classifier with the second classifier based on the comparison. [0024] In a ninth implementation form of the apparatus according to the second aspect as such or any implementation form of the second aspect, the at least one processor further executes the instructions to perform a preliminary step of selecting the data object from the training data set. [0025] In a tenth implementation form of the apparatus according to the second aspect as such or any implementation form of the second aspect, the at least one processor further executes the instructions to perform the step of re- training the classifier using the revised training data set. [0026] According to a third aspect of the present disclosure, there is provided a non-transitory computer-readable media storing instructions for training a machine learning (ML) model. When executed by at least one processor, the instructions cause the at least one processor to perform the following steps including performing data augmentation on a data object from a training data set to generate a plurality of augmented data sets. The training data set is associated with a classifier of the machine learning model. The steps further include generating a plurality of testing accuracies of the classifier based on the plurality of augmented data sets and determining at least one testing accuracy of the plurality of testing accuracies below a pre-configured threshold accuracy. The at least one testing accuracy corresponds to an augmented data set of the plurality of augmented data sets. The steps further include supplementing the training data set with augmented data from the augmented data set to generate a revised training data set. [0027] In a first implementation form of the non-transitory computer- readable media according to the third aspect as such, the performing the data augmentation further includes detecting that the data object comprises image data, and selecting a plurality of data transformations from available data transformations associated with the image data. [0028] In a second implementation form of the non-transitory computer- readable media according to the third aspect as such or any implementation form of the third aspect, the performing the data augmentation further includes applying the plurality of data transformations to the image data to generate the plurality of augmented data sets. Each data transformation of the plurality of data transformations corresponds to a respective augmented data set of the plurality of augmented data sets. [0029] In a third implementation form of the non-transitory computer- readable media according to the third aspect as such or any implementation form of the third aspect, the plurality of data transformations includes at least two of: image translation, image reflection, image rotation, image enlargement, image filtering, image color enhancement, image affine transformation, or image noise enhancement. [0030] In a fourth implementation form of the non-transitory computer- readable media according to the third aspect as such or any implementation form of the third aspect, the determining the plurality of testing accuracies further includes selecting an augmented data object from the augmented data set of the plurality of augmented data sets and applying the classifier to the augmented data object to determine a data type of the augmented data object. [0031] In a fifth implementation form of the non-transitory computer- readable media according to the third aspect as such or any implementation form of the third aspect, the steps further include determining the at least one testing accuracy corresponding to the augmented data set of the plurality of augmented data sets based on a match between the data type of the augmented data object and a data type of the data object. [0032] In a sixth implementation form of the non-transitory computer- readable media according to the third aspect as such or any implementation form of the third aspect, the steps further include re-training the classifier using the revised training data set, with the supplementing the training data set and the re- training of the classifier being performed in a processing loop. The steps further include exiting the processing loop when each testing accuracy of the plurality of testing accuracies is equal to or is above the pre-configured threshold accuracy. [0033] In a seventh implementation form of the non-transitory computer- readable media according to the third aspect as such or any implementation form of the third aspect, the steps further include generating a probability array with a plurality of characteristics of the classifier. Each characteristic of the plurality of characteristics indicates a probability that a corresponding testing accuracy of the plurality of testing accuracies is above the pre-configured threshold accuracy. [0034] In an eighth implementation form of the non-transitory computer- readable media according to the third aspect as such or any implementation form of the third aspect, the steps further include performing a comparison between the plurality of characteristics of the classifier and a second plurality of characteristics in a second probability array associated with a second classifier. The steps further include determining whether to replace the classifier with the second classifier based on the comparison. [0035] In a ninth implementation form of the non-transitory computer- readable media according to the third aspect as such or any implementation form of the third aspect, the steps further include a preliminary step of selecting the data object from the training data set. [0036] In a tenth implementation form of the non-transitory computer- readable media according to the third aspect as such or any implementation form of the third aspect, the steps further include re-training the classifier using the revised training data set. [0037] According to a fourth aspect of the present disclosure, there is provided an apparatus for training a machine learning (ML) model. The apparatus includes means for performing data augmentation on a data object from a training data set to generate a plurality of augmented data sets. The training data set is associated with a classifier of the machine learning model. The apparatus further includes means for generating a plurality of testing accuracies of the classifier based on the plurality of augmented data sets. The apparatus further includes means for determining at least one testing accuracy of the plurality of testing accuracies is below a pre-configured threshold accuracy. The at least one testing accuracy corresponds to an augmented data set of the plurality of augmented data sets. The apparatus further includes means for supplementing the training data set with augmented data from the augmented data set to generate a revised training data set. [0038] Any of the foregoing examples may be combined with any one or more of the other foregoing examples to create a new embodiment within the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS [0039] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document. [0040] FIG. 1 is a diagram of a system for generating a trained machine learning model including a trained classifier, according to example embodiments. [0041] FIG. 2 is a diagram of the generation of a trained machine learning model within a deep learning architecture (DLA) associated with the system of FIG. 1, according to example embodiments. [0042] FIG. 3A, FIG. 3B, FIG. 3C, and FIG. 3D illustrate data augmentation techniques for image data, according to example embodiments. [0043] FIG. 4 and FIG. 5 illustrate data augmentation techniques for text data, according to example embodiments. [0044] FIG. 6 illustrates the sampling of augmented data in connection with a classifier training, according to example embodiments. [0045] FIG. 7 is a radar chart of distinguishing abilities associated with a classifier of a machine learning network, according to example embodiments. [0046] FIG. 8 is a flowchart of a method for training a machine learning model, according to example embodiments. [0047] FIG. 9 is a diagram of a representative software architecture, which may be used in conjunction with various device hardware described herein, according to example embodiments. [0048] FIG. 10 is a diagram of circuitry for a device that implements algorithms and performs methods, according to example embodiments. DETAILED DESCRIPTION [0049] It should be understood at the outset that although an illustrative implementation of one or more embodiments is provided below, the disclosed systems and methods described with respect to FIGS. 1-10 may be implemented using any number of techniques, whether currently known or not yet in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents. [0050] In the following description, reference is made to the accompanying drawings that form a part hereof, and which are shown, by way of illustration, specific embodiments which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the inventive subject matter, and it is to be understood that other embodiments may be utilized, and that structural, logical, and electrical changes may be made without departing from the scope of the present disclosure. The following description of example embodiments is, therefore, not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims. [0051] As used herein, the term “classifier” indicates a classification algorithm used by a trained machine learning (ML) model. In machine learning networks, a classification algorithm can be used when the outputs are restricted to a limited set of values (e.g., text categorization, image classification, etc.). In machine learning, a “classifier” can also be referred to as an “identifier”. [0052] As used herein, the term “data object” (or “object”) indicates image data, text data, or a combination thereof. [0053] As used herein, the term “data augmentation” indicates one or more data transformations that can be applied to the data object to generate augmented data. For example, the following data transformations can be applied to image data: image translation, image reflection, image rotation, image enlargement, image filtering, image color enhancement, image affine transformation, and image noise enhancement. Example data transformations that can be performed on text data include replacing one or more words in the text data with a synonym, and translating the text data from an original language (e.g., English) to a second language (e.g., French) and then translating back to the original language. [0054] In a machine learning network, configuring a robust classifier can be challenging. For example, a slight change in a testing object (e.g., image data used as input to a trained classifier) may not change the object category often causes the classifier to crash and/or generate an incorrect output. Moreover, the classifier may not be aware of its weakness and would not be able to initiate active learning and re-training by itself. For example, an image recognition classifier can be applied to an image of a panda, which can result in panda detection with a higher than 50% confidence. However, if noise is applied to the image without visibly distorting it, the same classifier can detect a different animal (e.g., a gibbon) with a higher than 90% confidence. [0055] Techniques disclosed herein can be used to configure a classifier training module (CTM) to assess the capabilities of a classifier in a machine learning network and perform additional training (or replacement) of the classifier based on an augmented training data set. For example, the CTM can generate additional training data based on data augmentation. The CTM can also assess the classifier’s performance, augment the training data, and re-train the classifier using the augmented training data until a desired performance is achieved. In some aspects, data augmentation is used to generate additional training data and initiate additional training (or re-training) of the machine learning network classifier. In this regard, the CTM can determine the classifier’s weakness based on the performance of classification on the augmented data of distinct types, to strengthen the training in a particular classification application. In some embodiments, using the disclosed techniques can be used to increase the efficiency and accuracy of classifiers used in ML models. [0056] Existing prior art techniques for selecting and training a classifier do not use training data augmentation (e.g., via data transformation techniques) for classifier assessment and re-training. [0057] FIG. 1 is a diagram of a system 100 for generating a trained machine learning model including a trained classifier, according to example embodiments. Referring to FIG. 1, system 100 includes a computing device 107, which receives as input training data 102 or new data 114 and generates assessments 116 as output. The computing device 107 is configured to execute (e.g., as one or more software modules as part of an application or the device operating system) a machine learning (ML) architecture (MLA) 106 with a machine learning (ML) model 109 (also referred to as a convolutional graph neural network, or graph GNN). The MLA 106 can perform ML model training 108 to train the ML model 109 and generate the trained ML model 110 (e.g., using the ML model 109, as illustrated in FIG. 2). The training data 102 includes training input data 104 and training desired output data 118 that can be used during the DL model training 108. [0058] In some embodiments, the trained ML model 110 includes a classifier 112, which can be used to assess new data 114 and generate assessments 116 (e.g., when the trained ML model 110 is applied to the new data 114). In some aspects, the trained ML model 110 can be stored in a storage location (not illustrated in FIG. 1) of computing device 107 (e.g., device memory). [0059] Deep learning is part of machine learning, which is a field of study that gives computers the ability to learn without being explicitly programmed. Machine learning explores the study and construction of algorithms, also referred to herein as tools, that may learn from existing data, may correlate data, and may make predictions about new data. Such machine learning tools operate by building a model from example training data (e.g., training data 102) to make data-driven predictions or decisions expressed as outputs or assessments 116. Although example embodiments are presented with respect to a few machine-learning tools (e.g., a deep learning training architecture), the principles presented herein may be applied to other machine- learning tools. [0060] In some example embodiments, different machine-learning tools may be used. For example, Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), matrix factorization, and Support Vector Machines (SVM) tools may be used during the program training process (e.g., DL model training 108 for correlating the training data 102 and generating the trained DL model 110). [0061] Two common types of problems in machine learning are classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several categories using a classifier of a trained ML model (e.g., classifier 112 of trained ML model 110). Regression algorithms aim at quantifying some items (for example, by providing a value that is a real number). In some embodiments, the MLA 106 can be configured to use machine learning algorithms that utilize the training data 102 to find correlations among identified features that affect the outcome. [0062] The machine learning algorithms utilize features from the training data 102 for analyzing the new data 114 and generating the assessments 116. The features include individual measurable properties of a phenomenon being observed and used for training the ML program. The concept of a feature is related to that of an explanatory variable used in statistical techniques such as linear regression. Choosing informative, discriminating, and independent features are important for the effective operation of the ML program in pattern recognition, classification, and regression. Features may be of different types, such as numeric features, strings, and graphs. In some aspects, training data can be of different types, with the features being numeric for use by a computing device. [0063] During a training (or learning) phase, the ML model 109 is developed against the training dataset of inputs to optimize the model to correctly predict the target output (e.g., training desired output data 118) for a given input (e.g., training input data 104). Generally, the training phase may be supervised, semi-supervised, or unsupervised; indicating a decreasing level to which the “correct” outputs are provided in correspondence to the training inputs. The training phase also includes the training of one or more classifiers (e.g., classifier 112) of the ML model. In a supervised learning phase, all of the target outputs are provided to the model and the model is directed to develop a general rule or algorithm that maps the input to the output. In contrast, in an unsupervised learning phase, the desired output is not provided for the inputs so that the model may develop its own rules to discover relationships within the training dataset. In a semi-supervised learning phase, an incompletely labeled training set is provided, with some of the outputs known and with some of the outputs unknown for the training dataset. [0064] An ML model may be run against a training dataset for several epochs, in which the training dataset is repeatedly fed into the model to refine its results (i.e., the entire dataset is processed during an epoch). During an iteration, the ML model is run against a mini-batch (or a portion) of the entire dataset. In a supervised learning phase, a model is developed to predict the target output for a given set of inputs (e.g., training data 102) and is evaluated over several epochs to more reliably provide the output that is specified as corresponding to the given input for the greatest number of inputs for the training dataset. In another example, for an unsupervised learning phase, a model is developed to cluster the dataset into n groups and is evaluated over several epochs as to how consistently it places a given input into a given group and how reliably it produces the n desired clusters across each epoch. [0065] Once an epoch is run, the ML model is evaluated and the values of its variables (e.g., weights, biases, or other parameters) are adjusted to attempt to better refine the ML model iteratively. As used herein, the term “weights” is used to refer to the parameters used by a machine learning model. The weights are values used by individual nodes and affect a signal or data as the data passes through the node during the processing of the data in the machine-learning model. During a backward computation, a neural network (NN) model can output gradients, which can be used for updating weights associated with a forward computation. [0066] Each model refines the values of its nodes or layer of nodes over several epochs by varying the values of one or more variables, affecting the inputs to more closely map to the desired result. But, as the training dataset may be varied and is preferably very large, perfect accuracy and precision may not be achievable. Several epochs that make up a learning phase, therefore, may be set as a given number of trials, may be set as a fixed time/computing budget, or may be terminated before that number/budget is reached when the accuracy of a given model is high enough or low enough or an accuracy plateau has been reached. For example, if the training phase is designed to run n epochs and produce a model with at least 95% accuracy, and such a model is produced before the nth epoch, the learning phase may end early and use the produced model, as it satisfies the end-goal accuracy threshold. Similarly, if a given model is inaccurate enough to satisfy a random chance threshold (e.g., the model is only 55% accurate in determining true/false outputs for given inputs), the learning phase for that model may be terminated early, although other models in the learning phase may continue training. Similarly, when a given model continues to provide similar accuracy or vacillate in its results across multiple epochs – having reached a performance plateau – the learning phase for the given model may terminate before the epoch number/computing budget is reached. [0067] Once the learning phase is complete, a trained model is generated based on the final weights that produce results close to the training desired output data. In example embodiments, models that have been finalized are evaluated against testing criteria. In a first example, a testing dataset (that includes known target outputs for its inputs) is fed into the finalized models to determine the accuracy of the model in handling data that the model has not been trained on. In a second example, a false positive rate or false-negative rate may be used to evaluate the models after finalization. In a third example, a delineation between data clusters in each model is used to select a model that produces the clearest bounds for its clusters of data. [0068] During the training of a DL model, a regression, which is structured as a set of statistical processes for estimating the relationships among variables, can include the minimization of a cost function. The cost function may be implemented as a function to return a number representing how well the neural network performed in mapping training examples to correct output. In training, if the cost function value is not within a predetermined range, based on the known training images, backpropagation is used. Backpropagation is a common method of training artificial neural networks that are used with an optimization method such as the stochastic gradient descent (SGD) method. [0069] The use of backpropagation can include propagation and weight updates. When an input is presented to the neural network, it is propagated forward through the neural network, layer by layer, until it reaches the output layer. The output of the neural network is then compared to the desired target output, using the cost function, and an error value is calculated for each of the nodes in the output layer. The error values are propagated backward, starting from the output, until each node has an associated error value that roughly represents its contribution to the original output. Backpropagation can use these error values to calculate the gradient of the cost function concerning the weights in the neural network. The calculated gradient is fed to the selected optimization method to update the weights and to attempt to minimize the cost function. [0070] Referring again to FIG. 1, during an example training phase, the MLA 106 receives the training data 102, which includes training input data 104 and training desired output data 118 and initiates the training of the ML model 109. The training input data 104 is run through the convolutional layers of the ML model 109 and is changed by the ML model 109 according to current node values in the convolutional layers of the ML model 109 (a more detailed view of the ML model 109 is illustrated in FIG. 2). The output of the ML model 109 is compared to the training desired output data 118, and the differences between the target output values and the current output values are fed back and are used to modify the current node values of the ML model 109. This is an iterative process, where the system converges on a solution over time as the node values eventually produce the target output values when the training data is run through the ML model 109. The final node values are then used to generate the trained ML model 110 with classifier 112. [0071] In example embodiments, different machine learning tools/models may be used during the deep learning model training 108. In example embodiments, during a new data processing operation, the trained ML model 110 utilizes features from the training data 102 for analyzing the new data 114 using the classifier 112, resulting in the generation of assessments 116 as an output. The features include, for example, individual measurable properties of a phenomenon being observed and used for training the machine learning model. Choosing informative, discriminating, and independent features are important for the effective operation of MLA 106 in pattern recognition, classification, and regression. Features may be of different types, such as numeric features, strings, and graphs. In some aspects, training data can be of different types, with the features being numeric for use by a computing device. [0072] In some embodiments, the MLA 106 includes a classifier training module (CTM) 120. CTM 120 includes suitable circuitry, interfaces, and/or code and is configured to perform classifier training functionalities discussed herein. More specifically, CTM 120 is configured to select a data object from a training data set (e.g., training data 102) associated with classifier 112, and perform data augmentation on the data object to generate a plurality of augmented data sets. The CTM 120 further determines a plurality of testing accuracies of the classifier based on the plurality of augmented data sets. The CTM 120 is configured to detect that at least one testing accuracy of the plurality of testing accuracies is below a pre-configured threshold accuracy, where the at least one testing accuracy corresponds to an augmented data set of the plurality of augmented data sets. The CTM 120 is configured to supplement the training data set with augmented data from the augmented data set to generate a revised training data set and re-train classifier 112 using the revised training data set. Example techniques for generating the plurality of augmented data sets are discussed in connection with FIG. 3A-FIG. 5. Example additional functionalities of the CTM 120 are discussed further in connection with FIG. 6- FIG. 10. [0073] FIG. 2 is diagram 200 of the generation of a trained ML model 110 within the MLA 106 associated with the system of FIG. 1, according to example embodiments. Referring to FIG. 2, source data 202 is analyzed by the convolutional layers 204 of the ML model 109 (or another type of machine learning algorithm or technique) to generate the trained ML model 110 (which also includes the trained classifier 112). The source data 202 can include a training set of data, such as training data 102, including data identified by one or more features. [0074] In example embodiments, the ML model 110 is trained by a neural network model (e.g., deep learning, deep convolutional, or recurrent neural network), which comprises a series of “neurons,” such as Long Short- Term Memory (LSTM) nodes, arranged into a network. A neuron is an architectural element used in data processing and artificial intelligence, particularly machine learning, that includes a memory that may determine when to “remember” and when to “forget” values held in that memory based on the weights of inputs provided to the given neuron. Each of the neurons used herein is configured to accept a predefined number of inputs from other neurons in the network to provide relational and sub-relational outputs for the content of the frames being analyzed. Individual neurons may be chained together or organized into tree structures in various configurations of neural networks to provide interactions and relationship-learning modeling for how each of the frames in an utterance is related to one another. [0075] For example, an LSTM node serving as a neuron includes several gates to handle input vectors (e.g., phonemes from an utterance), a memory cell, and an output vector (e.g., contextual representation). The input gate and output gate control the information flowing into and out of the memory cell, respectively, whereas forget gates optionally remove information from the memory cell based on the inputs from linked cells earlier in the neural network. Weights and bias vectors for the various gates are adjusted throughout a training phase, and once the training phase is complete, those weights and biases are finalized for normal operation. One of skill in the art will appreciate that neurons and neural networks may be constructed programmatically (e.g., via software instructions) or via specialized hardware linking each neuron to form the neural network. [0076] Even though the MLA 106 is referred to as a machine learning architecture using an ML model 109 (and the model that is generated as a result of the training is referred to as a machine learning model, such as the trained ML model 110), the disclosure is not limited in this regard and other types of machine learning training architectures may also be used for model training, using the techniques disclosed herein. [0077] FIG. 3A, FIG. 3B, FIG. 3C, and FIG. 3D illustrate data augmentation techniques for image data, according to example embodiments. For example, FIG. 3A illustrates image translation 302 where the original image is used to generate the translated image. FIG. 3B illustrates image reflection 304 where the reflected image is generated from the original image. FIG. 3C illustrates image rotation 306 where the rotated image is generated from the original image. FIG. 3D illustrates image enlargement 308 where the enlarged image is generated from the original image. In some aspects, the CTM 120 can apply the data augmentation techniques of FIGS. 3A-3C to perform data augmentation on a data object that includes image data to generate one or more augmented data sets. [0078] FIG. 4 and FIG. 5 illustrate diagrams 400 and 500 of data augmentation techniques for text data, according to example embodiments. Referring to FIG. 4, data augmentation 404 is applied to text data 402 to generate augmented data 406. More specifically, data augmentation 404 is based on substituting at least one word (or phrase) in the text data with a synonym (e.g., the phrase “very cool” is substituted with one of the following synonyms “pretty cool,” “really cool,” “super cool,” “kinda cool,” or “quite cool”). [0079] Referring to FIG. 5, data augmentation in diagram 500 is based on translating text data from a first language to a second language and then translating back from the second language to the first language. For example, original text data includes sentence 502, which is translated from English into French to obtain sentence 504. Sentence 504 is translated from French back into English to obtain augmented text data in the form of sentence 506. [0080] FIG. 6 illustrates a diagram 600 of sampling augmented data in connection with a classifier training, according to example embodiments. Referring to FIG. 6, CTM 120 is configured to perform data augmentation 601 by applying one or more of the data augmentation techniques discussed herein to training data 102 and generating a plurality of augmented data sets. More specifically, the data augmentation techniques can include N kinds of data transformations, which when applied to object x in category X, generate a total of N types of augmented data - augmented data set 602 (including augmented data of type 1), augmented data set 604 (including augmented data of type 2), …, and augmented data set 606 (including augmented data of type N). All of the augmented data sets 602, 604, …, 606 include data in the same category X as the original object x in training data 102. For example, object x in category X can include a particular image in category image data. The different types of data augmentations can include data transformation techniques for image data (e.g., as illustrated in FIGS. 3A-3C), including flipping, rotation, cropping, scaling, adding noise, etc. [0081] In some embodiments, CTM 120 can determine one or more distinguishing characteristics (or abilities) of classifier 112 (also referred to as classifier C). For example, the distinguishing abilities of classifier C on the augmented testing data of object x are described by an array (p1(x), p2(x), …, p_n(x)), where p₁, p₂, …, p_n are the testing accuracies of the classifier associated with augmented data of type 1, type 2, …, type N_. [0082] As used herein, the term “testing accuracy” for a specific augmented data set (e.g., testing accuracy p_k) is the accuracy of a classifier (e.g., measured in percentage with a maximum being 100%) when classifying a data object from an augmented data set of type k as being object x in category X. [0083] If a testing accuracy (e.g., p_k) is less than a pre-configured threshold accuracy T, then one or more data objects of the augmented data of type k will be added into the training data 102 to upgrade the classifier C (e.g., the classifier can be re-trained after the training data set is supplemented/augmented). In some aspects, the above procedure of supplementing the training data with augmented data of a specific type is repeated until all testing accuracies p₁, p₂, …, p_n of classifier C are greater than the threshold accuracy T. [0084] In some embodiments, CTM 120 can determine the distinguishing abilities of classifier C on category X. For example, CTM 120 can determine the distinguishing abilities of the classifier C on category X by determining a probability array (P₁, P₂, …, P_n), where P_k is the probability of p_k(x)>T for any x in X, T is a pre-configured threshold probability, and k = 1, 2, …, n. [0085] In some embodiments, CTM 120 can determine the distinguishing abilities of multiple classifiers as indicated by probability arrays (P₁, P₂, …, P_n) on data object category X, and use such distinguishing abilities to perform a comparison between ML models (and a selection of one of the ML models and the associated classifier). In some aspects, the distinguishing abilities of a classifier as indicated by probability array (P₁, P₂, …, P_n) can be used to generate a radar chart (e.g., as illustrated in FIG. 7). [0086] FIG. 7 is a radar chart 700 of distinguishing abilities associated with a classifier of a machine learning network, according to example embodiments. Referring to FIG. 7, radar chart 700 indicates the probabilities P₁, P2, …, Pn associated with the probability array (P1, P2, …, Pn) of classifier C in relation to data objects of category X. In some embodiments, CTM 120 can determine the probability arrays for multiple classifiers and select one based on a desired probability value (or values) in the probability array. Such selection of a classifier can be performed before using an ML network or can be performed dynamically (e.g., during the use of the ML network). [0087] In an exemplary processing sequence, X=“horse” is a category that classifier C can identify. After the testing on the augmented data, CTM 120 determines that the distinguishing ability of image rotation is weak (e.g., testing accuracy is below a threshold) and further training of classifier C on this augmented data type can be initiated. If classifier C is trained using the disclosed techniques associated with data augmentation, its convergent distinguishing abilities on the category of “horse” describe its limitation of identifying a “horse” in practice. In some aspects, given a testing image of a horse, the augmented data sets are generated. If classifier C identifies most of the data objects in the augmented data set as a “horse”, then its category is determined as “horse”. [0088] FIG. 8 is a flowchart of method 800 for training a machine learning model, according to example embodiments. Method 800 includes operations 802, 804, 806, and 808. By way of example and not limitation, method 800 is described as being performed by the CTM 120, which can be configured to execute within a mobile device such as device 1000 illustrated in FIG. 10. [0089] At operation 802, data augmentation is performed on a data object from a training data set to generate a plurality of augmented data sets. The training data set can be associated with a classifier of the machine learning model. For example, CTM 120 selects object x from training data 102. The training data set is associated with a classifier of a machine learning model (e.g., classifier 112 of trained ML model 110). CTM 120 generates augmented data sets 602, 604, …, 606 using N different types of data augmentation techniques. [0090] At operation 804, a plurality of testing accuracies of the classifier is generated based on the plurality of augmented data sets. A determination is made that at least one testing accuracy of the plurality of testing accuracies is below a pre-configured threshold accuracy. The at least one testing accuracy corresponds to an augmented data set of the plurality of augmented data sets. [0091] For example, a plurality of testing accuracies of the classifier is determined based on the plurality of augmented data sets. For example, the distinguishing abilities of classifier 112 on the augmented testing data of object x are described by an array (p₁(x), p₂(x), …, p_n(x)), where p₁, p₂, …, p_n are the testing accuracies of the classifier associated with augmented data of type 1, type 2, …, type N_. The at least one testing accuracy p_k is the accuracy of classifier 112 when classifying a data object from an augmented data set of type k as being object x in category X. In this regard, the at least one testing accuracy p_k corresponds to an augmented data set of the plurality of augmented data sets. [0092] At operation 806, the training data set is supplemented with the augmented data set to generate a revised training data set. For example, training data 102 is supplemented with augmented data from augmented data set of type k. In some aspects, the classifier is re-trained using the revised training data set. [0093] In some embodiments, performing the data augmentation includes detecting the data object includes image data, and selecting a plurality of data transformations from available data transformations associated with the image data. [0094] In some embodiments, the plurality of data transformations is applied to the image data to generate the plurality of augmented data sets. Each data transformation of the plurality of data transformations corresponds to a respective augmented data set of the plurality of augmented data sets. [0095] In some embodiments, the plurality of data transformations includes at least two of the following transformations: image translation, image reflection, image rotation, image enlargement, image filtering, image color enhancement, image affine transformation, and image noise enhancement. [0096] In some embodiments, determining the plurality of testing accuracies further includes selecting an augmented data object from the augmented data set of the plurality of augmented data sets, and applying the classifier to the augmented data object to determine a data type of the augmented data object. [0097] In some embodiments, the at least one testing accuracy corresponding to the augmented data set of the plurality of augmented data sets is determined based on a match between the data type of the augmented data object and a data type of the data object. [0098] In some embodiments, the supplementing of the training data set and the re-training of the classifier are performed in a processing loop. In some aspects, the processing loop can be exited from when each of the plurality of testing accuracies is equal to or is above, the pre-configured threshold accuracy. [0099] In some embodiments, a probability array with a plurality of characteristics of the classifier is generated. In some aspects, each characteristic of the plurality of characteristics indicates a probability that a corresponding testing accuracy of the plurality of testing accuracies is above the pre-configured threshold accuracy. [0100] In some aspects, a comparison is performed between the plurality of characteristics of the classifier and a second plurality of characteristics in a second probability array associated with a second classifier. A determination can be made to replace the classifier with the second classifier based on the comparison. [0101] FIG. 9 is a diagram of a representative software architecture 900, which may be used in conjunction with various device hardware described herein, according to example embodiments. FIG. 9 is merely a non-limiting example of a software architecture 902 and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture 902 executes on hardware, such as computing device 107 in FIG. 1 which can be the same as device 1000 of FIG. 10 that includes, among other things, processor 1005, memory 1010, storage 1015 and/or 1020, and I/O interfaces 1025 and 1030. [0102] A representative hardware layer 904 is illustrated and can represent, for example, the device 1000 of FIG. 10. The representative hardware layer 904 comprises one or more processing units 906 having associated executable instructions 908. Executable instructions 908 represent the executable instructions of the software architecture 902, including the implementation of the methods, modules, and so forth of FIGS. 1-8. Hardware layer 904 also includes memory or storage modules 910, which also have executable instructions 908. Hardware layer 904 may also comprise other hardware 912, which represents any other hardware of the hardware layer 904, such as the other hardware illustrated as part of device 1000. [0103] In the example architecture of FIG. 9, software architecture 902 may be conceptualized as a stack of layers where each layer provides particular functionality. For example, software architecture 902 may include layers such as an operating system 914, libraries 916, frameworks/middleware 918, applications 920, and presentation layer 944. Operationally, the applications 920 or other components within the layers may invoke application programming interface (API) calls 924 through the software stack and receive a response, returned values, and so forth illustrated as messages 926 in response to the API calls 924. The layers illustrated in FIG. 9 are representative and not all software architectures 902 have all layers. For example, some mobile or special-purpose operating systems may not provide frameworks/middleware 918, while others may provide such a layer. Other software architectures may include additional or different layers. [0104] The operating system 914 may manage hardware resources and provide common services. The operating system 914 may include, for example, a kernel 928, services 930, and drivers 932. The kernel 928 may act as an abstraction layer between the hardware and the other software layers. For example, kernel 928 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and so on. Services 930 may provide other common services for the other software layers. Drivers 932 may be responsible for controlling or interfacing with the underlying hardware. For instance, drivers 932 may include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audio drivers, power management drivers, and so forth, depending on the hardware configuration. [0105] The libraries 916 may provide a common infrastructure that may be utilized by the applications 920 or other components or layers. The libraries 916 typically provide functionality that allows other software modules to perform tasks more easily than to interface directly with the underlying operating system 914 functionality (e.g., kernel 928, services 930, or drivers 932). The libraries 916 may include system libraries 934 (e.g., C standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like. In addition, libraries 916 may include API libraries 936 such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., an OpenGL framework that may be used to render 2D and 3D in a graphic content on a display), database libraries (e.g., SQLite that may provide various relational database functions), web libraries (e.g., WebKit that may provide web browsing functionality), and the like. Libraries 916 may also include a wide variety of other libraries 938 to provide many other APIs to the applications 920 and other software components/modules. [0106] The frameworks/middleware 918 (also sometimes referred to as middleware) may provide a higher-level common infrastructure that may be utilized by the applications 920 or other software components/modules. For example, the frameworks/middleware 918 may provide various graphical user interface (GUI) functions, high-level resource management, high-level location services, and so forth. The frameworks/middleware 918 may provide a broad spectrum of other APIs that may be utilized by the applications 920 or other software components/modules, some of which may be specific to a particular operating system 914 or platform. [0107] The applications 920 include built-in applications 940, third-party applications 942, and CTM 960. In some aspects, the CTM 960 comprises suitable circuitry, logic, interfaces, or code and can be configured to perform one or more of the classifier configuration functions performed by the CTM 120 and discussed in connection with FIGS. 1-8. [0108] Examples of representative built-in applications 940 may include but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, or a game application. Third-party applications 942 may include any of the built-in applications 940 as well as a broad assortment of other applications. In a specific example, the third-party application 942 (e.g., an application developed using the Android™ or iOS™ software development kit (SDK) by an entity other than the vendor of the particular platform) may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or other mobile operating systems. In this example, the third-party application 942 may invoke the API calls 924 provided by the mobile operating system such as operating system 914 to facilitate the functionality described herein. [0109] The applications 920 may utilize built-in operating system functions (e.g., kernel 928, services 930, and drivers 932), libraries (e.g., system libraries 934, API libraries 936, and other libraries 938), and frameworks/middleware 918 to create user interfaces to interact with users of the system. Alternatively, or additionally, in some systems, interactions with a user may occur through a presentation layer, such as presentation layer 944. In these systems, the application/module "logic" can be separated from the aspects of the application/module that interact with a user. [0110] Some software architectures utilize virtual machines. In the example of FIG. 9, this is illustrated by virtual machine 948. A virtual machine creates a software environment where applications/modules can execute as if they were executing on a hardware machine (such as the device 1000 of FIG. 10, for example). A virtual machine 948 is hosted by a host operating system (e.g., operating system 914) and typically, although not always, has a virtual machine monitor 946, which manages the operation of the virtual machine 948 as well as the interface with the host operating system (i.e., operating system 914). The software architecture 902 executes within the virtual machine 948 such as an operating system 950, libraries 952, frameworks/middleware 954, applications 956, or presentation layer 958. These layers of software architecture executing within the virtual machine 948 can be the same as the corresponding layers previously described or may be different. [0111] FIG. 10 is a diagram of circuitry for a device that implements algorithms and performs methods, according to example embodiments. All components need not be used in various embodiments. For example, clients, servers, and cloud-based network devices may each use a different set of components, or in the case of servers, larger storage devices. [0112] One example computing device in the form of a computer 1000 (also referred to as computing device 1000, computer system 1000, or computer 1000) may include a processor 1005, memory 1010, removable storage 1015, non-removable storage 1020, input interface 1025, the output interface 1030, and communication interface 1035, all connected by a bus 1040. Although the example computing device is illustrated and described as the computer 1000, the computing device may be in different forms in different embodiments. [0113] Memory 1010 may include volatile memory 1045 and non- volatile memory 1050 and may store a program 1055. The computing device 1000 may include – or have access to a computing environment that includes – a variety of computer-readable media, such as the volatile memory 1045, the non- volatile memory 1050, the removable storage 1015, and the non-removable storage 1020. Computer storage includes random-access memory (RAM), read- only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions. [0114] Computer-readable instructions stored on a computer-readable medium (e.g., the program 1055 stored in memory 1010) are executable by the processor 1005 of the computing device 1000. A hard drive, CD-ROM, or RAM are some examples of articles including a non-transitory computer-readable medium such as a storage device. The terms “computer-readable medium” and “storage device” do not include carrier waves to the extent that carrier waves are deemed too transitory. “Computer-readable non-transitory media” includes all types of computer-readable media, including magnetic storage media, optical storage media, flash media, and solid-state storage media. It should be understood that software can be installed in and sold with a computer. Alternatively, the software can be obtained and loaded into the computer, including obtaining the software through a physical medium or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator. The software can be stored on a server for distribution over the Internet, for example. As used herein, the terms “computer-readable medium” and “machine-readable medium” are interchangeable. [0115] The program 1055 may utilize a CTM 1060, which can be configured to perform one or more of the classifier configuration functions performed by the CTM 120 and discussed in connection with FIGS. 1-8. [0116] Any one or more of the modules described herein may be implemented using hardware (e.g., a processor of a machine, an application- specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any suitable combination thereof). Moreover, any two or more of these modules may be combined into a single module, and the functions described herein for a single module may be subdivided among multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices. [0117] In some aspects, the CTM 1060 as well as one or more other modules that are part of the program 1055, can be integrated as a single module, performing the corresponding functions of the integrated modules. [0118] Although a few embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from the described systems. Other embodiments may be within the scope of the following claims. [0119] It should be further understood that software including one or more computer-executable instructions that facilitate processing and operations as described above regarding any one or all of the steps of the disclosure can be installed in and sold with one or more computing devices consistent with the disclosure. Alternatively, the software can be obtained and loaded into one or more computing devices, including obtaining the software through a physical medium or distribution system, including, for example, from a server owned by the software creator or from a server not owned but used by the software creator. The software can be stored on a server for distribution over the Internet, for example. [0120] Also, it will be understood by one skilled in the art that this disclosure is not limited in its application to the details of construction and the arrangement of components outlined in the description or illustrated in the drawings. The embodiments herein are capable of other embodiments and capable of being practiced or carried out in various ways. Also, it will be understood that the phraseology and terminology used herein are for description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless limited otherwise, the terms “connected,” “coupled,” and “mounted,” and variations thereof herein are used broadly and encompass direct and indirect connections, couplings, and mountings. In addition, the terms “connected” and “coupled,” and variations thereof, are not restricted to physical or mechanical connections or couplings. Further, terms such as up, down, bottom, and top are relative, and are employed to aid illustration, but are not limiting. [0121] The components of the illustrative devices, systems, and methods employed by the illustrated embodiments can be implemented, at least in part, in digital electronic circuitry, analog electronic circuitry, computer hardware, firmware, software, or in combinations of them. These components can be implemented, for example, as a computer program product such as a computer program, program code, or computer instructions tangibly embodied in an information carrier, or a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus such as a programmable processor, a computer, or multiple computers. [0122] A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other units suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or multiple computers at one site or distributed across multiple sites and interconnected by a communication network. Also, functional programs, codes, and code segments for accomplishing the techniques described herein can be easily construed as within the scope of the claims by programmers skilled in the art to which the techniques described herein pertain. Method steps associated with the illustrative embodiments can be performed by one or more programmable processors executing a computer program, code, or instructions to perform functions (e.g., by operating on input data or generating an output). Method steps can also be performed by (and the apparatus for performing the methods can be implemented as) special-purpose logic circuitry, e.g., an FPGA (field- programmable gate array) or an ASIC (application-specific integrated circuit), for example. [0123] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. [0124] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The required elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example, semiconductor memory devices, e.g., electrically programmable read-only memory or ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, or data storage disks (e.g., magnetic disks, internal hard disks, or removable disks, magneto-optical disks, or CD-ROM and DVD-ROM disks). The processor and the memory can be supplemented by or incorporated into special-purpose logic circuitry. [0125] Those with skill in the art understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof. [0126] As used herein, “machine-readable medium” (or “computer- readable medium”) comprises a device able to store instructions and data temporarily or permanently and may include, but is not limited to, random- access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., Erasable Programmable Read-Only Memory (EEPROM)), or any suitable combination thereof. The term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store processor instructions. The term “machine-readable medium” shall also be taken to include any medium or a combination of multiple media, that is capable of storing instructions for execution by one or more processors, such that the instructions, when executed by one or more processors, cause the one or more processors to perform any one or more of the methodologies described herein. Accordingly, a “machine- readable medium” refers to a single storage apparatus or device, as well as “cloud-based” storage systems or storage networks that include multiple storage apparatus or devices. The term “machine-readable medium” as used herein excludes signals per se. [0127] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the scope disclosed herein. [0128] Although the present disclosure has been described concerning specific features and embodiments thereof, it is evident that various modifications and combinations can be made thereto without departing from the scope of the disclosure. For example, other components may be added to, or removed from the described systems. The specification and drawings are, accordingly, to be regarded simply as an illustration of the disclosure as defined by the appended claims, and are contemplated to cover any modifications, variations, combinations, or equivalents that fall within the scope of the present disclosure. Other aspects may be within the scope of the following claims. Finally, as used herein, the conjunction “or” refers to a non-exclusive “or,” unless specifically stated otherwise.

Claims

CLAIMS What is claimed is: 1. A computer-implemented method for training a machine learning (ML) model, the method comprising: performing data augmentation on a data object from a training data set to generate a plurality of augmented data sets, the training data set being associated with a classifier of the machine learning model; generating a plurality of testing accuracies of the classifier based on the plurality of augmented data sets and determining at least one testing accuracy of the plurality of testing accuracies is below a pre-configured threshold accuracy, the at least one testing accuracy corresponding to an augmented data set of the plurality of augmented data sets; and supplementing the training data set with the augmented data set to generate a revised training data set.

2. The computer-implemented method of claim 1, the performing the data augmentation further comprising: detecting that the data object comprises image data; and selecting a plurality of data transformations from available data transformations associated with the image data.

3. The computer-implemented method of claim 2, the performing the data augmentation further comprising: applying the plurality of data transformations to the image data to generate the plurality of augmented data sets, wherein each data transformation of the plurality of data transformations corresponds to a respective augmented data set of the plurality of augmented data sets.

4. The computer-implemented method of any of claims 2-3, the plurality of data transformations comprising at least two of: image translation, image reflection, image rotation, image enlargement, image filtering, image color enhancement, image affine transformation, or image noise enhancement.

5. The computer-implemented method of any of claims 1-4, the determining the plurality of testing accuracies further comprising: selecting an augmented data object from the augmented data set of the plurality of augmented data sets; and applying the classifier to the augmented data object to determine a data type of the augmented data object.

6. The computer-implemented method of claim 5, further comprising: determining the at least one testing accuracy corresponding to the augmented data set of the plurality of augmented data sets based on a match between the data type of the augmented data object and a data type of the data object.

7. The computer-implemented method of any of claims 1-6, further comprising: re-training the classifier using the revised training data set, with the supplementing the training data set and the re-training of the classifier being performed in a processing loop; and exiting the processing loop when each testing accuracy of the plurality of testing accuracies is equal to or is above the pre-configured threshold accuracy.

8. The computer-implemented method of any of claims 1-7, further comprising: generating a probability array with a plurality of characteristics of the classifier, wherein each characteristic of the plurality of characteristics indicates a probability that a corresponding testing accuracy of the plurality of testing accuracies is above the pre-configured threshold accuracy.

9. The computer-implemented method of claim 8, further comprising: performing a comparison between the plurality of characteristics of the classifier and a second plurality of characteristics in a second probability array associated with a second classifier; and determining to replace the classifier with the second classifier based on the comparison.

10. The computer-implemented method of any of claims 1-9, further comprising a preliminary step of selecting the data object from the training data set.

11. The computer-implemented method of any of claims 1-10, further comprising re-training the classifier using the revised training data set.

12. An apparatus for training a machine learning (ML) model, the apparatus comprising: a memory storing instructions; and at least one processor in communication with the memory, the at least one processor configured, upon execution of the instructions, to perform the following steps: performing data augmentation on a data object from a training data set to generate a plurality of augmented data sets, the training data set being associated with a classifier of the machine learning model; generating a plurality of testing accuracies of the classifier based on the plurality of augmented data sets and determining at least one testing accuracy of the plurality of testing accuracies is below a pre- configured threshold accuracy, the at least one testing accuracy corresponding to an augmented data set of the plurality of augmented data sets; and supplementing the training data set with augmented data from the augmented data set to generate a revised training data set.

13. The apparatus of claim 12, the performing the data augmentation further comprises: detecting that the data object comprises image data; and selecting a plurality of data transformations from available data transformations associated with the image data.

14. The apparatus of claim 13, the performing the data augmentation further comprises: applying the plurality of data transformations to the image data to generate the plurality of augmented data sets, wherein each data transformation of the plurality of data transformations corresponds to a respective augmented data set of the plurality of augmented data sets.

15. The apparatus of any of claims 13-14, the plurality of data transformations comprising at least two of: image translation, image reflection, image rotation, image enlargement, image filtering, image color enhancement, image affine transformation, or image noise enhancement.

16. The apparatus of any of claims 12-15, the determining the plurality of testing accuracies further comprising: selecting an augmented data object from the augmented data set of the plurality of augmented data sets; and applying the classifier to the augmented data object to determine a data type of the augmented data object.

17. The apparatus of claim 16, the at least one processor further executing the instructions to perform the steps of: determining the at least one testing accuracy corresponding to the augmented data set of the plurality of augmented data sets based on a match between the data type of the augmented data object and a data type of the data object.

18. The apparatus of any of claims 12-17, the at least one processor further executing the instructions to perform the steps of: re-training the classifier using the revised training data set, with the supplementing the training data set and the re-training of the classifier being performed in a processing loop; and exiting the processing loop when each testing accuracy of the plurality of testing accuracies is equal to or is above the pre-configured threshold accuracy.

19. The apparatus of any of claims 12-18, the at least one processor further executing the instructions to perform the steps of: generating a probability array with a plurality of characteristics of the classifier, wherein each characteristic of the plurality of characteristics indicates a probability that a corresponding testing accuracy of the plurality of testing accuracies is above the pre-configured threshold accuracy.

20. The apparatus of claim 19, the at least one processor further executing the instructions to perform the steps of: performing a comparison between the plurality of characteristics of the classifier and a second plurality of characteristics in a second probability array associated with a second classifier; and determining to replace the classifier with the second classifier based on the comparison.

21. The apparatus of any of claims 12-20, the at least one processor further executing the instructions to perform a preliminary step of selecting the data object from the training data set.

22. The apparatus of any of claims 12-21, the at least one processor further executing the instructions to perform the step of re-training the classifier using the revised training data set.

23. A non-transitory computer-readable media storing computer instructions for training a machine learning model, that configure at least one processor, upon execution of the instructions, to perform the following steps comprising: performing data augmentation on a data object from a training data set to generate a plurality of augmented data sets, the training data set being associated with a classifier of the machine learning model; generating a plurality of testing accuracies of the classifier based on the plurality of augmented data sets and determining at least one testing accuracy of the plurality of testing accuracies is below a pre-configured threshold accuracy, the at least one testing accuracy corresponding to an augmented data set of the plurality of augmented data sets; and supplementing the training data set with augmented data from the augmented data set to generate a revised training data set.

24. The computer-readable media of claim 23, the performing the data augmentation further comprising: detecting that the data object comprises image data; and selecting a plurality of data transformations from available data transformations associated with the image data.

25. The computer-readable media of claim 24, the performing the data augmentation further comprising: applying the plurality of data transformations to the image data to generate the plurality of augmented data sets, wherein each data transformation of the plurality of data transformations corresponds to a respective augmented data set of the plurality of augmented data sets.

26. The computer-readable media of any of claims 24-25, the plurality of data transformations comprising at least two of: image translation, image reflection, image rotation, image enlargement, image filtering, image color enhancement, image affine transformation, or image noise enhancement.

27. The computer-readable media of any of claims 23-26, the determining the plurality of testing accuracies further comprising: selecting an augmented data object from the augmented data set of the plurality of augmented data sets; and applying the classifier to the augmented data object to determine a data type of the augmented data object.

28. The computer-readable media of claim 27, the steps further comprising: determining the at least one testing accuracy corresponding to the augmented data set of the plurality of augmented data sets based on a match between the data type of the augmented data object and a data type of the data object.

29. The computer-readable media of any of claims 23-28, the steps further comprising: re-training the classifier using the revised training data set, with the supplementing the training data set and the re-training of the classifier being performed in a processing loop; and exiting the processing loop when each testing accuracy of the plurality of testing accuracies is equal to or is above the pre-configured threshold accuracy.

30. The computer-readable media of any of claims 23-29, the steps further comprising: generating a probability array with a plurality of characteristics of the classifier, wherein each characteristic of the plurality of characteristics indicates a probability that a corresponding testing accuracy of the plurality of testing accuracies is above the pre-configured threshold accuracy.

31. The computer-readable media of claim 30, the steps further comprising: performing a comparison between the plurality of characteristics of the classifier and a second plurality of characteristics in a second probability array associated with a second classifier; and determining to replace the classifier with the second classifier based on the comparison.

32. The computer-readable media of any of claims 23-31, the steps further comprising a preliminary step of selecting the data object from the training data set.

33. The computer-readable media of any of claims 23-32, the steps further comprising re-training the classifier using the revised training data set.

34. An apparatus for training a machine learning (ML) model, the apparatus comprising: means for performing data augmentation on a data object from a training data set to generate a plurality of augmented data sets, the training data set being associated with a classifier of the machine learning model; means for generating a plurality of testing accuracies of the classifier based on the plurality of augmented data sets and determining at least one testing accuracy of the plurality of testing accuracies is below a pre-configured threshold accuracy, the at least one testing accuracy corresponding to an augmented data set of the plurality of augmented data sets; and means for supplementing the training data set with augmented data from the augmented data set to generate a revised training data set.