JP2022056412A - Method, system, and program (mobile ai) for selecting machine learning model - Google Patents

Abstract

To allow a machine learning model to be optimized for development into a device on the basis of hardware specifications of the device.SOLUTION: An existing model is acquired and pruned in order to reduce hardware resource consumption of the model. Next, the pruned model is trained on the basis of training data. The pruned model is also trained on the basis of a collection of teacher models. Next, the performance of the trained model is evaluated and is compared with performance requirements which can be based on hardware specifications of a device.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to machine learning, and more specifically to optimizing the performance of machine learning models.

Machine learning models such as neural networks are becoming more and more versatile. While machine learning models can be useful, machine learning models still have associated drawbacks and costs. For example, training a model can be particularly resource-intensive, and accuracy and required computing resources are often inversely proportional.

Specifically, even modern systems automatically configure and deploy machine learning models on relatively low power devices, ensuring that the deployed model meets both accuracy and resource performance requirements. It is not possible. For example, machine learning models used in low power devices are often limited in performance (due to the hardware overhead required by more advanced machine learning models). In addition, in many areas (such as medicine), the amount of training data available may be limited, making training new models particularly difficult. Therefore, even if a given task (such as classification) can be reliably accomplished through prior art machine learning models, such a model would not be able to perform within a given hardware specification of a low power device. It is likely to require considerable effort. Similarly, many prior art models can function with limited hardware resources, but their performance (eg, accuracy) is typically compromised as a result of their "lightweight" configuration.

While machine learning models can be useful, machine learning models still have associated drawbacks and costs. For example, training a model can be particularly resource-intensive, and accuracy and required computing resources are often inversely proportional.

Some embodiments of the present disclosure may be exemplified as a first method. The first method involves receiving the hardware specifications of the device. The first method also includes determining performance requirements based on the hardware specifications of the device. The first method also includes acquiring a machine learning model with a set of layers. The first method also includes the acquisition of a teacher model. The first method also involves removing one or more layers of a set of layers from the machine learning model, resulting in a student model with a set of pruned layers. The first method also includes training the student model based on the training data and the teacher model. The first method also includes evaluating the performance of the student model and comparing the performance of the student model with the performance requirements. The first method also involves making a judgment (based on comparison) to select the model with the best performance. This first method makes it possible to advantageously configure the model to function within the requirements based on the hardware specifications of the device while training the model to meet performance criteria.

Some embodiments of the present disclosure may be exemplified as a second method. The second method includes the first method as discussed above, and the training (of the first method as discussed above) involves inputting training data into the student and teacher models and the student. Receiving student features from the model, receiving teacher features from the teacher model, performing the first comparison between student features and teacher features, and receiving student output from the student model. Performing a second comparison between student output and training data, and adjusting at least one layer of the student model's set of pruned layers based on the first comparison and said second comparison. Including what to do. This second method favorably configures the student model to perform similar to the advanced teacher model, even though the student model has significantly reduced resource overhead compared to the teacher model. to enable.

Some embodiments of the present disclosure may be exemplified as a third method. The third method includes the first method as discussed above, and removing one or more layers (of the first method as discussed above) is based on performance requirements. It involves determining the number of layers to remove, randomly selecting one or more layers based on the number, and deleting the selected layers. This third method advantageously allows the model to be pruned based on the hardware specifications of the device with improved efficiency.

Some embodiments of the present disclosure may be exemplified as a fourth method. A fourth method involves retrieving labeled data. The fourth method also includes retrieving unlabeled data. The fourth method also includes comparing the labeled data with the unlabeled data. The fourth method also includes labeling unlabeled data based on comparison, resulting in weakly labeled data. The fourth method also involves retraining the model based on weakly labeled data, resulting in retraining of the model. This fourth method advantageously allows the model to be retrained based on the initially unlabeled data.

Some embodiments of the present disclosure may be exemplified as a fifth method. The fifth method includes a fourth method as discussed above. The fifth method also involves obtaining the teacher output from the teacher model, and the retraining (of the fourth method as discussed above) is further based on the strongly labeled data and the teacher output. This fifth method first makes it possible to advantageously retrain the model based on performance-enhanced unlabeled data.

Some embodiments of the present disclosure may be exemplified as a sixth method. The sixth method includes a fourth method as discussed above, and retrieving labeled data (of the fourth method as discussed above) takes the labeled input data. Labeled feature data is labeled data, including receiving and generating labeled feature data via a model based on the labeled input data. Further, in the sixth method, retrieving unlabeled data (of the fourth method as discussed above) is labeled as collecting unlabeled input data. Unlabeled feature data is unlabeled data, including generating unlabeled feature data via a model based on no input data. This sixth method first makes it possible to advantageously retrain the model based on unlabeled data with improved privacy security and further effectiveness.

Some embodiments of the present disclosure may also be exemplified as a computer program product having a computer-readable storage medium having the program instructions embodied in the computer program product, where the program instructions are any of the methods discussed above. Can be run by the computer to get the computer to run. This makes it possible to advantageously configure the model to function within the requirements based on the hardware specifications of the device while training the model to meet performance criteria.

Some embodiments of the present disclosure may be exemplified as a system. The system may include memory and a central processing unit (CPU). The CPU may be configured to execute an instruction to perform any of the methods discussed above. This makes it possible to advantageously configure the model to function within the requirements based on the hardware specifications of the device while training the model to meet performance criteria.

The above overview is not intended to illustrate each of the exemplary embodiments or implementations of the present disclosure.

The drawings included in this application are incorporated herein and form part of this specification. They exemplify embodiments of the present disclosure and, along with explanations, serve to explain the principles of disclosure. The drawings merely illustrate certain embodiments and are not intended to limit the disclosure. The features and advantages of the various embodiments of the claimed subject matter will become apparent as the embodiments for carrying out the following invention progress, and by reference to the drawings in which similar numbers show similar parts. Let's do it.

A high level of automated complex knowledge distillation method consistent with some embodiments of the present disclosure.

It is an exemplary diagram of model selection and pruning to generate a student model, consistent with some embodiments of the present disclosure.

It is an exemplary diagram of a data flow for training a student model based in part on training data and multiple teacher models, consistent with some embodiments of the present disclosure.

Demonstrates a high level method for updating machine learning models based on unlabeled data, consistent with some embodiments of the present disclosure.

It is a diagram showing an exemplary example of how unlabeled observational data can be automatically labeled based on a known dataset, consistent with some embodiments of the present disclosure.

Demonstrates a high-level way to update a machine learning model based on anonymized feature data derived from unlabeled input data, consistent with some embodiments of the present disclosure.

It is a diagram showing an example of labeling of unlabeled input data via transductive learning, consistent with some embodiments of the present disclosure.

A high-level block diagram of an exemplary computer system that can be used in implementing the embodiments of the present disclosure is shown.

The present invention can accept various modified and alternative forms, the specific examples of which are shown in the drawings as examples and will be described in detail. However, it should be understood that the intent is not limited to the particular embodiment described of the present invention. On the contrary, the intent is to cover all modifications, equivalents, and alternatives within the spirit and scope of the invention.

Aspects of the present disclosure relate to systems and methods for optimizing models with respect to device specifications. A more specific embodiment relates to a system that acquires a machine learning model structure, compresses the structure, trains the compression model, evaluates the performance of the trained model, and selects the model with the highest performance.

A further aspect of the disclosure relates to a system and method for improving the performance of a model based on measurement data. A more specific aspect of the present disclosure is to receive unlabeled data, compare the unlabeled data with known data, label the unlabeled data based on the comparison, and add new labels. Regarding a system that retrains a model based on labeled data.

The systems and methods described in this disclosure allow for automated compact model compression via "complex knowledge distillation", such as neural networks, based on hardware specifications and / or related performance requirements. It makes it possible to optimize the machine learning model in an advantageous way. In other words, most state-of-the-art systems accept trade-offs between accuracy / speed and resource requirements, but models trained through the systems and methods disclosed are mounted on mobile devices or wrists. Despite working on relatively low power hardware such as devices, it can be optimized to maximize accuracy and / or speed.

Moreover, systems and methods consistent with the present disclosure can advantageously allow the model to be trained to operate reliably on lower power devices, even with relatively limited training data sets.

Throughout this disclosure, "machine learning models" are referred to and abbreviated to "models" for simplicity. The model may include, for example, an artificial neural network such as a convolutional neural network (CNN), a multi-layer perceptron (MLP), a recurrent neural network (RNN), a long short-term memory (LSTM) RNN. The model can be used for various purposes such as classification and prediction. Typically, the model receives an input of data and passes that data to several "layers". For example, the data is input to a first layer that manipulates the data (based on the composition of the layer) and produces an output of one or more "features" (often grouped into "feature vectors"). obtain. The layer configuration includes several different "channels" and "filters" that are responsible for the lower level aspects of the data manipulation performed by the layer.

As an example, the input data can be an image, the first layer identifies the edges in the image and a list of data (“vectors”) that describes all the edges (“features”) detected in the image. Can be generated. The features produced by the first layer are then sent to the second layer, where the second layer performs further operations to generate a second feature vector. The same applies hereinafter. The final result is "output" at the end. As used herein, "output" refers to the final output / determination / classification of a model, unless otherwise stated. The layout / configuration and order of the various layers is commonly referred to as the "structure" of the model.

Machine learning models are often "trained" and one or more layers are tuned to change their inputs in different ways. As a simple example, the first layer may initially double the input, but during the training process the first layer may be adjusted to multiply the input by 1.5 instead of doubling. Of course, more complex layers can be adjusted in more subtle ways.

Machine learning models trained and / or optimized according to the systems and methods of the present disclosure may be optimized to meet one or more performance requirements. The performance requirements used herein are resource-based requirements (maximum memory footprint, maximum CPU utilization, maximum estimated power consumption, etc.), output-based requirements (accuracy evaluation, inference speed, etc.), or two of them. May include combinations.

In many areas, even a well-trained and scrutinized model can become less accurate over time. The model can be configured to learn over time (which can inadvertently lead to model degradation), but another common cause of poor accuracy is changes in the input data to the model. obtain. As an example, a patient's electroencephalogram (EEG) data can change spontaneously over time, regardless of changes in the patient's condition. The EEG of a healthy patient recorded today can be significantly different from the EEG of the same healthy patient years (or months) ago, even if it is relatively consistent over a shorter period of time. In particular, this is not (necessarily) caused by factors other than the patient's own physiology, and the EEG emission pattern can simply "drift" over time. Patterns of various problems (such as seizures) can still be relatively consistent, but "background" or "baseline" states detect the presence (or absence) of such significant patterns. Therefore, it may change significantly to the point where it may not be possible to analyze the model. Therefore, a model that was initially accurate in classifying a patient's EEG can eventually become obsolete. Models can be retrained or improved over time, but known datasets can be relatively small, especially in narrow areas (such as neurology).

In addition, background / baseline patterns or noise or combinations thereof can vary widely between patients, so known datasets may not be sufficient to train (or retrain) a particular patient's model. be. One solution in the art is to record datasets from patients on a regular basis, have specialists (health care providers / specialists, etc.) evaluate the recorded datasets and manually annotate the datasets. / Labeling and using the patient-specific dataset for model retraining. However, this can be exorbitantly time consuming, inconvenient, and / or costly for both patients and professionals. In addition, this can raise concerns about the confidentiality / personally identifiable nature of the patient's data.

The systems and methods described in this disclosure make it possible to further improve the accuracy of machine learning models based on observed (ie, initially unlabeled) data and manually annotate the observed data. It makes it possible to advantageously improve the performance of the model without the need. This changes the patient's physiology and over time, even if new patterns (ie, patterns not represented in the model's initial training dataset or, in some cases, known datasets) are observed. The model can grow over time and become more robust.

FIG. 1 is a high level automated composite knowledge distillation method 100 consistent with some embodiments of the present disclosure. Method 100 advantageously enables training and optimization of machine learning models with respect to device-specific performance requirements.

Method 100 comprises collecting performance requirements and / or hardware specifications in operation 102. Operation 102 may include receiving data from a device, such as a mobile device, that describes the hardware of the device (available memory, processor manufacturer / model, etc.). Performance requirements, especially resource-based requirements such as maximum memory footprint, can be collected from this data. In some embodiments, operation 102 may include receiving output-based requirements outside the hardware specifications. For example, the operation 102 may include receiving a minimum accuracy rating, a minimum inference speed, etc. from the user or device based on the circumstances in which the device is normally used.

Method 100 further comprises acquiring a model in motion 104. Operation 104 may include, for example, downloading a machine learning model from a repository of models known to be accurate. The model acquired in operation 104 can be one of a plurality of different types of models (eg, CNN, RNN, etc.). In particular, the model obtained may be for a different purpose than the model created and optimized via method 100. As an example, method 100 can be performed to train and optimize seizure detection models intended for deployment to wrist-mounted computing devices. Nevertheless, operation 104 may include selecting an established CNN (eg, an EFFICIENTNET model) that has been previously trained to perform image recognition based on the input image. This is possible because of the purpose of Method 100, the structure of the selected model (eg, the composition of the individual layers) is the actual original purpose of the selected model (eg, face recognition, playing chess, etc.). This is because it is often more important than. The repository accessed in operation 104 may be generally available, stored in a cloud database, stored in local long-term memory, and so on.

Method 100 further comprises pruning the selected machine learning model based on the performance requirements in motion 106. Operation 106 may include, for example, removing one or more layers from the selected model. In some embodiments, operation 106 may include removing one or more blocks (consecutive groups of layers) from the selected model. The layer and / or block selected to be deleted in operation 106 may, in some cases, select a layer adjacent to (or, in some embodiments, not) adjacent to another selected layer, and the like. It can be randomly selected with some restrictions. In some embodiments, the layer may be removed (partially) based on the performance requirements received in operation 102. Action 106 may include determining the number of layers in the selected model and removing the layers until the number falls below the threshold. The total number of layers in the model (the "number of layers" in the model) is a key factor in the model's resource requirements (other factors are the size of the filter used in each layer, the resolution / dimension of the output of each layer). Etc.). For example, a model with 300 layers generally consumes significantly more memory and CPU overhead than a model with 100 layers. With this in mind, the threshold can be determined, for example, based on the device's hardware maximum memory and / or CPU model. Thresholds can limit certain characteristics of the model, such as the maximum number of layers, the maximum number of channels per layer, and so on. As an example, the device may have 512 megabytes (MB) of memory (such as random access memory (RAM)) and a CPU operating at 120 megahertz (MHz). These hardware specifications can be taken into account when generating thresholds. As a simple example, the maximum number of layers in a model deployed on a device is r / 10 layers (where r is the total memory of the device (in MB)) or c / 2 layers (where c is the device). Whichever is lower (rounded down to the nearest integer) of the CPU frequency (in MHz). Therefore, the maximum number of layers of the model developed in the above-mentioned device example may be 51 layers (512/10 <120/2). It should be noted that these indicators are not necessarily model resource requirements (eg, a 3-tier model may not necessarily require 30 MB of RAM, or may consume as much as half the resources of a 6-tier model). sea bream. In some embodiments, the threshold may represent the maximum amount of memory the model can consume, the maximum amount of CPU overhead, and so on.

The pruned and selected model is referred to herein as the "student model". In particular, in some embodiments, the remaining layers of the student model (ie, the layers that were not removed as part of movement 106) may remain relatively unchanged. For example, the internal composition of the layer (such as individual "neurons") can remain unaffected and the student model can retain some of the structure of the original selected model. However, the connection between the deleted layer and the rest of the layers may be broken, resulting in inaccuracies or complete inoperability of the student model. In some embodiments, rather than pruning the layer (or in addition), the channels and / or filters within one or more layers are removed to reduce the overall complexity of the layer. obtain.

As an exemplary example, the model may have six layers (“Layer 1-6”), and the output of each layer is fed to the input of the next layer (eg, each of Layers 1-5 is: "Connected" to the layer). In other words, layer 1 is connected to layer 2 (ie, the output of layer 1 is used as the input of layer 2) and layer 2 is connected to layer 3 (ie, the output of layer 2 is the input of layer 3). Used as). The same applies hereinafter. The layers themselves are generally not the same, and each layer manipulates its inputs differently to produce individual outputs. The layer itself evolves over time in the process of training the model (note that the model acquired in motion 104 has already been trained). For example, layer 1 may manipulate the input data to generate an output feature vector if the input data includes an image so that the output feature vector represents an edge of the image. Layer 2 may manipulate the input feature vector to generate the output feature vector so that if the input feature vector represents the edge of the image, the output feature vector represents the recognized shape in the image. The same applies hereinafter. Therefore, if the operation 106 involves the deletion of layer 1, the input data may be supplied directly to layer 2. Therefore, the input to layer 2 is no longer a feature vector representing the edges of the image (instead, the input is the image itself). However, since layer 2 itself has not changed, layer 2 performs the same operation on the input data as before. Therefore, the output feature vector of layer 2 is less likely to accurately represent the shape of the recognized image. On the contrary, the output feature vector of layer 2 can be generally irrelevant "garbage data". Since layers 3 to 6 have not changed as well, this discrepancy can result in a cascading effect throughout the model. As a result, the removal of motion 106 can result in an accuracy penalty on the final output of the model. In particular, some layers may make only minor changes to the input. Therefore, depending on the removed layer, the overall accuracy penalty can range from relatively small to severe enough to make the model functionally inoperable.

Pruning can result in significant accuracy penalties for the entire model, but the structure of the model can remain largely intact. In other words, the model may be in a state where the function of the model can be restored by some training to "fill the gap" left by the deleted layer. Continuing with the above example, layers 3-6 can function as before, so if layer 2 can be adjusted to convert the input image into an output feature vector representing the recognized shape in the image. , The model can come to produce (relatively) accurate results.

Method 100 further comprises training a student model with motion 108. The operation 108 is, for example, inputting known training data into the student model, receiving an output from the student model, comparing the output with a known result, and one of the student models based on the comparison. It may include adjusting multiple layers. Motion 108 may further include inputting training data into one or more "teacher" models (other established / known good models). The operation 108 may include selecting a teacher model from the repository, for example, the teacher model may be another model in the repository for which the model was selected in operation 104. In some cases, operation 108 may utilize all teacher models in a given repository. In some cases, motion 108 may select several teacher models (eg, randomly). As an example, motion 104 may include randomly selecting a model from a repository having 15 models, and motion 108 may include utilizing some or all of the remaining 14 models. In some cases, the model selected as part of motion 104 may also be selected as a teacher model in motion 108.

In some embodiments, the adjustments made to the student model layer as part of motion 108 may be further based on a comparison between the output from the teacher model and the output from the student model. In general, the output of the student model (“student output”) and the output of the teacher model (“teacher output”) are compared to minimize their differences during the training process. For example, if the student output is significantly different from the teacher output, the layer of the student model can be adjusted even if the output is relatively similar to the expected output (eg, the student output of "35" is "36". It can be quite similar to the expected output, but even if the teacher output is "37", the layer of the student model can be adjusted).

In some embodiments, rather than comparing (or in addition to) the output of the student and teacher models, motion 108 compares the features output by the "middle" layer of the student and teacher models. May include. In some embodiments, the student model layer selected for comparison may be randomly selected. In some embodiments, layers may be selected based on pruning, for example, after the last adjusted (or deleted) layer, a first unadjusted layer may be selected. As an example, the model selected in operation 104 may have four layers A, B, C and D, and operation 106 may include removing layer B. In such an example, motion 108 may include comparing the feature vector output by layer C of the student with the feature vector output by layer C of the four-layer teacher model. As an additional example, the last unadjusted layer may be selected before the first adjusted (or deleted) layer (eg, the output of student layer A is the output of teacher layer B). Can be compared with). For teacher models with a different number of layers than the model selected in motion 104, and after pruning in motion 106, layers may be selected for output comparison based on their order in the model (eg, each). The feature vector output by the penultimate layer of the model can be selected for comparison). This may advantageously allow the student model to be trained to perform in a manner similar to the teacher model, albeit substantially simple (due to the pruning of motion 106).

Method 100 further comprises assessing the performance of the student model pruned / trained at motion 110. The operation 110 includes, for example, inputting different known data (eg, a second set of known data different from the training data) into the student model and analyzing the performance of the student model in determining the output. obtain. As an example, the operation 110 is the accuracy of the student model, the inference speed (the time it takes for the student model to judge the output), the memory footprint (the amount of memory required for the student model), the peak CPU usage, and the estimated power consumption. It may include determining the amount and the like.

Method 100 further comprises determining if the trained student model meets the performance requirements for motion 112. Motion 112 may include, for example, comparing the performance of the student model as determined by motion 110 with the performance requirements collected in motion 102. As an example, operation 112 determines whether the student model's memory usage is below the maximum threshold indicated by the device hardware, whether the student model's inference speed is above the user-set minimum, and so on. May include doing.

If the student model does not meet one or more performance requirements (112 "no"), method 100 proceeds to generate a new student model, starting with acquiring the model in motion 105 and retrying. The operation 105 can be performed in substantially the same manner as the operation 104. In particular, the model selected in motion 105 is the same model or "new" model selected in motion 104 (or the iteration prior to motion 105) (eg, a model that has not yet been pruned / trained / evaluated by method 100). Can be. Operation 105 may include, for example, selecting a model from a repository or other external source, optionally including the source accessed by operation 104. In some embodiments, a copy of the model selected in operation 104 may be cached, and therefore operation 105 may include loading a copy. In some embodiments, the model selected in motion 105 is obtained based on a comparison of motion 112, eg, if the student model is significantly inferior to the performance requirements, motion 105 is the same model previously used. It may include selecting a new model instead. In some embodiments, trends can be tracked. For example, if the student model generated from a particular model is consistently inferior regardless of pruning / training, the particular model may be excluded from selection in the iterations after motion 105.

Once the model is selected in motion 105, method 100 then returns to pruning, training, and evaluation of the model in motions 106-110 to generate a new student model. In some embodiments, motion 105 may select a new model at each iteration until all available models have been "tried". In some cases, a new student model can be generated by selecting the same model and pruning in different ways, such as removing more layers / less layers / different layers, but otherwise the same method. Trained at. Even if only pruning was performed in different ways, this would still be a different trained student model. Therefore, when different student models are evaluated in motion 110, they may be judged to perform better (or inferior) than the previous model. In some cases, a new student model is generated by selecting the same model and pruning in the same way (eg, deleting the same layers / blocks deleted in the previous iteration of operation 106 in operation 106). You get, but you are trained differently in motion 108, such as using a different dataset than that used in the previous iteration of motion 108. In some cases, a new student model can be generated by choosing another model, but in other cases, the same layer can be pruned and trained in the same way. Therefore, several different methods for generating new student models can be implemented and considered.

If the student model meets all performance requirements (112 "yes"), method 100 further comprises determining in motion 114 whether the model's performance is inferior to the previous model. Motion 114 may include, for example, comparing the performance of the currently studied student model (the "current" student model) with the performance of a previous set of student models. The performance of the current student model can be compared to the performance of each previous student model in the set, or the average or maximum of the performance of each previous student model in the set. Depending on the embodiment / use case, the previous set of student models may exclude student models that do not meet the performance requirements of motion 112. Depending on the embodiment / use case, the set of previous student models can be all previous student models only or only some (eg, previous student model only, last three student models, etc.).

As an exemplary example, 35 student models can be generated and evaluated (referred to as "model 1", "model 2", etc.). Model 2, Model 32, Model 33, and Model 35 (in this example, Model 35 is the "current" student model) can meet all performance requirements, respectively, Model 1, Models 3-31, and Model 34. Each cannot meet all performance requirements. Thus, operation 114 lists the performance of model 35, model 34, models 32-34, model 33, models 2, 32, and 33, models 1-34, depending on the use case and / or embodiment. It may include comparing with performance such as average and / or maximum of any of the groups.

Performance comparisons may be organized by "category", for example, operation 114 may include comparing the amount of memory consumed, inference speed, accuracy, etc. of different models. Depending on the embodiment / use case, the behavior 114 is a set of previous student models in which the performance of the current student model is in each performance category, most of the categories, or in a particular group of one or more categories. It may include determining if it is better than performance. For example, if the performance of the current student model has higher accuracy, slower inference speed, etc., the performance can be considered "excellent". In some embodiments, the performance of each category can be used to calculate an overall performance rating (eg, a number from 0 to 1), where operation 114 compares the performance ratings of the various models. Can include that.

If the performance of the current student model is superior to that of the previous set of models (114 "no"), method 100 returns to operation 105 to generate and evaluate another student model. In this way, student models are continuously generated (through model selection, pruning, and training) and evaluated as long as performance improves. If the current student model is the generated first student model, the action 114 defaults to 114 "no". If the current student model is inferior to the previous model (114 "yes"), the loop may be terminated and method 100 may be terminated by selecting the model with the best performance in motion 116. As used herein, "best performance" refers to the model with the highest performance in one or more evaluated categories (eg, fastest inference speed, highest accuracy, lowest resource requirements, etc.). obtain. If different models are superior in different categories, one category may take precedence over the other (eg, accuracy) as long as the corresponding best model meets the minimum performance requirements of the remaining categories. Other criteria for selecting the "best performance" model will also be understood by those of skill in the art. For example, the performance of the student model in each category may be assigned a quantifiable rating (eg, ranging from 0 to 1). Performance requirements can be expressed by evaluation in a similar manner. The difference between the minimum required evaluation and the evaluation of each model can be determined, and the model with the largest total difference (or the largest average difference, etc.) can be selected.

Looping operations 105-114 allows the system performing method 100 to advantageously generate machine learning models optimized for specific performance requirements such as device hardware specifications. In some embodiments, the loop must be executed a minimum number of times before the method 100 can proceed to operation 116. In some embodiments, for example, has the current student model exceeded a set of performance goals (eg, higher thresholds based on performance requirements), has the maximum number of models tried, or has exceeded the performance requirements? There may be other conditions for ending the loop, such as whether. Different conditions can be used in different use cases.

FIG. 2 is an exemplary diagram 200 of model selection and pruning for generating student models, consistent with some embodiments of the present disclosure. The machine learning model repository 202 includes a set of machine learning models 220, 240, 260 and 280 (collectively "models 220-280"). Repository 202 may include, for example, an online database of scrutinized and trained models. Models 220-280 can each be of a different type (eg, model 220 can be a convolutional neural network, model 280 can be a recurrent neural network, etc.), or part of models 220-280. (Or all) can be models of the same type. Models 220-280 can be trained to perform various tasks such as image recognition, natural language processing and the like.

Each of the models 220-280 comprises one or more layers. Model 220 includes layers 222, 224, and 226, model 240 contains layers 242, 244, and 246, model 260 contains layers 262, 264, and 266, and model 280 contains layers 282, 284, and 286. Includes (collectively, layers 222-286). Layers 222-286 are configured to receive data inputs, manipulate the data, and generate outputs. The input to each layer can be either the initial input dataset at the first layer (ie, layers 222, 242, 262, or 282), or the feature vector output from the previous layer.

One of models 220-280 may be selected for use as a student model. In the example shown in FIG. 2, the model 240 is selected, as shown by the dashed box 201. In some embodiments, various selection criteria may be implemented to attempt to select an "ideal" initial model such as accuracy: layer ratio, memory footprint, inference speed, etc. However, in some embodiments, random selection can be beneficial because it is relatively easy to implement and fast to execute. Moreover, due to the unpredictable effects of pruning and training processes, random selection can still be relatively effective. The selection of model 240 may be performed in a manner similar to that described for operation 104 of method 100 discussed above in connection with FIG.

When model 240 is selected, it is pruned and in the example shown in FIG. 2, layer 244 of model 240 is removed, resulting in student model 241 having only layer 242 and layer 246. As a result, the student model 241 may have smaller size and resource requirements when compared to the "original" model 240. The selection of layers to be deleted can be performed in a manner similar to operation 106 of method 100 discussed above in connection with FIG. The specific layer deleted can be randomly selected or selected based on one or more criteria. For example, one or more contiguous blocks (having a minimum size of 10% of the total number of layers) may be randomly selected, the central 20 layers may be selected, and so on. The number of layers to be deleted can also be determined randomly, but limits can be set according to performance requirements. For example, in a 150-layer model selected as the basis for a student model implemented on a mobile device, 100 layers can be deleted and the exact layer deleted can be randomly selected.

It is important to note that models 220-280 can significantly exceed three layers, and machine learning models typically have hundreds of layers. However, models 220-280 are shown to have only three layers each for simplicity of explanation. Similarly, the student model 241 has fewer layers than the model 240, but the student model 241 can have significantly more (and possibly hundreds) of layers than the two. On the other hand, the student model 241 is shown to be missing only a single layer compared to the models 220, 240, 260, and 280, whereas the student model 241 is from any of the models 220-280. Can also contain significantly fewer layers. Moreover, the models 220-280 may not necessarily have the same structure, and as an exemplary example, some models are not four models with three layers and a student model with two layers, respectively. Use cases may include 15 teacher models (although all 15 have at least 200 layers) and a student model with 30 layers, each with a different number of layers.

Pruning can significantly improve resource-based performance. In other words, after pruning, the student model 241 can be significantly "lighter" than the model 240. For example, the student model 241 may consume significantly less memory to operate than the model 240. Therefore, pruning can also result in a significant accuracy penalty for student model 241 when compared to model 240. This is because the connection and order of the layers is an important factor in the effectiveness of the model, and even a small change in the order of the layers can have a significant impact on the accuracy of the model. Pruning involves removing multiple layers, which can result in a significant reduction in the accuracy of the student model 241. However, these penalties can be advantageously mitigated by special training as described below with reference to FIG.

FIG. 3 is an exemplary diagram 300 of a data flow for training a student model 340 based in part on training data 302 and a plurality of teacher models 320 and 360, consistent with some embodiments of the present disclosure. .. This training can greatly improve the accuracy of the student model and can advantageously reduce the accuracy penalty incurred by pruning. The training data 302 includes annotated data such as a series of X-ray images showing a specific medical condition (potential for cancerous growth, fracture, etc.), a series of electroencephalogram (EEG) charts of a patient in a diagnosed state, and the like. Can be famous.

Teacher models 320 and 360 can be established / pre-trained models selected from the model repository. For example, teacher models 320 and 360 may correspond to models 220 and 260 in the model repository 202, as described above with reference to FIG.

Data 302 can be input to several models such as teacher models 320 and 360 as well as student models 340. For example, the data 302 may be input to the first layer 322 of the model 320. Layer 322 may generate features 323 (such as feature vector morphology), which may then be input to layer 324. Similarly, layer 324 may manipulate feature 323 and then produce output feature 325. Feature 325 can then be input to the final layer 326, resulting in output 328. In some embodiments, the output 328 may be compared to the label of the data 302 to check the accuracy of the teacher model 320. However, the accuracy of the teacher model 320 may be considered sufficient to train the student model 340 (thus eliminating the need to compare the output 328 with the data 302).

Similarly, the data 302 may be input to the first layer 362 of the model 360. The layer 362 may generate the output feature 363 and then the output feature 363 may be input to the layer 364. Similarly, layer 364 may manipulate feature 363 and then produce output feature 365. Feature 365 can then be input to the final layer 366, resulting in output 368. In some embodiments, the output 368 may be compared to the data 302 to check the accuracy of the teacher model 360. However, the accuracy of the teacher model 360 may be considered sufficient to train the student model 340 (thus eliminating the need to compare the output 368 with the data 302).

The data 302 can also be input to the first layer 342 of the student model 340. The first layer 342 may manipulate the input data 302 to generate the first output feature 343. These features can then be input to layer 346 to generate student output 348. Output 348 may be compared to known values of input data 302 and adjustments may be made to layers 342 and 346 based on the comparison. However, function 343 may also be compared to one or more of functions 323, 325, 363, and 365. Further, adjustments to layers 342 and 346 may be based on these comparisons, at least in part. For example, if the difference between feature 342 and feature 323 is relatively large, the magnitude of the change applied to the weight of layer 342 can be relatively large. This is similar to the teacher models 320 and 360, with the training student model 340 having the lowest accuracy cost (because the comparison with the data 302 of the output 348 is also taken into account when the layers 342/346 are adjusted). It may be possible to favorably function in a way (despite the fact that the teacher models 320 and 360 are fairly complex and have many layers).

Throughout this disclosure, various forms of data, in particular "labeled" and "unlabeled" data, are referred to. "Labeled" data is further divided into groups of "strongly" labeled data (usually received from external sources) and "weakly" labeled data. In general, strongly labeled data refers to data that has already been categorized, annotated, or otherwise known and understood. For example, in the context of image recognition, known data can be an image containing information (ie, metadata) detailing the item or object shown in the image. As an additional example, an electrocardiogram (ECG or EKG) received from a public health database containing metadata describing the ECG as a normal cardiac rhythm (scrutinized by one or more medical professionals) is strongly labeled. It can be data. Strongly labeled data can be published from a variety of sources, including model training databases and artificial intelligence (AI) communities. Data without annotations or labels is called "unlabeled" data. As a non-limiting example, raw sensor data can be unlabeled data.

The "weakly" labeled data used herein are not previously labeled and are labeled by means other than manual annotation by human experts, such as through systems and methods consistent with this disclosure. Refers to the data that has been created. The name "weak" can imply the fact that the data has not been labeled by traditional methods such as manual annotation and has not been confirmed to be correctly labeled.

Further, as used herein, "data" can refer to "input data" (such as data recorded from a device) or "feature data" (such as output from the middle layer of a machine learning model). As an example, the ECG (“input data”) may be input to the first layer of the model, which may output a feature vector containing the plurality of features (“feature data”). The feature vector can then be input to the second layer of the model, and then the second layer can output the second feature vector, the feature of which is also feature data. The same applies hereinafter. The input data is either labeled or unlabeled. Feature data generated from labeled input data may inherit the same label. For example, the labeled ECG can be transformed into a first feature vector by the first model layer. The first feature vector can then be labeled based on the ECG label to obtain the labeled feature data. Therefore, retrieving data means receiving input data (eg, by collecting patient data or downloading labeled data, etc.), or inputting feature data into the model. It can also refer to generating feature data (by fetching feature data output from the layers of the model).

FIG. 4 shows a high level method 400 for updating a machine learning model based on unlabeled data, consistent with some embodiments of the present disclosure. Method 400 can be performed by a device on which a machine learning model is installed, such as a computer, for example. Method 400 comprises collecting data in operation 402. Operation 402 may include recording data, for example, via one or more sensors. The data collected in operation 402 can be unlabeled input data. For example, motion 402 may include receiving electroencephalogram recording data (eg, an EEG read image of data from a set of electrodes). Whether the EEG data indicate normal brain activity, for example, seizures, is initially unclear because the data are unlabeled.

Method 400 further comprises receiving known data in operation 404. Operation 404 may include, for example, downloading a dataset from an online database, loading a dataset from a local storage device, and the like. Known data may include a set of measurements annotated by a specialist / specialist or the like. This known data is also referred to herein as "strongly" labeled data. Operation 404 may include receiving a single data set or multiple data sets.

Method 400 further comprises comparing the labeled data with the unlabeled data in operation 406. Operation 406 may include, for example, calculating the difference between a data point in an unlabeled data set and a data point in a labeled data set. In some embodiments, operation 406 may include clustering of data. As a non-limiting high-level example, each data point in a dataset can be represented as an XY coordinate pair, where operation 406 is a group or "cluster" of data points close to each other in the XY plane. It may include plotting each data point for recognition. As a more specific example, a group of labeled publicly available EEG images showing different states of brain function (each image acts as a "data point" corresponding to a single read). Can be plotted so that similarly labeled EEG images are close to each other. In other words, EEG images labeled as indicating normal brain function can be plotted in a first cluster, and EEG images labeled as indicating a first type of seizure can be plotted in a second cluster. EEG images that can be plotted in and labeled as indicating a second type of seizure can be plotted in a third cluster. The same applies hereinafter. The algorithm used to plot the various labeled images can be iteratively modified until the images are clustered according to the labels. Images from unlabeled data can then be plotted and organized into clusters using the same algorithm. An example of such a comparison is shown in Diagram 500 of FIG. Operation 406 may include comparing labeled and unlabeled data via a conversion algorithm.

Some clusters identified via operation 406 cannot contain data points from unlabeled datasets. As an example, the first cluster may contain known data points that are each ECG labeled "normal cardiac rhythm" and the second cluster may contain each labeled "atrial fibrillation". A third cluster may contain known data points, which may be ECGs, and a third cluster may contain known data points, which are ECGs labeled as "ventricular fibrillation", and unlabeled data may be specific. It can be a set of ECGs measured for a patient with. An unlabeled dataset (eg, the patient's ECG) may contain some data points in the first cluster and some data points in the second cluster, but the third cluster. Cannot contain data points. If the patient is relatively healthy and has no history of heart disease, it cannot be expected that unlabeled data will contain ECG indicating ventricular fibrillation (usually cardiac arrest in ventricular fibrillation). (Because it is usually fatal).

Method 400 further comprises labeling the data collected in operation 408. The operation 408 may include determining the label of each data point contained in the unlabeled data set received via the operation 402. The operation 408 may include determining the label based on the comparison of the operation 406. For example, operation 408 may include determining the label of each data point based on the label of the most similar data point from a known data set received. In some embodiments, the operation 408 may be performed by a computer system that implements the conversion algorithm. Labels determined via motion 408 are "weak" because labels determined via motion 408 are not determined by "conventional" methods (such as through manual analysis and annotation by experts). Can be considered a label.

In some embodiments, the output of the teacher model can be considered a weak label. As an exemplary example, unlabeled data can be input to a teacher model that can output a vector containing 7 entries representing the probabilities of 7 target classes (related to 7 classes of seizure type classification). You can apply the argmax operation to this vector to find the index with the highest probability. This index of vector can be treated as a weak / synthetic label because it can be considered the most likely class of unlabeled data.

As an example, a first unlabeled EEG image (eg, collected patient data acquired via motion 402) can be within the first cluster, and the first cluster is also how many. Includes such labeled images (eg, EEG images from known "strongly" labeled datasets received via operation 404). Action 408 identifies that each of the labeled EEG images in the first cluster is labeled as "normal brain activity" and therefore its first label is first labeled. It may include assigning to an unlabeled image (and any other unlabeled data point in the first cluster). Therefore, the first unlabeled EEG image can be assigned a "weak" label that identifies the image as representing "normal brain activity".

Operation 408 may further include assigning the determined label to the dataset (eg, by adding metadata containing the label to the dataset). Metadata may further indicate that the label is a "weak" label.

In some cases, not all data points can be labeled, for example, unlabeled data points are of clusters identified via operation 406 (“unlabeled outliers”). It can exist on the outside or directly between clusters. The procedure for handling unlabeled outliers may vary from embodiment / use case to embodiment. For example, in some embodiments, operation 408 may further comprise discarding unlabeled outliers from the unlabeled dataset. In some embodiments, operation 408 may further comprise labeling unlabeled outliers based on a "best guess" (eg, the cluster closest to the outliers), or Operation 408 may include performing additional "paths" to attempt to label outliers, such as by implementing different algorithms (eg, nearest neighbors). Data points can be considered "outliers" based on threshold comparisons, and the difference threshold is the maximum between the unlabeled data point and the labeled data point (or the center of the cluster). It can be defined to represent the difference. Labeled if the unlabeled data points differ from the closest labeled data point (or the closest cluster of labeled data points) by an amount greater than the difference threshold. No data points can be considered outliers.

Method 400 further comprises retraining the machine learning model with motion 410. Machine learning models can already be trained to analyze or classify input data, such as data collected via motion 402. As the nature of the input data changes, the model can become less accurate over time, which can be mitigated by retraining. Specifically, motion 410 may include using weakly labeled data as training data for retraining the machine learning model. For example, operation 410 inputs weakly labeled data into the model, receives output from the model, calculates the error between the output and the label, and layers the model based on the error. Can include adjusting and repeating until the error is no longer reduced. In some embodiments, motion 410 further comprises training the model with a strongly labeled dataset, in addition to training with weakly labeled data. Further, in some embodiments, strongly labeled data can be input into the teacher model to generate a teacher output, in a manner similar to operation 108 of method 100, strongly considering the teacher output. The student model can be retrained based on the labeled and weakly labeled data.

Action 410 runs against a "working copy" of the model without changing the "live" version of the model, taking into account the possibility that retraining may not improve (or degrade) performance. Can be. In some cases, weakly labeled data and strongly labeled data can be combined into an aggregate training dataset, which can be used to retrain the model. In some cases, weakly labeled data can be used to train the first student model, and strongly labeled data can be used to train the second student model. The two student models can later be combined with the ensemble model.

Method 400 further comprises evaluating a model trained in motion 412. Operation 412 may include, for example, inputting strongly labeled data into the retrained model, receiving output, and determining the accuracy of the retrained model. Motion 412 may include evaluating the model using "evaluation data" with reference to strongly labeled data that was not used to retrain the model as part of motion 410.

Method 400 further determines whether the performance of the retrained model is improved or reduced compared to the performance of the previous version of the model in operation 414 (the "live" model). Be prepared. Motion 414 may include, for example, comparing the accuracy of the retrained model with the accuracy of the live model. For example, a retrained model may correctly classify 98% of the evaluation data in motion 412. If the accuracy of the live model is 95%, then the retrained model is considered superior because of the high accuracy (414 "yes"). Other performance metrics (other than accuracy) such as f1 score, sensitivity, and specificity can be considered. For example, motion 414 may include comparing the sensitivity of the retrained model with the sensitivity of the live model.

If the retrained model results in improved performance (414 "yes"), method 400 further comprises replacing the live model with the retrained model in motion 416. Motion 416 may include, for example, updating the layers of the model based on the changes made when retraining the model in motion 410. In some use cases, older versions of the model can be archived and rollback enabled if needed. As an exemplary example, the performance of a retrained model can be degraded in some contexts, as the evaluation at 412 does not necessarily thoroughly test the performance of the model in all aspects (eg,). , The accuracy is relatively low when certain subtypes of input data are displayed, such as ECG, which indicates a rare display of atrial fibrillation that was not displayed in the evaluation data). Therefore, it may be advantageous to keep a backup of one or more previous iterations of the model in case the retrained model later turns out to be inadequate. However, in some cases, backups may not be available due to storage constraints on the device running the model (although backups can still be maintained elsewhere, such as on a cloud server).

If the retrained model does not provide a performance improvement compared to the live model (414 "no"), method 400 discards the retrained model in motion 418 and continues to use the live model. Further have.

The method 400 may be performed in response to user input (eg, the user may have the device or system perform method 400). In some embodiments, method 400 may be performed on a regular basis, eg, every 3 months, weekly, and so on. Retraining, including the use of weakly labeled data, advantageously allows the system performing Method 400 to update the machine learning model. Specifically, the "real world" nature of weakly labeled data makes it advantageous to update machine learning models to maintain robustness and accuracy as input data moves. become. For example, as the user's physiology changes over time, the data entered into the model may include patterns in which the model has not been trained. However, since the basic pattern can be consistent, Method 400 is to maintain accuracy even when a new pattern is presented, without the need for additional specialized data analysis or annotation. , It can be advantageous to update the model.

In some embodiments, the retrained model can also be compared to resource requirements based on the hardware specifications of the device. If the resource overhead of the retrained model exceeds the requirements, the retrained model can be pruned and retrained. This can be performed in a manner similar to operations 112, 105, and 106 of method 100 (discussed above with reference to FIG. 1). This makes it possible to advantageously retrain existing models in order to achieve both better execution and functionality within resource requirements (eg, by increasing accuracy).

FIG. 5 is a diagram 500 illustrating an exemplary example of how unlabeled observational data can be automatically labeled based on a known dataset, consistent with some embodiments of the present disclosure. .. Diagram 500 can show a semi-supervised machine learning process, such as transformation. Diagram 500 may represent, for example, a 2D plot of data points generated by assigning (X, Y) coordinates to various data points. Figure 500 shows an observed unlabeled dataset containing data points 512, 514, and 516 (shown in squares in FIG. 5), known "strongly" labeled data points 522, 523, and 526. Three different data of the first set of (indicated by a triangle in FIG. 5) and the second set of known strongly labeled data points 532, 534, 535, and 536 (indicated by a circle in FIG. 5). Includes shape from the set. For example, the observed data set can be a patient's EEG scan (ie, data point 512 can be the first set of EEG data from the first patient record, and data point 514 is the same patient. Can be a second set of EEG data from a second recording by, etc.). In addition, the first set of strongly labeled data points can be the first group of publicly available EEG datasets with known results. For example, data point 522 may be an EEG scan identified to indicate a first type of seizure, and data point 523 may also be identified by an expert to indicate a first type of seizure. It can be an EEG scan, data point 526 can be an EEG scan identified to indicate normal brain activity. A second set of strongly labeled data points can be a different group of annotated EEG scans (possibly from the source). The use of three or more labeled datasets is also being considered and may be useful for labeling and retraining. In some embodiments, the first dataset may be used to retrain the model and the second dataset may be used to evaluate the performance (ie, accuracy) of the retrained model. ..

Diagram 500 may represent, for example, a 2D plot of data points generated by assigning (X, Y) coordinates to various data points. The X and Y values can be assigned via the algorithm and the algorithm can be iteratively modified until the data points in the labeled group are plotted in separate clusters with similarly labeled data points. For example, a set of labeled EEGs may be labeled as indicating a first type of seizure (“seizure EEG”) and some labeled as indicating normal brain activity. EEG (“normal EEG”) may be included. In the first iteration of the algorithm, each of these EEGs can be plotted using the X coordinate determined based on the file size of the EEG and the Y coordinate determined based on the EEG creation date. This makes it unlikely that the seizure EEGs will be plotted close to each other (or the normal EEGs will be plotted close to each other). Therefore, the algorithm could be modified to determine and re-evaluate the (X, Y) coordinates in different ways, and this process would allow all seizure EEGs to have a specified radius without the usual EEG nearby. It can be plotted within one region (first cluster) and repeated until all normal EEGs are plotted within a second region (second cluster) of the same radius without nearby seizure EEGs. If the algorithm reliably plots the labeled data points close to each other, the unlabeled data points can be considered to be properly plotted (eg, unlabeled to represent normal brain activity). EEG can be considered to be plotted in the second cluster).

It should be noted that the (X, Y) coordinates do not necessarily represent the characteristic properties of the EEG and they can be seen arbitrarily by an outside observer. In other words, the diagram 500 does not have to be a plot of data on well-understood metrics such as mean, frequency and magnitude. Simply put, axes are necessary, and as long as the algorithm plots data points with the same label in the same cluster, the system will have the same algorithm unlabeled data in the appropriate cluster. It can have a relatively high reliability of plotting.

In particular, all the data points in Diagram 500 are relatively separated into separate groups or "clusters" so that all members of a given cluster are located relatively close to each other. In the exemplary diagram 500, data points 512, 522, 523 and 532 are in cluster 502, data points 514, 534 and 535 are in cluster 504, and data points 516, 526 and 536 are in cluster 506. .. The various strongly labeled data points also have one of three different labels, "A", "B", "C", respectively. In addition, all known labels for each given cluster in Diagram 500 are equal. For example, data points 522, 523, and 532 are all labeled "A". This can be the result of how the data points are plotted.

Entering the value of a data point into the algorithm, receiving the coordinates as the output of the algorithm, plotting the data points according to the output coordinates, and plotting the data points of a particular value relatively close to each other. Coordinates can be assigned to various data points by adjusting the algorithm up to. Thus, if the value of an unlabeled data point is entered into the algorithm, the cluster containing the resulting position may correspond to the "correct" label of the data point. For example, the data points 512 are plotted because the unlabeled data points 512 are plotted in the cluster 502 and the plotting algorithm prefers to plot the data points labeled "A" in the cluster 502. It can be inferred that it should be labeled "A". This process can be repeated for each data point in the unlabeled dataset, which results in the dataset being "weakened" and favorably retraining the model based on the observed data. To.

As a non-limiting example, "A" may refer to a first type of seizure, "B" may refer to normal brain activity, and "C" may refer to a second type of seizure. Therefore, the algorithm shows all EEGs indicating a first type of seizure in cluster 502, all EEGs indicating normal brain activity in cluster 504, and all EEGs indicating a second type of seizure in cluster 506. EEG can be plotted. Even if the exact nature of the sort seems arbitrary or is not well understood (eg, the X-axis and Y-axis do not correspond to well-known properties of the data), the algorithm does EEG. You can confirm that they are sorted reliably. Therefore, if the same algorithm plots unlabeled EEGs within cluster 502, it can be inferred that the unlabeled EEGs indicate a first type of seizure.

FIG. 6 shows a high level method 600 for updating a machine learning model based on anonymized feature data derived from unlabeled input data, consistent with some embodiments of the present disclosure. In general, method 600 includes some actions that may be substantially similar to those performed by method 400. However, Method 600 retrains the model based on unlabeled "anonymized" features (ie, feature data) based on the data itself collected (ie, input data). This can be advantageous, for example, if the data collected by the device performing method 600 can be highly confidential, such as personally identifiable information (PII), health information, and the like.

Method 600 comprises acquiring labeled and unlabeled input data in operation 602. Operation 602, for example, records unlabeled data through one or more sensors, downloads a labeled dataset from an online database, and loads the dataset from a local storage device. It can include things like doing. For example, operation 602 may include receiving unlabeled EEG data from a set of electrodes and downloading one or more labeled EEG data sets from a public database.

Method 600 further comprises inputting unlabeled input data into the machine learning model in operation 604. The operation 604 may be performed in a manner substantially similar to (or as part of) the normal operation of a machine learning model, such as a machine learning model performed on a device performing method 600.

Method 600 further comprises generating unlabeled feature data (ie, one or more unlabeled features) in operation 606. A machine learning model can include multiple "intermediate" layers, each of which receives input features, manipulates them, produces output features, and inputs them to subsequent layers. Operation 606 may include receiving features output from one of these layers. In general, the features themselves can be relatively "abstract" and can be esoteric / meaningless in nature except when used as an input to the next layer of a machine learning model. This is an effect of the nature of the training method (by repeatedly adjusting different layers) of the machine learning model. For human observers, and for various machine learning models, the middle layer (sometimes called the "hidden layer") acts as a "black box." However, they are usually deterministic (although this can vary from model to model), and input data trends are generally reflected in these intermediate features. Therefore, an intermediate feature of a machine learning model presents a viable alternative to raw data when it can be sensitive. In addition, features generated in the high-level or middle layers of the model can capture useful information from raw input data (eg edges, corners, homogeneous patches, etc.) and therefore compared to the input data. , Can be more effective in distinguishing target classes. In some embodiments, the machine learning model may be configured to output a particular feature vector in addition to or instead of its typical output. For example, a machine learning model with 50 layers may be configured to output the feature vector generated by layer 37 in addition to the final output. The feature vector of layer 37 can then be used as the "generated unlabeled feature data" of operation 606.

Method 600 further comprises inputting the labeled input data into the same machine learning model in operation 608. The operation 608 can be performed in substantially the same manner as the operation 604. The label for each input data point can be retained or cached to allow labeling of the corresponding generated feature data.

Method 600 further comprises generating feature data labeled with operation 610. The operation 610 can be performed in substantially the same manner as the operation 606. In addition, the generated features can be labeled based on the label associated with the corresponding input data.

Method 600 further comprises labeling unlabeled feature data based on a comparison between the labeled feature and the unlabeled feature in operation 612. Operation 612 compares unlabeled features with labeled features and with unlabeled features and labeled features, for example, in a manner substantially similar to operation 406 of method 400. May include calculating the difference between. The operation 612 may further include determining the label of each feature contained in the unlabeled features generated via the operation 606 in a manner substantially similar to the operation 408 of the method 400. The various features can differ significantly from the initial input data from which the features were generated, but due to the nature of the machine learning model, comparing labeled and unlabeled features is still equally useful. , (Similar to the example discussed with reference to FIG. 5, by plotting features into clusters, etc.), it is possible to label unlabeled feature data in a similar manner.

Method 600 further comprises retraining the machine learning model via the newly labeled (“weakly” labeled) feature data in motion 614. In some embodiments, motion 614 may further include retraining via feature data that is strongly labeled and feature data from one or more teacher models. Operation 614 may include, for example, entering a weakly labeled feature in the "next layer" of the model. As an example, the feature data labeled at the time generated in operation 606 can be output from layer 37 of the 50 layer model. Operation 614 inputs weakly labeled feature data into layer 38 of the model, receives output from the model, calculates the error between the output and the label, and is based on the error. It may include adjusting one or more of layers 38-50 of the model and repeating until the error is no longer reduced. In some embodiments, motion 614 further comprises training the model with a strongly labeled dataset, in addition to training with weakly labeled feature data.

Operation 614 is performed on a "working copy" of the model without changing the "live" version of the model, taking into account the possibility that retraining may not improve (or reduce) performance. Can be. The model can then be evaluated and updated as needed in a manner substantially similar to the operations 412-416 of Method 400. In some embodiments, strongly labeled data can be input into the teacher model to generate teacher output, using the combination of strongly labeled data and weakly labeled data to teach. The student model can be retrained along with the output. In some embodiments, operation 614 may include retraining using complex knowledge distillation, such as via method 100.

FIG. 7 is a diagram 700 showing an example of labeling of unlabeled input data (such as sensor data) via inductive learning, consistent with some embodiments of the present disclosure. Diagram 700 is presented as an exemplary example of how Method 600 can be performed. The unlabeled input data 702 may be collected via observations such as from one or more sensors. The unlabeled input data 702 is input to the machine learning model 701. Model 701 may output unlabeled feature data 712.

The known data 704 may include a plurality of datasets, such as a first labeled input data 706 and a second labeled input data 708. Both input data sets 706 and 708 are input to model 701, resulting in labeled feature data 716 and labeled feature data 718, respectively. For clarity, model 701 may be run multiple times per dataset. For example, each of the input datasets 702, 706, and 708 may contain five data points. Therefore, the model 701 has one labeled input in the known input data set 706, once for each unlabeled input data point in the unlabeled input data set 702 (five features 712 are obtained). Once per data point (5 labeled features 716 are obtained) and once per labeled input data point 708 of a known input dataset (5 labeled features 718 are obtained) Can be performed 15 times. These iterations of model 701 can be performed in any order.

In particular, the model 701 can usually be configured to output a classification (not shown in FIG. 7), but for the purpose of updating the model 701, the model 701 is specific in addition to / classification instead of classification. It may be reconfigured to output intermediate features (such as unlabeled feature data 712). This reconstruction can be temporary or permanent. In some embodiments, the system may be configured to fetch the output of the layer without reconfiguring the model 701. The output of specific feature data can vary between update attempts as long as the same features are utilized for each dataset (unknown and known) within the same trial. For example, a first update attempt, such as a first iteration of method 600, in which model 701 may be configured to output the feature vector generated by layer 42, may be performed. A second update attempt, such as a second iteration of method 600, in which the model 701 is configured to output the feature vector generated by layer 37 may be performed.

The unlabeled feature data 712, the strongly labeled feature data 716, and the strongly labeled feature data 718 are input to the conversion algorithm 705. The conversion algorithm 705 may, for example, organize each of the features 712-716 into clusters, each cluster containing a feature with a particular label. Thus, each of the unlabeled features 712 can be labeled based on inference (ie, based on the cluster in which each of the features 712 is plotted). The result is a "weakly" labeled feature 722, which is combined with a strongly labeled feature 726 and a strongly labeled feature 728 to form a retraining feature 720, which is used. The model 701 can be retrained and evaluated.

Referring here to FIG. 8, a high level block diagram of an exemplary computer system 800 that may be configured to perform various aspects of the present disclosure, including, for example, methods 100, 400, and 600, is shown. An exemplary computer system 800 is the one or more methods or modules described herein (eg, using one or more processor circuits or computer processors of a computer), and the embodiments of the present disclosure. It can be used to implement any related function or behavior. In some embodiments, the main components of the computer system 800 are one or more CPUs 802, a memory subsystem 808, a terminal interface 816, a storage interface 818, an I / O (input / output) device interface 820, and The network interface 822 may all be communicable directly or indirectly for inter-component communication via the memory bus 806, I / O bus 814, and I / O bus interface unit 812. ..

The computer system 800 may include one or more general purpose programmable central processing units (CPUs) 802, some or all of which are cores 804A, 804B, 804C, collectively referred to herein as CPU802. , And 804D may be included. In some embodiments, the computer system 800 may include multiple processors typical of a relatively large system, whereas in other embodiments, the computer system 800 is an alternative, single CPU system. Can be. Each CPU 802 may execute instructions stored in the memory subsystem 808 on the CPU core 804 and may include one or more levels of onboard cache.

In some embodiments, the memory subsystem 808 may include a random access semiconductor memory, a storage device, or a storage medium (either volatile or non-volatile) for storing data and programs. In some embodiments, the memory subsystem 808 may represent the entire virtual memory of the computer system 800 and may also include virtual memory of another computer system coupled to or connected via a network to the computer system 800. obtain. The memory subsystem 808 can be conceptually a single monolithic entity, but in some embodiments the memory subsystem 808 can be a more complex arrangement, such as a hierarchy of caches and other memory devices. .. For example, memory can reside in multiple levels of cache, and these caches can be further divided by function, so one cache holds instructions and another cache is used by one or more processors, non. Holds instruction data. As is known in any of the various so-called non-uniform memory access (NUMA) computer architectures, memory can be further distributed and associated with different CPUs or sets of CPUs. In some embodiments, the main memory or memory subsystem 804 may include elements for control and flow of memory used by the CPU 802. This may include a memory controller 810.

Although the memory bus 806 is shown in FIG. 8 as a single bus structure that provides a direct communication path between the CPU 802, the memory subsystem 808, and the I / O bus interface 812, there are several memory buses 806. In an embodiment of, it is placed in one of various forms, such as a hierarchical, star, or web-configured point-to-point link, multiple hierarchical buses, parallel and redundant paths, or any other suitable type of configuration. It may contain several different buses or communication paths. Further, while the I / O bus interface 812 and the I / O bus 814 are shown as a single unit, the computer system 800, in some embodiments, is a plurality of I / O bus interface units 812. , Multiple I / O buses 814, or both. Further, a plurality of I / O interface units are shown that separate the I / O bus 814 from various communication paths to various I / O devices, but in other embodiments, some of the I / O devices. Or all may be directly connected to one or more system I / O buses.

In some embodiments, the computer system 800 has little or no multi-user mainframe computer system, single user system, or direct user interface, but receives requests from other computer systems (clients). It can be a server computer or a similar device. Further, in some embodiments, the computer system 800 is a desktop computer, portable computer, laptop or notebook computer, tablet computer, pocket computer, telephone, smartphone, mobile device, or any other suitable type of electronic. Can be implemented as a device.

It should be noted that FIG. 8 is intended to show the representative key components of an exemplary computer system 800. However, in some embodiments, the individual components can be more or less complex than those shown in FIG. 8, or there are components other than those shown in FIG. 8 or in addition to those components. It can be obtained or the number, type, and composition of such components can vary.

The present invention can be a system, method, and / or computer program product in any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or a plurality of media) having computer-readable program instructions on it for causing the processor to perform aspects of the invention.

The computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by the instruction executing device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination described above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory). ), Static Random Access Memory (SRAM), Portable Compact Disk Read-Only Memory (CD-ROM), Digital Versatile Disk (DVD), Memory Stick, Flop (Registered Trademark) Disk, Punch with instructions recorded on it Includes mechanically encoded devices such as cards or embossed structures in grooves, and any suitable combination described above. The computer-readable storage medium used herein is an electromagnetic wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (eg, an optical pulse through an optical fiber cable), or. It should not be construed as a temporary signal in itself, such as an electrical signal transmitted through an electric wire.

The computer-readable program instructions described herein are external from a computer-readable storage medium to each computing / processing device, or via a network such as the Internet, local area network, wide area network, wireless network, or both. Can be downloaded to your computer or external storage device. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers or combinations thereof. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers computer-readable program instructions for storage on a computer-readable storage medium within each computing / processing device. do.

The computer-readable program instructions for performing the operations of the present invention are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcodes, firmware instructions, state setting data, integrated circuit configuration data, or , Smalltalk®, object-oriented programming languages such as C ++, and any combination of programming languages, including procedural programming languages such as the "C" programming language or similar programming languages. It can be either source code or object code. Computer-readable program instructions are all on the user's computer, some on the user's computer, as a stand-alone software package, some on the user's computer, some on the remote computer, or all on the remote computer. It can be run on or on the server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or (eg, an internet service provider). Can be connected to an external computer (using via the Internet). In some embodiments, for example, an electronic circuit comprising a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) is used to personalize the electronic circuit in order to carry out aspects of the invention. Computer-readable program instructions can be executed by using the state information of the computer-readable program instructions.

Aspects of the invention are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the invention. It will be appreciated that each block of the flowchart and / or block diagram, and the combination of blocks of the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

These computer-readable program instructions are provided to the computer's processor, or other programmable data processing device, and can generate a machine, resulting in instructions executed through the computer or other programmable data processing device's processor. Creates means for performing a function / operation specified in one or more blocks of a flowchart and / or a block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that can instruct the computer, programmable data processor, and / or other device to function in a particular way, and as a result, in it. A computer-readable storage medium having stored instructions comprises a product comprising instructions that perform a mode of function / operation specified in one or more blocks of a flowchart and / or a block diagram.

Computer-readable program instructions can also be loaded onto a computer, other programmable data processor, or other device to perform a series of operating steps on the computer, other programmable device, or other device to complete the computer implementation process. As a result, the instructions that can be generated and executed on a computer, other programmable device, or other device implement the function / behavior specified in one or more blocks of the flowchart and / or block diagram. ..

The flowcharts and block diagrams in the figure show the architecture, function, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or part of an instruction that contains one or more executable instructions for implementing a given logical function. In some alternative implementations, the functions shown in the blocks may differ from the order shown in the figure. For example, two blocks shown in succession can actually be performed as one step and can be performed simultaneously or substantially simultaneously in a partially or totally temporally overlapping manner, or , The blocks may be executed in reverse order depending on the associated function. Also, each block of the block diagram and / or the flowchart diagram, and the combination of the blocks of the block diagram and / or the flowchart diagram perform a specified function or operation, or a combination of special purpose hardware and computer instructions. Also note that it can be implemented by a special purpose hardware-based system that runs.

The description of the various embodiments of the present disclosure is presented for purposes of illustration but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of skill in the art without departing from the scope and spirit of the embodiments described. The terminology used herein is to describe the principles of the embodiment, the actual application, or a technological improvement beyond the technology found in the market, or embodiments disclosed herein by those of ordinary skill in the art. It was chosen to help you understand.

Claims (20)

How to choose a machine learning model
At the stage of receiving the hardware specifications of the device,
At the stage of determining performance requirements based on the hardware specifications of the device,
At the stage of acquiring a set of machine learning models, the set of machine learning models is
A first machine learning model with a set of first layers,
With a teacher model, with stages,
A step of removing one or more layers of the first set of layers from the first machine learning model, with the result creating a student model with a set of pruned layers. ,
Training data and
Based on the teacher model
The stage of training the student model and
At the stage of evaluating the performance of the student model,
At the stage of comparing the performance of the student model with the performance requirements,
A method comprising: determining to select the model with the highest performance, at least based on the comparison steps. The method of claim 1, further comprising receiving additional performance requirements from the user, wherein the comparison stage further comprises comparing the performance of the student model with the additional performance requirements. The step of deleting one or more layers is
At the stage of determining the number of layers to be deleted based on the performance requirements,
The step of selecting the one or more layers based on the number, and
The method of claim 1 or 2, comprising the step of removing the selected layer. The method of claim 3, wherein the one or more layers are randomly selected. The step of deleting one or more layers has a step of deleting one or more blocks, and each block has a set of consecutive layers, according to any one of claims 1 to 4. The method described. The training stage is
At the stage of inputting training data into the student model and the teacher model,
At the stage of receiving student characteristics from the student model,
The stage of receiving teacher features from the teacher model and
The stage of performing the first comparison between the student characteristics and the teacher characteristics, and
At the stage of receiving student output from the student model,
The stage of performing a second comparison between the student output and the training data,
One of claims 1-5, comprising the step of adjusting at least one layer of the set of pruned layers of the student model based on the first comparison and the second comparison. The method described. The method according to any one of claims 1 to 6, wherein each of the set of machine learning models is selected from a model repository. It ’s a system,
With memory
A central processing unit (CPU) coupled to a memory is provided, and the CPU is
Receiving the hardware specifications of the device and
Determining performance requirements based on the hardware specifications and
To acquire a set of machine learning models, wherein the set of machine learning models has a first machine learning model having a set of first layers and a teacher model.
Removing one or more layers of the first set of layers from the first machine learning model results in a first student model having a first set of pruned layers. Created, deleted, and
Training data and
Based on the teacher model
Training the first student model and
To evaluate the performance of the first student model and
Comparing the performance of the first student model with the performance requirements,
A system configured to execute instructions to determine and do to generate a second student model, at least on the basis of said comparison. The CPU is further configured to receive additional performance requirements from the user, and the comparison further comprises comparing the performance of the first student model with the additional performance requirements. The system according to claim 8. Removing the one or more layers can
Determining the number of layers to remove based on the performance requirements
Selecting the one or more layers based on the number,
The system of claim 8 or 9, wherein the selected layer is removed. 10. The system of claim 10, wherein the one or more layers are randomly selected. 13. The system described. The training mentioned above
Inputting training data into the first student model and the teacher model,
Receiving student characteristics from the first student model and
Receiving teacher features from the teacher model and
Performing a first comparison between the student characteristics and the teacher characteristics,
Receiving student output from the first student model and
Performing a second comparison between the student output and the training data,
8-12, which comprises adjusting at least one layer of the first pruned layer set of the first student model based on the first comparison and the second comparison. The system according to any one of the above. The system according to any one of claims 8 to 13, wherein each of the set of machine learning models is selected from a model repository. On the computer
The procedure for receiving the hardware specifications of the device and
A procedure for determining performance requirements based on the hardware specifications of the device, and
A procedure for acquiring a set of machine learning models, wherein the set of machine learning models is
A first machine learning model with a set of first layers,
With the teacher model, the procedure to get,
A procedure for removing one or more layers of the first set of layers from the first machine learning model, resulting in a student model with a set of pruned layers being created and deleted. Procedure and
Training data and
Based on the teacher model
The procedure for training the student model and
The procedure for evaluating the performance of the student model and
A procedure for comparing the performance of the student model with the performance requirements,
A computer program that causes the decision to select the model with the best performance, at least based on the above comparisons. 15. 15. The listed computer program. The procedure for removing one or more layers is
A procedure for determining the number of layers to be deleted based on the performance requirements and
The procedure for selecting the one or more layers based on the number and
15. The computer program of claim 15 or 16, comprising the procedure of removing the selected layer. 13. The listed computer program. The training procedure is
The procedure for inputting training data into the student model and the teacher model,
The procedure for receiving student characteristics from the student model and
The procedure for receiving teacher features from the teacher model and
The procedure for performing the first comparison between the student characteristics and the teacher characteristics,
The procedure for receiving student output from the student model and
A procedure for performing a second comparison between the student output and the training data,
15. One of claims 15-18, comprising the procedure of adjusting at least one layer of the set of pruned layers of the student model based on the first comparison and the second comparison. Described computer program. The computer program according to any one of claims 15 to 19, wherein each of the set of machine learning models is selected from a model repository.

Images