BR112021009042A2 - automated generation of machine learning models - Google Patents

Abstract

automated generation of machine learning models. the present invention relates to the automated generation of machine learning models such as neural networks. an example system includes a hardware processing unit and a storage facility. the storage resource can store computer-readable instructions causes the hardware processing unit to perform an iterative model growth process that involves modifying parent models to obtain child models. the iterative model growth process may also include selecting candidate layers to include in the child models based on at least learned weights in an initialization process of the candidate layers. the system can still output a final model selected from the child models.

Description

Descriptive Report of the Patent of Invention for "AUTOMATED GENERATION OF MACHINE LEARNING MODELS". background

[001] Traditionally, machine learning models were built manually by experts who would define a model structure and then use automated techniques to train the model. As machine learning models have become more complex, several attempts have been made to automate the process of generating machine learning models. However, these efforts have met with limited success. summary

[002] This Table of Contents is provided to present a selection of concepts in a simplified form which are described in more detail below in the Detailed Description. This Table of Contents is not intended to identify key features or essential features of the claimed subject, nor is it intended to be used to limit the scope of the claimed subject.

[003] The present invention generally refers to techniques for automated generation of machine learning models. An example includes a method or technique that can be performed on a computing device. The method or technique may include performing two or more iterations of an iterative model growth process. The iterative model growth process may include selecting a particular parent model from a parent model group of one or more main models, generating a plurality of candidate layers, and initializing the plurality of candidate layers while reusing learned parameters and/or the structure. of the particular parent model. The iterative model growth process may also include selecting particular candidate components as layers to include in child models for training. The respective child models can include the particular parent model and one or more of the particular candidate layers or other structures. The iterative model growth process may further include training the plurality of child models to obtain trained child models, and evaluating the trained child models using one or more criteria. The iterative model growth process may also include designating an individual trained child model as a new parent model based on at least evaluation and adding the new parent model to the parent model group. The method or technique may further include selecting at least one trained child model as a final model after the two or more iterations, and emitting the final model.

[004] Another example includes a system that involves a hardware processing unit and a storage facility. The storage facility can store computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to perform an iterative model growth process that involves modifying parent models to obtain child models. The iterative model growth process may include selecting candidate layers to include in child models based on at least weights learned in an initialization process of the candidate layers. Computer-readable instructions can also cause the hardware processing unit to output a final model selected from the child models.

[005] Another example includes a computer-readable storage medium that stores instructions that, when executed by a processing device, cause the processing device to perform actions. Actions can include performing two or more iterations of an iterative model growth process. The iterative model growth process may include selecting a particular parent model from a parent model group of one or more parent models, initializing a plurality of candidate layers, and selecting a plurality of child models for training. The respective child models can include a structure inherited from the particular parent model and at least one of the candidate layers. The iterative model growth process may further include training the plurality of child models to obtain trained child models, and designating an individual trained child model as a new parent model based on at least one or more criteria. The iterative model growth process may also include adding the new parent model to the parent model group. Actions may also include selecting at least one trained child model as a final model after the two or more iterations, and emitting the final model.

[006] The examples listed above are intended to provide a quick reference to assist the reader and are not intended to define the scope of the concepts described in this document. Brief Description of Drawings

[007] The detailed description is described with reference to the attached Figures. In Figures, the leftmost digit(s) of a reference number identifies the Figure in which the reference number first appears. The use of similar reference numbers in different instances in the description and Figures may indicate similar or identical items.

[008] Figure 1 illustrates an example of a method or technique for automated generation of machine learning models, consistent with some implementations of the present concepts.

[009] Figure 2 illustrates an exemplary approach to generate candidate layers of a machine learning model,

consistent with some implementations of the present concepts.

[0010] Figure 3 illustrates an exemplary approach to initializing candidate layers of a machine learning model, consistent with some implementations of the present concepts.

[0011] Figure 4 illustrates another exemplary approach to initializing candidate layers of a machine learning model, consistent with some implementations of the present concepts.

[0012] Figure 5 illustrates an exemplary approach to training a child model, consistent with some implementations of the present concepts.

[0013] Figures 6-8 illustrate scatter plots associated with consecutive iterations of an iterative model growth process, consistent with some implementations of the present concepts.

[0014] Figure 9 illustrates an example of a processing flow for automated generation of machine learning models, consistent with some implementations of the present concepts.

[0015] Figure 10 illustrates an example of the system, consistent with some implementations of the present concepts.

[0016] Figure 11 illustrates an example of the graphical user interface, consistent with some implementations of the present concepts. Detailed Description Overview

[0017] There are several types of machine learning frameworks that can be trained using supervised and/or unsupervised learning. Support vector machines, decision trees, and neural networks are just a few examples of machine learning frameworks suitable for supervised learning, where models can learn from labeled training data. Some machine learning frameworks, such as neural networks, use layers of nodes that perform specific operations.

[0018] In a neural network, nodes are connected to each other through one or more edges. A neural network may include an input layer, an output layer, and one or more intermediate layers. Individual nodes can process their respective inputs according to a predefined function and provide an output to a subsequent layer or, in some cases, a previous layer. Inputs for a given node can be multiplied by a corresponding weight value for an edge between the input and the node. In addition, nodes can have individual bias values that are also used to produce outputs. Various training procedures can be applied to learn edge weights and/or polarization values. For the purposes of this document, the term "learned parameters" refers to parameters such as edges and polarization values that are learned by training a layered machine learning model, such as a neural network.

[0019] A neural network structure can be built in a modular way. For example, one or more layers of nodes can collectively perform a specific operation, such as a grouping or convolution operation. Then different layers can be connected to form the overall structure of the network. For the purposes of this document, the term "layer" refers to a group of nodes that share connectivity with one or more input layers and one or more destination layers that receive output from nodes in that layer. The term "operation" refers to a function that can be performed by one or more layers of nodes. The term "model structure" refers to an overall architecture of a layered model, including the number of layers, the connectivity of layers, and the type of operations performed by individual layers. The term "neural network structure" refers to the model structure of a neural network. The disclosed implementations primarily use neural network frameworks as example model frameworks for layered machine learning models. The term "trained model" refers to a model structure along with the learned parameters for the model structure. Note that two trained models can share the same model structure and still have different parameters learned, for example if the two models trained on different training data or if there are underlying stochastic processes in the training process.

[0020] As noted earlier, one way to generate a model structure is for a human to manually define the model structure. Then the model structure can be trained on some training dataset by a computer to get a trained model, and then the trained model can be validated using a validation dataset. Later, modifications to the model structure can be generated manually, for example, adding or removing layers or connections between layers. Then, the modified frameworks can be retrained from scratch to obtain additional trained models, and the additional trained models can be compared against each other to select a final model and corresponding framework that works well for a given task. However, this approach requires the involvement of a human with domain expertise to create the initial model structure and modifications, and also to select the final model structure.

[0021] Another approach is to automate the process using a computer to generate different model structures and select a final model structure among the generated structures. However, previous efforts to automate the generation of model structures have had limited success. Although modern computers have advanced tremendously in computational power, existing approaches to automated generation of model structures, such as neural network structures, tend to exploit limited search spaces or require impractical amounts of computational resources. In practice, generating model structures tends to be computationally feasible, but independent training of many different model structures tends to be computationally infeasible given the computing hardware currently available.

[0022] One way to quantify the training time for a model structure is to define a reference computational resource, such as a virtual machine or a processor, and the amount of time that training takes to complete on the reference computational resource. For example, a day of training on a specific type of graphics processing unit ("GPU") can be referred to as a GPU day, and the computational cost of training a particular model can be specified as a number of GPU days. Other approaches are also possible, for example the number of hours of training on a given CPU or FPGA model, the amount of time spent in a virtual machine, etc.

[0023] One approach to automating the generation of model structures is to simply define a search space and generate all model structures within the search space. Then, all possible model structures can be trained independently to obtain trained models and one of the trained models can be selected as a final model. Generally, the search space can be defined by constraining the depth of the model structure, the operations performed by the model, and the connectivity of the model. For example, a computer can be programmed to consider all fully connected model structures of depth five, where each of the five layers performs one of several possible operations.

[0024] However, this approach is not feasible for many models. First, model structures can grow very large, with many different layers and nodes. Second, training can take a long time for such large model structures. Finally, this approach does not consider other model structures outside the search space that may be better suited to the task at hand. For example, if the task is image recognition and a six-layer model is significantly better for this task than any plausible five-layer model, the six-layer model will not be found if the search space is constrained to five layers. .

[0025] A more sophisticated approach to automated generation of model structures involves creating a separate "controller" model that is trained to generate new model structures for a given task. However, previous efforts to use controller models to generate new model structures still suffer from some of the drawbacks mentioned above. If a large search space is considered, for example the controller model is expected to consider a wide range of potential model structures, each model structure needs to be generated, fully trained and then evaluated against other models. Because training can take days on large datasets, efforts to automate learning new model structures with relatively unconstrained search spaces have met with limited success. For example, such approaches can take hundreds or thousands of days per GPU to produce a final model framework with acceptable levels of performance.

[0026] Another approach to automated generation of model structures is to narrow the search space significantly and then search among a relatively limited set of models within the restricted search space. For example, some previous approaches define an external model skeleton with a specified number of modularized layers and types of connections between modules. These approaches then generate and evaluate different candidate substructures or "cells", which are then repeated within the skeletons of the external model. However, these approaches do not consider alternative external skeletons, alternative connections between cells, or the use of different types of substructures in each cell.

[0027] In contrast to the shortcomings of the conventional techniques described above, the disclosed implementations can generate model structures in a relatively unconstrained search space, while saving significant training time over the techniques discussed above. When considering a large search space, disclosed implementations can potentially find new model structures that offer better performance than would be possible in a more restricted search space. For example, disclosed implementations may find model structures that are not limited to a predefined external skeleton, and they may also find model structures that have various types of operations on different layers of the model.

[0028] The disclosed implementations may use various techniques to avoid separately generating and training each model structure in the search space. Instead, the disclosed implementations guide the growth of new model structures towards a portion of the search space that is expected to contain improved model structures over structures that have already been evaluated, avoiding looking for other portions of the search space. that are less likely to contain improved model structures. Model Structure Generation Method

[0029] The following discussion presents an overview of functionality that can allow the automated generation of model structures, such as neural network structures, to be performed. Figure 1 illustrates an example method 100, consistent with the present concepts. As discussed further below, method 100 can be implemented on many different types of devices, for example by one or more cloud servers, by a client device such as a laptop, tablet or smartphone, or by combinations of one or more servers. , client devices, etc.

[0030] Method 100 starts at block 102, where a set of parent models is initialized. For example, one or more initial parent models can be added to the set of parent models. Initial parent models can be known models that have been predetermined to perform well on a specific task, they can be randomly generated, or they can simply be predefined default seed models.

[0031] Method 100 continues at block 104, where a particular parent model is selected from the group. For example, master models can be selected from the group randomly or deterministically. The particular parent model has a corresponding model structure that can be modified, as discussed in more detail below.

[0032] Method 100 continues at block 106, where candidate layers are generated and initialized. Generally speaking, generating a candidate layer may include selecting one or more layers from the particular parent model to provide inputs to the candidate layer, and selecting another "target" particular parent model layer to receive output from the candidate layer. Thus, candidate layers can be considered as additions to the structure of the particular parent model. Furthermore, generating a candidate layer may include selecting operations to perform on inputs provided by selected layers of the particular parent model. Initializing a candidate layer may include performing some initial training on the candidate layer, as discussed later in this document.

[0033] Method 100 continues at block 108, where particular candidate layers are selected to include in one or more child models derived from the parent model. For example, particular candidate layers can be selected based on initialized parameters learned when initializing candidate layers, as discussed in more detail below. In other cases, each of the candidate layers is selected for inclusion in a child model. In any case, a child model can be considered a model that inherits the parent model structure and additionally includes one or more of the selected candidate layers.

[0034] Method 100 continues in block 110, where child models are trained. In some cases, child models are trained by reusing parameters learned from parent models, as discussed in more detail below. In these cases, the child models may still be considered to initially inherit learned parameters from the parent model, although these learned parameters may still be adjusted during training of the child models.

[0035] Method 100 continues at block 112, where the trained child models are evaluated against one or more criteria. Generally, the criteria can relate to the performance of the model in a given task, for example, accuracy and/or other factors such as the computational cost of training the child model.

[0036] Method 100 continues at block 114, where an individual child model is designated as a new parent model based on the evaluation in block 112. The new parent model is added to the parent model group.

[0037] Method 100 continues at decision block 116, where a determination is made whether a stop condition has been met. The stop condition can define a specified amount of compute resources to be used (e.g. a budget in GPU days), a specified performance criterion (e.g. a threshold precision), a specified amount of time, etc.

[0038] If the stop condition has not been reached, the method continues at block 104, where subsequent iterations of blocks 104-116 can be performed. Generally speaking, blocks 104-116 can be considered an iterative process of model growth that can be repeated over multiple iterations until a stop condition is reached.

[0039] If the stop condition has been reached, the method moves to block 118, where a final model is produced. For example, the final model may be selected from all child models according to one or more criteria, such as those discussed above in relation to block 112.

[0040] In many cases, method 100 is performed to generate models that are suitable for a specific application. For example, in a face recognition scenario, the training data might include labeled examples of images, indicating whether faces are included in the images and also where the faces are located in the images. In a scene segmentation example, the training data might include labeled examples of video that was segmented into predetermined segments. By iteratively generating new child models and evaluating them against a training dataset for a specific task, method 100 can produce a final model that performs well on that specific task. Candidate Layer Generation

[0041] Figure 2 shows a parent model 210 and a candidate layer 220. The following describes an example of how candidate layers can be derived from parent model structures. Note that Figure 2 represents an approach to generating multiple candidate layers from a given parent model, rather than single instances of a parent model and a candidate layer, as discussed in more detail below.

[0042] In Figure 2, elements from the parent model are shown in solid lines and elements from the candidate layer are shown in dotted lines. In this case, the parent model includes model inputs 211, layers 212, 213, 214, and 215, and model outputs 216. The parent model may also include one or more other layers that are not shown, represented by the ellipses shown in Figure 2. Generally, model inputs can include features that are processed by the model, such as raw image, video, audio, and/or text data. The outputs can represent computational results of the model, for example, identification of faces in images, segmented video, transcribed audio, semantic representations of text, etc.

[0043] In this example, the candidate layer includes individual operations 221, 222, and 223 and aggregate operation 224. Generally, individual operations 221, 223, and 223 may involve convolutions, grouping, etc., as discussed later in this document. The aggregate operation 224 may involve manipulating the outputs of the individual operations to conform to a target layer of the parent model that will receive the output of the aggregate operation, for example, the layer 215. For example, the aggregate operation may concatenate the outputs of the individual operations and project them to a shape or shape that matches the input from the target layer.

[0044] As discussed above, block 106 of method 100 can be used to generate candidate layers. Generating candidate layers may involve selecting one or more input layers from the parent model, selecting one or more target layers from the parent model to receive the candidate layer's outputs, and selecting one or more operations to be performed by the candidate layer on the inputs. Furthermore, generating candidate layers may include selecting operational parameters of operations, e.g. convolution sizes, convolution steps, cluster windows, etc.

[0045] In this example, the search space of possible candidate layers can be defined as follows. Let x1, ..., xL be all existing layers in the parent model. A candidate layer can be defined by a tuple (xout,xin,1,op1,xin,2,op2, ..., xin,J,opJ), where J is a positive integer, xout,xin,1,. ..,Xίn,k are existing layers and op1...opJ are operations like convolutions and grouping. xout can be defined as strictly behind all xin,i in the topological order of the parent model's computational graph, so no direct cycle can be formed. A candidate layer can be computed from , and then added to take inputs from one or more input layers and output its output to xout.

[0046] A specific algorithm for forming a candidate layer xc is as follows. First, randomly sample a target layer x0ut from layers that were in the parent model 210. Next, three input layers xin,i for i = 1,2,3 can be chosen. To ensure that xc has access to local layers, xin1 can be selected as the deepest entry of xout that was in the initial parent model xin,2 and xin,3 can be shown with evenly random replacement of all layers of the parent model that are topologically earlier than xout. Next, the operations to be performed on each input layer can be selected evenly at random from a group of operations.

[0047] For example, the operation group can be predefined. Specific examples of operations include convolution, grouping, and identity operations. Each operation type can have different operation parameters. For example, convolution operations can have kernel size parameters specified - 1x1, 3x3, 5x5, 7x7, etc. Convolutions can also have a filter size parameter, for example 16, 32, or 64 filters and so on, as well as step parameters, fill parameters, etc. Grouping operations can include maximum and average grouping operations and can be applied to a window that varies based on a window size parameter. These parameters are generally referred to in this document as "operating parameters" to distinguish them from "learned parameters", such as weights and bias values obtained through model training. The generation of candidate layers can include the selection of different operations and/or different operation parameters, deterministically or randomly.

[0048] In some implementations, separable convolutions may be applied twice. The outputs of each operation can be concatenated via aggregate operation 224 and then projected to the same shape as the destination layer xout using a 1x1 convolution. The result is the candidate layer xc.

[0049] Some implementations may restrict candidate layer connectivity to reduce search space. At depth i in a given parent model, there are i - 1 potential initial layers and any subset of the previous layers can be chosen as input to the candidate layer. As a consequence, there are an exponential number of input options for the candidate layer. Therefore, some implementations may limit input options by learning only a repeatable cell module and deploying the cell in a manually designed skeleton macro structure. In these implementations, layers in cell modules can only receive input from other layers in the same cell and from the outputs of the two most recent previous cells. Other implementations are not limited to any specific skeleton or connectivity arrangement and may allow sparse hop connections between arbitrary layers of a model skeleton that vary as the model structure grows.

[0050] Block 106 of method 100 may involve generating one or more candidate layers using the approach discussed above for each iteration of the method. In some cases, model structures can include thousands or millions of different layers that are connected in any number of different combinations, and thus the number of potential candidate layers that can be generated from a single parent model in a single iteration can be large. The following discusses some approaches that can reduce the computational burden of independently training the entire space of possible child model structures.

Candidate Layer Initialization

[0051] As noted above, block 106 of method 100 may also include initialization of candidate layers. Generally, candidate layer initialization can serve several purposes. Firstly, initializing the parameters of candidate layers allows child models to be trained from initialization rather than from scratch. In other words, the child models are already partially trained when the final training takes place in block 110 of method 100. Second, candidate layer initialization can provide information about candidate layers so that candidate layers can be selectively added to models. parent, while omitting other candidate layers, as discussed further below.

[0052] Figure 3 illustrates exemplary initialization operations for an implementation where the candidate layer 220 is trained, initially preventing the candidate layer from affecting the main model 210. To do this, operations 221, 222 and 223 can be augmented with a stop gradient (sg) operation that applies to the respective inputs of each operation. sg(x) is x during forward propagation and is zero during backpropagation. During backpropagation, gradients are calculated for each layer to adjust learned model parameters. The sg operation prevents the gradient from affecting any of the parameters learned in any previous layer of the parent model.

[0053] A stop-forward (sf) operation 302 can be applied to the output of the aggregate operation 224 before the output is provided to the target layer 215. Here, sf(x) is zero during forward propagation and is the identity function during backward propagation. This allows candidate layer 220 to receive gradient information during initialization without affecting the target layer.

Therefore, during initialization, the candidate layer can accumulate the gradient of the loss with respect to the target layer, without actually affecting the values emitted by the target layer or any subsequent outputs.

[0054] Thus, at the initialization of the candidate layer, the parameters learned from the parent model can remain stable while determining the initialized values of the parameters learned for the candidate layer. In some cases, different candidate layers generated in the same iteration may share margins. When this is the case, weights for the shared margins can be independently initialized and/or trained for each candidate layer. Alternatively, initialization and/or training can be performed so that different candidate layers share weights for margins they have in common. For example, forward propagation and backpropagation operations can be shared during initialization and/or training of shared margins and performed independently for other margins that are not shared between candidate layers. Significant savings in computational costs can be realized by shared initialization and/or margin training on a given candidate layer and/or child model.54 Pruning Candidate Layers

[0055] In some cases, block 108 of method 100 may involve selecting all candidate layers generated in block 106 for inclusion in child models for subsequent training and evaluation. When this is the case, method 100 can still provide significant benefits over previous techniques, because only certain child models are designated as parent models in block 114. As a consequence, the child models in each iteration are derived from known good parent models. . This approach allows a significant part of the search space to be omitted when developing new child models.

[0056] However, depending on the potential operations, operation parameters and connectivity types under consideration, the number of possible candidate layers in any iteration can be very large. Therefore, the number of possible immediate child models of a given parent can be large. As a further refinement to reduce the computational burden of training new child models, some implementations may filter out certain candidate layers before training the child models in block 110. This can further reduce the space of child models that need to be trained and evaluated in each iteration, as discussed above.

[0057] One approach to reducing the number of children for training is to randomly sample the possible candidate layers in block 108 of method 100, so that fewer children are trained. In other implementations, block 108 may involve using initialized parameters from candidate layers to determine which candidate layers should be incorporated into child models for training. Consider a scenario where multiple candidate layers share connectivity to one or more input layers and a target layer of a given parent model, still performing different operations. Different weights can be initialized to the margins of the candidate layers that are inserted and/or emitted by the different operations. These initialized weights can be used to select certain candidate layers to include in child models for further training. Other candidate layers can be pruned so that no child models with these candidate layers are trained, thus saving additional training time.

[0058] Figure 4 illustrates a technique for using initialized parameters to prune candidate layers that are unlikely to be useful. Specifically, Figure 4 shows a scenario where multiple operations can be initialized together by deriving an aggregate candidate layer 402 from a parent model 210. Generally speaking, aggregate candidate layer 402 represents multiple candidate layers that share connectivity to the parent model, but perform different operations. As discussed further below, Figure 4 offers an alternative to adding all candidate layers to the main models to get the child models, or randomly selecting candidate layers. Note that, for brevity, Figure 4 omits layer 212 that was shown in Figures 2 and 3.

[0059] The following is a specific algorithm for pruning candidate layers. For each input xin,i, the possible operations up to xin,i can be instantiated as opi,1,...opi,k, where k is the number of possible operations. In Figure 4, these inputs are provided by layers 213 and 214, for example. Given the inputs J xin,i this gives a total of tensors Jk o1,o2,...oJK. These operations can be trained simultaneously. Once trained, a subset of the operations can be summed to end an aggregate candidate layer operation 404 via the formula xc as xc =

[0060] Once trained, the aggregate candidate layer 402 includes different weights for edges connected to different operations. Thus, the aggregate candidate layer can be thought of as multiple candidate layers initialized with different sets of edge weights connecting different operations. These weights can be fed into a feature selection technique to select one or more of the candidate layers for use in child models and similarly prune one or more other candidate layers so that they are not used in child models. This resource selection process is represented in Figure 4 by the selected trade 406. Note that Figure 4 also shows the employment of the stop-forward trade 408 and stop-gradient trades 410 and 412 in a similar manner as discussed above with respect to Figure 3.

[0061] One approach to achieve sparse feature selection in selection operation 406 is to linearly combine options o1,...oJK and use L - 1 norm regularization such as absolute reduction and selection operator ("lasso"), in linear weights. Scattering can be applied by adding the following regularization to the overall loss: where ƛout is a parameter associated with destination layer 215 (xout) to manage the sparsity level. Another alternative is to replace linear loss with squared loss as follows:

[0062] Note that in some implementations and thus the previous equation is equivalent to:

[0063] Viewed from one perspective, the implementations discussed above employ feature selection and learned parameter sharing to initialize a combinatorial number of candidate layers and select a subset of them to include in child models for further training. This approach may favor the selection of candidate layers that are likely to improve model performance and disfavor the selection of candidate layers that are unlikely to improve model performance. As a consequence, fewer total child models need to be trained for evaluation as potential parent models, and the trained child models are more likely to produce offspring that exhibit desirable performance. Son Model Training

[0064] After initialization, a given child model can include learned parameters inherited from the parent model and also initialized parameters for any candidate layers included in this child model. As discussed above, block 110 of method 100 can include training of child models as a whole, for example, candidate layers can be trained together with the rest of the parent model. This can be seen as a "hot start" technique, where the child model has both initialized candidate layer parameters and parent model parameters inherited at the beginning of training, both of which can be modified when training a given child model as a whole.

[0065] Figure 5 illustrates an example approach to training a child model. In Figure 5, stop-gradient operations on entries can be removed. In addition, the stop-

forward can be replaced by a scalar multiplier of 502. The scalar is trainable and can be initialized to be zero. Therefore, immediately after initialization, the child models represent the same functions as the parent models. Likewise, the child model has a different structure than the parent model, but the functionality of the parent model is preserved. The child models are then trained from the combination of parent parameters and initialized candidate parameters, as the scalar multiplier can change over time and the added candidate layer slowly starts to contribute to the target layer and subsequent model outputs. This approach can prevent candidate layers from destabilizing learned parameters inherited from the parent model, which can be close to ideal given that the parent model has been fully trained.

[0066] Note that some implementations may omit the sg and sf operations shown in Figure 3 and instead immediately allow candidate layers to affect the parent model while candidate layers are being trained. Using sf-sg as a pre- and post-correction of the candidate layer, as shown in Figure 3, can make the candidate layers converge faster. However, this may involve formulating additional goals for candidates during startup.

[0067] On the other hand, allowing values to flow freely between the parent and candidate layers allows the new candidate layers to directly contribute to adjusting the final loss. However, the initial values for candidate layers can be very far from ideal compared to models in the parent, and this can negatively affect the parameters of the parent model. An alternative approach to solve this problem is to use a learning rate to initialize the candidate layers that is much lower than the one used in training the parent model (0.1 ~ 0.02 times the original). Evaluate and Assign Child Models as Parents

[0068] As noted earlier, certain child models are added to the set of parent models in block 114 of the method

100. One approach to deciding which child models to add to the parent model pool involves using one or more criteria to predict which child models are likely to produce offspring that, in subsequent iterations, will exhibit improvements over previously discovered models. Generally, the criteria can consider factors such as the loss of a given child model as well as the computational effort to train a given child model. Higher loss implies lower accuracy during model validation and higher cost implies longer training times on a training dataset. Child models that exhibit relatively low loss and low training effort may be preferred for inclusion in the parent model group.

[0069] A specific approach to selecting child models for the parent set is shown here with respect to Figure 6. This Figure illustrates an example 600 scatterplot for various trained models. For each child model that completes training, the computational cost to train that child model can be calculated and plotted on the x-axis 602. In addition, the loss of this child model can be calculated and plotted on the y-axis 604. Since all models for a given iteration has been plotted, a lower convex hull 606 can be calculated from the plotted values. Note that the computational cost can be calculated as a value, such as GPU days, that reflects the amount of time required to train the model on standardized hardware (for example, a specific GPU model). In other implementations, the computational cost reflects the cost of testing time, for example, the number of operations involved in using a given model to make a prediction, regardless of specific training values, such as the number of training periods and/or or data augmentations involved in training. The computational cost can be normalized to a number between 0 and 1, as shown in Figure 6.

[0070] The lower convex hull 606 can be used as a mechanism to decide whether a particular child model is added to the parent model set. For example, a child model in the lower convex hull can be added to the set of parent models with a defined probability using the following specific algorithm. If m1 and m2 are two adjacent models in the hull, with computational costs c1 and c2 (c1 < c2), then the probability weight of in 1 can be defined proportionally to c2 - c1. The most accurate model, which does not have the next model on the curve, can be selected for inclusion in the parent model pool with probability 0.5. In Figure 6, the most accurate model is the 608 model, as this model has the lowest loss.

[0071] Generally, a lower convex hull is a subset of the Pareto efficiency and therefore another approach is to select child models in the Pareto efficiency for inclusion in the parent group. Either approach can provide good performance for selecting child models to add to the parent model group. One way to visualize the lower convex hull and/or the Pareto boundary is as follows. A given model on the lower convex hull or Pareto frontier cannot be improved with respect to one criterion by switching to another model on the lower convex hull/Pareto efficiency without degrading the other criteria.

[0072] Note that the same models may have different validation errors due to randomness in the formation of stochastic gradients. As a consequence, the lower convex hull or Pareto boundary can be relaxed with a multiplicative bandwidth. Thus, a child model whose validation error is within (1+y) times the validation error of the lower convex hull with the same computational cost can be considered to be in the lower convex hull and can be chosen as a parent. Some implementations may set y = 0.025. This approach allows certain child models that are close to the lower convex hull, but not strictly located on it, to still be designated as parent models.

[0073] Other approaches can also be used to allow child models that have locations within a predetermined neighborhood of the lower convex hull to be selected as parent models. For example, some implementations may define a lower convex hull threshold distance and allow child models within the lower convex hull threshold distance to be selected as parent models. This is just one of several approaches that can be used to select a subset of one or more child models as a parent model, based on one or more criteria.

[0074] Figure 6 shows the models that have completed training as black dots. For the sake of explanation, suppose Figure 6 represents the state of the scatterplot 600 after iteration N. One or more of the child models in the convex hull below or close to 606 can be selected as parent models for a subsequent iteration N+1 , where additional candidate layers can be added and initialized to form more child models, as discussed above.

[0075] Figure 7 shows the scatterplot 600 in a subsequent state after the N+1 iteration. Child models trained during the N+1 iteration are shown in Figure 7 using squares. A new lower convex hull 702 can be calculated. The anterior lower convex hull 606 is shown in dotted lines to illustrate the downward movement of the lower convex hull in the N+1 iteration.

[0076] Again, one or more of the child models at or near the lower convex hull 702 may be selected for a subsequent iteration N+2. Child models trained during the N 2 iteration are shown in Figure 8 as triangles. A new lower convex hull 802 can be calculated and the previous lower convex hulls 606 and 702 are shown in dotted lines to illustrate their position relative to the lower convex hull 802.

[0077] One way to visualize the approach shown in Figures 6-8 is a greedy approach to finding low-cost predictors. Note that this is a multi-objective approach, considering both the computational cost of training/validation and the accuracy of the model. Alternative implementations may use different and/or additional criteria, e.g. multidimensional plots of three or more criteria, an objective function defined on one or more criteria, etc.

[0078] The approach set out above generally develops networks using a random approach. However, instead of a purely random approach that may be computationally infeasible, the approach is guided by prediction criteria that tend to favor the use of known good models as a basis for further modifications. As noted earlier, training a model from scratch can be very computationally intensive. For example, a training dataset might include millions of training data items, and a given model might need to be trained at multiple training epochs before convergence. A training epoch can involve a forward propagation operation and a backpropagation operation through an entire model for each data item in the training dataset.

[0079] The approach set out above offers several benefits over conventional approaches to automated model generation. Note that not every child model is used to derive candidate layers for subsequent training. Instead, using a subset of child models that occur along the lower convex hull as new parent models, the disclosed implementations begin each new iteration with child model structures that inherit the parent model structure from known good models. This allows subsequent iterations to proceed without training models that occupy a significant part of the search space that is far from the lower convex hull and can save a huge amount of training time. Furthermore, by using not only accuracy, but training cost as a criterion for selecting which child models to use as new parent models, the disclosed implementations disfavor the generation of new models that are more computationally intensive. This, in turn, reduces the training time for training not only these models, but the child models derived from them.

[0080] Also, remember that candidate layer initialization can be performed on child models that inherit not only the structure of the parent model, but also the parameters learned from the parent model. As a consequence, the parameters of new candidate layers can be initialized relatively quickly to reasonable values. In addition, this allows for "hot start" training of the child model as a whole. This can speed up convergence, for example by requiring far fewer training periods or training samples than would be the case if each child model were fully trained from scratch.

[0081] Also, remember that the margins shared between any two child models can be initialized and trained together. Considering that child models are derived from a shared set of parent models, the probability of shared edges is relatively high. Thus, significant training time can be saved over alternatives where shared margins are initialized and trained separately for each child model.

[0082] Remember that previous techniques for automated generation of machine learning models suffered from computationally infeasible training loads or only considered models in a very restricted search space. Taken together, the benefits noted above allow the automated generation of large and accurate machine learning models from a relatively unconstrained search space, without imposing inordinate computational loads on the computational resources used to generate the models. Processing Flow Example

[0083] Figure 9 provides a high-level overview of a processing flow 900 for incremental growth of a model structure. The processing flow illustrates how a set of parent models can be iteratively augmented. This processing flow is an example of an approach to assigning certain tasks to worker processes such as virtual machines. For example, the processing flow might be suitable for distributed systems or tower servers where a hypervisor or operating system is scheduling new work over time as work processes become available.

[0084] In phase 910, a parent model group 912 is determined. For example, in some cases, a human or automated technique may choose one or more models for the group based on the known performance of the models. For simplicity, suppose the group includes a single model 914 at startup.

[0085] In phase 920, the candidate layers are initialized from the main models. For example, Figure 9 shows candidate layers 922, 924, and 926. As noted, each candidate layer can be added to the parent model group 912. Candidate layers can be added to a queue of candidate layers 928. When a given process becomes available, the worker process can pull a candidate layer from the candidate layer queue and initialize that candidate layer as discussed in this document, for example, performing several initial training iterations.

[0086] Once initialized, a subset of the candidate layers can be selected for inclusion in a child model queue 932 for training in phase 930. An available worker process can pull a given child model from the child model queue for training in phase 930. Figure 9 shows a child model 934 in phase 930 including candidate layer 924, but not candidate layer 922 or 926. This represents the idea that candidate layers 922 and/or 926 were removed in previous phase 920, for example, as discussed in the selection implementations feature mentioned above.

[0087] As mentioned earlier, child models are not necessarily trained from scratch, but can be trained from the parameters learned from the parent model and also initialized parameters determined in phase 920 when initializing the candidate layers that are added to the child models. Also, note that multiple child models can be trained together as part of a larger model, as also discussed above.

[0088] In phase 930, certain child models from the child model queue 932 can be chosen to add to the parent model set 912, as shown by arrow 936. As discussed earlier, child models added to the parent model set can be chosen based on various criteria. For example, some implementations can evaluate trained child models on validation datasets and determine the respective loss for each child model, and the loss can be used as a criterion for which child models are added to the parent model set. Alternatively and/or additionally, the computational expense of training a given child model can be used as a criterion to determine whether that child model is added to the set of parent models. Other approaches can leverage formal calculations of the net expected value of adding a child model to the parent model pool based on consideration of computational expense and calculated projections of the expected value of the addition. Once the parent model pool has been updated, processing can return to phase 910 for subsequent iterations. System Example

[0089] The present implementations can be performed in various scenarios on various devices. Figure 10 shows an example system 1000 in which the present implementations may be employed, as discussed further below.

[0090] As shown in Figure 10, the system 1000 includes a client device 1010, a server 1020, a server 1030 and a client device 1040, connected by one or more network(s) 1050.

Note that client devices can be embedded either as mobile devices such as smartphones or tablets, as well as fixed devices such as desktops, server devices, etc. Likewise, servers can be implemented using various types of computing devices. In some cases, any of the devices shown in Figure 10, but particularly servers, can be deployed in data centers, server towers, etc.

[0091] Certain components of the devices shown in Figure 10 may be referred to in this document by reference numbers in parentheses. For the purposes of the following description, the parentheses (1) indicate an occurrence of a particular component on client device 1010, (2) indicate an occurrence of a particular component on server 1020, (3) indicate an occurrence on server 1030, and (4) indicates an occurrence on the client device

1040. Unless identifying a specific instance of a particular component, this document will generally refer to components without parentheses.

[0092] Generally, devices 1010, 1020, 1030, and/or 1040 may have respective processing capabilities 1001 and storage capabilities 1002, which are discussed in more detail below. Devices can also have multiple modules that work by using processing and storage resources to perform the techniques discussed here. Storage resources can include persistent storage resources, such as magnetic or solid-state drives, and volatile storage, such as one or more random-access memory devices. In some cases, modules are provided as executable instructions that are stored on persistent storage devices, loaded into random-access memory devices, and read from random-access memory by processing resources for execution.

[0093] Client device 1010 may include a configuration module 1011 that can interact with a template generation module 1021 on server 1020. Generally speaking, the configuration module may provide certain configuration parameters to the generation module of model. The model generation module uses these configuration parameters to perform model generation functionality as discussed in this document. In particular, the model generation module can execute method 100 based on configuration parameters. As noted above, the iterative model growth process may involve modifying parent models to obtain child models, selecting candidate layers to include in child models based at least on the weights learned in an initialization process of the candidate layers and output of a final model selected from the children's models. The model generation module can include multiple submodules (not shown) for each method block 100.

[0094] Model generation module 1021 may send a final model to server 1030. Model execution module 1031 may execute the final model in response to incoming inputs. For example, the recording module 1041 in the client device 1040 can record data such as images or voice for processing by the final model, and a local application 1042 can upload the recorded data to the server 1030 for processing. The model execution module can process the data loaded using the final model and provide output from the model to the local application for further processing. Example of Graphical Interface

[0095] As noted above, the configuration module

1011 on client device 1010 may provide initial configuration parameters to template generation module 1021. Template generation module 1021 may perform method 100 in accordance with configuration parameters provided by the configuration module. Figure 11 illustrates an example configuration graphical user interface ("GUI") 1100 that can be presented on the client device 1010 for a user to set these configuration parameters.

[0096] Parent model element 1101 allows the user to specify what type of initial parent model or models should be used to start the model growth process. In Figure 11, the user is shown having selected a default parent model. For example, the 1021 model generation module can provide a standard neural network structure for use as a generic initial parent model. Other options might include a randomly generated model, where the module generation module selects a random model structure to use as the initial parent. Another option is for the user to navigate to an existing model that is known to provide relatively good performance for a specific task. In this case, the configuration module 1011 can load the model assigned to the model generation module for use as the initial parent model.

[0097] Operations element 1102 allows the user to specify which types of operations are considered by model generation module 1021. For example, model generation module can provide various options for groups of enumerated operations that can be selected for layers candidates. A first option, group 1, can define two operations, a single convolution operation (eg 5x5) and a single grouping operation (eg maximum grouping). A second option, group 2, can define a total of four possible operations - two convolution operations, for example, 5x5 and 7x7, and two grouping operations, average grouping and maximum grouping. A third option, group 3, can define a total of seven possible operations - 1x1, 3x3, 5x5, and 7x7 convolutions, average and maximum grouping, and an identity operation. Enumerated operations can also have predetermined or selectable operation parameters, e.g. adjustable window sizes, passes, etc.

[0098] Budget input element 1103 allows the user to specify a computational budget for model generation. For example, the user can specify a budget of

10,000 GPU-days, and model generation module 1021 can use this budget as a stopping condition in decision block 116 of method 100. Alternative implementations can use other metrics such as a series of processing operations, a series of virtual machines , an amount of time, etc., such as computational budgets.

[0099] Criterion 1 element 1104 allows the user to specify a first criterion for evaluating models, and criterion 2 element 1105 allows the user to specify a second criterion. In Figure 11, these criteria are shown as cost and computational loss, respectively, as discussed above. However, users may wish to specify other criteria, such as the overall size of the model structure, number of connections, the total number of filterable parameters of the model, etc. For applications with limited resources, such as running the model on a mobile device or an Internet of Things (IoT) device, model size in bytes can be an important criterion, as these devices tend to have storage and/or memory capabilities. limited. In addition, computational time to run the model can also be a useful criterion in these scenarios, as these devices may have constrained processing resources that can cause the user to notice latency when a given model takes too long to run.

[00100] Note that the configuration parameters shown in Figure 11 are merely examples and several other implementations are contemplated. For example, in some cases, users can specify connectivity parameters. As an example, a user can specify that candidate layers receive input from a specified number of previous layers or a variable number (eg random) from previous layers. As another example, the user can specify whether hop connections are allowed, for example, where a layer may not provide input to an immediately subsequent layer, but instead can ignore the immediately subsequent layer and connect to another subsequent layer. Users can also specify a DenseNet architecture where each layer is connected to all previous layers in the model.

[00101] Also note that some implementations may provide one or more GUIs to show the progress of method 100. For example, some implementations may generate GUIs that show the scatterplot 600 changing in different iterations of model growth in a similar to that shown in Figures 6-8. Other implementations may show graphical representations of individual models and/or candidate layers as they are generated. Application example

[00102] The techniques discussed here can be used for various applications without limitation. However, the following presents some specific examples by way of illustration.

[00103] As a first example, suppose an entity wants to market an application that tags all of the user's friends in its photo collection. This entity may have a pre-existing model that they currently use for this purpose, and that model may be executed by model execution module 1031 on server 1030 on photos provided by local application 1042 on client device 1040. However, the entity may feel that the pre-existing model is not accurate enough and can provide a poor user experience in some circumstances.

[00104] First, the entity can load the preexisting model to model generation module 1021 on server 1020 and can configure various initial parameters as discussed above. Then the model generation module can iteratively augment the preexisting model until a stop condition is reached and export the final model to server 1030. Server 1030 can then replace the initial model with the final model received and continue tagging photos user as discussed.

[00105] Alternatively, the client device can provide recorded audio data for voice transcription by the final model. As another alternative, the customer device can provide text in a first language for translation into a second language by the final model. As additional examples, the final model can perform scene segmentation on video, object detection on images or video (e.g. identifying faces, vehicles, etc.), performing parsing on text, etc.

[00106] As yet another example, the final model can be used to map documents and queries into a semantic vector. In this case, the server 1030 can provide search engine functionality that uses vector distances between indexed documents and incoming queries to sort search results for a user of the client device 1040. More generally, the final model can perform any task suitable for which training data is available, including but not limited to classification, machine translation, and pattern recognition tasks.

[00107] Additionally, some implementations may allow entities to provide datasets for training, validation, and/or testing. For example, a first entity may send a dataset from client device 1010 to a second entity that controls server 1020. The second entity may generate a final model using the dataset provided by the first entity. In some cases, the second entity may provide the final model for the first entity, for example, for execution on server 1030 by the first entity. In other cases, the second entity does not provide the model itself, but allows the first entity to call the model. In that case, the second entity may control server 1020 and server 1030 and/or implement model generation and execution on the same device.

[00108] Also, note that the term "final model" is only used here to imply that a particular child model is designed for practical use in an application. In some implementations, a final model output used by an entity can also be used as an initial parent model for subsequent iterations of the model growth process described here. In some cases, an entity will obtain new data over the course of using a given final model, and the new data can be used as training/validation/testing data for subsequent iterations of the process. This can result in the output of a new final model, which can then be used for some time as new data is obtained, and the process can be repeated continuously over the lifetime of a given practical application driven by a given entity. .

[00109] Also, note that the disclosed implementations focus on examples where candidate layers are added to model structures. However, in some cases, other types of candidate components may be used instead of layers. Generally, candidate components can be implemented using any type of math and/or logic functionality, from simple arithmetic or Boolean operators to more complex, trainable modules. Device Implementations

[00110] As noted above with respect to Figure 10, the system 1000 includes a number of devices, including a client device 1010, a server 1020, a server 1030, and a client device.

1040. As also noted, not all device implementations can be illustrated and other device implementations should be apparent to the person skilled in the art from the description above and below.

[00111] The term "device", "computer", "computing device", "client device", and/or "server device", as used in this document, may mean any type of device that has some amount of processing power hardware and/or hardware storage/memory capacity. Processing power may be provided by one or more hardware processors (eg, hardware processing units/cores) that can execute data in the form of computer-readable instructions to provide functionality. Computer readable instructions and/or data may be stored in storage, such as storage/memory and/or data storage. The term "system" as used in this document may refer to a single device, multiple devices, etc.

[00112] Storage resources can be internal or external to the respective devices they are associated with. Storage resources may include any one or more of volatile or non-volatile memory, hard drives, flash storage devices, and/or optical storage devices (eg, CDs, DVDs, etc.), among others. As used in this document, the term "computer readable media" may include signs. In contrast, the term "computer readable storage media" excludes signals. Computer readable storage media includes "computer readable storage devices". Examples of computer-readable storage devices include volatile storage media such as RAM and non-volatile storage media such as hard drives, optical discs, and flash memory, among others.

[00113] In some cases, devices are configured with a general purpose hardware processor and storage capabilities. In other cases, a device may include a system-on-a-chip (SOC) design. In SOC design implementations, the functionality provided by the device can be integrated into a single SOC or into multiple coupled SOCs. One or more associated processors can be configured to coordinate with shared resources, such as memory, storage, etc., and/or one or more dedicated resources, such as hardware blocks configured to perform certain specific functionality. Thus, the term "processor", "hardware processor" or "hardware processing unit", as used herein, may also refer to central processing units (CPUs), graphics processing units (GPUs), controllers, microcontrollers, processor cores, or other types of processing devices suitable for implementation in both conventional computing architectures and SOC designs.

[00114] Alternatively, or in addition, the functionality described herein may be performed, at least in part, by one or more logical hardware components. For example, and without limitation, illustrative types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System- on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[00115] In some configurations, any of the modules/code discussed in this document may be implemented in software, hardware and/or firmware. In any case, the modules/code may be provided during device manufacturing or by an intermediary who prepares the device for sale to the end user. In other cases, the end user can install these modules/code later, for example by downloading the executable code and installing the executable code on the corresponding device.

[00116] Also note that devices can often have input and/or output functionality. For example, computing devices may have multiple input mechanisms such as keyboards, mice, touchpads, speech recognition, gesture recognition (e.g. using depth cameras such as stereoscopic or time-of-flight camera systems, infrared camera, RGB camera systems or using accelerometers/gyroscopes, face recognition, etc.). Devices can also have multiple output mechanisms such as printers, monitors, etc.

[00117] Also note that the devices described in this document may work autonomously or cooperatively to implement the techniques described. For example, the methods and functionality described herein may be performed on a single computing device and/or distributed across multiple computing devices that communicate over the 1050 network(s). Without limitation, the network(s) s) 1050 may include one or more local area networks (LANs), wide area networks (WANs), the Internet and the like. Additional Examples

[00118] Several example devices are described above. Additional examples are described below. An example includes a method performed on a computing device, and the method may include performing two or more iterations of an iterative model growth process. The iterative model growth process may include selecting a particular parent model from a parent model group of one or more main models, generating a plurality of candidate layers, and initializing the plurality of candidate layers while reusing parameters learned from the particular parent model. The iterative model growth process may also include selecting particular candidate layers to include in child models for training, and respective child models may include the particular parent model and one or more of the particular candidate layers. The iterative model growth process may further include training the plurality of child models to obtain trained child models, evaluating the trained child models using one or more criteria, and, based at least on the evaluation, designating an individual trained child model as a new parent model and add the new parent model to the parent model group. The method may also include selecting at least one trained child model as a final model after two or more iterations, and emitting the final model.

[00119] Another example may include any of the examples above and/or below where the method still includes determining computational costs of training or testing the child models and using computational costs as a first criterion when evaluating the trained child models.

[00120] Another example may include any of the examples above and/or below where the method still includes determining losses associated with the trained child models and using the losses as a second criterion when evaluating the trained child models.

[00121] Another example may include any of the examples above and/or below where the method still includes representing the child models on a graph having a first axis reflecting computational costs and a second axis reflecting losses, and selecting the new parent model based on at least one matching location of the new parent model in the chart.

[00122] Another example may include any of the examples above and/or below where the method still includes determining at least one of a lower convex hull or Pareto efficiency on the graph, and selecting the new parent model based on at least proximity from the new parent model to the lower convex hull or Pareto efficiency.

[00123] Another example may include any of the examples above and/or below where the selection includes identifying a subset of trained child models that are within a predetermined proximity of the lower convex hull or Pareto efficiency, determining the respective probabilities for the subset of the trained child models, and select the new parent model based on at least the respective probabilities.

[00124] Another example may include any of the above and/or below examples where generating an individual candidate layer includes selecting a target layer of the particular parent model to receive outputs from the individual candidate layer, selecting one or more input layers from the parent model to provide inputs to the individual candidate layer, and to select a particular operation to be performed by the individual candidate layer on the inputs.

[00125] Another example may include any of the above and/or below examples where selecting the particular operation includes defining a group of operations and randomly selecting the particular operation from the group of operations.

[00126] Another example may include any of the examples above and/or below where the method still includes selecting the target layer and at least one input layer randomly from the particular parent model

[00127] Another example may include any of the examples above and/or below where the final model is a neural network.

[00128] Another example includes a system that includes a hardware processing unit and a storage facility storing computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to perform a process of iterative model growth that involves modifying parent models to get child models. The iterative model growth process may include selecting candidate layers to include in child models based on at least weights learned in an initialization process of the candidate layers. Computer readable instructions can still cause the hardware processing unit to still output a final model selected from child models.

[00129] Another example may include any of the examples above and/or below where computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to generate different candidate layers that share connectivity to the models. and perform different operations, and initialize the different candidate layers together to obtain different weights for the different candidate layers.

[00130] Another example may include any of the examples above and/or below where computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to select individual candidate layers for inclusion in models child, based at least on the different weights of different candidate layers.

[00131] Another example may include any of the above and/or below examples where computer readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to perform a feature selection technique on different weights of the different candidate layers to select individual candidate layers for inclusion in child models.

[00132] Another example may include any of the above and/or below examples where the feature selection technique includes least absolute shrinkage and selection operator (LASSO).

[00133] Another example may include any of the examples above and/or below where the different operations include convolution operations and grouping operations.

[00134] Another example may include any of the examples above and/or below where computer readable instructions, when executed by the hardware processing unit,

cause the hardware processing unit to train the final model using training data for at least one classification, machine translation, or pattern recognition task, and provide the final model for execution, the final model being adapted to perform at least a classification, machine translation, or pattern recognition task.

[00135] Another example includes a computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to perform actions that include performing two or more iterations of an iterative model growth process. The iterative model growth process may include: selecting a particular parent model from a parent model group of one or more parent models, initializing a plurality of candidate layers, selecting a plurality of child models for training, respective child models inheriting a structure of the particular parent model and including at least one of the candidate layers, training the plurality of child models to obtain trained child models, and designating an individual trained child model as a new parent model based on at least part of one or more criteria, and add the new parent model to the parent model group. Actions may also include selecting at least one trained child model as a final model after two or more iterations, and emitting the final model.

[00136] Another example may include any of the examples above and/or below where the actions still include simultaneously initializing a plurality of candidate layers to get initialized parameters and selecting individual candidate layers to include in individual child models based on at least the initialized parameters .

[00137] Another example may include any of the examples above and/or below where the actions further include randomly selecting operations from a group of enumerated operations to include in the plurality of candidate layers. Conclusion

[00138] Although the subject has been described in language specific to structural features and/or methodological acts, it should be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific remedies and acts described above are disclosed as exemplary forms of implementation of the claims and other remedies and acts that would be recognized by one skilled in the art are intended to be within the scope of the claims.

Claims (15)

1. Method performed on a computing device, characterized in that it comprises: performing two or more iterations of an iterative model growth process, the iterative model growth process comprising: selecting a particular parent model from a model group parent of one or more parent models; generating a plurality of candidate layers and initializing the plurality of candidate layers while reusing parameters learned from the particular parent model; select particular candidate layers to include in child models for training, the respective child models including the particular parent model and one or more of the particular candidate layers; train the plurality of child models to get trained child models; evaluate trained child models using one or more criteria; and based at least on evaluation, designating an individual trained child model as a new parent model and adding the new parent model to the parent model group; and after the two or more iterations, select at least one trained child model as a final model and output the final model. 2. Method, according to claim 1, further characterized by the fact that it comprises: determining computational costs for training or testing the child models; and using computational costs as a first criterion when evaluating trained child models. 3. Method, according to claim 2, further characterized by the fact that it comprises: determining losses associated with the trained child models; and using losses as a second criterion when evaluating trained child models. 4. Method, according to claim 3, further characterized by the fact that it comprises: representing the child models in a graph having a first axis reflecting computational costs and a second axis reflecting losses; and select the new parent model based on at least a matching location of the new parent model in the chart. 5. Method, according to claim 4, further characterized by the fact that it comprises: determining at least one of a lower convex hull or a Pareto efficiency in the graph; and selecting the new parent model based at least on the proximity of the new parent model to the lower convex hull or Pareto efficiency. 6. Method according to claim 5, characterized in that the selection comprises: identifying a subset of the trained child models that are within a predetermined proximity of the lower convex hull or Pareto efficiency; determine the respective probabilities for the subset of the trained child models; and select the new parent model based at least on the respective probabilities. 7. Method according to claim 1, characterized in that generating an individual candidate layer comprises: selecting a target layer of the particular parent model to receive outputs from the individual candidate layer; selecting one or more input layers from the particular parent model to provide inputs to the individual candidate layer; and selecting a particular operation to be performed by the individual candidate layer on the inputs. 8. Method, according to claim 7, the selection of the particular operation characterized by the fact that it comprises: defining a group of operations; and randomly select the particular operation from the group of operations. 9. Method according to claim 7, further characterized by the fact that it comprises: selecting the target layer and at least one input layer randomly from the particular parent model. 10. Method, according to claim 1, characterized by the fact that the final model being a neural network. 11. System, characterized in that it comprises: a hardware processing unit; and a storage facility storing computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to: perform an iterative model growth process that involves modifying parent models to obtain child models, the iterative model growth process comprising selecting candidate layers to include in child models based on at least the weights learned in an initialization process of the candidate layers; and emit a final model, the final model being selected from the child models. 12. System according to claim 11, characterized in that computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to: generate different candidate layers that share connectivity to the models principals and carry out different operations; and initialize the different candidate layers together to get different weights for the different candidate layers. 13. System according to claim 12, characterized in that computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit: based at least on the different weights of the different candidate layers, select individual candidate layers for inclusion in child models. 14. System according to claim 11, characterized in that computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to: train the final model using training data to at least one classification, machine translation, or pattern recognition task; and provide the final model for execution, the final model being adapted to perform at least one classification, machine translation, or pattern recognition task. 15. System according to claim 11, characterized in that the computer-readable instructions, when performed by the hardware processing unit, they cause the hardware processing unit to: execute the final model to perform at least one classification, machine translation, or pattern recognition task.