CN111931916A - Exploration method and device of deep learning model - Google Patents

Abstract

The invention discloses a method and device for exploring a deep learning model. The method includes: determining one cloud computing resource as a master node and multiple other cloud computing resources as multiple slave nodes, wherein the master node is used to schedule multiple slave nodes Perform a training operation on the deep learning model; perform a training operation on the deep learning model based on each slave node, and obtain the training result of the slave node, and the training result of each slave node includes the target deep learning model and the score of the target deep learning model, The target deep learning model included in the training result of each slave node is the trained deep learning model; the optimal deep learning model is determined from all the target deep learning models according to the scores included in all the training results. It can be seen that the implementation of the present invention can realize parallel training operations of deep learning models, improve the efficiency of deep learning model training, reduce training time, and facilitate the application of deep learning model exploration technology in business scenarios.

Description

Exploring method and device for deep learning model

technical field

The present invention relates to the technical field of deep learning, and in particular, to a method and device for exploring a deep learning model.

Background technique

In recent years, deep learning technology has been quickly applied to business scenarios in various industries due to its characteristics of reducing the complexity of users' use and the difficulty of users' technical understanding. In addition, due to the variability of business scenarios in which deep learning technology is applied, in order to fully tap the potential of deep learning technology and improve the accuracy of deep learning technology in practical applications, a deep learning model suitable for the business scenario is obtained by training for different business scenarios. appear particularly important.

In practical applications, in order to obtain a deep learning model suitable for specific business scenarios, deep learning model exploration can be performed (by continuously training and verifying various deep learning network structures and various hyperparameters, and then from the training and verification results. Select the optimal deep learning model) to get a suitable deep learning model. However, due to the fact that the exploration of deep learning models requires a lot of computing resources, the process of deep learning model generation and training is highly correlated, and the generation process of deep learning models is highly serialized. The efficiency of the exploration method is low and the exploration time is long, which is not conducive to the application of the exploration technology of the deep learning model in business scenarios.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a method and device for exploring deep learning models, which can determine multiple cloud computing resources to perform training operations of deep learning models in parallel, thereby improving the efficiency of deep learning model training and reducing training time. Time is conducive to the application of deep learning model exploration technology in business scenarios.

In order to solve the above technical problems, a first aspect of the present invention discloses a method for exploring a deep learning model, the method comprising:

Determining one cloud computing resource as the master node and multiple other cloud computing resources as multiple slave nodes, wherein the master node is used to schedule a plurality of the slave nodes to perform training operations on the deep learning model;

Perform a training operation on the deep learning model based on each slave node, and obtain the training result of the slave node. The training result of each slave node includes the target deep learning model and the score of the target deep learning model. Each The target deep learning model included in the training result of the slave node is the deep learning model after training;

An optimal deep learning model is determined from all the target deep learning models according to the scores included in all the training results.

As an optional implementation manner, in the first aspect of the present invention, performing a training operation on the deep learning model based on each slave node to obtain the training result of the slave node, including:

Create a first process and a second process corresponding to each of the slave nodes, and generate a plurality of deep learning models corresponding to the slave node based on the first process of each of the slave nodes;

Perform training and verification operations on each of the deep learning models corresponding to the slave node based on the second process of each slave node, the local cloud computing resources of each slave node, and the determined hyperparameters, and obtain the The training result corresponding to the slave node.

As an optional implementation manner, in the first aspect of the present invention, the generation of multiple deep learning models corresponding to the slave node based on the first process of each slave node includes:

Based on the first process of each slave node, multiple historical models are selected from the determined historical model pool as multiple basic models corresponding to the slave node, and the historical models are the depths generated by all the slave nodes learning model;

Based on the determined simulated annealing method, a plurality of the basic models are selected from all the basic models corresponding to each of the secondary nodes as the target basic models corresponding to the secondary node;

Perform a model deformation operation on each of the target base models corresponding to each of the slave nodes, obtain multiple variant models corresponding to the target base model, and screen out the target from the multiple variant models corresponding to the target base model The target variant model corresponding to the base model;

Score each of the target variant models, and screen out a plurality of deep learning models corresponding to the slave node from all the target variant models corresponding to each of the slave nodes according to the score of each of the target variant models ;

The model deformation operation refers to randomly performing at least one of a network structure deepening operation, a network structure widening operation, and a skip layer structure operation on the neural network model.

As an optional implementation manner, in the first aspect of the present invention, the target variant model corresponding to the target basic model is selected from the plurality of variant models corresponding to the target basic model, including:

Calculate the model distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool;

Determine whether each of the variant models corresponding to the target base model has a matching history model, and the matching history model corresponding to the variant model is the history model whose model distance from the variant model is less than a preset threshold. When there is a matching historical model in the model, delete the variant model from the plurality of variant models corresponding to the target base model;

Each of the variant models corresponding to the target basic model is scored, and the variant model with the highest score is selected from the plurality of variant models corresponding to the target basic model as the target variant model corresponding to the target basic model.

As an optional implementation manner, in the first aspect of the present invention, calculating the model distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool includes: :

Calculate the common layer distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool;

Calculate the jumping layer distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool;

adding the normal layer distance and the jump layer distance of each of the variant models corresponding to the target base model and each of the historical models in the historical model pool as models for the variant model and the historical model distance.

As an optional implementation manner, in the first aspect of the present invention, the calculating the common layer distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool, include:

Information coding is performed on the common layer of each of the variant models corresponding to the target base model to obtain a list of common layer information corresponding to the variant model, and the common layer of each of the historical models in the historical model pool is coded. Information coding, to obtain the common layer information list corresponding to the historical model;

The normal layer corresponding to the variant model and the historical model is constructed according to the normal layer information list corresponding to each variant model corresponding to the target base model and the normal layer information list corresponding to each historical model in the historical model pool matrix;

Perform each element of the normal layer matrix corresponding to each of the variant models corresponding to the target base model and each of the historical models in the historical model pool in the order from left to right and from top to bottom. Assignment, where the formula for assigning each element of the normal layer matrix is as follows:

matrix _ij =min(matrix _ij +dist(M1 _i ,M2 _j ),matrix _i(j-1) ,matrix _(i-1)j )

Wherein, matrix _ij represents the element in the ith row and jth column of the common layer matrix, M1 _i represents the ith common layer of the variant model, and M2 _j represents the jth common layer of the historical model;

Among them, the function dist is used to calculate the distance between the two layers of M1 _i and M2 _j . When M1 _i and M2 _j are different types of layers, the value of dist(M1 _i , M2 _j ) is 1. When M1 _i and M2 j are different types of layers When M2 _j are layers of the same type, the value of dist(M1 _i ,M2 _j ) is calculated by:

Among them, a _k is the k-th parameter information in the information coding of the common layer represented by M1 _i , b _k is the k-th parameter information in the information coding of the common layer represented by M2 _j , and n is the parameter information contained in the information coding. number;

Each of the variant models corresponding to the target base model and the element in the lower right corner of the normal layer matrix corresponding to each of the historical models in the historical model pool are taken as the normal layer distance between the variant model and the historical model.

As an optional implementation manner, in the first aspect of the present invention, calculating the layer-hopping distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool, include:

Information coding is performed on the skip layer of each of the variant models corresponding to the target base model, to obtain a layer skip information list corresponding to the variant model, and information on the skip layer of each of the historical models in the historical model pool is performed. Encoding to obtain the layer-hopping information list corresponding to the historical model;

According to the layer jump information list corresponding to each of the variant models corresponding to the target base model and the layer jump information list corresponding to each of the historical models in the historical model pool, construct the layer jump corresponding to the variant model and the historical model matrix, the number of rows of the layer-hopping matrix is the length of the layer-hopping information list corresponding to the variant model, and the number of columns of the layer-hopping matrix is the length of the layer-hopping information list corresponding to the historical model;

Each element of the layer jump matrix corresponding to each of the variant models corresponding to the target basic model and each of the historical models in the historical model pool is assigned a value according to the following formula:

skip_connection_matrix _pq = dist2(S1 _p , S2 _q )

Among them, skip_connection_matrix represents the skip layer matrix, skip_connection_matrix _pq represents the element in the pth row and the qth column of the skip layer matrix, S1 _p represents the pth skip layer of the variant model, and S2 _q represents the qth skip of the historical model Floor;

Among them, the function dist2 is used to calculate the distance between the two layers of S1 _p and S2 _q . When S1 _p and S2 _q are different types of layers, the value of dist2 (S1 _p , S2 _q ) is 1. When S1 _p and S2 q are different types of layers, the value of dist2 is 1. When S2 _q are layers of the same type, the value of dist2(S1 _p , S2 _q ) is calculated by:

Among them, p _s represents the layer position index of the starting layer in the p-th layer jumping layer in the variant model, and q _s represents the layer position of the starting layer in the q-th layer jumping layer in the historical model. index, p _l represents the depth of the p-th layer jumping layer in the variant model, q _l represents the q-th layer jumping layer depth in the historical model;

Calculate the jump distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool according to the following formula:

dist _s =sum(skip_connection_matrix)+|s1-s2|

Among them, dist _s represents the jumping distance between the variant model and the historical model, sum(skip_connection_matrix) means adding and summing each element in the skip_connection_matrix, and s1 represents the jumping information corresponding to the variant model The length of the list, s2 represents the length of the layer jump information list corresponding to the history model.

As an optional implementation manner, in the first aspect of the present invention, based on the second process of each of the slave nodes, the local cloud computing resources of each of the slave nodes, and the determined hyperparameters, the Each of the deep learning models corresponding to the slave node performs training and verification operations, and obtains the training result corresponding to the slave node, including:

Based on the second process of each slave node and the local cloud computing resources of each slave node, for each of the deep learning models corresponding to the slave node, the hyperparameter space corresponding to the deep learning model is formulated, so The hyperparameter space includes at least the number of batches and the learning rate;

Set the number of searches corresponding to each of the deep learning models corresponding to each of the slave nodes;

Construct a set corresponding to each of the deep learning models corresponding to each of the slave nodes, and the set is used to save the score on the verification set after the deep learning model is trained;

The objective function corresponding to each of the deep learning models corresponding to each of the slave nodes is set according to the following formula:

F=max(SC)

Among them, F is the objective function, and SC is the score on the validation set after the deep learning model is trained;

Randomly select a starting point in the hyperparameter space corresponding to each of the deep learning models corresponding to each of the slave nodes, and then perform a circular search in the hyperparameter space through Gaussian process mapping to select the deep learning model The corresponding multiple target hyperparameters, where the Gaussian process map can be expressed as:

T=G(C, R, F, J)

Among them, T is the target hyperparameter corresponding to the deep learning model, each target hyperparameter is the value of the hyperparameter worth trying recommended by G, C is the hyperparameter space corresponding to the deep learning model, and R is the depth The set corresponding to the learning model, F is the objective function corresponding to the deep learning model, and J is the number of searches corresponding to the deep learning model;

Perform training and verification operations on the deep learning model based on each of the target hyperparameters corresponding to each of the deep learning models corresponding to each of the slave nodes, to obtain a plurality of intermediate deep learning models corresponding to the deep learning model and a score corresponding to each of the intermediate deep learning models;

From all the intermediate deep learning models corresponding to each of the deep learning models corresponding to each of the slave nodes, select the intermediate deep learning model with the highest score and the score corresponding to the intermediate deep learning model as the training result corresponding to the slave node.

A second aspect of the present invention discloses a device for exploring a deep learning model, the device comprising:

a determination module, configured to determine one cloud computing resource as the master node and multiple other cloud computing resources as multiple slave nodes, wherein the master node is used to schedule a plurality of the slave nodes to perform training operations on the deep learning model;

A training module, configured to perform a training operation on the deep learning model based on each of the slave nodes to obtain a training result of the slave node, where the training result of each slave node includes the target deep learning model and the target deep learning model The score, the target deep learning model included in the training result of each described slave node is the deep learning model after training;

The determining module is further configured to determine an optimal deep learning model from all the target deep learning models according to the scores included in all the training results.

As an optional implementation manner, in the second aspect of the present invention, the training module includes:

creating a submodule for creating a first process and a second process corresponding to each of the slave nodes;

generating a submodule for generating a plurality of deep learning models corresponding to the slave node based on the first process of each slave node;

A training submodule for performing training on each of the deep learning models corresponding to the slave node based on the second process of each slave node, the local cloud computing resources of each slave node, and the determined hyperparameters With the verification operation, the training result corresponding to the slave node is obtained.

As an optional implementation manner, in the second aspect of the present invention, the generating submodule includes:

The selection unit is used to select a plurality of historical models from the determined historical model pool as a plurality of basic models corresponding to the slave node based on the first process of each of the slave nodes, and the historical models are all the slave nodes. The deep learning model that the node has generated;

The selection unit is further configured to select a plurality of the basic models from all the basic models corresponding to each of the secondary nodes based on the determined simulated annealing method as the target basic models corresponding to the secondary node;

a deformation unit, configured to perform a model deformation operation on each of the target basic models corresponding to each of the slave nodes, to obtain a plurality of variant models corresponding to the target basic model;

a first screening unit, configured to screen out a target variant model corresponding to the target basic model from a plurality of variant models corresponding to the target basic model;

The second screening unit is configured to score each of the target variant models, and filter out the subordinate node from all the target variant models corresponding to each of the subordinate nodes according to the score of each of the target variant models Corresponding multiple deep learning models;

The model deformation operation refers to randomly performing at least one of a network structure deepening operation, a network structure widening operation, and a skip layer structure operation on the neural network model.

As an optional embodiment, in the second aspect of the present invention, the first screening unit includes:

A calculation subunit for calculating the model distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool;

A judging subunit for judging whether each of the variant models corresponding to the target base model has a matching history model, and the matching history model corresponding to the variant model is the history model whose model distance from the variant model is less than a preset threshold , when it is determined that there is a matching historical model in the variant model, delete the variant model from the plurality of variant models corresponding to the target base model;

Selecting a subunit for scoring each of the variant models corresponding to the target basic model, and selecting the variant model with the highest score from the plurality of variant models corresponding to the target basic model as the target basic model The corresponding target variant model.

As an optional implementation manner, in the second aspect of the present invention, the calculation subunit includes:

A second-level subunit is calculated for calculating the common layer distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool;

The calculating secondary subunit is also used to calculate the jump distance between each of the variant models corresponding to the target basic model and each of the historical models in the historical model pool;

The addition secondary subunit is used to add the normal layer distance and the jump layer distance of each of the variant models corresponding to the target basic model and each of the historical models in the historical model pool as The model distance between the variant model and the historical model.

As an optional implementation manner, in the second aspect of the present invention, the calculating secondary subunit calculates each of the variant models corresponding to the target basic model and each of the historical models in the historical model pool The specific way of the common layer distance is:

Information coding is performed on the common layer of each of the variant models corresponding to the target base model to obtain a list of common layer information corresponding to the variant model, and the common layer of each of the historical models in the historical model pool is coded. Information coding, to obtain the common layer information list corresponding to the historical model;

The normal layer corresponding to the variant model and the historical model is constructed according to the normal layer information list corresponding to each variant model corresponding to the target base model and the normal layer information list corresponding to each historical model in the historical model pool matrix;

Perform each element of the normal layer matrix corresponding to each of the variant models corresponding to the target base model and each of the historical models in the historical model pool in the order from left to right and from top to bottom. Assignment, where the formula for assigning each element of the normal layer matrix is as follows:

matrix _ij =min(matrix _ij +dist(M1 _i ,M2 _j ),matrix _i(j-1) ,matrix _(i-1)j )

Wherein, matrix _ij represents the element in the ith row and jth column of the common layer matrix, M1 _i represents the ith common layer of the variant model, and M2 _j represents the jth common layer of the historical model;

Among them, the function dist is used to calculate the distance between the two layers of M1 _i and M2 _j . When M1 _i and M2 _j are different types of layers, the value of dist(M1 _i , M2 _j ) is 1. When M1 _i and M2 j are different types of layers When M2 _j are layers of the same type, the value of dist(M1 _i ,M2 _j ) is calculated by:

Among them, a _k is the k-th parameter information in the information coding of the common layer represented by M1 _i , b _k is the k-th parameter information in the information coding of the common layer represented by M2 _j , and n is the parameter information contained in the information coding. number;

Each of the variant models corresponding to the target base model and the element in the lower right corner of the normal layer matrix corresponding to each of the historical models in the historical model pool are taken as the normal layer distance between the variant model and the historical model.

As an optional implementation manner, in the second aspect of the present invention, the calculating secondary subunit calculates each of the variant models corresponding to the target basic model and each of the historical models in the historical model pool The specific way of the jump distance is:

Information coding is performed on the skip layer of each of the variant models corresponding to the target base model, to obtain a layer skip information list corresponding to the variant model, and information on the skip layer of each of the historical models in the historical model pool is performed. Encoding to obtain the layer-hopping information list corresponding to the historical model;

According to the layer jump information list corresponding to each of the variant models corresponding to the target base model and the layer jump information list corresponding to each of the historical models in the historical model pool, construct the layer jump corresponding to the variant model and the historical model matrix, the number of rows of the layer-hopping matrix is the length of the layer-hopping information list corresponding to the variant model, and the number of columns of the layer-hopping matrix is the length of the layer-hopping information list corresponding to the historical model;

Each element of the layer jump matrix corresponding to each of the variant models corresponding to the target basic model and each of the historical models in the historical model pool is assigned a value according to the following formula:

skip_connection_matrix _pq = dist2(S1 _p , S2 _q )

Among them, skip_connection_matrix represents the skip layer matrix, skip_connection_matrix _pq represents the element in the pth row and the qth column of the skip layer matrix, S1 _p represents the pth skip layer of the variant model, and S2 _q represents the qth skip of the historical model Floor;

Among them, the function dist2 is used to calculate the distance between the two layers S1 _p and S2 _q . When S1 _p and S2 _q are different types of layers, the value of dist2(S1 _p , S2 _q ) is 1, when S1 _p and S2 q When S2 _q are layers of the same type, the value of dist2(S1 _p , S2 _q ) is calculated by:

Among them, p _s represents the layer position index of the starting layer in the p-th layer jumping layer in the variant model, and q _s represents the layer position of the starting layer in the q-th layer jumping layer in the historical model. index, p _l represents the depth of the p-th layer jumping layer in the variant model, q _l represents the q-th layer jumping layer depth in the historical model;

Calculate the jump distance between each of the variant models corresponding to the target base model and each of the historical models in the historical model pool according to the following formula:

dist _s =sum(skip_connection_matrix)+|s1-s2|

Among them, dist _s represents the jumping distance between the variant model and the historical model, sum(skip_connection_matrix) means adding and summing each element in the skip_connection_matrix, and s1 represents the jumping information corresponding to the variant model The length of the list, s2 represents the length of the layer jump information list corresponding to the history model.

As an optional implementation manner, in the second aspect of the present invention, the training submodule is based on the second process of each of the slave nodes, the local cloud computing resources of each of the slave nodes, and the determined overtime The parameters perform training and verification operations on each of the deep learning models corresponding to the slave node, and the specific way to obtain the training result corresponding to the slave node is:

Based on the second process of each slave node and the local cloud computing resources of each slave node, for each of the deep learning models corresponding to the slave node, the hyperparameter space corresponding to the deep learning model is formulated, so The hyperparameter space includes at least the number of batches and the learning rate;

Set the number of searches corresponding to each of the deep learning models corresponding to each of the slave nodes;

Construct a set corresponding to each of the deep learning models corresponding to each of the slave nodes, and the set is used to save the score on the verification set after the deep learning model is trained;

The objective function corresponding to each of the deep learning models corresponding to each of the slave nodes is set according to the following formula:

F=max(SC)

Among them, F is the objective function, and SC is the score on the validation set after the deep learning model is trained;

Randomly select a starting point in the hyperparameter space corresponding to each of the deep learning models corresponding to each of the slave nodes, and then perform a circular search in the hyperparameter space through Gaussian process mapping to select the deep learning model The corresponding multiple target hyperparameters, where the Gaussian process map can be expressed as:

T=G(C, R, F, J)

Among them, T is the target hyperparameter corresponding to the deep learning model, each target hyperparameter is the value of the hyperparameter worth trying recommended by G, C is the hyperparameter space corresponding to the deep learning model, and R is the depth The set corresponding to the learning model, F is the objective function corresponding to the deep learning model, and J is the number of searches corresponding to the deep learning model;

Perform training and verification operations on the deep learning model based on each of the target hyperparameters corresponding to each of the deep learning models corresponding to each of the slave nodes, to obtain a plurality of intermediate deep learning models corresponding to the deep learning model and a score corresponding to each of the intermediate deep learning models;

From all the intermediate deep learning models corresponding to each of the deep learning models corresponding to each of the slave nodes, select the intermediate deep learning model with the highest score and the score corresponding to the intermediate deep learning model as the training result corresponding to the slave node.

A third aspect of the present invention discloses a device for exploring a deep learning model, the device comprising:

a memory in which executable program code is stored;

a processor coupled to the memory;

The processor invokes the executable program code stored in the memory to execute the deep learning model exploration method disclosed in the first aspect of the present invention.

A fourth aspect of the present invention discloses a computer storage medium. The computer storage medium stores computer instructions, and when the computer instructions are invoked, they are used to execute the exploration method for the deep learning model disclosed in the first aspect of the present invention.

Compared with the prior art, the present invention has the following beneficial effects:

In the embodiment of the present invention, one cloud computing resource is determined as the master node and multiple other cloud computing resources are used as multiple slave nodes, and then the training operation of the deep learning model is performed based on each slave node to obtain the training result of the slave node. The scoring of the results determines the optimal deep learning model from all the training results, so that the training operation of the deep learning model can be performed in parallel, the efficiency of the deep learning model training is improved, the training time is reduced, and it is beneficial to the deep learning model exploration technology. Applications in business scenarios.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a schematic flowchart of a method for exploring a deep learning model disclosed in an embodiment of the present invention;

2 is a schematic flowchart of another method for exploring a deep learning model disclosed in an embodiment of the present invention;

3 is a schematic structural diagram of a device for exploring a deep learning model disclosed in an embodiment of the present invention;

4 is a schematic structural diagram of another apparatus for exploring a deep learning model disclosed in an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of another apparatus for exploring a deep learning model disclosed in an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes For other steps or units inherent to these processes, methods, products or devices.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

The invention discloses a method and device for exploring a deep learning model. By determining one cloud computing resource as a master node and multiple other cloud computing resources as multiple slave nodes, and then performing a deep learning model training operation based on each slave node The training result of the slave node is obtained, and finally the optimal deep learning model is determined from all the training results according to the score of the training result, so that the training operation of the deep learning model can be performed in parallel, the efficiency of deep learning model training can be improved, and the The training time is conducive to the application of deep learning model exploration technology in business scenarios.

Example 1

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a method for exploring a deep learning model disclosed in an embodiment of the present invention. As shown in Figure 1, the exploration method of the deep learning model can include the following operations:

101. Determine one cloud computing resource as the master node and multiple other cloud computing resources as multiple slave nodes, where the master node is used to schedule multiple slave nodes to perform training operations on the deep learning model.

In the above step 101, the master node is mainly responsible for the interaction with the user, the startup of the slave node, the interaction of the slave node, and the aggregation of the training results of the deep learning model. The hub of the master node is a message loop module, which is used to manage the interaction of each module of the master node itself. The modules of the master node include parameter adjusters, evaluators, training service modules, etc. The network structure of the deep learning model, the parameter adjuster needs to specify the method in the initialization stage, such as Network Morphism. The evaluator is used to evaluate the quality of the network structure. The training service module is used to schedule slave nodes for training. The interaction of each module of the master node itself, the interaction between the master node and the slave node, and the interaction between the master node and the user can all be carried out through network communication. In the process of parallel training, the master node sends the parameter adjuster to the slave node, so that the slave node can generate and train the deep learning model autonomously.

Specifically, the startup and scheduling process of cloud computing resources can refer to the following description. The user configures relevant task information (such as image classification related) through yml, and then starts the master node through a shell script, and the master node starts nodejs. In this process, the user You can use the web to view the cloud running status. The master node starts its own message loop module and zmq module, and then starts to interact with the slave node based on the typescript management module (typescript is a kind of java software, here for network communication). The master node parses the user's shell request information, initializes the business use case, parameter adjuster and historical information table (used to save all the training results uploaded from the slave nodes) according to it, and stores a 16-layer initial structure in the historical information table (Combination of several types of deep learning layers, such as convolution, sampling layer), as the basis for network structure generation, and according to the obtained slave node information, the slave node is started based on the shell script through the training service module, and then the business use cases (including The corresponding configuration parameters and business data for training. For example, when the use case is image classification, the configuration parameters are the image input dimension and the image output dimension, and the business data is the image classification training data set), the parameter adjuster and the history information table are passed to the slave through the network. node. Next, the slave node starts to maintain the local history information table (used to save the training results of the slave node), the local parameter adjuster and the local business use case.

102. Perform a training operation on the deep learning model based on each slave node to obtain a training result of the slave node.

In the above step 102, the training result of each slave node includes the target deep learning model and the score of the target deep learning model, and the target deep learning model included in the training result of each slave node is the trained deep learning model. Here, the score of the deep learning model may be the accuracy score of the deep learning model on the validation set. After each slave node receives the business use case, parameter adjuster and historical information table transmitted by the master node through the network, it can start to maintain the local historical information table (used to save the training results of the slave node), local The parameter tuner and local business use cases start to train the deep learning model independently. Specifically, use the local parameter adjuster and local business use cases to train the deep learning model, and then save the obtained training results to the local historical information table.

In an optional embodiment, a training operation is performed on the deep learning model based on each slave node, and the training result of the slave node is obtained, including:

Create a first process and a second process corresponding to each slave node, and generate multiple deep learning models corresponding to the slave node based on the first process of each slave node;

Perform training and verification operations on each deep learning model corresponding to the slave node based on the second process of each slave node, the local cloud computing resources of each slave node, and the determined hyperparameters, and obtain the training result corresponding to the slave node .

In this optional embodiment, a first process and a second process of the slave node are created in each slave node, and the first process of the slave node is used to continuously cyclically execute the task of generating a deep learning model to continuously A new deep learning model is generated, and the second process of the slave node is used for continuously cyclically executing the tasks of training and validating the deep learning model generated by the first process.

It can be seen that, by implementing this optional embodiment, the first process and the second process corresponding to the slave node are generated in each slave node, and a new deep learning model is continuously generated through the first process, and a new deep learning model is continuously generated through the second process. The newly generated deep learning model is trained and verified to obtain the training result, which can realize the parallelization of the generation and training of the deep learning model, so that the generation and training of the deep learning model do not need to wait for each other, and the training efficiency of the deep learning model can be improved.

103. Determine an optimal deep learning model from all target deep learning models according to the scores included in all training results.

In the above step 103, since the history information table is used to save all the training results uploaded from the nodes, the optimal deep learning model can be determined according to the information in the history information table. Here, the target deep learning model with the highest score can be selected as the optimal deep learning model.

It can be seen that the exploration method of the deep learning model described in FIG. 1 is implemented by determining one cloud computing resource as the master node and multiple other cloud computing resources as multiple slave nodes, and then performing the training operation of the deep learning model based on each slave node. The training result of the slave node is obtained, and finally the optimal deep learning model is determined from all the training results according to the score of the training result, so that the training operation of the deep learning model can be performed in parallel, the efficiency of deep learning model training can be improved, and the The training time is conducive to the application of deep learning model exploration technology in business scenarios. In addition, by generating the first process and the second process corresponding to the slave node in each slave node, the parallelization of the generation and training of the deep learning model is realized, so that the generation and training of the deep learning model do not need to wait for each other. Training efficiency of deep learning models.

Embodiment 2

Please refer to FIG. 2. FIG. 2 is a schematic flowchart of another method for exploring a deep learning model disclosed in an embodiment of the present invention. As shown in Figure 2, the exploration method of the deep learning model can include the following operations:

201. Determine one cloud computing resource as the master node and multiple other cloud computing resources as multiple slave nodes, where the master node is used to schedule multiple slave nodes to perform training operations on the deep learning model.

202. Create a first process and a second process corresponding to each slave node.

203. Based on the first process of each slave node, select multiple historical models from the determined historical model pool as multiple basic models corresponding to the slave node, where the historical models are deep learning models that have been generated by all slave nodes.

In the above step 203, the historical model pool may be determined according to the historical information table, and the historical information table may include the model number and model information (network structure, model score, model weight, etc.) of the historical model. Here, the top ten historical models in the model score can be selected as the ten basic models corresponding to the slave nodes.

204. Based on the determined simulated annealing method, select multiple base models from all base models corresponding to each slave node as multiple target base models corresponding to the slave node.

In the above step 204, the simulated annealing method is an algorithm for finding a global optimal solution, which can effectively avoid falling into a local optimal solution. The target basic model is selected based on the simulated annealing method, which can more efficiently explore deep learning models with higher scores.

205. Perform a model deformation operation on each target base model corresponding to each slave node, obtain multiple variant models corresponding to the target base model, and screen out the target base model from the multiple variant models corresponding to the target base model The corresponding target variant model.

In the above step 205, the model deformation operation refers to randomly performing at least one of a network structure deepening operation, a network structure widening operation, and a skip layer structure operation on the neural network model. By performing the model deformation operation, multiple variant models corresponding to the target base model can be obtained, which can expand the exploration space and make the exploration results more accurate.

In an optional embodiment, the target variant model corresponding to the target basic model is selected from the plurality of variant models corresponding to the target basic model, including:

Calculate the model distance between each variant model corresponding to the target base model and each historical model in the historical model pool;

Determine whether each variant model corresponding to the target base model has a matching history model. The matching history model corresponding to the variant model is a historical model whose model distance from the variant model is less than the preset threshold. When it is judged that the variant model has a matching history model, delete the variant model from the multiple variant models corresponding to the target base model;

Each variant model corresponding to the target base model is scored, and the variant model with the highest score is selected from the plurality of variant models corresponding to the target base model as the target variant model corresponding to the target base model.

In this optional embodiment, since the model deformation operation is a random operation, the structure of the obtained variant model is also random. If the structural deformation of the variant model is not large, then the variant model is subjected to subsequent training and The verification operation is of little significance and will cause a waste of computing resources. Therefore, by calculating the model distance between the variant model and the historical model, the variant model with too small model distance is deleted, and then the remaining variant models are scored, and the variant model with the highest score is selected from the remaining variant models as the target variant. Model. Optionally, when the number of historical models in the historical model pool is greater than 40, a process pool can be created in the first process, and each subprocess in the process pool is used to perform the task of calculating the model distance between the variant model and the historical model, In this way, the model distance of each variant model and each historical model can be calculated in parallel, and the exploration efficiency of the deep learning model can be improved.

Specifically, the scoring rules for variant models are:

Among them, n is the number of historical models, d _i is the distance between the variant model and the i-th historical model, and acc is the scoring result.

For example, if the number of historical models is 2, and the distances between variant models and them are 2 and 4, respectively, the score is: (2+4)/2=3.

It can be seen that, by implementing this optional embodiment, by deleting the variant model whose distance from the historical model is too small, and selecting the variant model with the highest score from the remaining variant models as the target variant model, it is possible to realize the transformation from multiple variants. The target variant model is screened out from the model, which reduces the amount of subsequent data processing and improves the efficiency of training.

In this optional embodiment, further optional, calculating the model distance between each variant model corresponding to the target base model and each historical model in the historical model pool, including:

Calculate the common layer distance between each variant model corresponding to the target base model and each historical model in the historical model pool;

Calculate the jumping distance between each variant model corresponding to the target base model and each historical model in the historical model pool;

Each variant model corresponding to the target base model is added with the common layer distance and the jump layer distance of each historical model in the historical model pool as the model distance between the variant model and the historical model.

In this further optional embodiment, the common layer of the deep learning model includes a single-layer convolutional layer, a batch normalization layer, a pooling layer, etc., and the skip layer of the deep learning model refers to a residual connection layer. Since a deep learning model is usually composed of multiple ordinary layers and multiple jump layers, the ordinary layer distance and the jump layer distance between the two deep learning models are calculated, and then the ordinary layer distance and the jump layer distance are added as two The model distance between the two models can effectively compare the two models. Optionally, you can first count the number of jump layers of the variant model to be compared and the historical model. When the number of jump layers is greater than 5, create several sub-processes in the first process of the slave node to calculate the difference between the two models in parallel. The layer-jump distance can realize parallel calculation of the layer-jump distance between the two models and improve the training efficiency.

It can be seen that, by implementing this further optional embodiment, the model distance between the two models can be obtained by calculating the normal layer distance and the jump layer distance between the two models, which can effectively compare the two models, so that the two models can be accurately and effectively screened out. Target variant model.

In this further optional embodiment, it is further optional to calculate the common layer distance between each variant model corresponding to the target basic model and each historical model in the historical model pool, including:

Perform information encoding on the common layer of each variant model corresponding to the target basic model to obtain a list of common layer information corresponding to the variant model, and encode information on the common layer of each historical model in the historical model pool to obtain the history A list of common layer information corresponding to the model;

According to the normal layer information list corresponding to each variant model corresponding to the target basic model and the normal layer information list corresponding to each historical model in the historical model pool, construct the normal layer matrix corresponding to the variant model and the historical model;

Assign values to each variant model corresponding to the target base model and each element of the normal layer matrix corresponding to each historical model in the historical model pool in the order from left to right and from top to bottom, where the normal layer The formula for assigning a value to each element of the matrix is as follows:

matrix _ij =min(matrix _ij +dist(M1 _i ,M2 _j ),matrix _i(j-1) ,matrix _(i-1)j )

Wherein, matrix _ij represents the element in the ith row and jth column of the common layer matrix, M1 _i represents the ith common layer of the variant model, and M2 _j represents the jth common layer of the historical model;

Among them, the function dist is used to calculate the distance between the two layers of M1 _i and M2 _j . When M1 _i and M2 _j are different types of layers, the value of dist(M1 _i , M2 _j ) is 1. When M1 _i and M2 j are different types of layers When M2 _j are layers of the same type, the value of dist(M1 _i ,M2 _j ) is calculated by:

Among them, a _k is the k-th parameter information in the information coding of the common layer represented by M1 _i , b _k is the k-th parameter information in the information coding of the common layer represented by M2 _j , and n is the parameter information contained in the information coding. number;

Take each variant model corresponding to the target base model and the element in the lower right corner of the normal layer matrix corresponding to each historical model in the historical model pool as the normal layer distance between the variant model and the historical model.

In this further optional embodiment, the process of performing information encoding on the common layer of the model to obtain the common layer information list corresponding to the model can be expressed as:

M=(Lm ₁ ,Lm ₂ ,...,Lm _N )

Among them, M represents the list of common layer information of the model, N represents the number of common layers of the model, and Lm _i is the information code of the first common layer of the model. For example, the information of an activation layer is encoded as the string "ReLU".

Specifically, the specific process of constructing the common layer matrix corresponding to the variant model and the historical model according to the common layer information list corresponding to the variant model and the common layer information list corresponding to the historical model may be as follows:

According to the length m1 of the normal layer information list corresponding to the variant model and the length m2 of the normal layer information list corresponding to the historical model, construct a normal layer matrix with m1+1 rows and m2+1 columns, and then calculate the first layer of the normal layer matrix. One row is assigned as 0 to m1, and the first column is assigned as 0 to m2, so as to realize the initialization of the ordinary layer matrix and complete the construction of the ordinary layer matrix.

Specifically, the description of the function dist(M1 _i , M2 _j ) is as follows:

The layers of the deep learning model usually include convolutional layers, sampling layers, etc. For example, if M1 _i is a convolutional layer and M2 _j is a sampling layer, then these two layers are different types of layers, then dist(M1 _i , M2 _j ) value of 1. For another example, the information corresponding to M1 _i is coded as "conv(32, 8)", and the information corresponding to M2 _j is coded as "conv(16, 4)", when calculating the value of the function dist(M1 _i , M2 _j ), according to The first four letters of the information code determine whether the types of the two layers are the same. The first four letters of the information code of the two layers are all "conv", so the two layers are of the same type. In addition, the information code "conv(32, 8)" corresponding to M1 _i indicates that the first parameter information a1 (the number of convolution kernels) of the M1 _i layer is 32, and the second parameter information a2 (the size of the convolution kernel) is 32. ) is 8, and the information code corresponding to M2 _j "conv(16, 4)" indicates that the first parameter information b1 (the number of convolution kernels) of the M2 _j layer is 16, and the second parameter information b2 (the number of convolution kernels) The size of ) is 4, so dist(M1 _i , M2 _j )=(32-16)/2*32+(8-4)/2*8=0.5.

It can be seen that, implementing this further optional embodiment, the common layer information list is obtained by encoding the common layer of the model, and then the common layer matrix is constructed according to the common layer information list, and the common layer matrix is assigned in a specific way to obtain The common layer distance between models, so that the common layer distance between two models can be calculated.

In this further optional embodiment, it is further optional to calculate the jumping distance between each variant model corresponding to the target basic model and each historical model in the historical model pool, including:

Perform information coding on the skip layer of each variant model corresponding to the target basic model to obtain a list of skip layer information corresponding to the variant model, and perform information coding on the skip layer of each historical model in the historical model pool to obtain the historical model. Corresponding layer jump information list;

According to the layer jump information list corresponding to each variant model corresponding to the target basic model and the layer jump information list corresponding to each historical model in the historical model pool, the layer jump matrix corresponding to the variant model and the historical model is constructed. The number of rows is the length of the layer-hopping information list corresponding to the variant model, and the number of columns of the layer-hopping matrix is the length of the layer-hopping information list corresponding to the historical model;

Each element of the jump-layer matrix corresponding to each variant model corresponding to the target base model and each historical model in the historical model pool is assigned a value according to the following formula:

skip_connection_matrix _pq = dist2(S1 _p , S2 _q )

Among them, skip_connection_matrix represents the skip layer matrix, skip_connection_matrix _pq represents the element in the pth row and the qth column of the skip layer matrix, S1 _p represents the pth skip layer of the variant model, and S2 _q represents the qth skip of the historical model Floor;

Among them, the function dist2 is used to calculate the distance between the two layers S1 _p and S2 _q . When S1 _p and S2 _q are different types of layers, the value of dist2(S1 _p , S2 _q ) is 1, when S1 _p and S2 q When S2 _q are layers of the same type, the value of dist2(S1 _p , S2 _q ) is calculated by:

Among them, p _s represents the layer position index of the starting layer in the p-th layer jumping layer in the variant model, and q _s represents the layer position of the starting layer in the q-th layer jumping layer in the historical model. index, p _l represents the depth of the p-th layer jumping layer in the variant model, q _l represents the q-th layer jumping layer depth in the historical model;

Calculate the jump distance between each variant model corresponding to the target base model and each historical model in the historical model pool according to the following formula:

dist _s =sum(skip_connection_matrix)+|s1-s2|

Among them, dist _s represents the jumping distance between the variant model and the historical model, sum(skip_connection_matrix) means adding and summing each element in the skip_connection_matrix, and s1 represents the jumping information corresponding to the variant model The length of the list, s2 represents the length of the layer jump information list corresponding to the history model.

In this further optional embodiment, the information coding is performed on the layer-hopping information of the model, and the process of obtaining the layer-hopping information list corresponding to the model can be expressed as:

S=(Ls ₁ , Ls ₂ , ..., Ls _N )

Among them, S represents the list of skip layer information of the model, N represents the number of skip layers of the model, and Ls _i is the information code of the ith layer skip layer of the model.

Specifically, the description of the function dist2(S1 _p , S2 _q ) is as follows:

For example, the starting position of the p-th jump layer S1 _p of the variant model is the 3rd layer and the end position is the 7th layer, then the corresponding jump layer S1 _p corresponds to _ps =3, p _l =7-3=4, The starting position of the qth jump layer S2 _q of the historical model is the 4th floor, and the end position is the 9th floor, then the corresponding jump layer S2 _q corresponds to q _s =4, q _l =9-4=5, then dist2 (S1 _p , S2 _q )=((4-3)+(5-4))/(4+5)=2/9.

It can be seen that, implementing this further optional embodiment, the layer-hopping information list is obtained by encoding the layer-hopping information of the model, then the layer-hopping matrix is constructed according to the layer-hopping information list, and the layer-hopping matrix is assigned in a specific way, and finally The layer-jump distance between the models is calculated according to the layer-jump matrix, so that the layer-jump distance between the two models can be calculated.

206. Score each target variant model, and screen out multiple deep learning models corresponding to the slave node from all target variant models corresponding to each slave node according to the score of each target variant model.

In the above step 206, the scoring rules for the target variant model may be consistent with the scoring rules for the above variant models, which will not be repeated here. Here, all target variant models corresponding to each slave node can be classified according to the target basic model on which the target variant model is based, and then the target variant with the highest score can be selected from all target variant models in the classification corresponding to each target basic model. model to form multiple deep learning models corresponding to the slave node.

207. Based on the second process of each slave node, the local cloud computing resources of each slave node, and the determined hyperparameters, perform training and verification operations on each deep learning model corresponding to the slave node, and obtain the corresponding slave node. training results.

In another optional embodiment, training and verification are performed on each deep learning model corresponding to the slave node based on the second process of each slave node, the local cloud computing resources of each slave node, and the determined hyperparameters operation to get the training result corresponding to the slave node, including:

Based on the second process of each slave node and the local cloud computing resources of each slave node, for each deep learning model corresponding to the slave node, formulate a hyperparameter space corresponding to the deep learning model, and the hyperparameter space includes at least batch processing number and learning rate;

Set the number of searches corresponding to each deep learning model corresponding to each slave node;

Construct a set corresponding to each deep learning model corresponding to each slave node, and the set is used to save the score on the validation set after the deep learning model is trained;

The objective function corresponding to each deep learning model corresponding to each slave node is set according to the following formula:

F=max(SC)

Among them, F is the objective function, and SC is the score on the validation set after the deep learning model is trained;

Randomly select a starting point in the hyperparameter space corresponding to each deep learning model corresponding to each slave node, and then cyclically search in the hyperparameter space through Gaussian process mapping to select multiple target hyperparameters corresponding to the deep learning model. parameters, where the Gaussian process map can be expressed as:

T=G(C, R, F, J)

Among them, T is the target hyperparameter corresponding to the deep learning model, each target hyperparameter is the value of the hyperparameter worth trying recommended by G, C is the hyperparameter space corresponding to the deep learning model, and R is the deep learning model. The corresponding set, F is the objective function corresponding to the deep learning model, and J is the number of searches corresponding to the deep learning model;

Perform training and verification operations on the deep learning model based on each target hyperparameter corresponding to each deep learning model corresponding to each slave node, and obtain multiple intermediate deep learning models corresponding to the deep learning model and each intermediate deep learning model. corresponding score;

From all the intermediate deep learning models corresponding to each deep learning model corresponding to each slave node, the intermediate deep learning model with the highest score and the score corresponding to the intermediate deep learning model are selected as the training result corresponding to the slave node.

In this another optional embodiment, the configuration range of the number of batches included in the proposed hyperparameter space may be [64, 512], and the configuration range of the learning rate may be [0.0002, 0.001]. The set number of searches is 20 by default. When the set number of searches is 20, 20 searches are performed in the hyperparameter space to generate 20 target hyperparameters. Gaussian process mapping is a common prediction algorithm used here to predict the value of the next hyperparameter. The score corresponding to each intermediate deep learning model may be the highest accuracy that the intermediate deep learning model can achieve on the validation set.

It can be seen that by implementing this other optional embodiment, multiple hyperparameters can be selected from the hyperparameter space to perform training and verification operations on the deep learning model, so as to obtain the training result corresponding to each hyperparameter, so that the hyperparameters based on the hyperparameter can be obtained. The parameters perform training and validation of the deep learning model.

208. Determine an optimal deep learning model from all target deep learning models according to the scores included in all training results.

For the specific description of the foregoing steps 201 and 208, reference may be made to the specific description of the foregoing steps 101 and 103, which will not be repeated here.

It can be seen that the exploration method of the deep learning model described in FIG. 2 is implemented by determining one cloud computing resource as the master node and multiple other cloud computing resources as multiple slave nodes, and then performing the training operation of the deep learning model based on each slave node. The training result of the slave node finally determines the optimal deep learning model from all the training results according to the score of the training result, so that the training operation of the deep learning model can be performed in parallel, the efficiency of the deep learning model training is improved, and the training is reduced. The time is conducive to the application of deep learning model exploration technology in business scenarios. In addition, by continuously generating variant models, screening the generated variant models based on the model distance, and finally training and verifying the filtered variant models based on different hyperparameters, so as to expand the exploration space of the model and ensure the model training efficiency.

Embodiment 3

Please refer to FIG. 3 , which is a schematic structural diagram of a device for exploring a deep learning model disclosed in an embodiment of the present invention. As shown in Figure 3, the exploration device of the deep learning model may include:

A determination module 301, configured to determine one cloud computing resource as the master node and multiple other cloud computing resources as multiple slave nodes, wherein the master node is used to schedule multiple slave nodes to perform training operations on the deep learning model;

The training module 302 is configured to perform a training operation on the deep learning model based on each slave node, and obtain the training result of the slave node. The training result of each slave node includes the target deep learning model and the score of the target deep learning model. The target deep learning model included in the training result of the slave node is the trained deep learning model;

The determining module 301 is further configured to determine the optimal deep learning model from all target deep learning models according to the scores included in all training results.

It can be seen that the exploration device for implementing the deep learning model described in FIG. 3 determines one cloud computing resource as the master node and multiple other cloud computing resources as multiple slave nodes, and then performs the training operation of the deep learning model based on each slave node. The training result of the slave node is obtained, and finally the optimal deep learning model is determined from all the training results according to the score of the training result, so that the training operation of the deep learning model can be performed in parallel, the efficiency of deep learning model training can be improved, and the The training time is conducive to the application of deep learning model exploration technology in business scenarios.

In an optional embodiment, the training module 302 includes:

Create a submodule 3021 for creating a first process and a second process corresponding to each slave node;

generating submodule 3022, for generating a plurality of deep learning models corresponding to the slave node based on the first process of each slave node;

The training submodule 3023 is used to perform training and verification operations on each deep learning model corresponding to the slave node based on the second process of each slave node, the local cloud computing resources of each slave node, and the determined hyperparameters, and obtain The training result corresponding to the slave node.

It can be seen that the exploration device implementing the deep learning model described in FIG. 4 generates a first process and a second process corresponding to the slave node in each slave node, and continuously generates a new deep learning model through the first process, Through the second process, the newly generated deep learning model is continuously trained and verified to obtain the training result, which can realize the parallelization of the generation and training of the deep learning model, so that the generation and training of the deep learning model do not need to wait for each other, and the deep learning is improved. The training efficiency of the model.

In this optional embodiment, further optional, the generating sub-module 3022 includes:

The selection unit 30221 is used to select multiple historical models from the determined historical model pool as the multiple basic models corresponding to the slave node based on the first process of each slave node, and the historical model is the depth that all slave nodes have generated. learning model;

The selecting unit 30221 is further configured to select multiple base models from all base models corresponding to each slave node based on the determined simulated annealing method as multiple target base models corresponding to the slave node;

Deformation unit 30222, configured to perform model deformation operation on each target base model corresponding to each slave node, to obtain multiple variant models corresponding to the target base model;

The first screening unit 30223 is used to screen out the target variant model corresponding to the target basic model from the plurality of variant models corresponding to the target basic model;

The second screening unit 30224 is configured to score each target variant model, and screen out a plurality of deep learning models corresponding to the slave node from all target variant models corresponding to each slave node according to the score of each target variant model ;

The model deformation operation refers to randomly performing at least one of a network structure deepening operation, a network structure widening operation, and a skip layer structure operation on the neural network model.

It can be seen that the exploration device implementing the deep learning model described in FIG. 4 continuously generates variant models, screens the generated variant models based on the model distance, and finally performs training and verification operations on the filtered variant models based on different hyperparameters. Therefore, the training efficiency of the model can be ensured while expanding the exploration space of the model.

In this further optional embodiment, still further optional, the first screening unit 30223 includes:

The calculation subunit 302231 is used to calculate the model distance between each variant model corresponding to the target basic model and each historical model in the historical model pool;

The judging subunit 302232 is used to judge whether each variant model corresponding to the target basic model has a matching history model, and the matching history model corresponding to the variant model is a history model whose model distance from the variant model is less than the preset threshold. When judging When it is found that the variant model has a matching historical model, the variant model is deleted from the multiple variant models corresponding to the target base model;

The selection subunit 302233 is used to score each variant model corresponding to the target base model, and select the variant model with the highest score from the multiple variant models corresponding to the target base model as the target variant model corresponding to the target base model .

It can be seen that the exploration device implementing the deep learning model described in FIG. 4 can delete the variant model whose distance from the historical model is too small, and select the variant model with the highest score from the remaining variant models as the target variant model. The target variant model can be screened from multiple variant models, reducing the amount of subsequent data processing and improving the efficiency of training.

In this further optional embodiment, still further optional, the calculation subunit 302231 includes:

The second-level calculation subunit 3022311 is used to calculate the common layer distance between each variant model corresponding to the target basic model and each historical model in the historical model pool;

The calculation secondary subunit 3022311 is also used to calculate the jump distance between each variant model corresponding to the target basic model and each historical model in the historical model pool;

The addition secondary subunit 3022312 is used to add the normal layer distance and jump layer distance of each variant model corresponding to the target base model and each historical model in the historical model pool as the model of the variant model and the historical model distance.

It can be seen that the exploration device implementing the deep learning model described in FIG. 4 obtains the model distance between the two models by calculating the normal layer distance and the jump layer distance between the two models, which can effectively compare the two models, so as to accurately Effectively filter out target variant models.

In this still further optional embodiment, still further optional, the specific method for calculating the second-level subunit 3022311 to calculate the common layer distance between each variant model corresponding to the target basic model and each historical model in the historical model pool for:

Perform information encoding on the common layer of each variant model corresponding to the target basic model to obtain a list of common layer information corresponding to the variant model, and encode information on the common layer of each historical model in the historical model pool to obtain the history A list of common layer information corresponding to the model;

According to the normal layer information list corresponding to each variant model corresponding to the target basic model and the normal layer information list corresponding to each historical model in the historical model pool, construct the normal layer matrix corresponding to the variant model and the historical model;

Assign values to each variant model corresponding to the target base model and each element of the normal layer matrix corresponding to each historical model in the historical model pool in the order from left to right and from top to bottom, where the normal layer The formula for assigning a value to each element of the matrix is as follows:

matrix _ij =min(matrix _ij +dist(M1 _i ,M2 _j ),matrix _i(j-1) ,matrix _(i-1)j )

Wherein, matrix _ij represents the element in the ith row and jth column of the common layer matrix, M1 _i represents the ith common layer of the variant model, and M2 _j represents the jth common layer of the historical model;

Among them, the function dist is used to calculate the distance between the two layers of M1 _i and M2 _j . When M1 _i and M2 _j are different types of layers, the value of dist(M1 _i , M2 _j ) is 1. When M1 _i and M2 j are different types of layers When M2 _j are layers of the same type, the value of dist(M1 _i ,M2 _j ) is calculated by:

Among them, a _k is the k-th parameter information in the information coding of the common layer represented by M1 _i , b _k is the k-th parameter information in the information coding of the common layer represented by M2 _j , and n is the parameter information contained in the information coding. number;

Take each variant model corresponding to the target base model and the element in the lower right corner of the normal layer matrix corresponding to each historical model in the historical model pool as the normal layer distance between the variant model and the historical model.

It can be seen that the exploration device implementing the deep learning model described in FIG. 4 obtains a list of common layer information by encoding the common layer of the model, and then constructs a common layer matrix according to the list of common layer information, and performs a specific method on the common layer matrix. Assign value to get the common layer distance between models, so that the common layer distance between two models can be calculated.

In this still further optional embodiment, still further optional, the specific method for calculating the second-level subunit 3022311 to calculate the jump distance between each variant model corresponding to the target basic model and each historical model in the historical model pool for:

Perform information coding on the skip layer of each variant model corresponding to the target basic model to obtain a list of skip layer information corresponding to the variant model, and perform information coding on the skip layer of each historical model in the historical model pool to obtain the historical model. Corresponding layer jump information list;

According to the layer jump information list corresponding to each variant model corresponding to the target basic model and the layer jump information list corresponding to each historical model in the historical model pool, the layer jump matrix corresponding to the variant model and the historical model is constructed. The number of rows is the length of the layer-hopping information list corresponding to the variant model, and the number of columns of the layer-hopping matrix is the length of the layer-hopping information list corresponding to the historical model;

Each element of the jump-layer matrix corresponding to each variant model corresponding to the target base model and each historical model in the historical model pool is assigned a value according to the following formula:

skip_connection_matrix _pq = dist2(S1 _p , S2 _q )

Among them, skip_connection_matrix represents the skip layer matrix, skip_connection_matrix _pq represents the element in the pth row and the qth column of the skip layer matrix, S1 _p represents the pth skip layer of the variant model, and S2 _q represents the qth skip of the historical model Floor;

Among them, the function dist2 is used to calculate the distance between the two layers S1 _p and S2 _q . When S1 _p and S2 _q are different types of layers, the value of dist2(S1 _p , S2 _q ) is 1, when S1 _p and S2 q When S2 _q are layers of the same type, the value of dist2(S1 _p , S2 _q ) is calculated by:

Among them, p _s represents the layer position index of the starting layer in the p-th layer jumping layer in the variant model, and q _s represents the layer position of the starting layer in the q-th layer jumping layer in the historical model. index, p _l represents the depth of the p-th layer jumping layer in the variant model, q _l represents the q-th layer jumping layer depth in the historical model;

Calculate the jump distance between each variant model corresponding to the target base model and each historical model in the historical model pool according to the following formula:

dist _s =sum(skip_connection_matrix)+|s1-s2|

Among them, dist _s represents the jumping distance between the variant model and the historical model, sum(skip_connection_matrix) means adding and summing each element in the skip_connection_matrix, and s1 represents the jumping information corresponding to the variant model The length of the list, s2 represents the length of the layer jump information list corresponding to the history model.

It can be seen that the exploration device for implementing the deep learning model described in FIG. 4 obtains a layer-jump information list by encoding the layer-jump of the model, then constructs a layer-jump matrix according to the layer-jump information list, and performs the layer-jump matrix in a specific way. Assignment, and finally calculate the jump distance between the models according to the jump matrix, so that the jump distance between the two models can be calculated.

In another optional embodiment, the training sub-module 3023 is based on the second process of each slave node, the local cloud computing resources of each slave node, and the determined hyperparameters for each deep learning model corresponding to the slave node The specific way to perform training and verification operations to obtain the training result corresponding to the slave node is as follows:

Based on the second process of each slave node and the local cloud computing resources of each slave node, for each deep learning model corresponding to the slave node, formulate a hyperparameter space corresponding to the deep learning model, and the hyperparameter space includes at least batch processing number and learning rate;

Set the number of searches corresponding to each deep learning model corresponding to each slave node;

Construct a set corresponding to each deep learning model corresponding to each slave node, and the set is used to save the score on the validation set after the deep learning model is trained;

The objective function corresponding to each deep learning model corresponding to each slave node is set according to the following formula:

F=max(SC)

Among them, F is the objective function, and SC is the score on the validation set after the deep learning model is trained;

Randomly select a starting point in the hyperparameter space corresponding to each deep learning model corresponding to each slave node, and then cyclically search in the hyperparameter space through Gaussian process mapping to select multiple target hyperparameters corresponding to the deep learning model. parameters, where the Gaussian process map can be expressed as:

T=G(C, R, F, J)

Among them, T is the target hyperparameter corresponding to the deep learning model, each target hyperparameter is the value of the hyperparameter worth trying recommended by G, C is the hyperparameter space corresponding to the deep learning model, and R is the deep learning model. The corresponding set, F is the objective function corresponding to the deep learning model, and J is the number of searches corresponding to the deep learning model;

Perform training and verification operations on the deep learning model based on each target hyperparameter corresponding to each deep learning model corresponding to each slave node, and obtain multiple intermediate deep learning models corresponding to the deep learning model and each intermediate deep learning model. corresponding score;

From all the intermediate deep learning models corresponding to each deep learning model corresponding to each slave node, the intermediate deep learning model with the highest score and the score corresponding to the intermediate deep learning model are selected as the training result corresponding to the slave node.

It can be seen that the exploration device implementing the deep learning model described in FIG. 4 can select multiple hyperparameters from the hyperparameter space to perform training and verification operations on the deep learning model, so as to obtain the training results corresponding to each hyperparameter. Implements training and validation of deep learning models based on hyperparameters.

For the specific description of the device for exploring the deep learning model, reference may be made to the specific description of the method for exploring the deep learning model, which will not be repeated here.

It can be seen that the exploration device for implementing the deep learning model described in FIG. 4 determines one cloud computing resource as the master node and multiple other cloud computing resources as multiple slave nodes, and then performs the training operation of the deep learning model based on each slave node. The training result of the slave node is obtained, and finally the optimal deep learning model is determined from all the training results according to the score of the training result, so that the training operation of the deep learning model can be performed in parallel, the efficiency of deep learning model training can be improved, and the reduction of The training time is conducive to the application of deep learning model exploration technology in business scenarios. In addition, by generating the first process and the second process corresponding to the slave node in each slave node, the parallelization of the generation and training of the deep learning model is realized, so that the generation and training of the deep learning model do not need to wait for each other, improving the Training efficiency of deep learning models. In addition, by continuously generating variant models, the generated variant models are screened based on the model distance, and finally the filtered variant models are trained and verified based on different hyperparameters, so as to expand the exploration space of the model while ensuring that The training efficiency of the model.

Embodiment 3

Please refer to FIG. 5. FIG. 5 is a schematic structural diagram of another apparatus for exploring a deep learning model disclosed in an embodiment of the present invention. As shown in Figure 5, the apparatus may include:

a memory 501 storing executable program code;

a processor 502 coupled to the memory 501;

The processor 502 invokes the executable program code stored in the memory 501 to execute the deep learning model exploration method described in the first embodiment or the second embodiment.

Embodiment 4

An embodiment of the present invention discloses a computer-readable storage medium, which stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute the deep learning model exploration method described in the first embodiment or the second embodiment .

Embodiment 5

An embodiment of the present invention discloses a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the first or second embodiment. Describes an exploration method for deep learning models.

The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in One place, or it can be distributed over multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation manner can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by means of hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or that make contributions to the prior art. The computer software products can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory. (Read-Only Memory, ROM), Random Access Memory (Random Access Memory, RAM), Programmable Read-only Memory (Programmable Read-only Memory, PROM), Erasable Programmable Read Only Memory (Erasable Programmable Read Only Memory, EPROM) , One-time Programmable Read-Only Memory (OTPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM), CompactDisc Read-Only Memory , CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

Finally, it should be noted that the method and device for exploring a deep learning model disclosed in the embodiments of the present invention are only preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, but not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that; it can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements to some of the technical features; However, these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A method for exploring a deep learning model, the method comprising:

determining one cloud computing resource as a master node and a plurality of other cloud computing resources as a plurality of slave nodes, wherein the master node is used for scheduling the slave nodes to execute training operation on the deep learning model;

executing a training operation on the deep learning model based on each slave node to obtain a training result of the slave node, wherein the training result of each slave node comprises a target deep learning model and a score of the target deep learning model, and the target deep learning model included in the training result of each slave node is a trained deep learning model;

and determining an optimal deep learning model from all the target deep learning models according to the scores included in all the training results.

2. The method for exploring a deep learning model according to claim 1, wherein said performing a training operation on the deep learning model based on each slave node to obtain a training result of the slave node comprises:

creating a first process and a second process corresponding to each slave node, and generating a plurality of deep learning models corresponding to each slave node based on the first process of each slave node;

and performing training and verification operation on each deep learning model corresponding to each slave node based on the second process of each slave node, the local cloud computing resource of each slave node and the determined hyper-parameter to obtain a training result corresponding to the slave node.

3. The method for exploring a deep learning model according to claim 2, wherein the generating a plurality of deep learning models corresponding to each slave node based on the first process of the slave node includes:

based on the first process of each slave node, selecting a plurality of historical models from the determined historical model pool as a plurality of base models corresponding to the slave node, wherein the historical models are deep learning models generated by all the slave nodes;

selecting a plurality of basic models from all the basic models corresponding to each slave node based on the determined simulated annealing method as a plurality of target basic models corresponding to the slave node;

executing model deformation operation on each target base model corresponding to each slave node to obtain a plurality of variant models corresponding to the target base model, and screening out a target variant model corresponding to the target base model from the plurality of variant models corresponding to the target base model;

scoring each target variant model, and screening a plurality of deep learning models corresponding to the slave nodes from all the target variant models corresponding to the slave nodes according to the score of each target variant model;

the model deformation operation refers to at least one of network structure deepening operation, network structure widening operation and jump layer structure adding operation which are randomly performed on the neural network model.

4. The method for exploring a deep learning model as claimed in claim 3, wherein said step of screening out a target variant model corresponding to the target base model from a plurality of said variant models corresponding to the target base model comprises:

calculating the model distance between each variant model corresponding to the target base model and each historical model in the historical model pool;

judging whether each variant model corresponding to the target base model has a matching historical model, wherein the matching historical model corresponding to the variant model is the historical model of which the model distance from the variant model is smaller than a preset threshold value, and deleting the variant model from a plurality of variant models corresponding to the target base model when the variant model is judged to have the matching historical model;

and scoring each variant model corresponding to the target base model, and selecting the variant model with the highest score from the plurality of variant models corresponding to the target base model as the target variant model corresponding to the target base model.

5. The method for exploring a deep learning model as claimed in claim 4, wherein said calculating a model distance between each said variant model corresponding to the target base model and each said historical model in said historical model pool comprises:

calculating the distance between each variant model corresponding to the target base model and each historical model in the historical model pool;

calculating the layer jump distance between each variant model corresponding to the target base model and each historical model in the historical model pool;

and adding the common layer distance and the jump layer distance of each variant model corresponding to the target base model and each historical model in the historical model pool to serve as the model distance of the variant model and the historical model.

6. The method for exploring a deep learning model as claimed in claim 5, wherein said calculating a distance between a common layer of each said variant model corresponding to the target base model and each said historical model in said historical model pool comprises:

performing information coding on the common layer of each variant model corresponding to the calculated target base model to obtain a common layer information list corresponding to the variant model, and performing information coding on the common layer of each historical model in the historical model pool to obtain a common layer information list corresponding to the historical model;

constructing a common layer matrix corresponding to the variant model and the historical model according to a common layer information list corresponding to each variant model corresponding to the target base model and a common layer information list corresponding to each historical model in the historical model pool;

assigning values to each variant model corresponding to the target base model and each element of the common layer matrix corresponding to each historical model in the historical model pool from left to right and from top to bottom, wherein the formula for assigning each element of the common layer matrix is as follows:

matrix_ij＝min(matrix_ij+dist(M1_i,M2_j),matrix_i(j-1),matrix_(i-1)j)

wherein, matrix_ijM1, an element representing the ith row and the jth column in the normal layer matrix_iThe i-th common layer, M2, representing the variant model_jA jth common layer representing the history model;

where the function dist is used to compute M1_iAnd M2_jDistance between two layers, when M1_iAnd M2_jIn the case of layers of different types, dist (M1)_i,M2_j) When M1 is 1_iAnd M2_jIs the same type of layer, dist (M1)_i,M2_j) The value of (d) is calculated by:

wherein, a_kIs M1_iInformation of the k-th parameter in the information coding of the indicated normal layer, b_kIs M2_jThe kth parameter information in the information coding of the indicated common layer, and n is the number of the parameter information contained in the information coding;

and taking the element of the lower right corner of the common layer matrix corresponding to each variant model corresponding to the target base model and each historical model in the historical model pool as the common layer distance between the variant model and the historical model.

7. The method for exploring a deep learning model according to claim 5 or 6, wherein said calculating a jump distance between each said variant model corresponding to the target base model and each said historical model in said historical model pool comprises:

carrying out information coding on the jump layer of each variant model corresponding to the target basic model to obtain a jump layer information list corresponding to the variant model, and carrying out information coding on the jump layer of each historical model in the historical model pool to obtain a jump layer information list corresponding to the historical model;

constructing a layer jump matrix corresponding to the variant model and the historical model according to a layer jump information list corresponding to each variant model corresponding to the target base model and a layer jump information list corresponding to each historical model in the historical model pool, wherein the row number of the layer jump matrix is the length of the layer jump information list corresponding to the variant model, and the column number of the layer jump matrix is the length of the layer jump information list corresponding to the historical model;

assigning a value to each element of the layer jump matrix corresponding to each of the variety models corresponding to the target base model and each of the historical models in the historical model pool according to the following formula:

skip_connection_matrix_pq＝dist2(S1_p,S2_q)

wherein the skip _ connection _ matrix represents the skip layer matrix, and the skip _ connection _ matrix represents the skip layer matrix_pqRepresenting the elements of the p-th row and q-th column in the layer-hopping matrix, S1_pP-th jump representing the variant model, S2_qRepresenting the qth hop level of the historical model;

wherein the function dist2 is used to calculate S1_pAnd S2_qDistance between two layers when S1_pAnd S2_qIs a different type of layer, dist2 (S1)_p，S2_q) When S1 is 1_pAnd S2_qIs the same type of layer, dist2 (S1)_p，S2_q) The value of (d) is calculated by:

wherein p is_sRepresenting the layer position index of the starting layer in the p-th layer jump in the variant model, q_sAn index of layer positions in the model representing the starting layer in the q-th layer jump in the historical model, p_lRepresenting the depth of the p-th layer jump in the variant model, q_lRepresenting the depth of a q-th layer jump in the historical model;

calculating the jump layer distance between each variant model corresponding to the target base model and each historical model in the historical model pool according to the following formula:

dist_s＝sum(skip_connection_matrix)+|s1-s2|

wherein, dist_sRepresents the skip-layer distance between the variant model and the history model, sum (skip _ connection _ matrix) represents that each element in a skip-layer matrix skip _ connection _ matrix is added and summed, s1 represents the length of the skip-layer information list corresponding to the variant model, and s2 represents the length of the skip-layer information list corresponding to the history model.

8. The method for exploring a deep learning model according to any one of claims 2 to 7, wherein the performing a training and verification operation on each deep learning model corresponding to each slave node based on the second process of each slave node, the local cloud computing resource of each slave node and the determined hyper-parameter to obtain a training result corresponding to the slave node comprises:

drawing up a hyper-parameter space corresponding to each deep learning model aiming at each deep learning model corresponding to each slave node based on the second process of each slave node and the local cloud computing resource of each slave node, wherein the hyper-parameter space at least comprises batch processing quantity and learning rate;

setting the number of searching times corresponding to each deep learning model corresponding to each slave node;

constructing a set corresponding to each deep learning model corresponding to each slave node, wherein the set is used for storing scores of the deep learning models on a verification set after training;

setting an objective function corresponding to each deep learning model corresponding to each slave node according to the following formula:

F＝max(SC)

f is a target function, and SC is the score of the deep learning model on the verification set after training;

randomly selecting a starting point in the hyper-parameter space corresponding to each deep learning model corresponding to each slave node, and then circularly searching in the hyper-parameter space through Gaussian process mapping to select a plurality of target hyper-parameters corresponding to the deep learning model, wherein the Gaussian process mapping can be expressed as:

T＝G(C、R、F、J)

the method comprises the following steps that T is a target hyper-parameter corresponding to the deep learning model, each target hyper-parameter is a value of a hyper-parameter worth trying recommended by G, C is a hyper-parameter space corresponding to the deep learning model, R is a set corresponding to the deep learning model, F is a target function corresponding to the deep learning model, and J is the number of search times corresponding to the deep learning model;

training and verifying the deep learning model based on each target hyper-parameter corresponding to each deep learning model corresponding to each slave node to obtain a plurality of intermediate deep learning models corresponding to the deep learning model and a score corresponding to each intermediate deep learning model;

and selecting the intermediate deep learning model with the highest score and the score corresponding to the intermediate deep learning model from all the intermediate deep learning models corresponding to the slave nodes as training results corresponding to the slave nodes.

9. An apparatus for exploring a deep learning model, the apparatus comprising:

the determining module is used for determining one cloud computing resource as a master node and a plurality of other cloud computing resources as a plurality of slave nodes, wherein the master node is used for scheduling the slave nodes to execute training operation on the deep learning model;

the training module is used for executing training operation on the deep learning model based on each slave node to obtain a training result of the slave node, the training result of each slave node comprises a target deep learning model and the score of the target deep learning model, and the target deep learning model included in the training result of each slave node is a trained deep learning model;

the determining module is further configured to determine an optimal deep learning model from all the target deep learning models according to the scores included in all the training results.

10. An apparatus for exploring a deep learning model, the apparatus comprising:

a memory storing executable program code;

a processor coupled with the memory;

the processor calls the executable program code stored in the memory to execute the exploration method of the deep learning model according to any one of claims 1-8.

Images