TWI773100B - Neural network structure search method and apparatus - Google Patents

Abstract

The present invention discloses a neural network structure search method and apparatus, and the method includes: obtaining configuration information of a search task, and generating a plurality of candidate model structures and their corresponding parameters according to the configuration information; creating a plurality of training tasks according to the plurality of candidate model structures and parameters; scheduling the plurality of training tasks to their corresponding physical nodes, so that each physical node parallelly executes its corresponding training task, and calculating the statistics of training results of the plurality of training tasks; when the training result satisfies the predetermined training conditions of the configuration information, the corresponding search result of the search task is obtained. The invention not just improves the search efficiency only, but also improves the utilization of cluster resources and the used hardware resources, and the scalability of hardware acceleration resources to meet the requirement of large-scale search.

Description

Neural network architecture retrieval method and device

The invention relates to the field of deep learning, in particular to the field of neural network architecture retrieval.

With the development of artificial intelligence technology, deep learning neural network architecture retrieval technology has gradually matured. The deep learning neural network architecture retrieval technology can automatically build a deep learning neural network that meets the needs, reduce manpower input, improve manpower efficiency, and has high industrial value.

However, with the current deep learning neural network architecture retrieval technology, the retrieval task is submitted to the cluster management system as a single task, and the retrieval task is executed on a single machine, which is difficult to perform in a distributed manner on a cluster composed of multiple machines. The retrieval tasks are executed in parallel, resulting in lower retrieval efficiency and poor utilization of cluster resources.

Embodiments of the present invention provide a neural network architecture retrieval method and device to solve the problems existing in the related art. The technical solutions are as follows:

A first aspect provides a neural network architecture retrieval method, comprising: acquiring configuration information of a retrieval task, generating multiple candidate model structures and corresponding parameters according to the configuration information; creating multiple training tasks according to the multiple candidate model structures and parameters; Scheduling multiple training tasks to the corresponding physical nodes, so that each physical node executes the corresponding training tasks in parallel, and counts the training results of the multiple training tasks; Under the condition that the training result satisfies the preset training condition in the configuration information, the retrieval result corresponding to the retrieval task is obtained.

In an embodiment, the method further includes: if the training result does not meet the preset training condition in the configuration information, returning to execute the step of generating multiple candidate model structures and corresponding parameters according to the configuration information.

In one embodiment, the configuration information is configuration information input by a user, and the configuration information includes retrieval task information and training task information.

In one embodiment, generating multiple candidate model structures and corresponding parameters according to the configuration information includes: creating a corresponding algorithm instance according to the retrieval task information; loading the training task information into the algorithm instance to generate multiple candidate model structures and corresponding parameters. parameter.

In one embodiment, the retrieval task information includes a retrieval space and a retrieval algorithm of a traditional model structure; creating a corresponding algorithm instance according to the retrieval task information includes: creating a corresponding algorithm instance according to the retrieval space and the retrieval algorithm of the traditional model structure.

In one embodiment, the retrieval task information includes a retrieval algorithm of a retrieval space and a single model structure; creating a corresponding algorithm instance according to the retrieval task information includes: creating a corresponding algorithm instance according to the retrieval algorithm of the retrieval space and the single model structure .

In one embodiment, the method further includes: when the candidate model structure needs to be tuned, creating a plurality of model tuning tasks according to the plurality of candidate model structures and corresponding parameters; Schedule multiple model tuning tasks to the corresponding physical nodes, so that each physical node executes the corresponding model tuning tasks in parallel, obtains multiple optimized candidate model structures, and counts the training of multiple optimized candidate model structures result.

In one embodiment, scheduling multiple training tasks to corresponding physical nodes includes: calculating multiple resource type weights corresponding to multiple training tasks according to the number of resources used by all training tasks corresponding to the retrieval task; The multiple resource weights corresponding to the training tasks determine the scheduling sequence of the multiple training tasks; the multiple training tasks are scheduled to the corresponding physical nodes according to the scheduling sequence.

In one embodiment, the training result includes actual training times and actual values of the training index, and the preset training conditions include the maximum retrieval times and the expected value of the training index; the training results of multiple training tasks satisfy the preset training conditions in the configuration information In the case of , obtaining the retrieval result corresponding to the retrieval task, including: obtaining the optimal network structure and the corresponding The parameters, the optimal network structure and the corresponding parameters are used as the retrieval results.

In an embodiment, the method further includes: creating a corresponding temporary task for each training task, where the temporary task is used to store the training result in the data storage.

In a second aspect, an embodiment of the present invention provides a neural network architecture retrieval device, including: a configuration information acquisition module for acquiring configuration information of a retrieval task; The candidate model structure generation module is used to generate multiple candidate model structures and corresponding parameters according to the configuration information; the training task creation module is used to create multiple training tasks according to the multiple candidate model structures and parameters; the training task scheduling module , which is used to schedule multiple training tasks to the corresponding physical nodes, so that each physical node executes the corresponding training tasks in parallel, and counts the training results of multiple training tasks; In the case of the preset training conditions in the information, the retrieval result corresponding to the retrieval task is obtained.

In one embodiment, it further includes: an iterative retrieval triggering module, configured to trigger the candidate model structure generation module to perform the generation of multiple Steps for candidate model structures and corresponding parameters.

In one embodiment, the configuration information is configuration information input by a user, and the configuration information includes retrieval task information and training task information.

In one embodiment, the candidate model structure generation module includes: an algorithm instance creation submodule for creating a corresponding algorithm instance according to the retrieval task information; a candidate model structure generation submodule for loading the training task information into In the algorithm instance, multiple candidate model structures and corresponding parameters are generated.

In one embodiment, the retrieval task information includes a retrieval space and a retrieval algorithm of a traditional model structure, and the algorithm instance creation sub-module includes: a first algorithm instance creation unit for creating a retrieval algorithm based on the retrieval space and the traditional model structure. The corresponding algorithm example.

In one embodiment, the retrieval task information includes a retrieval space and a retrieval algorithm for a single model structure, an algorithm instance creation sub-module further includes: a second algorithm instance creation unit for retrieval based on the retrieval space and the single model structure Algorithm, create the corresponding algorithm instance.

In one embodiment, it further includes: a model tuning task creation module, configured to create multiple model tuning tasks according to multiple candidate model structures and corresponding parameters when the candidate model structure needs to be tuned; The optimal task scheduling module is used to schedule multiple model tuning tasks to the corresponding physical nodes, so that each physical node executes the corresponding model tuning tasks in parallel, obtains multiple optimized candidate model structures, and counts multiple The training results of the optimized candidate model structure.

In one embodiment, the training task scheduling module includes: a resource type weight calculation sub-module, configured to calculate multiple resource type weights corresponding to multiple training tasks according to the number of resources used by all training tasks corresponding to the retrieval task ; The scheduling order determination sub-module is used to determine the scheduling order of multiple training tasks according to multiple resource weights corresponding to multiple training tasks; the training task scheduling sub-module is used to schedule multiple training tasks to the corresponding physical node.

In one embodiment, the training result includes the actual number of times of training and the actual value of the training index, the preset training condition includes the maximum number of retrieval times and the expected value of the training index, and the retrieval result generation module includes: The retrieval result generation sub-module is used to obtain the optimal network structure and corresponding parameters when the actual training times are greater than or equal to the maximum retrieval times, or the actual value of the training index is greater than or equal to the expected value of the training index. The structure and the corresponding parameters are used as the retrieval result.

In one embodiment, the method further includes: a temporary task creation module, configured to create a corresponding temporary task for each training task, and the temporary task is used to store the training result in the data storage.

In a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor , to enable at least one processor to perform any of the above methods.

In a fourth aspect, there is provided a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause a computer to perform any of the above methods.

An embodiment in the above application has the following advantages or beneficial effects: because the configuration information of the retrieval task is used to dynamically generate multiple candidate model structures and corresponding parameters, and create training tasks for multiple candidate model structures, the multiple training tasks are The physical node corresponding to the scheduling executes multiple training tasks in parallel, and then obtains the retrieval results, so not only improves the retrieval efficiency, but also improves the utilization rate of cluster resources, and the used hardware resources and hardware acceleration resources are more scalable. Meet the needs of large-scale retrieval. Other effects of the above-mentioned optional manners will be described below with reference to specific embodiments.

110: Configuration information acquisition module

120: Candidate model structure generation module

121: Algorithm instance creation submodule

122: Candidate model structure generation submodule

130: Training task creation module

140: Training task scheduling module

141: Resource type weight calculation sub-module

142: scheduling order to determine submodules

143: Training Task Scheduling Submodule

150: Retrieval result generation module

151: Search result generation submodule

160: Iterative retrieval trigger module

170: Model tuning task creation module

180: Model Tuning Task Scheduling Module

190: Temporary task creation mod

901: Processor

902: Memory

903: Input Device

904: Output device

S110: Obtain the configuration information of the retrieval task

S120: Generate multiple candidate model structures and corresponding parameters according to the configuration information

S130: Create multiple training tasks according to multiple candidate model structures and parameters

S140: Schedule multiple training tasks to corresponding physical nodes, so that each physical node executes the corresponding training tasks, and statistics the training results of multiple training tasks

S150: Obtain a retrieval result corresponding to the retrieval task under the condition that the training result satisfies the preset training condition in the configuration information

S121: Create a corresponding algorithm instance according to the retrieval space and the retrieval algorithm of the single model structure

141: Calculate the weights of multiple resource types corresponding to multiple training tasks according to the number of resources used by all training tasks corresponding to the retrieval task

142: Determine the scheduling order of multiple training tasks according to multiple resource weights corresponding to multiple training tasks

143: Schedule multiple training tasks to the corresponding physical nodes according to the scheduling order

S151: Under the condition that the actual training times is greater than or equal to the maximum retrieval times, or the actual value of the training index is greater than or equal to the expected value of the training index, obtain the optimal network structure and corresponding parameters, and the optimal network structure and corresponding parameters are used as retrieval result

S160: When the candidate model structure needs to be tuned, create multiple model tuning tasks according to the multiple candidate model structures and corresponding parameters

S170: Scheduling multiple model tuning tasks to corresponding physical nodes, so that each physical node executes the corresponding model tuning tasks in parallel, obtains multiple optimized candidate model structures, and counts the multiple optimized candidate model structures The training result of

S180: Create a corresponding temporary task for each training task, and the temporary task is used to store the training result in the data storage

[Fig. 1] is a schematic diagram of a neural network architecture retrieval method according to an embodiment of the present invention; [Fig. 2] is a schematic diagram of a neural network architecture retrieval scenario according to an embodiment of the present invention; [Fig. 3] is according to the present invention A schematic diagram of another neural network architecture retrieval method according to an embodiment; [FIG. 4] is a schematic diagram of another neural network architecture retrieval method according to an embodiment of the present invention; [FIG. 5] is a training task according to an embodiment of the present invention A schematic diagram of a scheduling method; [Fig. 6] is a schematic diagram of a neural network architecture retrieval device according to an embodiment of the present invention; [Fig. 7] is a schematic diagram of a neural network architecture retrieval device according to another embodiment of the present invention; [Fig. 8] is a schematic diagram of a training task scheduling module according to an embodiment of the present invention; [Fig. 9] is a block diagram of an electronic device used to implement a neural network architecture retrieval method according to an embodiment of the present invention.

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

As shown in FIG. 1 , in a specific implementation manner, a neural network architecture retrieval method is provided, including: step S110 : obtaining configuration information of a retrieval task; step S120 : generating multiple candidate model structures and corresponding parameters according to the configuration information Step S130: create multiple training tasks according to multiple candidate model structures and parameters; Step S140: schedule multiple training tasks to corresponding physical nodes, so that each physical node executes the corresponding training tasks in parallel, and counts multiple training tasks The training result of the task; Step S150: Obtain a retrieval result corresponding to the retrieval task when the training result satisfies the preset training condition in the configuration information.

In one example, as shown in Figure 2, you can choose to run components such as controllers, schedulers, data storage, and API (application) servers in the K8S system (Kubernetes, container cluster management system). The application container engine) container runs on the master node of the K8S system.

As shown in FIG. 3, first, the controller obtains the configuration information of the retrieval task. The configuration information may be default configuration information stored in a configuration file, or configuration information input by the user through an interface operation, or retrieval task information input by the user through a command line on the client. The configuration information includes retrieval task information and training task information. Add missing information or default initial values to the configuration information, and set the configuration information completely. And verify the complete configuration information, for example, determine that the search space entered by the user is an appropriate search space: if the user selects an integer variable, the search space needs to provide the minimum and maximum integers; if the user selects a discrete variable, then The search space needs to provide a discrete list, indicating all search options; if the user selects a floating-point variable, the search space needs to provide the minimum value, maximum value and step size. Check whether a valid operation type is selected: if the operation type is a convolution operation, the parameters corresponding to the convolution operation include the size of the filter, the number of filters, and the stride. After verifying the configuration information, the controller sends a request to create a training task to the API server through the Golang client provided by Kubernetes, and the API server persists the request to create a training task to the data storage. The API server relies on the data store, which is the storage function provided by the ETCD (distributed consistent key-value store). The API server sends the request to create the training task to the processor of the retrieval task for further processing.

The processor of the retrieval task uses the component of Informer (a client tool with a local cache and indexing mechanism) in the K8S system to monitor the events of the retrieval task in the API server.

The Informer component registers callback handlers for events such as retrieval tasks, training tasks, and algorithms. When any events in the retrieval tasks, training tasks, and algorithm events change, the processor uses the client of the K8S system to process these changes. After receiving the above events, the processor first puts these events into the event queue maintained in the memory, and a special Worker (worker server) coroutine is responsible for fetching events from the event queue. The processor obtains the configuration information of the retrieval task from the API server through the client of the K8S system according to the reference object of the event.

Secondly, after receiving the request to create a training task, the processor of the retrieval task obtains the configuration information of the retrieval task from the API server through the client provided by the K8S system. The processor creates an algorithm instance according to the retrieval space and retrieval algorithm defined by the user in the retrieval task information, and uses the algorithm instance to run the retrieval algorithm. For example, the processor creates corresponding algorithm instances according to different types of retrieval algorithms (retrieval algorithms for traditional model structures, retrieval algorithms for single-shot model structures, and retrieval algorithms for optimized single-shot model structures, etc.). Depending on the architecture of the algorithm, the algorithm instance will use different implementations. After the initialization of the algorithm instance is completed, the processor will communicate with the controller and the processor through the communication mechanism of the gRPC (google remote procedure call, remote procedure call) framework, and transfer the training task information in the controller with the training task information in the processor. Algorithm instances interact to generate multiple candidate model structures and corresponding parameters (eg, weights). The processor accesses the algorithm instance through the HTTP interface, and obtains multiple candidate model structures and corresponding parameters through the HTTP interface.

Then, after the processor obtains multiple candidate model structures and corresponding parameters by accessing the algorithm instance, a training task of the candidate model structures is created. A candidate model structure corresponds to creating a training task, and a training task is distributed to physical nodes on multiple container groups (pods) for parallel execution. and multiple The training task of the candidate model structure is sent to the API server. Since the processor can monitor the API server, when there is a change in the creation of training tasks in the API server, the processor obtains the training task information in the configuration information from the API server through the client of the K8S system, and creates a bearer based on the training task information. The container group (pod) of the training load. The container group (pod) runs with physical nodes. This function is implemented by Kubeflow. After the container group (pod) is created, the scheduler determines the physical nodes to which multiple training tasks need to be scheduled according to the training task information, so that these training tasks can be executed by multiple physical nodes in parallel. The processor will determine whether the training task requires hardware accelerator resources such as GPU or CPU according to the training task information, so as to improve the used hardware resources and the scalability of hardware acceleration resources. Each training task will accept the candidate model structure as input, and use the training code in the training task information to train. After the training task is completed, the training task exits, and the processor modifies the state of the training task to complete. Subsequently, the processor will count the training results, and the training results include: the total number of training tasks completed in the same retrieval task, the actual value of the training indicators obtained by each training task, and the like.

Finally, the processor compares the maximum retrieval times in the retrieval task information and the expected value of the training index with the actual training times in the training result and the actual value of the training index. If the training result is greater than the corresponding value in the retrieval task information, then The processor marks the status of this retrieval task as complete.

In this embodiment, the deep learning neural network architecture retrieval technology utilizes the distributed execution capability of the K8S system to realize large-scale deep learning neural network architecture retrieval. When performing a retrieval task, the user only needs to provide configuration information of the retrieval task, such as retrieval task information and training task information, to initiate a retrieval task of the deep learning neural network architecture. Users do not need to build the operating environment for deep learning neural network architecture retrieval by themselves, do not need to apply for cluster resources required for training tasks, and do not need to design content related to retrieval algorithms by themselves, but are automatically implemented by the K8S system, which improves the convenience of retrieval. , adaptability, and retrieval speed. Dynamically based on the configuration information of the retrieval task entered by the user Generate multiple candidate model structures and corresponding parameters, and create training tasks for multiple candidate model structures, use the K8S system to schedule the corresponding physical nodes of multiple training tasks, execute multiple training tasks in parallel, and then obtain retrieval results . It not only improves the retrieval efficiency, but also improves the utilization rate of cluster resources, and the used hardware resources and hardware acceleration resources have enhanced scalability to meet the needs of large-scale retrieval.

In an embodiment, as shown in FIG. 4 , the method further includes: if the training result does not meet the preset training conditions in the configuration information, returning to execute the step of generating multiple candidate model structures and corresponding parameters according to the configuration information.

In one example, the processor compares the maximum number of retrievals in the retrieval task information and the expected value of the training index with the actual number of trainings in the training result and the actual value of the training index, if the training result is less than the corresponding value in the retrieval task information. , the processor triggers to return to step S120-step S140. Specifically, the processor will use the training indicators and candidate model structures of the historical training tasks as input, and re-interact with the algorithm instance. The algorithm instance will recommend new candidate model structures based on the training indicators and candidate model structures of the historical training tasks, until the maximum The retrieval times are less than or equal to the actual training times, or the actual value of the training index is better than the expected value.

In one embodiment, the configuration information is configuration information input by a user, and the configuration information includes retrieval task information and training task information.

In an example, the configuration information input by the user may include: retrieval task information and training task information. The retrieval task information includes: the namespace where the retrieval task is located (NameSpace), the name of the retrieval task, the data set and version that the retrieval task needs to use, the model structure and version required to execute the retrieval task, the training index name of the retrieval task, and Expected values of training indicators, definition of retrieval space (operations, such as convolution operations, etc.; operation types, such as discrete variables, shaping variables, etc.), retrieval failure times, maximum retrieval times, retrieval algorithm. The retrieval algorithm can include Including: search algorithm name (such as providing a mirror image of the search algorithm when using a custom algorithm), search algorithm type (such as one-time algorithm type, traditional algorithm type, etc.), hardware resources used by the search algorithm (such as GPU, CPU, persistent storage, memory, etc.). The training task information includes: the training code for a single execution of the training task, the hardware resources (such as CPU, memory, GPU, etc.) for the single execution of the training task, the training mode for the single execution of the training task (such as the distribution of TensorFlow based on data flow programming mode, Parameter Server parameter server, Worker work server mode, etc.) and so on.

In one embodiment, as shown in FIG. 4 , step S120 includes: step S121 : creating a corresponding algorithm instance according to the retrieval task information; step S122 : loading the training task information into the algorithm instance, generating multiple candidate model structures and corresponding parameters.

In an example, as shown in FIG. 3 , the processor creates a corresponding algorithm instance according to the algorithm type in the retrieval task information. For example, for the retrieval algorithm of the traditional model structure, the processor uses the algorithm code in the retrieval task information to create a corresponding algorithm instance. For retrieval algorithms that require only one model training, such as DARTS (framework of neural network retrieval technology based on gradient backpropagation), the processor uses the algorithm code in the retrieval task information to create a corresponding algorithm instance. Then, the processor loads the training task information into the created algorithm instance, and generates multiple candidate model structures and corresponding parameters.

In this embodiment, when performing a retrieval task, the user only needs to provide configuration information of the retrieval task, for example, retrieval task information and training task information, to initiate a retrieval task of a deep learning neural network architecture, which improves the convenience of retrieval. At the same time, since corresponding algorithm instances are created according to different algorithm types, and then multiple candidate model structures and corresponding parameters are generated, the adaptability of retrieval is improved.

In one embodiment, the retrieval task information includes a retrieval space and a retrieval algorithm of the traditional model structure; step S121 includes: creating a corresponding algorithm instance according to the retrieval space and the retrieval algorithm of the traditional model structure.

In an example, as shown in Figure 3, if the algorithm type is a retrieval algorithm with a traditional model structure, for example, a retrieval algorithm selected by the user for random retrieval, the algorithm instance will randomly select values in the retrieval space. The processor will find the corresponding image for random retrieval according to the image list in the configuration information, and use this image to create an algorithm container. The algorithm container is an API server in the form of RESTful (REpresentational State Transfer) externally. Provide services.

In another example, the user selects a retrieval algorithm based on deep learning, and the processor will create an algorithm instance according to the relevant information in the configuration information. The algorithm instance will establish an LSTM (Long Short-Term Memory, long short-term memory network), and then the generated candidate model structure is the model structure of deep learning.

In one embodiment, the retrieval task information includes a retrieval space and a retrieval algorithm for the single model structure; step S121 includes: creating a corresponding algorithm instance according to the retrieval space and the retrieval algorithm for the single model structure.

In an example, as shown in FIG. 3 , if the algorithm type is a retrieval algorithm of a single model structure, for example, a subgraph retrieval algorithm such as DARTS selected by the user, both the model structure and the model weight are generated. The processor creates an instance of the algorithm to complete the training. After the training is completed, the processor uses the obtained subgraph to create a model tuning task, and sends the model tuning task to the API server of K8S. If the user does not need tuning, the retrieved model is pushed directly to the data store or other persistent store.

In another example, if the user selects a subgraph retrieval algorithm that only generates model structures and does not generate model weights, the processor will create an algorithm instance to complete the training. After the training is completed, the processor uses the obtained model substructure to create a training task, and sends the training task to the K8S system API server.

In one embodiment, as shown in FIG. 4 , the method further includes: Step S160 : in the case that the candidate model structure needs to be tuned, create a plurality of model tuning tasks according to the plurality of candidate model structures and corresponding parameters; Step S170 : Schedule multiple model tuning tasks to the corresponding physical nodes, so that each physical node executes the corresponding model tuning tasks in parallel, obtains multiple optimized candidate model structures, and counts the training of multiple optimized candidate model structures result.

In one example, if the configuration information input by the user contains the tuning information related to the candidate model structure, after obtaining the tuning information, the processor sends a request to create a model tuning task to the API server, and the API server receives the request to create a model tuning task. After the model tuning task is requested, the processor listens to the event of creating a model tuning task from the API server, and creates multiple model tuning tasks using multiple candidate model structures and corresponding parameters according to the tuning information. For example, create a complete full graph from subgraphs, or use the full full graph entered by the user for model optimization and evaluation.

In one embodiment, as shown in FIG. 5 , step 140 includes: Step 141 : Calculate the weights of multiple resource types corresponding to multiple training tasks according to the number of resources used by all training tasks corresponding to the retrieval task; Step 142 : determine the scheduling sequence of the multiple training tasks according to the multiple resource weights corresponding to the multiple training tasks; Step 143 : schedule the multiple training tasks to the corresponding physical nodes according to the scheduling sequence.

In an example, when the scheduler executes the scheduling task, the following formula can be used to calculate the resources being used by all retrieval tasks:

Among them, Resource represents the resource type, which can be memory, CPU or hardware accelerator resources, etc.; i is the retrieval task; j is the training task of retrieval task i; k is the maximum retrieval times defined by retrieval task i; Resourcejobj represents the jth task resources used.

The scheduler calculates the resource weight of each training task j: W _j = w _CPU . CPU _i + w _Mem ． Mem _i + w _GPU ． GPU _i

Among them, wcpu, wMem, and wGPU are the resource weights of CPU, memory and GPU respectively, all of which are less than 1; CPUi is the amount of CPU resources used by retrieval task i, Memi is the amount of memory resources used by retrieval task i, and GPUi is the amount of memory resources used by retrieval task i. The number of GPU resources.

When the resources are insufficient to run multiple training tasks in the queue, the scheduler will decide which task to run based on this weighting formula. When the high-weight task resources cannot be satisfied, the scheduler will defer scheduling the second-highest-weight training or tuning task, and so on.

In one embodiment, as shown in FIG. 4 , the training result includes the actual training times and the actual value of the training index, and the preset training condition includes the maximum number of retrieval times and the expected value of the training index; step S150 includes: step S151 : in the actual training When the number of times is greater than or equal to the maximum number of retrieval times, or the actual value of the training index is greater than or equal to the expected value of the training index, the optimal network structure and corresponding parameters are obtained, and the optimal network structure and corresponding parameters are used as the retrieval results.

In one example, the training metrics refer to performance evaluation metrics such as accuracy or speed, including error, accuracy, and variance.

In an embodiment, as shown in FIG. 4 , the method further includes: Step S180 : Create a corresponding temporary task for each training task, and the temporary task is used to store the training result in the data storage.

In one example, training tasks are executed after scheduling is complete. During execution, the processor will create a temporary task for each training task. The temporary task and the training task are in the same namespace. The temporary task is responsible for obtaining training tasks or model tuning from the API server through the client of the K8S system. log of the task and store the training results in the data store. For example, in the logs of streaming training tasks, different processing mechanisms are used to obtain training indicators for different training frameworks, and temporary tasks feed back the training indicators to the data storage.

In another specific embodiment, as shown in FIG. 6 , a neural network architecture retrieval apparatus is provided, including: a configuration information acquisition module 110 for acquiring configuration information of a retrieval task; a candidate model structure generation module 120 for Generate multiple candidate model structures and corresponding parameters according to the configuration information; the training task creation module 130 is used to create multiple training tasks according to the multiple candidate model structures and parameters; the training task scheduling module 140 is used to create multiple training tasks The task is scheduled to the corresponding physical node, so that each physical node executes the corresponding training task in parallel, and counts the training results of multiple training tasks; the retrieval result generation module 150 is used for the training results to meet the preset training in the configuration information In the case of the condition, the retrieval result corresponding to the retrieval task is obtained.

In one embodiment, as shown in Figure 7, it also includes: The iterative retrieval trigger module 160 is used to trigger the candidate model structure generation module 120 to perform the step of generating a plurality of candidate model structures and corresponding parameters according to the configuration information when the training result does not meet the preset training conditions in the configuration information .

In one embodiment, the configuration information is configuration information input by a user, and the configuration information includes retrieval task information and training task information.

In one embodiment, as shown in FIG. 7 , the candidate model structure generation module 120 includes: an algorithm instance creation submodule 121 for creating corresponding algorithm instances according to the retrieval task information; the candidate model structure generation submodule 122 , which is used to load the training task information into the algorithm instance to generate multiple candidate model structures and corresponding parameters.

In one embodiment, the retrieval task information includes a retrieval space and a retrieval algorithm of the traditional model structure, and the algorithm instance creation sub-module 121 includes: a first algorithm instance creation unit, which is used for the retrieval algorithm according to the retrieval space and the traditional model structure, Create the corresponding algorithm instance.

In one embodiment, the retrieval task information includes a retrieval algorithm for the retrieval space and the single-shot model structure, and the algorithm instance creation sub-module 121 further includes: a second algorithm instance creation unit for searching according to the retrieval space and the single-shot model structure. Retrieve the algorithm and create the corresponding algorithm instance.

In one embodiment, as shown in FIG. 7 , it further includes: a model tuning task creation module 170, configured to create a plurality of candidate model structures and corresponding parameters when the candidate model structures need to be tuned Model tuning tasks; The model tuning task scheduling module 180 is used for scheduling multiple model tuning tasks to corresponding physical nodes, so that each physical node executes the corresponding model tuning tasks in parallel to obtain multiple optimized candidate model structures, and Count the training results of multiple optimized candidate model structures.

In one embodiment, as shown in FIG. 8 , the training task scheduling module 140 includes: a resource type weight calculation sub-module 141 for calculating a plurality of training tasks according to the number of resources used by all training tasks corresponding to the retrieval task The weights of multiple resource types corresponding to the tasks; the scheduling sequence determination submodule 142 is used to determine the scheduling sequence of multiple training tasks according to multiple resource weights corresponding to multiple training tasks; the training task scheduling submodule 143 is used for Multiple training tasks are scheduled to corresponding physical nodes according to the scheduling sequence.

In one embodiment, as shown in FIG. 7 , the training result includes the actual training times and the actual value of the training index, the preset training condition includes the maximum number of retrieval times and the expected value of the training index, and the retrieval result generation module 150 includes: The retrieval result generation sub-module 151 is used to obtain the optimal network structure and corresponding parameters when the actual training times are greater than or equal to the maximum retrieval times, or the actual value of the training index is greater than or equal to the expected value of the training index. The network structure and corresponding parameters are used as retrieval results.

In one embodiment, as shown in FIG. 7 , it further includes: a temporary task creation module 190 for creating a corresponding temporary task for each training task, and the temporary task is used for storing the training result in the data storage.

Please refer to the corresponding description in the above method for the functions of each module in each device of the embodiment, and details are not repeated here.

According to an embodiment of the present invention, the present invention further provides an electronic device and a readable storage medium.

As shown in FIG. 9 , it is a block diagram of an electronic device of a neural network architecture retrieval method according to an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the inventions described and/or claimed herein.

As shown in FIG. 9, the electronic device includes: one or more processors 901, a memory 902, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or otherwise as desired. The processor may process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and multiple memories, if desired. Likewise, multiple electronic devices may be connected, each providing some of the necessary operations (eg, as a server array, a group of blade servers, or a multiprocessor system). A processor 901 is taken as an example in FIG. 9 .

The memory 902 is the non-transitory computer-readable storage medium provided by the present invention. Wherein, the memory stores instructions executable by at least one processor, so that the at least one processor executes a neural network architecture retrieval method provided by the present application. The non-transitory computer readable of the present invention The storage medium stores computer instructions, and the computer instructions are used to make the computer execute the neural network architecture retrieval method provided by the present invention.

As a non-transitory computer-readable storage medium, the memory 902 can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as program instructions/modules corresponding to a neural network architecture retrieval method in an embodiment of the present invention (For example, the configuration information acquisition module 110, the candidate model structure generation module 120, the training task creation module 130, the training task scheduling module 140, and the retrieval result generation module 150 shown in FIG. 6). The processor 901 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 902, ie, implements a neural network architecture retrieval method in the above method embodiments.

The memory 902 can include a stored program area and a stored data area, wherein the stored program area can store an operating system, an application program required for at least one function; the stored data area can store a memory created by the use of an electronic device according to a neural network architecture retrieval method data etc. Additionally, memory 902 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 902 may optionally include memory located remotely relative to processor 901, and these remote memories may be connected via a network to an electronic device of a neural network architecture retrieval method. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The electronic device of a neural network architecture retrieval method may further include: an input device 903 and an output device 904 . The processor 901 , the memory 902 , the input device 903 and the output device 904 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 9 .

The input device 903 can receive input numerical or character information, and generate key signal input related to user settings and function control of an electronic device of a neural network architecture retrieval method, such as Input devices such as a touch screen, keypad, mouse, trackpad, touchpad, pointing stick, one or more mouse buttons, trackball, joystick, etc. Output devices 904 may include display devices, auxiliary lighting devices (eg, LEDs), haptic feedback devices (eg, vibration motors), and the like. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.

Various implementations of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that may is a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

These computational programs (also referred to as programs, software, software applications, or codes) include machine instructions for programmable processors, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages calculation program. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or apparatus for providing machine instructions and/or data to a programmable processor ( For example, magnetic disks, optical disks, memories, programmable logic devices (PLDs), including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present invention may be executed in parallel or sequentially or in sequence. It can be performed in different orders, as long as the desired results of the technical solutions disclosed in the present invention can be achieved, and are not limited herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

S110: Obtain the configuration information of the retrieval task

S120: Generate multiple candidate model structures and corresponding parameters according to the configuration information

S130: Create multiple training tasks according to multiple candidate model structures and parameters

S140: Schedule multiple training tasks to corresponding physical nodes, so that each physical node executes the corresponding training tasks in parallel, and counts the training results of the multiple training tasks

S150: Obtain a retrieval result corresponding to the retrieval task under the condition that the training result satisfies the preset training condition in the configuration information

Claims (20)

A neural network architecture retrieval method, comprising: acquiring configuration information of a retrieval task, generating a plurality of candidate model structures and corresponding parameters according to the configuration information; creating a plurality of training candidates according to the plurality of candidate model structures and parameters task; schedule the multiple training tasks to the corresponding physical nodes, so that each physical node executes the corresponding training tasks in parallel, and count the training results of the multiple training tasks; when the training results satisfy the configuration information In the case of the preset training conditions in , obtain the retrieval result corresponding to the retrieval task, the configuration information includes retrieval task information and training task information, and generate a plurality of candidate model structures and corresponding parameters according to the configuration information. The steps include: creating a corresponding algorithm instance according to the retrieval task information; loading the training task information into the algorithm instance to generate the multiple candidate model structures and corresponding parameters. The method according to claim 1, further comprising: in the case that the training result does not meet the preset training condition in the configuration information, returning to executing the generating of multiple candidates according to the configuration information Steps for the model structure and the corresponding parameters. The method according to claim 1, wherein the configuration information is configuration information input by a user. The method according to claim 1, wherein the retrieval task information includes a retrieval space and a retrieval algorithm of a traditional model structure; creating a corresponding algorithm instance according to the retrieval task information, including: A corresponding algorithm instance is created according to the retrieval space and the retrieval algorithm of the traditional model structure. The method according to claim 1, wherein the retrieval task information includes a retrieval space and a retrieval algorithm of a single model structure; creating a corresponding algorithm instance according to the retrieval task information includes: according to the retrieval space and For the retrieval algorithm of the single model structure, a corresponding algorithm instance is created. The method according to claim 5, further comprising: when the candidate model structures need to be tuned, creating multiple model tuning tasks according to the multiple candidate model structures and corresponding parameters; The multiple model tuning tasks are scheduled to corresponding physical nodes, so that each physical node executes the corresponding model tuning tasks in parallel, obtains multiple optimized candidate model structures, and counts the multiple optimized candidate models The training results of the structure. The method according to claim 1, wherein scheduling multiple training tasks to corresponding physical nodes comprises: calculating the multiple training tasks according to the number of resources used by all training tasks corresponding to the retrieval task corresponding multiple resource type weights; according to the multiple resource weights corresponding to the multiple training tasks, determine the scheduling sequence of the multiple training tasks; according to the scheduling sequence, schedule the multiple training tasks to the corresponding physical node. The method according to claim 1, wherein the training results include actual training times and actual values of training indicators, and the preset training conditions include the maximum retrieval times in the retrieval task information and the training indicators When the training results of the multiple training tasks meet the preset training conditions in the configuration information, the retrieval results corresponding to the retrieval tasks are obtained, including: Under the condition that the actual training times is greater than or equal to the maximum retrieval times, or the actual value of the training index is greater than or equal to the expected value of the training index, the optimal network structure and corresponding parameters are obtained, and the optimal network structure and corresponding parameters are obtained. The network structure and corresponding parameters are used as the retrieval result. The method according to claim 1, further comprising: creating a corresponding temporary task for each training task, wherein the temporary task is used to store the training result in a data storage. A neural network architecture retrieval device, characterized in that it includes: a configuration information acquisition module for acquiring configuration information of a retrieval task; a candidate model structure generation module for generating a plurality of candidate model structures and corresponding parameters; a training task creation module for creating multiple training tasks according to the multiple candidate model structures and parameters; a training task scheduling module for scheduling the multiple training tasks to corresponding physical nodes to Make each physical node execute the corresponding training task in parallel, and count the training results of the plurality of training tasks; the retrieval result generation module is used when the training results meet the preset training conditions in the configuration information , obtain the retrieval result corresponding to the retrieval task, the configuration information includes retrieval task information and training task information, and the candidate model structure generation module includes: an algorithm instance creation sub-module, which creates a sub-module according to the retrieval task information A corresponding algorithm instance; a candidate model structure generation sub-module, configured to load the training task information into the algorithm instance, and generate the multiple candidate model structures and corresponding parameters. The device according to claim 10, further comprising: an iterative retrieval trigger module for triggering the candidate model structure generation module to perform the generation of multiple candidates according to the configuration information when the training result does not meet the preset training conditions in the configuration information Steps for the model structure and the corresponding parameters. The device according to claim 10, wherein the configuration information is configuration information input by a user. The device according to claim 10, wherein the retrieval task information includes retrieval space and retrieval algorithms of traditional model structures, and the algorithm instance creation sub-module includes: a first algorithm instance creation unit for The retrieval space and the retrieval algorithm of the traditional model structure create corresponding algorithm instances. The device according to claim 10, wherein the retrieval task information includes a retrieval space and a retrieval algorithm for a single model structure, the algorithm instance creation sub-module further includes: a second algorithm instance creation unit, which uses Create a corresponding algorithm instance according to the retrieval algorithm of the retrieval space and the single-shot model structure. The device according to claim 14, further comprising: a model tuning task creation module, configured to, when the candidate model structures need tuning The parameters create multiple model tuning tasks; the model tuning task scheduling module is used to schedule the multiple model tuning tasks to the corresponding physical nodes, so that each physical node executes the corresponding model tuning tasks in parallel, obtaining A plurality of optimized candidate model structures are obtained, and the training results of the plurality of optimized candidate model structures are counted. The device according to claim 10, wherein the training task scheduling module includes: A resource type weight calculation sub-module, used for calculating the multiple resource type weights corresponding to the multiple training tasks according to the number of resources used by all the training tasks corresponding to the retrieval task; the scheduling sequence determination sub-module is used for According to the multiple resource weights corresponding to the multiple training tasks, the scheduling sequence of the multiple training tasks is determined; the training task scheduling sub-module is configured to schedule the multiple training tasks to the corresponding training tasks according to the scheduling sequence. physical node. The device according to claim 10, wherein the training results include actual training times and actual values of training indicators, and the preset training conditions include the maximum retrieval times in the retrieval task information and the training indicators The expected value of , the retrieval result generation module includes: a retrieval result generation sub-module for when the actual training times are greater than or equal to the maximum retrieval times, or the actual value of the training index is greater than or equal to the In the case of the expected value of the training index, the optimal network structure and corresponding parameters are obtained, and the optimal network structure and corresponding parameters are used as the retrieval result. The device according to claim 10, further comprising: a temporary task creation module, configured to create a corresponding temporary task for each training task, and the temporary task is used to store the training result in a data storage middle. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions Executed by the at least one processor to enable the at least one processor to perform the method of any of claim 1-9. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in any one of claim items 1-9.

Images