CN116432028A - Method, system, equipment and medium for model training based on training platform - Google Patents
Method, system, equipment and medium for model training based on training platform Download PDFInfo
- Publication number
- CN116432028A CN116432028A CN202310283803.1A CN202310283803A CN116432028A CN 116432028 A CN116432028 A CN 116432028A CN 202310283803 A CN202310283803 A CN 202310283803A CN 116432028 A CN116432028 A CN 116432028A
- Authority
- CN
- China
- Prior art keywords
- model file
- file data
- training
- model
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012549 training Methods 0.000 title claims abstract description 264
- 238000000034 method Methods 0.000 title claims abstract description 64
- 238000013136 deep learning model Methods 0.000 claims abstract description 81
- 238000012545 processing Methods 0.000 claims abstract description 42
- 238000012795 verification Methods 0.000 claims description 23
- 238000004891 communication Methods 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000007726 management method Methods 0.000 description 79
- 230000006870 function Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Abstract
The invention provides a method, a system, equipment and a medium for model training based on a training platform, wherein the method comprises the following steps: the model file management module is used for maintaining model file data; when a training task is created, a model file management module is started; according to the target model file data corresponding to the training task, a model file processing module carries out mounting of the target model file data through a model file management module; and the training task carries out deep learning model training based on the mounted target model file data. The training efficiency of the deep learning model training is improved by improving the convenience of the deep learning model training.
Description
Technical Field
The present invention relates to the field of operating system installation technologies, and in particular, to a method, a system, an apparatus, and a medium for model training based on a training platform.
Background
Several ways of performing deep learning model training through model file data currently include: before deep learning model training begins, copying model file data into a public directory, and mounting the public directory when a training task is run, so that a training script can read the model file data in the public directory mounted in a container; one is to write a code for downloading model file data in a training script, download the model file data in real time when the script is run, and then perform deep learning model training after the downloading is completed. The two modes have the problem of easy use of the deep learning model training, so that the model file data needs to be downloaded or copied in real time every time the deep learning model training is carried out, and the convenience of the deep learning model training is insufficient.
Disclosure of Invention
In view of the above, the present invention provides a method, system, device and medium for model training based on a training platform. The training efficiency of the deep learning model training is improved by improving the convenience of the deep learning model training.
In a first aspect of an embodiment of the present invention, there is provided a method for model training based on a training platform, the method including:
the model file management module is used for maintaining model file data;
when a training task is created, a model file management module is started;
according to the target model file data corresponding to the training task, a model file processing module carries out mounting of the target model file data through a model file management module;
and the training task carries out deep learning model training based on the mounted target model file data.
Optionally, the model file data maintained by the model file management module includes: model file data imported from outside, and model file data generated during execution of training tasks.
Optionally, the model file management module is configured to maintain model file data, including:
adding model file data to a database corresponding to the model file management module;
And adding the position information and the parameter information of the model file data in a corresponding model table through the model file management module.
Optionally, the model file processing module carries out mounting of the target model file data through a model file management module according to the target model file data corresponding to the training task, and the method includes:
creating a training task, and displaying a maintained model table through the model file management module;
selecting corresponding target model file data from the model table according to the training task;
and according to the selected target model file data, the model file processing module acquires the model file data from the corresponding database for mounting.
Optionally, the method further comprises:
according to the training task, a mounting path of the corresponding target model file data is customized;
and according to the mounting path, the model file processing module mounts the target model file data from the mounting path.
Optionally, the method further comprises:
verifying the determined target model file data;
when the verification is passed, according to the determined target model file data, the model file processing module acquires the target model file data for mounting;
When the verification fails, prompting corresponding alarm information to a user according to the verification result, wherein the alarm information at least comprises: the determined object model file data does not exist and the determined object model file data is repeated.
Optionally, the method further comprises:
when executing a training task through a script, customizing a mounting path of model file data in a container in the script execution process;
and according to the mounting path, the model file processing module mounts the model file data from the mounting path.
In a second aspect of the embodiment of the present invention, there is provided a system for performing a model training method based on a training platform, the system comprising:
the system comprises:
the model file management module is used for maintaining model file data;
the starting module is used for starting the model file management module when the training task is created;
the mounting module is used for mounting the target model file data through the model file management module according to the target model file data corresponding to the training task;
and the training module is used for training the deep learning model based on the mounted target model file data by the training task.
In a third aspect of the embodiment of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing the steps of the model training method based on the training platform when executing the program stored in the memory.
In a fourth aspect of embodiments of the present invention, there is also provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method for model training based on a training platform according to the first aspect of the present invention.
Aiming at the prior art, the invention has the following advantages:
according to the model training method based on the training platform, a model file management module is built in the deep learning model training platform, and a model file data group is maintained through the model file management module; starting a model file management module when a user creates a training task, providing a maintained model file data group for the user to select, and mounting target model file data by the model file management module according to target model file data corresponding to the training task selected by the user; model training is performed based on the mounted target model file data when the training task is executed. Therefore, the invention constructs a model file management module, maintains a large model file data group through the model file management module, and when a user creates a training task, the model file management module is directly used for carrying out mounting according to the target model file data for the training task selected by the user, and copying and real-time downloading of the model file data are not needed, thereby improving the training convenience of the deep learning model, and further improving the training efficiency of the deep learning model training.
The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flowchart of a method for model training based on a training platform according to an embodiment of the present invention;
FIG. 2 is another flow chart of a method for model training based on a training platform according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a system for performing model training based on a training platform according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings.
Before explaining the present invention, the background of the present invention will be described. At present, the training of the deep learning model through model file data is mainly carried out in the following two modes, namely copying the model file data into a public directory before the training of the deep learning model is started, and mounting the public directory when a training task is operated, so that a training script can read the model file data in the public directory mounted in a container; a code for downloading model file data is written in a training script, the model file data is downloaded in real time when the script is run, and deep learning model training is performed after the downloading is completed. The two modes have the problem of easy usability of the deep learning model training, so that the model file data needs to be downloaded or copied in real time each time the deep learning model training is carried out, the convenience of the deep learning model training is insufficient, and the training efficiency of the deep learning model training is reduced.
In view of this, the invention provides a new model training method based on training platform, by constructing a model file management module, the model file management module is used to maintain a large model file data group; the model file management module is started while the user creates the training task, and the model file management module provides the maintained model file data group to the user for selection by the user. At this time, the user can directly select one or more target model file data from the provided model file data group for the training task, and after the user selects the corresponding one or more target model file data, the model file management module will immediately mount the one or more target model file data from the maintained model file data group for the training task, without downloading or copying the model file data in real time, thereby improving the convenience of training the deep learning model and the training efficiency of training the deep learning model.
Fig. 1 is a flowchart of a method for performing model training based on a training platform according to an embodiment of the present invention, as shown in fig. 1, where the method includes:
Step S101: the model file management module is used for maintaining model file data;
step S102: when a training task is created, a model file management module is started;
step S103: according to the target model file data corresponding to the training task, a model file processing module carries out mounting of the target model file data through a model file management module;
step S104: and the training task carries out deep learning model training based on the mounted target model file data.
In the embodiment of the invention, a model file management module is integrated in a deep learning model training platform and is used for maintaining and managing a model file data group, wherein the model file data group comprises a large amount of model file data, and the model file data group is stored in a database corresponding to a model file management model. When a user creates a training task through the deep learning model training platform, the model file management module is synchronously started. After the model file management module is synchronously started, all model file data in the model file data group maintained by the model file management module are displayed to a user through the deep learning model training platform, and the user can select target model file data for the training task from all model file data in the model file data group maintained by the displayed model file management module. Wherein the object model file data may include one or more model file data. After the user selects the target model file data for the training task from all model file data in the model file data group maintained by the displayed model file management module, the model file data mounting processor in the model file processing module mounts the selected target model file data from the corresponding database into the container, so that the training task can be executed based on the target model file data selected by the user. The container characterizes the unit (dock container) in which the training task runs. After the model file data mounting processor in the model file processing module completes mounting of the target model file data, training is performed on the basis of the mounted target model file data in a corresponding container for training of the model.
According to the model training method based on the training platform, a model file management module is built in the deep learning model training platform, and a model file data group is maintained through the model file management module; starting a model file management module when a user creates a training task, providing a maintained model file data group for the user to select, and mounting target model file data by a model file data mounting processor in the model file processing module according to target model file data corresponding to the training task selected by the user; model training is performed based on the mounted target model file data when the training task is executed. Therefore, the invention maintains a large model file data group through the model file management module, and when a user creates a training task, the model file data is directly mounted according to the target model file data for the training task selected by the user through the model file data mounting processor in the model file processing module, and copying and real-time downloading of the model file data are not needed, thereby improving the training convenience of the deep learning model, and further improving the training efficiency of the deep learning model training.
In the present invention, the model file data maintained by the model file management module includes: model file data imported from outside, and model file data generated during execution of training tasks.
In the embodiment of the invention, in order to improve the diversity of the model file data managed and maintained by the model file management module and the convenience of adding the model file data into the database corresponding to the model file management module, the invention can import the external model file data into the database corresponding to the model file management module, and can directly store the corresponding generated model file data into the database corresponding to the model file management module after the deep learning model is trained to be qualified when the training task is executed.
In the present invention, the model file management module is used for maintaining model file data, and includes: adding model file data to a database corresponding to the model file management module; and adding the position information and the parameter information of the model file data in a corresponding model table through the model file management module.
In an embodiment of the present invention, one implementation manner of the model file management module in the present invention for maintaining model file data is: and adding the model file data which needs to be managed and maintained into a database corresponding to the model file management module. Meanwhile, the model file data group maintained by the model file management module is provided with a corresponding model table, wherein the position information and the parameter information of each model file data in the model file data group stored in the database are recorded. After the model file data which needs to be managed and maintained is added into the database corresponding to the model file management module, the model file data which is added to the model file management module at present is added into the model table, and the position information and the parameter information of the model file data which is added to the model file management module at present are recorded in the model table. The parameter information of the model file data at least comprises scenes, names and version numbers.
In the present invention, the model file processing module carries out mounting of the target model file data through a model file management module according to the target model file data corresponding to the training task, and the method includes: creating a training task, and displaying a maintained model table through the model file management module; selecting corresponding target model file data from the model table according to the training task; and according to the selected target model file data, the model file processing module acquires the model file data from the corresponding database for mounting.
In an embodiment of the present invention, according to the target model file data corresponding to the training task, an implementation manner of the model file data mounting processor in the model file processing module to mount the target model file data through the model file management module is as follows: when a user creates a training task through the deep learning model training platform, the model file management module displays the managed and maintained model table to the user through the deep learning model training platform, and the user selects target model file data for the training task from the model table according to the training task. Wherein the object model file data may include one or more. According to the target model file data selected by the user from the displayed model table, the model file data mounting processor in the model file processing module acquires the target model file data from the corresponding database and mounts the target model file data into the corresponding container, so that the training task can be executed based on the target model file data selected by the user.
In the present invention, the method further comprises: according to the training task, a mounting path of the corresponding target model file data is customized; and according to the mounting path, the model file processing module mounts the target model file data from the mounting path.
In the embodiment of the invention, the invention improves the training diversity of the deep learning model. Another implementation mode of the model file data mounting processor in the model file processing module of the invention for mounting the target model file data is as follows: when a user creates a training task through the deep learning model training platform, the model file management module displays the managed and maintained model table to the user through the deep learning model training platform, and when the model table does not have model file data required by the user or the user wants to use model file data in other places according to own needs, the model file management module can also provide a user-defined function of a mounting path of the model file data for the user. The user can customize the mounting path of the model file data according to the self requirement, and according to the mounting path, the model file data mounting processor in the model file processing module obtains the corresponding target model file data from the mounting path and mounts the corresponding target model file data into the corresponding container, so that the training task can be executed based on the target model file data. For example, the user may customize one or more local mounting paths, at which time the model file data mounting processor in the model file processing module may obtain and mount, based on the one or more mounting paths, one or more corresponding target model file data from a location pointed by the one or more mounting paths into a corresponding container, so that execution of the training task may be performed based on the one or more target model file data.
In the embodiment of the invention, as the multiple target model file data can be selected for the deep learning model training for one training task, when the user determines that the multiple target model file data are subjected to the deep learning model training, part of the target model file data in the multiple target model file data can be from the target model file data selected by the user from the model table, and the other part of the target model file data in the multiple target model file data can be from the target model file data acquired by the user-defined mounting path. Wherein the partial object model file data may include one or more, and the other partial object model file data may also include one or more.
For example, the user creates a training task through the deep learning model training platform, and the training task requires a plurality of target model file data for deep learning model training, wherein the target model file data in the target model file data group maintained by the model file management module in the plurality of target model file data has target model file data a, target model file data b and target model file data c, and the target model file data in the target model file data group not maintained by the model file management module in the plurality of target model file data has target model file data d and target model file data e. For the object model file data a, the object model file data b and the object model file data c, the model table maintained by the model file management module can be directly displayed to the user through the training platform, the user directly selects the object model file data a, the object model file data b and the object model file data c from the model table, and for the object model file data which do not belong to the object model file data group maintained by the model file management module, the object model file data d and the object model file data e exist, and under the condition that the storage paths of the object model file data d and the object model file data e are determined, the mounting paths of the object model file data d and the object model file data e are defined based on the storage paths of the object model file data d and the object model file data e in a self-defining mode. The complete training task is created based on the target model file data a, the target model file data b and the target model file data c which are determined by a user through a model table maintained by the model file management module and based on the target model file data d and the target model file data e which are determined by a user through a self-defined mounting path, so that the training task can perform deep learning model training through the target model file data a, the target model file data b, the target model file data c, the target model file data d and the target model file data e.
In the present invention, the method further comprises: verifying the determined target model file data; when the verification is passed, according to the determined target model file data, the model file processing module acquires the target model file data for mounting; when the verification fails, prompting corresponding alarm information to a user according to the verification result, wherein the alarm information at least comprises: the determined object model file data does not exist and the determined object model file data is repeated.
In the embodiment of the invention, as a training task can select a plurality of target model file data simultaneously for deep learning model training, in order to prevent the training task which is caused by that the target model file data to be mounted or the target model file data selected by a user from a model table is deleted from not being executed in a user-defined mounting path, and in order to prevent the target model file data in the user-defined mounting path and the target model file data selected by the user from the model table from being repeated, a plurality of identical target model file data are simultaneously subjected to the deep learning model training, thereby causing invalid training and causing waste of calculation resources. Thus, the target model file data determined by the user through the model table and/or through the custom mounting path is checked by the model file data checker in the model file processing module to determine whether these determined target model file data exist and whether the same duplicate target model file data exist. When verification passes, namely the determined target model files exist, and repeated identical target model file data do not exist at the same time, and a model file data mounting processor in the model file processing module mounts all target model file data. And when the verification result is that certain target model file data do not exist, alarming the user through the deep learning model training platform to inform the user that the certain target model file data do not exist, and when the verification result is that certain target model file data are repeated, alarming the user through the deep learning model training platform to inform the user that the certain target model file data are repeated, and prompting the user to select one target model file data from the certain target model file data to perform subsequent deep learning model training.
Illustratively, the user creates a training task through the deep learning model training platform, and determines a plurality of target model file data through a model table and/or through a custom mounting path manner, including: object model file data a1, object model file data a2, object model file data a3, object model file data a4, and object model file data a5. The object model file data a1, the object model file data a2 and the object model file data a3 are all determined by a model table maintained by the model file management module, and the object model file data a4 and the object model file data a5 are all determined by a user in a mode of customizing a mounting path of the model file data. And checking whether the target model file data a1, the target model file data a2, the target model file data a3, the target model file data a4 and the target model file data a5 exist and are repeated through a model file data checker in the model file processing module, so as to obtain a corresponding checking result. And finally, the verification result is that the target model file data a1 does not exist, and the target model file data a2 and the target model file data a4 are displayed to belong to repeated model file data. According to the verification result, alarming is carried out on a user through a page of the deep learning model training platform so as to inform the user that the target model file data a1 determined through a model table maintained by the model file management module does not exist, and simultaneously alarming is carried out on the user through the page of the deep learning model training platform so as to inform the user that the target model file data a2 determined through the model table maintained by the model file management module and the target model file data a4 determined through a mode of a mounting path of the user-defined model file data belong to repeated model file data. After the user deletes the selected object model file data a1 based on the alarm information and retains one object model file data a2 from the object model file data a2 and the object model file data a4, the respective object model file data determined by the user, that is, the object model file data a2, the object model file data a3 and the object model file data a5 are checked again, and at this time, the non-existing object model file data and the repeated object model file data are not existed any more, and the verification passes. At this time, the model file data mounting processor in the model file processing module mounts the target model file data a2, the target model file data a3 and the target model file data a5 into corresponding containers, so that the training task can normally perform deep learning model training.
In the present invention, the method further comprises: when executing a training task through a script, customizing a mounting path of model file data in a container in the script execution process; and according to the mounting path, the model file processing module mounts the model file data from the mounting path.
In an embodiment of the invention, the method further comprises: when executing a training task through a script, customizing a mounting path of model file data in a container in the script execution process; and according to the mounting path, a model file data mounting processor in the model file processing module mounts the model file data from the mounting path.
In the embodiment of the invention, in the case that the user wants to perform the deep learning model training through one script used by one other user, the invention is applicable to the scene that the user does not want to modify the script. The invention also provides an implementation mode: when a user executes a deep learning model training task through a script used by other users, the deep learning model training platform will mount model file data under a default mounting path, and the model file data pointed by the mounting path in the script is different from the model file data under the model mounting path mounted by the platform, at this time, the deep learning model training platform will not be able to execute the script to perform deep learning model training. This is because the mounting path of script execution and the default mounting path of the deep learning model training platform are different, for example, the mounting path of script execution is/model/xxx, the model file data 1 is pointed to, the default mounting path of the platform is/a/1 (model name is a, version is 1), the model file data 2 is pointed to, the script wants to execute the model file data 1 under the mounting path/model/xxx, and the actual default mounting of the deep learning model training platform is the model file data 2 under the mounting path/a/1, which are inconsistent. In order to achieve the aim of realizing correct execution of the script without modifying the script, the model file management module constructed by the invention also has the function of customizing the mounting path of the model file data in the container. When a user executes a deep learning model training task through a script used by other users, a model file management module in the deep learning model training platform is used for customizing a mounting path of model file data in a container in the script execution process, namely, a default mounting path of the deep learning model training platform is customized to be a mounting path in the script. In the above example, the default mounting path/a/1 of the deep learning model training platform is customized to be the mounting path/model/xxx of the model file data in the script.
In an embodiment of the present invention, as shown in fig. 2, one implementation manner of a method for performing model training based on a training platform provided by the present invention is: firstly, a user creates a training task through a deep learning model training platform, and simultaneously, determines target model file data for the training task from a model file data group managed by a model file management module and/or determines the target model file data for the training task through a mounting path of custom model file data. Wherein the sources of the model file data in the model file data group managed and maintained by the model file management module comprise model file data which is imported through the outside and model file data which is generated in the training process of the training task. After the user selects corresponding target model file data from the model table provided by the model file management module through the deep learning model training platform and/or determines the corresponding target model file data through the mounting path of the custom model file data, the model file processing module starts working at the moment. Firstly, a model file data verifier in a model file processing module starts to work, and target model file data determined by a user through a model table and/or through a custom mounting path is verified through the model file data verifier in the model file processing module so as to determine whether the determined target model file data exist or not and whether the same repeated target model file data exist or not.
When verification passes, that is, when the determined target model files exist and repeated identical target model file data do not exist, the model file data mounting processor in the model file processing module works. The model file data mounting processor in the model file processing module mounts the target model file data determined by the user through the model table and/or through the custom mounting path into a corresponding container so that the training task can be correctly executed in the container.
And when the verification result is that certain target model file data A does not exist, alarming the user through the deep learning model training platform to inform the user that the certain target model file data A does not exist, and when the verification result is that certain target model file data (such as target model file data B, target model file data C and target model file data D) are repeated, alarming the user through the deep learning model training platform to inform the user that certain target model file data (target model file data B, target model file data C and target model file data D) are repeated, prompting the user to select one target model file data (namely only one target model file data of the target model file data B, the target model file data C and the target model file data D) from the certain target model file data (namely, only one target model file data of the target model file data B, the target model file data C and the target model file data D) to carry out subsequent deep learning model training. After the user performs corresponding correction operation based on the alarm information (if the alarm information prompts that the target model file does not exist, new model file data is redetermined or the target model file data which does not exist is deleted, if the alarm information prompts that a plurality of repeated target model file data exist, one target model file data which is used for executing a subsequent training task is reserved from the repeated target model file data), the target model file data which is determined by the user through a model table and/or a custom mounting path is checked again, and when the user passes the check, that is, the determined target model files exist, and meanwhile, the repeated same target model file data do not exist, at this time, a model file data mounting processor in a model file processing module works. The model file data mounting processor in the model file processing module mounts the target model file data determined by a user through a model table and/or through a custom mounting path into a corresponding container so that a training task can be correctly executed in the container; and when the verification is not passed again, continuing to give a corresponding alarm to the user until the final verification is passed, wherein the model file data mounting processor in the model file processing module can not work so as to mount the target model file data determined by the user through the model table and/or the custom mounting path into a corresponding container, so that the training task can be correctly executed in the container.
The invention provides a method for training a model based on a training platform, which is mainly suitable for training tasks by using one or more model file data, a training script directly uses the model file data to train, the problem of usability when the training tasks train by using the model file data is solved, the deep learning model training method in the invention can effectively select the model file data to be used and/or the mounting path of the self-defined model file data through a page of the deep learning model training platform when the training tasks are created, the deep learning model training platform can automatically mount the model file data with the mounting path selected and/or self-defined into a task container after the training tasks are successfully created, the training script can directly use the model file data with the mounting path selected and/or self-defined, and can aim at the scene that a user wants to use the historical script but does not want to modify the script, and the mounting path of the model file data in the container can be self-defined by the function of the self-defined model file data, the mounting path of the model file data in the deep learning model training platform is self-defined as the mounting path of the model file data defined in the container, and the requirement of the model is directly realized on the premise that the model is not required to modify the model. According to the method for training the model based on the training platform, provided by the invention, a user can efficiently use the model file data to train the deep learning model of the training task, the model file parameters do not need to be modified in a training script, or the model file data is copied in a public directory, and the model file data is downloaded, so that the training efficiency of algorithm personnel can be effectively improved.
A second aspect of the present invention provides a system for performing model training based on a training platform, as shown in fig. 3, the system 300 includes:
a model file management module 301, configured to maintain model file data;
the starting module 302 is used for starting the model file management module when the training task is created;
the mounting module 303 is configured to mount, according to target model file data corresponding to the training task, the target model file data by using a model file management module;
and the training module 304 is used for the training task to perform deep learning model training based on the mounted target model file data.
Optionally, the model file data maintained by the model file management module 301 includes: model file data imported from outside, and model file data generated during execution of training tasks.
Optionally, the model file management module 301 includes:
the storage module is used for adding the model file data to a database corresponding to the model file management module;
and the data adding module is used for adding the position information and the parameter information of the model file data in the corresponding model table through the model file management module.
Optionally, the mounting module 303 includes:
the model table display module is used for creating training tasks and displaying the maintained model table through the model file management module;
the target model file data determining module is used for selecting corresponding target model file data from the model table according to the training task;
and the mounting sub-module is used for obtaining the model file data from the corresponding database according to the selected target model file data and mounting the model file data by the model file processing module.
Optionally, the system 300 further comprises:
the first mounting path self-defining module is used for self-defining the mounting path of the corresponding target model file data according to the training task;
and the first mounting module is used for mounting the target model file data from the mounting path according to the mounting path.
Optionally, the system 300 further comprises:
the verification module is used for verifying the determined target model file data;
the first verification module is used for obtaining the target model file data for mounting according to the determined target model file data when verification is passed;
The second checking module is used for prompting corresponding alarm information to the user according to the checking result when the checking is not passed, and the alarm information at least comprises: the determined object model file data does not exist and the determined object model file data is repeated.
Optionally, the system 300 further comprises:
the second mounting path self-defining module is used for self-defining the mounting path of the model file data in the container in the script executing process when the training task is executed through the script;
and the second mounting module is used for mounting the model file data from the mounting path according to the mounting path.
The invention provides a system for carrying out model training based on a training platform, which is mainly suitable for a scene that a training task uses one or more model file data to train, a training script directly uses the model file data to train, the problem of usability when the training task uses the model file data to train is solved, the deep learning model training method in the invention can effectively select the model file data to be used and/or the mounting path of the self-defined model file data through a page of the deep learning model training platform when the training task is created, the deep learning model training platform can automatically mount the model file data with the mounting path selected and/or self-defined into a task container after the training task is successfully created, the training script can directly use the model file data with the mounting path selected and/or self-defined, and can self-define the mounting path of one model file data in the deep learning model training platform as the mounting path of the self-defined model file data in the container according to the scene that a user wants to use the history script instead of the script, and the mounting path of the model file data in the container can be self-defined, so that the model file data can not be directly required to be directly modified by the model file. According to the system for carrying out model training based on the training platform, provided by the invention, a user can efficiently use model file data to carry out deep learning model training of training tasks, model file parameters do not need to be modified in a training script, or model file data are copied in a public directory, and the model file data are downloaded, so that the training efficiency of algorithm personnel can be effectively improved.
In a third aspect of the embodiments of the present invention, there is further provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing the steps of the model training method based on the training platform when executing the program stored in the memory.
In a fourth aspect of embodiments of the present invention, there is also provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method for model training based on a training platform according to the first aspect of the present invention.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (10)
1. A method for model training based on a training platform, the method comprising:
the model file management module is used for maintaining model file data;
when a training task is created, a model file management module is started;
according to the target model file data corresponding to the training task, a model file processing module carries out mounting of the target model file data through a model file management module;
and the training task carries out deep learning model training based on the mounted target model file data.
2. The method of claim 1, wherein the model file data maintained by the model file management module comprises: model file data imported from outside, and model file data generated during execution of training tasks.
3. The method of claim 1, wherein the model file management module is configured to maintain model file data, comprising:
Adding model file data to a database corresponding to the model file management module;
and adding the position information and the parameter information of the model file data in a corresponding model table through the model file management module.
4. The method according to claim 3, wherein the model file processing module performs mounting of the target model file data through a model file management module according to the target model file data corresponding to the training task, and the method comprises:
creating a training task, and displaying a maintained model table through the model file management module;
selecting corresponding target model file data from the model table according to the training task;
and according to the selected target model file data, the model file processing module acquires the model file data from the corresponding database for mounting.
5. The method according to claim 4, wherein the method further comprises:
according to the training task, a mounting path of the corresponding target model file data is customized;
and according to the mounting path, the model file processing module mounts the target model file data from the mounting path.
6. The method according to claim 4, wherein the method further comprises:
verifying the determined target model file data;
when the verification is passed, according to the determined target model file data, the model file processing module acquires the target model file data for mounting;
when the verification fails, prompting corresponding alarm information to a user according to the verification result, wherein the alarm information at least comprises: the determined object model file data does not exist and the determined object model file data is repeated.
7. The method according to claim 1, wherein the method further comprises:
when executing a training task through a script, customizing a mounting path of model file data in a container in the script execution process;
and according to the mounting path, the model file processing module mounts the model file data from the mounting path.
8. A system for a method of model training based on a training platform, the system comprising:
the model file management module is used for maintaining model file data;
the starting module is used for starting the model file management module when the training task is created;
The mounting module is used for mounting the target model file data through the model file management module according to the target model file data corresponding to the training task;
and the training module is used for training the deep learning model based on the mounted target model file data by the training task.
9. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the steps of a method for model training based on a training platform according to any of claims 1-7 when executing a program stored on a memory.
10. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a method of model training based on a training platform according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310283803.1A CN116432028A (en) | 2023-03-22 | 2023-03-22 | Method, system, equipment and medium for model training based on training platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310283803.1A CN116432028A (en) | 2023-03-22 | 2023-03-22 | Method, system, equipment and medium for model training based on training platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116432028A true CN116432028A (en) | 2023-07-14 |
Family
ID=87084658
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310283803.1A Pending CN116432028A (en) | 2023-03-22 | 2023-03-22 | Method, system, equipment and medium for model training based on training platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116432028A (en) |
-
2023
- 2023-03-22 CN CN202310283803.1A patent/CN116432028A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7657884B2 (en) | Electronic device supporting multiple update agents | |
| US9319508B2 (en) | Method and device for adding menu item to android menu | |
| US20110107137A1 (en) | System and method for providing automated support to electronic devices | |
| CN107729031B (en) | Code administration method, apparatus, computer equipment and computer readable storage medium | |
| CN105827683A (en) | Data synchronization method, server and electronic device | |
| CN107896244A (en) | A kind of distribution method of version file, client and server | |
| CN110673907B (en) | Multi-role page jumping method and device, computer equipment and readable storage medium | |
| CN105765533A (en) | Method and apparatus for firmware virtualization | |
| CN105550325A (en) | Data management method and device | |
| CN110806916B (en) | Method and system for realizing personalized login page of each tenant of SAAS platform | |
| CN104011698A (en) | Access to supplemental data based on identifier derived from corresponding primary application data | |
| CN107147711A (en) | Application service update method and device | |
| CN111984239A (en) | Page configuration method, device, server and storage medium | |
| CN105117499A (en) | File display method and device based on cloud disk | |
| CN107360265A (en) | A kind of equipment exclusive identification code acquisition methods, device and mobile terminal | |
| CN110825395B (en) | Multi-plug-in layered deployment system, device and medium | |
| CN106445479B (en) | Information-pushing method and device | |
| US9734449B2 (en) | Method and apparatus for run-time user control of system-generated content | |
| CN116432028A (en) | Method, system, equipment and medium for model training based on training platform | |
| CN106557317A (en) | A kind of method and apparatus that shortcut is created in desktop | |
| CN109582487A (en) | Method and apparatus for sending information | |
| CN104717227A (en) | Method and device for updating login information of login control | |
| US7231448B1 (en) | System and method for automated network element database population | |
| CN113328895A (en) | Error code management system, abnormality positioning method, and storage medium | |
| CN112860311A (en) | Application program issuing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |