CN111190690A - Intelligent training device based on container arrangement tool - Google Patents
Intelligent training device based on container arrangement tool Download PDFInfo
- Publication number
- CN111190690A CN111190690A CN201911357094.7A CN201911357094A CN111190690A CN 111190690 A CN111190690 A CN 111190690A CN 201911357094 A CN201911357094 A CN 201911357094A CN 111190690 A CN111190690 A CN 111190690A
- Authority
- CN
- China
- Prior art keywords
- module
- training
- container arrangement
- arrangement tool
- parameters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
Abstract
The application discloses intelligence trainer based on container arrangement tool, the device includes: integrating an API interaction module, a data processing module and a mirror image library module on the basis of a container arrangement tool module; and the user inputs training parameters through the API interaction module, and after the training parameters are converted by the container arrangement tool module and analyzed and optimized by the data processing module, the container arrangement tool module pulls the application mirror image of the mirror image library module and starts the container for training. It is an object of the present application to enable deep learning to be managed using a container orchestration tool to avoid careless mistakes caused by human operations and to provide a user with an easily understandable and acceptable visual graphical interface.
Description
Technical Field
The application relates to the technical field of intelligent training based on container (a container is a lightweight independent executable software package of software and contains all needed for running the software, namely code, running time, system tools, system libraries and settings), and particularly relates to an intelligent training device based on a container arrangement tool.
Background
The trend of deep learning research continues to rise, and various open source deep learning frameworks are also developed, including TensorFlow, cafe (a common deep learning framework mainly applied to video and image processing), and so on. In the face of challenges of deep learning computing power, software and hardware have the same key, and it cannot provide usability and expansibility by means of hardware alone, taking Caffe as an example, it can be used for face recognition, picture classification, position detection, target tracking, etc., but because it has insufficient support for time series RNN (recurrent neural Network), LSTM (Long Short-Term Memory Network), etc., it is troublesome to define RNN structure, when the model structure is very complicated, it may need to write very lengthy configuration files to design a Network, and it is also laborious to read, and the model file generated along with its training is not only lengthy, but also key fields need to be screened, increasing overhead intangibly.
The problems existing in the prior art are as follows:
at present, deep learning tasks, resource allocation and task scheduling need to depend on personal experience, and when the resources are in short or the task scheduling is unreasonable, not only is insufficient resource utilization easily caused, but also training progress is blocked.
The process of forming the deep training needs a long time from the conversion of a data set and the compiling of a configuration file to the training without the participation of a platform, and the result data needs to be extracted manually, so carelessness is easy to cause.
Disclosure of Invention
In view of the foregoing problems in the related art, the present application provides an intelligent training apparatus based on a container arrangement tool, which at least can utilize the container arrangement tool to manage deep learning, thereby avoiding careless mistakes caused by manual operations and providing a visual graphical interface that is easy to understand and accept for a user.
The technical scheme of the application is realized as follows:
the utility model provides an intelligence trainer based on container arrangement tool, includes:
on the basis of a container arrangement tool module, an Application Programming Interface (API) interaction module, a data processing module and a mirror image (the mirror image is a file storage format, the mirror image file is similar to a ZIP (ZIP) compression package in nature, and a specific series of files are manufactured into a single file according to a certain format so as to be convenient for a user to download and use) library module are integrated;
and the user inputs training parameters through the API interaction module, and after the training parameters are converted by the container arrangement tool module and analyzed and optimized by the data processing module, the container arrangement tool module pulls the application mirror image of the mirror image library module and starts the container for training.
According to the embodiment of the application, the method further comprises the following steps:
the user only needs to input the training parameters through the API interactive module and obtains the training result displayed by the graphical interface through the API interactive module, and the training process is completed by the data processing module, the container arrangement tool module and the mirror image library module.
According to an embodiment of the application, the training parameters comprise at least:
selecting an original data set;
selecting a network structure;
entering solving parameters;
resource types and training samples are selected.
According to an embodiment of the application, the container arrangement tool module comprises at least:
the data set conversion module is used for converting the data set;
the optimization processing module is used for optimizing the training parameters and the recommendation model and configuring the parameters, and the parameters at least comprise: display and snapshot;
and the application mirror image pulling module is used for pulling the application mirror image of the mirror image library module.
According to an embodiment of the application, the data processing module comprises at least:
the optimization processing module is used for analyzing and optimizing the training parameters and the training results;
and the configuration file generation module is used for generating a configuration file for training according to the training parameters and the conversion data set.
According to an embodiment of the application, the configuration file comprises a solution file and a network file, wherein:
solving the file, at least comprising the parameters:
base lr、lr policy、weight decay、momentum；
a network file comprising at least:
the device comprises a pooling layer, a data layer, an activation layer and an accuracy layer.
The beneficial technical effect of this application lies in:
the platform saves a large amount of time in the process of deep learning of a user through an abstract training process, guidance between modules after abstraction, automatic construction of configuration files and the like;
the platform analyzes data generated by deep learning training and maps the data into a graphical interface, so that a user can easily understand and receive the data, and careless mistakes caused by negligence in manual operation are avoided.
Drawings
In order to more clearly illustrate the embodiments of the present application, the drawings needed for the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
FIG. 1 is a schematic diagram of an intelligent training apparatus based on a container orchestration tool according to an embodiment of the present application;
FIG. 2 is a flow chart of an intelligent training apparatus based on a container arrangement tool according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments that can be derived from the embodiments given herein by a person of ordinary skill in the art are intended to be within the scope of the present disclosure.
According to an embodiment of the application, an intelligent training device based on a container arrangement tool is provided. FIG. 1 shows a schematic diagram of an intelligent training apparatus based on a container arrangement tool according to an embodiment of the present application. Referring to fig. 1, the intelligent training device based on the container arrangement tool of the present invention comprises: on the basis of the container arrangement tool module, integrating a mirror image library module, a data processing module and an API interaction module; and the user inputs training parameters through the API interaction module, and after the training parameters are converted by the container arrangement tool module and analyzed and optimized by the data processing module, the container arrangement tool module pulls the application mirror image of the mirror image library module and starts the container for training.
According to the technical scheme, a deep learning software platform is built, deep learning framework applications and dependency packages are packaged into a portable container, a container arranging tool K8s is used for management, the processes of data set conversion, model building, training and the like related to deep learning are integrated, a visual interface is provided for a user, the user is concentrated on data analysis and result collection in the training process, algorithm research and development thresholds are reduced by using the usability, a data scientist and an algorithm researcher can describe and realize deep learning service requirements without knowing bottom level details, and the flexible and changeable requirements of upper developers are met in an abstract and easily understood mode.
On the basis of a container arrangement tool (the container arrangement tool is a Kubernetes tool, is a Google open-source container cluster management system, provides functions of application deployment, maintenance, extension mechanism and the like, and can conveniently manage cross-machine operation containerization application by utilizing the Kubernetes, wherein the main implementation language is Go language, and the Kubernetes is abbreviated as K8s), a mirror image library, a data processing platform, an interaction platform and the like are integrated to form modularization and orientation of a training process; the analysis result is visualized; the process is linked with the intelligent training platform of deep learning of integration.
As shown in fig. 1, the implementation architecture of the present invention is divided into four parts:
API interaction layer as shown at 11 in FIG. 1: and an interface for user input and data export is provided, the layer is responsible for presenting the division of the training modules in the platform to the user, handing over the control right of resource domination and parameter specification to the user, and displaying the result generated in the training in a graphical interface in real time.
Data processing layers as shown at 12 in fig. 1: the system is responsible for standardizing and integrating the entries of various deep learning tasks; and the results of the training in the container are analyzed and processed.
The container orchestration tool, as shown at 13 in fig. 1: uniformly planning and distributing resources according to a scheduling strategy; mapping the container port and the node port to provide external services; and pulling the mirror image of the mirror library according to different-depth learning applications selected by the user, and instantiating the mirror image into a container to perform related operations.
Mirror libraries as shown at 14 in FIG. 1: and providing a mirror image of a deep learning framework for a container editing tool to pull and instantiate a container.
As shown in fig. 2, a flow chart and related implementations of the intelligent training apparatus based on the container arrangement tool are described below:
based on an interactive interface and a Kubernets container arrangement tool provided by the platform, taking Caffe training task as an example, after a user finishes the operations of selecting a data set, selecting a network file, inputting solving parameters, selecting resources and training samples, the Caffe training task can be finished once. The processes of data set conversion, configuration file generation, resource scheduling, result file analysis and the like are not perceived by a user:
preprocessing a data set: compared with the converted data (lmdb, leveldb format), the original image has a large difference in efficiency of reading data, and a user can call the K8s starting container shown at 202 to perform conversion shown at 203 by merely typing in the relevant information of the data set and the index file thereof through an interface provided by the platform shown at 201, so as to generate a file format for direct processing by Caffe, that is, a converted data set shown at 204.
Generating a configuration file: the configuration file consists of two parts, a solution file shown at 208 and a network file shown at 207. Wherein, the 208 solved files contain base lr, lr policy, weight decay, momentum and other parameters, and the 207 network files contain pooling layer, data layer, activation layer, accuracy layer and the like. Through an interface provided by the platform, a user only needs to key in the solving parameters shown at the 206 position, selects the recommended network structure template shown at the 205 position (or performs customization on the basis of the recommended network structure template), the platform can generate the network file shown at the 207 position according to the selected network structure 205 matched with the converted data set shown at the 204 position, and the solving file shown at the 208 position is regenerated according to the solving parameters and the network file which are input by the user at the 206 position for use in subsequent training tasks. The process of file generation is also transparent to the user.
Submitting a training model: the resource types and quantity and training model samples shown at 209 are selected, the platform can reasonably configure parameters such as display, snapshot and the like according to the model selected by the user, and pull the corresponding application mirror image to the mirror image warehouse, and start a container for training as shown at 210 by K8 s.
And (3) displaying a training result: throughout the training process and after completion, the platform parses the generated training logs, data, as shown at 211, and presents them in a graphical interface as shown at 212.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (6)
1. An intelligent training device based on a container arrangement tool, comprising:
integrating an API interaction module, a data processing module and a mirror image library module on the basis of a container arrangement tool module;
and a user inputs training parameters through the API interaction module, and after the training parameters are converted by the container arrangement tool module and analyzed and optimized by the data processing module, the container arrangement tool module pulls the application mirror image of the mirror image library module and starts the container for training.
2. The intelligent training device based on a container arrangement tool according to claim 1, further comprising:
the user only needs to input the training parameters through the API interactive module and obtains the training result displayed by the graphical interface through the API interactive module, and the training process is completed by the data processing module, the container arrangement tool module and the mirror image library module.
3. Intelligent training device based on a container arrangement tool according to claim 1 or 2, characterized in that said training parameters comprise at least:
selecting an original data set;
selecting a network structure;
entering solving parameters;
resource types and training samples are selected.
4. The intelligent training device based on the container arrangement tool as claimed in claim 1 or 2, wherein the container arrangement tool module at least comprises:
the data set conversion module is used for converting the data set;
an optimization processing module, configured to optimize the training parameters and the recommendation model and configure parameters, where the parameters at least include: display and snapshot;
and the application mirror image pulling module is used for pulling the application mirror image of the mirror image library module.
5. The intelligent training device based on the container arrangement tool as claimed in claim 1 or 2, wherein the data processing module at least comprises:
the optimization processing module is used for analyzing and optimizing the training parameters and the training results;
and the configuration file generation module is used for generating a configuration file for the training according to the training parameters and the conversion data set.
6. The intelligent training apparatus based on container arrangement tool according to claim 5, wherein the configuration file comprises a solution file and a network file, wherein:
the solution file at least comprises the parameters:
base lr、lr policy、weight decay、momentum；
the network file at least comprises:
the device comprises a pooling layer, a data layer, an activation layer and an accuracy layer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911357094.7A CN111190690A (en) | 2019-12-25 | 2019-12-25 | Intelligent training device based on container arrangement tool |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911357094.7A CN111190690A (en) | 2019-12-25 | 2019-12-25 | Intelligent training device based on container arrangement tool |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111190690A true CN111190690A (en) | 2020-05-22 |
Family
ID=70709372
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911357094.7A Pending CN111190690A (en) | 2019-12-25 | 2019-12-25 | Intelligent training device based on container arrangement tool |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111190690A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148269A (en) * | 2020-10-22 | 2020-12-29 | 深圳市思迅软件股份有限公司 | Research and development continuous integration method and device, computer equipment and storage medium |
| CN112295617A (en) * | 2020-09-18 | 2021-02-02 | 济南大学 | Intelligent beaker based on experimental scene situation perception |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108881446A (en) * | 2018-06-22 | 2018-11-23 | 深源恒际科技有限公司 | A kind of artificial intelligence plateform system based on deep learning |
| CN109272116A (en) * | 2018-09-05 | 2019-01-25 | 郑州云海信息技术有限公司 | A kind of method and device of deep learning |
| CN109447248A (en) * | 2018-11-09 | 2019-03-08 | 深圳增强现实技术有限公司 | Deep learning platform and method |
| CN109508238A (en) * | 2019-01-05 | 2019-03-22 | 咪付（广西）网络技术有限公司 | A kind of resource management system and method for deep learning |
| US20190180145A1 (en) * | 2016-08-03 | 2019-06-13 | Infervision | Method and device for performing transformation-based learning on medical image |
-
2019
- 2019-12-25 CN CN201911357094.7A patent/CN111190690A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190180145A1 (en) * | 2016-08-03 | 2019-06-13 | Infervision | Method and device for performing transformation-based learning on medical image |
| CN108881446A (en) * | 2018-06-22 | 2018-11-23 | 深源恒际科技有限公司 | A kind of artificial intelligence plateform system based on deep learning |
| CN109272116A (en) * | 2018-09-05 | 2019-01-25 | 郑州云海信息技术有限公司 | A kind of method and device of deep learning |
| CN109447248A (en) * | 2018-11-09 | 2019-03-08 | 深圳增强现实技术有限公司 | Deep learning platform and method |
| CN109508238A (en) * | 2019-01-05 | 2019-03-22 | 咪付（广西）网络技术有限公司 | A kind of resource management system and method for deep learning |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112295617A (en) * | 2020-09-18 | 2021-02-02 | 济南大学 | Intelligent beaker based on experimental scene situation perception |
| CN112295617B (en) * | 2020-09-18 | 2022-04-01 | 济南大学 | Intelligent beaker based on experimental scene situation perception |
| CN112148269A (en) * | 2020-10-22 | 2020-12-29 | 深圳市思迅软件股份有限公司 | Research and development continuous integration method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Warden et al. | Tinyml: Machine learning with tensorflow lite on arduino and ultra-low-power microcontrollers | |
| US11205143B2 (en) | Utilizing a machine learning model and natural language processing to manage and allocate tasks | |
| Chen et al. | {TVM}: An automated {End-to-End} optimizing compiler for deep learning | |
| US11074107B1 (en) | Data processing system and method for managing AI solutions development lifecycle | |
| Rasure et al. | The khoros application development environment | |
| Limbourg | Multi-path development of user interfaces | |
| CN109155025A (en) | Across multi-platform autotask flow management | |
| CN110674295A (en) | Data labeling system based on deep learning | |
| CN111611240A (en) | Method, apparatus and device for executing automatic machine learning process | |
| US11861469B2 (en) | Code generation for Auto-AI | |
| CN115080021A (en) | Zero code modeling method and system based on automatic machine learning | |
| CN111190690A (en) | Intelligent training device based on container arrangement tool | |
| WO2023138437A1 (en) | Application development platform, micro-program generation method, and device and storage medium | |
| Rahman et al. | An assistive model for visually impaired people using YOLO and MTCNN | |
| CN115794341A (en) | Task scheduling method, device, equipment and storage medium based on artificial intelligence | |
| CN104657141B (en) | A kind of gui software wrapper and its implementation based on computer vision | |
| Michael et al. | The HBMS story | |
| EP3842940A1 (en) | Method and system for uniformly performing feature extraction | |
| CN112148926A (en) | Graph data stream processing method, processing device and storage medium | |
| CN116075899A (en) | Generation of organic synthetic programs from simplified molecular input linear entry system reactions | |
| Clarke et al. | An intelligent front-end for computer-aided building design | |
| CN103678383B (en) | A kind of data processing method and electronic equipment | |
| CN112307177A (en) | Generating a process flow model using an unstructured conversational robot | |
| Sophocleous et al. | AI-Driven intent-based networking for 5G enhanced robot autonomy | |
| Rafner et al. | Towards hybrid intelligence workflows: integrating interface design and scalable deployment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |