CN117149361A - Multi-terminal collaborative optimization system for training model - Google Patents
Multi-terminal collaborative optimization system for training model Download PDFInfo
- Publication number
- CN117149361A CN117149361A CN202311413682.4A CN202311413682A CN117149361A CN 117149361 A CN117149361 A CN 117149361A CN 202311413682 A CN202311413682 A CN 202311413682A CN 117149361 A CN117149361 A CN 117149361A
- Authority
- CN
- China
- Prior art keywords
- optimization
- training
- model
- script
- cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000005457 optimization Methods 0.000 title claims abstract description 237
- 238000012549 training Methods 0.000 title claims abstract description 142
- 238000013515 script Methods 0.000 claims abstract description 91
- 238000007726 management method Methods 0.000 claims description 42
- 238000004422 calculation algorithm Methods 0.000 claims description 34
- 238000000034 method Methods 0.000 claims description 20
- 238000013523 data management Methods 0.000 claims description 18
- 230000009471 action Effects 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 11
- 238000012544 monitoring process Methods 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 3
- 238000012800 visualization Methods 0.000 claims description 3
- 230000010354 integration Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000739 chaotic effect Effects 0.000 description 1
- 230000006854 communication Effects 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 210000001503 joint Anatomy 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
Abstract
The invention provides a training model multi-terminal collaborative optimization system, and relates to the field of computing resource management. According to the invention, the automatic hosting execution is carried out through the optimized script hosting unit, so that the tasks to be handled of each operator can be adjusted at any time; in addition, through the mutual coordination among the model optimization editing unit, the optimization script hosting unit, the multi-terminal personnel collaborative management unit and the optimization model cloud training unit, a plurality of users can work cooperatively without personnel management at any time, and the quality and performance of the model are improved, so that a unified management platform, standardized management and task allocation are provided, and model optimization work can be better performed cooperatively.
Description
Technical Field
The invention relates to the field of computing resource management, in particular to a training model multi-terminal collaborative optimization system.
Background
The cloud computing platform is a cloud platform for providing computing leasing services, so that enterprises with more and more assistance get rid of the constraint of hardware and more investment in solving the actual problems is realized; machine learning is the creation of a model through sample data, training algorithms, and model training, and users train models through cloud computing platforms, through which the intended prediction or decision tasks are accomplished to make predictions or decisions without explicit programming.
In order to obtain a predicted or better-decided training model, model optimization is needed to be carried out on the training model; model optimization needs to be jointly performed by multiple users in some cases (such as a large model formed by multiple models), and the existing processing mode is as follows: the cloud computing platform only provides training model service, different users download training models on the cloud computing platform on the platform and locally optimize the training models, then task butt joint is carried out outside the platform, corresponding optimization tasks are respectively completed, and collaborative integration is carried out, so that multi-terminal collaborative optimization of the models is completed.
However, there are a number of problems with user-by-user task interfacing and multi-terminal collaboration, such as: 1. the lack of a standardized unified management paradigm leads to frequent problems in the collaborative optimization process; 2. different optimization results can be obtained by different optimization task combinations, and when the coordination of the front and back optimization tasks is not uniform, the white work can be caused; 3. the traceability of optimization is poor, and the optimization is not positioned to a specific step when abnormal.
Therefore, there is a need to provide a training model multi-terminal collaborative optimization system to solve the above-mentioned technical problems.
Disclosure of Invention
In order to solve the technical problems, the multi-terminal collaborative optimization system for training the model is deployed on a cloud computing platform and comprises a cloud data management unit, a model optimization editing unit, an optimization script hosting unit, a multi-terminal personnel collaborative management unit and an optimization model cloud training unit;
cloud data management unit: the cloud computing platform is used for managing and calling the training model and the corresponding model data stored by the cloud computing platform; when model optimization is carried out, a user selects an original training model, and the cloud data management unit invokes original model data corresponding to the original training model;
model optimization editing unit: the cloud optimization editing space is used for providing cloud optimization editing space for operators and comprises an optimization content editing container, an editing operation monitor and an associated description editor; the optimized content editing container is used for providing environment support and operation space for optimized projects of the model data; the editing operation monitor is used for monitoring and recording the optimizing operation in the project container and generating a corresponding difference file according to the data change before and after the optimizing operation; the association description editor is used for describing the front-back association relation among all the optimized items set by the user definition and generating corresponding association description items;
an optimization script hosting unit: generating a corresponding script file according to the hosting requirement of the user and performing automatic hosting execution; the script file comprises a container management script, a personnel collaboration script and a training execution script; the container management script is used for managing the creation, start-stop and optimization task allocation of each container; the personnel collaboration script is used for creating, starting, stopping and assigning tasks to be handled by operators; the training execution script is used for creating and executing a training task;
the multi-terminal personnel cooperative management unit: the method comprises the steps of connecting multi-end users for executing model collaborative optimization, feeding back the current operation state of the users, and assigning optimization tasks to be handled of the corresponding users according to personnel collaborative scripts;
and the cloud training unit of the optimization model: and the method is used for executing the training execution script control, loading the corresponding model data and the difference file into cloud hardware resources according to the associated description items, and carrying out cloud training on the optimization model to obtain an optimization training model.
As a further solution, a training model multi-terminal collaborative optimization is performed by:
step 1: the cloud computing platform performs optimization demand detection, and a user initiates a multi-terminal collaborative optimization request of a training model to the cloud computing platform and selects an original training model to be optimized;
step 2: finding out original training data, an original model algorithm and an original training rule corresponding to the original training model through a cloud data management unit;
step 3: the cloud computing platform initiates an optimization requirement query to a user, and the user fills in the optimization requirement according to a question-answer paradigm provided by a WEB interface; the optimization requirements comprise an optimization flow, optimization personnel and an optimization project;
step 4: the cloud computing platform pulls the corresponding user account into the multi-end personnel collaborative management unit according to the optimization personnel, generates the corresponding association description item through the association description editor according to the optimization flow, and newly establishes the corresponding item container in the optimized content editing container according to the optimized item;
step 5: the cloud computing platform initiates a hosting requirement query to a user, and the user fills in the hosting requirement according to a question-answer paradigm provided by a WEB interface; wherein the hosting requirements include hosting flows, hosting personnel, and hosting training;
step 6: the optimized script hosting unit generates a corresponding personnel collaboration script according to hosting personnel, generates a corresponding container management script according to hosting flow, and generates a corresponding training execution script according to hosting training;
step 7: starting an optimization script hosting unit to automatically host and execute, and monitoring and recording the optimization operation in the project container through an editing operation monitor in the process;
step 8: the personnel collaboration script builds a task to be handled to a corresponding operator, the container management script builds an optimized content editing container for an optimized project of the model data, and the operator enters the corresponding project container to perform optimized operation;
step 9: the editing operation monitor monitors and records the optimizing operation in the project container, and generates a corresponding difference file according to the data change before and after the optimizing operation;
step 10: the training execution script judges whether each optimizing item is completed or not through carrying out continuous monitoring on the associated description items; when the cloud training units are all completed, starting the cloud training units of the optimization model and performing the next operation;
step 11: the cloud training unit of the optimization model acquires corresponding difference files, combines the difference files according to the front-back association relation among the optimization items of the association description items, and corrects the original model data to obtain optimization training data;
step 12: and loading the optimized training data into cloud hardware resources, and carrying out cloud training on the optimized model to obtain an optimized training model, thereby completing multi-end collaborative optimization of the one-time training model.
As a still further solution, the model data includes training data, model algorithms and training rules; the optimized content editing container comprises a data optimizing container, an algorithm optimizing container and a rule optimizing container; the optimization items comprise data optimization items, algorithm optimization items and rule optimization items; the difference file alarm data difference file, the algorithm difference file and the rule difference file.
As a further solution, the association description item includes a first association description item and a second association description item; the first association description items are used for describing the front-back association relations among the data optimization items, the algorithm optimization items and the data optimization items and the algorithm optimization items; the second association description items are used for describing the front-back association relations among algorithm optimization items, rule optimization items and algorithm optimization items and rule optimization items.
As a further solution, the script file judges whether to execute the corresponding trigger execution action according to the pre-trigger condition; the front trigger condition of the container management script is an optimized item, and the trigger execution action is to create a corresponding item container; the personnel collaborative script prepositive trigger condition is an optimization project and an optimization personnel, and the trigger execution action is used as an assigned corresponding operator; and the training execution script is an optimization flow, and the triggering execution action is cloud training for creating an optimization model.
As a still further solution, the personnel collaboration script includes automatic assignment and directional assignment when assigning the corresponding operator; wherein, automatic assignment: judging according to the current task to be handled by the operators, and preferentially distributing the operators with the least tasks to be handled to execute the optimization operation; orientation assignment: and specific operators are directionally assigned to execute the optimization operation according to the list of the personnel items used for presetting.
As a further solution, the cloud computing platform also performs cloud visualization on the script execution condition of the optimized script hosting unit, and provides a WEB remote viewing page.
As a further solution, the cloud data management unit trains the model and the corresponding model data to perform version management; and realizing multiple iterative optimization of the training model through the version line.
Compared with the related art, the training model multi-terminal collaborative optimization system provided by the invention has the following beneficial effects:
aiming at the problems of difficult coordination, chaotic management and the like in the training model optimization process, the cloud data management unit is used for managing model data, and the automatic hosting and execution are carried out through the optimizing script hosting unit, so that the tasks to be handled of operators can be adjusted at any time; in addition, through the mutual coordination among the model optimization editing unit, the optimization script hosting unit, the multi-terminal personnel collaborative management unit and the optimization model cloud training unit, a plurality of users can work cooperatively without personnel management at any time, and the quality and performance of the model are improved, so that a unified management platform, standardized management and task allocation are provided, and model optimization work can be better performed cooperatively.
Drawings
FIG. 1 is a schematic diagram of a training model multi-terminal collaborative optimization system according to the present invention;
FIG. 2 is a schematic flow chart of a training model multi-terminal collaborative optimization system according to the present invention.
Detailed Description
The invention will be further described with reference to the drawings and embodiments.
As shown in fig. 1, the training model multi-terminal collaborative optimization system provided in this embodiment is deployed on a cloud computing platform, and includes a cloud data management unit, a model optimization editing unit, an optimization script hosting unit, a multi-terminal personnel collaborative management unit, and an optimization model cloud training unit;
cloud data management unit: the cloud computing platform is used for managing and calling the training model and the corresponding model data stored by the cloud computing platform; when model optimization is carried out, a user selects an original training model, and the cloud data management unit invokes original model data corresponding to the original training model;
model optimization editing unit: the cloud optimization editing space is used for providing cloud optimization editing space for operators and comprises an optimization content editing container, an editing operation monitor and an associated description editor; the optimized content editing container is used for providing environment support and operation space for optimized projects of the model data; the editing operation monitor is used for monitoring and recording the optimizing operation in the project container and generating a corresponding difference file according to the data change before and after the optimizing operation; the association description editor is used for describing the front-back association relation among all the optimized items set by the user definition and generating corresponding association description items;
an optimization script hosting unit: generating a corresponding script file according to the hosting requirement of the user and performing automatic hosting execution; the script file comprises a container management script, a personnel collaboration script and a training execution script; the container management script is used for managing the creation, start-stop and optimization task allocation of each container; the personnel collaboration script is used for creating, starting, stopping and assigning tasks to be handled by operators; the training execution script is used for creating and executing a training task;
the multi-terminal personnel cooperative management unit: the method comprises the steps of connecting multi-end users for executing model collaborative optimization, feeding back the current operation state of the users, and assigning optimization tasks to be handled of the corresponding users according to personnel collaborative scripts;
and the cloud training unit of the optimization model: and the method is used for executing the training execution script control, loading the corresponding model data and the difference file into cloud hardware resources according to the associated description items, and carrying out cloud training on the optimization model to obtain an optimization training model.
It should be noted that: the current processing mode involves the downloading of a model by a user and the optimization at the local, and then the manual collaborative integration, and has the problems of management, standardization, optimization traceability and the like. Thus, this embodiment proposes a system comprising the following main components: the system comprises a cloud data management unit, a model optimization editing unit, an optimization script hosting unit, a multi-terminal personnel collaborative management unit and an optimization model cloud training unit. The core functions of the system comprise model data management, optimization editing, script hosting, collaborative management and cloud training. Through the system, a user can more effectively perform model optimization work, and the cooperativity and the model quality are improved.
As a still further solution, as shown in fig. 2, a training model multi-terminal collaborative optimization is performed by:
step 1: the cloud computing platform performs optimization demand detection, and a user initiates a multi-terminal collaborative optimization request of a training model to the cloud computing platform and selects an original training model to be optimized;
step 2: finding out original training data, an original model algorithm and an original training rule corresponding to the original training model through a cloud data management unit;
step 3: the cloud computing platform initiates an optimization requirement query to a user, and the user fills in the optimization requirement according to a question-answer paradigm provided by a WEB interface; the optimization requirements comprise an optimization flow, optimization personnel and an optimization project;
step 4: the cloud computing platform pulls the corresponding user account into the multi-end personnel collaborative management unit according to the optimization personnel, generates the corresponding association description item through the association description editor according to the optimization flow, and newly establishes the corresponding item container in the optimized content editing container according to the optimized item;
step 5: the cloud computing platform initiates a hosting requirement query to a user, and the user fills in the hosting requirement according to a question-answer paradigm provided by a WEB interface; wherein the hosting requirements include hosting flows, hosting personnel, and hosting training;
step 6: the optimized script hosting unit generates a corresponding personnel collaboration script according to hosting personnel, generates a corresponding container management script according to hosting flow, and generates a corresponding training execution script according to hosting training;
step 7: starting an optimization script hosting unit to automatically host and execute, and monitoring and recording the optimization operation in the project container through an editing operation monitor in the process;
step 8: the personnel collaboration script builds a task to be handled to a corresponding operator, the container management script builds an optimized content editing container for an optimized project of the model data, and the operator enters the corresponding project container to perform optimized operation;
step 9: the editing operation monitor monitors and records the optimizing operation in the project container, and generates a corresponding difference file according to the data change before and after the optimizing operation;
step 10: the training execution script judges whether each optimizing item is completed or not through carrying out continuous monitoring on the associated description items; when the cloud training units are all completed, starting the cloud training units of the optimization model and performing the next operation;
step 11: the cloud training unit of the optimization model acquires corresponding difference files, combines the difference files according to the front-back association relation among the optimization items of the association description items, and corrects the original model data to obtain optimization training data;
step 12: and loading the optimized training data into cloud hardware resources, and carrying out cloud training on the optimized model to obtain an optimized training model, thereby completing multi-end collaborative optimization of the one-time training model.
It should be noted that: according to the embodiment, the optimization demand inquiry is initiated to the user so as to acquire the optimization flow, the optimization personnel and the optimization project, then the hosting demand inquiry is initiated to the user so as to know hosting demands such as the hosting flow, the hosting personnel and the hosting training, and corresponding script files are generated according to the optimization demands and the hosting demands, so that script hosting management is realized, the multi-terminal optimization process of the whole-process supervision coordination training model is realized, the cloud training unit of the coordination optimization model starts the optimization model training after the optimization training data is obtained, the resource occupation time and the occupation amount are reduced, and the hardware resources are fully utilized.
As a still further solution, the model data includes training data, model algorithms and training rules; the optimized content editing container comprises a data optimizing container, an algorithm optimizing container and a rule optimizing container; the optimization items comprise data optimization items, algorithm optimization items and rule optimization items; the difference file alarm data difference file, the algorithm difference file and the rule difference file.
It should be noted that: the implementation is described by optimizing a difference file, and has the following advantages: the original model data can be converted into the optimized model data through describing differences and recording steps without managing multiple model data, so that the process can be restored and traced, difficult positioning and restoration caused by problems are avoided, the script is convenient for process management and control, the combination relation among all optimization strategies can be flexibly adjusted, and different optimization results obtained by different optimization task combinations can be orderly managed.
As a further solution, the association description item includes a first association description item and a second association description item; the first association description items are used for describing the front-back association relations among the data optimization items, the algorithm optimization items and the data optimization items and the algorithm optimization items; the second association description items are used for describing the front-back association relations among algorithm optimization items, rule optimization items and algorithm optimization items and rule optimization items.
It should be noted that: as shown in fig. 1, in a specific embodiment, two optimization training models of different optimization lines are needed; a first strip: [ data optimization (data optimization term 1, data optimization term 2), algorithm optimization (algorithm optimization term 1, algorithm optimization term 2), rule optimization (rule optimization term 1) ]; and a second strip: [ data optimization (data optimization term 1, data optimization term 2), algorithm optimization (algorithm optimization term 1, algorithm optimization term 2), rule optimization (rule optimization term 2) ]
The traditional method needs to reconstruct the data content into two parts of model data, and the data multiplexing cannot be performed; according to the method, only a container is needed to be newly built for each optimization project, and difference files are obtained, on the basis, description of different optimization flows is achieved through the first association description item and the second association description item, therefore, only the difference of the last rule optimization item is needed to be described, data expression of the optimization training models of two different optimization lines can be achieved, and due to the fact that decomposition steps are carried out in the whole process, corresponding personnel can be distributed through a script hosting unit to operate, automatic management is facilitated, and each optimization project can be located and traced.
As a further solution, the script file judges whether to execute the corresponding trigger execution action according to the pre-trigger condition; the front trigger condition of the container management script is an optimized item, and the trigger execution action is to create a corresponding item container; the personnel collaborative script prepositive trigger condition is an optimization project and an optimization personnel, and the trigger execution action is used as an assigned corresponding operator; and the training execution script is an optimization flow, and the triggering execution action is cloud training for creating an optimization model.
It should be noted that: according to the embodiment, the complicated management process is automatically managed, and a user can automatically allocate tasks and arrange training plans in the whole process only by constructing the whole requirement when optimizing projects, so that the complicated communication process is avoided.
As a still further solution, the personnel collaboration script includes automatic assignment and directional assignment when assigning the corresponding operator; wherein, automatic assignment: judging according to the current task to be handled by the operators, and preferentially distributing the operators with the least tasks to be handled to execute the optimization operation; orientation assignment: and specific operators are directionally assigned to execute the optimization operation according to the list of the personnel items used for presetting.
As a further solution, the cloud computing platform also performs cloud visualization on the script execution condition of the optimized script hosting unit, and provides a WEB remote viewing page.
As a further solution, the cloud data management unit trains the model and the corresponding model data to perform version management; and realizing multiple iterative optimization of the training model through the version line.
It should be noted that: the embodiment can continuously circulate iteration, thereby realizing the generation of training models of different versions and the orderly management of data, and carrying out problem backtracking and reduction.
The foregoing is only illustrative of the present invention and is not to be construed as limiting the scope of the invention, and all equivalent structures or equivalent flow modifications which may be made by the teachings of the present invention and the accompanying drawings or which may be directly or indirectly employed in other related art are within the scope of the invention.
Claims (8)
1. The multi-terminal collaborative optimization system for the training model is deployed on a cloud computing platform and is characterized by comprising a cloud data management unit, a model optimization editing unit, an optimization script hosting unit, a multi-terminal personnel collaborative management unit and an optimization model cloud training unit;
cloud data management unit: the cloud computing platform is used for managing and calling the training model and the corresponding model data stored by the cloud computing platform; when model optimization is carried out, a user selects an original training model, and the cloud data management unit invokes original model data corresponding to the original training model;
model optimization editing unit: the cloud optimization editing space is used for providing cloud optimization editing space for operators and comprises an optimization content editing container, an editing operation monitor and an associated description editor; the optimized content editing container is used for providing environment support and operation space for optimized projects of the model data; the editing operation monitor is used for monitoring and recording the optimizing operation in the project container and generating a corresponding difference file according to the data change before and after the optimizing operation; the association description editor is used for describing the front-back association relation among all the optimized items set by the user definition and generating corresponding association description items;
an optimization script hosting unit: generating a corresponding script file according to the hosting requirement of the user and performing automatic hosting execution; the script file comprises a container management script, a personnel collaboration script and a training execution script; the container management script is used for managing the creation, start-stop and optimization task allocation of each container; the personnel collaboration script is used for creating, starting, stopping and assigning tasks to be handled by operators; the training execution script is used for creating and executing a training task;
the multi-terminal personnel cooperative management unit: the method comprises the steps of connecting multi-end users for executing model collaborative optimization, feeding back the current operation state of the users, and assigning optimization tasks to be handled of the corresponding users according to personnel collaborative scripts;
and the cloud training unit of the optimization model: and the method is used for executing the training execution script control, loading the corresponding model data and the difference file into cloud hardware resources according to the associated description items, and carrying out cloud training on the optimization model to obtain an optimization training model.
2. The training model multi-terminal collaborative optimization system according to claim 1, wherein the training model multi-terminal collaborative optimization is performed by:
step 1: the cloud computing platform performs optimization demand detection, and a user initiates a multi-terminal collaborative optimization request of a training model to the cloud computing platform and selects an original training model to be optimized;
step 2: finding out original training data, an original model algorithm and an original training rule corresponding to the original training model through a cloud data management unit;
step 3: the cloud computing platform initiates an optimization requirement query to a user, and the user fills in the optimization requirement according to a question-answer paradigm provided by a WEB interface; the optimization requirements comprise an optimization flow, optimization personnel and an optimization project;
step 4: the cloud computing platform pulls the corresponding user account into the multi-end personnel collaborative management unit according to the optimization personnel, generates the corresponding association description item through the association description editor according to the optimization flow, and newly establishes the corresponding item container in the optimized content editing container according to the optimized item;
step 5: the cloud computing platform initiates a hosting requirement query to a user, and the user fills in the hosting requirement according to a question-answer paradigm provided by a WEB interface; wherein the hosting requirements include hosting flows, hosting personnel, and hosting training;
step 6: the optimized script hosting unit generates a corresponding personnel collaboration script according to hosting personnel, generates a corresponding container management script according to hosting flow, and generates a corresponding training execution script according to hosting training;
step 7: starting an optimization script hosting unit to automatically host and execute, and monitoring and recording the optimization operation in the project container through an editing operation monitor in the process;
step 8: the personnel collaboration script builds a task to be handled to a corresponding operator, the container management script builds an optimized content editing container for an optimized project of the model data, and the operator enters the corresponding project container to perform optimized operation;
step 9: the editing operation monitor monitors and records the optimizing operation in the project container, and generates a corresponding difference file according to the data change before and after the optimizing operation;
step 10: the training execution script judges whether each optimizing item is completed or not through carrying out continuous monitoring on the associated description items; when the cloud training units are all completed, starting the cloud training units of the optimization model and performing the next operation;
step 11: the cloud training unit of the optimization model acquires corresponding difference files, combines the difference files according to the front-back association relation among the optimization items of the association description items, and corrects the original model data to obtain optimization training data;
step 12: and loading the optimized training data into cloud hardware resources, and carrying out cloud training on the optimized model to obtain an optimized training model, thereby completing multi-end collaborative optimization of the one-time training model.
3. The training model multi-terminal collaborative optimization system of claim 1, wherein the model data includes training data, model algorithms, and training rules; the optimized content editing container comprises a data optimizing container, an algorithm optimizing container and a rule optimizing container; the optimization items comprise data optimization items, algorithm optimization items and rule optimization items; the difference file alarm data difference file, the algorithm difference file and the rule difference file.
4. A training model multi-terminal collaborative optimization system according to claim 3, wherein the association description includes a first association description and a second association description; the first association description items are used for describing the front-back association relations among the data optimization items, the algorithm optimization items and the data optimization items and the algorithm optimization items; the second association description items are used for describing the front-back association relations among algorithm optimization items, rule optimization items and algorithm optimization items and rule optimization items.
5. The training model multi-terminal collaborative optimization system according to claim 1, wherein the script file determines whether to execute a corresponding trigger execution action according to a pre-trigger condition; the front trigger condition of the container management script is an optimized item, and the trigger execution action is to create a corresponding item container; the personnel collaborative script prepositive trigger condition is an optimization project and an optimization personnel, and the trigger execution action is used as an assigned corresponding operator; and the training execution script is an optimization flow, and the triggering execution action is cloud training for creating an optimization model.
6. The training model multi-terminal collaborative optimization system according to claim 5, wherein the personnel collaboration script, when assigning corresponding operators, includes automatic assignment and directional assignment; wherein, automatic assignment: judging according to the current task to be handled by the operators, and preferentially distributing the operators with the least tasks to be handled to execute the optimization operation; orientation assignment: and specific operators are directionally assigned to execute the optimization operation according to the list of the personnel items used for presetting.
7. The training model multi-terminal collaborative optimization system according to claim 1, wherein the cloud computing platform further performs cloud visualization of script execution conditions of the optimization script hosting unit and provides a WEB remote viewing page.
8. The training model multi-terminal collaborative optimization system according to claim 1, wherein the cloud data management unit performs version management on the training model and corresponding model data; and realizing multiple iterative optimization of the training model through the version line.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311413682.4A CN117149361B (en) | 2023-10-30 | 2023-10-30 | Multi-terminal collaborative optimization system for training model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311413682.4A CN117149361B (en) | 2023-10-30 | 2023-10-30 | Multi-terminal collaborative optimization system for training model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117149361A true CN117149361A (en) | 2023-12-01 |
| CN117149361B CN117149361B (en) | 2023-12-29 |
Family
ID=88904660
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311413682.4A Active CN117149361B (en) | 2023-10-30 | 2023-10-30 | Multi-terminal collaborative optimization system for training model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117149361B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685139A (en) * | 2021-01-11 | 2021-04-20 | 东北大学 | K8S and Kubeedge-based cloud edge deep learning model management system and model training method |
| CN113067873A (en) * | 2021-03-19 | 2021-07-02 | 北京邮电大学 | Edge cloud collaborative optimization method based on deep reinforcement learning |
| CN114359781A (en) * | 2021-12-02 | 2022-04-15 | 国家石油天然气管网集团有限公司 | Intelligent recognition system for cloud-side collaborative autonomous learning |
| US20220309393A1 (en) * | 2019-07-16 | 2022-09-29 | Northeastern University | Cloud-edge collaboration forecasting system and method for aluminum oxide production indexes |
| WO2022227105A1 (en) * | 2021-04-30 | 2022-11-03 | 广州中国科学院软件应用技术研究所 | Cooperative control system and method based on device-cloud fusion |
-
2023
- 2023-10-30 CN CN202311413682.4A patent/CN117149361B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220309393A1 (en) * | 2019-07-16 | 2022-09-29 | Northeastern University | Cloud-edge collaboration forecasting system and method for aluminum oxide production indexes |
| CN112685139A (en) * | 2021-01-11 | 2021-04-20 | 东北大学 | K8S and Kubeedge-based cloud edge deep learning model management system and model training method |
| CN113067873A (en) * | 2021-03-19 | 2021-07-02 | 北京邮电大学 | Edge cloud collaborative optimization method based on deep reinforcement learning |
| WO2022227105A1 (en) * | 2021-04-30 | 2022-11-03 | 广州中国科学院软件应用技术研究所 | Cooperative control system and method based on device-cloud fusion |
| CN114359781A (en) * | 2021-12-02 | 2022-04-15 | 国家石油天然气管网集团有限公司 | Intelligent recognition system for cloud-side collaborative autonomous learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117149361B (en) | 2023-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109118347B (en) | Automatic cooperation method and system | |
| US8301935B2 (en) | Distributed batch runner | |
| CN106844198B (en) | Distributed dispatching automation test platform and method | |
| CN107992392B (en) | Automatic monitoring and repairing system and method for cloud rendering system | |
| CN109032945B (en) | Software reliability engineering integrated environment framework design method | |
| CN112882813A (en) | Task scheduling method, device and system and electronic equipment | |
| CN110740053A (en) | Service arranging method and device | |
| CN107508869B (en) | Tracing data acquisition method and client | |
| CN111526049B (en) | Operation and maintenance system, operation and maintenance method, electronic device and storage medium | |
| CN112698952A (en) | Unified management method and device for computing resources, computer equipment and storage medium | |
| US20240058955A1 (en) | System using natural conversation for monitoring a facility | |
| CN115687486A (en) | Method and device for lightweight data acquisition based on button | |
| CN116414521A (en) | Virtual machine remote command interaction method, device and medium for distributed heterogeneous shooting ranges | |
| CN112686580B (en) | Workflow definition method and system capable of customizing flow | |
| CN117149361B (en) | Multi-terminal collaborative optimization system for training model | |
| CN111667250A (en) | Cloud-based management method and system for automatic test execution machine | |
| CN116149701A (en) | Online software upgrading method and system for edge terminal | |
| CN103164776B (en) | A kind of medicine traceability system and medicine retroactive method thereof and flow monitoring method | |
| CN115695472A (en) | Server fault response processing method and device, computer equipment and storage medium | |
| CN103914339A (en) | Server management system and server management method | |
| CN113034048A (en) | Task processing method, device, server and storage medium | |
| CN113220480A (en) | Distributed data task cross-cloud scheduling system and method | |
| CN111399971A (en) | Network element state analyzing method, device and storage medium | |
| CN117196539B (en) | Automatic checking method, system, equipment and medium for security base line | |
| CN114510282B (en) | Method, device, equipment and storage medium for running automation application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |