CN112506673B - Intelligent edge calculation-oriented collaborative model training task configuration method - Google Patents

Intelligent edge calculation-oriented collaborative model training task configuration method Download PDF

Info

Publication number
CN112506673B
CN112506673B CN202110153068.3A CN202110153068A CN112506673B CN 112506673 B CN112506673 B CN 112506673B CN 202110153068 A CN202110153068 A CN 202110153068A CN 112506673 B CN112506673 B CN 112506673B
Authority
CN
China
Prior art keywords
training
edge
time slot
global
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110153068.3A
Other languages
Chinese (zh)
Other versions
CN112506673A (en
Inventor
邹昊东
张明明
俞俊
陈海洋
夏飞
王鹏飞
范磊
陶晔波
许明杰
王琳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nari Technology Co Ltd
Information and Telecommunication Branch of State Grid Jiangsu Electric Power Co Ltd
Original Assignee
Nari Technology Co Ltd
Information and Telecommunication Branch of State Grid Jiangsu Electric Power Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nari Technology Co Ltd, Information and Telecommunication Branch of State Grid Jiangsu Electric Power Co Ltd filed Critical Nari Technology Co Ltd
Priority to CN202110153068.3A priority Critical patent/CN112506673B/en
Publication of CN112506673A publication Critical patent/CN112506673A/en
Application granted granted Critical
Publication of CN112506673B publication Critical patent/CN112506673B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5072—Grid computing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/50—Indexing scheme relating to G06F9/50
    • G06F2209/502—Proximity

Abstract

The invention discloses a collaborative model training task configuration method facing intelligent edge computing, which is used for edge computing nodes and comprises one or more training time slots, wherein each training time slot comprises the following steps: sending a model training request to one or more mobile devices; receiving an availability status and a user data size of a current time slot reported from one or more mobile devices; determining the number of training small wheels required by the training of the mobile equipment and the interaction model participating in the training based on the previously obtained task configuration result and the current available state of each mobile equipment; and performing interactive model training with the mobile equipment participating in training until the number of the determined training small rounds is reached, constructing and solving an optimization problem aiming at minimizing the use of edge training resources according to the training effect and the user data scale reported by each mobile equipment, and obtaining a new task configuration result. Compared with other methods, the method has the advantages of much less training resource consumption and little difference in precision.

Description

Intelligent edge calculation-oriented collaborative model training task configuration method
Technical Field
The invention relates to a collaborative model training task configuration method, in particular to a collaborative model training task configuration method for intelligent edge computing.
Background
During the process of using a mobile device, such as a mobile phone and a tablet computer, a large amount of user data is generated, including browsing records, typing records, various log information, and the like. After being analyzed and processed, the data can help the service provider to perform better service deployment and provision. Such analytical processing approaches often rely on machine learning models. Specifically, a machine learning model includes a model structure and model parameters, and the accuracy of the machine learning model on a specific data set, such as a classification model for classifying a data set, and the obtained correct classification ratio is used as the model accuracy. The goal of the service provider is then to train a particular machine learning model with the user data generated in a distributed manner for each user in order to achieve the best model accuracy. In this way, service providers can leverage these machine learning models to provide better inferential class services for outsourcing. For example, when a user browses commodities, commodity recommendation is carried out on the user in combination with commodity classification; when a user types, hot word recommendation is carried out in combination with context; or during navigation, a more accurate navigation model is utilized for making a route.
Although all the user data are gathered to the data center for processing, the machine learning model can be obtained according to the training mode. However, in a marginal environment, such raw data aggregation is prohibited. The reason is that: 1) due to privacy protection, users are often reluctant to upload their original data. 2) The service provider often leases the edge device of the operator for computation and transmission. Transferring user data on each mobile device to a data center can result in costly cross-domain transfers. The cross-domain here contains two meanings at the same time: cross-regional transmission and cross-operator to data center transmission.
Due to different habits of users in using mobile devices in marginal scenes, namely different time and frequency of using the devices and different scale and content of data generated in the using process, uncertainty exists in the process of performing distributed machine learning training. Even if the equipment is fixed in a certain period of time and all user data are generated, how to carry out distributed machine learning training by utilizing the mobile equipment and the edge computing nodes is a key problem to save edge training resources as far as possible under the condition of ensuring the model training precision.
Disclosure of Invention
The purpose of the invention is as follows: in order to overcome the defects in the prior art, the invention provides a collaborative model training task configuration method facing intelligent edge computing on one hand, so as to solve the problem that distributed machine learning training data is difficult to share, and save resource consumption as much as possible under the condition of ensuring precision.
The technical scheme is as follows: the invention discloses a collaborative model training task configuration method facing intelligent edge computing, which is used for edge computing nodes and comprises one or more training time slots. Each training time slot of the method comprises the following steps: sending a model training request to one or more edge devices; receiving an availability status and a user data size of a current time slot reported by the one or more edge devices in response to the model training request; selecting edge equipment participating in training from the current available edge equipment based on a task configuration result obtained in the last training time slot, and determining the number of training small wheels required by interactive model training; performing interactive model training with edge equipment participating in training until the number of the determined training small wheels is reached; and according to the training effect and the user data scale of the current time slot reported by each edge device, constructing and solving an optimization problem aiming at minimizing the use of edge training resources to obtain a new task configuration result for the next training time slot.
Further, the task configuration result comprises: for deciding whether to select firstiAn edge device in a training time slottParticipant decision making for internal participation training
Figure 791043DEST_PATH_IMAGE001
And for decision training time slotstNumber of internal training wheelsAmount of aid decision
Figure 887044DEST_PATH_IMAGE002
。
Further, training time slotstNumber of small wheels for training
Figure 159894DEST_PATH_IMAGE003
Calculated by the following formula:
Figure 577231DEST_PATH_IMAGE003
= K
Figure 328149DEST_PATH_IMAGE002
，
wherein the content of the first and second substances,Kis a constant.
Further, when the interactive model training is carried out with the edge equipment participating in the training, the time slot is trainedtEach training small wheel specifically comprises:
(1) the edge computing node compares the parameters of the previously trained global training model
Figure 809815DEST_PATH_IMAGE004
Local accuracy correction gradient of each edge device
Figure 987986DEST_PATH_IMAGE005
And global precision correction gradient
Figure 938625DEST_PATH_IMAGE006
Sending to all available edge devices; the edge device participating in training according to the received data and the local precision loss function of the edge device
Figure 40704DEST_PATH_IMAGE007
Separately computing respective updates to global training model parameters
Figure 65292DEST_PATH_IMAGE008
；tFor the ordinal number of the current interactive training,jfor the current trainingThe ordinal number of the small wheel,iordinal number for each edge device;
Figure 742261DEST_PATH_IMAGE009
=0；
(2) the edge computing node receives the update of the global training model parameters sent by all the edge devices participating in the training
Figure 101567DEST_PATH_IMAGE008
On the basis of the global model parameters, new global model parameters are obtained by calculation
Figure 256605DEST_PATH_IMAGE010
And sending the data to all the edge devices participating in training for verification; all edge devices participating in training are based on
Figure 169589DEST_PATH_IMAGE010
Respectively calculating to obtain new local precision
Figure 751880DEST_PATH_IMAGE011
New local accuracy correction gradient
Figure 349215DEST_PATH_IMAGE012
New local convergence performance
Figure 557211DEST_PATH_IMAGE013
And sending the data to the edge computing node for updating the record;
(3) the edge computing node corrects the gradient based on the received local precision of each edge device
Figure 556391DEST_PATH_IMAGE012
Calculating to obtain a new global precision correction gradient
Figure 309583DEST_PATH_IMAGE014
；
(4) If the current training small wheel reaches the current training time slottNumber of training wheels required
Figure 207263DEST_PATH_IMAGE003
The edge computing node also updates the global model parameters
Figure 641787DEST_PATH_IMAGE015
Sending the data to the edge device which does not participate in training; edge device not participating in training based on
Figure 557790DEST_PATH_IMAGE015
Calculate to obtain the respective second
Figure 934414DEST_PATH_IMAGE003
New local accuracy after each training small wheel
Figure 568657DEST_PATH_IMAGE016
And sending the data to the edge computing node for updating the record.
Further, in step (1), the edge devices participating in the training respectively calculate updates to the global training model parameters according to the received data
Figure 557604DEST_PATH_IMAGE008
The method specifically comprises the following steps: each edge device involved in training utilizes the obtained
Figure 328114DEST_PATH_IMAGE004
、
Figure 626372DEST_PATH_IMAGE005
And local loss of precision function of itself
Figure 997179DEST_PATH_IMAGE017
Constructing an optimization function
Figure 773505DEST_PATH_IMAGE018
And to minimize said optimization function
Figure 664101DEST_PATH_IMAGE018
In such a manner as to obtain
Figure 883992DEST_PATH_IMAGE008
(ii) a The optimization function
Figure 227248DEST_PATH_IMAGE018
Expressed as:
Figure 322112DEST_PATH_IMAGE019
，
wherein
Figure 67214DEST_PATH_IMAGE020
、
Figure 504012DEST_PATH_IMAGE021
Are all determined parameters.
Further, in step (2), new global model parameters
Figure 17121DEST_PATH_IMAGE010
Calculated by the following formula:
Figure 197566DEST_PATH_IMAGE022
，
wherein the content of the first and second substances,
Figure 437DEST_PATH_IMAGE023
time slot for current trainingtThe set of inner available edge devices,
Figure 122983DEST_PATH_IMAGE024
for training time slotstFor indicating the firstiA variable of whether an individual edge device is involved in training,
Figure 83242DEST_PATH_IMAGE024
equal to 0 or 1.
Further, in step (2), a new local precision
Figure 270641DEST_PATH_IMAGE011
Is formed by edge devices
Figure 442865DEST_PATH_IMAGE010
Substituting into its own local loss of precision function
Figure 424728DEST_PATH_IMAGE017
And then obtaining; new local precision correction gradient
Figure 229873DEST_PATH_IMAGE012
Based on new local precisions
Figure 706115DEST_PATH_IMAGE011
And then obtaining; new local convergence performance
Figure 280316DEST_PATH_IMAGE025
Is obtained by the following formula:
Figure 682348DEST_PATH_IMAGE026
。
further, in step (3), a new global precision correction gradient
Figure 443630DEST_PATH_IMAGE027
Obtained by the following formula:
Figure 35149DEST_PATH_IMAGE028
，
wherein the content of the first and second substances,
Figure 146412DEST_PATH_IMAGE023
time slot for current trainingtA set of inner available edge devices.
Further, the training effect comprises: training time slottReach a certain number of training wheels internally
Figure 532394DEST_PATH_IMAGE003
Latter global model parameters
Figure 46552DEST_PATH_IMAGE015
Local convergence performance actually observed by each edge device
Figure 363133DEST_PATH_IMAGE029
And local accuracy of each edge device updated in each training small round
Figure 646347DEST_PATH_IMAGE011
(ii) a Wherein the content of the first and second substances,
Figure 406492DEST_PATH_IMAGE030
。
further, the optimization problem is represented as:
an objective function:
Figure 158679DEST_PATH_IMAGE031
，
constraint conditions are as follows:
1)
Figure 764103DEST_PATH_IMAGE032
，
2)
Figure 167403DEST_PATH_IMAGE033
，
3)
Figure 347717DEST_PATH_IMAGE034
，
4)
Figure 570888DEST_PATH_IMAGE035
，
5)
Figure 42321DEST_PATH_IMAGE036
，
wherein the content of the first and second substances,
Figure 988542DEST_PATH_IMAGE037
for training time slots for decision makingt+1 internal training small wheelA number of aiding decision quantities;
Figure 887228DEST_PATH_IMAGE038
for training time slotstA set of inner available edge devices, determined by the current availability status of the report;
Figure 50225DEST_PATH_IMAGE039
for deciding whether to selectiAn edge device in a training time slott+1 participant decision volume for internal participation in training;man upper limit of capacity that the mobile network can concurrently transmit;
Figure 13764DEST_PATH_IMAGE040
the scale of one transmission of the model parameters and the gradient in the current edge network is determined;
Figure 543054DEST_PATH_IMAGE041
for training time slotstAvailable bandwidth in the inner edge network;
Figure 330750DEST_PATH_IMAGE042
for training time slotstInner firstiThe computational cost of each edge device for a single data sample;
Figure 466196DEST_PATH_IMAGE043
for training time slotstInner firstiThe user data size of each edge device;
Figure 545011DEST_PATH_IMAGE044
is a global loss of precision function, an
Figure 465825DEST_PATH_IMAGE044
=
Figure 706313DEST_PATH_IMAGE045
，
Figure 843902DEST_PATH_IMAGE046
；
Figure 460828DEST_PATH_IMAGE047
Is a set global precision loss;
Figure 485416DEST_PATH_IMAGE048
time slot for current trainingtMaximum value of local convergence performance of all edge devices after internal interaction training, and
Figure 647538DEST_PATH_IMAGE048
=
Figure 554314DEST_PATH_IMAGE049
=
Figure 522402DEST_PATH_IMAGE050
，
Figure 401496DEST_PATH_IMAGE029
for training time slotstReach a certain number of training wheels internally
Figure 701896DEST_PATH_IMAGE003
After thatiLocal convergence performance actually observed by each edge device.
Has the advantages that: compared with the prior art, the invention has the following advantages:
1. by constructing the optimization function for updating in the interactive training process, the edge device does not need to directly send the original data of the model parameters of the edge device to the edge computing node, but sends the updated data related to the original data of the edge device. The problem that users are often unwilling to upload own original data due to privacy protection can be solved, and the method is suitable for distributed machine learning training.
2. And constructing an optimization problem taking the use of the minimum edge training resources as a target based on the effect of interactive training and solving the optimization problem, so as to decide and obtain the task configuration of the next training time slot, wherein the task configuration comprises the selection preference and the training small wheel number of each edge device of the next training time slot. According to the decision result, the edge devices participating in training are selected from the available edge devices to train, and all the edge devices do not need to participate in training, so that the resource consumption expense of the cooperative training task configuration method is reduced by at least 27%, and the training precision is reduced by 4% at most. In other words, the invention can greatly reduce the consumption of training resources while ensuring the training precision.
Drawings
FIG. 1 is a schematic structural diagram of a collaborative model training system facing intelligent edge computing;
FIG. 2 is a graph of training resource cost variation for training using edge computing resources after applying a task configuration method;
FIG. 3 is a variation of global precision after application of a task configuration method;
fig. 4 is a variation of the maximum local convergence performance after applying the task configuration method.
Detailed Description
The disclosed method is described in further detail below with reference to the accompanying drawings.
The intelligent edge computing-oriented collaborative model training task configuration method is used for edge computing nodes and comprises one or more training time slots. Each training time slot comprises the following steps:
s1: a model training request is sent to one or more edge devices. The edge device can be a mobile device, a notebook computer and the like connected with the edge computing node.
S2: receiving an availability status and a user data size of a current time slot reported from the one or more edge devices in response to the model training request.
S3: and selecting edge devices participating in training from the currently available edge devices based on a task configuration result obtained in the last training time slot, and determining the number of training small rounds required by the interactive model training. Wherein, the task configuration result obtained from the last training time slot in the step comprises: for deciding whether to select firstiAn edge device in a training time slottParticipant decision making for internal participation training
Figure 564810DEST_PATH_IMAGE001
And for decision training time slotstInternal trainingAmount of aid decision making for training the number of small wheels
Figure 523539DEST_PATH_IMAGE002
。
Wherein the current training time slottNumber of small wheels for training
Figure 556608DEST_PATH_IMAGE003
Calculated by the following formula:
Figure 513063DEST_PATH_IMAGE003
= K
Figure 643699DEST_PATH_IMAGE002
，
wherein the content of the first and second substances,Kis a constant.
Current training time slottInternally trained edge devices are then based on participant decision-making
Figure 343802DEST_PATH_IMAGE001
Selected from among the edge devices that are in a usable state. Participant decision volume
Figure 213800DEST_PATH_IMAGE051
The preference degree is used as probability for selecting among edge devices to determine the edge devices participating in training, and the preference degree can be obtained by solving by using a Pair-wise Rounding algorithm or a DepRound algorithm. Participant decision volume for each training slot
Figure 403473DEST_PATH_IMAGE001
The specific calculation method is introduced in the following contents, which is obtained from the last training time slot decision. For the initial training slot, all available edge devices will be selected as participants since there is no training effect of the last training slot to reference.
S4: and performing interactive model training with the edge equipment participating in the training until the determined number of training small rounds is reached.
In step S4, when performing interactive model training with the edge device participating in training, each training small round specifically includes the following processes:
s41: the edge computing node compares the parameters of the previously trained global training model
Figure 975400DEST_PATH_IMAGE004
Local accuracy correction gradient of each edge device
Figure 10352DEST_PATH_IMAGE005
And global precision correction gradient
Figure 780862DEST_PATH_IMAGE006
Sending to all available edge devices; the edge device participating in training according to the received data and the local precision loss function of the edge device
Figure 79119DEST_PATH_IMAGE007
Separately computing respective updates to global training model parameters
Figure 685812DEST_PATH_IMAGE008
；tFor the ordinal number of the current interactive training,jfor the ordinal number of the current training small round,iordinal number for each edge device;
Figure 727717DEST_PATH_IMAGE009
=0。
wherein, the edge devices participating in the training respectively calculate the updating of the global training model parameters according to the received data
Figure 618313DEST_PATH_IMAGE008
The method specifically comprises the following steps:
each edge device involved in training utilizes the obtained
Figure 336739DEST_PATH_IMAGE004
、
Figure 679996DEST_PATH_IMAGE005
And local loss of precision function of itself
Figure 4886DEST_PATH_IMAGE017
Constructing an optimization function
Figure 749988DEST_PATH_IMAGE018
And to minimize said optimization function
Figure 390048DEST_PATH_IMAGE018
In such a manner as to obtain
Figure 204289DEST_PATH_IMAGE008
(ii) a Wherein the optimization function
Figure 587997DEST_PATH_IMAGE018
Expressed as:
Figure 141600DEST_PATH_IMAGE019
，
wherein
Figure 14878DEST_PATH_IMAGE020
、
Figure 254098DEST_PATH_IMAGE021
Are all determined parameters.
By constructing an optimization function
Figure 503814DEST_PATH_IMAGE018
The edge device does not need to directly send the original data of the model parameters of the edge device to the edge computing node, but sends the updated data related to the original data of the edge device. The problem that users are often unwilling to upload own original data due to privacy protection can be solved, and the method is suitable for distributed machine learning training.
S42: the edge computing node receives the update of the global training model parameters sent by all the edge devices participating in the training
Figure 161192DEST_PATH_IMAGE008
On the basis of the global model parameters, new global model parameters are obtained by calculation
Figure 956103DEST_PATH_IMAGE010
And sending the data to all the edge devices participating in training for verification; all edge devices participating in training are based on
Figure 433352DEST_PATH_IMAGE010
Respectively calculating to obtain new local precision
Figure 221180DEST_PATH_IMAGE011
New local accuracy correction gradient
Figure 779069DEST_PATH_IMAGE012
New local convergence performance
Figure 197412DEST_PATH_IMAGE013
And sending the data to the edge computing node for updating the record.
The method comprises the following steps: new global model parameters
Figure 178268DEST_PATH_IMAGE010
Calculated by the following formula:
Figure 238628DEST_PATH_IMAGE022
，
wherein the content of the first and second substances,
Figure 932915DEST_PATH_IMAGE023
time slot for current trainingtThe set of inner available edge devices,
Figure 505848DEST_PATH_IMAGE024
for training time slotstFor indicating the firstiA variable of whether an individual edge device is involved in training,
Figure 20006DEST_PATH_IMAGE024
equal to 0 or 1 and based on participant decision quantity
Figure 87319DEST_PATH_IMAGE052
Thus obtaining the product. In general terms, the amount of the solvent to be used,
Figure 380985DEST_PATH_IMAGE024
equal to 0 indicates the corresponding edge deviceiThe training is not carried out, and the training is not carried out,
Figure 875551DEST_PATH_IMAGE024
indicating a corresponding edge device when equal to 1iParticipate in training, or may be used in reverse.
New local precision
Figure 877005DEST_PATH_IMAGE011
Is formed by edge devices
Figure 794014DEST_PATH_IMAGE010
Substituting into its own local loss of precision function
Figure 134997DEST_PATH_IMAGE017
And then obtaining; new local precision correction gradient
Figure 862782DEST_PATH_IMAGE012
Based on new local precisions
Figure 39947DEST_PATH_IMAGE011
And then obtaining; new local convergence performance
Figure 776959DEST_PATH_IMAGE025
Is obtained by the following formula:
Figure 221716DEST_PATH_IMAGE026
。
s43: edge computing node corrects gradients based on received local precisions
Figure 120402DEST_PATH_IMAGE053
Calculating to obtain a new global precision correction gradient
Figure 830869DEST_PATH_IMAGE054
。
In this step, a new global precision correction gradient
Figure 794408DEST_PATH_IMAGE027
Obtained by the following formula:
Figure 906720DEST_PATH_IMAGE028
，
wherein the content of the first and second substances,
Figure 179570DEST_PATH_IMAGE023
time slot for current trainingtA set of inner available edge devices.
If the current training small wheel reaches the current training time slottNumber of training wheels required
Figure 626601DEST_PATH_IMAGE003
The edge computing node also updates the global model parameters
Figure 643098DEST_PATH_IMAGE015
Sending the data to the edge device which does not participate in training; edge device not participating in training based on
Figure 609917DEST_PATH_IMAGE015
Calculate to obtain the respective second
Figure 601138DEST_PATH_IMAGE003
New local accuracy after each training small wheel
Figure 223880DEST_PATH_IMAGE016
And sending the data to the edge computing node for updating the record.
S5: and according to the training effect and the user data scale of the current time slot reported by each edge device, constructing and solving an optimization problem aiming at minimizing the use of edge training resources to obtain a new task configuration result for the next training time slot.
In step S5, the training effect includes: training time slottReach a certain number of training wheels internally
Figure 840806DEST_PATH_IMAGE003
Latter global model parameters
Figure 114662DEST_PATH_IMAGE015
Local convergence performance actually observed by each edge device
Figure 791631DEST_PATH_IMAGE029
And local accuracy of each edge device updated in each training small round
Figure 646542DEST_PATH_IMAGE011
(ii) a Wherein the content of the first and second substances,
Figure 67159DEST_PATH_IMAGE030
。
the overall goal of the edge compute node is to compute all the training: (
Figure 742991DEST_PATH_IMAGE055
Training of sub-training time slots), minimizing the use of edge training resources under the condition that the respective training satisfies the precision, and therefore, establishing an optimization problem as follows:
optimizing the target:
Figure 777812DEST_PATH_IMAGE056
；
constraint conditions are as follows:
1) for edge transmission limitation:
Figure 437464DEST_PATH_IMAGE057
2) for participant selection control:
Figure 333876DEST_PATH_IMAGE058
3) for global posterior accuracy requirements:
Figure 146105DEST_PATH_IMAGE059
4) for domain-defined restrictions of the decision:
Figure 836980DEST_PATH_IMAGE060
in the formula (I), the compound is shown in the specification,
Figure 983928DEST_PATH_IMAGE003
for training time slotstThe number of small rounds required for the internal training,
Figure 933298DEST_PATH_IMAGE061
in order to train the resulting global convergence accuracy,
Figure 583723DEST_PATH_IMAGE062
is a global loss of precision;
Figure 773395DEST_PATH_IMAGE023
for training time slotstA set of inner available edge devices, determined by the current availability status of the report;
Figure 830475DEST_PATH_IMAGE001
for deciding whether to selectiAn edge device in a training time slottThe decision making quantity of the participants who participate in the training;
Figure 131007DEST_PATH_IMAGE040
the scale of one transmission of the model parameters and the gradient in the current edge network is determined;
Figure 354047DEST_PATH_IMAGE041
for training time slotstAvailable bandwidth in the inner edge network;
Figure 449041DEST_PATH_IMAGE042
for trainingExercise time slottInner firstiThe computational cost of each edge device for a single data sample;
Figure 305002DEST_PATH_IMAGE043
for training time slotstInner firstiThe user data size of each edge device;man upper limit of capacity that the mobile network can concurrently transmit;
Figure 97640DEST_PATH_IMAGE048
time slot for current trainingtMaximum value of local convergence performance of all edge devices after internal interaction training, and
Figure 988235DEST_PATH_IMAGE048
=
Figure 457394DEST_PATH_IMAGE049
=
Figure 49918DEST_PATH_IMAGE050
，
Figure 629935DEST_PATH_IMAGE029
for training time slotstReach a certain number of training wheels internally
Figure 109458DEST_PATH_IMAGE003
After thatiLocal convergence performance actually observed by each edge device;
Figure 580145DEST_PATH_IMAGE015
for training time slotstInner part
Figure 348381DEST_PATH_IMAGE003
Model parameters after training the small wheel;
Figure 794406DEST_PATH_IMAGE044
is a global loss of precision function, an
Figure 580965DEST_PATH_IMAGE044
=
Figure 454244DEST_PATH_IMAGE045
，
Figure 460508DEST_PATH_IMAGE046
。
Because of the online scene, the actual training effect cannot be accurately observed during decision making, so that the optimization problem can only be decomposed into each training time slot in practice to solve the subproblems each time. Furthermore, in each solving of the sub-problem, the local convergence accuracy of the training cannot be observed in advance
Figure 444644DEST_PATH_IMAGE029
=
Figure 898760DEST_PATH_IMAGE050
And global convergence accuracy
Figure 129890DEST_PATH_IMAGE061
. Therefore, it is necessary to use the training effect of each edge device in the last training time slot as a reference to approximately replace the global and local convergence accuracy that cannot be obtained by the current training. In summary, for training time slotstThe training in, the subproblems are actually:
an objective function:
Figure 935035DEST_PATH_IMAGE031
，
constraint conditions are as follows:
1)
Figure 660545DEST_PATH_IMAGE032
，
2)
Figure 985478DEST_PATH_IMAGE033
，
3)
Figure 138242DEST_PATH_IMAGE034
，
4)
Figure 430683DEST_PATH_IMAGE035
，
5)
Figure 943573DEST_PATH_IMAGE036
，
wherein the content of the first and second substances,
Figure 372280DEST_PATH_IMAGE037
for training time slots for decision makingt+1 training small round number of assistant decision quantity;
Figure 712257DEST_PATH_IMAGE039
for deciding whether to selectiAn edge device in a training time slott+1 participation in the decision making volume of the trained participants.
In the above sub-problem solution, albeit in the objective function
Figure 226415DEST_PATH_IMAGE039
Is quadratic, but has the same meaning as without quadratic, which is to say that it is a convex function in the real number domain. Among the above sub-problems
Figure 293728DEST_PATH_IMAGE039
The solution can be performed using a sophisticated solution tool such as IPOPT + AMPL.
Fig. 1 shows a structure of the intelligent edge computing oriented collaborative model training system of the present invention by taking a selection of four edge devices as an example, where all the edge devices are connected to the same edge computing node and perform data interaction, and the maximum capacity that an edge network can allow to transmit may include four edge devices; the intelligent edge computing-oriented collaborative model training task configuration method of the present invention is further explained below by taking two times of global model training as an example:
(1) when a first model training request arrives, data needing training is distributed on three available edge devices; because no previous trained effect is taken as a reference, all three available edge devices are regarded as participants of distributed machine learning training, and the three participants report the user data scale to the edge computing node;
(2) initializing a global model (maintenance of edge calculation nodes), precision correction gradients of each edge device and a global precision correction gradient by the edge calculation nodes;
(3) the edge calculation node issues the global model parameters, the precision correction gradient of each edge device and the global precision correction gradient to the three edge devices;
(4) after each edge device receives the information from the edge computing device, the accuracy loss function is constructed by using the user data on the own device so as to minimize
Figure 91789DEST_PATH_IMAGE063
Is obtained in the form of
Figure 586355DEST_PATH_IMAGE064
The obtained process is continuously iterative updating
Figure 587809DEST_PATH_IMAGE064
；
(5) Edge device utilization
Figure 203686DEST_PATH_IMAGE064
Updating a local model of the self, and performing primary verification by using a precision loss function of the self to obtain local precision, local convergence performance and a local precision correction gradient;
(6) each edge device is to
Figure 606986DEST_PATH_IMAGE064
The local precision, the local convergence performance and the local precision correction gradient are sent to an edge computing node;
(7) sent by edge computing nodes according to each edge device
Figure 69191DEST_PATH_IMAGE064
Updating a global model on the basis of the global model; office transmitted by edge equipmentThe partial precision correction gradient is updated with the global precision correction gradient; and recording the local convergence performance;
(8) since all the edge devices participate currently, the global precision is only the weighted average of the local precision of each edge device;
(9) continuously performing the steps (3) to (8) until the number of training rounds reaches the value of
Figure 744892DEST_PATH_IMAGE048
Determined
Figure 216325DEST_PATH_IMAGE003
；
(10) Observing the local training effect of the three edge devices, namely the local convergence performance of each small wheel, and correcting the preference of the three edge devices according to the local convergence performance;
(11) a second distributed machine learning model training request arrives, and four edge devices are available currently;
(12) because the local convergence performance of the first edge device in the last training is not good, the edge computing node selects other edge devices except the first edge device as participants by combining the selection preference of each edge device;
(13) performing steps (2) to (10) for a second distributed machine learning training;
(14) in the step (8) of the second distributed machine learning training, although the first edge device does not participate in the distributed machine learning training, during verification, the latest model parameters still need to be obtained from the edge computing node, and verification is performed once by using the accuracy loss function of the first edge device, so that local accuracy is obtained and sent to the edge computing node.
The effect of the experiment is shown in fig. 2 to 4, and fig. 2 shows the change of the edge computing resource consumption (normalized according to the maximum value) in the process of continuously performing distributed machine learning training after the dynamic task adjustment method is applied, and the edge training resource consumption is set for the edge computing nodes and each edgeThe sum of the computing resource cost and the transmission cost of each small round is prepared, the training resource consumption is always minimum compared with other methods, and the overhead is reduced by at least 27%; FIG. 3 shows the global accuracy change during the continuous distributed machine learning training process after applying the dynamic task adjustment method, which actually corresponds to the global posterior accuracy in modeling
Figure 162546DEST_PATH_IMAGE065
In order to verify the obtained precision on all devices, the proposed method reduces the training precision by at most 4%; FIG. 4 shows the variation of maximum local convergence performance during continuous distributed machine learning training, i.e., modeling, after applying the dynamic task adjustment method
Figure 326811DEST_PATH_IMAGE048
=
Figure 240541DEST_PATH_IMAGE049
=
Figure 499352DEST_PATH_IMAGE050
。
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: although the present invention has been described in detail with reference to the above embodiments, the interaction between the control node and the edge computing node, the feedback information content collection and the online scheduling method in the present invention are applicable to all systems, and it should be understood by those skilled in the art that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.

Claims (9)

1. A collaborative model training task configuration method facing intelligent edge computing is used for edge computing nodes and comprises one or more training time slots, and is characterized in that each training time slot comprises the following steps:
sending a model training request to one or more edge devices;
receiving an availability status and a user data size of a current time slot reported by the one or more edge devices in response to the model training request;
selecting edge equipment participating in training from the current available edge equipment based on a task configuration result obtained in the last training time slot, and determining the number of training small wheels required by interactive model training;
performing interactive model training with edge equipment participating in training until the number of the determined training small wheels is reached; according to the training effect and the user data scale of the current time slot reported by each edge device, constructing an optimization problem aiming at minimizing the use of edge training resources and solving the optimization problem to obtain a new task configuration result for the next training time slot;
wherein, when the interactive model training is carried out with the edge equipment participating in the training, the training time slottEach training small wheel specifically comprises:
(1) the edge computing node compares the parameters of the previously trained global training model
Figure 429638DEST_PATH_IMAGE001
Local accuracy correction gradient of each edge device
Figure 815620DEST_PATH_IMAGE002
And global precision correction gradient
Figure 909871DEST_PATH_IMAGE003
Sending to all available edge devices; the edge device participating in training according to the received data and the local precision loss function of the edge device
Figure 898556DEST_PATH_IMAGE004
Separately computing respective updates to global training model parameters
Figure 447349DEST_PATH_IMAGE005
； jIs at presentThe number of the small round is trained,iis shown asiAn edge device;
Figure 614019DEST_PATH_IMAGE006
=0；
(2) the edge computing node receives the update of the global training model parameters sent by all the edge devices participating in the training
Figure 677790DEST_PATH_IMAGE005
On the basis of the global model parameters, new global model parameters are obtained by calculation
Figure 345532DEST_PATH_IMAGE007
And sending the data to all the edge devices participating in training for verification; all edge devices participating in training are based on
Figure 857154DEST_PATH_IMAGE007
Respectively calculating to obtain new local precision
Figure 584938DEST_PATH_IMAGE008
New local accuracy correction gradient
Figure 932743DEST_PATH_IMAGE009
New local convergence performance
Figure 138596DEST_PATH_IMAGE010
And sending the data to the edge computing node for updating the record;
(3) the edge computing node corrects the gradient based on the received local precision of each edge device
Figure 6189DEST_PATH_IMAGE009
Calculating to obtain a new global precision correction gradient
Figure 232771DEST_PATH_IMAGE011
；
(4) If the current training small wheel reaches the current training time slottNumber of training wheels required
Figure 943238DEST_PATH_IMAGE012
The edge computing node also updates the global model parameters
Figure 52316DEST_PATH_IMAGE013
Sending the data to the edge device which does not participate in training; edge device not participating in training based on
Figure 164628DEST_PATH_IMAGE013
Calculate to obtain the respective second
Figure 358849DEST_PATH_IMAGE012
New local accuracy after each training small wheel
Figure 556612DEST_PATH_IMAGE014
And sending the data to the edge computing node for updating the record.
2. The intelligent edge computing-oriented collaborative model training task configuration method according to claim 1, wherein the task configuration result includes: for deciding whether to select firstiAn edge device in a training time slottParticipant decision making for internal participation training
Figure 979635DEST_PATH_IMAGE015
And for decision training time slotstDecision-making aid for number of inner training small wheel
Figure 946454DEST_PATH_IMAGE016
。
3. The intelligent edge computing-oriented collaborative model training task configuration method according to claim 2, wherein the training time slot is a training time slottNumber of small wheels for training
Figure 311576DEST_PATH_IMAGE012
Calculated by the following formula:
Figure 731056DEST_PATH_IMAGE012
= K
Figure 659566DEST_PATH_IMAGE016
，
wherein the content of the first and second substances,Kis a constant.
4. The intelligent edge computing-oriented collaborative model training task configuration method according to claim 1, wherein in step (1), the edge devices involved in training respectively calculate respective updates to global training model parameters according to the received data
Figure 605526DEST_PATH_IMAGE005
The method specifically comprises the following steps:
each edge device involved in training utilizes the obtained
Figure 16915DEST_PATH_IMAGE001
、
Figure 64637DEST_PATH_IMAGE002
And local loss of precision function of itself
Figure 954096DEST_PATH_IMAGE017
Constructing an optimization function
Figure 20141DEST_PATH_IMAGE018
And to minimize said optimization function
Figure 602432DEST_PATH_IMAGE018
In such a manner as to obtain
Figure 373335DEST_PATH_IMAGE005
；
The optimization function
Figure 66484DEST_PATH_IMAGE018
Expressed as:
Figure 924719DEST_PATH_IMAGE019
，
wherein
Figure 553278DEST_PATH_IMAGE020
、
Figure 434646DEST_PATH_IMAGE021
Are all determined parameters.
5. The intelligent edge computing-oriented collaborative model training task configuration method according to claim 1, wherein in the step (2), new global model parameters
Figure 56120DEST_PATH_IMAGE007
Calculated by the following formula:
Figure 706544DEST_PATH_IMAGE022
，
wherein the content of the first and second substances,
Figure 4539DEST_PATH_IMAGE023
time slot for current trainingtThe set of inner available edge devices,
Figure 373204DEST_PATH_IMAGE024
for training time slotstFor indicating the firstiA variable of whether an individual edge device is involved in training,
Figure 532790DEST_PATH_IMAGE024
equal to 0 or 1.
6. The intelligent edge computing-oriented collaborative model training task configuration method according to claim 1, wherein in the step (2), the new local precision
Figure 303300DEST_PATH_IMAGE008
Is formed by edge devices
Figure 8082DEST_PATH_IMAGE007
Substituting into its own local loss of precision function
Figure 864042DEST_PATH_IMAGE017
And then obtaining; new local precision correction gradient
Figure 30581DEST_PATH_IMAGE009
Based on new local precisions
Figure 766849DEST_PATH_IMAGE008
And then obtaining; new local convergence performance
Figure 32746DEST_PATH_IMAGE025
Is obtained by the following formula:
Figure 235057DEST_PATH_IMAGE026
。
7. the intelligent edge computing-oriented collaborative model training task configuration method according to claim 1, wherein in the step (3), a new global accuracy correction gradient is adopted
Figure 877391DEST_PATH_IMAGE027
Obtained by the following formula:
Figure 497859DEST_PATH_IMAGE028
，
wherein the content of the first and second substances,
Figure 934657DEST_PATH_IMAGE023
time slot for current trainingtA set of inner available edge devices.
8. The intelligent edge computing-oriented collaborative model training task configuration method according to claim 1, wherein the training effect includes: training time slottReach a certain number of training wheels internally
Figure 624264DEST_PATH_IMAGE012
Latter global model parameters
Figure 804710DEST_PATH_IMAGE013
Local convergence performance actually observed by each edge device
Figure 715903DEST_PATH_IMAGE029
And local accuracy of each edge device updated in each training small round
Figure 448235DEST_PATH_IMAGE008
(ii) a Wherein the content of the first and second substances,
Figure 766084DEST_PATH_IMAGE030
。
9. the intelligent edge computing-oriented collaborative model training task configuration method according to claim 1, wherein the optimization problem is expressed as:
an objective function:
Figure 360008DEST_PATH_IMAGE031
，
constraint conditions are as follows:
Figure 814123DEST_PATH_IMAGE032
，
Figure 717357DEST_PATH_IMAGE033
，
Figure 256923DEST_PATH_IMAGE034
，
Figure 359264DEST_PATH_IMAGE035
，
Figure 58099DEST_PATH_IMAGE036
，
wherein the content of the first and second substances,
Figure 7600DEST_PATH_IMAGE037
for training time slots for decision makingt+1 training small round number of assistant decision quantity;
Figure 847511DEST_PATH_IMAGE038
for training time slotstA set of inner available edge devices, determined by the current availability status of the report;
Figure 563663DEST_PATH_IMAGE039
for deciding whether to selectiAn edge device in a training time slott+1 participant decision volume for internal participation in training;
Figure 726792DEST_PATH_IMAGE040
the scale of one transmission of the model parameters and the gradient in the current edge network is determined;
Figure 221096DEST_PATH_IMAGE041
for training time slotstAvailable bandwidth in the inner edge network;man upper limit of capacity that the mobile network can concurrently transmit;
Figure 735254DEST_PATH_IMAGE042
for training time slotstInner firstiThe computational cost of each edge device for a single data sample;
Figure 927201DEST_PATH_IMAGE043
for training time slotstInner firstiThe user data size of each edge device;
Figure 820201DEST_PATH_IMAGE044
is a global loss of precision function, an
Figure 377085DEST_PATH_IMAGE044
=
Figure 237593DEST_PATH_IMAGE045
，
Figure 905335DEST_PATH_IMAGE046
；
Figure 431605DEST_PATH_IMAGE047
Is a set global precision loss;
Figure 159390DEST_PATH_IMAGE048
time slot for current trainingtMaximum value of local convergence performance of all edge devices after internal interaction training, and
Figure 507194DEST_PATH_IMAGE048
=
Figure 791676DEST_PATH_IMAGE049
=
Figure 783903DEST_PATH_IMAGE050
，
Figure 807223DEST_PATH_IMAGE029
for training time slotstReach a certain number of training wheels internally
Figure 252111DEST_PATH_IMAGE012
After thatiLocal convergence performance actually observed by each edge device.
CN202110153068.3A 2021-02-04 2021-02-04 Intelligent edge calculation-oriented collaborative model training task configuration method Active CN112506673B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110153068.3A CN112506673B (en) 2021-02-04 2021-02-04 Intelligent edge calculation-oriented collaborative model training task configuration method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110153068.3A CN112506673B (en) 2021-02-04 2021-02-04 Intelligent edge calculation-oriented collaborative model training task configuration method

Publications (2)

Publication Number Publication Date
CN112506673A CN112506673A (en) 2021-03-16
CN112506673B true CN112506673B (en) 2021-06-08

Family

ID=74953076

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110153068.3A Active CN112506673B (en) 2021-02-04 2021-02-04 Intelligent edge calculation-oriented collaborative model training task configuration method

Country Status (1)

Country Link
CN (1) CN112506673B (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111898763A (en) * 2020-06-03 2020-11-06 东南大学 Robust Byzantine fault-tolerant distributed gradient descent algorithm

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020024585A1 (en) * 2018-08-03 2020-02-06 华为技术有限公司 Method and apparatus for training object detection model, and device
CN111339554B (en) * 2020-02-17 2022-04-01 电子科技大学 User data privacy protection method based on mobile edge calculation
CN111367657B (en) * 2020-02-21 2022-04-19 重庆邮电大学 Computing resource collaborative cooperation method based on deep reinforcement learning
CN111582016A (en) * 2020-03-18 2020-08-25 宁波送变电建设有限公司永耀科技分公司 Intelligent maintenance-free power grid monitoring method and system based on cloud edge collaborative deep learning
CN111459505B (en) * 2020-05-22 2021-06-25 南京大学 Multi-version inference model deployment method, device and system in edge computing environment
CN112085321A (en) * 2020-07-30 2020-12-15 国家电网有限公司 Station area state evaluation method based on edge calculation

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111898763A (en) * 2020-06-03 2020-11-06 东南大学 Robust Byzantine fault-tolerant distributed gradient descent algorithm

Also Published As

Publication number Publication date
CN112506673A (en) 2021-03-16

Similar Documents

Publication Publication Date Title
CN110782042B (en) Method, device, equipment and medium for combining horizontal federation and vertical federation
CN112367109B (en) Incentive method for digital twin-driven federal learning in air-ground network
CN110659678B (en) User behavior classification method, system and storage medium
CN110490335A (en) A kind of method and device calculating participant's contribution rate
CN105897616B (en) Resource allocation method and server
CN108665089B (en) Robust optimization model solving method for site selection problem
CN112416986B (en) User portrait realizing method and system based on hierarchical personalized federal learning
CN114418035A (en) Decision tree model generation method and data recommendation method based on decision tree model
CN113536097B (en) Recommendation method and device based on automatic feature grouping
CN110717671A (en) Method and device for determining contribution degree of participants
CN113360777B (en) Content recommendation model training method, content recommendation method and related equipment
CN109067583A (en) A kind of resource prediction method and system based on edge calculations
CN111783810A (en) Method and apparatus for determining attribute information of user
CN116627970A (en) Data sharing method and device based on blockchain and federal learning
CN115034836A (en) Model training method and related device
CN116910373B (en) House source recommendation method and device, electronic equipment and storage medium
CN112506673B (en) Intelligent edge calculation-oriented collaborative model training task configuration method
US20160342899A1 (en) Collaborative filtering in directed graph
WO2023087933A1 (en) Content recommendation method and apparatus, device, storage medium, and program product
CN114092162B (en) Recommendation quality determination method, and training method and device of recommendation quality determination model
CN115577301A (en) Personalized splitting learning method and application for distributed system
CN115249175A (en) Self-attention sequence recommendation method, system and equipment integrating time information
US20120151491A1 (en) Redistributing incomplete segments for processing tasks in distributed computing
CN115345635A (en) Processing method and device for recommended content, computer equipment and storage medium
CN116720003B (en) Ordering processing method, ordering processing device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant