WO2022134809A1 - 模型训练处理方法、装置、计算机设备及介质 - Google Patents

模型训练处理方法、装置、计算机设备及介质 Download PDF

Info

Publication number
WO2022134809A1
WO2022134809A1 PCT/CN2021/125440 CN2021125440W WO2022134809A1 WO 2022134809 A1 WO2022134809 A1 WO 2022134809A1 CN 2021125440 W CN2021125440 W CN 2021125440W WO 2022134809 A1 WO2022134809 A1 WO 2022134809A1
Authority
WO
WIPO (PCT)
Prior art keywords
training
server
target
affiliated
model training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/125440
Other languages
English (en)
French (fr)
Inventor
张山
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
OneConnect Smart Technology Co Ltd
Original Assignee
OneConnect Smart Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by OneConnect Smart Technology Co Ltd filed Critical OneConnect Smart Technology Co Ltd
Publication of WO2022134809A1 publication Critical patent/WO2022134809A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Definitions

  • the present application relates to the technical field of data processing, and in particular, to a model training processing method, apparatus, computer equipment and medium.
  • Deep neural networks have been successfully applied in many fields, including image recognition, texture classification, speech recognition, etc. In recent years, the performance of deep neural networks has been significantly improved due to the use of deeper network architectures and larger training sample sets for training.
  • the inventor found that the prior art has at least the following problems: deep neural network learning requires too high time cost, and deep neural network learning often takes several hours or even costs It takes a few days. Once there is a problem with the algorithm, the deep neural network learning model must be retrained, and the efficiency of model training is extremely low.
  • a first aspect of the embodiments of the present application provides a model training processing method, where the model training processing method includes:
  • the affiliate server determines the affiliate model training request assigned to the target affiliate server that does not output the training response message, and output the affiliate model training request to the target affiliate server corresponding to the target affiliate server. in the target slave node;
  • the target training model is modified based on the abnormal item, and the training quantity of the affiliated server is dynamically adjusted according to the model training result to retrain the target training model.
  • model training processing apparatus includes:
  • the request receiving module is used to obtain the subsidiary server corresponding to the target server when the model training request is received;
  • a quantity calculation module configured to calculate the optimal training quantity of the affiliated servers, and allocate the model training request to each of the affiliated servers based on the optimal training quantity;
  • a message detection module for detecting whether the affiliated server outputs a training response message
  • the node determination module is configured to determine, when the detection result is that the affiliated server does not output the training response message, the affiliated model training request assigned to the target affiliated server that does not output the training response message, and output the affiliated model training request to the corresponding in the target subordinate node of the target subordinate server;
  • An anomaly detection module configured to collect the model training results of each of the affiliated servers, and detect whether there are any abnormal items in the model training results;
  • a quantity adjustment module configured to modify the target training model based on the abnormal item when the detection result is that there is an abnormal item in the model training result, and dynamically adjust the training quantity of the affiliated server to retrain according to the model training result The target training model.
  • a third aspect of the embodiments of the present application further provides a computer device, where the computer device includes a processor, and the processor is configured to execute computer-readable instructions stored in a memory to implement the following steps:
  • the affiliate server determines the affiliate model training request assigned to the target affiliate server that does not output the training response message, and output the affiliate model training request to the target affiliate server corresponding to the target affiliate server. in the target slave node;
  • the target training model is modified based on the abnormal item, and the training quantity of the affiliated server is dynamically adjusted according to the model training result to retrain the target training model.
  • a fourth aspect of the embodiments of the present application further provides a computer-readable storage medium, where computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the following steps are implemented:
  • the affiliate server determines the affiliate model training request assigned to the target affiliate server that does not output the training response message, and output the affiliate model training request to the target affiliate server corresponding to the target affiliate server. in the target slave node;
  • the target training model is modified based on the abnormal item, and the training quantity of the affiliated server is dynamically adjusted according to the model training result to retrain the target training model.
  • the present application can reduce the time required for model training and improve model training efficiency.
  • FIG. 1 is a flowchart of a model training processing method provided in Embodiment 1 of the present application.
  • FIG. 2 is a structural diagram of a model training processing apparatus provided in Embodiment 2 of the present application.
  • FIG. 3 is a schematic structural diagram of a computer device provided in Embodiment 3 of the present application.
  • the model training processing method provided by the embodiment of the present application is executed by computer equipment, and correspondingly, the model training processing apparatus runs in the computer equipment.
  • FIG. 1 is a flowchart of a model training processing method provided in Embodiment 1 of the present application.
  • the model training processing method may include the following steps. According to different requirements, the order of the steps in the flowchart may be changed, and some may be omitted:
  • the model training request refers to a request for requesting to train a target training model
  • the model training request is a request sent to a target application to start training of the target training model
  • the target application refers to an application that includes the target training model.
  • the application forwards the model training request to the target server cluster, and passes the multiple servers in the target server cluster.
  • the model training requests are executed in parallel.
  • the target server cluster includes a target server and an auxiliary server corresponding to the target server.
  • the target server refers to a server that receives a model training request forwarded by the target application, and the affiliated server refers to a server associated with the target server.
  • acquiring an affiliate server corresponding to a target server includes: parsing the model training request to obtain target routing information corresponding to the model training request; determining a target according to the target routing information application; acquiring a target server cluster corresponding to the target application; acquiring a target server and an affiliated server in the target server cluster.
  • each model training request includes target routing information
  • the target routing information is used to indicate the routing information of the target application corresponding to the target training model, and the target corresponding to the model training request can be obtained according to the target routing information.
  • application is associated with a target server cluster, and a corresponding target server cluster can be obtained according to the target application.
  • the target server cluster refers to a cluster including one target server and multiple subsidiary servers, and the number of the subsidiary servers may be one or multiple, which is not limited herein.
  • the number of the auxiliary servers may be determined according to the training quantity carried in the model training request, and the larger the training quantity, the larger the corresponding number of the auxiliary servers.
  • the obtaining the subsidiary server corresponding to the target server further includes: parsing the model training request to obtain the training quantity corresponding to the target training model; determining the training magnitude corresponding to the training quantity; determining the subsidiary server according to the training magnitude quantity.
  • the model training request includes not only target routing information, but also information such as the training quantity of the corresponding target training model, which is not limited here.
  • the training quantity refers to the quantity of training data, and the training quantity can be set according to the actual situation of the target training model.
  • there is a preset first mapping relationship between the training quantity and the training magnitude and a preset second mapping relationship exists between the training magnitude and the number of affiliated servers.
  • the quantity traverses the first mapping relationship, and the training magnitude corresponding to the training quantity can be determined, and the training magnitude may be low-level, intermediate-level, and high-level.
  • the data processing capabilities of different affiliated servers may or may not be the same.
  • the training quantity carried by the simulated training request can be divided into equal amounts to obtain an affiliated model training request, and the affiliated model training request can be allocated to the affiliated server .
  • the data processing capabilities of different affiliated servers are not the same, since the data processing capabilities of each server are the same, the amount of training data processed by different affiliated servers in the same or similar time period is different. Allocate the optimal training quantity to each of the affiliated servers, so that the affiliated server with high data processing capability can process more training data, while the affiliated server with less data processing capability can process less training data, so that the affiliated servers can all process the same training data. or similar time periods to process their respective training quantities, reducing the overall time required for model training and improving the efficiency of model training.
  • the data processing capability of the auxiliary server may be determined according to the load level of the server. The higher the load level, the higher the corresponding data processing capability.
  • the load level can be determined by three indicators of the server's CPU data, memory data and network load data.
  • the CPU data can refer to the CPU load parameter
  • the memory data can refer to the memory occupancy parameter
  • the network load data Flow parameters can be included, which are not limited here.
  • the load level of the corresponding server is obtained by acquiring the above three indicators and inputting them into a pre-trained load level determination model for processing.
  • the calculating the optimal training quantity of the auxiliary server includes: obtaining the data processing capability corresponding to each auxiliary server; obtaining the training quantity of the corresponding target training model carried in the model training request; processing according to the data The capacity and the number of trainings calculate the optimal number of trainings for each of the satellites.
  • the training quantity carried in the model training request may refer to the total amount of training data, and the data processing capability may refer to the quantity of model training data processed by the server per unit time. By configuring the same or similar training time, it is possible to Get the optimal number of trainings allocated to each server.
  • the model training request is allocated and processed, and the model training request is divided into multiple affiliated model trainings
  • Each of the auxiliary model training requests corresponds to the auxiliary server, and is used for requesting to call the auxiliary server to perform training processing on the training data.
  • the assigning the model training request based on the optimal training quantity includes: obtaining the number of the affiliated servers and the optimal training quantity of each affiliated server; The model training request is divided according to the optimal training quantity to obtain several auxiliary model training requests; the auxiliary model training requests are allocated to the corresponding auxiliary servers.
  • each of the affiliated model training requests carries the identity of the affiliated server and the training quantity that the affiliated server should train.
  • the affiliated model training request can be allocated to the corresponding on an affiliated server.
  • step S13 Detect whether the affiliate server outputs a training response message, and when the detection result is that the affiliate server does not output a training response message, step S14 is performed.
  • each of the affiliated servers will output a training response message to the target server before starting training.
  • the affiliated server By checking whether the affiliated server outputs a training response message, it can be determined whether the affiliated server has Normal training model data.
  • the detection result is that the affiliate server outputs a training response message, it is determined that the affiliate server training process is normal; when the detection result is that the affiliate server does not output a training response message, it is determined that the affiliate server training process is abnormal.
  • the detecting whether the auxiliary server outputs a training response message includes: acquiring the operation log corresponding to each of the auxiliary servers; detecting whether there is target response data corresponding to each of the auxiliary servers in the operation log; When the detection result is that there is target response data corresponding to each of the auxiliary servers in the operation log, it is determined that the auxiliary server outputs a training response message; when the detection result is that there is no target corresponding to the auxiliary server in the operation log When responding to the data, it is determined that the affiliate server does not output a training response message.
  • the operation log of each affiliated server can be stored in the target node of the blockchain, and each affiliated server stores data such as training response messages, abnormal operation messages, and operation end messages generated during the operation in the target node. , by querying the running log, the running process of each subsidiary server can be obtained.
  • the operation log may also record the address information, port information, slave node information, port information of slave nodes (that is, slave port information), node type (the node type may include main task and subordinate node information) of each server. tasks) and auxiliary model training request information, etc., which are not limited here.
  • step S15 is executed to collect model training results of each of the affiliated servers.
  • each of the affiliated servers is provided with an affiliated node, and the affiliated node can be configured with model training processing logic corresponding to the affiliated server.
  • the target affiliated server does not output training When responding to the message, it indicates that the running process of the target affiliate server is abnormal at this time.
  • the target slave node corresponding to the target slave server with abnormal running process to execute the corresponding model training request, it ensures that the model training request can be executed normally, saves the time required for model training, and improves the efficiency of model training.
  • the determining the affiliate model training request assigned to the target affiliate server that does not output the training response message, and outputting the affiliate model training request to the target affiliate node corresponding to the target affiliate server includes: obtaining the target affiliate server. A subordinate node set of a target subordinate server; determining a target subordinate node from the subordinate node set; outputting the subordinate model training request to the target subordinate node, and the target subordinate node performs model training.
  • the number of slave nodes corresponding to the target slave server may be one or more.
  • the method further includes: acquiring subordinate node sets of the target subordinate server; determining execution logic corresponding to each subordinate node in the subordinate node set; detecting Whether there is a target execution logic consistent with the execution logic of the target affiliate server; when the detection result is that there is a target execution logic consistent with the execution logic of the target affiliate server, determine the subordinate node corresponding to the target execution logic as the target subordinate node .
  • most of the subordinate nodes are used to provide query and question-answer matching logic, and a subordinate node can be configured to configure task processing logic for calling the subordinate node corresponding to the subordinate server when the subordinate server cannot normally perform the model training task. Perform model training tasks.
  • a target slave node is preconfigured for executing task processing logic.
  • the task processing logic of the slave node does not need to be pre-configured, and when it is detected that the task execution process of the subsidiary server is abnormal, the target execution logic information corresponding to the subsidiary server is obtained, and the target execution logic information is output to the Any subordinate node in the subordinate node set is configured, and the subordinate node is configured according to the target execution logic information, so that the subordinate node has task processing logic.
  • the present application configures the slave nodes in real time according to the target execution logic information when detecting an abnormal task execution of the slave server, so as to avoid the waste of resources caused by the pre-configuration, and improve the development efficiency.
  • the slave node is preset with relevant configuration information for executing the corresponding query question-answer matching logic, and configuring the slave node according to the target execution logic information also means storing the target execution logic information in the configuration of the corresponding slave node information, and call the target execution logic information so that the slave node has the task processing capability.
  • step S15 Collect the model training results of each of the affiliated servers, and detect whether there are abnormal items in the model training results. When the detection result is that there are abnormal items in the model training results, step S16 is performed.
  • each affiliated server will output relevant training data to the operation log of the target node of the blockchain.
  • the model training result will be output to the running log of the target node.
  • the method further includes: acquiring the association relationship between the target server and the affiliated server; Model training logic diagram, wherein each server corresponds to the target button in the model training logic diagram; in the visualized model training logic diagram, obtain the model training progress and training rate of each of the affiliated servers; The training progress and the training rate determine the actualized display area and the actualized display rate of each of the target buttons.
  • the model training logic diagram corresponds to the target button of each server, each button corresponds to an affiliated server, the target button and the affiliated server can be associated with a preset identifier, and by viewing the preset identifier, That is, the target server associated with each of the target buttons can be determined, and the preset identification can be a digital identification, an alphabetic identification and a color identification, which is not limited herein.
  • the present application can visually display the training progress and training rate of each of the affiliated servers by performing virtual and actual display on each of the target buttons. When the actualized display area of the target button is the entire target button area, it is determined that the server training is completed; the faster the actualized display rate of the target button, the faster the training rate of the corresponding server.
  • the method further includes: detecting the Whether a training abnormality item occurs during the operation of the affiliated server; when the detection result is that a training abnormality item occurs during the operation of the affiliated server, obtain the number of the abnormality item; Set the preset flag in the target button of the logic diagram.
  • the number of the abnormal items is different, and the preset identification is also different.
  • the preset identification is a color identification
  • the preset identification is closer to red , which is not limited here.
  • the developer when the detection result is that an abnormal item exists in the model training result, the developer can modify the target training model based on the abnormal item, so that the modified target training model can solve the above abnormality item.
  • the target training model needs to be retrained according to the training data to ensure that the target training model has no other abnormal items when it is officially used.
  • the model training result also includes the training duration of each subsidiary server, and the present application dynamically adjusts the amount of training data of the subsidiary server according to the training duration, which further saves model training time and improves model training efficiency.
  • the dynamically adjusting the training quantity of the affiliated servers according to the model training results includes: acquiring and parsing the model training results of each of the affiliated servers, and obtaining the actual model training duration corresponding to each of the affiliated servers. ; Detect whether the actual model training duration satisfies the target adjustment condition; when the detection result is that the actual model training duration meets the target adjustment condition, determine the target affiliate server that satisfies the target adjustment condition; dynamically adjust the target affiliate server's number of training sessions.
  • the target adjustment condition refers to an abnormal duration in the actual model training duration.
  • the time interval between the training durations is within the preset time interval, and the time interval between the actual model training duration of the affiliate server 5 and the actual model training duration of one or more of the remaining 4 affiliate servers exceeds the preset time interval, then the affiliate server
  • the actual model training duration of 5 satisfies the target adjustment condition, and it is necessary to dynamically adjust the training quantity of the subsidiary server 5, so that the training duration of the subsidiary server 5 and the training duration of several other subsidiary servers are at a preset time interval.
  • the time required for model training is further reduced as a whole, and the efficiency of model training is improved.
  • the dynamically adjusting the training quantity of the affiliated servers includes: determining the first affiliated server with the longest actual model training duration and the second affiliated server with the shortest actual model training duration; calculating the difference between the first affiliated server and all the affiliated servers. the training duration interval of the second affiliated server; obtain the first training quantity of the first affiliated server and the second training quantity of the second affiliated server; adjust the first training quantity and all the training durations according to the training duration interval Describe the second training number.
  • the above-mentioned model training processing method provided by this application expands the processing of training tasks of a single server and a single thread into parallel processing of multiple servers, which greatly reduces model training time and improves model training efficiency;
  • the optimal training quantity allocates model training requests, so that the training time interval of each subsidiary server is smaller, which reduces the model training time, and the training time can be shortened to less than 30 minutes, which improves the model training efficiency;
  • this application modifies the target based on abnormal items
  • the concurrent query capability of the system is enhanced, and the reliability of the system is strengthened.
  • the present application can be applied to various functional modules of smart cities such as smart government affairs, smart transportation, etc., such as a model training processing module of smart government affairs, etc., which can promote the rapid development of smart cities.
  • FIG. 2 is a structural diagram of a model training processing apparatus provided in Embodiment 2 of the present application.
  • the model training processing device 20 may include a plurality of functional modules composed of computer program segments.
  • the computer program of each program segment in the model training processing device 20 can be stored in the memory of the computer device and executed by at least one processor to perform the function of model training processing (see description in FIG. 1 for details).
  • the model training processing apparatus 20 can be divided into a plurality of functional modules according to the functions performed by the model training processing apparatus 20 .
  • the functional modules may include: a request receiving module 201 , a quantity calculation module 202 , a message detection module 203 , a node determination module 204 , an abnormality detection module 205 and a quantity adjustment module 206 .
  • a module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, and are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.
  • the request receiving module 201 may be configured to acquire the affiliated server corresponding to the target server when a model training request is received.
  • the quantity calculation module 202 may be configured to calculate the optimal training quantity of the affiliated servers, and allocate the model training request to each of the affiliated servers based on the optimal training quantity.
  • the message detection module 203 can be used to detect whether the affiliate server outputs a training response message.
  • the node determination module 204 may be configured to determine the affiliate model training request allocated to the target affiliate server that has not output the training response message when the detection result is that the affiliate server does not output the training response message, and assign the affiliate model training request to the target affiliate server that does not output the training response message. Output to the target slave node corresponding to the target slave server.
  • the abnormality detection module 205 can be used to collect the model training results of each of the affiliated servers, and detect whether there are abnormal items in the model training results.
  • the quantity adjustment module 206 may be configured to modify the target training model based on the abnormal item when the detection result is that there is an abnormal item in the model training result, and dynamically adjust the training quantity of the affiliated server according to the model training result to renew the training quantity of the affiliated server. Train the target training model.
  • the computer device 3 includes a memory 31 , at least one processor 32 , at least one communication bus 33 and a transceiver 34 .
  • FIG. 3 does not constitute a limitation of the embodiments of the present application, and may be a bus-type structure or a star-shaped structure. more or less other hardware or software, or a different arrangement of components is shown.
  • the computer device 3 is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, Programmable gate arrays, digital processors and embedded devices, etc.
  • the computer equipment 3 may also include client equipment, including but not limited to any electronic product that can interact with the client through a keyboard, a mouse, a remote control, a touchpad or a voice-activated device, etc., for example, Personal computers, tablets, smartphones, digital cameras, etc.
  • a computer program is stored in the memory 31, and when the computer program is executed by the at least one processor 32, all or part of the steps in the model training processing method as described above are implemented.
  • the computer program may be divided into one or more modules/units, and the one or more modules/units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe The execution process of the computer program in the computer device.
  • each module described in FIG. 2 is a computer program stored in the memory 31 and executed by the at least one processor 32, thereby realizing the functions of the various modules to achieve information processing based on big data the goal of.
  • Described memory 31 comprises read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM) , One-time Programmable Read-Only Memory (OTPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read- Only Memory, CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.
  • Read-Only Memory Read-Only Memory
  • PROM Programmable Read-Only Memory
  • EPROM Erasable Programmable Read-Only Memory
  • OTPROM One-time Programmable Read-Only Memory
  • EEPROM Electronically-Erasable Programmable Read-Only Memory
  • CD-ROM Compact Disc Read- Only Memory
  • CD-ROM Compact Disc Read- Only Memory
  • the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function, and the like; The data created by the use of the node, etc.
  • the computer-readable storage medium may be non-volatile or volatile.
  • the blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.
  • Blockchain essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information to verify its Validity of information (anti-counterfeiting) and generation of the next block.
  • the blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.
  • the at least one processor 32 is a control core (Control Unit) of the computer device 3, using various interfaces and lines to connect various components of the entire computer device 3, and by running or executing storage in the computer device 3
  • the programs or modules in the memory 31 and the data stored in the memory 31 are called to perform various functions of the computer device 3 and process data.
  • the at least one processor 32 executes the computer program stored in the memory, it implements all or part of the steps of the model training processing method described in the embodiments of the present application; or implements all or part of the functions of the model training processing apparatus.
  • the at least one processor 32 may be composed of integrated circuits, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more central processing units. (Central Processing unit, CPU), microprocessor, digital processing chip, graphics processor and combination of various control chips, etc.
  • CPU Central Processing unit
  • microprocessor digital processing chip
  • graphics processor and combination of various control chips, etc.
  • the at least one communication bus 33 is configured to enable connection communication between the memory 31 and the at least one processor 32 and the like.
  • the computer device 3 may also include a power source (such as a battery) for supplying power to various components.
  • the power source may be logically connected to the at least one processor 32 through a power management device, so as to be implemented by the power management device Manage charging, discharging, and power management functions.
  • the power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and any other components.
  • the computer device 3 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.
  • the above-mentioned integrated units implemented in the form of software functional modules may be stored in a computer-readable storage medium.
  • the above-mentioned software function modules are stored in a storage medium, and include several instructions to make a computer device (which may be a personal computer, a computer device, or a network device, etc.) or a processor (processor) to execute the methods described in the various embodiments of the present application. part.
  • modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, and may be located in one place or distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Information Transfer Between Computers (AREA)
  • Debugging And Monitoring (AREA)

Abstract

本申请涉及数据处理技术，提供一种模型训练处理方法、装置、计算机设备与存储介质，包括：获取目标服务器对应的附属服务器；计算附属服务器的最优训练数量，并向每一附属服务器分配模型训练请求；检测附属服务器是否输出训练应答消息；当检测结果为否时，确定附属模型训练请求，并将附属模型训练请求输出至对应目标从属节点中；采集每一附属服务器的模型训练结果，并检测模型训练结果中是否存在异常项；当检测结果为是时，基于异常项修改目标训练模型，并根据模型训练结果动态调整附属服务器的训练数量以重新训练目标训练模型。本申请能够提高模型训练效率，促进智慧医疗及智慧城市的建设。

Description

模型训练处理方法、装置、计算机设备及介质
本申请要求于2020年12月24日提交中国专利局，申请号为202011553263.7发明名称为“模型训练处理方法、装置、计算机设备及介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及数据处理技术领域，尤其涉及一种模型训练处理方法、装置、计算机设备及介质。
背景技术
深度神经网络已成功应用于很多领域，包括图像识别、纹理分类、语音识别等领域。近年来，深度神经网络由于采用更深的网络架构和更大的训练样本集来进行训练，所以在性能上有了显著的提升。
针对深度神经网络训练的情况，在实现本申请的过程中，发明人发现现有技术至少存在如下问题：深度神经网络学习需要花费太高的时间成本，深度神经网络学习动辄就几个小时甚至花费几天时间来进行，一旦算法出现问题，就要重新训练深度神经网络学习模型，模型训练的效率极低。
因此，有必要提供一种模型训练处理方法，能够减少模型训练所需时间，提高模型训练效率。
发明内容
鉴于以上内容，有必要提出一种模型训练处理方法、模型训练处理装置、计算机设备及介质，能够减少模型训练所需时间，提高模型训练效率。
本申请实施例第一方面提供一种模型训练处理方法，所述模型训练处理方法包括：
当接收到模型训练请求时，获取目标服务器对应的附属服务器；
计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器分配所述模型训练请求；
检测所述附属服务器是否输出训练应答消息；
当检测结果为所述附属服务器未输出训练应答消息时，确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中；
采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项；
当检测结果为所述模型训练结果中存在异常项时，基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
本申请实施例第二方面提供的上述一种模型训练处理装置，所述模型训练处理装置包括：
请求接收模块，用于当接收到模型训练请求时，获取目标服务器对应的附属服务器；
数量计算模块，用于计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器分配所述模型训练请求；
消息检测模块，用于检测所述附属服务器是否输出训练应答消息；
节点确定模块，用于当检测结果为所述附属服务器未输出训练应答消息时，确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中；
异常检测模块，用于采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项；
数量调整模块，用于当检测结果为所述模型训练结果中存在异常项时，基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
本申请实施例第三方面还提供一种计算机设备，所述计算机设备包括处理器，所述处理器用于执行存储器中存储的计算机可读指令以实现以下步骤：
当接收到模型训练请求时，获取目标服务器对应的附属服务器；
计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器分配所述模型训练请求；
检测所述附属服务器是否输出训练应答消息；
当检测结果为所述附属服务器未输出训练应答消息时，确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中；
采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项；
当检测结果为所述模型训练结果中存在异常项时，基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
本申请实施例第四方面还提供一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机可读指令，所述计算机可读指令被处理器执行时实现以下步骤：
当接收到模型训练请求时，获取目标服务器对应的附属服务器；
计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器分配所述模型训练请求；
检测所述附属服务器是否输出训练应答消息；
当检测结果为所述附属服务器未输出训练应答消息时，确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中；
采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项；
当检测结果为所述模型训练结果中存在异常项时，基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
本申请能够减少模型训练所需时间，提高模型训练效率。
附图说明
图1是本申请实施例一提供的模型训练处理方法的流程图。
图2是本申请实施例二提供的模型训练处理装置的结构图。
图3是本申请实施例三提供的计算机设备的结构示意图。
如下具体实施方式将结合上述附图进一步说明本申请。
具体实施方式
为了能够更清楚地理解本申请的上述目的、特征和优点，下面结合附图和具体实施例对本申请进行详细描述。需要说明的是，在不冲突的情况下，本申请的实施例及实施例中的特征可以相互组合。
在下面的描述中阐述了很多具体细节以便于充分理解本申请，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
除非另有定义，本文所使用的所有的技术和科学术语与属于本申请的技术领域的技术人员通常理解的含义相同。本文中在本申请的说明书中所使用的术语只是为了描述具体的实施例的目的，不是旨在于限制本申请。
本申请实施例提供的模型训练处理方法由计算机设备执行，相应地，模型训练处理装置运行于计算机设备中。
图1是本申请实施例一提供的模型训练处理方法的流程图。如图1所示，所述模型训练处理方法可以包括如下步骤，根据不同的需求，该流程图中步骤的顺序可以改变，某些可以省略：
S11、当接收到模型训练请求时，获取目标服务器对应的附属服务器。
在本申请的至少一实施例中，所述模型训练请求是指用于请求训练目标训练模型的请求，所述模型训练请求是向目标应用发出的请求，用于开启所述目标训练模型的训练， 所述目标应用是指包含所述目标训练模型应用，所述应用在接收到模型训练请求时，将所述模型训练请求转发至目标服务器集群中，通过所述目标服务器集群中的多个服务器并行执行所述模型训练请求。所述目标服务器集群包括目标服务器以及对应所述目标服务器的附属服务器。所述目标服务器是指接收所述目标应用转发模型训练请求的服务器，所述附属服务器是指与所述目标服务器关联的服务器。可选地，所述当接收到模型训练请求时，获取目标服务器对应的附属服务器包括：解析所述模型训练请求，得到对应所述模型训练请求的目标路由信息；根据所述目标路由信息确定目标应用；获取对应所述目标应用的目标服务器集群；获取所述目标服务器集群中的目标服务器与附属服务器。
其中，每一所述模型训练请求均包含目标路由信息，所述目标路由信息用于指示对应目标训练模型的目标应用的路由信息，根据所述目标路由信息能够得到对应所述模型训练请求的目标应用。所述目标应用与目标服务器集群相关联，根据目标应用能够得到对应的目标服务器集群。所述目标服务器集群是指包含一个目标服务器以及多个附属服务器的集群，所述附属服务器的数量可以为1个，也可以为多个，在此不作限制。
在本申请的至少一实施例中，所述附属服务器的数量可以根据所述模型训练请求中携带的训练数量确定，所述训练数量越大，其对应的所述附属服务器的数量也越大。可选地，所述获取目标服务器对应的附属服务器还包括：解析所述模型训练请求，得到对应目标训练模型的训练数量；确定所述训练数量对应的训练数量级；根据所述训练数量级确定附属服务器的数量。
其中，所述模型训练请求不仅包含目标路由信息，还包括对应目标训练模型的训练数量等信息，在此不作限制。所述训练数量是指训练数据的数量，所述训练数量可以根据目标训练模型的实际情况进行设定。在一实施例中，所述训练数量与所述训练数量级间存在预先设置的第一映射关系，所述训练数量级与所述附属服务器的数量间存在预先设置的第二映射关系，通过所述训练数量遍历所述第一映射关系，能够确定对应所述训练数量的训练数量级，所述训练数量级可以为低级、中级与高级。通过所述训练数量级遍历所述第二映射关系，能够确定对应所述训练数量级的附属服务器的数量，避免调用过多或过少的附属服务器造成的资源浪费或模型训练时间长的问题。
S12、计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器分配所述模型训练请求。
在本申请的至少一实施例中，不同附属服务器的数据处理能力可能相同，也可能不相同。当每个附属服务器的数据处理能力均相同时，可以将所述模拟训练请求携带的训练数量进行等量拆分，得到附属模型训练请求，并将所述附属模型训练请求分配至所述附属服务器。当不同附属服务器的数据处理能力不相同时，由于各个服务器的数据处理能力相同，导致相同或者相近的时间段内不同附属服务器处理训练数据的数量不同，本申请根据附属服务器的不同数据处理能力，为每一所述附属服务器分配最优训练数量， 使得数据处理能力高的附属服务器处理较多的训练数据，而数据处理能力少的附属服务器处理较少的训练数据，从而附属服务器均能够在相同或相近的时间段处理完各自的训练数量，减少模型训练整体所需的时间，提高模型训练的效率。
其中，所述附属服务器的数据处理能力可根据服务器的负载等级确定，所述负载等级越高，其对应的数据处理能力越高，所述负载等级与所述数据处理能力间存在映射关系，通过查询该映射关系，能够得到所述负载等级对应的数据处理能力。所述负载等级可通过服务器的CPU数据、内存数据与网络负载数据这3个指标确定，所述CPU数据可以是指CPU负载参数，所述内存数据可以是指内存占用参数，所述网络负载数据可以包括流量参数，在此不做限制。在一实施例中，通过获取上述3个指标，并将其输入至预先训练的负载等级确定模型中进行处理，得到对应服务器的负载等级。
可选地，所述计算所述附属服务器的最优训练数量包括：获取对应每一附属服务器的数据处理能力；获取所述模型训练请求携带的对应目标训练模型的训练数量；根据所述数据处理能力与所述训练数量计算每一所述附属服务器的最优训练数量。其中，所述模型训练请求中携带的训练数量可以是指训练数据的总量，所述数据处理能力可以是指单位时间内服务器处理模型训练数据的数量，通过配置相同或相近的训练时间，能够得到分配至各个服务器的最优训练数量。
在本申请的至少一实施例中，当每一所述附属服务器的最优训练数量确定后，开始对所述模型训练请求进行分配处理，将所述模型训练请求拆分为多个附属模型训练请求，每一所述附属模型训练请求与所述附属服务器相对应，用于请求调用所述附属服务器对训练数据进行训练处理。可选地，所述基于所述最优训练数量分配所述模型训练请求包括：获取所述附属服务器的数量与每一所述附属服务器的最优训练数量；基于所述附属服务器的数量与所述最优训练数量拆分所述模型训练请求，得到若干个附属模型训练请求；将所述附属模型训练请求分配至对应的附属服务器。
其中，每一所述附属模型训练请求中均携带有该附属服务器的标识以及该附属服务器应训练的训练数量，通过解析所述附属模型训练请求即可将所述附属模型训练请求分配至对应的附属服务器中。
S13、检测所述附属服务器是否输出训练应答消息，当检测结果为所述附属服务器未输出训练应答消息时，执行步骤S14。
在本申请的至少一实施例中，每一所述附属服务器在开始训练前，均会向目标服务器输出训练应答消息，通过检查所述附属服务器是否输出训练应答消息，能够确定所述附属服务器是否正常训练模型数据。当检测结果为所述附属服务器输出训练应答消息时，确定所述附属服务器训练进程正常；当检测结果为所述附属服务器未输出训练应答消息时，确定所述附属服务器训练进程异常。
可选地，所述检测所述附属服务器是否输出训练应答消息包括：获取对应各个所述 附属服务器的运行日志；检测所述运行日志中是否存在每一所述附属服务器对应的目标应答数据；当检测结果为所述运行日志中存在每一所述附属服务器对应的目标应答数据时，确定所述附属服务器输出训练应答消息；当检测结果为所述运行日志中不存在所述附属服务器对应的目标应答数据时，确定所述附属服务器未输出训练应答消息。
其中，各个所述附属服务器的运行日志可以存储于区块链的目标节点中，各个附属服务器将运行过程中产生的训练应答消息、运行异常消息以及运行结束消息等数据存储至所述目标节点中，通过查询所述运行日志，能够得到各个附属服务器的运行进程。此外，所述运行日志内还可以记录有各个服务器的地址信息、端口信息、从属节点信息、从属节点的端口信息(也即从端口信息)、节点类型(所述节点类型可以包括主任务与附属任务)以及附属模型训练请求信息等，在此不做限制。
在本申请的至少一实施例中，当检测结果为所述附属服务器均输出训练应答消息时，则执行步骤S15，用于采集每一所述附属服务器的模型训练结果。
S14、确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中。
在本申请的至少一实施例中，每一所述附属服务器均设有从属节点，所述从属节点可配置与所述附属服务器相对应的模型训练处理逻辑，当所述目标附属服务器未输出训练应答消息时，说明此时所述目标附属服务器运行进程存在异常。本申请通过调用运行进程存在异常的目标附属服务器对应的目标从属节点执行相应的模型训练请求，保证模型训练请求能够正常执行，节省模型训练所需的时间，提高模型训练的效率。
可选地，所述确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中包括：获取所述目标附属服务器的从属节点集；从所述从属节点集中确定目标从属节点；将所述附属模型训练请求输出至所述目标从属节点，由所述目标从属节点执行模型训练。
其中，所述目标附属服务器对应的从属节点的数量可以为1个，也可以为多个。当所述目标附属服务器对应的从属节点的数量为多个时，所述方法还包括：获取所述目标附属服务器的从属节点集；确定所述从属节点集中每一从属节点对应的执行逻辑；检测是否存在与目标附属服务器的执行逻辑一致的目标执行逻辑；当检测结果为存在与所述目标附属服务器的执行逻辑一致的目标执行逻辑时，确定对应所述目标执行逻辑的从属节点为目标从属节点。
可选地，所述从属节点大多用于提供查询问答匹配逻辑，可配置一个从属节点用于配置任务处理逻辑，用于在附属服务器无法正常执行模型训练任务时，调用该附属服务器对应的从属节点执行模型训练任务。在一实施例中，针对每一附属服务器，均预先配置目标从属节点，用于执行任务处理逻辑。在其他实施例中，无需预先配置从属节点的任务处理逻辑，当检测到附属服务器的任务执行进程异常时，获取对应所述附属服务器 的目标执行逻辑信息，将所述目标执行逻辑信息输出至所述从属节点集中的任一从属节点中，并根据所述目标执行逻辑信息配置该从属节点，使得该从属节点具有任务处理逻辑。本申请在检测到附属服务器的任务执行异常时，根据目标执行逻辑信息实时配置从属节点，避免预先配置导致的资源浪费的情况，提高开发效率。
其中，所述从属节点预先设有相关配置信息，用于执行对应查询问答匹配逻辑，所述根据所述目标执行逻辑信息配置该从属节点也即将所述目标执行逻辑信息存储至对应从属节点的配置信息中，并调用所述目标执行逻辑信息使得所述从属节点具备任务处理能力。
S15、采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项，当检测结果为所述模型训练结果中存在异常项时，执行步骤S16。
在本申请的至少一实施例中，每一所述附属服务器在模型训练过程中，均会向所述区块链的目标节点的运行日志中输出相关训练数据，当每一所述附属服务器的模型训练结束后，会向所述目标节点的运行日志中输出模型训练结果。本申请除了通过查询所述运行日志得到每一所述附属服务器的模型训练结果外，所述方法还包括：获取所述目标服务器与所述附属服务器的关联关系；根据所述关联关系确定可视化的模型训练逻辑图，其中，每一服务器对应所述模型训练逻辑图中的目标按钮；在所述可视化的模型训练逻辑图中，获取每一所述附属服务器的模型训练进度与训练速率；根据所述训练进度与所述训练速率确定每一所述目标按钮的实化显示面积与实化显示速率。
其中，所述模型训练逻辑图对应各个服务器的目标按钮，每一按钮均对应一附属服务器，所述目标按钮与所述附属服务器间可通过预设标识进行关联，通过查看所述预设标识，即可确定每一所述目标按钮关联的目服务器，所述预设标识可以为数字标识、字母标识与颜色标识，在此不做限制。本申请通过对每一所述目标按钮进行虚化与实化显示，能够直观地显示每一所述附属服务器的训练进度与训练速率。当所述目标按钮的实化显示面积为整个目标按钮面积时，确定所述服务器训练完成；所述目标按钮的实化显示速率越快，其对应服务器的训练速率越快。
在本申请的至少一实施例中，在可视化的模型逻辑训练图中，除了可以通过虚化显示的方式直观地表示附属服务器的模型训练进度与模型训练速率，所述方法还包括：检测所述附属服务器的运行过程中是否出现训练异常项；当检测结果为所述附属服务器的运行过程中出现训练异常项时，获取所述异常项的数量；根据所述异常项的数量在可视化的模型训练逻辑图的目标按钮中设置预设标识。
其中，所述异常项的数量不同，所述预设标识也不相同，例如，所述预设标识为颜色标识时，当所述异常项的数量越多时，所述预设标识的越接近红色，在此不作限制。通过在可视化的模型训练逻辑图中显示异常项的预设标识，能够直观地看出附属服务器在模型训练过程中是否出现异常项以及出现的异常项的数量，并及时对异常情况进行查 看。
S16、基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
在本申请的至少一实施例中，当检测结果为所述模型训练结果中存在异常项时，开发人员可基于所述异常项修改目标训练模型，以使得修改好的目标训练模型能够解决上述异常项。在实际训练过程中，当开发人员根据所述异常项修改好目标训练模型后，还需将所述目标训练模型按照训练数据重新进行训练，以确保目标训练模型在正式使用时无其他异常项。所述模型训练结果中还包含各个附属服务器的训练时长，本申请根据所述训练时长动态微调整所述附属服务器的训练数据数量，进一步节省模型训练的时间，提高模型训练效率。
可选地，所述根据所述模型训练结果动态调整所述附属服务器的训练数量包括：获取并解析每一所述附属服务器的模型训练结果，得到对应每一所述附属服务器的实际模型训练时长；检测所述实际模型训练时长是否满足目标调整条件；当检测结果为所述实际模型训练时长满足目标调整条件时，确定满足目标调整条件的所述目标附属服务器；动态调整所述目标附属服务器的训练数量。
其中，所述目标调整条件是指所述实际模型训练时长中存在异常时长，举例来说，存在5个附属服务器，分别为附属服务器1-5，其中，附属服务器1-4彼此间的实际模型训练时长相差时间间隔均在预设时间间隔内，而附属服务器5的实际模型训练时长与其余4个附属服务器中一个或多个的实际模型训练时长的时间间隔超过预设时间间隔，则附属服务器5的实际模型训练时长满足目标调整条件，需要对所述附属服务器5的训练数量进行动态调整，以使得所述附属服务器5的训练时长与其他几个附属服务器的训练时长的时间间隔在预设时间间隔内，进一步从整体上减少模型训练所需时间，提高模型训练的效率。
可选地，所述动态调整所述附属服务器的训练数量包括：确定实际模型训练时长最长的第一附属服务器与实际模型训练时长最短的第二附属服务器；计算所述第一附属服务器与所述第二附属服务器的训练时长间隔；获取所述第一附属服务器的第一训练数量以及所述第二附属服务器的第二训练数量；根据所述训练时长间隔调整所述第一训练数量与所述第二训练数量。
本申请提供的上述模型训练处理方法，将单台服务器单线程的训练任务处理扩展为多台服务器并行处理，使得模型训练时间大幅降低，提高模型训练效率；且本申请针对每台附属服务器的最优训练数量分配模型训练请求，使得每台附属服务器的训练时间间隔差距较小，使得模型训练时间降低，训练时间能够缩短至30分钟以内，提高模型训练效率；此外，本申请基于异常项修改目标训练模型后，根据此前的模型训练结果动态调整附属服务器的训练数量，进一步优化模型训练过程，提高模型训练效率；此外，本申 请通过对集群从属节点的配置，使得从属节点支持逻辑匹配查询，增加了系统的并发查询能力，加强了系统的可靠性。本申请可应用于智慧政务、智慧交通等智慧城市的各个功能模块中，比如智慧政务的模型训练处理模块等，能够促进智慧城市的快速发展。
图2是本申请实施例二提供的模型训练处理装置的结构图。
在一些实施例中，所述模型训练处理装置20可以包括多个由计算机程序段所组成的功能模块。所述模型训练处理装置20中的各个程序段的计算机程序可以存储于计算机设备的存储器中，并由至少一个处理器所执行，以执行(详见图1描述)模型训练处理的功能。
本实施例中，所述模型训练处理装置20根据其所执行的功能，可以被划分为多个功能模块。所述功能模块可以包括：请求接收模块201、数量计算模块202、消息检测模块203、节点确定模块204、异常检测模块205以及数量调整模块206。本申请所称的模块是指一种能够被至少一个处理器所执行并且能够完成固定功能的一系列计算机程序段，其存储在存储器中。在本实施例中，关于各模块的功能将在后续的实施例中详述。
请求接收模块201可以用于当接收到模型训练请求时，获取目标服务器对应的附属服务器。
数量计算模块202可以用于计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器分配所述模型训练请求。
消息检测模块203可以用于检测所述附属服务器是否输出训练应答消息。
节点确定模块204可以用于用于当检测结果为所述附属服务器未输出训练应答消息时，确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中。
异常检测模块205可以用于采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项。
数量调整模块206可以用于当检测结果为所述模型训练结果中存在异常项时，基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
参阅图3所示，为本申请实施例三提供的计算机设备的结构示意图。在本申请较佳实施例中，所述计算机设备3包括存储器31、至少一个处理器32、至少一条通信总线33及收发器34。
本领域技术人员应该了解，图3示出的计算机设备的结构并不构成本申请实施例的限定，既可以是总线型结构，也可以是星形结构，所述计算机设备3还可以包括比图示更多或更少的其他硬件或者软件，或者不同的部件布置。
在一些实施例中，所述计算机设备3是一种能够按照事先设定或存储的指令，自动进行数值计算和/或信息处理的设备，其硬件包括但不限于微处理器、专用集成电路、可编程门阵列、数字处理器及嵌入式设备等。所述计算机设备3还可包括客户设备，所述客户设备包括 但不限于任何一种可与客户通过键盘、鼠标、遥控器、触摸板或声控设备等方式进行人机交互的电子产品，例如，个人计算机、平板电脑、智能手机、数码相机等。
需要说明的是，所述计算机设备3仅为举例，其他现有的或今后可能出现的电子产品如可适应于本申请，也应包含在本申请的保护范围以内，并以引用方式包含于此。
在一些实施例中，所述存储器31中存储有计算机程序，所述计算机程序被所述至少一个处理器32执行时实现如所述的模型训练处理方法中的全部或者部分步骤。示例性的，所述计算机程序可以被分割成一个或多个模块/单元，所述一个或多个模块/单元可以是能够完成特定功能的一系列计算机可读指令段，该指令段用于描述所述计算机程序在所述计算机设备中的执行过程。例如，图2中所述的各个模块是存储在所述存储器31中的计算机程序，并由所述至少一个处理器32所执行，从而实现所述各个模块的功能以达到基于大数据的信息处理的目的。所述存储器31包括只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable Read-Only Memory，PROM)、可擦除可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM)、一次可编程只读存储器(One-time Programmable Read-Only Memory，OTPROM)、电子擦除式可复写只读存储器(Electrically-Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact Disc Read-Only Memory，CD-ROM)或其他光盘存储器、磁盘存储器、磁带存储器、或者能够用于携带或存储数据的计算机可读的任何其他介质。
进一步地，所述计算机可读存储介质可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序等；存储数据区可存储根据区块链节点的使用所创建的数据等。所述计算机可读存储介质可以是非易失性，也可以是易失性。
本申请所指区块链是分布式数据存储、点对点传输、共识机制、加密算法等计算机技术的新型应用模式。区块链(Blockchain)，本质上是一个去中心化的数据库，是一串使用密码学方法相关联产生的数据块，每一个数据块中包含了一批次网络交易的信息，用于验证其信息的有效性(防伪)和生成下一个区块。区块链可以包括区块链底层平台、平台产品服务层以及应用服务层等。
在一些实施例中，所述至少一个处理器32是所述计算机设备3的控制核心(Control Unit)，利用各种接口和线路连接整个计算机设备3的各个部件，通过运行或执行存储在所述存储器31内的程序或者模块，以及调用存储在所述存储器31内的数据，以执行计算机设备3的各种功能和处理数据。例如，所述至少一个处理器32执行所述存储器中存储的计算机程序时实现本申请实施例中所述的模型训练处理方法的全部或者部分步骤；或者实现模型训练处理装置的全部或者部分功能。所述至少一个处理器32可以由集成电路组成，例如可以由单个封装的集成电路所组成，也可以是由多个相同功能或不同功能封装的集成电路所组成，包括一个或者多个中央处理器(Central Processing unit，CPU)、微处理器、数字处理芯片、图形处理器及各种控制芯片的组合等。
在一些实施例中，所述至少一条通信总线33被设置为实现所述存储器31以及所述至少一个处理器32等之间的连接通信。
尽管未示出，所述计算机设备3还可以包括给各个部件供电的电源(比如电池)，优选的，电源可以通过电源管理装置与所述至少一个处理器32逻辑相连，从而通过电源管理装置实现管理充电、放电、以及功耗管理等功能。电源还可以包括一个或一个以上的直流或交流电源、再充电装置、电源故障检测电路、电源转换器或者逆变器、电源状态指示器等任意组件。所述计算机设备3还可以包括多种传感器、蓝牙模块、Wi-Fi模块等，在此不再赘述。
上述以软件功能模块的形式实现的集成的单元，可以存储在一个计算机可读取存储介质中。上述软件功能模块存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，计算机设备，或者网络设备等)或处理器(processor)执行本申请各个实施例所述方法的部分。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
所述作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的部件可以是或者也可以不是物理单元，既可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能模块可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能模块的形式实现。
对于本领域技术人员而言，显然本申请不限于上述示范性实施例的细节，而且在不背离本申请的精神或基本特征的情况下，能够以其他的具体形式实现本申请。因此，无论从哪一点来看，均应将实施例看作是示范性的，而且是非限制性的，本申请的范围由所附权利要求而不是上述说明限定，因此旨在将落在权利要求的等同要件的含义和范围内的所有变化涵括在本申请内。不应将权利要求中的任何附图标记视为限制所涉及的权利要求。此外，显然“包括”一词不排除其他单元或，单数不排除复数。说明书中陈述的多个单元或装置也可以由一个单元或装置通过软件或者硬件来实现。第一，第二等词语用来表示名称，而并不表示任何特定的顺序。
最后应说明的是，以上实施例仅用以说明本申请的技术方案而非限制，尽管参照较佳实施例对本申请进行了详细说明，本领域的普通技术人员应当理解，可以对本申请的技术方案进行修改或等同替换，而不脱离本申请技术方案的精神和范围。

Claims (20)

  1. 一种模型训练处理方法，其中，所述模型训练处理方法包括：
    当接收到模型训练请求时，获取目标服务器对应的附属服务器；
    计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器分配所述模型训练请求；
    检测所述附属服务器是否输出训练应答消息；
    当检测结果为所述附属服务器未输出训练应答消息时，确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中；
    采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项；
    当检测结果为所述模型训练结果中存在异常项时，基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
  2. 根据权利要求1所述的模型训练处理方法，其中，所述当接收到模型训练请求时，获取目标服务器对应的附属服务器包括：
    解析所述模型训练请求，得到对应所述模型训练请求的目标路由信息；
    根据所述目标路由信息确定目标应用；
    获取对应所述目标应用的目标服务器集群；
    获取所述目标服务器集群中的目标服务器与附属服务器。
  3. 根据权利要求1所述的模型训练处理方法，其中，所述获取目标服务器对应的附属服务器还包括：
    解析所述模型训练请求，得到对应目标训练模型的训练数量；
    确定所述训练数量对应的训练数量级；
    根据所述训练数量级确定附属服务器的数量。
  4. 根据权利要求1所述的模型训练处理方法，其中，所述计算所述附属服务器的最优训练数量包括：
    获取对应每一附属服务器的数据处理能力；
    获取所述模型训练请求携带的对应目标训练模型的训练数量；
    根据所述数据处理能力与所述训练数量计算每一所述附属服务器的最优训练数量。
  5. 根据权利要求1所述的模型训练处理方法，其中，所述基于所述最优训练数量分配所述模型训练请求包括：
    获取所述附属服务器的数量与每一所述附属服务器的最优训练数量；
    基于所述附属服务器的数量与所述最优训练数量拆分所述模型训练请求，得到若干个附属模型训练请求；
    将所述附属模型训练请求分配至对应的附属服务器。
  6. 根据权利要求1所述的模型训练处理方法，其中，所述检测所述附属服务器是否输出训练应答消息包括：
    获取对应各个所述附属服务器的运行日志；
    检测所述运行日志中是否存在每一所述附属服务器对应的目标应答数据；
    当检测结果为所述运行日志中存在每一所述附属服务器对应的目标应答数据时，确定所述附属服务器输出训练应答消息；
    当检测结果为所述运行日志中不存在所述附属服务器对应的目标应答数据时，确定所述附属服务器未输出训练应答消息。
  7. 根据权利要求1所述的模型训练处理方法，其中，所述方法还包括：
    获取所述目标服务器与所述附属服务器的关联关系；
    根据所述关联关系确定可视化的模型训练逻辑图，其中，每一服务器对应所述模型训练逻辑图中的目标按钮；
    在所述可视化的模型训练逻辑图中，获取每一所述附属服务器的模型训练进度与训练速率；
    根据所述训练进度与所述训练速率确定每一所述目标按钮的实化显示面积与实化显示速率。
  8. 一种模型训练处理装置，其中，所述模型训练处理装置包括：
    请求接收模块，用于当接收到模型训练请求时，获取目标服务器对应的附属服务器；
    数量计算模块，用于计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器分配所述模型训练请求；
    消息检测模块，用于检测所述附属服务器是否输出训练应答消息；
    节点确定模块，用于当检测结果为所述附属服务器未输出训练应答消息时，确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中；
    异常检测模块，用于采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项；
    数量调整模块，用于当检测结果为所述模型训练结果中存在异常项时，基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
  9. 一种计算机设备，其中，所述计算机设备包括处理器，所述处理器用于执行存储器中存储的计算机可读指令以实现以下步骤：
    当接收到模型训练请求时，获取目标服务器对应的附属服务器；
    计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器 分配所述模型训练请求；
    检测所述附属服务器是否输出训练应答消息；
    当检测结果为所述附属服务器未输出训练应答消息时，确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中；
    采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项；
    当检测结果为所述模型训练结果中存在异常项时，基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
  10. 根据权利要求9所述的计算机设备，其中，所述处理器执行所述计算机可读指令以实现所述当接收到模型训练请求时，获取目标服务器对应的附属服务器时，包括：
    解析所述模型训练请求，得到对应所述模型训练请求的目标路由信息；
    根据所述目标路由信息确定目标应用；
    获取对应所述目标应用的目标服务器集群；
    获取所述目标服务器集群中的目标服务器与附属服务器。
  11. 根据权利要求9所述的计算机设备，其中，所述处理器执行所述计算机可读指令以实现所述获取目标服务器对应的附属服务器时，还包括：
    解析所述模型训练请求，得到对应目标训练模型的训练数量；
    确定所述训练数量对应的训练数量级；
    根据所述训练数量级确定附属服务器的数量。
  12. 根据权利要求9所述的计算机设备，其中，所述处理器执行所述计算机可读指令以实现所述计算所述附属服务器的最优训练数量时，包括：
    获取对应每一附属服务器的数据处理能力；
    获取所述模型训练请求携带的对应目标训练模型的训练数量；
    根据所述数据处理能力与所述训练数量计算每一所述附属服务器的最优训练数量。
  13. 根据权利要求9所述的计算机设备，其中，所述处理器执行所述计算机可读指令以实现所述基于所述最优训练数量分配所述模型训练请求时，包括：
    获取所述附属服务器的数量与每一所述附属服务器的最优训练数量；
    基于所述附属服务器的数量与所述最优训练数量拆分所述模型训练请求，得到若干个附属模型训练请求；
    将所述附属模型训练请求分配至对应的附属服务器。
  14. 根据权利要求9所述的计算机设备，其中，所述处理器执行所述计算机可读指令以实现所述检测所述附属服务器是否输出训练应答消息时，包括：
    获取对应各个所述附属服务器的运行日志；
    检测所述运行日志中是否存在每一所述附属服务器对应的目标应答数据；
    当检测结果为所述运行日志中存在每一所述附属服务器对应的目标应答数据时，确定所述附属服务器输出训练应答消息；
    当检测结果为所述运行日志中不存在所述附属服务器对应的目标应答数据时，确定所述附属服务器未输出训练应答消息。
  15. 根据权利要求9所述的计算机设备，其中，所述处理器执行所述计算机可读指令还用以实现以下步骤：
    获取所述目标服务器与所述附属服务器的关联关系；
    根据所述关联关系确定可视化的模型训练逻辑图，其中，每一服务器对应所述模型训练逻辑图中的目标按钮；
    在所述可视化的模型训练逻辑图中，获取每一所述附属服务器的模型训练进度与训练速率；
    根据所述训练进度与所述训练速率确定每一所述目标按钮的实化显示面积与实化显示速率。
  16. 一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机可读指令，其中，所述计算机可读指令被处理器执行时实现以下步骤：
    当接收到模型训练请求时，获取目标服务器对应的附属服务器；
    计算所述附属服务器的最优训练数量，并基于所述最优训练数量向每一所述附属服务器分配所述模型训练请求；
    检测所述附属服务器是否输出训练应答消息；
    当检测结果为所述附属服务器未输出训练应答消息时，确定分配至未输出训练应答消息的目标附属服务器的附属模型训练请求，并将所述附属模型训练请求输出至对应所述目标附属服务器的目标从属节点中；
    采集每一所述附属服务器的模型训练结果，并检测所述模型训练结果中是否存在异常项；
    当检测结果为所述模型训练结果中存在异常项时，基于所述异常项修改目标训练模型，并根据所述模型训练结果动态调整所述附属服务器的训练数量以重新训练所述目标训练模型。
  17. 根据权利要求16所述的计算机可读存储介质，其中，所述计算机可读指令被处理器执行以实现所述当接收到模型训练请求时，获取目标服务器对应的附属服务器时，包括：
    解析所述模型训练请求，得到对应所述模型训练请求的目标路由信息；
    根据所述目标路由信息确定目标应用；
    获取对应所述目标应用的目标服务器集群；
    获取所述目标服务器集群中的目标服务器与附属服务器。
  18. 根据权利要求16所述的计算机可读存储介质，其中，所述计算机可读指令被处理器 执行以实现所述获取目标服务器对应的附属服务器时，还包括：
    解析所述模型训练请求，得到对应目标训练模型的训练数量；
    确定所述训练数量对应的训练数量级；
    根据所述训练数量级确定附属服务器的数量。
  19. 根据权利要求16所述的计算机可读存储介质，其中，所述计算机可读指令被处理器执行以实现所述计算所述附属服务器的最优训练数量时，包括：
    获取对应每一附属服务器的数据处理能力；
    获取所述模型训练请求携带的对应目标训练模型的训练数量；
    根据所述数据处理能力与所述训练数量计算每一所述附属服务器的最优训练数量。
  20. 根据权利要求16所述的计算机可读存储介质，其中，所述计算机可读指令被处理器执行以实现所述基于所述最优训练数量分配所述模型训练请求时，包括：
    获取所述附属服务器的数量与每一所述附属服务器的最优训练数量；
    基于所述附属服务器的数量与所述最优训练数量拆分所述模型训练请求，得到若干个附属模型训练请求；
    将所述附属模型训练请求分配至对应的附属服务器。
PCT/CN2021/125440 2020-12-24 2021-10-21 模型训练处理方法、装置、计算机设备及介质 Ceased WO2022134809A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011553263.7A CN112598135A (zh) 2020-12-24 2020-12-24 模型训练处理方法、装置、计算机设备及介质
CN202011553263.7 2020-12-24

Publications (1)

Publication Number Publication Date
WO2022134809A1 true WO2022134809A1 (zh) 2022-06-30

Family

ID=75202408

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/125440 Ceased WO2022134809A1 (zh) 2020-12-24 2021-10-21 模型训练处理方法、装置、计算机设备及介质

Country Status (2)

Country Link
CN (1) CN112598135A (zh)
WO (1) WO2022134809A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116382599A (zh) * 2023-06-07 2023-07-04 之江实验室 一种面向分布式集群的任务执行方法、装置、介质及设备
CN118093167A (zh) * 2024-01-31 2024-05-28 三六零数字安全科技集团有限公司 模型训练方法、装置、电子设备及存储介质

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112598135A (zh) * 2020-12-24 2021-04-02 深圳壹账通智能科技有限公司 模型训练处理方法、装置、计算机设备及介质
CN117407493A (zh) * 2023-09-01 2024-01-16 阿里巴巴达摩院(杭州)科技有限公司 数据处理方法、系统、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10152676B1 (en) * 2013-11-22 2018-12-11 Amazon Technologies, Inc. Distributed training of models using stochastic gradient descent
CN109543726A (zh) * 2018-11-06 2019-03-29 联动优势科技有限公司 一种训练模型的方法及装置
CN111753997A (zh) * 2020-06-28 2020-10-09 北京百度网讯科技有限公司 分布式训练方法、系统、设备及存储介质
CN111917579A (zh) * 2020-07-30 2020-11-10 云知声智能科技股份有限公司 分布式训练方法、装置、设备和存储介质
CN112598135A (zh) * 2020-12-24 2021-04-02 深圳壹账通智能科技有限公司 模型训练处理方法、装置、计算机设备及介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107688493B (zh) * 2016-08-05 2021-06-18 阿里巴巴集团控股有限公司 训练深度神经网络的方法、装置及系统
CN108734293B (zh) * 2017-04-13 2023-05-02 北京京东尚科信息技术有限公司 任务管理系统、方法和装置
US11275991B2 (en) * 2018-04-04 2022-03-15 Nokia Technologies Oy Coordinated heterogeneous processing of training data for deep neural networks
CN108986792B (zh) * 2018-09-11 2021-02-12 苏州思必驰信息科技有限公司 用于语音对话平台的语音识别模型的训练调度方法及系统
CN111445027B (zh) * 2019-01-16 2024-01-16 顺丰科技有限公司 机器学习模型的训练方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10152676B1 (en) * 2013-11-22 2018-12-11 Amazon Technologies, Inc. Distributed training of models using stochastic gradient descent
CN109543726A (zh) * 2018-11-06 2019-03-29 联动优势科技有限公司 一种训练模型的方法及装置
CN111753997A (zh) * 2020-06-28 2020-10-09 北京百度网讯科技有限公司 分布式训练方法、系统、设备及存储介质
CN111917579A (zh) * 2020-07-30 2020-11-10 云知声智能科技股份有限公司 分布式训练方法、装置、设备和存储介质
CN112598135A (zh) * 2020-12-24 2021-04-02 深圳壹账通智能科技有限公司 模型训练处理方法、装置、计算机设备及介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116382599A (zh) * 2023-06-07 2023-07-04 之江实验室 一种面向分布式集群的任务执行方法、装置、介质及设备
CN116382599B (zh) * 2023-06-07 2023-08-29 之江实验室 一种面向分布式集群的任务执行方法、装置、介质及设备
CN118093167A (zh) * 2024-01-31 2024-05-28 三六零数字安全科技集团有限公司 模型训练方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN112598135A (zh) 2021-04-02

Similar Documents

Publication Publication Date Title
WO2022134809A1 (zh) 模型训练处理方法、装置、计算机设备及介质
WO2022142016A1 (zh) 基于规则引擎的参数转换方法、装置、设备及介质
US11770456B1 (en) System and method for distributed management of storage systems based on subscription changes
US11831706B1 (en) System and method for distributed management of storage systems based on intent
US20160142475A1 (en) Shard management service
CN113361913A (zh) 一种通信业务编排方法、装置、计算机设备及存储介质
CN114221908A (zh) 动态限流熔断处理方法、装置、计算机设备及存储介质
CN113159464A (zh) 一种数据处理方法、装置和服务器
CN110162407A (zh) 一种资源管理方法及装置
CN112948723A (zh) 接口调用方法、装置及相关设备
WO2023226205A1 (zh) 基于kubernetes的GPU共享方法、装置、设备及介质
CN111327447B (zh) 一种分布式系统、仲裁方法、节点设备及存储介质
CN114124968A (zh) 基于行情数据的负载均衡方法、装置、设备及介质
US12309162B2 (en) System and method for distributed management of hardware with intermittent connectivity
WO2021212636A1 (zh) 表单页面生成方法、装置、终端及存储介质
CN113886086B (zh) 云平台计算资源分配方法、系统、终端及存储介质
CN113760940B (zh) 应用于分布式系统的配额管理方法、装置、设备及介质
CN114036218A (zh) 一种数据模型切换方法、装置、服务器和存储介质
CN115480911A (zh) 一种集群的资源分配方法、服务器和存储介质
US12124351B2 (en) System and method for distributed management of hardware based on performance validation
US11929891B1 (en) System and method for distributed management of hardware through relationship management
CN112181599A (zh) 模型训练方法、装置及存储介质
CN116723191B (zh) 利用加速装置执行数据流加速计算的方法和系统
CN112698945A (zh) 基于多模型的资源分配方法、装置、电子设备及存储介质
CN116095007A (zh) 负载调度方法、装置、计算机设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21908810

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 31.10.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21908810

Country of ref document: EP

Kind code of ref document: A1