WO2025001026A1 - 一种用于混合专家模型的并行训练方法及装置 - Google Patents

一种用于混合专家模型的并行训练方法及装置 Download PDF

Info

Publication number
WO2025001026A1
WO2025001026A1 PCT/CN2023/142843 CN2023142843W WO2025001026A1 WO 2025001026 A1 WO2025001026 A1 WO 2025001026A1 CN 2023142843 W CN2023142843 W CN 2023142843W WO 2025001026 A1 WO2025001026 A1 WO 2025001026A1
Authority
WO
WIPO (PCT)
Prior art keywords
parallel
expert
strategy
training
hybrid
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/142843
Other languages
English (en)
French (fr)
Inventor
翟季冬
翟明书
何家傲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Publication of WO2025001026A1 publication Critical patent/WO2025001026A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
    • Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
    • Y04S10/00—Systems supporting electrical power generation, transmission or distribution
    • Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications

Definitions

  • the present application relates to the field of deep learning technology, and in particular, to a parallel training method and device for a hybrid expert model.
  • the Mixture of Experts (MoE) model is a neural network, but unlike general neural networks, it can train multiple models based on data separation. Each model can be called an expert network. That is, the idea of the Mixture of Experts model is to train multiple expert networks, each of which is applied to different parts of the data set. As an emerging sparse activation deep learning model architecture, the Mixture of Experts model can achieve ultra-large-scale model training.
  • the present application provides a parallel training method and device for a hybrid expert model, which can effectively solve the load balancing problem on the basis of introducing expert strategies and improve the training efficiency of the hybrid expert model.
  • An embodiment of the present application provides a parallel training method for a hybrid expert model, the method comprising:
  • the hybrid expert model is trained based on a better one of the historical parallel strategy and the target parallel strategy.
  • Another embodiment of the present application provides a parallel training device for a hybrid expert model, the device comprising:
  • a candidate parallel strategy generation module is configured to generate multiple sets of candidate parallel strategies according to the configuration information of the hybrid expert model and the hardware information;
  • a target parallel strategy generation module is configured to determine a target parallel strategy from the plurality of groups of candidate parallel strategies according to a current load condition of the hybrid expert model in a current training cycle;
  • a historical parallel strategy determination module configured as a historical parallel strategy used in a previous training cycle of a current training cycle
  • the parallel strategy switching module is configured to train the hybrid expert model based on a better one of the historical parallel strategy and the target parallel strategy.
  • Another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the parallel training method for a hybrid expert model when executing the program.
  • Yet another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the parallel training method for a hybrid expert model.
  • Yet another embodiment of the present application provides a computer program product, including a computer program/instruction, which, when executed by a processor, implements the steps of the parallel training method for a hybrid expert model.
  • the beneficial effects of the present application are: generating multiple groups of candidate parallel strategies through the configuration information and hardware information of the hybrid expert model; and in the current training cycle, determining the target parallel strategy from the multiple groups of candidate parallel strategies according to the current load situation of the hybrid expert model; further determining the historical parallel strategy used in the previous training cycle of the current training cycle; training the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy, being able to switch the parallel strategy according to the load situation, effectively solving the load balancing problem on the basis of introducing the expert strategy, and improving the parallel training efficiency of the hybrid expert model.
  • FIG1 is a flow chart of a parallel training method for a hybrid expert model of the present application.
  • FIG. 2 is a schematic diagram of a dense large model and a MOE model in the prior art.
  • FIG3 is a flow chart of a method for determining multiple groups of candidate parallel strategies in the present application.
  • FIG4 is a flow chart of a method for determining a target parallel strategy in the present application.
  • FIG5 is a schematic diagram of the load condition of the hybrid expert model of the present application.
  • FIG6 is a flow chart of the method for parallel strategy switching of the present application.
  • FIG. 7 is a schematic diagram of a parallel training device for a hybrid expert model according to the present application.
  • FIG8 is a block diagram of an electronic device for implementing a parallel training method for a hybrid expert model of the present application.
  • This application proposes a parallel training method for a hybrid expert model, as shown in FIG1 , the method comprising:
  • Step 101 Generate multiple groups of candidate parallel strategies according to the configuration information and hardware information of the hybrid expert model
  • Step 102 In the current training cycle, according to the current load of the hybrid expert model, determine the target parallel strategy from multiple groups of candidate parallel strategies;
  • Step 103 determine the historical parallel strategy used in the previous training cycle of the current training cycle
  • Step 104 training a hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy.
  • the present application provides a parallel training method for a hybrid expert model, which generates multiple groups of candidate parallel strategies through the configuration information and hardware information of the hybrid expert model; and in the current training cycle, determines the target parallel strategy from the multiple groups of candidate parallel strategies according to the current load of the hybrid expert model; further determines the historical parallel strategy used in the previous training cycle of the current training cycle; trains the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy, can switch the parallel strategy according to the load situation, effectively solves the load balancing problem based on the introduction of the expert strategy, and improves the parallel training efficiency of the hybrid expert model.
  • Step 101 Generate multiple groups of candidate parallel strategies according to the configuration information and hardware information of the hybrid expert model.
  • multiple sets of candidate parallel strategies can be generated based on the configuration information and hardware information of the hybrid expert model.
  • the configuration information of the hybrid expert model can be the number of expert networks and the size of the expert networks; the hybrid expert model is generally configured on a computing device of a distributed computing platform, and the hardware information can be the number of computing devices on the computing platform, and can also be the bandwidth of the computing device, etc.; the overhead of internal conversion between candidate parallel strategy sets is low, and can be quickly converted during runtime.
  • multiple groups of candidate parallel strategies are generated according to the configuration information and hardware information of the hybrid expert model, including:
  • Step 1011 According to the number of expert networks and the number of computing devices, tensor parallelism, data parallelism, pipeline parallelism and expert parallelism are arranged and combined to generate multiple groups of initial parallel strategies.
  • tensor parallelism, data parallelism, pipeline parallelism, and expert parallelism are all ways to train neural networks in parallel.
  • tensor parallelism divides the model between different computing units, and each computing unit independently calculates subtasks during training; data parallelism copies and stores the model between different computing units and uses different training data for training; pipeline parallelism trains in the form of a multi-stage pipeline between different computing units; and expert parallelism is a training method unique to the hybrid expert model.
  • tensor parallelism, data parallelism, pipeline parallelism and expert parallelism are arbitrarily arranged and combined to generate multiple groups of initial parallel strategies, for example, 50% tensor parallelism + 50% expert parallelism, 50% data parallelism + 50% expert parallelism, etc.
  • Step 1012 Perform performance prediction on the multiple groups of initial parallel strategies, and select multiple groups of candidate parallel strategies from the multiple groups of initial parallel strategies based on the performance prediction results.
  • the number of expert networks of the hybrid expert model placed in each computing device is fixed in advance, that is, its segmentation and repeated placement scheme is fixed; the placement position of the expert network is not fixed, and has a mapping relationship with the candidate parallel strategies, that is, different candidate parallel strategies correspond to different placement positions of the expert networks.
  • Step 102 In the current training cycle, according to the current load of the hybrid expert model, a target parallel strategy is determined from multiple groups of candidate parallel strategies.
  • a target parallel strategy that best suits the current load condition can be determined from multiple groups of candidate parallel strategies.
  • a target parallel strategy is determined from multiple groups of candidate parallel strategies, including:
  • Step 1021 Determine the current load of each expert network in the hybrid expert model.
  • the load of each expert network can be determined according to the number of inputs received by the expert network. As shown in FIG5 , the expert network 0 is in an overloaded state, and the expert network 2 is in an idle state.
  • Step 1022 sort the expert networks in the hybrid expert model in descending order according to the current load of each expert network to obtain the placement order of each expert network.
  • the expert networks in the hybrid expert model are further sorted in descending order according to the current load of each expert network to obtain the placement order of each expert network, that is, the computing device to be placed is selected for each expert network in order of load from large to small.
  • Step 1023 Select computing devices to be placed for each expert network in turn according to the placement order until the computing devices to be placed for the last expert network are selected, and determine a group of candidate parallel strategies that match the current placement method from multiple groups of candidate parallel strategies, and use the group of candidate parallel strategies as the target parallel strategies.
  • the preset number threshold of expert networks that can be placed on each computing device can be set as N/M; thus, the computing devices where the number of placed expert networks is less than the preset number threshold are regarded as a computing device set, and the number of expert networks on this computing device set is not saturated; further, from this computing device set, determine a computing device with the smallest load of placed expert networks, and use this computing device as the computing device to be placed.
  • the load balancing problem caused by expert parallelism can be overcome, overload and idle conditions can be avoided, and the number of expert networks placed on each computing device and the expert network load can be balanced.
  • the hybrid expert model includes four expert networks, and the placement order can be determined as expert network 0, expert network 1, expert network 2, and expert network 3 (not shown in Figure 5) according to the load conditions.
  • the computing devices to be placed are selected in turn as expert network 0, expert network 1, expert network 2, and expert network 3.
  • expert network 0 it can arbitrarily select a computing device from computing devices 0-computing devices 2, for example, placing expert network 0 on computing device 0; for expert network 1, the computing devices on which the number of placed expert networks is less than 2 include computing device 0, computing device 1 and computing device 2 as a computing device set, wherein the loads of computing device 1 and computing device 2 are the smallest, and it can arbitrarily select a computing device from computing device 1 and computing device 2, for example, placing expert network 1 on computing device 1; for expert network 2, the computing devices on which the number of placed expert networks is less than 2 include computing device 0, computing device 1 and computing device 2 as a computing device set, wherein the load of computing device 2 is the smallest, and expert network 2 is placed on computing device 2; for expert network 3, the computing devices on which the number of placed expert networks is less than 2 include computing device 0, computing device 1 and computing device 2 as a computing device set, obviously, the load of expert network 2 is the smallest, and expert network 3 can be placed on computing device 2.
  • a group of candidate parallel strategies that match the current placement method is further determined from multiple groups of candidate parallel strategies, and this group of candidate parallel strategies is used as the target parallel strategy. It should be understood that the target parallel strategy is the preferred parallel strategy for the current load situation.
  • Step 103 Determine the historical parallel strategy used in the previous training cycle of the current training cycle.
  • the historical parallel strategy used in the previous training cycle of the current training cycle is determined.
  • a parallel strategy can be arbitrarily selected from multiple groups of candidate parallel strategies for training.
  • the arbitrarily selected parallel strategy is the historical parallel strategy.
  • the historical parallel strategy used in the previous training cycle of the current training cycle can be arbitrarily determined. It should be noted that the historical parallel strategy is one of multiple groups of candidate parallel strategies.
  • Step 104 training a hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy.
  • the target parallel strategy is the preferred parallel strategy for the current load situation, the time required for strategy switching also needs to be considered.
  • the training cycle includes K rounds of training. It should be understood that K is an integer greater than 0.
  • the hybrid expert model is trained, including:
  • Step 1041 Determine, according to the current load situation, a first estimated execution time T1 required for a round of training of the hybrid expert model based on the historical parallel strategy and a second estimated execution time T2 required for a round of training of the hybrid expert model based on the target parallel strategy.
  • Step 1042 Determine the switching time TC required to switch the historical parallel strategy to the target parallel strategy.
  • Step 1043A In response to the existence of T1 ⁇ K>T2 ⁇ K+TC, determine that the target parallel strategy is a better one, switch the historical parallel strategy to the target parallel strategy, and train the hybrid expert model based on the target parallel strategy.
  • a strategy switch can be performed to switch the historical parallel strategy to the target parallel strategy, and the hybrid expert model can be trained based on the target parallel strategy.
  • Step 1043B in response to the existence of T1 ⁇ K ⁇ T2 ⁇ K+TC, determine that the historical parallel strategy is a better one, do not switch the historical parallel strategy to the target parallel strategy, and continue to train the hybrid expert model based on the historical parallel strategy.
  • the historical parallel strategy is further compared with the target parallel strategy taking the switching time into consideration, so that the hybrid expert model is trained based on the better one of the historical parallel strategy and the target parallel strategy, thereby further improving the accuracy and training efficiency of parallel training.
  • the present application provides a parallel training method for a hybrid expert model, which generates multiple groups of candidate parallel strategies through the configuration information and hardware information of the hybrid expert model; and in the current training cycle, determines the target parallel strategy from the multiple groups of candidate parallel strategies according to the current load of the hybrid expert model; further determines the historical parallel strategy used in the previous training cycle of the current training cycle; trains the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy, can switch the parallel strategy according to the load situation, effectively solves the load balancing problem based on the introduction of the expert strategy, and improves the parallel training efficiency of the hybrid expert model.
  • the embodiments of the present application also provide a parallel training device for a hybrid expert model, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of the problem solved by the parallel training device for a hybrid expert model is similar to that of the parallel training method for a hybrid expert model, the implementation of the parallel training device for a hybrid expert model can refer to the implementation of the parallel training method for a hybrid expert model, and the repetitions will not be repeated.
  • the term "unit” or "module” can be a combination of software and/or hardware that implements a predetermined function.
  • the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
  • the parallel training device for the hybrid expert model includes:
  • a candidate parallel strategy generation module 701 is configured to generate multiple groups of candidate parallel strategies according to the configuration information and hardware information of the hybrid expert model;
  • the target parallel strategy generating module 702 is configured to determine the target parallel strategy from the plurality of groups of candidate parallel strategies according to the current load of the hybrid expert model in the current training cycle;
  • a historical parallel strategy determination module 703 is configured to determine the historical parallel strategy used in a previous training cycle of the current training cycle
  • the parallel strategy switching module 704 is configured to train the hybrid expert model based on the better one of the historical parallel strategy and the target parallel strategy.
  • the configuration information includes the number of expert networks
  • the hardware information includes the number of computing devices
  • the candidate parallel strategy generation module is further configured as follows:
  • tensor parallelism, data parallelism, pipeline parallelism and expert parallelism are arranged and combined to generate multiple groups of initial parallel strategies
  • Performance prediction is performed on the multiple groups of initial parallel strategies, and based on the performance prediction results, multiple groups of candidate parallel strategies are screened out from the multiple groups of initial parallel strategies.
  • the number of expert networks of the hybrid expert model placed in each computing device is fixed in advance; the placement position of the expert network is not fixed and has a mapping relationship with the candidate parallel strategies.
  • the target parallel strategy generation module includes:
  • an expert network load determination unit configured to determine a current load condition of each expert network in the hybrid expert model
  • a placement order determination unit is configured to sort the expert networks in the hybrid expert model in descending order according to the current load of each expert network to obtain a placement order for each expert network;
  • the target parallel strategy generation unit is configured to select computing devices to be placed for each expert network in turn according to the placement order until the computing devices to be placed of the last expert network are selected, determine a group of candidate parallel strategies that match the current placement method from the multiple groups of candidate parallel strategies, and use the group of candidate parallel strategies as the target parallel strategy.
  • the target parallel strategy generation unit is further configured as follows:
  • a computing device with the smallest load of the placed expert network is determined, and the computing device is used as the computing device to be placed.
  • the training cycle includes K rounds of training, and the parallel strategy switching module performs One step is configured as:
  • the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.
  • An electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps of the parallel training method for a hybrid expert model of the aforementioned embodiment.
  • a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the steps of the parallel training method for a hybrid expert model of the aforementioned embodiment.
  • a computer program product includes a computer program/instruction, which, when executed by a processor, implements the steps of the parallel training method for a hybrid expert model of the aforementioned embodiment.
  • FIG8 shows a schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure.
  • the electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.
  • the electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices.
  • the components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and/or required herein.
  • the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803.
  • ROM read-only memory
  • RAM random access memory
  • various programs and data required for the operation of the device 800 can also be stored.
  • the computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804.
  • An input/output (I/O) interface 805 is also connected to the bus 804.
  • a number of components in the device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc.
  • the communication unit 809 allows the device 800 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.
  • the computing unit 801 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc.
  • the computing unit 801 performs the various methods and processes described above, such as a parallel training method for a hybrid expert model.
  • a parallel training method for a hybrid expert model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808.
  • part or all of the computer program may be loaded and/or installed on the device 800 via the ROM 802 and/or the communication unit 809.
  • the computing unit 801 may be configured to perform a parallel training method for a hybrid expert model in any other suitable manner (e.g., by means of firmware).
  • Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof.
  • FPGAs field programmable gate arrays
  • ASICs application specific integrated circuits
  • ASSPs application specific standard products
  • SOCs systems on chips
  • CPLDs load programmable logic devices
  • Various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
  • a programmable processor which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
  • the program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions/operations specified in the flow chart and/or block diagram to be implemented.
  • the program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment.
  • a machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • a machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing.
  • a more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read-only memory
  • EPROM or flash memory erasable programmable read-only memory
  • CD-ROM portable compact disk read-only memory
  • CD-ROM compact disk read-only memory
  • magnetic storage device or any suitable combination of the foregoing.
  • the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • a keyboard and pointing device e.g., a mouse or trackball
  • Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
  • the systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components.
  • the components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
  • a computer system may include a client and a server.
  • the client and the server are generally remote from each other and usually interact through a communication network.
  • the relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other.
  • the server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种用于混合专家模型的并行训练方法及装置，属于深度学习技术领域，其中一实施例的并行训练方法包括：根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略；在当前训练周期中，根据混合专家模型的当前负载情况，从多组候选并行策略中确定目标并行策略；确定在当前训练周期的上一个训练周期中所使用的历史并行策略；基于历史并行策略和目标并行策略中较优的一个，训练混合专家模型。本申请技术方案提供一种用于混合专家模型的自动并行训练方法，通过首先确定并行策略集合，并根据负载情况对并行策略进行切换，能够在引入专家策略的基础上有效解决负载均衡问题，提高对混合专家模型的并行训练效率。

Description

一种用于混合专家模型的并行训练方法及装置
相关申请
本申请要求于2023年06月30日递交的申请号为202310789205.1的中国发明专利申请的优先权，并引用上述专利申请公开的全部内容作为本申请的一部分。
技术领域
本申请涉及深度学习技术领域，具体地讲，涉及一种用于混合专家模型的并行训练方法及装置。
背景技术
混合专家(MoE，Mixure‐of‐Experts)模型是一种神经网络，但不同于一般的神经网络是它可以根据数据分离训练多个模型，每个模型可分别称为一个专家网络，即混合专家模型的思想是训练多个专家网络，每个专家网络分别应用于数据集的不同部分。混合专家模型作为一种新兴的稀疏激活深度学习模型架构，可实现超大规模的模型训练。
现有的自动化并行训练主要针对具有稠密架构的传统模型，一方面没有考虑到专家并行策略，另一方面未考虑到专家并行策略可能带来的负载均衡问题
发明内容
为了解决上述问题，本申请提供了一种用于混合专家模型的并行训练方法及装置，能够在引入专家策略的基础上有效解决负载均衡问题，提高对混合专家模型的训练效率。
本申请一实施例提供一种用于混合专家模型的并行训练方法，该方法包括：
根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略；
在当前训练周期中，根据所述混合专家模型的当前负载情况，从所述多组候选并行策略中确定目标并行策略；
确定在当前训练周期的上一个训练周期中所使用的历史并行策略；
基于所述历史并行策略和所述目标并行策略中较优的一个，训练所述混合专家模型。
本申请又一实施例提供一种用于混合专家模型的并行训练装置，该装置包括：
候选并行策略生成模块，被配置为根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略；
目标并行策略生成模块，被配置为在当前训练周期中，根据所述混合专家模型的当前负载情况，从所述多组候选并行策略中确定目标并行策略；
历史并行策略确定模块，被配置为在当前训练周期的上一个训练周期中所使用的历史并行策略；
并行策略切换模块，被配置为基于所述历史并行策略和所述目标并行策略中较优的一个，训练所述混合专家模型。
本申请又一实施例提供一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现所述一种用于混合专家模型的并行训练方法的步骤。
本申请又一实施例提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现所述一种用于混合专家模型的并行训练方法的步骤。
本申请又一实施例提供一种计算机程序产品，包括计算机程序/指令，该计算机程序/指令被处理器执行时实现所述一种用于混合专家模型的并行训练方法的步骤。
本申请的有益效果是：通过混合专家模型的配置信息和硬件信息，生成多组候选并行策略；并在当前训练周期中，根据混合专家模型的当前负载情况，从多组候选并行策略中确定目标并行策略；进一步确定在当前训练周期的上一个训练周期中所使用的历史并行策略；基于历史并行策略和目标并行策略中较优的一个，训练所述混合专家模型，能够根据负载情况对并行策略进行切换，在引入专家策略的基础上有效解决负载均衡问题，提高对混合专家模型的并行训练效率。
通过以下参照附图对本申请的示例性实施例的详细描述，本申请的其它特征及其优点将会变得清楚。
附图说明
被结合在说明书中并构成说明书的一部分的附图示出了本申请的实施例，并且连同其说明一起用于解释本申请的原理。
图1为本申请一种用于混合专家模型的并行训练方法的流程图。
图2为现有技术中的稠密大模型和MOE模型的示意图。
图3为本申请确定多组候选并行策略的方法流程图。
图4为本申请确定目标并行策略的方法流程图。
图5为本申请混合专家模型的负载情况的示意图。
图6为本申请并行策略切换的方法流程图。
图7为本申请一种用于混合专家模型的并行训练装置的示意图。
图8为用来实现本申请的一种用于混合专家模型的并行训练方法的电子设备的框图。
具体实施方式
现在将参照附图来详细描述本申请的各种示例性实施例。应注意到：除非另外具体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本申请的范围。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，绝不作为对本申请及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
在这里示出和讨论的所有例子中，任何具体值应被解释为仅仅是示例性的，而不是作为限制。因此，示例性实施例的其它例子可以具有不同的值。
本申请提出了一种用于混合专家模型的并行训练方法，如图1所示，该方法包括：
步骤101、根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略；
步骤102、在当前训练周期中，根据混合专家模型的当前负载情况，从多组候选并行策略中确定目标并行策略；
步骤103、确定在当前训练周期的上一个训练周期中所使用的历史并行策略；
步骤104、基于历史并行策略和目标并行策略中较优的一个，训练混合专家模型。
本申请提供的一种用于混合专家模型的并行训练方法，通过混合专家模型的配置信息和硬件信息，生成多组候选并行策略；并在当前训练周期中，根据混合专家模型的当前负载情况，从多组候选并行策略中确定目标并行策略；进一步确定在当前训练周期的上一个训练周期中所使用的历史并行策略；基于历史并行策略和目标并行策略中较优的一个，训练混合专家模型，能够根据负载情况对并行策略进行切换，在引入专家策略的基础上有效解决负载均衡问题，提高对混合专家模型的并行训练效率。
下面对图1的每个步骤进行详细阐述：
步骤101、根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略。
应当理解，近年来深度神经网络设计的一个有前途的方向是增加模型大小。从模型设计方面，已经提出了稀疏架构来打破具有密集架构的神经网络模型中模型大小和计算成本之间的正比关系。例如，不同于图2中直接将小模型尺寸扩大为稠密大模型，混合专家(MoE，Mixure‐of‐Experts)模型MoE模型包含若干小模型,其中每个小模型被称作一个专家网络，对于一个训练样本,通过门网络模块选择性地激活少数几个专家对该样本进行计算。
在本实施例中，可以根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略。其中，混合专家模型的配置信息可以为专家网络数量和专家网络尺寸；混合专家模型一般配置在分布式的计算平台的运算设备上，硬件信息可以为计算平台上运算设备数量，还可以为运算设备的带宽等；候选并行策略集合内部互相转换的开销较低，可以在运行时进行快速转换。
在本实施例的一些可选方式中，如图3所示，根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略，包括：
步骤1011、根据专家网络数量和运算设备数量，对张量并行、数据并行、流水线并行以及专家并行进行排列组合，生成多组初始并行策略。
在本实施例中，张量并行、数据并行、流水线并行以及专家并行均为并行训练神经网络的方式。其中，张量并行在不同的计算单元之间切分模型，训练时各个计算单元独立计算子任务；数据并行在不同的计算单元之间复制存储模型，并使用不同的训练数据进行训练；流水线并行在不同的计算单元之间以多阶段流水线的形式进行训练；专家并行是混合专家模型特有的训练方式。
在本实施例中，根据专家网络数量和运算设备数量，基于搜索算法，对张量并行、数据并行、流水线并行以及专家并行进行任意的排列组合，能够生成多组初始并行策略。例如，50％张量并行+50％的专家并行、50％数据并行+50％的专家并行等。
步骤1012、对多组初始并行策略进行性能预测，并基于性能预测结果，从多组初始并行策略中筛选出多组候选并行策略。
在本实施例中，在得到多组初始并行策略后，为进一步缩小搜索空间，提高对混合专家模型的并行训练效率，还需要对前述的多组初始并行策略进行性能预测。例如，对于给定的MoE模型的路由算法,其可以给出对于每个专家子模型的运算负载的上限,并使用该上限和性能预测公式对每个初始并行策略的训练性能进行估计,最终选择性能靠前的并行策略作为候选并行策略，来对混合专家模型进行训练。
在本实施例的一些可选方式中，在多组候选并行策略中，混合专家模型在每个运算设备的专家网络的摆放数量预先固定，即其切分和重复放置的方案是固定的；专家网络的摆放位置不固定，与候选并行策略具有映射关系，即不同的候选并行策略分别对应不同的专家网络的摆放位置。
步骤102、在当前训练周期中，根据混合专家模型的当前负载情况，从多组候选并行策略中确定目标并行策略。
在本实施例中，在当前训练周期内，能够基于该轮训练中专家网络的负载情况，从多组候选并行策略中确定最适合当前负载情况的目标并行策略。
在本实施例的一些可选方式中，如图4所示，根据混合专家模型的当前负载情况，从多组候选并行策略中确定目标并行策略，包括：
步骤1021、确定混合专家模型中的每个专家网络的当前负载情况。
在本实施例中，可根据专家网络接收的输入数量来确定每个专家网络的负载情况，如图5所示，专家网络0处于过载状态，专家网络2处于空闲状态。
步骤1022、根据每个专家网络的当前负载情况，对混合专家模型中的专家网络进行降序排序，得到每个专家网络的放置顺序。
在本实施例中，进一步根据每个专家网络的当前负载情况，对混合专家模型中的专家网络进行降序排序，得到每个专家网络的放置顺序，即按照负载从大到小的顺序为每个专家网络选择所放置的运算设备。
步骤1023、根据放置顺序，依次为每个专家网络选择待放置运算设备，直至最后一个专家网络的待放置运算设备被选择完成，从多组候选并行策略中确定与当前放置方式相匹配的一组候选并行策略，并将该组候选并行策略作为目标并行策略。
在本实施例的一些可选方式中，依次对每一个专家网络均执行如下操作：
首先，确定已放置专家网络的数量小于预设数量阈值的运算设备集合，例如，设共有N个专家网络，M个不同的运算设备，则可以规定在每个运算设备上可放置的专家网络的预设数量阈值为N/M；由此，将已放置专家网络的数量小于预设数量阈值的运算设备作为运算设备集合，该运算设备集合上的专家网络数量未处于饱和状态；进一步，从该运算设备集合中，确定已放置专家网络的负载为最小的一个运算设备，并将该运算设备作为待放置运算设备。通过该放置方式，能够克服专家并行带来的负载均衡问题，能够避免出现过载和空闲情况，使得各运算设备上放置的专家网络数量和专家网络负载实现均衡配置。
以图5为示例进行说明，设定每个运算设备上可以放置2个专家网络，混合专家模型部署在3个运算设备上，混合专家模型包括4个专家网络，且根据负载情况可确定放置顺序为专家网络0、专家网络1、专家网络2以及专家网络3(图5中未示出)，依次为专家网络0、专家网络1、专家网络2以及专家网络3选择所放置的运算设备。
其中，对于专家网络0，其可以从运算设备0-运算设备2中任意选择一运算设备，例如将专家网络0放置在运算设备0；对于专家网络1，已放置专家网络的数量小于2的运算设备作为运算设备集合包括运算设备0、运算设备1和运算设备2，其中，运算设备1和运算设备2的负载最小，其可以从运算设备1和运算设备2中任意选择一运算设备，例如将专家网络1放置在运算设备1；对于专家网络2，已放置专家网络的数量小于2的运算设备作为运算设备集合包括运算设备0、运算设备1和运算设备2，其中，运算设备2的负载最小，将专家网络2放置在运算设备2上；对于专家网络3，已放置专家网络的数量小于2的运算设备作为运算设备集合包括运算设备0、运算设备1和运算设备2，显然，专家网络2的负载最小，可将专家网络3放置在运算设备2上。
在本实施例中，依次为每个专家网路选择了所放置的设备后，进一步从多组候选并行策略中确定与当前放置方式相匹配的一组候选并行策略，并将该组候选并行策略作为目标并行策略，应当理解，该目标并行策略为针对当前负载情况的优选并行策略。
步骤103、确定在当前训练周期的上一个训练周期中所使用的历史并行策略。
在本实施例中，确定在当前训练周期的上一个训练周期中所使用的历史并行策略。其中，在首个训练周期内，可以从多组候选并行策略中任意选择一个并行策略进行训练，在首个训练周期的下一个训练周期内，该任意选择的并行策略即为历史并行策略，依次类推，可任意确定在当前训练周期的上一个训练周期中所使用的历史并行策略。需要说明的是，历史并行策略为多组候选并行策略的一个。
步骤104、基于历史并行策略和目标并行策略中较优的一个，训练混合专家模型。
需要说明的是，虽然目标并行策略为针对当前负载情况的优选并行策略，但还需要考虑策略切换所需的时间，为此，在本实施例的一些可选方式中，如图6所示，设训练周期包括K轮训练，应当理解，K为大于0的整数，基于历史并行策略和目标并行策略中较优的一个，训练混合专家模型，包括：
步骤1041、根据当前负载情况，确定基于历史并行策略对混合专家模型进行一轮训练所需的第一预计执行时间T1以及基于目标并行策略对混合专家模型进行一轮训练所需的第二预计执行时间T2。
步骤1042、确定将历史并行策略切换为目标并行策略所需的切换时间TC。
步骤1043A、响应于存在T1×K>T2×K+TC，确定目标并行策略为较优的一个，将历史并行策略切换为目标并行策略，并基于目标并行策略训练混合专家模型。
在本实施中，即历史并行策略执行K轮训练的时间大于使用目标并行策略时间执行K轮训练的时间与切换时间的和，则可以进行策略切换，将历史并行策略切换为目标并行策略，并基于目标并行策略训练混合专家模型。
步骤1043B、响应于存在T1×K≤T2×K+TC，确定历史并行策略为较优的一个，不将历史并行策略切换为目标并行策略，并继续基于历史并行策略训练混合专家模型。
在本实施例中，即历史并行策略执行K轮训练的时间不大于使用目标并行策略时间执行K轮训练的时间与切换时间的和，则不可以进行策略切换，并继续基于历史并行策略训练混合专家模型。
在本实施例中，考虑切换时间，将历史并行策略与目标并行策略进一步对比，从而将基于历史并行策略和目标并行策略中较优的一个，训练混合专家模型，进一步提高并行训练的准确性和训练效率。
本申请提供的一种用于混合专家模型的并行训练方法，通过混合专家模型的配置信息和硬件信息，生成多组候选并行策略；并在当前训练周期中，根据混合专家模型的当前负载情况，从多组候选并行策略中确定目标并行策略；进一步确定在当前训练周期的上一个训练周期中所使用的历史并行策略；基于历史并行策略和目标并行策略中较优的一个，训练混合专家模型，能够根据负载情况对并行策略进行切换，在引入专家策略的基础上有效解决负载均衡问题，提高对混合专家模型的并行训练效率。
基于同一发明构思，本申请实施例还提供了一种用于混合专家模型的并行训练装置，可以用于实现上述实施例中所描述的方法，如下面实施例所述。由于该用于混合专家模型的并行训练装置所解决问题的原理与用于混合专家模型的并行训练方法相似，因此用于混合专家模型的并行训练装置的实施可以参见用于混合专家模型的并行训练方法的实施，重复之处不再赘述。以下所使用的，术语“单元”或者“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的系统较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。
如图7所示，该用于混合专家模型的并行训练装置包括：
候选并行策略生成模块701，被配置为根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略；
目标并行策略生成模块702，被配置为在当前训练周期中，根据所述混合专家模型的当前负载情况，从所述多组候选并行策略中确定目标并行策略；
历史并行策略确定模块703，被配置为在当前训练周期的上一个训练周期中所使用的历史并行策略；
并行策略切换模块704，被配置为基于所述历史并行策略和所述目标并行策略中较优的一个，训练所述混合专家模型。
在本实施例的一些可选方式中，所述配置信息包括专家网络数量，所述硬件信息包括运算设备数量，候选并行策略生成模块进一步被配置为：
根据所述专家网络数量和所述运算设备数量，对张量并行、数据并行、流水线并行以及专家并行进行排列组合，生成多组初始并行策略；
对所述多组初始并行策略进行性能预测，并基于性能预测结果，从所述多组初始并行策略中筛选出多组候选并行策略。
在本实施例的一些可选方式中，在所述多组候选并行策略中，所述混合专家模型在每个运算设备的专家网络的摆放数量预先固定；专家网络的摆放位置不固定，与所述候选并行策略具有映射关系。
在本实施例的一些可选方式中，目标并行策略生成模块包括：
专家网络负载确定单元，被配置为确定所述混合专家模型中的每个专家网络的当前负载情况；
放置顺序确定单元，被配置为根据所述每个专家网络的当前负载情况，对所述混合专家模型中的专家网络进行降序排序，得到所述每个专家网络的放置顺序；
目标并行策略生成单元，被配置为根据所述放置顺序，依次为每个专家网络选择待放置运算设备，直至最后一个专家网络的待放置运算设备被选择完成，从所述多组候选并行策略中确定与当前放置方式相匹配的一组候选并行策略，并将该组候选并行策略作为目标并行策略。
在本实施例的一些可选方式中，目标并行策略生成单元进一步被配置为：
对每一个专家网络执行如下操作：
确定已放置专家网络的数量小于预设数量阈值的运算设备集合；
从所述运算设备集合中，确定所述已放置专家网络的负载为最小的一个运算设备，并将该运算设备作为所述待放置运算设备。
在本实施例的一些可选方式中，所述训练周期包括K轮训练，并行策略切换模块进 一步被配置为：
根据所述当前负载情况，确定基于所述历史并行策略对所述混合专家模型进行一轮训练所需的第一预计执行时间T1以及基于所述目标并行策略对所述混合专家模型进行一轮训练所需的第二预计执行时间T2；
确定将所述历史并行策略切换为所述目标并行策略所需的切换时间TC；
响应于存在T1×K>T2×K+TC，确定所述目标并行策略为较优的一个，将所述历史并行策略切换为所述目标并行策略，并基于所述目标并行策略训练所述混合专家模型；
响应于存在T1×K≤T2×K+TC，确定所述历史并行策略为较优的一个，不将所述历史并行策略切换为所述目标并行策略，并继续基于所述历史并行策略训练所述混合专家模型。
根据本公开的实施例，本公开还提供了一种电子设备、一种可读存储介质以及一种计算机程序产品。
一种电子设备，包括：至少一个处理器；以及与至少一个处理器通信连接的存储器；其中，存储器存储有可被至少一个处理器执行的指令，指令被至少一个处理器执行，以使至少一个处理器能够执行前述实施例的用于混合专家模型的并行训练方法的步骤。
一种存储有计算机指令的非瞬时计算机可读存储介质，其中，计算机指令用于使计算机执行前述实施例的用于混合专家模型的并行训练方法的步骤。
一种计算机程序产品，包括计算机程序/指令，该计算机程序/指令被处理器执行时实现前述实施例的用于混合专家模型的并行训练方法的步骤。
图8示出了可以用来实施本公开的实施例的示例电子设备800的示意性框图。电子设备旨在表示各种形式的数字计算机，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本公开的实现。
如图8所示，设备800包括计算单元801，其可以根据存储在只读存储器(ROM)802中的计算机程序或者从存储单元808加载到随机访问存储器(RAM)803中的计算机程序，来执行各种适当的动作和处理。在RAM 803中，还可存储设备800操作所需的各种程序和数据。计算单元801、ROM 802以及RAM 803通过总线804彼此相连。输入/输出(I/O)接口805也连接至总线804。
设备800中的多个部件连接至I/O接口805，包括：输入单元806，例如键盘、鼠标等；输出单元807，例如各种类型的显示器、扬声器等；存储单元808，例如磁盘、光盘等；以及通信单元809，例如网卡、调制解调器、无线通信收发机等。通信单元809允许设备800通过诸如因特网的计算机网络和/或各种电信网络与其他设备交换信息/数据。
计算单元801可以是各种具有处理和计算能力的通用和/或专用处理组件。计算单元801的一些示例包括但不限于中央处理单元(CPU)、图形处理单元(GPU)、各种专用的人工智能(AI)计算芯片、各种运行机器学习模型算法的计算单元、数字信号处理器(DSP)、以及任何适当的处理器、控制器、微控制器等。计算单元801执行上文所描述的各个方法和处理，例如一种用于混合专家模型的并行训练方法。
例如，在一些实施例中，一种用于混合专家模型的并行训练方法可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元808。在一些实施例中，计算机程序的部分或者全部可以经由ROM 802和/或通信单元809而被载入和/或安装到设备800上。当计算机程序加载到RAM 803并由计算单元801执行时，可以执行上文描述的一种用于混合专家模型的并行训练方法的一个或多个步骤。备选地，在其他实施例中，计算单元801可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行一种用于混合专家模型的并行训练方法。
本文中以上描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电路系统、场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、芯片上系统的系统(SOC)、负载可编程逻辑设备(CPLD)、计算机硬件、固件、软件、和/或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和/或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。
用于实施本公开的方法的程序代码可以采用一个或多个编程语言的任何组合来编写。这些程序代码可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器或控制器，使得程序代码当由处理器或控制器执行时使流程图和/或框图中所规定的功能/操作被实施。程序代码可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)和互联网。
计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。服务器可以是云服务器，也可以为分布式系统的服务器，或者是结合了区块链的服务器。
应该理解，可以使用上面所示的各种形式的流程，重新排序、增加或删除步骤。例如，本发公开中记载的各步骤可以并行地执行也可以顺序地执行也可以不同的次序执行，只要能够实现本公开的技术方案所期望的结果，本文在此不进行限制。
虽然已经通过例子对本申请的一些特定实施例进行了详细说明，但是本领域的技术 人员应该理解，以上例子仅是为了进行说明，而不是为了限制本申请的范围。本领域的技术人员应该理解，可在不脱离本申请的范围和精神的情况下，对以上实施例进行修改。本申请的范围由所附权利要求来限定。

Claims (10)

  1. 一种用于混合专家模型的并行训练方法，其特征在于，包括：
    根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略；
    在当前训练周期中，根据所述混合专家模型的当前负载情况，从所述多组候选并行策略中确定目标并行策略；
    确定在当前训练周期的上一个训练周期中所使用的历史并行策略；
    基于所述历史并行策略和所述目标并行策略中较优的一个，训练所述混合专家模型。
  2. 根据权利要求1所述的方法，其特征在于，所述配置信息包括专家网络数量，所述硬件信息包括运算设备数量，所述根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略，包括：
    根据所述专家网络数量和所述运算设备数量，对张量并行、数据并行、流水线并行以及专家并行进行排列组合，生成多组初始并行策略；
    对所述多组初始并行策略进行性能预测，并基于性能预测结果，从所述多组初始并行策略中筛选出多组候选并行策略。
  3. 根据权利要求2所述的方法，其特征在于，在所述多组候选并行策略中，所述混合专家模型在每个运算设备的专家网络的摆放数量预先固定；专家网络的摆放位置不固定，与所述候选并行策略具有映射关系。
  4. 根据权利要求3所述的方法，其特征在于，所述根据所述混合专家模型的当前负载情况，从所述多组候选并行策略中确定目标并行策略，包括：
    确定所述混合专家模型中的每个专家网络的当前负载情况；
    根据所述每个专家网络的当前负载情况，对所述混合专家模型中的专家网络进行降序排序，得到所述每个专家网络的放置顺序；
    根据所述放置顺序，依次为每个专家网络选择待放置运算设备，直至最后一个专家网络的待放置运算设备被选择完成，从所述多组候选并行策略中确定与当前放置方式相匹配的一组候选并行策略，并将该组候选并行策略作为目标并行策略。
  5. 根据权利要求4所述的方法，其特征在于，所述根据所述放置顺序，依次为每个专家网络选择待放置运算设备，包括：
    对每一个专家网络执行如下操作：
    确定已放置专家网络的数量小于预设数量阈值的运算设备集合；
    从所述运算设备集合中，确定所述已放置专家网络的负载为最小的一个运算设备，并将该运算设备作为所述待放置运算设备。
  6. 根据权利要求1所述的方法，其特征在于，所述训练周期包括K轮训练，所述基于所述历史并行策略和所述目标并行策略中较优的一个，训练所述混合专家模型，包括：
    根据所述当前负载情况，确定基于所述历史并行策略对所述混合专家模型进行一轮训练所需的第一预计执行时间T1以及基于所述目标并行策略对所述混合专家模型进行一轮训练所需的第二预计执行时间T2；
    确定将所述历史并行策略切换为所述目标并行策略所需的切换时间TC；
    响应于存在T1×K>T2×K+TC，确定所述目标并行策略为较优的一个，将所述历史并行策略切换为所述目标并行策略，并基于所述目标并行策略训练所述混合专家模型；
    响应于存在T1×K≤T2×K+TC，确定所述历史并行策略为较优的一个，不将所述历史并行策略切换为所述目标并行策略，并继续基于所述历史并行策略训练所述混合专家模型。
  7. 一种用于混合专家模型的并行训练装置，其特征在于，包括：
    候选并行策略生成模块，被配置为根据混合专家模型的配置信息和硬件信息，生成多组候选并行策略；
    目标并行策略生成模块，被配置为在当前训练周期中，根据所述混合专家模型的当前负载情况，从所述多组候选并行策略中确定目标并行策略；
    历史并行策略确定模块，被配置为在当前训练周期的上一个训练周期中所使用的历史并行策略；
    并行策略切换模块，被配置为基于所述历史并行策略和所述目标并行策略中较优的一个，训练所述混合专家模型。
  8. 一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，其特征在于，所述处理器执行所述程序时实现权利要求1至6任一项所述的用于混合专家模型的并行训练方法的步骤。
  9. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该计算机程序被处理器执行时实现权利要求1至6任一项所述的用于混合专家模型的并行训练方法的步骤。
  10. 一种计算机程序产品，包括计算机程序/指令，其特征在于，该计算机程序/指令被处理器执行时实现权利要求1至6任一项所述的用于混合专家模型的并行训练方法的步骤。
PCT/CN2023/142843 2023-06-30 2023-12-28 一种用于混合专家模型的并行训练方法及装置 Ceased WO2025001026A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310789205.1 2023-06-30
CN202310789205.1A CN116502680B (zh) 2023-06-30 2023-06-30 一种用于混合专家模型的并行训练方法及装置

Publications (1)

Publication Number Publication Date
WO2025001026A1 true WO2025001026A1 (zh) 2025-01-02

Family

ID=87328926

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/142843 Ceased WO2025001026A1 (zh) 2023-06-30 2023-12-28 一种用于混合专家模型的并行训练方法及装置

Country Status (2)

Country Link
CN (1) CN116502680B (zh)
WO (1) WO2025001026A1 (zh)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119416822A (zh) * 2025-01-06 2025-02-11 山东浪潮科学研究院有限公司 一种基于多专家层的任务处理方法、系统、终端及介质
CN120013301A (zh) * 2025-04-22 2025-05-16 浙江大学 一种单位乘客的公交碳排放因子预测方法、设备、介质
CN120066731A (zh) * 2025-02-27 2025-05-30 北京智源人工智能研究院 异构硬件集群分布式训练方法、装置、电子设备及介质
CN120234286A (zh) * 2025-05-30 2025-07-01 苏州元脑智能科技有限公司 数据的处理方法
CN120654501A (zh) * 2025-08-15 2025-09-16 温州大学 基于有限元分析的工业用减速器疲劳损伤分析方法、系统
CN120806040A (zh) * 2025-09-09 2025-10-17 浪潮电子信息产业股份有限公司 专家模型的训练方法和装置、存储介质及电子设备
CN121483575A (zh) * 2026-01-09 2026-02-06 山东建筑大学 用于医学图像的多任务专家模型的跨模态推理方法及系统

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116502680B (zh) * 2023-06-30 2023-09-19 清华大学 一种用于混合专家模型的并行训练方法及装置
CN117827404B (zh) * 2023-12-20 2025-03-28 北京百度网讯科技有限公司 任务处理方法、任务执行方法、数据处理方法及装置
CN117972293B (zh) * 2024-03-28 2024-06-07 北京思凌科半导体技术有限公司 基于混合专家模型的计算方法、装置、设备及存储介质
CN118551220B (zh) * 2024-05-14 2025-03-28 哈尔滨工业大学(深圳)(哈尔滨工业大学深圳科技创新研究院) 一种基于混合专家结构大模型训练的多模态数据解析方法
CN119759554B (zh) * 2024-12-10 2025-11-04 北京百度网讯科技有限公司 跨数据中心的分布式训练方法、装置及计算机程序产品
CN121478468A (zh) * 2025-09-28 2026-02-06 上海清程极智科技有限公司 本地化大规模语言模型服务方法及相关设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190188577A1 (en) * 2017-12-20 2019-06-20 Advanced Micro Devices, Inc. Dynamic hardware selection for experts in mixture-of-experts model
CN114841315A (zh) * 2022-04-22 2022-08-02 北京百度网讯科技有限公司 混合专家模型实现方法、系统、电子设备及存储介质
CN115330058A (zh) * 2022-08-16 2022-11-11 北京百度网讯科技有限公司 对象选择模型的训练方法、对象选择方法及装置
CN115688917A (zh) * 2022-11-10 2023-02-03 北京百度网讯科技有限公司 神经网络模型的训练方法、装置、电子设备及存储介质
CN116502680A (zh) * 2023-06-30 2023-07-28 清华大学 一种用于混合专家模型的并行训练方法及装置

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11295171B2 (en) * 2019-10-18 2022-04-05 Google Llc Framework for training machine-learned models on extremely large datasets
US12518135B2 (en) * 2021-02-05 2026-01-06 Google Llc Sparse and differentiable mixture of experts neural networks
CN114186633B (zh) * 2021-12-10 2023-04-07 北京百度网讯科技有限公司 模型的分布式训练方法、装置、设备以及存储介质
CN114429195B (zh) * 2022-01-21 2024-07-19 清华大学 混合专家模型训练的性能优化方法和装置
CN114968587A (zh) * 2022-06-01 2022-08-30 北京潞晨科技有限公司 并行异构平台计算moe的训练方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190188577A1 (en) * 2017-12-20 2019-06-20 Advanced Micro Devices, Inc. Dynamic hardware selection for experts in mixture-of-experts model
CN114841315A (zh) * 2022-04-22 2022-08-02 北京百度网讯科技有限公司 混合专家模型实现方法、系统、电子设备及存储介质
CN115330058A (zh) * 2022-08-16 2022-11-11 北京百度网讯科技有限公司 对象选择模型的训练方法、对象选择方法及装置
CN115688917A (zh) * 2022-11-10 2023-02-03 北京百度网讯科技有限公司 神经网络模型的训练方法、装置、电子设备及存储介质
CN116502680A (zh) * 2023-06-30 2023-07-28 清华大学 一种用于混合专家模型的并行训练方法及装置

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119416822A (zh) * 2025-01-06 2025-02-11 山东浪潮科学研究院有限公司 一种基于多专家层的任务处理方法、系统、终端及介质
CN120066731A (zh) * 2025-02-27 2025-05-30 北京智源人工智能研究院 异构硬件集群分布式训练方法、装置、电子设备及介质
CN120013301A (zh) * 2025-04-22 2025-05-16 浙江大学 一种单位乘客的公交碳排放因子预测方法、设备、介质
CN120234286A (zh) * 2025-05-30 2025-07-01 苏州元脑智能科技有限公司 数据的处理方法
CN120654501A (zh) * 2025-08-15 2025-09-16 温州大学 基于有限元分析的工业用减速器疲劳损伤分析方法、系统
CN120806040A (zh) * 2025-09-09 2025-10-17 浪潮电子信息产业股份有限公司 专家模型的训练方法和装置、存储介质及电子设备
CN121483575A (zh) * 2026-01-09 2026-02-06 山东建筑大学 用于医学图像的多任务专家模型的跨模态推理方法及系统

Also Published As

Publication number Publication date
CN116502680B (zh) 2023-09-19
CN116502680A (zh) 2023-07-28

Similar Documents

Publication Publication Date Title
CN116502680B (zh) 一种用于混合专家模型的并行训练方法及装置
US12573477B2 (en) Molecular structure acquisition method and apparatus, electronic device and storage medium
US20250036920A1 (en) Mixture-of-experts model implementation method and system, electronic device, and storage medium
KR20250166738A (ko) 대규모 언어 모델에 적용되는 텍스트 추론 가속 방법 및 관련 장치
EP3649582A1 (en) System and method for automatic building of learning machines using learning machines
CN118761472A (zh) 混合专家模型推理加速方法、装置、设备、介质及程序
CN119862360A (zh) 数据处理方法、装置、电子设备和存储介质
CN117669751B (zh) 量子电路模拟方法、装置及电子设备
CN117455005B (zh) 模型算子的处理方法、装置、电子设备及存储介质
CN120851217B (zh) 基于多级缓存机制的大模型推理方法、系统、电子设备及存储介质
CN114998649B (zh) 图像分类模型的训练方法、图像分类方法及装置
CN120893495A (zh) 用于训练隔离路由模型的方法、计算机存储介质、程序产品和设备及混合推理系统
CN115578583A (zh) 图像处理方法、装置、电子设备和存储介质
CN112784962B (zh) 超网络的训练方法、装置、电子设备和存储介质
CN120218182A (zh) 模型蒸馏方法、答复信息生成方法及装置
CN119960929A (zh) 大模型加载方法、装置、电子设备及存储介质
CN119807356A (zh) 信息获取方法、装置及电子设备
CN112560987A (zh) 图像样本处理方法、装置、设备、存储介质和程序产品
CN115759209B (zh) 神经网络模型的量化方法、装置、电子设备及介质
AU2023285953A1 (en) Method and apparatus of processing quantum noise circuit, device, and storage medium
CN113807397B (zh) 语义表示模型的训练方法、装置、设备和存储介质
CN117273069A (zh) 一种基于神经网络模型的推理方法、装置、设备及介质
CN120598063B (zh) 模型训练优化方法、装置、电子设备、存储介质及程序
CN114331379B (zh) 用于输出待办任务的方法、模型训练方法和装置
CN112528123A (zh) 模型搜索方法、装置、电子设备、存储介质和程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23943497

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE