WO2023142828A1 - 信息处理方法及装置 - Google Patents

信息处理方法及装置 Download PDF

Info

Publication number
WO2023142828A1
WO2023142828A1 PCT/CN2022/141926 CN2022141926W WO2023142828A1 WO 2023142828 A1 WO2023142828 A1 WO 2023142828A1 CN 2022141926 W CN2022141926 W CN 2022141926W WO 2023142828 A1 WO2023142828 A1 WO 2023142828A1
Authority
WO
WIPO (PCT)
Prior art keywords
quantization
groups
indication information
parameters
group
Prior art date
Application number
PCT/CN2022/141926
Other languages
English (en)
French (fr)
Inventor
谢俊文
马梦瑶
李佳徽
马江镭
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023142828A1 publication Critical patent/WO2023142828A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the present disclosure relates to the field of distributed learning, and more specifically, to an information processing method and device.
  • Federated Learning is an artificial intelligence (AI) basic technology. It ensures information security during big data exchange, protects terminal data and personal data privacy, and ensures legal compliance. Efficient machine learning between parties or multiple computing nodes.
  • the machine learning algorithms that can be used in federated learning are not limited to neural networks, but also include other algorithms such as random forests.
  • Distributed learning such as federated learning involves data interaction between multiple parties, such as the interaction of model parameters and so on.
  • the interactive data needs to be compressed before being sent.
  • the current compression process uses compression schemes based on deep learning, such as model pruning, knowledge distillation, format conversion, etc. These schemes can effectively use transmission resources and ensure communication efficiency. However, these schemes set some smaller parameter values to zero or migrate larger models to smaller models to reduce the amount of transmitted data, resulting in loss of model performance.
  • Embodiments of the present disclosure provide a data processing solution, which can be applied to distributed learning such as federated learning, and can ensure model performance while ensuring communication efficiency.
  • a data processing method includes: the first device divides the parameters of the distributed training model into multiple groups, and at least two of the multiple groups have different numerical intervals; the first device is based on the quantization bits corresponding to each of the multiple groups number, determine a plurality of quantization groups corresponding to the plurality of groups, and the number of quantization bits is determined based on at least one of the following: the amount of transmission resources between the first device and the second device, the value of the loss function, or each group The value range of the parameter; and the first device sends multiple quantization groups to the second device.
  • the first device can quantize the parameters of the distributed training model, thereby realizing parameter compression.
  • This process not only considers factors such as transmission resources, but also considers factors such as the value of parameters, so as to ensure that the compression of parameters can not only ensure communication efficiency, but also ensure the performance of parameters. On the one hand, it avoids the performance loss caused by excessive compression, and on the other hand, it avoids the excessive delay caused by the low compression ratio.
  • the method further includes: the first device determines the quantization boundary value corresponding to the first group based on the multiple first parameters in the first group of the multiple groups.
  • the value range of the parameters in the divided groups can be specified by the quantization boundary value, so that the amount of data transmitted in the quantization process can be simplified.
  • the quantization boundary value includes at least one of the following: a maximum value, or a minimum value, where the maximum value is the maximum value among the multiple first parameters or the maximum value is a multiple of the multiple first parameters.
  • the maximum value of the absolute value, the minimum value is the minimum value among the multiple first parameters or the minimum value is the minimum value of the multiple absolute values of the multiple first parameters.
  • the maximum value and/or the minimum value as the quantization boundary value, it can be used to specify the numerical interval or absolute value range of the parameter in the divided group.
  • the quantization group corresponding to the first group includes quantization boundary values and a plurality of quantization parameters corresponding to the plurality of first parameters.
  • the first device receives first indication information from the second device, the first indication information indicates whether to determine the quantization boundary value according to the absolute value; and the first device determines the same as the first
  • the quantization boundary value corresponding to the group includes: the first device determines the quantization boundary value based on the first indication information.
  • the first device can determine the quantization boundary value based on the indication of the second device, which can reduce the amount of calculation at the first device for determining the quantization mode.
  • the method further includes: the first device sends second indication information to the second device, where the second indication information indicates whether the quantization boundary value is determined according to an absolute value.
  • the first device can inform the second device of the quantization method through the indication information, so that the second device can correctly perform inverse quantization.
  • the number of quantization bits for each group is equal. In this manner, each group uses the same number of quantization bits, which can simplify operations and improve processing efficiency.
  • the plurality of groups includes a first group and a second group, and the number of quantization bits corresponding to the first group is not equal to the number of quantization bits corresponding to the second group. In this way, different groups use different numbers of quantization bits, which can fully consider the parameter differences between different groups, and avoid the impact of quantization on performance.
  • the method further includes: the first device determining multiple quantization bit numbers corresponding to multiple groups.
  • the first device determining the multiple quantization bit numbers corresponding to the multiple groups includes: the first device determining the multiple quantization bit numbers corresponding to the multiple groups based on the configuration of the quantization bit numbers from the second device The number of quantization bits, the configuration of the number of quantization bits includes the quantization threshold and the corresponding number of quantization bits.
  • the first device can determine the number of quantization bits based on the configuration of the number of quantization bits from the second device, which can meet the actual needs of the first device and the second device at the same time, and the determined number of quantization bits can take into account communication efficiency and compression performance.
  • the method further includes: the first device receives the quantization bit number configuration from the second device.
  • the method further includes: the first device sends third indication information to the second device, where the third indication information indicates a plurality of quantization bit numbers.
  • the third indication information may implicitly or indirectly indicate multiple quantization bit numbers through indexes in the quantization bit number configuration.
  • the first device can send the determined number of quantization bits to the second device, so that the second device can determine the number of quantization bits of each group based on the index, and ensure correct inverse quantization of parameters by the second device.
  • the number of quantization bits is determined by the second device, and the method further includes: the first device receives fourth indication information from the second device, and the fourth indication information indicates the The number of quantized bits corresponding to each group.
  • the number of quantization bits is indicated by the second device, so that the first device can perform parameter quantization directly based on the indication.
  • the operation of the first device is simplified, and the processing efficiency is improved.
  • the dividing the parameters of the distributed training model into multiple groups by the first device includes: dividing the parameters of the distributed training model into multiple groups by the first device using a model parameter division manner.
  • the first device determines the model parameter division method based on the values of the parameters of the distributed training model; and the first device sends fifth instruction information to the second device, and the fifth The instruction information indicates how the model parameters are divided.
  • the first device can determine the parameter division method based on the parameter characteristics, so that different quantization processes can be performed subsequently for different groups, which can ensure compression performance.
  • the method further includes: the first device receives sixth indication information from the second device, where the sixth indication information indicates a model parameter division manner.
  • the second device can indicate the parameter division manner, so that the first device can perform parameter division directly based on the indication.
  • the operation of the first device is simplified, and the processing efficiency is improved.
  • the model parameter division mode indicates that the division is based on at least one of the following: the type of the parameter, the type of the network layer where the parameter is located, and the like.
  • the division of parameters can be achieved by taking into account different characteristics of the parameters, so that parameters located in the same group can be quantified similarly.
  • the first device further includes: the first device performs entropy encoding on multiple quantization groups; and sending the multiple quantization groups from the first device to the second device includes: sending the entropy encoded Encoded multiple quantization groups.
  • the first device can further improve the compression rate through entropy coding, so that the communication efficiency can be further improved.
  • the method further includes: the first device sends first entropy encoding indication information to the second device, the first entropy encoding indication information having entropy encoded multiple quantization groups.
  • the second device can know whether entropy coding is performed, so that it can correctly decode the received parameters.
  • the method further includes: the first device receives the second entropy coding indication information from the second device; and performing entropy coding on the multiple quantization groups by the first device includes: the first device based on the second entropy Encoding indication information, performing entropy encoding on multiple quantization groups.
  • the first device can perform entropy encoding based on the instruction of the second device, which can further improve the compression rate.
  • a data processing method includes: the second device receives multiple quantization groups from the first device; the second device determines multiple groups corresponding to the multiple quantization groups based on the number of quantization bits corresponding to each of the multiple quantization groups, The number of quantized bits is determined based on at least one of the following: the amount of transmission resources between the first device and the second device, the value of the loss function, or the value range of the parameters of each group; and the second device based on the model parameters The division method and multiple groups determine the parameters of the distributed training model.
  • the first quantization group of the plurality of quantization groups includes a quantization boundary value and a plurality of quantization parameters.
  • the quantization boundary value includes at least one of the following: a maximum value, or a minimum value, the maximum value being the maximum value among the plurality of first parameters in the first group corresponding to the first quantization group or The maximum value is the maximum value of the multiple absolute values of the multiple first parameters, and the minimum value is the minimum value among the multiple first parameters or the minimum value is the minimum value of the multiple absolute values of the multiple first parameters.
  • the second device further includes: the second device sends first indication information to the first device, and the first indication information indicates whether to determine the quantization boundary value according to an absolute value; or, the second device receives the information from the first Second indication information of the device, where the second indication information indicates whether the quantization boundary value is determined according to an absolute value.
  • each quantization group has the same number of quantization bits.
  • the plurality of quantization groups includes a first quantization group and a second quantization group, and the number of quantization bits corresponding to the first quantization group is not equal to the number of quantization bits corresponding to the second quantization group.
  • the method further includes: the second device receives third indication information from the first device, where the third indication information indicates a plurality of quantization bit numbers.
  • the second device sends the quantization bit number configuration to the first device, where the quantization bit number configuration includes a quantization threshold and a corresponding quantization bit number.
  • the method further includes: the second device sending fourth indication information to the first device, where the fourth indication information indicates the number of quantization bits corresponding to each of the multiple groups.
  • the method further includes: the second device receives fifth indication information from the first device, where the fifth indication information indicates a model parameter division manner.
  • the method further includes: the second device sending sixth indication information to the first device, where the sixth indication information indicates a model parameter division manner.
  • the model parameter division mode indicates that the division is based on at least one of the following: the type of the parameter, the type of the network layer where the parameter is located, and the like.
  • the second device receiving the plurality of quantization groups from the first device includes: the second device receiving the plurality of entropy encoded quantization groups.
  • the method further includes: the second device receives first entropy encoding indication information from the first device, the first entropy encoding indication information having entropy encoded multiple quantization groups.
  • the method further includes: the second device sends second entropy encoding indication information to the first device, so as to instruct the first device to perform entropy encoding on multiple quantization groups.
  • a communication device configured to include: a division module, configured to divide the parameters of the distributed training model into multiple groups, at least two of the multiple groups have different numerical intervals; a determination module, configured to be based on the parameters of the multiple groups The number of quantized bits corresponding to each group determines multiple quantized groups corresponding to multiple groups, and the number of quantized bits is determined based on at least one of the following: the amount of transmission resources between the communication device and the second device, and the loss function value, or a range of values of parameters of each group; and a sending module configured to send multiple quantization groups to the second device.
  • the determining module is configured to determine the quantization boundary value corresponding to the first group based on the plurality of first parameters in the first group of the plurality of groups.
  • the quantization boundary value includes at least one of the following: a maximum value, or a minimum value, where the maximum value is the maximum value of the multiple first parameters or the maximum value is a multiple of the multiple first parameters.
  • the maximum value of the absolute value, the minimum value is the minimum value among the multiple first parameters or the minimum value is the minimum value of the multiple absolute values of the multiple first parameters.
  • the quantization group corresponding to the first group includes quantization boundary values and a plurality of quantization parameters corresponding to the plurality of first parameters.
  • it further includes a receiving module configured to receive first indication information from the second device, where the first indication information indicates whether to determine the quantization boundary value according to an absolute value. And the determination module is configured to determine the quantization boundary value based on the first indication information.
  • the sending module is configured to send second indication information to the second device, where the second indication information indicates whether the quantization boundary value is determined according to an absolute value.
  • the number of quantization bits for each group is equal.
  • the plurality of groups includes a first group and a second group, and the number of quantization bits corresponding to the first group is not equal to the number of quantization bits corresponding to the second group.
  • the determining module is configured to determine a plurality of quantization bit numbers corresponding to a plurality of groups.
  • the determination module is configured to determine a plurality of quantization bit numbers corresponding to a plurality of groups based on the quantization bit number configuration from the second device, the quantization bit number configuration including a quantization threshold and a corresponding quantization bit number number of bits.
  • the receiving module is configured to receive the quantization bit number configuration from the second device.
  • the sending module is configured to send third indication information to the second device, where the third indication information indicates a plurality of quantization bit numbers.
  • the number of quantized bits is determined by the second device, and the receiving module is configured to receive fourth indication information from the second device, where the fourth indication information indicates that each group in the plurality of groups The corresponding number of quantized bits.
  • the division module is configured to divide the parameters of the distributed training model into multiple groups using a model parameter division manner.
  • the determining module is configured to determine the model parameter division method based on the values of the parameters of the distributed training model; and the sending module is configured to send fifth indication information to the second device, the first The five indication information indicates the division mode of the model parameters.
  • the receiving module is configured to receive sixth indication information from the second device, where the sixth indication information indicates a model parameter division manner.
  • the model parameter division mode indicates that the division is based on at least one of the following: the type of the parameter, the type of the network layer where the parameter is located, and the like.
  • an entropy encoding module configured to entropy encode the plurality of quantization groups; and the sending module configured to send the entropy encoded plurality of quantization groups to the second device.
  • the sending module is configured to send first entropy encoding indication information to the second device, where the first entropy encoding indication information has entropy encoded a plurality of quantization groups.
  • the receiving module is configured to receive second entropy encoding indication information from the second device; and the entropy encoding module is configured to perform entropy entropy on multiple quantization groups based on the second entropy encoding indication information coding.
  • a communication device configured to include: a receiving module configured to receive a plurality of quantization groups from the first device; a first determination module configured to determine the number of quantization bits corresponding to each quantization group in the plurality of quantization groups multiple quantization groups corresponding to multiple groups, the number of quantization bits is determined based on at least one of the following: the amount of transmission resources between the first device and the communication device, the value of the loss function, or the value range of the parameters of each group ; and a second determination module configured to determine parameters of the distributed training model based on the model parameter division method and multiple groups.
  • the first quantization group of the plurality of quantization groups includes a quantization boundary value and a plurality of quantization parameters.
  • the quantization boundary value includes at least one of the following: a maximum value, or a minimum value, and the maximum value is the maximum value of a plurality of first parameters in the first group corresponding to the first quantization group or The maximum value is the maximum value of the multiple absolute values of the multiple first parameters, and the minimum value is the minimum value among the multiple first parameters or the minimum value is the minimum value of the multiple absolute values of the multiple first parameters.
  • it further includes a sending module configured to send first indication information to the first device, where the first indication information indicates whether to determine the quantization boundary value according to an absolute value.
  • the receiving module is configured to receive second indication information from the first device, the second indication information indicating whether the quantization boundary value is determined according to an absolute value.
  • the number of quantization bits of each quantization group is equal.
  • the plurality of quantization groups includes a first quantization group and a second quantization group, and the number of quantization bits corresponding to the first quantization group is not equal to the number of quantization bits corresponding to the second quantization group.
  • the receiving module is configured to receive third indication information from the first device, where the third indication information indicates a plurality of quantization bit numbers.
  • the sending module is configured to send the quantization bit number configuration to the first device, where the quantization bit number configuration includes a quantization threshold and a corresponding quantization bit number.
  • the sending module is configured to send fourth indication information to the first device, where the fourth indication information indicates the number of quantized bits corresponding to each of the multiple groups.
  • the receiving module is configured to receive fifth indication information from the first device, where the fifth indication information indicates a model parameter division manner.
  • the sending module is configured to send sixth indication information to the first device, where the sixth indication information indicates a model parameter division manner.
  • the model parameter division mode indicates that the division is based on at least one of the following: the type of the parameter, the type of the network layer where the parameter is located, and the like.
  • the receiving module is configured to receive the entropy encoded plurality of quantization groups.
  • the receiving module is configured to receive first entropy encoding indication information from the first device, the first entropy encoding indication information having entropy encoded a plurality of quantization groups.
  • the sending module is configured to send the second entropy encoding indication information to the first device, so as to instruct the first device to perform entropy encoding on the multiple quantization groups.
  • a communication device in a fifth aspect of the present disclosure, includes a processor and a memory, and the memory stores instructions executed by the processor, and when the instructions are executed by the processor, the communication device implements the operation of the method in the aforementioned first aspect or any embodiment thereof , or implement the operations according to the method in the above second aspect or any embodiment thereof.
  • the communication device further includes a transceiver for receiving information from another device, sending information to another device, and the like.
  • a computer-readable storage medium on which computer-executable instructions are stored.
  • the above-mentioned first aspect or its The operation of the method in any embodiment, or the operation of the method according to the above second aspect or any embodiment thereof.
  • a chip or a chip system in a seventh aspect of the present disclosure, includes a processing circuit configured to perform operations according to the method in the above first aspect or any embodiment thereof, or perform operations according to the method in the above second aspect or any embodiment thereof.
  • a computer program or computer program product is provided.
  • the computer program or computer program product is tangibly stored on a computer-readable medium and comprises computer-executable instructions which, when executed, implement the operations according to the method in the above-mentioned first aspect or any of its embodiments, Or implement the operations according to the method in the above second aspect or any embodiment thereof.
  • FIG. 1 shows a schematic diagram of an example scenario where an embodiment of the present disclosure can be applied
  • Fig. 2 shows a signaling interaction diagram of an information processing process according to some embodiments of the present disclosure
  • Fig. 3 shows a schematic diagram of parameter division according to some embodiments of the present disclosure
  • Fig. 4 shows a schematic diagram of parameter division according to some embodiments of the present disclosure
  • Fig. 5 shows a schematic diagram of parameter division according to some embodiments of the present disclosure
  • Fig. 6 shows a signaling interaction diagram of an information transmission process according to some embodiments of the present disclosure
  • FIG. 7 shows a schematic diagram of a compressed configuration message according to some embodiments of the present disclosure
  • Fig. 8 shows a schematic diagram of compressed data according to some embodiments of the present disclosure
  • FIG. 9 shows a signaling interaction diagram of an information transmission process according to some embodiments of the present disclosure.
  • Fig. 10 shows a schematic diagram of a quantization indication message according to some embodiments of the present disclosure
  • Fig. 11 shows a schematic diagram of compressed data according to some embodiments of the present disclosure
  • Fig. 12 shows a schematic block diagram of a communication device according to some embodiments of the present disclosure
  • Figure 13 shows a schematic block diagram of a communication device according to some embodiments of the present disclosure.
  • Fig. 14 shows a schematic block diagram of an example device that may be used to implement embodiments of the present disclosure.
  • the term “comprising” and its similar expressions should be interpreted as an open inclusion, that is, “including but not limited to”.
  • the term “based on” should be understood as “based at least in part on”.
  • the term “one embodiment” or “the embodiment” should be read as “at least one embodiment”.
  • the terms “first”, “second”, etc. may refer to different or the same object.
  • the term “plurality” means two or more. Other definitions, both express and implied, may also be included below.
  • Embodiments of the present disclosure may be implemented according to appropriate communication protocols, including but not limited to, third generation (3rd Generation, 3G), fourth generation (4G), fifth generation (5G), sixth generation (6G), etc.
  • Cellular communication protocols including but not limited to, third generation (3rd Generation, 3G), fourth generation (4G), fifth generation (5G), sixth generation (6G), etc.
  • Cellular communication protocols wireless local area network communication protocols such as Institute of Electrical and Electronics Engineers (Institute of Electrical and Electronics Engineers, IEEE) 802.11, future evolved mobile communication systems such as 6G communication system protocols, and/or currently known or future developed other agreements.
  • the technical solution of the embodiment of the present disclosure is applied to a communication system following any appropriate communication protocol, such as: Universal Mobile Telecommunications System (Universal Mobile Telecommunications Service, UMTS), Long Term Evolution (Long Term Evolution, LTE) system, wideband code division multiple access System (Wideband Code Division Multiple Access, WCDMA), Code Division Multiple Access 2000 system (Code Division Multiple Access, CDMA2000), Time Division Synchronization Code Division Multiple Access system (Time Division-Synchronization Code Division Multiple Access, TD-SCDMA), frequency division Duplex (Frequency Division Duplex, FDD) system, Time Division Duplex (Time Division Duplex, TDD), fifth generation (5G) system or new radio (New Radio, NR), future evolution of 6G, 7G communication systems, etc. .
  • Universal Mobile Telecommunications System Universal Mobile Telecommunications Service, UMTS
  • Long Term Evolution Long Term Evolution, LTE
  • WCDMA Wideband Code Division Multiple Access
  • CDMA2000 Code Division Multiple Access 2000 system
  • WLAN wireless local area network
  • wired communication system any communication system with similar problems, such as a Wi-Fi network, a Wi-Fi network, or other communication systems developed in the future.
  • terminal device refers to any terminal device capable of wired or wireless communication with network devices or with each other.
  • the terminal equipment may sometimes be called user equipment (User Equipment, UE).
  • a terminal device may be any type of mobile terminal, stationary terminal or portable terminal.
  • terminal equipment may include a mobile handset, station, unit, device, mobile terminal (Mobile Terminal, MT), subscription station, portable subscription station, Internet node, communicator, desktop computer, laptop computer, notebook computer, tablet Computers, personal communication system devices, personal navigation devices, personal digital assistants (Personal Digital Assistant, PDA), positioning devices, radio broadcast receivers, e-book devices, game devices, Internet of Things (IoT) devices, vehicle-mounted devices , aircraft, virtual reality (Virtual Reality, VR) devices, augmented reality (Augmented Reality, AR) devices, wearable devices, terminal devices in 5G networks or evolved public land mobile networks (Public Land Mobile Network, PLMN) Any terminal device, other device that can be used for communication, or any combination of the above. Embodiments of the present disclosure do not limit this.
  • the term "network device” used in this disclosure is an entity or node that can be used to communicate with a terminal device, such as an access network device.
  • the access network device may be a device deployed in the radio access network to provide a wireless communication function for the mobile terminal, for example, it may be a radio access network (Radio Access Network, RAN) network device.
  • Access network equipment may include various types of base stations.
  • the access network equipment may include various forms of macro base stations, micro base stations, pico base stations, femto base stations, relay stations, access points, satellites, remote radio units (Remote Radio Unit, RRU), radio head (Radio Head , RH), remote radio head (Remote Radio Head, RRH), etc., can also be a device that undertakes the base station function in device-to-device (D2D) communication and machine-to-machine (M2M) communication.
  • D2D device-to-device
  • M2M machine-to-machine
  • the names of access network equipment may be different, for example, in a Long Term Evolution (LTE) network, it is called an evolved NodeB (evolved NodeB, eNB or eNodeB), which is called Node B (NodeB, NB) in 3G network, can be called gNode B (gNB) or NR Node B (NR NB) in 5G network, and so on.
  • the access network device may include a central unit (Central Unit, CU) and/or a distributed unit (Distributed Unit, DU).
  • CU and DU can be placed in different places, for example: DU is remote and placed in a high-traffic area, and CU is placed in the central computer room.
  • the CU and DU can also be placed in the same equipment room.
  • the CU and DU can also be different components under one rack.
  • the above-mentioned apparatuses for providing wireless communication functions for mobile terminals are collectively referred to as network devices, which are not specifically limited in the embodiments of the present disclosure.
  • Federated learning was proposed in 2016. It is essentially a distributed learning framework, which aims to realize the joint construction of a global model on the basis of ensuring data privacy and security. Federated learning allows users to form a federation on the premise that the data is kept locally and not shared, and the global model is obtained through joint training, thereby avoiding data privacy and security issues.
  • the embodiments of the present disclosure provide a parameter compression scheme, which can divide the parameters into multiple groups and quantize each group to compress the parameters, so that the performance of the data can be guaranteed, and the parameters can be compressed. Improve communication efficiency.
  • FIG. 1 shows a schematic diagram of an example scenario 100 in which embodiments of the present disclosure can be applied.
  • the scene 100 includes a first device 110 - 1 to a first device 110 -N and a second device 120 .
  • the first device 110 - 1 to the first device 110 -N may be collectively referred to as the first device 110 .
  • the first device 110 may be implemented as a client device, and the second device 120 may be implemented as a server device. In some examples, the first device 110 may be implemented as a terminal device, and the second device 120 may be implemented as an access network device. In some examples, the first device 110 may be called an execution node in distributed machine learning, and the second device 120 may be called an execution anchor in distributed machine learning. This disclosure is not limited thereto.
  • the second device 120 can send the initial model parameters of the neural network to N1 first devices 110, for example, in the form of broadcast or multicast .
  • the N1 first devices 110 may perform learning, such as training or iteration, on the basis of initial model parameters based on the local data set, so as to obtain processed parameters, which may include, for example, local model parameters or gradient parameters.
  • Each of the N1 first devices 110 compresses the obtained processed parameters and sends them to the second device 120 .
  • the second device 120 decompresses the processed parameters from each of the N1 first devices 110 , and obtains updated model parameters through aggregation. In this way, the model parameters of the neural network at the second device 120 are changed from the initial model parameters to the updated model parameters.
  • the next round of training or learning can be performed to obtain further updated model parameters. That is to say, the second device 120 may send the updated model parameters to the N2 first devices 110 , and the N2 first devices 110 obtain reprocessed parameters and send them back to the second device 120 . This is repeated until the parameters converge, or until the training fails, such as reaching the maximum number of iterations.
  • the parameters sent by the second device 120 to the first device 110 may be collectively referred to as the parameters to be trained, such as the initial model parameters of the neural network, or the aggregated and updated parameters obtained after the last round of training. Model parameters.
  • the same or different first devices 110 may be equal or unequal, for example, the above-mentioned N1 first devices 110 and N2 first devices 110 They may be completely different, partly the same, or all of them may be the same.
  • the N1 first devices 110 include the first device 110-1, and the N2 first devices 110 include or do not include the first device 110-1.
  • the second device 120 receives compressed parameters from one or more first devices 110 in each round.
  • the compression scheme used by each first device 110 is fixed. In some other embodiments, in multiple rounds of interaction, the compression scheme used by each first device 110 may be dynamically adjusted, for example, in an intermediate round of interaction, the compression scheme may be updated based on the attribute of the parameter. In order to simplify the description, in the following embodiment, one compression by the first device 110 is taken as an example for illustration, but it can be understood that the processes for each compression in multiple rounds of interaction are similar.
  • Fig. 2 shows a signaling interaction diagram of an information processing procedure 200 according to some embodiments of the present disclosure.
  • the process 200 involves a first device 110 and a second device 120, and the first device 110 in FIG. 3 refers to any one of the first device 110-1 to the first device 110-N in FIG. 1 .
  • the first device 110 divides 210 the parameters of the distributed training model into a plurality of groups.
  • Each group includes multiple parameters, and the number of parameters in different groups may be the same or different.
  • the first group includes M1 parameters
  • the second group includes M2 parameters, and M1 and M2 may or may not be equal.
  • the values of multiple parameters in different groups can be in the same interval or different intervals. Taking the first group among multiple groups as an example, the first group includes M1 parameters, then the value interval of the first group can be determined based on the minimum value of the values of the M1 parameters and the maximum value of the values of the M1 parameters , for example, it can be assumed to be a closed interval [a1,b1].
  • the value range of the second group can be determined, for example, it can be assumed to be [a2, b2].
  • a1 ⁇ a2 and/or b1 ⁇ b2, that is, the value range of the first group is different from the value range of the second group.
  • the first group here may be any one of the multiple groups, and the second group may be any one of the multiple groups that is different from the first group. Exemplarily, at least two of the multiple groups have different value intervals.
  • the closed interval [a1, b1] of this application can also be an open interval (a1, b1), an open and closed interval (a1, b1], or a closed and open interval [a1, b1), which does not affect the essence of this application.
  • [a1, b1] is taken as an example in this application, and [a2, b2] is also similar.
  • any two groups in the multiple groups have the following differences: the value interval of the value of the parameter, the type of the parameter, or the type of the network layer to which the parameter belongs, where the type of the parameter is, for example, Weight, bias, variance, mean, etc., where the type of network layer to which the parameter belongs is, for example, a convolutional layer, a fully connected layer, etc.
  • the parameters of the distributed training model may be that the first device 110 utilizes a local (that is, stored in the memory of the first device 110) data set, based on the model parameters from the last round of the second device 120, after learning or training owned.
  • the first device 110 may divide the parameters of the distributed training model into multiple groups based on the parameter division manner.
  • the parameter division method may indicate that the parameter division is based on at least one of the following: the type of the parameter, the type of the network layer where the parameter is located, and the like.
  • Types of network layers include, for example, convolutional layers, fully connected layers, and the like.
  • Types of parameters include, for example, weight parameters, bias parameters, mean parameters, and the like.
  • FIG. 3 shows a schematic diagram of parameter division 300 according to some embodiments of the present disclosure.
  • parameters representing weights in each convolutional layer may be divided into a group, such as group 310 .
  • the parameters representing the bias in each convolutional layer are divided into a group, such as group 320 .
  • Parameters representing weights in each fully connected layer are divided into a group, such as group 330 .
  • the parameter division manner described with reference to FIG. 3 may be called a type-based division manner.
  • FIG. 4 shows a schematic diagram of parameter division 400 according to some embodiments of the present disclosure.
  • parameters representing weights in the convolutional layer 401 can be divided into a group, such as group 410 .
  • Parameters representing biases in the convolutional layer 401 are divided into groups, such as group 420 .
  • Parameters representing weights in the convolutional layer 402 are divided into groups, such as group 430 .
  • the parameters representing biases in the convolutional layer 402 are divided into groups, such as group 440 .
  • the parameter division manner described with reference to FIG. 4 may be referred to as a matrix-based division manner.
  • FIG. 5 shows a schematic diagram of parameter division 500 according to some embodiments of the present disclosure.
  • the parameters in the convolutional layer 501 can be divided into a group, such as group 510 .
  • the parameters in batch normalization layer 502 are divided into groups, such as group 520 .
  • the parameter division manner described with reference to FIG. 5 may be referred to as a network layer-based division manner.
  • the parameter division method may also indicate that the parameters are divided according to other characteristics of the parameters, which will not be listed here.
  • the parameter division method may be determined by the second device 120 and then indicated to the first device 110, or the parameter division method may be determined by the first device 110, as follows in conjunction with the embodiments of FIG. 6 to FIG. 11 Elaborate.
  • the first device 110 determines 220 a plurality of quantization groups corresponding to the plurality of groups based on the number of quantization bits corresponding to each of the plurality of groups.
  • the number of quantization bits may be used to indicate the length of each quantization parameter in the corresponding quantization group, that is, the number of bits occupied.
  • the storage space occupied by multiple quantization groups is smaller than the storage space occupied by multiple quantization groups, for example, it may be 50% or other values, so as to realize the compression of multiple groups.
  • Each quantization group may include a plurality of quantization parameters, wherein the length of each quantization parameter in the plurality of quantization parameters is equal to or smaller than the corresponding number of quantization bits.
  • the first group among multiple groups as an example, assuming that the first group includes M1 parameters, and the number of quantization bits corresponding to the first group is n1, then the first quantization group corresponding to the first group includes M1 quantization bits parameter, and the number of bits of each quantization parameter in the M1 quantization parameters may be equal to n1.
  • Each quantization group may include a quantization boundary value and a plurality of quantization parameters, wherein the quantization boundary value may be determined based on the value range of a plurality of parameters in the corresponding group, wherein each quantization parameter in the plurality of quantization parameters The length is equal to or less than the corresponding number of quantization bits.
  • the first device 110 may determine a quantization boundary value determination manner for each of the multiple groups. In one example, the first device 110 may determine a quantization boundary value determination manner, and the quantization boundary value determination manner is applied to each group. In another example, for each of the multiple groups, the first device 110 may respectively determine a corresponding quantization boundary value determination manner.
  • the determination method of the quantization boundary value may indicate whether to determine the quantization boundary value according to an absolute value.
  • determining the quantization boundary value not according to the absolute value includes: determining the maximum and minimum values of the multiple parameters in the group as the quantization boundary value. Taking the first group among the multiple groups as an example, the first group includes M1 parameters, and the value interval of the M1 parameters is a closed interval [a1, b1], then the quantization boundary value can be determined correspondingly including a1 and b1 .
  • determining the quantization boundary value according to the absolute value includes: determining the maximum and minimum values of the absolute values of the multiple parameters in the group as the quantization boundary value; or determining the maximum value of the absolute values of the multiple parameters in the group as Quantize boundary values.
  • the first group includes M1 parameters, and the value range of M1 parameters is a closed interval [a1, b1]. If a1 and b1 are both positive values, then optional It can be determined that the quantization boundary value includes a1 and b1; if a1 is a negative value and b1 is a positive value, then optionally the quantization boundary value can be determined to include max(-a1, b1), where max means taking the maximum value.
  • the quantization boundary value determination method that does not determine the quantization boundary value according to the absolute value can be called the traditional quantization method, and the quantization boundary value determination method that determines the quantization boundary value according to the absolute value can be called the symbol quantization method, but should It is understood that this designation is only indicative and should not be construed as limiting the embodiments of the present disclosure.
  • the method of determining the quantization boundary value may also indicate to determine according to other value characteristics of the parameters, which will not be listed here.
  • the quantization boundary value determination method may be determined by the second device 120 and then indicated to the first device 110, or the quantization boundary value determination method may be determined by the first device 110, or may be determined by the first device 110 is determined based on the configuration of the second device 120 , which will be described in detail below in conjunction with the embodiments of FIG. 6 to FIG. 11 .
  • the number of quantization bits for each group may be equal.
  • the first device 110 or the second device 120 may determine the number of quantization bits and use the number of quantization bits for each group. In this way, the lengths of the individual quantization parameters in each quantization group are equal.
  • at least two of the plurality of groups have different numbers of quantization bits.
  • the first device 110 or the second device 120 may respectively determine the corresponding number of quantization bits for each group. In this way, the lengths of quantization parameters located in different quantization groups may or may not be equal.
  • the first group includes M1 parameters, and the number of quantization bits corresponding to the first group is n1
  • the first quantization group corresponding to the first group includes M1 quantization parameters, and each of the M1 quantization parameters is quantized
  • the number of bits of the parameter may be equal to n1.
  • the second group includes M2 parameters, and the number of quantization bits corresponding to the second group is n2, then the second quantization group corresponding to the second group includes M2 quantization parameters, and the bit of each quantization parameter in the M2 quantization parameters
  • the number can be equal to n2.
  • each quantization parameter may include at least one bit for indicating the sign of the corresponding parameter.
  • the quantization boundary value determination method of a certain quantization group is the symbol quantization method (that is, the quantization boundary value is determined according to the absolute value)
  • the number of bits of the quantization parameter in the quantization group is equal to n1
  • n1 bits Any bit in (for example, the first bit) indicates its positive or negative, and the remaining n1-1 bits indicate the parameter size.
  • the number of quantized bits may be determined by the second device 120 and then indicated to the first device 110, or the number of quantized bits may be determined by the first device 110, or may be determined by the first device 110 It is determined based on the configuration of the second device 120 , which is described in detail below in conjunction with the embodiments of FIG. 6 to FIG. 11 .
  • the first device 110 may perform entropy encoding 222 on multiple quantization groups.
  • entropy encoding may be performed based on an indication from the second device 120 .
  • the first device 110 may decide to perform entropy encoding by itself and notify the second device 120 .
  • the first device 110 sends 230 quantization groups to the second device 120 .
  • the multiple quantization groups that are sent are entropy coded multiple quantization groups.
  • the second device 120 can receive multiple quantization groups from the first device 110 . It can be understood that, in some examples, the second device 120 previously sent the model parameters of the last round to multiple first devices 110, then correspondingly, the second device 120 can receive data from each of the multiple first devices 110. 110 multiple quantized groups sent.
  • the second device 120 determines 240 a plurality of groups corresponding to the plurality of quantization groups.
  • the second device 120 may perform inverse quantization based on the quantization manner and the number of quantization bits, so as to obtain multiple groups.
  • the second device 120 determines 250 model parameters.
  • the second device 120 may determine a position in the network model where each parameter in the dequantized multiple groups is located based on a parameter division manner.
  • the second device 120 can obtain the model parameters obtained through the current round of training.
  • the second device 120 may obtain the parameters to be trained in the next round by aggregation based on model parameters from multiple first devices.
  • the subsequent process is similar to the above and will not be repeated in this article.
  • FIG. 6 shows a signaling interaction diagram of an information transmission process 600 according to some embodiments of the present disclosure.
  • the process 600 involves the first device 110 and the second device 120, and the first device 110 in FIG. 6 may include any one of the first device 110-1 to the first device 110-N in FIG. 1 .
  • the second device 120 determines 610 to compress the configuration message.
  • the compressed configuration message includes first indication information, and the first indication information indicates a parameter division manner.
  • the second device 120 may determine a parameter division manner.
  • the second device 120 may determine the parameter division method based on at least one of the following factors: parameter attributes of the parameters to be trained, network structure of the model to be trained, communication efficiency, and model performance.
  • the parameters to be trained may be initial model parameters, or may be updated model parameters obtained based on a previous round of training.
  • Parameter attributes may include: the value range of the parameter, the type of the parameter, the network layer where the parameter is located, etc., where the type of the parameter includes, for example, weight, bias, variance, mean, etc.
  • the network structure may include the type of the network layer in the model, the number of the network layer, etc., where the type of the network layer includes, for example, a convolutional layer, a fully connected layer, and the like.
  • the parameter division manner determined by the second device 120 may be a type-based division manner as shown in FIG. 3 or a matrix-based division manner as shown in FIG. 4 or a network layer-based division manner as shown in FIG. Divide the way.
  • the first indication information may be indicated by 1 bit, for example, "0” indicates a matrix-based division manner, and "1" indicates a type-based division manner. In some examples, the first indication information may be indicated by 2 bits, for example, "00” indicates a matrix-based division manner, “01” indicates a type-based division manner, and "10” indicates a network layer-based division manner. In some other examples, the first indication information may also be indicated by using more bits, which is not limited in the present disclosure.
  • the signaling overhead of the type-based partitioning method is smaller than that of the matrix-based partitioning method, but the type-based partitioning method has The model performance is not as good as that of the matrix-based partitioning approach. In practice, a better compromise can be made based on communication efficiency and model performance to determine the appropriate parameter division method.
  • the number of parameters to be trained is s
  • it can be divided into r1 groups according to the type-based division method, and can be divided into r2 groups according to the matrix-based division method. Then if (r2-r1) ⁇ 32 ⁇ 2 ⁇ s ⁇ d is satisfied, then the parameter division method is determined to be the matrix-based division method. On the contrary, if (r2-r1) ⁇ 32 ⁇ 2 ⁇ s ⁇ d is satisfied, it is determined that the parameter division method is a type-based division method.
  • s, r1 and r2 are all positive integers, 32 represents the number of bits occupied by floating point numbers in the computer system, d is a predetermined value, for example, d can be the number of bits of two different parameter division methods determined by the user based on empirical information difference.
  • the number of parameters to be trained is s
  • s/r1 is smaller than the preset threshold, it is determined that the parameter division method is the type-based division method.
  • s/r1 is greater than or equal to the preset threshold, it is determined that the parameter division method is a matrix-based division method.
  • the preset threshold may be predefined by the user, or the preset threshold may be determined based on a historical division manner, for example equal to the maximum value of the number of average parameters of each group in the historically trained model.
  • a threshold value of the number of parameters in each group after division can also be set
  • the number of parameters in each group is between a first number threshold and a second number threshold, and the first number threshold is smaller than the second number threshold.
  • group 1 may be further divided into at least two groups. In this way, the performance loss caused by an excessive number of parameters in a certain group can be avoided.
  • group 2 may be merged with other groups. In this way, compression efficiency can be further improved, and excessive signaling overhead can be avoided.
  • the compression configuration message may include second indication information. That is to say, the compression configuration message includes first indication information and second indication information.
  • the second indication information may indicate a quantization boundary value determination mode (or quantization mode for short).
  • the second device 120 may determine a quantization boundary value determination manner, and the quantization boundary value determination manner will be used for each group, that is, the quantization boundary value determination manner of each group is the same.
  • the second device 120 may use the quantization manner most frequently used by other models trained in history as the quantization boundary value determination manner.
  • the second device 120 may determine the manner of determining the quantization boundary value based on the value distribution of the parameter. For example, if the value distribution of the parameter is uneven, it can be determined to use the traditional quantization method, that is, to consider the value of positive and negative numbers, which can reduce the error. For example, multiple parameters in a certain group are all positive or negative, for example, the difference between the maximum value (positive) and the minimum value (negative) in a certain group is large.
  • the parameter is the gradient of the reverse update or the residual parameter of the model, the sign of the parameter is more important at this time, then it can be determined to use the symbol quantization method, that is, quantization based on the absolute value, so as to ensure that each parameter remains Correct update direction to avoid inaccurate gradient direction of parameters near 0.
  • the symbol quantization method that is, quantization based on the absolute value
  • the minimum value of the parameter value may be 0 or close to 0.
  • the quantization boundary value may include the maximum value of the absolute value but not the minimum value of the absolute value, so that Save signaling overhead.
  • the second indication information may be indicated by 1 bit, for example, "0" indicates a traditional quantization mode, and "1" indicates a symbol quantization mode.
  • the second indication information may also be indicated by more bits, for example, it may be indicated by 2 bits, wherein one bit indicates that multiple groups use the same quantization method, and the other bit indicates the quantization method used by each group way, which is not limited in the present disclosure.
  • the second indication information may indicate the quantization mode configuration, and the second indication information may be used by the first device 110 to determine the quantization mode.
  • the second indication information may be indicated by 1 bit, for example, "0" indicates that multiple groups use the same quantization mode, and "1" indicates that a quantization mode is determined for each group.
  • the second indication information may be indicated by m1 bits, for example, one bit (such as the first bit) is used to indicate whether each group uses the same quantization method, and the other m1-1 bits indicate whether to use Used to determine the threshold of the quantization method.
  • the second indication information may be indicated by m1-1 bits, indicating a threshold for determining a quantization manner. Exemplarily, for a specific embodiment of determining the quantization mode based on the configuration of the quantization mode, reference may be made to the description in conjunction with 650 below.
  • the compression configuration message further includes third indication information.
  • the compression configuration message may include first indication information, second indication information, and third indication information.
  • the third indication information may indicate the number of quantization bits.
  • the second device 120 may determine the number of quantization bits, and the number of quantization bits will be used for each group, that is, the number of quantization bits for each group is the same.
  • the number of quantization bits used for all groups may be referred to as the number of round quantization bits or the number of coarse-grained quantization bits.
  • the second device 120 may determine the number of quantization bits based on transmission resources between the first device 110 and the second device 120 and/or the number of network layers of the distributed neural network.
  • the second device 120 may allocate transmission resources from the first device 110 to the second device 120 for the first device 110, and determine the number of quantization bits based on the allocated transmission resources.
  • the third indication information may be indicated by m2 bits, which may indicate that the maximum number of quantization bits is 2 m2 . In some other examples, the third indication information may also be indicated by more or less bits, which is not limited in the present disclosure.
  • the third indication information may indicate the configuration of the number of quantization bits, and the third indication information may be used by the first device 110 to determine the number of quantization bits.
  • the quantization bit number configuration may indicate quantization granularity and a quantization threshold and quantization bit number list for determining the quantization bit number, wherein the quantization granularity may indicate whether multiple groups use the same quantization bit number.
  • the quantization granularity may indicate that multiple groups use the same quantization bit number, for example, the round quantization bit number or the coarse-grained quantization bit number.
  • the quantization granularity may indicate a coarse granularity.
  • the quantization granularity may indicate that multiple groups use different quantization bit numbers, for example, a fine-grained quantization bit number, and optionally, the quantization granularity may indicate fine granularity.
  • quantization granularity may indicate a combination of coarse and fine granularity.
  • the quantization bit number configuration may include a quantization threshold number and a quantization threshold list, where the quantization threshold number is used to indicate the number of quantization thresholds in the quantization threshold list, and the quantization threshold list may include multiple quantization thresholds.
  • the number of quantization thresholds is K, and K is a positive integer, and it is assumed that the list of quantization thresholds includes (T 1 , T 2 , . . . , T K ).
  • the configuration of quantization bit numbers may also include a list continuity indication and a quantization list, wherein the list continuity indication is used to indicate the representation form of the quantization list, and the quantization list includes one or more quantization bit numbers.
  • the list continuity indication may be represented by 1 bit.
  • the quantization list may include (b 0 , b 1 , b 2 , . . . , b K ). If continuous is indicated, the quantization list may include b 0 and/or b K .
  • quantization granularity may indicate a combination of coarse and fine granularity.
  • the quantization bit number configuration may include the first quantization threshold number corresponding to the coarse grain, the first quantization threshold list, the first list continuity indication and the first quantization list, and the quantization bit number configuration further includes the second quantization threshold corresponding to the fine grain Quantity, Second Quantization Threshold List, Second List Continuity Indicator, and Second Quantization List.
  • the configuration of the number of quantization bits may not include the second list continuity indication and the second quantization list, and in actual scenarios, the first list continuity indication and the first quantization list corresponding to the coarse grain may be used to determine the second list corresponding to the coarse grain. Two lists of continuity indication and a second quantization list.
  • the configuration of the number of quantization bits may indicate a preset condition, so that when the first device 110 determines that the preset condition is satisfied, it may consider that the number of quantization bits is 0, or it may be understood that the first device 110 is After the current round of iterative training is completed, the processed parameters are not sent to the second device 120 .
  • the second device 120 can determine the processed parameters of the current round based on the parameters of the previous round, so the second device 120 can prevent the first device 110 from sending the processed parameters of the current round through this configuration, which can save signaling overhead.
  • the compression configuration message further includes entropy coding indication information.
  • the compression configuration message may include first indication information, second indication information, and entropy coding indication information.
  • the compression configuration message may include first indication information, second indication information, third indication information, and entropy coding indication information.
  • the entropy coding indication information may indicate whether to perform entropy coding. For example, it may indicate whether the first device 110 performs arithmetic coding after quantizing the processed parameters.
  • the entropy encoding indication information may be indicated by 1 bit, for example, "0" indicates that entropy encoding is performed, and "1" indicates that entropy encoding is not performed.
  • the entropy coding indication information may also be indicated by more bits, which is not limited in the present disclosure.
  • FIG. 7 shows a schematic diagram of a compressed configuration message 700 according to some embodiments of the present disclosure.
  • the compression configuration message 700 includes first indication information 710 and second indication information 720 , and may optionally further include third indication information 730 and/or entropy coding indication information 740 .
  • the second indication information 720 may indicate a quantization mode 722 or may indicate a quantization mode configuration 724 .
  • the third indication information 730 may indicate a quantization bit number 732 or may indicate a quantization bit number configuration 734 , wherein the quantization bit number configuration 734 may include a quantization granularity 7342 and a quantization threshold and a quantization bit number list 7344 .
  • the second device 120 sends 620 a compressed configuration message to the first device 110 .
  • the second device 120 may send the compressed configuration message through signaling, for example, the second device 120 is an access network device, and the first device 110 is a terminal device, then the signaling may optionally be a physical layer signal
  • the signaling may be radio resource control layer (Radio Resource Control, RRC) signaling.
  • RRC Radio Resource Control
  • the second device 120 sends 630 the parameters to be trained to the first device 110 .
  • the parameters to be trained may be the initial model parameters of the above-mentioned neural network, or may be updated model parameters obtained after a previous round of training.
  • the parameters to be trained may include parameters associated with the network structure of the neural network, such as weight parameters and the like.
  • the parameters to be trained may include parameters used for iteration of the neural network, such as gradient parameters and the like.
  • the second device 120 may send the compression configuration message together with the parameters to be trained to the first device 110 . In some other examples, the second device 120 may send the compressed configuration message and the parameters to be trained to the first device 110 through different signaling, which is not limited in the present disclosure.
  • the second device 120 sends a compression configuration message and parameters to be trained to multiple first devices 110, for example, N1 first devices 110 or N2 first devices 110 described in conjunction with FIG. 1 .
  • first devices 110 for example, N1 first devices 110 or N2 first devices 110 described in conjunction with FIG. 1 .
  • the embodiment of the present disclosure is described with one first device 110 , and the operations performed by other first devices 110 are similar, and will not be repeated herein.
  • the first device 110 may receive the compressed configuration message and the parameters to be trained from the second device 120 .
  • the first device 110 may obtain 640 parameters of the distributed training model based on the parameters to be trained.
  • the first device 110 may use the local data set to train the network model on the basis of the parameters to be trained, so as to obtain the parameters after the current round of training, that is, the parameters of the distributed training model.
  • the first device 110 may determine 650 a parameter division mode, a quantization mode, and a number of quantization bits based on the compression configuration message. Corresponding embodiments in which the first device 110 determines the parameter division mode, the quantization mode, and the number of quantization bits will be described below through different compression configuration messages.
  • the compression configuration message includes first indication information, second indication information and third indication information, wherein the first indication information indicates the parameter division method, the second indication information indicates the quantization mode, and the third indication information Indicates the number of quantization bits. Then, the first device 110 can read the parameter division mode, quantization mode and quantization bit number from the compression configuration message. It can be understood that the quantization method is used for a plurality of groups divided using the parameter division method, and the quantization bit number is used for a plurality of groups.
  • the compression configuration message includes first indication information and second indication information, wherein the first indication information indicates a parameter division manner, and the second indication information indicates a quantization manner.
  • the compressed configuration message does not include the third indication information. Then the first device 110 can read the parameter division mode and quantization mode from the compression configuration message, and the first device 110 can determine the number of quantization bits.
  • the first device 110 may determine the number of quantization bits based on the value of the parameter, the distribution of the value of the parameter, and the like.
  • the first device 110 may determine the number of quantization bits based on one or more of the following: transmission resources from the first device 110 to the second device 120, the number of network layers of the distributed neural network, and the training loss function.
  • the first device 110 may determine the number of quantization bits and use it for each group, that is, the number of quantization bits in a coarse grain. Alternatively, the first device 110 may separately determine the number of quantization bits for each group, that is, the number of quantization bits in a fine-grained manner.
  • the compression configuration message includes first indication information and second indication information, wherein the first indication information indicates a parameter division mode, and the second indication information indicates a quantization mode configuration.
  • the compressed configuration message does not include the third indication information. Then the first device 110 can read the parameter division manner from the compressed configuration message.
  • the first device 110 may determine the quantization scheme based on the quantization scheme configuration.
  • the quantization mode configuration indicates that multiple groups use the same quantization mode, then the first device 110 may determine the quantization mode, and the quantization mode is used for each group.
  • the configuration of the quantization mode indicates that the quantization modes are respectively determined for multiple groups, then the first device 110 may determine the quantization mode of the corresponding group based on the threshold in the configuration of the quantization mode.
  • the maximum value of multiple parameters in the i-th group is max i
  • the minimum value is min i
  • ) ⁇ thres it can be determined that the quantization mode of the i-th group is a symbol quantization mode.
  • ) ⁇ thres it may be determined that the quantization mode of the i-th group is a symbol quantization mode.
  • the first device 110 may determine the number of quantization bits.
  • the first device 110 may determine the number of quantization bits based on the value of the parameter, the distribution of the value of the parameter, and the like.
  • the first device 110 may determine the number of quantization bits based on one or more of the following: transmission resources from the first device 110 to the second device 120, the number of network layers of the distributed neural network, and the training loss function.
  • the first device 110 may determine the number of quantization bits and use it for each group, that is, the number of quantization bits in a coarse grain. Alternatively, the first device 110 may separately determine the number of quantization bits for each group, that is, the number of quantization bits in a fine-grained manner.
  • the compression configuration message includes first indication information, second indication information and third indication information, wherein the first indication information indicates the parameter division method, the second indication information indicates the quantization mode configuration, and the third indication information The information indicates quantization bit number configuration. Then the first device 110 can read the parameter division manner from the compressed configuration message.
  • the first device 110 may determine the quantization scheme based on the quantization scheme configuration.
  • the quantization mode configuration indicates that multiple groups use the same quantization mode, then the first device 110 may determine the quantization mode, and the quantization mode is used for each group.
  • the configuration of the quantization mode indicates that the quantization modes are respectively determined for multiple groups, then the first device 110 may determine the quantization mode of the corresponding group based on the threshold in the configuration of the quantization mode.
  • the first device 110 may determine the quantization bit number based on the quantization bit number configuration, where the quantization bit number configuration includes a quantization granularity, a quantization threshold list, and a quantization list. For example, the first device 110 may determine the value to be compared (denoted as d), and compare the value to be compared with two thresholds in the quantization threshold list to determine the quantization threshold interval in which the value to be compared is located, and then determine The number of quantization bits in the quantization list corresponding to the quantization threshold interval.
  • the quantization threshold list includes (T 1 , T 2 , ..., T K )
  • the quantization list may include (b 0 , b 1 , b 2 , ..., b K ). Denote the value to be compared as d, and denote the determined number of quantization bits as quan bit , then the following formula is satisfied:
  • the value to be compared may be determined based on a loss function or based on the values of multiple parameters in the divided groups.
  • the quantization bit configuration includes quantization granularity, and the quantization granularity indicates coarse granularity, that is to say, multiple groups use the same quantization bit number.
  • the first device 110 may compare the loss function obtained in the current round of training with the quantization threshold in the quantization bit number configuration, and determine the quantization bit number based on the comparison result.
  • the quantization threshold list includes (T 1 , T 2 , . . . , T K )
  • the quantization list may include (b 0 , b 1 , b 2 , . . . , b K ). If T 1 ⁇ abs(yx) ⁇ T 2 is satisfied, the number of quantization bits can be determined to be b 1 .
  • the first device 110 may also determine an index corresponding to the number of quantization bits, where the index may indicate a position of the determined number of quantization bits in the quantization list, for example, the index corresponding to b2 may be 2.
  • the quantization bit configuration includes quantization granularity, and the quantization granularity indicates fine granularity, that is to say, the quantization bit numbers need to be determined separately for different groups.
  • the first device 110 may compare the value ranges of the multiple parameters in the group with the quantization threshold in the configuration of the number of quantization bits, and determine the number of quantization bits based on the comparison result.
  • the quantization threshold list includes (T 1 , T 2 , . . . , T K )
  • the quantization list may include (b 0 , b 1 , b 2 , . . . , b K ). If T 1 ⁇ abs(max i ⁇ min i ) ⁇ T 2 is satisfied, then the number of quantization bits of the i-th group can be determined to be b 1 .
  • the first device 110 may determine a plurality of quantization bit numbers respectively corresponding to a plurality of groups.
  • the first device 110 may also determine an index corresponding to each quantization bit number, and the index may indicate the position of the quantization bit number used by the group in the quantization list, for example, the quantization bit number b2 of the i-th group corresponds to The index of can be 2.
  • the values to be compared in the embodiments of the present disclosure may also be determined based on other factors, such as the number of multiple parameters in a group, such as the size of transmission resources, etc., which are not listed in the present disclosure. It can be understood that in some examples, T 1 in the quantization threshold can be set close to 0, such as 10 ⁇ 3 or other values, and b 0 in the quantization list can be set to 0 accordingly. If the number of quantized bits is determined to be b 0 for the jth group among multiple groups, it can be considered that there is no need to send data corresponding to the number of quantized bits to the second device 120. For example, the quantized parameters for the jth group may only include quantization Boundary values such as max j and min j .
  • the first device 110 determines 660 a plurality of quantization groups based on parameters of the distributed training model.
  • each quantization group may include quantization boundary values and multiple quantization parameters.
  • the first device 110 may use a parameter division manner to divide the parameters of the distributed training model into multiple groups.
  • the first device 110 may use a quantization manner to determine a quantization boundary value for each of the multiple groups.
  • the first device 110 may determine a plurality of quantization parameters for each of the plurality of groups based on the number of quantization bits.
  • the first device 110 may use the quantization method of the group to determine the quantization boundary value of the group, and determine the multiple quantization parameters.
  • the process 660 of determining multiple quantization groups may also be referred to as a quantization process or a compression process. It should be noted that the process 660 and the above-mentioned process 650 can be interleaved.
  • the parameter division method can be used to divide the parameters into multiple groups, and then the quantization method can be determined based on the values of multiple parameters in each group. and quantization bits.
  • the quantization process may include: using a parameter division method to divide the parameters into multiple groups; for each group, determining a quantization boundary value based on the quantization method, and determining multiple quantization parameters corresponding to multiple parameters in the group based on the number of quantization bits.
  • a parameter division method to divide the parameters into multiple groups; for each group, determining a quantization boundary value based on the quantization method, and determining multiple quantization parameters corresponding to multiple parameters in the group based on the number of quantization bits.
  • the first device 110 may perform entropy encoding 670 .
  • the compression configuration message may include entropy encoding indication information, and the entropy encoding indication information indicates entropy encoding. Then the first device 110 may also perform entropy coding on the quantization parameter.
  • the first device 110 may determine to perform entropy coding by itself. entropy coding, then the first device 110 may also perform entropy coding on the quantization parameter. For example, if the first device 110 determines that the compression rate of the quantization parameter is lower than the compression rate threshold, it may decide to perform entropy encoding by itself.
  • the first device 110 sends more than 680 quantization groups to the second device 120 .
  • the multiple quantization groups may be entropy encoded multiple quantization groups.
  • the second device 120 may receive multiple quantization groups from the first device 110 .
  • FIG. 8 shows a schematic diagram of compressed data 800 according to some embodiments of the present disclosure.
  • the compressed data 800 includes a quantization boundary value data field 810 and a quantization parameter data field 820 .
  • Quantization boundary value data field 810 may include quantization boundary values in multiple quantization groups. As an example, (min1, max1), (min2, max2), . . . may be included. As another example, (min1, min2, . . . ) and (max1, max2, . . . ) may be included. As yet another example, (max1, max2, . . . ) may be included.
  • the quantization parameter data field 820 may include quantization parameters in multiple quantization groups, or may include entropy-encoded quantization parameters.
  • the first device 110 may send 6822 a quantization bit feedback message to the second device 120 .
  • the compression configuration message may include first indication information, second indication information and third indication information, wherein the first indication information indicates the parameter division mode, the second indication information indicates the quantization mode, and the third indication information indicates the number of quantization bits .
  • the first device 110 may determine another number of quantization bits based on parameters of the distributed training model.
  • the quantization bit feedback message may indicate another number of quantization bits, for example, another minimum number of quantization bits determined by the first device 110 to quantize the parameters of the distributed training model.
  • the quantization bit feedback message may be sent to the second device 120 together with multiple quantization groups via the same signaling.
  • the quantization bit feedback message and the multiple quantization groups may be sent to the second device 120 via different signaling.
  • the second device 120 may determine whether to accept another number of quantization bits included in the quantization bit feedback message based on the amount of transmission resources or the like.
  • the second device 120 may send 6824 a quantization bit response message to the first device 110, where the quantization bit response message may indicate acceptance or rejection of another quantization bit number. If the second device 120 accepts it, the second device 120 may then send a compression configuration message for the next round of training to the first device 110, and the third indication information in the compression configuration message may indicate the other quantization bit number .
  • the first device 110 may send 6842 indication information of the number of quantization bits to the second device 120 .
  • the compression configuration message may include first indication information and second indication information, wherein the first indication information indicates a parameter division method, the second indication information indicates a quantization mode, and the compression configuration message does not include the third indication information.
  • the first device 110 may send quantization bit numbers for obtaining multiple quantization groups to the second device 120, so that the second device 120 performs subsequent inverse quantization based on the quantization bit numbers.
  • the number of quantization bits and multiple quantization groups may be sent to the second device 120 via the same signaling.
  • the number of quantization bits and the multiple quantization groups may be sent to the second device 120 via different signaling.
  • the first device 110 may send 6862 a quantization indication message to the second device 120, where the quantization indication message may indicate the quantization method and the number of quantization bits used by the first device 110.
  • the compression configuration message includes the first indication information, the second indication information and the third indication information, wherein the first indication information indicates the parameter division mode, the second indication information indicates the quantization mode configuration, and the third indication information indicates the quantization bit number configuration.
  • the first device 110 may determine the quantization mode based on the configuration of the quantization mode, and may determine the number of quantization bits based on the configuration of the number of quantization bits.
  • the second indication information indicates a quantization scheme configuration, and the quantization scheme configuration indicates that multiple groups use the same quantization scheme. Then, the quantization indication message may use 1 bit to indicate the quantization mode used.
  • the second indication information indicates a quantization scheme configuration, and the quantization scheme configuration indicates that different groups use different quantization schemes. Then, the quantization indication message may respectively indicate the quantization modes of multiple groups through multiple bits.
  • the third indication information indicates the configuration of the number of quantization bits, and the configuration of the number of quantization bits includes a quantization granularity, a quantization threshold list, and a quantization list.
  • the quantization indication message may indicate the number of quantization bits used by using an index. It can be understood that for a coarse-grained situation, the number of indexes is one. For fine-grained situations, the number of indexes is multiple, corresponding to the number of multiple groups. For the description about the index, refer to the above-mentioned corresponding embodiment in the process 650 .
  • the first device 110 may also send 6882 the entropy encoding indication information to the second device 120 .
  • the first device 110 may use the entropy coding indication information to inform the second device 120 whether entropy coding is performed, for example, 1 bit may be used for indication. Therefore, the second device 120 can successfully decompress the received data based on this.
  • the second device 120 dequantizes 690 the plurality of quantization groups. Specifically, the second device 120 may perform inverse quantization on multiple quantization groups based on the parameter division manner, the quantization manner, and the number of quantization bits.
  • the description about inverse quantization can refer to the above corresponding embodiment in conjunction with FIG. 2 , and for the sake of brevity, it will not be repeated here.
  • the second device 120 obtains the parameters to be trained in the next round through inverse quantization and aggregation based on multiple quantization groups from multiple first devices 110 .
  • the second device 120 may send the compressed configuration message again, as shown in processes 610 and 620 .
  • the second device 120 may omit sending the compression configuration message for the next round.
  • the compression configuration message of the previous round may be used. In this way, repeated transmission of the same content can be avoided, and signaling overhead can be reduced.
  • the first device 110 can quantize the parameters of the distributed training model based on the compression configuration message, thereby realizing the compression of the parameters. This process not only considers factors such as transmission resources, but also considers factors such as the value of parameters, so as to ensure that the compression of parameters can not only ensure communication efficiency, but also ensure the performance of parameters. On the one hand, it avoids the performance loss caused by excessive compression, and on the other hand, it avoids the excessive delay caused by the low compression rate.
  • the transmission resources for the first device 110 to send a plurality of quantized bits to the second device 120 may be allocated by the second device 120, or may be allocated by the second device 120 after the first device 110 requests a resource. distributed.
  • the first device 110 is a terminal device
  • the second device 120 is an access network device.
  • the second device 120 may use the resource indication information to indicate the transmission resource for the first device 110 to transmit to the second device 120, and may optionally indicate a coding and modulation scheme and the like.
  • the first device 110 may send a resource request to the second device 120 , and the first device 110 may receive a resource allocation message from the second device 120 .
  • the first device 110 may then send the plurality of quantization groups to the second device 120 on the allocated resources.
  • the resource request may be a buffer state report (Buffer State Report, BSR) of a medium access control (Medium Access Control, MAC) layer.
  • BSR buffer State Report
  • MAC medium access control
  • FIG. 9 shows a signaling interaction diagram of an information transmission process 900 according to some embodiments of the present disclosure.
  • the process 900 involves the first device 110 and the second device 120, and the first device 110 in FIG. 9 may include any one of the first device 110-1 to the first device 110-N in FIG. 1 .
  • the second device 120 sends 910 the parameters to be trained to the first device 110 .
  • the parameters to be trained may be the initial model parameters of the above-mentioned neural network, or may be updated model parameters obtained after a previous round of training.
  • the parameters to be trained may include parameters associated with the network structure of the neural network, such as weight parameters and the like.
  • the parameters to be trained may include parameters used for iteration of the neural network, such as gradient parameters and the like.
  • the second device 120 sends the parameters to be trained to multiple first devices 110 , for example, N1 first devices 110 or N2 first devices 110 described in conjunction with FIG. 1 .
  • the embodiment of the present disclosure is described with one first device 110 , and the operations performed by other first devices 110 are similar, and will not be repeated herein.
  • the first device 110 may receive parameters to be trained from the second device 120 .
  • the first device 110 may obtain 920 parameters of the distributed training model based on the parameters to be trained.
  • the first device 110 may use the local data set to train the network model on the basis of the parameters to be trained, so as to obtain the parameters after the current round of training, that is, the parameters of the distributed training model.
  • the first device 110 determines 930 parameters based on the division method, the quantization method and the number of quantization bits. Specifically, the first device 110 may determine the parameter division method, the quantization method, and the number of quantization bits based on one or more of the following factors: the transmission resources of the first device 110 to the second device 120, the distributed neural network The number of network layers, and the loss function for training.
  • the first device 110 may similarly determine the parameter division mode, which is not repeated here for brevity.
  • the parameter division manner is a type-based division manner, a matrix-based division manner, or a network layer-based division manner.
  • the first device 110 may use the determined parameter division manner to divide the parameters of the distributed training model into multiple groups.
  • the first device 110 may similarly determine the quantization mode, and details are not repeated here for brevity.
  • the first device 110 may determine a quantization scheme, and the quantization scheme is used for multiple groups.
  • the first device 110 may respectively determine quantization modes for multiple groups.
  • the quantization method is a conventional quantization method or a symbolic quantization method.
  • the first device 110 may use the determined quantization manner to determine a quantization boundary value for each of the multiple groups.
  • the first device 110 may determine the number of quantization bits and apply the number of quantization bits to multiple groups, in other words, the number of quantization bits may be a coarse-grained number of quantization bits. In some other examples, the first device 110 may separately determine the number of quantization bits for each group, in other words, the number of quantization bits may be a fine-grained number of quantization bits.
  • the first device 110 may use the determined number of quantization bits to determine multiple quantization parameters for each of the multiple groups.
  • the first device 110 determines 940 a plurality of quantization groups based on parameters of the distributed training model.
  • each quantization group may include quantization boundary values and multiple quantization parameters.
  • the first device 110 may perform entropy encoding 950 .
  • the first device 110 may determine whether to perform entropy encoding based on compression ratios of multiple quantization groups and the like.
  • the first device 110 sends 962 a quantization indication message to the second device 120 .
  • the quantization indication message may include first indication information and second indication information.
  • the quantization indication message may include third indication information.
  • the first indication information may indicate a parameter division manner determined by the first device 110, for example, a type-based division manner, a matrix-based division manner, or a network layer-based division manner, and the like.
  • the second indication information may indicate the quantization manner determined by the first device 110 .
  • multiple groups are quantized in the same manner.
  • the second indication information may be indicated by 1 bit, for example, "0" indicates a traditional quantization mode, and "1" indicates a symbol quantization mode.
  • the second indication information may be indicated by 2 bits, wherein one bit indicates that each group uses the same quantization mode, and the other bit indicates the quantization mode.
  • multiple groups are quantized differently. Assume that the parameters are divided into m3 groups using the determined parameter division method.
  • the second indication information can be indicated by m3+1 bits, for example, one bit (such as the first bit) is used to indicate whether each group uses the same quantization method, and the other m3 bits represent the quantization of m3 groups respectively Way.
  • the second indication information may be indicated by m3 bits, respectively representing the quantization manners of the m3 groups.
  • the quantization indication message may not include the second indication information.
  • the second device 120 can determine the quantization manner. For example, the second device 120 can calculate the total length of quantization parameters of all quantization groups, and then calculate the number of quantization boundary values. If the number is equal to the parameter set, it means that the quantization boundary value includes the maximum value of the absolute value, and the quantization mode can be determined as the symbol quantization mode. If the number is equal to twice the parameter group, it means that the quantization boundary value includes the maximum value and the minimum value, and the quantization method can be determined as the traditional quantization method.
  • the third indication information may indicate the number of quantization bits determined by the first device 110 .
  • multiple groups have the same number of quantization bits.
  • the third indication information may be indicated by m2 bits, for example, the number of coarse-grained quantization bits is less than 2 m2 .
  • the quantization bit numbers of the multiple groups are different, the third indication information may indicate the quantization granularity and the quantization list, and the third indication information further indicates the index of the quantization bit numbers of the multiple groups. So that the second device 120 can determine the number of quantization bits of each group based on the index and the quantization list.
  • the quantization granularity indicates whether multiple groups use the same number of quantization bits . b K ).
  • indexes corresponding to different groups may be the same. For example, there are two groups with the same number of quantization bits in multiple groups.
  • the quantization indication message may not include the third indication information, and in this case, the second device 120 may determine the number of quantization bits based on multiple quantization groups.
  • each quantization group includes a quantization boundary value and a plurality of quantization parameters. For example, when the overhead (number of bits) occupied by quantization boundary values is less than the number of multiple quantization parameters, assuming that the total number of bits occupied by multiple quantization groups is p1, and the number of multiple quantization parameters is n3, then able to pass To determine the number of quantized bits, where Indicates rounding down. In this manner, the second device 120 can determine the number of quantization bits, so that there is no need to indicate through the third indication information, which can save signaling overhead.
  • the quantization indication message may not include the third indication information.
  • the second device 120 can calculate the total length of quantization parameters of all quantization groups, and then calculate the number of quantization boundary values. If the number is equal to the parameter set, it means that the quantization boundary value includes the maximum value of the absolute value, and the quantization mode can be determined as the symbol quantization mode. If the number is equal to twice the parameter group, it means that the quantization boundary value includes the maximum value and the minimum value, and the quantization method can be determined as the traditional quantization method. The second device 120 can further determine the quantized parameter length, and divide it by the number of parameters to obtain the common number of quantized bits of each parameter group.
  • the first device 110 may determine not to send the quantization indication message for the current round.
  • the quantization indication message of the previous round may be used to dequantize the multiple quantization groups of the current round . In this way, repeated transmission of the same content can be avoided, and signaling overhead can be reduced.
  • the first device 110 may also send 964 the entropy coding indication information to the second device 120 .
  • the first device 110 may use the entropy coding indication information to inform the second device 120 whether entropy coding is performed, for example, 1 bit may be used to indicate. Therefore, the second device 120 can successfully decompress the received data based on this.
  • the quantization indication message and the multiple quantization groups may be carried in the same signaling, or the quantization indication message and the multiple quantization groups may be carried in different signalings.
  • the entropy coding indication information and the multiple quantization groups may be carried in the same signaling, or the entropy coding indication information and the multiple quantization groups may be carried in different signalings. This disclosure is not limited thereto.
  • FIG. 10 shows a schematic diagram of a quantization indication message 1000 according to some embodiments of the present disclosure.
  • the quantization indication message 1000 includes first indication information 1010 , second indication information 1020 and third indication information 1030 , and may also include entropy coding indication information 1040 optionally.
  • the third indication information 1030 may indicate the number of quantization bits 1032 , or the third indication information 1030 may indicate a quantization granularity 1034 and a quantization list 1036 .
  • the quantization indication message may not include the third indication information.
  • the quantization indication message may not include the first indication information.
  • the first device 110 and the second device 120 are preset with a method for determining the division method, then the first device 110 and the second device 120 can determine the division method based on the preset determination method, so that the first device 110 and the second device 120 do not need to The indication information is included in the quantization indication message, which can reduce signaling overhead.
  • the preset determination method includes, for example: assuming that the parameters can be divided into r1 groups according to the type-based division method, and assuming that the parameters can be divided into r2 groups according to the matrix-based division method, then based on r2-r1 and the predetermined The relationship between the values (such as s and d) to determine the division method, as described in conjunction with the process 610 above.
  • the first device 110 sends more than 970 quantization groups to the second device 120 .
  • the multiple quantization groups may be entropy encoded multiple quantization groups.
  • the second device 120 may receive multiple quantization groups from the first device 110 .
  • the third indication information in the quantization indication message indicates the quantization granularity and the quantization list
  • the corresponding index may also be sent to the second device 120 .
  • the index may indicate the position of the corresponding quantization bit number in the quantization bit number list as shown in FIG. 7 .
  • the first device 110 may determine the compressed data based on the plurality of quantization groups, and send the compressed data to the second device 120 .
  • FIG. 11 shows a schematic diagram of compressed data 1100 according to some embodiments of the present disclosure.
  • the compressed data 1100 includes a quantization boundary value data field 1110 and a quantization parameter data field 1120 , and may also include a list of indexes 1130 optionally.
  • Quantization boundary value data field 1110 may include quantization boundary values in multiple quantization groups. As an example, (min1, max1), (min2, max2), . . . may be included. As another example, (min1, min2, . . . ) and (max1, max2, . . . ) may be included.
  • the quantization parameter data field 1120 may include quantization parameters in a plurality of quantization groups, or may include entropy-encoded quantization parameters.
  • the index list 1130 may include indexes corresponding to quantization bit numbers of each group.
  • the index list 1130 may be independent from the quantization boundary value data field 1110 and the quantization parameter data field 1120 .
  • a list of indexes may be included in the quantization boundary value data field 1110, for example may include (min1, max1, index1), (min2, max2, index2), ..., or for example include (min1, min2, %), (max1, max2, ...) and (index1, index2, 7), where index1 represents the index of the number of quantization bits used by the group whose quantization boundary value is min1 and max1, and index2 represents the group whose quantization boundary value is min2 and max2 Index of the number of quantization bits used by the group.
  • the second device 120 dequantizes 980 the plurality of quantization groups. Specifically, the second device 120 may perform inverse quantization on multiple quantization groups based on the quantization indication information from the first device 110 .
  • the description about inverse quantization can refer to the above corresponding embodiment in conjunction with FIG. 2 , and for the sake of brevity, it will not be repeated here.
  • the transmission resources for the first device 110 to send a plurality of quantized bits to the second device 120 may be allocated by the second device 120, or may be allocated by the second device 120 after the first device 110 requests a resource. distributed. For example as described in the above embodiments of the present disclosure.
  • the first device 110 can quantize the parameters of the distributed training model, thereby realizing parameter compression. This process not only considers factors such as transmission resources, but also considers factors such as the value of parameters, so as to ensure that the compression of parameters can not only ensure communication efficiency, but also ensure the performance of parameters. On the one hand, it avoids the performance loss caused by excessive compression, and on the other hand, it avoids the excessive delay caused by the low compression ratio.
  • Fig. 12 shows a schematic block diagram of a communication device 1200 according to some embodiments of the present disclosure.
  • the apparatus 1200 may be implemented as the first device 110 shown in FIG. 1 or as a part of the first device 110 (such as a chip), etc., which is not limited in the present disclosure.
  • the apparatus 1200 may include a dividing module 1210 , a determining module 1220 and a sending module 1230 , and optionally, a receiving module 1240 and/or an entropy encoding module 1250 .
  • the dividing module 1210 is configured to divide the parameters of the distributed training model into multiple groups, and at least two of the multiple groups have different value intervals.
  • the determination module 1220 is configured to determine a plurality of quantization groups corresponding to the plurality of groups based on the number of quantization bits corresponding to each group in the plurality of groups, the number of quantization bits is determined based on at least one of the following: the communication device and The amount of transmission resources between the second devices, the value of the loss function, or the value range of the parameters of each group.
  • the sending module 1230 is configured to send multiple quantization groups to the second device.
  • the determining module 1220 may be configured to determine a quantization boundary value corresponding to the first group based on the plurality of first parameters in the first group of the plurality of groups.
  • the quantization boundary value includes a maximum value and/or a minimum value, the maximum value is the maximum value of the multiple first parameters or the maximum value is the maximum value of multiple absolute values of the multiple first parameters, and the minimum value is the multiple first parameters
  • the minimum value or the minimum value in is the minimum value of the absolute values of the plurality of first parameters.
  • the quantization group corresponding to the first group includes quantization boundary values and a plurality of quantization parameters corresponding to the plurality of first parameters.
  • the receiving module 1240 may be configured to receive first indication information from the second device, where the first indication information indicates whether to determine the quantization boundary value according to an absolute value.
  • the determination module 1220 may be configured to determine a quantization boundary value based on the first indication information.
  • the sending module 1230 may be configured to send second indication information to the second device, where the second indication information indicates whether the quantization boundary value is determined according to an absolute value.
  • the number of quantization bits for each group is equal. In some embodiments, the plurality of groups includes a first group and a second group, and the number of quantization bits corresponding to the first group is not equal to the number of quantization bits corresponding to the second group.
  • the determining module 1220 may be configured to determine a plurality of quantization bit numbers corresponding to a plurality of groups. In some examples, the determining module 1220 may be configured to determine a plurality of quantization bit numbers corresponding to multiple groups based on a quantization bit number configuration from the second device, the quantization bit number configuration including a quantization threshold and a corresponding quantization bit number. Correspondingly, the receiving module 1240 may be configured to receive the quantization bit number configuration from the second device.
  • the sending module 1230 is configured to send third indication information to the second device, where the third indication information indicates a plurality of quantization bit numbers.
  • the number of quantization bits is determined by the second device, and the receiving module 1240 is configured to receive fourth indication information from the second device, where the fourth indication information indicates the quantization bits corresponding to each of the multiple groups number.
  • the dividing module 1210 may be configured to divide the parameters of the distributed training model into multiple groups using a model parameter dividing manner.
  • the determination module 1220 is configured to determine the model parameter division method based on the values of the parameters of the distributed training model.
  • the sending module 1230 may be configured to send fifth indication information to the second device, where the fifth indication information indicates a model parameter division manner.
  • the receiving module 1240 is configured to receive sixth indication information from the second device, where the sixth indication information indicates a model parameter division manner.
  • the division mode of the model parameters indicates that the division is based on at least one of the following items: the type of the parameter, the type of the network layer where the parameter is located, and the like.
  • the entropy encoding module 1250 is configured to entropy encode the multiple quantization groups; and the sending module is configured to send the entropy encoded multiple quantization groups to the second device.
  • the sending module 1230 is configured to send first entropy encoding indication information to the second device, where the first entropy encoding indication information has entropy encoded multiple quantization groups.
  • the receiving module is configured to receive second entropy encoding indication information from the second device; and the entropy encoding module is configured to perform entropy encoding on the multiple quantization groups based on the second entropy encoding indication information.
  • each function in the disclosed embodiments Units can be integrated into one unit, or physically exist separately, or two or more units can be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the apparatus 1200 in FIG. 12 can be used to implement each process described by the first device 110 in the foregoing embodiments, and for the sake of brevity, details are not repeated here.
  • Fig. 13 shows a schematic block diagram of a communication device 1300 according to some embodiments of the present disclosure.
  • the apparatus 1300 may be implemented as the second device 120 shown in FIG. 1 or as a part of the second device 120 (such as a chip), etc., which is not limited in the present disclosure.
  • the apparatus 1300 may include a receiving module 1310 , a first determining module 1320 and a second determining module 1330 , and optionally further includes a sending module 1340 .
  • the receiving module 1310 is configured to receive multiple quantization groups from the first device.
  • the first determination module 1320 is configured to determine a plurality of groups corresponding to the plurality of quantization groups based on the number of quantization bits corresponding to each of the plurality of quantization groups, the number of quantization bits is determined based on at least one of the following : the amount of transmission resources between the first device and the communication device, the value of the loss function, or the value range of the parameters of each group.
  • the second determination module 1330 is configured to determine parameters of the distributed training model based on the model parameter division method and multiple groups.
  • the first quantization group of the plurality of quantization groups includes quantization boundary values and a plurality of quantization parameters.
  • the quantization boundary value includes a maximum value and/or a minimum value, and the maximum value is the maximum value among the multiple first parameters in the first group corresponding to the first quantization group or the maximum value is the maximum value of the multiple absolute values of the multiple first parameters.
  • the maximum value and the minimum value are the minimum value among the multiple first parameters or the minimum value is the minimum value of the multiple absolute values of the multiple first parameters.
  • the sending module 1340 is configured to send first indication information to the first device, where the first indication information indicates whether to determine the quantization boundary value according to an absolute value.
  • the receiving module 1310 is configured to receive second indication information from the first device, where the second indication information indicates whether the quantization boundary value is determined according to an absolute value.
  • the number of quantization bits for each quantization group is equal. In some embodiments, the multiple quantization groups include a first quantization group and a second quantization group, and the number of quantization bits corresponding to the first quantization group is not equal to the number of quantization bits corresponding to the second quantization group.
  • the receiving module 1310 is configured to receive third indication information from the first device, where the third indication information indicates a plurality of quantization bit numbers.
  • the sending module 1340 is configured to send the quantization bit number configuration to the first device, where the quantization bit number configuration includes a quantization threshold and a corresponding quantization bit number.
  • the sending module 1340 is configured to send fourth indication information to the first device, where the fourth indication information indicates the number of quantization bits corresponding to each of the multiple groups.
  • the receiving module 1310 is configured to receive fifth indication information from the first device, where the fifth indication information indicates a model parameter division manner.
  • the sending module 1340 is configured to send sixth indication information to the first device, where the sixth indication information indicates a model parameter division manner.
  • the model parameter division method indicates that the division is based on at least one of the following: the type of the parameter, the type of the network layer where the parameter is located, and the like.
  • the receiving module 1310 is configured to receive the entropy encoded plurality of quantization groups.
  • the receiving module 1310 is configured to receive first entropy encoding indication information from the first device, where the first entropy encoding indication information has entropy encoded a plurality of quantization groups.
  • the sending module 1340 is configured to send second entropy encoding indication information to the first device, so as to instruct the first device to perform entropy encoding on multiple quantization groups.
  • each function in the disclosed embodiments Units can be integrated into one unit, or physically exist separately, or two or more units can be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the apparatus 1300 in FIG. 13 can be used to implement the various processes described by the second device 120 in the foregoing embodiments, and details are not repeated here for brevity.
  • FIG. 14 shows a schematic block diagram of an example device 1400 that may be used to implement embodiments of the present disclosure.
  • the device 1400 may be implemented as or included in the first device 110 of FIG. 1 , or the device 1400 may be implemented as or included in the second device 120 of FIG. 1 .
  • device 1400 includes one or more processors 1410 , one or more memories 1420 coupled to processors 1410 , and communication module 1440 coupled to processors 1410 .
  • the communication module 1440 can be used for two-way communication.
  • the communication module 1440 may have at least one communication interface for communication.
  • Communication interfaces may include any interface necessary to communicate with other devices.
  • the processor 1410 may be of any type suitable for the local technology network, and may include, but is not limited to, at least one of the following: a general purpose computer, a special purpose computer, a microcontroller, a digital signal processor (Digital Signal Processor, DSP), or a control-based One or more of the multi-core controller architectures of the processor.
  • Device 1400 may have multiple processors, such as application specific integrated circuit chips, that are time slaved to a clock that is synchronized to a main processor.
  • Memory 1420 may include one or more non-volatile memories and one or more volatile memories.
  • non-volatile memory include but are not limited to at least one of the following: read-only memory (Read-Only Memory, ROM) 1424, erasable programmable read-only memory (Erasable Programmable Read Only Memory, EPROM), flash memory, hard disk , Compact Disc (CD), Digital Video Disk (Digital Versatile Disc, DVD) or other magnetic and/or optical storage.
  • Examples of volatile memory include, but are not limited to, at least one of: Random Access Memory (RAM) 1422, or other volatile memory that does not persist for the duration of a power outage.
  • RAM Random Access Memory
  • the memory 1420 may also be integrated with the processor 1410 .
  • the computer program 1430 includes computer-executable instructions executed by the associated processor 1410 .
  • Program 1430 may be stored in ROM 1424.
  • Processor 1410 may perform any suitable actions and processes by loading program 1430 into RAM 1422.
  • Embodiments of the present disclosure may be implemented by means of a program 1430 such that the device 1400 may perform any process as discussed with reference to FIGS. 2 to 11 .
  • Embodiments of the present disclosure can also be realized by hardware or by a combination of software and hardware.
  • Program 1430 may be tangibly embodied on a computer-readable medium, which may be included in device 1400 , such as in memory 1420 , or other storage device accessible by device 1400 .
  • Program 1430 may be loaded from a computer readable medium into RAM 1422 for execution.
  • the computer readable medium may include any type of tangible nonvolatile memory such as ROM, EPROM, flash memory, hard disk, CD, DVD, and the like.
  • the communication module 1440 in the device 1400 can be implemented as a transmitter and receiver (or transceiver), which can be configured to send/receive information such as compression configuration messages, parameters to be trained, multiple quantization groups, etc. .
  • the device 1400 may further include one or more of a scheduler, a controller, and a radio frequency/antenna, which will not be described in detail in this disclosure.
  • the device 1400 in FIG. 14 may be implemented as a communication device, or may be implemented as a chip or chip system in the communication device, which is not limited by the embodiments of the present disclosure.
  • Embodiments of the present disclosure also provide a chip, which may include an input interface, an output interface, and a processing circuit.
  • a chip which may include an input interface, an output interface, and a processing circuit.
  • the interaction of signaling or data may be completed by the input interface and the output interface, and the generation and processing of signaling or data information may be completed by the processing circuit.
  • Embodiments of the present disclosure also provide a chip system, including a processor, configured to support a device to implement the functions involved in any of the foregoing embodiments.
  • the system-on-a-chip may further include a memory for storing necessary program instructions and data, and when the processor runs the program instructions, the device installed with the system-on-a-chip can implement the program described in any of the above-mentioned embodiments.
  • the chip system may consist of one or more chips, and may also include chips and other discrete devices.
  • the memory and processor can be integrated.
  • Embodiments of the present disclosure further provide a processor, configured to be coupled with a memory, where instructions are stored in the memory, and when the processor executes the instructions, the processor executes the methods and functions involved in any of the foregoing embodiments.
  • Embodiments of the present disclosure also provide a computer program product containing instructions, which, when run on a computer, cause the computer to execute the methods and functions involved in any of the above embodiments.
  • Embodiments of the present disclosure also provide a computer-readable storage medium on which computer instructions are stored, and when a processor executes the instructions, the processor is made to execute the methods and functions involved in any of the above embodiments.
  • the various embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor or other device. While various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flowcharts, or using some other pictorial representation, it should be understood that the blocks, devices, systems, techniques or methods described herein can be implemented as, without limitation, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers or other devices, or some combination thereof.
  • the present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium.
  • the computer program product comprises computer-executable instructions, eg included in program modules, which are executed in a device on a real or virtual processor of a target to perform the process/method as above with reference to the accompanying drawings.
  • program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
  • the functionality of the program modules may be combined or divided as desired among the program modules.
  • Machine-executable instructions for program modules may be executed within local or distributed devices. In a distributed device, program modules may be located in both local and remote storage media.
  • Computer program codes for implementing the methods of the present disclosure may be written in one or more programming languages. These computer program codes can be provided to processors of general-purpose computers, special-purpose computers, or other programmable data processing devices, so that when the program codes are executed by the computer or other programmable data processing devices, The functions/operations specified in are implemented.
  • the program code may execute entirely on the computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server.
  • computer program code or related data may be carried by any suitable carrier to enable a device, apparatus or processor to perform the various processes and operations described above.
  • carriers include signals, computer readable media, and the like.
  • signals may include electrical, optical, radio, sound, or other forms of propagated signals, such as carrier waves, infrared signals, and the like.
  • a computer readable medium may be any tangible medium that contains or stores a program for or related to an instruction execution system, apparatus, or device.
  • the computer readable medium may be a computer readable signal medium or a computer readable storage medium.
  • a computer readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More detailed examples of computer-readable storage media include electrical connections with one or more wires, portable computer diskettes, hard disks, random storage access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash), optical storage, magnetic storage, or any suitable combination thereof.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Bioethics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Databases & Information Systems (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

提供了一种信息处理方法及装置。方法包括：第一设备将分布式训练模型的参数划分为多个组，多个组中的至少两个组的数值区间不同；第一设备基于与多个组中每个组对应的量化比特数，确定与多个组对应的多个量化组，量化比特数是基于以下至少一项而被确定的：第一设备与第二设备之间的传输资源量、损失函数值或每个组的参数的取值范围；以及第一设备向第二设备发送多个量化组。

Description

信息处理方法及装置
本申请要求于2022年1月29日提交中国专利局，申请号为202210112944.2，发明名称为“信息处理方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开涉及分布式学习领域，更具体地，涉及一种信息处理方法及装置。
背景技术
联邦学习(Federated Learning，FL)属于人工智能(Artificial Intelligence，AI)基础技术，其在保障大数据交换时的信息安全、保护终端数据和个人数据隐私、保证合法合规的前提下，在多参与方或多计算结点之间开展高效率的机器学习。联邦学习可使用的机器学习算法不局限于神经网络，还包括随机森林等其他算法。
诸如联邦学习之类的分布式学习涉及多个参与方之间的数据交互，例如模型参数等的交互。为了保证通信效率，交互的数据需要经过压缩之后再被发送。
目前的压缩过程中采用基于深度学习的压缩方案，例如模型剪枝、知识蒸馏、格式转化等，这些方案可以有效地利用传输资源，确保通信效率。然而这些方案将一些较小的参数值置零或者将较大的模型迁移到较小的模型上来降低传输的数据量，从而造成了对模型性能的损失。
发明内容
本公开的实施例提供了一种数据处理的方案，可以被应用于诸如联邦学习之类的分布式学习中，能够在保证通信效率的同时确保模型的性能。
在本公开的第一方面，提供了一种数据处理方法。该方法包括：第一设备将分布式训练模型的参数划分为多个组，多个组中的至少两个组的数值区间不同；第一设备基于与多个组中每个组对应的量化比特数，确定与多个组对应的多个量化组，量化比特数是基于以下至少一项而被确定的：第一设备与第二设备之间的传输资源量、损失函数值、或每个组的参数的取值范围；以及第一设备向第二设备发送多个量化组。
因此，第一设备能够对分布式训练模型的参数进行量化，从而实现了对参数的压缩。该过程既考虑了诸如传输资源等因素，又考虑了参数的取值大小等因素，从而能够确保对参数的压缩既能保证通信效率，又能保证参数的性能。一方面避免过度压缩导致的性能损失，另一方面又避免了因压缩率低导致的时延过大。
在第一方面的一些实施例中，还包括：第一设备基于多个组中的第一组中的多个第一参数，确定与第一组对应的量化边界值。
以此方式，能够通过量化边界值指定所划分的组中的参数的取值大小的范围，从而能够在量化过程简化传输的数据量。
在第一方面的一些实施例中，量化边界值包括以下至少一项：最大值、或最小值，最大值为多个第一参数中的最大值或者最大值为多个第一参数的多个绝对值的最大值，最小值为多个第一参数中的最小值或者最小值为多个第一参数的多个绝对值的最小值。
以此方式，通过将最大值和/或最小值作为量化边界值，从而能够用于指定所划分的组中 参数的数值区间或绝对值范围。
在第一方面的一些实施例中，与第一组对应的量化组包括量化边界值以及与多个第一参数对应的多个量化参数。
在第一方面的一些实施例中，还包括：第一设备接收来自第二设备的第一指示信息，第一指示信息指示是否按照绝对值来确定量化边界值；并且第一设备确定与第一组对应的量化边界值包括：第一设备基于第一指示信息，确定量化边界值。
以此方式，第一设备可以基于第二设备的指示来确定量化边界值，这样能够减小第一设备处用于确定量化方式的计算量。
在第一方面的一些实施例中，还包括：第一设备向第二设备发送第二指示信息，第二指示信息指示量化边界值是否按照绝对值而被确定。
以此方式，第一设备能够通过指示信息将量化方式告知第二设备，从而第二设备能够正确地进行反量化。
在第一方面的一些实施例中，每个组的量化比特数都是相等的。以此方式，各个组使用相同的量化比特数，这样能够简化操作，提升处理效率。
在第一方面的一些实施例中，多个组包括第一组和第二组，并且与第一组对应的量化比特数不等于与第二组对应的量化比特数。以此方式，不同的组使用不同的量化比特数，能够充分考虑不同组之间的参数区别，避免量化对性能的影响。
在第一方面的一些实施例中，还包括：第一设备确定与多个组对应的多个量化比特数。
在第一方面的一些实施例中，第一设备确定与多个组对应的多个量化比特数包括：第一设备基于来自第二设备的量化比特数配置，确定与多个组对应的多个量化比特数，量化比特数配置包括量化阈值和对应的量化比特数。
以此方式，第一设备可以基于来自第二设备的量化比特数配置来确定量化比特数，这样能够同时满足第一设备和第二设备的实际需求，进而所确定的量化比特数能够兼顾通信效率和压缩性能。
在第一方面的一些实施例中，还包括：第一设备接收来自第二设备的量化比特数配置。
在第一方面的一些实施例中，还包括：第一设备向第二设备发送第三指示信息，第三指示信息指示多个量化比特数。
例如第三指示信息可以通过量化比特数配置中的索引来隐式或间接地指示多个量化比特数。以此方式，第一设备可以将所确定的量化比特数发送至第二设备，从而第二设备能够基于索引确定各个组的量化比特数，确保第二设备对参数正确的反量化。
在第一方面的一些实施例中，量化比特数是由第二设备确定的，方法还包括：第一设备接收来自第二设备的第四指示信息，第四指示信息指示与多个组中每个组对应的量化比特数。
以此方式，由第二设备指示量化比特数，从而第一设备可以直接基于该指示进行参数量化。简化了第一设备的操作，提升了处理效率。
在第一方面的一些实施例中，第一设备将分布式训练模型的参数划分为多个组包括：第一设备使用模型参数划分方式，将分布式训练模型的参数划分为多个组。
在第一方面的一些实施例中，还包括：第一设备基于分布式训练模型的参数的取值大小，确定模型参数划分方式；以及第一设备向第二设备发送第五指示信息，第五指示信息指示模型参数划分方式。
以此方式，第一设备可以基于参数特性来确定参数划分方式，从而后续可以针对不同组 进行不同的量化处理，这样能够确保压缩性能。
在第一方面的一些实施例中，还包括：第一设备接收来自第二设备的第六指示信息，第六指示信息指示模型参数划分方式。
以此方式，第二设备可以指示参数划分方式，从而第一设备可以直接基于该指示进行参数划分。简化了第一设备的操作，提升了处理效率。
在第一方面的一些实施例中，模型参数划分方式指示基于如下至少一项进行划分：参数的类型、参数所在的网络层的类型等。
以此方式，可以通过考虑参数的不同特性来实现参数的划分，从而位于同一个组内的参数能够进行类似的量化。
在第一方面的一些实施例中，还包括：第一设备对多个量化组进行熵编码；并且第一设备向第二设备发送多个量化组包括：第一设备向第二设备发送经熵编码的多个量化组。
以此方式，第一设备可以通过熵编码进一步提升压缩率，从而能够进一步提高通信效率。
在第一方面的一些实施例中，还包括：第一设备向第二设备发送第一熵编码指示信息，第一熵编码指示信息多个量化组已被熵编码。
以此方式，第二设备可以获知是否进行了熵编码，从而能够对接收到的参数正确进行解码。
在第一方面的一些实施例中，还包括：第一设备接收来自第二设备的第二熵编码指示信息；并且第一设备对多个量化组进行熵编码包括：第一设备基于第二熵编码指示信息，对多个量化组进行熵编码。
以此方式，第一设备可以基于第二设备的指示进行熵编码，这样能够进一步提升压缩率。
在本公开的第二方面，提供了一种数据处理方法。该方法包括：第二设备接收来自第一设备的多个量化组；第二设备基于与多个量化组中每个量化组对应的量化比特数，确定与多个量化组对应的多个组，量化比特数是基于以下至少一项而被确定的：第一设备与第二设备之间的传输资源量、损失函数值、或每个组的参数的取值范围；以及第二设备基于模型参数划分方式和多个组，确定分布式训练模型的参数。
在第二方面的一些实施例中，多个量化组中第一量化组包括量化边界值以及多个量化参数。
在第二方面的一些实施例中，量化边界值包括以下至少一项：最大值、或最小值，最大值为与第一量化组对应的第一组中多个第一参数中的最大值或者最大值为多个第一参数的多个绝对值的最大值，最小值为多个第一参数中的最小值或者最小值为多个第一参数的多个绝对值的最小值。
在第二方面的一些实施例中，还包括：第二设备向第一设备发送第一指示信息，第一指示信息指示是否按照绝对值来确定量化边界值；或者，第二设备接收来自第一设备的第二指示信息，第二指示信息指示量化边界值是否按照绝对值而被确定。
在第二方面的一些实施例中，每个量化组的量化比特数都是相等的。
在第二方面的一些实施例中，多个量化组包括第一量化组和第二量化组，并且与第一量化组对应的量化比特数不等于与第二量化组对应的量化比特数。
在第二方面的一些实施例中，还包括：第二设备接收来自第一设备的第三指示信息，第三指示信息指示多个量化比特数。
在第二方面的一些实施例中，还包括：第二设备向第一设备发送量化比特数配置，量化 比特数配置包括量化阈值和对应的量化比特数。
在第二方面的一些实施例中，还包括：第二设备向第一设备发送第四指示信息，第四指示信息指示与多个组中每个组对应的量化比特数。
在第二方面的一些实施例中，还包括：第二设备接收来自第一设备的第五指示信息，第五指示信息指示模型参数划分方式。
在第二方面的一些实施例中，还包括：第二设备向第一设备发送第六指示信息，第六指示信息指示模型参数划分方式。
在第二方面的一些实施例中，模型参数划分方式指示基于如下至少一项进行划分：参数的类型、参数所在的网络层的类型等。
在第二方面的一些实施例中，第二设备接收来自第一设备的多个量化组包括：第二设备接收经熵编码的多个量化组。
在第二方面的一些实施例中，还包括：第二设备接收来自第一设备的第一熵编码指示信息，第一熵编码指示信息多个量化组已被熵编码。
在第二方面的一些实施例中，还包括：第二设备向第一设备发送第二熵编码指示信息，以指示第一设备对多个量化组进行熵编码。
在本公开的第三方面，提供了一种通信装置。该通信装置包括：划分模块，被配置为将分布式训练模型的参数划分为多个组，多个组中的至少两个组的数值区间不同；确定模块，被配置为基于与多个组中每个组对应的量化比特数，确定与多个组对应的多个量化组，量化比特数是基于以下至少一项而被确定的：通信装置与第二设备之间的传输资源量、损失函数值、或每个组的参数的取值范围；以及发送模块，被配置为向第二设备发送多个量化组。
在第三方面的一些实施例中，确定模块被配置为基于多个组中的第一组中的多个第一参数，确定与第一组对应的量化边界值。
在第三方面的一些实施例中，量化边界值包括以下至少一项：最大值、或最小值，最大值为多个第一参数中的最大值或者最大值为多个第一参数的多个绝对值的最大值，最小值为多个第一参数中的最小值或者最小值为多个第一参数的多个绝对值的最小值。
在第三方面的一些实施例中，与第一组对应的量化组包括量化边界值以及与多个第一参数对应的多个量化参数。
在第三方面的一些实施例中，还包括接收模块，被配置为接收来自第二设备的第一指示信息，第一指示信息指示是否按照绝对值来确定量化边界值。并且确定模块被配置为基于第一指示信息，确定量化边界值。
在第三方面的一些实施例中，发送模块被配置为向第二设备发送第二指示信息，第二指示信息指示量化边界值是否按照绝对值而被确定。
在第三方面的一些实施例中，每个组的量化比特数都是相等的。
在第三方面的一些实施例中，多个组包括第一组和第二组，并且与第一组对应的量化比特数不等于与第二组对应的量化比特数。
在第三方面的一些实施例中，确定模块被配置为确定与多个组对应的多个量化比特数。
在第三方面的一些实施例中，确定模块被配置为基于来自第二设备的量化比特数配置，确定与多个组对应的多个量化比特数，量化比特数配置包括量化阈值和对应的量化比特数。
在第三方面的一些实施例中，接收模块被配置为接收来自第二设备的量化比特数配置。
在第三方面的一些实施例中，发送模块被配置为向第二设备发送第三指示信息，第三指 示信息指示多个量化比特数。
在第三方面的一些实施例中，量化比特数是由第二设备确定的，接收模块被配置为接收来自第二设备的第四指示信息，第四指示信息指示与多个组中每个组对应的量化比特数。
在第三方面的一些实施例中，划分模块被配置为使用模型参数划分方式，将分布式训练模型的参数划分为多个组。
在第三方面的一些实施例中，确定模块被配置为基于分布式训练模型的参数的取值大小，确定模型参数划分方式；以及发送模块被配置为向第二设备发送第五指示信息，第五指示信息指示模型参数划分方式。
在第三方面的一些实施例中，接收模块被配置为接收来自第二设备的第六指示信息，第六指示信息指示模型参数划分方式。
在第三方面的一些实施例中，模型参数划分方式指示基于如下至少一项进行划分：参数的类型、参数所在的网络层的类型等。
在第三方面的一些实施例中，还包括熵编码模块，被配置为对多个量化组进行熵编码；并且发送模块被配置为向第二设备发送经熵编码的多个量化组。
在第三方面的一些实施例中，发送模块被配置为向第二设备发送第一熵编码指示信息，第一熵编码指示信息多个量化组已被熵编码。
在第三方面的一些实施例中，接收模块被配置为接收来自第二设备的第二熵编码指示信息；并且熵编码模块被配置为基于第二熵编码指示信息，对多个量化组进行熵编码。
在本公开的第四方面，提供了一种通信装置。该通信装置包括：接收模块，被配置为接收来自第一设备的多个量化组；第一确定模块，被配置为基于与多个量化组中每个量化组对应的量化比特数，确定与多个量化组对应的多个组，量化比特数是基于以下至少一项而被确定的：第一设备与通信装置之间的传输资源量、损失函数值、或每个组的参数的取值范围；以及第二确定模块，被配置为基于模型参数划分方式和多个组，确定分布式训练模型的参数。
在第四方面的一些实施例中，多个量化组中第一量化组包括量化边界值以及多个量化参数。
在第四方面的一些实施例中，量化边界值包括以下至少一项：最大值、或最小值，最大值为与第一量化组对应的第一组中多个第一参数中的最大值或者最大值为多个第一参数的多个绝对值的最大值，最小值为多个第一参数中的最小值或者最小值为多个第一参数的多个绝对值的最小值。
在第四方面的一些实施例中，还包括发送模块，被配置为向第一设备发送第一指示信息，第一指示信息指示是否按照绝对值来确定量化边界值。
在第四方面的一些实施例中，接收模块被配置为接收来自第一设备的第二指示信息，第二指示信息指示量化边界值是否按照绝对值而被确定。
在第四方面的一些实施例中，每个量化组的量化比特数都是相等的。
在第四方面的一些实施例中，多个量化组包括第一量化组和第二量化组，并且与第一量化组对应的量化比特数不等于与第二量化组对应的量化比特数。
在第四方面的一些实施例中，接收模块被配置为接收来自第一设备的第三指示信息，第三指示信息指示多个量化比特数。
在第四方面的一些实施例中，发送模块被配置为向第一设备发送量化比特数配置，量化比特数配置包括量化阈值和对应的量化比特数。
在第四方面的一些实施例中，发送模块被配置为向第一设备发送第四指示信息，第四指示信息指示与多个组中每个组对应的量化比特数。
在第四方面的一些实施例中，接收模块被配置为接收来自第一设备的第五指示信息，第五指示信息指示模型参数划分方式。
在第四方面的一些实施例中，发送模块被配置为向第一设备发送第六指示信息，第六指示信息指示模型参数划分方式。
在第四方面的一些实施例中，模型参数划分方式指示基于如下至少一项进行划分：参数的类型、参数所在的网络层的类型等。
在第四方面的一些实施例中，接收模块被配置为接收经熵编码的多个量化组。
在第四方面的一些实施例中，接收模块被配置为接收来自第一设备的第一熵编码指示信息，第一熵编码指示信息多个量化组已被熵编码。
在第四方面的一些实施例中，发送模块被配置为向第一设备发送第二熵编码指示信息，以指示第一设备对多个量化组进行熵编码。
在本公开的第五方面，提供了一种通信设备。该通信设备包括处理器以及存储器，存储器上存储有由处理器执行的指令，当指令被处理器执行时使得该通信设备实现如前所述第一方面或其任一实施例中的方法的操作，或者实现根据上述第二方面或其任一实施例中的方法的操作。
在第五方面的一些实施例中，通信设备还包括收发器，以用于接收来自另一设备的信息或者向另一设备发送信息等。
在本公开的第六方面，提供了一种计算机可读存储介质，该计算机可读存储介质上存储有计算机可执行指令，该计算机可执行指令被处理器执行时实现根据上述第一方面或其任一实施例中的方法的操作，或者实现根据上述第二方面或其任一实施例中的方法的操作。
在本公开的第七方面，提供了一种芯片或芯片系统。该芯片或芯片系统包括处理电路，被配置为执行根据上述第一方面或其任一实施例中的方法的操作，或者执行根据上述第二方面或其任一实施例中的方法的操作。
在本公开的第八方面，提供了一种计算机程序或计算机程序产品。该计算机程序或计算机程序产品被有形地存储在计算机可读介质上并且包括计算机可执行指令，计算机可执行指令在被执行时实现根据上述第一方面或其任一实施例中的方法的操作，或者实现根据上述第二方面或其任一实施例中的方法的操作。
应当理解，发明内容部分中所描述的内容并非旨在限定本公开的实施例的关键或重要特征，亦非用于限制本公开的范围。本公开的其它特征将通过以下的描述变得容易理解。
附图说明
结合附图并参考以下详细说明，本公开各实施例的上述和其他特征、优点及方面将变得更加明显。在附图中，相同或相似的附图标记表示相同或相似的元素，其中：
图1示出了本公开的实施例能够应用于其中的示例场景的示意图；
图2示出了根据本公开的一些实施例的信息处理过程的信令交互图；
图3示出了根据本公开的一些实施例的参数划分的示意图；
图4示出了根据本公开的一些实施例的参数划分的示意图；
图5示出了根据本公开的一些实施例的参数划分的示意图；
图6示出了根据本公开的一些实施例的信息传输过程的信令交互图；
图7示出了根据本公开的一些实施例的压缩配置消息的示意图；
图8示出了根据本公开的一些实施例的压缩数据的示意图；
图9示出了根据本公开的一些实施例的信息传输过程的信令交互图；
图10示出了根据本公开的一些实施例的量化指示消息的示意图；
图11示出了根据本公开的一些实施例的压缩数据的示意图；
图12示出了根据本公开的一些实施例的通信装置的一个示意框图；
图13示出了根据本公开的一些实施例的通信装置的一个示意框图；以及
图14示出了可以用来实施本公开的实施例的示例设备的示意性框图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
在本公开的实施例的描述中，术语“包括”及其类似用语应当理解为开放性包含，即“包括但不限于”。术语“基于”应当理解为“至少部分地基于”。术语“一个实施例”或“该实施例”应当理解为“至少一个实施例”。术语“第一”、“第二”等等可以指代不同的或相同的对象。术语“多个”表示两个或者两个以上。下文还可能包括其他明确的和隐含的定义。
本公开的实施例可以根据适当的通信协议来实施，包括但不限于，第三代(3rd Generation，3G)、第四代(4G)、第五代(5G)、第六代(6G)等蜂窝通信协议、诸如电气与电子工程师协会(Institute of Electrical and Electronics Engineers，IEEE)802.11等的无线局域网通信协议、未来演进的移动通信系统例如6G通信系统协议、和/或目前已知或者将来开发的其他协议。
本公开的实施例的技术方案应用于遵循任何适当通信协议的通信系统，例如：通用移动通信系统(Universal Mobile Telecommunications Service，UMTS)、长期演进(Long Term Evolution，LTE)系统、宽带码分多址系统(Wideband Code Division Multiple Access，WCDMA)、码分多址2000系统(Code Division Multiple Access，CDMA2000)、时分同步码分多址系统(Time Division-Synchronization Code Division Multiple Access，TD-SCDMA)、频分双工(Frequency Division Duplex，FDD)系统、时分双工(Time Division Duplex，TDD)、第五代(5G)系统或新无线电(New Radio，NR)、未来演进的6G、7G通信系统，等等。
应当理解，本公开的实施例可以被应用到任何存在类似问题的通信系统中，例如无线局域网(WLAN)、有线通信系统、或者将来开发的其他通信系统等。
在本公开中使用的术语“终端设备”指能够与网络设备之间或者彼此之间进行有线或无线通信的任何终端设备。终端设备有时可以称为用户设备(User Equipment，UE)。终端设备可以是任意类型的移动终端、固定终端或便携式终端。作为示例，终端设备可以包括移动手机、站点、单元、设备、移动终端(Mobile Terminal，MT)、订阅台、便携式订阅台、互联网节点、通信器、台式计算机、膝上型计算机、笔记本计算机、平板计算机、个人通信系统设备、个人导航设备、个人数字助理(Personal Digital Assistant，PDA)、定位设备、无线电广播接收器、电子书设备、游戏设备、物联网(Internet of Things，IoT)设备、车载设备、飞行器、虚拟现实(Virtual Reality，VR)设备、增强现实(Augmented Reality，AR)设备、可 穿戴设备、5G网络中的终端设备或者演进的公用陆地移动网络(Public Land Mobile Network，PLMN)中的任何终端设备、可用于通信的其他设备、或者上述的任意组合。本公开的实施例对此并不做限定。
在本公开中使用的术语“网络设备”是可以用于与终端设备通信的实体或节点，例如可以是接入网设备。接入网设备可以是部署在无线接入网中为移动终端提供无线通信功能的装置，例如可以是无线接入网(Radio Access Network，RAN)网络设备。接入网设备可以包括各种类型的基站。作为示例，接入网设备可以包括各种形式的宏基站、微基站、微微基站、毫微微基站、中继站、接入点、卫星、远程无线电单元(Remote Radio Unit，RRU)、射频头(Radio Head，RH)、远程无线电头端(Remote Radio Head，RRH)等等，还可以是设备对设备(D2D)通信、机器对机器(M2M)通信中承担基站功能的设备。在采用不同的无线接入技术的系统中，接入网设备的名称可能会有所不同，例如在长期演进系统(Long Term Evolution，LTE)网络中称为演进的节点B(evolved NodeB，eNB或eNodeB)，在3G网络中称为节点B(NodeB，NB)，在5G网络中可以称为g节点B(gNB)或NR节点B(NR NB)，等等。在某些场景下，接入网设备可以包含集中单元(Central Unit，CU)和/或分布单元(Distributed Unit，DU)。CU和DU可以放置在不同的地方，例如：DU拉远，放置于高话务量的区域，CU放置于中心机房。或者，CU和DU也可以放置在同一机房。CU和DU也可以为一个机架下的不同部件。为方便描述，本公开后续的实施例中，上述为移动终端提供无线通信功能的装置统称为网络设备，本公开的实施例不再具体限定。
随着人工智能领域的发展，人们的生活方式发生了天翻地覆的变化，从AlphaGo大战人类顶级围棋手取得胜利，再到基于目标检测，图像分割的无人驾驶、AI医疗场景，以及机器翻译、文本处理、智能聊天等推动的自然语言领域，AI逐步融入人类社会的方方面面。
AI的兴起离不开大量优质的数据。但是，诸如安全隐私保护问题、利益阻碍交互问题等使得在不同实体或企业处的数据形成了数据孤岛。在此基础上，联邦学习应运而生，联邦学习在2016年提出，本质上是一种分布式学习框架，旨在在保证数据隐私安全的基础上，实现各端口共建全局模型。联邦学习允许用户在将数据保留在本地端不共享的前提下形成一个联合体，通过联合训练得到全局模型，从而避免数据隐私和安全问题。
在基于联邦学习进行模块训练的过程中，涉及频繁的交互，每次交互都涉及大量的数据。考虑到通信的效率，联邦学习的交互不应太过频繁，且交互的数据在传输前需要进行压缩，很多压缩的方法相继被提出。例如多步更新、周期平均的方法可以用来减小交互次数。例如参数剪枝、知识蒸馏等方式可以通过减少参数的数据量来提高传输效率。但是目前的模型压缩算法往往会以损失部分性能为代价，因此需要一种在保证数据性能的前提下，提升通信效率的方案。
针对上述问题以及其他潜在的问题，本公开的实施例提供了参数压缩方案，能够将参数划分为多个组并对各个组分别进行量化来将参数进行压缩，如此能够保证数据的性能，并且能够提升通信效率。
图1示出了本公开的实施例能够应用于其中的示例场景100的示意图。在场景100中包括第一设备110-1至第一设备110-N以及第二设备120。为了简洁，可以将第一设备110-1至第一设备110-N统称为第一设备110。
在一些示例中，第一设备110可以被实现为客户端设备，第二设备120可以被实现为服务器设备。在一些示例中，第一设备110可以被实现为终端设备，第二设备120可以被实现 为接入网设备。在一些示例中，第一设备110可以被称为分布式机器学习中的执行节点，第二设备120可以被称为分布式机器学习中的执行锚点。本公开对此不限定。
在分布式学习的过程中，以神经网络的模型训练为例，第二设备120可以将神经网络的初始的模型参数发送到N1个第一设备110，例如可以采用广播或组播的形式进行发送。N1个第一设备110可以基于本地数据集，在初始的模型参数的基础上进行学习，例如训练或迭代，从而得到处理后的参数，例如可以包括局部模型参数或梯度参数等。N1个第一设备110各自将得到的处理后的参数压缩之后发送到第二设备120。第二设备120对来自N1个第一设备110中各个第一设备110的处理后的参数进行解压，并通过聚合得到更新后的模型参数。如此，在第二设备120处的神经网络的模型参数从初始的模型参数变更为更新后的模型参数。
进一步地，基于类似的过程，可以执行下一轮的训练或学习，得到进一步更新后的模型参数。也就是说，第二设备120可以将更新后的模型参数发送到N2个第一设备110，由N2个第一设备110得到再次处理后的参数并发送回第二设备120。如此反复，直到参数收敛，或者直到训练失败，例如达到最大迭代次数。为了简化描述，在下文中可以将第二设备120发送到第一设备110的参数统称为待训练参数，如神经网络的初始的模型参数，或者经上一轮训练之后得到的聚合后的更新后的模型参数。
在训练过程中，不同次的迭代可以由相同或不同的第一设备110来执行，例如上述的N1与N2可以相等或不相等，例如上述的N1个第一设备110与N2个第一设备110可以完全不同，也可以部分相同，也可以全部相同。例如N1个第一设备110包括第一设备110-1，N2个第一设备110包括或不包括第一设备110-1。
可见，在分布式学习的过程中会涉及多轮数据交互，例如每一轮中第二设备120从一个或多个第一设备110接收压缩的参数。
在一些实施例中，多轮交互中，各个第一设备110所使用的压缩方案是固定的。在另一些实施例中，多轮交互中，各个第一设备110所使用的压缩方案可以是动态调整的，例如可以在中间轮交互中，基于参数的属性更新压缩方案。为了简化描述，下面的实施例中以第一设备110的一次压缩为例进行阐述，但是可理解，针对多轮交互中各次压缩的过程是类似的。
图2示出了根据本公开的一些实施例的信息处理过程200的信令交互图。过程200涉及第一设备110和第二设备120，图3中的第一设备110指的是如图1中的第一设备110-1至第一设备110-N中的任一个。
第一设备110将分布式训练模型的参数划分210为多个组。每个组包括多个参数，不同的组中的参数的数量可以相同也可以不同。例如第一组中包括M1个参数，第二组中包括M2个参数，M1和M2可以相等也可以不相等。不同组中多个参数的取值可以位于相同的区间或不同的区间。以多个组中的第一组为例，第一组包括M1个参数，那么可以基于M1个参数的取值的最小值与M1个参数的取值的最大值来确定第一组的数值区间，例如可以假设为闭区间[a1,b1]。类似的，针对包括M2个参数的第二组，可以确定第二组的数值区间，例如可以假设为[a2,b2]。在一例中，a1＝a2且b1＝b2，即第一组的数值区间与第二组的数值区间相同。在另一例中，a1≠a2和/或b1≠b2，即第一组的数值区间与第二组的数值区间不同。可理解，此处的第一组可以是多组中的任意一组，第二组可以为多组中不同于第一组的任意一组。示例性地，多个组中至少两个组的数值区间不同。注意本申请的闭区间[a1,b1]还可以是开区间(a1,b1),开闭区间(a1,b1]，或者闭开区间[a1,b1)，并不影响本申请的实质，不失一般性，本申请均以[a1,b1]为例，[a2,b2]也类似。
可选地，在一些示例中，多个组中任意两个组之间具有以下不同：参数的取值的数值区间、参数的类型、或参数所属的网络层类型，其中，参数的类型例如为权重、偏置、方差、均值等，其中参数所属的网络层类型例如为卷积层、全连接层等。
分布式训练模型的参数可以是第一设备110利用本地(即存储在第一设备110的存储器中)的数据集，基于来自第二设备120的上一轮的模型参数，通过学习或训练之后而得到的。
第一设备110可以基于参数划分方式，将分布式训练模型的参数划分为多个组。
在一些实施例中，参数划分方式可以指示基于如下至少一项来进行参数划分：参数的类型、参数所在的网络层的类型等。网络层的类型例如包括卷积层、全连接层等。参数的类型例如包括权重参数、偏置参数、均值参数等。
示例性地，可以将同一类型的网络层中且相同类型的参数划分为一组。图3示出了根据本公开的一些实施例的参数划分300的示意图。在图3中，可以将各个卷积层中的代表权重的参数划分为一组，如组310。将各个卷积层中的代表偏置的参数划分为一组，如组320。将各个全连接层中的代表权重的参数划分为一组，如组330。将各个全连接(Full Connection，FC)层中的代表偏置的参数划分为一组，如组340。如此，不同的神经网络层的参数可能属于同一个组。可选地，可以将参照图3所描述的参数划分方式称为基于类型的划分方式。
示例性地，可以将同一个网络层中的不同类型的参数划分为不同的组。图4示出了根据本公开的一些实施例的参数划分400的示意图。在图4中，可以将卷积层401中的代表权重的参数划分为一组，如组410。将卷积层401中的代表偏置的参数划分为一组，如组420。将卷积层402中的代表权重的参数划分为一组，如组430。将卷积层402中的代表偏置的参数划分为一组，如组440。如此，位于同一个组的参数必然属于同一个神经网络层。可选地，可以将参照图4所描述的参数划分方式称为基于矩阵的划分方式。
示例性地，可以将同一个网络层中的参数都划分为同一个组。图5示出了根据本公开的一些实施例的参数划分500的示意图。在图5中，可以将卷积层501中的参数划分为一组，如组510。将批归一化层502中的参数划分为一组，如组520。将卷积层503中的参数划分为一组，如图530。如此，位于同一个网络层的参数属于同一个组。可选地，可以将参照图5所描述的参数划分方式称为基于网络层的划分方式。
在一些实施例中，参数划分方式也可以指示按照参数的其他特性来进行划分，这里不再一一罗列。
示例性地，参数划分方式可以是由第二设备120确定后再指示给第一设备110的，或者参数划分方式可以是由第一设备110确定的，如下结合图6至图11的实施例的详细阐述。
第一设备110基于与多个组中每个组对应的量化比特数，确定220与多个组对应的多个量化组。示例性地，量化比特数可以用于指示对应的量化组中各个量化参数的长度，即所占用的比特数。多个量化组所占用的存储空间小于多个组所占用的存储空间，例如可以为50％或其他值等，如此实现了对多个组的压缩。
每个量化组可以包括多个量化参数，其中多个量化参数中每个量化参数的长度等于或小于对应的量化比特数。具体而言，以多个组中第一组为例，假设第一组包括M1个参数，与第一组对应的量化比特数为n1，那么第一组对应的第一量化组包括M1个量化参数，并且M1个量化参数中每个量化参数的比特位数可以等于n1。
每个量化组可以包括量化边界值以及多个量化参数，其中量化边界值可以是基于对应组中多个参数的取值的数值区间而被确定的，其中多个量化参数中每个量化参数的长度等于或 小于对应的量化比特数。
第一设备110可以确定多个组中每个组的量化边界值确定方式。在一例中，第一设备110可以确定量化边界值确定方式，并且该量化边界值确定方式被应用于每个组。在另一例中，针对多个组中各个组，第一设备110可以分别确定对应的量化边界值确定方式。
在一些实施例中，量化边界值确定方式可以指示是否按照绝对值来确定量化边界值。示例性地，不按照绝对值来确定量化边界值包括：将组中多个参数的最大值和最小值确定为量化边界值。以多个组中第一组为例，第一组包括M1个参数，且M1个参数的取值的数值区间为闭区间[a1,b1]，那么相应的可以确定量化边界值包括a1和b1。示例性地，按照绝对值来确定量化边界值包括：将组中多个参数的绝对值的最大值和最小值确定为量化边界值；或者将组中多个参数的绝对值的最大值确定为量化边界值。以多个组中第一组为例，第一组包括M1个参数，且M1个参数的取值的数值区间为闭区间[a1,b1]，如果a1和b1均为正值，那么可选地可以确定量化边界值包括a1和b1；如果a1为负值，b1为正值，那么可选地可以确定量化边界值包括max(-a1,b1)，其中max表示取最大值。作为一个示例，可以将不按照绝对值来确定量化边界值的量化边界值确定方式称为传统量化方式，将按照绝对值来确定量化边界值的量化边界值确定方式称为符号量化方式，但是应理解，该名称仅是示意，不应解释为对本公开的实施例的限制。
在一些实施例中，量化边界值确定方式也可以指示按照参数的其他取值特性来进行确定，这里不再一一罗列。
示例性地，量化边界值确定方式可以是由第二设备120确定后再指示给第一设备110的，或者量化边界值确定方式可以是由第一设备110确定的，或者可以是由第一设备110基于第二设备120的配置而确定的，如下结合图6至图11的实施例的详细阐述。
在一些示例中，对于多个组而言，各个组的量化比特数可以相等。例如，第一设备110或第二设备120可以确定量化比特数，并将该量化比特数用于每个组。以此方式，每个量化组中的各个量化参数的长度都相等。在一些示例中，多个组中至少两个组的量化比特数不同。例如，第一设备110或第二设备120可以针对各个组，分别确定对应的量化比特数。以此方式，位于不同量化组的量化参数的长度可能相等也可能不相等。举例而言，假设第一组包括M1个参数，与第一组对应的量化比特数为n1，那么第一组对应的第一量化组包括M1个量化参数，并且M1个量化参数中每个量化参数的比特位数可以等于n1。假设第二组包括M2个参数，与第二组对应的量化比特数为n2，那么第二组对应的第二量化组包括M2个量化参数，并且M2个量化参数中每个量化参数的比特位数可以等于n2。
可选地，在一些示例中，每个量化参数中可以包括用于表示对应的参数的正负的至少一个比特位。举例而言，假设某个量化组的量化边界值确定方式为符号量化方式(即按照绝对值来确定量化边界值)，假设该量化组中的量化参数的比特位数等于n1，那么n1个比特中的任一个比特(例如第一个比特)表示其正负，剩余的n1-1位比特表示参数大小。
本公开的实施例中，量化比特数可以是由第二设备120确定后再指示给第一设备110的，或者量化比特数可以是由第一设备110确定的，或者可以是由第一设备110基于第二设备120的配置而确定的，如下结合图6至图11的实施例的详细阐述。
可选地，如图2所示，第一设备110可以对多个量化组进行熵编码222。在一些示例中，可以基于来自第二设备120的指示，来执行熵编码。在一些示例中，第一设备110可以自行决定执行熵编码，并通知第二设备120。关于熵编码可以参照如下结合图6至图11的实施例 中的详细阐述。
第一设备110向第二设备120发送230多个量化组。可选地，在第一设备110已经进行熵编码的实施例中，被发送的多个量化组是经熵编码的多个量化组。
相应的，第二设备120能够接收来自第一设备110的多个量化组。可理解，在一些示例中，第二设备120先前将上一轮的模型参数发送到多个第一设备110，那么相应地第二设备120可以接收来自多个第一设备110中各个第一设备110发送的多个量化组。
第二设备120确定240与多个量化组对应的多个组。示例性地，第二设备120可以基于量化方式以及量化比特数，执行反量化，从而得到多个组。
第二设备120确定250模型参数。示例性地，第二设备120可以基于参数划分方式，确定经反量化的多个组中的各个参数所在的网络模型中的位置。从而第二设备120能够得到经本轮训练所得到的模型参数。
进一步地，可理解，第二设备120可以基于来自多个第一设备的模型参数，通过聚合得到下一轮的待训练参数。在这之后的流程与上面是类似的，本文中不再重复。
下面将结合图6至图11描述本公开的实施例中进行量化和反量化的更具体的实现。
图6示出了根据本公开的一些实施例的信息传输过程600的信令交互图。过程600涉及第一设备110和第二设备120，图6中的第一设备110可以包括如图1中的第一设备110-1至第一设备110-N中的任一个。
第二设备120确定610压缩配置消息。压缩配置消息包括第一指示信息，第一指示信息指示参数划分方式。示例性地，第二设备120可以确定参数划分方式。
在一些实施例中，第二设备120可以基于以下因素中的至少一项来确定参数划分方式：待训练参数的参数属性、待训练的模型的网络结构、通信效率、模型性能。待训练参数可以是初始的模型参数，或者可以是基于上一轮训练得到的更新后的模型参数。参数属性可以包括：参数的取值范围、参数的类型、参数所在的网络层等，其中参数的类型例如包括权重、偏置、方差、均值等。网络结构可以包括模型中的网络层的类型、网络层的数量等，其中网络层的类型例如包括卷积层、全连接层等。
示例性地，第二设备120所确定的参数划分方式可以为如图3所示的基于类型的划分方式或者如图4所示的基于矩阵的划分方式或者如图5所示的基于网络层的划分方式。
在一些示例中，第一指示信息可以通过1比特来指示，例如“0”表示基于矩阵的划分方式，“1”表示基于类型的划分方式。在一些示例中，第一指示信息可以通过2比特来指示，例如“00”表示基于矩阵的划分方式，“01”表示基于类型的划分方式，“10”表示基于网络层的划分方式。在另一些示例中，第一指示信息也可以通过更多比特来指示，本公开对此不限定。
以基于类型的划分方式和基于矩阵的划分方式两者比较而言，可理解，基于类型的划分方式的信令开销比基于矩阵的划分方式的信令开销更小，但是基于类型的划分方式的模型性能不如基于矩阵的划分方式的模型性能。在实际中，可以基于通信效率和模型性能进行更好的折中，从而确定合适的参数划分方式。
举例而言，假设待训练参数的数量为s，如果按照基于类型的划分方式可以被划分为r1组，如果按照基于矩阵的划分方式可以被划分为r2组。那么如果满足(r2-r1)×32×2<s×d，则确定参数划分方式为基于矩阵的划分方式。相反，如果满足(r2-r1)×32×2≥s×d，则确定参数划分方式为基于类型的划分方式。s、r1和r2都是正整数，32代表浮点数在计算机系统 中所占用的比特数，d为预定值，例如d可以是用户根据经验信息所确定的两种不同的参数划分方式的比特数的差值。
举例而言，假设待训练参数的数量为s，如果按照基于类型的划分方式可以被划分为r1组。那么如果s/r1小于预设的阈值，则确定参数划分方式为基于类型的划分方式。相反，如果满足s/r1大于或等于该预设的阈值，则确定参数划分方式为基于矩阵的划分方式。预设的阈值可以是用户预定义的，或者预设的阈值可以是基于历史划分方式所确定的，例如等于历史训练的模型中各个组的平均参数的数量的最大值。
为了表述方式，下文中可以假设参数划分方式为基于类型的划分方式或者基于矩阵的划分方式，但是应理解，本公开不限于此，例如还可以设定划分后每个组中的参数的数量阈值，例如可以限定每个组中的多个参数的数量位于第一数量阈值与第二数量阈值之间，第一数量阈值小于第二数量阈值。举例而言，如果采用某种参数划分方式导致组1中的多个参数的数量大于第二数量阈值，可以将该组1进一步拆分为至少两个组。这样能够避免某个组参数数量过大造成的性能损失。如果采用某种参数划分方式导致组2中的多个参数的数量小于第一数量阈值，可以将该组2与其余的组合并。这样能够进一步地提升压缩效率，避免过大的信令开销。
可选地，在本公开的一些实施例中，压缩配置消息可以包括第二指示信息。也就是说，压缩配置消息包括第一指示信息和第二指示信息。
在一些实施例中，第二指示信息可以指示量化边界值确定方式(或简称为量化方式)。示例性地，第二设备120可以确定量化边界值确定方式，并且该量化边界值确定方式将被用于每个组，也就是说，每个组的量化边界值确定方式都相同。
示例性地，第二设备120可以将历史训练的其余模型所使用的频次最多的量化方式作为量化边界值确定方式。
示例性地，第二设备120可以基于参数的取值分布情况来确定量化边界值确定方式。举例而言，如果参数的取值分布不均匀，那么可以确定使用传统量化方式，即考虑正负数的取值，这样能够减小误差。例如某个组中的多个参数都是正值或都是负值，例如某个组中最大取值(为正)与最小取值(为负)之差较大。举例而言，如果参数为反向更新的梯度或者为模型的残差参数，此时参数的正负更加重要，那么可以确定使用符号量化方式，即基于绝对值来量化，从而能够确保各个参数保持正确的更新方向，避免在0附近的参数的梯度方向不准确。
另外可理解，针对符号量化方式，参数的取值的最小值可能为0或者接近于0，在此情况下，量化边界值可以包括绝对值的最大值而不包括绝对值的最小值，这样能够节省信令开销。
在一些示例中，第二指示信息可以通过1比特来指示，例如“0”表示传统量化方式，“1”表示符号量化方式。在另一些示例中，第二指示信息也可以通过更多比特来指示，例如可以通过2比特来指示，其中一比特表示多个组使用相同的量化方式，另一比特表示各个组所使用的量化方式，本公开对此不限定。
在一些实施例中，第二指示信息可以指示量化方式配置，该第二指示信息可以被用于由第一设备110来确定量化方式。
在一些示例中，第二指示信息可以通过1比特来指示，例如“0”表示多个组使用相同的量化方式，“1”表示针对每个组分别确定量化方式。在一些示例中，第二指示信息可以通过 m1比特来指示，例如其中的一个比特(如第一个比特)用于表示是否每个组使用相同的量化方式，另外的m1-1个比特表示用于确定量化方式的阈值。在一些示例中，第二指示信息可以通过m1-1比特来指示，表示用于确定量化方式的阈值。示例性地，基于量化方式配置来确定量化方式的具体实施例可以参照下面结合650的描述。
可选地，在本公开的一些实施例中，压缩配置消息还包括第三指示信息。此时，压缩配置消息可以包括第一指示信息、第二指示信息和第三指示信息。
在一些实施例中，第三指示信息可以指示量化比特数。示例性地，第二设备120可以确定量化比特数，并且该量化比特数将被用于每个组，也就是说，每个组的量化比特数都相同。可选地，可以将用于所有组的量化比特数称为轮次量化比特数或者粗粒度量化比特数。
第二设备120可以基于第一设备110与第二设备120之间的传输资源和/或分布式神经网络的网络层数来确定量化比特数。可选地，第二设备120可以为第一设备110分配从第一设备110到第二设备120的传输资源，并基于该分配的传输资源来确定量化比特数。
在一些示例中，第三指示信息可以通过m2比特来指示，可以表示量化比特数的最大值为2 m2。在另一些示例中，第三指示信息也可以通过更多或更少的比特来指示，本公开对此不限定。
在一些实施例中，第三指示信息可以指示量化比特数配置，该第三指示信息可以被用于由第一设备110来确定量化比特数。
量化比特数配置可以指示量化粒度以及用于确定量化比特数的量化阈值和量化比特数列表，其中量化粒度可以指示多个组是否使用相同的量化比特数。
在一些示例中，量化粒度可以指示多个组使用相同的量化比特数，例如轮次量化比特数或者粗粒度量化比特数，可选地，量化粒度可以指示粗粒度。在一些示例中，量化粒度可以指示多个组使用不同的量化比特数，例如细粒度量化比特数，可选地，量化粒度可以指示细粒度。在一些示例中，量化粒度可以指示粗粒度和细粒度的结合。
举例而言，量化比特数配置可以包括量化阈值数量以及量化阈值列表，其中量化阈值数量用于表示量化阈值列表中的量化阈值的个数，量化阈值列表中可以包括多个量化阈值。例如，假设量化阈值数量为K，K为正整数，假设量化阈值列表包括(T 1,T 2,…,T K)。
量化比特数配置还可以包括列表连续性指示和量化列表，其中列表连续性指示用于表示量化列表的表示形式，量化列表中包括一个或多个量化比特数。例如，可以通过1比特来表示列表连续性指示。例如，如果指示不连续，量化列表可以包括(b 0,b 1,b 2,…,b K)。如果指示连续，量化列表可以包括b 0和/或b K。
可选地，在一些示例中，量化粒度可以指示粗粒度和细粒度的结合。量化比特数配置可以包括与粗粒度对应的第一量化阈值数量、第一量化阈值列表、第一列表连续性指示和第一量化列表，量化比特数配置还包括与细粒度对应的第二量化阈值数量、第二量化阈值列表、第二列表连续性指示和第二量化列表。或者，量化比特数配置可以不包括第二列表连续性指示和第二量化列表，实际场景中可以基于与粗粒度对应的第一列表连续性指示和第一量化列表来确定与粗粒度对应的第二列表连续性指示和第二量化列表。
示例性地，基于量化比特数配置来确定量化比特数的具体实施例可以参照下面结合650的描述。
可选地，在一个示例中，该量化比特数配置可以指示预设条件，从而第一设备110在确定满足预设条件时，可以认为量化比特数为0，或者可以理解为第一设备110在完成本轮迭 代训练后不向第二设备120发送处理后的参数。例如，第二设备120可以基于前一轮的参数自己确定出本轮的处理后参数，那么第二设备120可以通过该配置避免第一设备110发送本轮处理后的参数，这样能够节省信令开销。
可选地，在本公开的一些实施例中，压缩配置消息还包括熵编码指示信息。此时，压缩配置消息可以包括第一指示信息、第二指示信息和熵编码指示信息。或者此时，压缩配置消息可以包括第一指示信息、第二指示信息、第三指示信息和熵编码指示信息。
熵编码指示信息可以指示是否进行熵编码。例如，可以指示第一设备110在将处理后的参数进行量化之后是否进行算数编码。在一些示例中，熵编码指示信息可以通过1比特来指示，例如“0”表示进行熵编码，“1”表示不进行熵编码。在另一些示例中，熵编码指示信息也可以通过更多比特来指示，本公开对此不限定。
图7示出了根据本公开的一些实施例的压缩配置消息700的示意图。如图7所示，压缩配置消息700包括第一指示信息710和和第二指示信息720，可选地还可以包括第三指示信息730和/或熵编码指示信息740。第二指示信息720可以指示量化方式722或者可以指示量化方式配置724。第三指示信息730可以指示量化比特数732或者可以指示量化比特数配置734，其中量化比特数配置734可以包括量化粒度7342以及量化阈值和量化比特数列表7344。
第二设备120向第一设备110发送620压缩配置消息。在一些示例中，第二设备120可以通过信令发送该压缩配置消息，例如第二设备120为接入网设备，第一设备110为终端设备，那么可选地该信令可以为物理层信令或者可以为无线资源控制层(Radio Resource Control，RRC)信令。
第二设备120向第一设备110发送630待训练参数。待训练参数可以为如上所述的神经网络的初始的模型参数，或者可以为经上一轮训练之后得到的更新后的模型参数。待训练参数可以包括神经网络的与网络结构关联的参数，例如权重参数等。待训练参数可以包括神经网络的用于迭代的参数，例如梯度参数等。
在一些示例中，第二设备120可以将压缩配置消息可以与待训练参数一起发送到第一设备110。在另一些示例中，第二设备120可以通过不同的信令分别将压缩配置消息与待训练参数发送到第一设备110，本公开对此不限定。
可理解，本公开的实施例中，第二设备120向多个第一设备110发送压缩配置消息和待训练参数，例如结合图1所描述的N1个第一设备110或者N2个第一设备110。为了简化描述，本公开实施例以一个第一设备110进行阐述，针对其余第一设备110所执行的操作是类似的，本文中不再重复。
相应地，第一设备110可以接收来自第二设备120的压缩配置消息和待训练参数。第一设备110可以基于待训练参数得到640分布式训练模型的参数。具体而言，第一设备110可以在待训练参数的基础上，使用本地数据集对网络模型进行训练，从而得到本轮训练后的参数，即分布式训练模型的参数。
第一设备110可以基于压缩配置消息确定650参数划分方式、量化方式和量化比特数。下面将通过不同的压缩配置消息来分别描述第一设备110确定参数划分方式、量化方式和量化比特数的相应实施例。
在本公开的一些实施例中，压缩配置消息包括第一指示信息、第二指示信息和第三指示信息，其中第一指示信息指示参数划分方式，第二指示信息指示量化方式，第三指示信息指示量化比特数。那么，第一设备110可以从压缩配置消息中读取参数划分方式、量化方式和 量化比特数。可理解，量化方式被用于使用参数划分方式被划分后的多个组，并且量化比特数被用于多个组。
在本公开的一些实施例中，压缩配置消息包括第一指示信息和第二指示信息，其中第一指示信息指示参数划分方式，第二指示信息指示量化方式。但压缩配置消息不包括第三指示信息。那么第一设备110可以从压缩配置消息中读取参数划分方式和量化方式，并且第一设备110可以确定量化比特数。
可选地，第一设备110可以基于参数的取值大小、参数的取值的分布情况等来确定量化比特数。可选地，第一设备110可以基于以下中的一项或多项来确定量化比特数：第一设备110至第二设备120的传输资源、分布式神经网络的网络层数、以及训练的损失函数。
可理解，第一设备110可以确定量化比特数，且其被用于每个组，即粗粒度量化比特数。或者，第一设备110可以针对各个组分别确定量化比特数，即细粒度量化比特数。
在本公开的一些实施例中，压缩配置消息包括第一指示信息和第二指示信息，其中第一指示信息指示参数划分方式，第二指示信息指示量化方式配置。但压缩配置消息不包括第三指示信息。那么第一设备110可以从压缩配置消息中读取参数划分方式。
第一设备110可以基于量化方式配置来确定量化方式。在一些实施例中，量化方式配置指示多个组使用相同的量化方式，那么第一设备110可以确定量化方式，且该量化方式被用于每个组。在另一些实施例中，量化方式配置指示针对多个组分别确定量化方式，那么第一设备110可以基于量化方式配置中的阈值来确定对应组的量化方式。
举例而言，以多组中的第i组为例，假设该第i组中的多个参数的最大值为max i，最小值为min i，且满足min i<0<max i。在一例中，如果max(max i,|min i|)-min(max i,|min i|)<thres，则可以确定该第i组的量化方式为符号量化方式。在另一例中，如果max(max i,|min i|)/min(max i,|min i|)<thres，则可以确定该第i组的量化方式为符号量化方式。|min i|表示min i的绝对值，max()表示取最大值，min()表示取最小值，thres表示阈值。可理解，该举例仅是示意，在实际场景中，可以通过其他的方式来基于阈值确定量化方式，本公开不再一一罗列。
第一设备110可以确定量化比特数。可选地，第一设备110可以基于参数的取值大小、参数的取值的分布情况等来确定量化比特数。可选地，第一设备110可以基于以下中的一项或多项来确定量化比特数：第一设备110至第二设备120的传输资源、分布式神经网络的网络层数、以及训练的损失函数。
可理解，第一设备110可以确定量化比特数，且其被用于每个组，即粗粒度量化比特数。或者，第一设备110可以针对各个组分别确定量化比特数，即细粒度量化比特数。
在本公开的一些实施例中，压缩配置消息包括第一指示信息、第二指示信息和第三指示信息，其中第一指示信息指示参数划分方式，第二指示信息指示量化方式配置，第三指示信息指示量化比特数配置。那么第一设备110可以从压缩配置消息中读取参数划分方式。
第一设备110可以基于量化方式配置来确定量化方式。在一些实施例中，量化方式配置指示多个组使用相同的量化方式，那么第一设备110可以确定量化方式，且该量化方式被用 于每个组。在另一些实施例中，量化方式配置指示针对多个组分别确定量化方式，那么第一设备110可以基于量化方式配置中的阈值来确定对应组的量化方式。具体示例可以参照上面的实施例所述。
第一设备110可以基于量化比特数配置来确定量化比特数，其中量化比特数配置包括量化粒度、量化阈值列表和量化列表。举例而言，第一设备110可以确定待比较值(表示为d)，将待比较值与量化阈值列表中的两个阈值进行比较，以确定该待比较值所处的量化阈值区间，进而确定与量化阈值区间所对应的量化列表中的量化比特数。
具体地，假设量化阈值列表包括(T 1,T 2,…,T K)，量化列表可以包括(b 0,b 1,b 2,…,b K)。将待比较值表示为d，将所确定的量化比特数表示为quan bit，那么满足下式：
Figure PCTCN2022141926-appb-000001
示例性地，待比较值可以基于损失函数或者基于所划分的组中多个参数的取值大小等来确定。
在一些实施例中，量化比特数配置包括量化粒度，且该量化粒度指示粗粒度，也就是说多组使用同一个量化比特数。第一设备110可以将本轮训练得到的损失函数与量化比特数配置中的量化阈值进行比较，并基于比较结果来确定量化比特数。
举例而言，针对一定数量的训练样本，例如一个批块(batch)，假设在本轮训练之前的该batch的损失函数为x，在本轮训练之后的该batch的损失函数为y，可以将abs(y-x)与量化阈值进行比较，其中abs表示取绝对值或取模。假设量化阈值列表包括(T 1,T 2,…,T K)，量化列表可以包括(b 0,b 1,b 2,…,b K)。如果满足T 1≤abs(y-x)<T 2，则可以确定量化比特数为b 1。
可选地，第一设备110还可以确定与量化比特数对应的索引，该索引可以表示所确定的量化比特数在量化列表中的位置，例如与b 2对应的索引可以为2。
在一些实施例中，量化比特数配置包括量化粒度，且该量化粒度指示细粒度，也就是说针对不同的组需分别确定量化比特数。针对每个组，第一设备110可以将该组中多个参数的取值大小的范围与量化比特数配置中的量化阈值进行比较，并基于比较结果来确定量化比特数。
举例而言，以多组中的第i组为例，假设该第i组中的多个参数的最大值为max i，最小值为min i，那么可以将abs(max i-min i)与量化阈值进行比较，其中abs表示取绝对值或取模。假设量化阈值列表包括(T 1,T 2,…,T K)，量化列表可以包括(b 0,b 1,b 2,…,b K)。如果满足T 1≤abs(max i-min i)<T 2，则可以确定该第i组的量化比特数为b 1。
以此方式，第一设备110可以确定与多个组分别对应的多个量化比特数。可选地，第一设备110还可以确定与各个量化比特数对应的索引，该索引可以表示该组所使用的量化比特数在量化列表中的位置，例如第i组的量化比特数b 2对应的索引可以为2。
可理解，本公开实施例中的待比较值还可以基于其他的因素来确定，例如组中的多个参数的数量，例如传输资源的大小等，本公开不再一一罗列。可理解，在一些示例中，可以将量化阈值中的设置的T 1接近于0，例如10 -3或其他值等，相应地可以将量化列表中的b 0设置 为0。如果针对多组中的第j组，在所确定的量化比特数为b 0，可以认为无需向第二设备120发送与量化比特数对应的数据，例如针对第j组的量化参数可以仅包括量化边界值，如max j和min j。
第一设备110基于分布式训练模型的参数确定660多个量化组。示例性地，每个量化组可以包括量化边界值以及多个量化参数。
具体而言，第一设备110可以使用参数划分方式，将分布式训练模型的参数划分为多个组。在一些示例中，第一设备110可以使用量化方式，确定多个组中每个组的量化边界值。第一设备110可以基于量化比特数，确定多个组中每个组的多个量化参数。在另一些示例中，针对多个组中的每个组，第一设备110可以使用该组的量化方式，确定该组的量化边界值，基于该组的量化比特数，确定该组的多个量化参数。
示例性地，确定多个量化组的过程660也可以被称为量化过程或者压缩过程。应注意的是，过程660与上述的过程650可以是交叉进行的，例如可以先使用参数划分方式将参数划分为多组，然后在基于各组中的多个参数的取值等来确定量化方式和量化比特数。
量化处理可以包括：使用参数划分方式，将参数划分为多组；针对每个组，基于量化方式确定量化边界值，基于量化比特数确定与组中多个参数对应的多个量化参数。该量化方式可以参照前述结合图2的描述，为了简洁，这里不再重复。
可选地，第一设备110可以进行熵编码670。
在一些实施例中，压缩配置消息可以包括熵编码指示信息，且该熵编码指示信息指示进行熵编码。那么第一设备110还可以对量化参数进行熵编码。
在另一些实施例中，尽管压缩配置消息不包括熵编码指示信息，或者，压缩配置消息包括熵编码指示信息，且该熵编码指示信息指示不进行熵编码，但是第一设备110可以自行确定进行熵编码，那么第一设备110还可以对量化参数进行熵编码。举例而言，如果第一设备110确定量化参数的压缩率低于压缩率阈值，可以自行决定进行熵编码。
第一设备110向第二设备120发送680多个量化组。在一些示例中，多个量化组可以是经熵编码后的多个量化组。相应地，第二设备120可以接收来自第一设备110的多个量化组。
在一些示例中，第一设备110可以基于多个量化组确定压缩数据，并向第二设备120发送压缩数据。图8示出了根据本公开的一些实施例的压缩数据800的示意图。压缩数据800包括量化边界值数据域810和量化参数数据域820。量化边界值数据域810可以包括多个量化组中的量化边界值。作为一例，可以包括(min1，max1)，(min2，max2)，…。作为另一例，可以包括(min1，min2，…)和(max1，max2，…)。作为再一例，可以包括(max1，max2，…)。量化参数数据域820可以包括多个量化组中的量化参数，或者可以包括经熵编码的量化参数。
可选地，在本公开的一些实施例中，第一设备110可以向第二设备120发送6822量化比特反馈消息。如上所述，压缩配置消息可以包括第一指示信息、第二指示信息和第三指示信息，其中第一指示信息指示参数划分方式，第二指示信息指示量化方式，第三指示信息指示量化比特数。
第一设备110可以基于分布式训练模型的参数确定另一量化比特数。量化比特反馈消息可以指示另一量化比特数，例如第一设备110所确定的为了量化分布式训练模型的参数所需要的最小的另一量化比特数。可选地，可以经由同一信令将该量化比特反馈消息与多个量化 组一起发送到第二设备120。或者可选地，可以经由不同信令，分别将该量化比特反馈消息与多个量化组发送到第二设备120。
第二设备120可以基于传输资源量等确定是否接受该量化比特反馈消息所包括的另一量化比特数。可选地，第二设备120可以向第一设备110发送6824量化比特响应消息，该量化比特响应消息可以指示接受或拒绝另一量化比特数。如果第二设备120接受，那么第二设备120可以随后向第一设备110发送针对下一轮次训练的压缩配置消息，并且该压缩配置消息中的第三指示信息可以指示该另一量化比特数。
可选地，在本公开的一些实施例中，第一设备110可以向第二设备120发送6842量化比特数的指示信息。如上所述，压缩配置消息可以包括第一指示信息和第二指示信息，其中第一指示信息指示参数划分方式，第二指示信息指示量化方式，压缩配置消息不包括第三指示信息。
第一设备110可以将用于得到多个量化组的量化比特数发送到第二设备120，以便第二设备120基于该量化比特数进行后续的反量化。可选地，可以经由同一信令将该量化比特数与多个量化组一起发送到第二设备120。或者可选地，可以经由不同信令，分别将该量化比特数与多个量化组发送到第二设备120。
可选地，在本公开的一些实施例中，第一设备110可以向第二设备120发送6862量化指示消息，该量化指示消息可以指示第一设备110所使用的量化方式以及量化比特数。如上所述，如果压缩配置消息包括第一指示信息、第二指示信息和第三指示信息，其中第一指示信息指示参数划分方式，第二指示信息指示量化方式配置，第三指示信息指示量化比特数配置。如上所述，第一设备110可以基于量化方式配置确定量化方式，可以基于量化比特数配置确定量化比特数。
举例而言，第二指示信息指示量化方式配置，且该量化方式配置指示多个组使用相同的量化方式。那么，该量化指示消息可以通过1比特来指示所使用的量化方式。举例而言，第二指示信息指示量化方式配置，且该量化方式配置指示不同组使用不同的量化方式。那么，该量化指示消息可以通过多个比特来分别指示多个组的量化方式。
举例而言，第三指示信息指示量化比特数配置，且量化比特数配置包括量化粒度、量化阈值列表和量化列表。那么，该量化指示消息可以通过索引来指示所使用的量化比特数。可理解，针对粗粒度的情形，索引的数量为1个。针对细粒度的情形，索引的数量为多个，对应于多个组的数量。关于索引的描述可以参照上述在过程650中的相应实施例。
可选地，第一设备110还可以向第二设备120发送6882熵编码指示信息。举例而言，在压缩配置消息不包括熵编码指示信息的情况下，第一设备110可以通过该熵编码指示信息告知第二设备120是否进行了熵编码，例如可以通过1比特进行指示。从而，第二设备120可以基于此对接收到的数据成功地进行解压缩。
第二设备120对多个量化组进行反量化690。具体而言，第二设备120可以基于参数划分方式、量化方式以及量化比特数，对多个量化组进行反量化。关于反量化的描述可以参照上述结合图2部分的相应实施例，为了简洁，这里不再重复。
示例性地，第二设备120基于来自多个第一设备110的多个量化组，通过反量化和聚合得到下一轮的待训练参数。可选地，第二设备120可以再次发送压缩配置消息，如过程610和620所示。或者可选地，如果针对下一轮的压缩配置消息与上一轮的压缩配置消息相同，第二设备120可以省却发送针对下一轮的压缩配置消息。对于第一设备110而言，如果没有 收到针对当前轮次的压缩配置消息，则可以使用上一轮的压缩配置消息。这样能够避免相同内容的重复传输，减小信令开销。
根据如上结合图6至图8所描述的实施例，第一设备110能够基于压缩配置消息对分布式训练模型的参数进行量化，从而实现了对参数的压缩。该过程既考虑了诸如传输资源等因素，又考虑了参数的取值大小等因素，从而能够确保对参数的压缩既能保证通信效率，又能保证参数的性能。一方面避免过度压缩导致的性能损失，另一方面又避免了因压缩率低导致的时延过大。
本公开的实施例中，第一设备110向第二设备120发送多个量化比特的传输资源可以是第二设备120进行分配的，或者可以是第一设备110通过资源请求后由第二设备120分配的。
举例而言，假设第一设备110为终端设备，第二设备120为接入网设备。在一些示例中，第二设备120可以通过资源指示信息来指示第一设备110向第二设备120进行传输的传输资源，以及可选地可以指示编码调制方案等。在另一些示例中，第一设备110可以向第二设备120发送资源请求，并且第一设备110可以接收来自第二设备120的资源分配消息。随后第一设备110可以在分配的资源上向第二设备120发送多个量化组。例如，资源请求可以是媒体接入控制(Medium Access Control，MAC)层的缓存状态报告(Buffer State Report，BSR)。关于传输资源的调度等可以参照已有技术，本文中不再详细阐述。
图9示出了根据本公开的一些实施例的信息传输过程900的信令交互图。过程900涉及第一设备110和第二设备120，图9中第一设备110可以包括如图1中的第一设备110-1至第一设备110-N中的任一个。
第二设备120向第一设备110发送910待训练参数。待训练参数可以为如上所述的神经网络的初始的模型参数，或者可以为经上一轮训练之后得到的更新后的模型参数。待训练参数可以包括神经网络的与网络结构关联的参数，例如权重参数等。待训练参数可以包括神经网络的用于迭代的参数，例如梯度参数等。可理解，本公开的实施例中，第二设备120向多个第一设备110发送待训练参数，例如结合图1所描述的N1个第一设备110或者N2个第一设备110。为了简化描述，本公开实施例以一个第一设备110进行阐述，针对其余第一设备110所执行的操作是类似的，本文中不再重复。
相应地，第一设备110可以接收来自第二设备120的待训练参数。第一设备110可以基于待训练参数得到920分布式训练模型的参数。具体而言，第一设备110可以在待训练参数的基础上，使用本地数据集对网络模型进行训练，从而得到本轮训练后的参数，即分布式训练模型的参数。
第一设备110基于确定930参数划分方式、量化方式和量化比特数。具体而言，第一设备110可以基于以下因素中的一项或多项来确定参数划分方式、量化方式和量化比特数：第一设备110至第二设备120的传输资源、分布式神经网络的网络层数、以及训练的损失函数。
参照在图6中第二设备110确定参数划分方式的过程610，第一设备110可以类似地确定参数划分方式，为了简洁，这里不再赘述。例如，参数划分方式为基于类型的划分方式、基于矩阵的划分方式、或基于网络层的划分方式等。
相应地，第一设备110可以使用所确定的参数划分方式，将分布式训练模型的参数划分为多个组。
参照在图6中第二设备110确定量化方式的过程610，第一设备110可以类似地确定量化方式，为了简洁，这里不再赘述。在一些实施例中，第一设备110可以确定量化方式，并 且该量化方式被用于多个组。在另一些实施例中，第一设备110可以针对多个组分别确定量化方式。例如，量化方式为传统量化方式或者符号量化方式。
相应地，第一设备110可以使用所确定的量化方式，确定多个组中每个组的量化边界值。
在一些示例中，第一设备110可以确定量化比特数，并将该量化比特数应用于多个组，换句话说，该量化比特数可以为粗粒度量化比特数。在另一些示例中，第一设备110可以针对每个组分别确定量化比特数，换句话说，该量化比特数可以为细粒度量化比特数。
相应地，第一设备110可以使用所确定的量化比特数，确定多个组中每个组的多个量化参数。
第一设备110基于分布式训练模型的参数确定940多个量化组。示例性地，每个量化组可以包括量化边界值以及多个量化参数。该过程可以参照前述图6中的过程660，为了简洁，这里不再赘述。
可选地，第一设备110可以进行熵编码950。示例性地，第一设备110可以基于多个量化组的压缩率等来确定是否进行熵编码。
第一设备110向第二设备120发送962量化指示消息。该量化指示消息可以包括第一指示信息和第二指示信息。可选地，量化指示消息可以包括第三指示信息。
第一指示信息可以指示第一设备110所确定的参数划分方式，例如基于类型的划分方式、基于矩阵的划分方式、或基于网络层的划分方式等。
第二指示信息可以指示第一设备110所确定的量化方式。在一些示例中，多个组的量化方式相同。第二指示信息可以通过1比特来指示，例如“0”表示传统量化方式，“1”表示符号量化方式。或者，第二指示信息可以通过2比特来指示，其中一个比特表示每个组使用相同的量化方式，另一个比特表示量化方式。在另一些示例中，多个组的量化方式不同。假设使用所确定的参数划分方式将参数划分为m3个组。第二指示信息可以通过m3+1比特来指示，例如其中的一个比特(如第一个比特)用于表示是否每个组使用相同的量化方式，另外的m3个比特分别表示m3个组的量化方式。或者，第二指示信息可以通过m3比特来指示，分别表示m3个组的量化方式。
可选地，在一些示例中，量化指示消息可以不包括第二指示信息。例如如果第一设备110确定使用符号量化方式(各个组使用相同的量化方式)，并且量化边界值只包括绝对值的最大值。那么即使量化指示消息可以不包括第二指示信息，第二设备120也能够确定该量化方式。例如第二设备120能够计算出所有量化组的量化参数的总长度，进而推算出量化边界值的数量。如果该数量等于参数组，说明量化边界值包括绝对值的最大值，则可以确定量化方式为符号量化方式。如果该数量等于参数组的两倍，说明量化边界值包括最大值和最小值，则可以确定量化方式为传统量化方式。
第三指示信息可以指示第一设备110所确定的量化比特数。在一些示例中，多个组的量化比特数相同。第三指示信息可以通过m2比特来指示，例如粗粒度的量化比特数小于2 m2。在另一些示例中，多个组的量化比特数不同，第三指示信息可以指示量化粒度和量化列表，并且第三指示信息还指示多个组的量化比特数的索引。以便第二设备120能够基于索引和量化列表确定各个组的量化比特数。
关于量化粒度和量化列表可以参照前述图6的实施例中的相关描述，例如量化粒度指示是否多个组使用相同的量化比特数，例如量化列表包括(b 0,b 1,b 2,…,b K)。
关于索引可以参照前述图6的实施例中的相关描述，可理解，不同组对应的索引可以是 相同的。例如多组中存在两个组量化比特数相等。
可选地，在一些实施例中，量化指示消息可以不包括第三指示信息，此时第二设备120可以基于多个量化组确定量化比特数。举例而言，各个量化组包括量化边界值和多个量化参数。例如在量化边界值所占用的开销(比特数)小于多个量化参数的数量的情况下，假设多个量化组所占用的总比特数为p1个，多个量化参数的数量为n3个，那么可以通过
Figure PCTCN2022141926-appb-000002
来确定量化比特数，其中
Figure PCTCN2022141926-appb-000003
表示下取整。以此方式，第二设备120能够确定量化比特数，从而无需通过第三指示信息进行指示，这样能够节省信令开销。
可选地，在一些实施例中，量化指示消息可以不包括第三指示信息。例如第二设备120能够计算出所有量化组的量化参数的总长度，进而推算出量化边界值的数量。如果该数量等于参数组，说明量化边界值包括绝对值的最大值，则可以确定量化方式为符号量化方式。如果该数量等于参数组的两倍，说明量化边界值包括最大值和最小值，则可以确定量化方式为传统量化方式。第二设备120能够进一步确定量化后的参数长度，将其除以参数的数量就可以得到各参数组共同的量化比特数。
可选地，如果针对当前轮次的量化指示消息与上一轮的量化指示消息相同，那么第一设备110可以确定不发送当前轮次的量化指示消息。对于第二设备120而言，如果没有收到与当前轮次的多个量化组对应的量化指示消息，则可以使用上一轮的量化指示消息来对本轮次的多个量化组进行反量化。这样能够避免相同内容的重复传输，减小信令开销。
可选地，第一设备110还可以向第二设备120发送964熵编码指示信息。举例而言，第一设备110可以通过该熵编码指示信息告知第二设备120是否进行了熵编码，例如可以通过1比特进行指示。从而，第二设备120可以基于此对接收到的数据成功地进行解压缩。
可选地，量化指示消息与多个量化组可以被承载在同一信令中，或者，量化指示消息与多个量化组可以被承载在不同的信令中。可选地，熵编码指示信息与多个量化组可以被承载在同一信令中，或者，熵编码指示信息与多个量化组可以被承载在不同的信令中。本公开对此不限定。
在一些示例中，假设量化指示消息和熵编码指示信息被承载于同一信令中，例如量化指示消息包括第一指示信息、第二指示信息、第三指示信息和熵编码指示信息。图10示出了根据本公开的一些实施例的量化指示消息1000的示意图。如图所示，量化指示消息1000包括第一指示信息1010、第二指示信息1020和第三指示信息1030，可选地还可以包括熵编码指示信息1040。第三指示信息1030可以指示量化比特数1032，或者，第三指示信息1030可以指示量化粒度1034以及量化列表1036。
应注意，上述结合图10的描述仅是示意，不应解释为对本公开的实施例的限定。例如，如上结合过程962所述，量化指示消息可以不包括第三指示信息。再例如，在一些实施例中，量化指示消息可以不包括第一指示信息。举例而言，第一设备110和第二设备120被预置了划分方式的确定方法，那么第一设备110和第二设备120可以基于预置的确定方法来确定划分方式，如此无需将第一指示信息包括在量化指示消息中，这样能够减小信令开销。可选地，预置的确定方法例如包括：假设按照基于类型的划分方式可以将参数划分为r1组，假设按照基于矩阵的划分方式可以将参数划分为r2组，那么基于r2-r1与预先确定的值(如s和d)之间的关系来确定划分方式，如上结合过程610的相关描述。
第一设备110向第二设备120发送970多个量化组。在一些示例中，多个量化组可以是经熵编码后的多个量化组。相应地，第二设备120可以接收来自第一设备110的多个量化组。
可选地，在一些示例中，量化指示消息中的第三指示信息指示量化粒度和量化列表，那么还可以向第二设备120发送对应的索引。示例性地，该索引可以表示对应的量化比特数在如图7所示的量化比特数列表中的位置。
在一些示例中，第一设备110可以基于多个量化组确定压缩数据，并向第二设备120发送压缩数据。图11示出了根据本公开的一些实施例的压缩数据1100的示意图。压缩数据1100包括量化边界值数据域1110和量化参数数据域1120，可选地还可以包括索引的列表1130。量化边界值数据域1110可以包括多个量化组中的量化边界值。作为一例，可以包括(min1，max1)，(min2，max2)，…。作为另一例，可以包括(min1，min2，…)和(max1，max2，…)。作为再一例，可以包括(max1，max2，…)。量化参数数据域1120可以包括多个量化组中的量化参数，或者可以包括经熵编码的量化参数。索引的列表1130可以包括各个组的量化比特数对应的索引。可选地，如图11所示，索引的列表1130可以独立于量化边界值数据域1110和量化参数数据域1120。在另一些示例中，索引的列表可以被包括在量化边界值数据域1110内，例如可以包括(min1，max1，index1)，(min2，max2，index2)，…，或者例如包括(min1，min2，…)、(max1，max2，…)和(index1，index2，…)，其中index1表示量化边界值为min1和max1的组所使用的量化比特数的索引，index2表示量化边界值为min2和max2的组所使用的量化比特数的索引。
第二设备120对多个量化组进行反量化980。具体而言，第二设备120可以基于来自第一设备110的量化指示信息，对多个量化组进行反量化。关于反量化的描述可以参照上述结合图2部分的相应实施例，为了简洁，这里不再重复。
本公开的实施例中，第一设备110向第二设备120发送多个量化比特的传输资源可以是第二设备120进行分配的，或者可以是第一设备110通过资源请求后由第二设备120分配的。例如在本公开的上面实施例所述。
根据如上结合图9至图11所描述的实施例，第一设备110能够对分布式训练模型的参数进行量化，从而实现了对参数的压缩。该过程既考虑了诸如传输资源等因素，又考虑了参数的取值大小等因素，从而能够确保对参数的压缩既能保证通信效率，又能保证参数的性能。一方面避免过度压缩导致的性能损失，另一方面又避免了因压缩率低导致的时延过大。
应理解，在本公开的实施例中，“第一”，“第二”，“第三”等只是为了表示多个对象可能是不同的，但是同时不排除两个对象之间是相同的。“第一”，“第二”，“第三”等不应当解释为对本公开实施例的任何限制。
还应理解，本公开的实施例中的方式、情况、类别以及实施例的划分仅是为了描述的方便，不应构成特别的限定，各种方式、类别、情况以及实施例中的特征在符合逻辑的情况下，可以相互结合。
还应理解，上述内容只是为了帮助本领域技术人员更好地理解本公开的实施例，而不是要限制本公开的实施例的范围。本领域技术人员根据上述内容，可以进行各种修改或变化或组合等。这样的修改、变化或组合后的方案也在本公开的实施例的范围内。
还应理解，上述内容的描述着重于强调各个实施例之前的不同之处，相同或相似之处可以互相参考或借鉴，为了简洁，这里不再赘述。
图12示出了根据本公开的一些实施例的通信装置1200的一个示意框图。装置1200可以被实现为如图1所示的第一设备110或者被实现为第一设备110的一部分(如芯片)等，本公开对此不限定。
如图12所示，装置1200可以包括划分模块1210、确定模块1220和发送模块1230，可选地，还包括接收模块1240和/或熵编码模块1250。划分模块1210被配置为将分布式训练模型的参数划分为多个组，多个组中的至少两个组的数值区间不同。确定模块1220被配置为基于与多个组中每个组对应的量化比特数，确定与多个组对应的多个量化组，量化比特数是基于以下至少一项而被确定的：通信装置与第二设备之间的传输资源量、损失函数值、或每个组的参数的取值范围。发送模块1230被配置为向第二设备发送多个量化组。
确定模块1220可以被配置为基于多个组中的第一组中的多个第一参数，确定与第一组对应的量化边界值。量化边界值包括最大值和/或最小值，最大值为多个第一参数中的最大值或者最大值为多个第一参数的多个绝对值的最大值，最小值为多个第一参数中的最小值或者最小值为多个第一参数的多个绝对值的最小值。
在一些实施例中，与第一组对应的量化组包括量化边界值以及与多个第一参数对应的多个量化参数。
可选地，接收模块1240可以被配置为接收来自第二设备的第一指示信息，第一指示信息指示是否按照绝对值来确定量化边界值。并且确定模块1220可以被配置为基于第一指示信息，确定量化边界值。或者可选地，发送模块1230可以被配置为向第二设备发送第二指示信息，第二指示信息指示量化边界值是否按照绝对值而被确定。
在一些实施例中，每个组的量化比特数都是相等的。在一些实施例中，多个组包括第一组和第二组，并且与第一组对应的量化比特数不等于与第二组对应的量化比特数。
在一些示例中，确定模块1220可以被配置为确定与多个组对应的多个量化比特数。在一些示例中，确定模块1220可以被配置为基于来自第二设备的量化比特数配置，确定与多个组对应的多个量化比特数，量化比特数配置包括量化阈值和对应的量化比特数。相应地，接收模块1240可以被配置为接收来自第二设备的量化比特数配置。
可选地，发送模块1230被配置为向第二设备发送第三指示信息，第三指示信息指示多个量化比特数。或者可选地，量化比特数是由第二设备确定的，接收模块1240被配置为接收来自第二设备的第四指示信息，第四指示信息指示与多个组中每个组对应的量化比特数。
划分模块1210可以被配置为使用模型参数划分方式，将分布式训练模型的参数划分为多个组。在一些示例中，确定模块1220被配置为基于分布式训练模型的参数的取值大小，确定模型参数划分方式。发送模块1230可以被配置为向第二设备发送第五指示信息，第五指示信息指示模型参数划分方式。在另一些示例中，接收模块1240被配置为接收来自第二设备的第六指示信息，第六指示信息指示模型参数划分方式。示例性地，模型参数划分方式指示基于如下至少一项进行划分：参数的类型、参数所在的网络层的类型等。
熵编码模块1250被配置为对多个量化组进行熵编码；并且发送模块被配置为向第二设备发送经熵编码的多个量化组。可选地，发送模块1230被配置为向第二设备发送第一熵编码指示信息，第一熵编码指示信息多个量化组已被熵编码。或者可选地，接收模块被配置为接收来自第二设备的第二熵编码指示信息；并且熵编码模块被配置为基于第二熵编码指示信息，对多个量化组进行熵编码。
可理解，本公开的实施例中对模块或单元的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时也可以有另外的划分方式，另外，在公开的实施例中的各功能单元可以集成在一个单元中，也可以是单独物理存在，也可以两个或两个以上单元集成为一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
图12中的装置1200能够用于实现上述实施例中由第一设备110所述的各个过程，为了简洁，这里不再赘述。
图13示出了根据本公开的一些实施例的通信装置1300的一个示意框图。装置1300可以被实现为如图1所示的第二设备120或者被实现为第二设备120的一部分(如芯片)等，本公开对此不限定。
如图13所示，装置1300可以包括接收模块1310、第一确定模块1320和第二确定模块1330，可选地还包括发送模块1340。接收模块1310被配置为接收来自第一设备的多个量化组。第一确定模块1320被配置为基于与多个量化组中每个量化组对应的量化比特数，确定与多个量化组对应的多个组，量化比特数是基于以下至少一项而被确定的：第一设备与通信装置之间的传输资源量、损失函数值、或每个组的参数的取值范围。第二确定模块1330被配置为基于模型参数划分方式和多个组，确定分布式训练模型的参数。
在一些实施例中，多个量化组中第一量化组包括量化边界值以及多个量化参数。量化边界值包括最大值和/或最小值，最大值为与第一量化组对应的第一组中多个第一参数中的最大值或者最大值为多个第一参数的多个绝对值的最大值，最小值为多个第一参数中的最小值或者最小值为多个第一参数的多个绝对值的最小值。
可选地，发送模块1340被配置为向第一设备发送第一指示信息，第一指示信息指示是否按照绝对值来确定量化边界值。可选地，接收模块1310被配置为接收来自第一设备的第二指示信息，第二指示信息指示量化边界值是否按照绝对值而被确定。
在一些实施例中，每个量化组的量化比特数都是相等的。在一些实施例中，多个量化组包括第一量化组和第二量化组，并且与第一量化组对应的量化比特数不等于与第二量化组对应的量化比特数。
在一些示例中，接收模块1310被配置为接收来自第一设备的第三指示信息，第三指示信息指示多个量化比特数。在一些示例中，发送模块1340被配置为向第一设备发送量化比特数配置，量化比特数配置包括量化阈值和对应的量化比特数。
在一些实施例中，发送模块1340被配置为向第一设备发送第四指示信息，第四指示信息指示与多个组中每个组对应的量化比特数。
可选地，接收模块1310被配置为接收来自第一设备的第五指示信息，第五指示信息指示模型参数划分方式。或者可选地，发送模块1340被配置为向第一设备发送第六指示信息，第六指示信息指示模型参数划分方式。模型参数划分方式指示基于如下至少一项进行划分：参数的类型、参数所在的网络层的类型等。
在一些实施例中，接收模块1310被配置为接收经熵编码的多个量化组。可选地，接收模块1310被配置为接收来自第一设备的第一熵编码指示信息，第一熵编码指示信息多个量化组已被熵编码。或者可选地，发送模块1340被配置为向第一设备发送第二熵编码指示信息，以指示第一设备对多个量化组进行熵编码。
可理解，本公开的实施例中对模块或单元的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时也可以有另外的划分方式，另外，在公开的实施例中的各功能单元可以集成在一个单元中，也可以是单独物理存在，也可以两个或两个以上单元集成为一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
图13中的装置1300能够用于实现上述实施例中由第二设备120所述的各个过程，为了简洁，这里不再赘述。
图14示出了可以用来实施本公开的实施例的示例设备1400的示意性框图。设备1400可以被实现为或者被包括在图1的第一设备110中，或者设备1400可以被实现为或者被包括在图1的第二设备120中。如图所示，设备1400包括一个或多个处理器1410，耦合到处理器1410的一个或多个存储器1420，以及耦合到处理器1410的通信模块1440。
通信模块1440可以用于双向通信。通信模块1440可以具有用于通信的至少一个通信接口。通信接口可以包括与其他设备通信所必需的任何接口。
处理器1410可以是适合于本地技术网络的任何类型，并且可以包括但不限于以下至少一种：通用计算机、专用计算机、微控制器、数字信号处理器(Digital Signal Processor，DSP)、或基于控制器的多核控制器架构中的一个或多个。设备1400可以具有多个处理器，例如专用集成电路芯片，其在时间上从属于与主处理器同步的时钟。
存储器1420可以包括一个或多个非易失性存储器和一个或多个易失性存储器。非易失性存储器的示例包括但不限于以下至少一种：只读存储器(Read-Only Memory，ROM)1424、可擦除可编程只读存储器(Erasable Programmable Read Only Memory，EPROM)、闪存、硬盘、光盘(Compact Disc，CD)、数字视频盘(Digital Versatile Disc，DVD)或其他磁存储和/或光存储。易失性存储器的示例包括但不限于以下至少一种：随机存取存储器(Random Access Memory，RAM)1422、或不会在断电持续时间中持续的其他易失性存储器。可选地，存储器1420还可以和处理器1410集成在一起。
计算机程序1430包括由关联处理器1410执行的计算机可执行指令。程序1430可以存储在ROM 1424中。处理器1410可以通过将程序1430加载到RAM 1422中来执行任何合适的动作和处理。
可以借助于程序1430来实现本公开的实施例，使得设备1400可以执行如参考图2至图11所讨论的任何过程。本公开的实施例还可以通过硬件或通过软件和硬件的组合来实现。
程序1430可以有形地包含在计算机可读介质中，该计算机可读介质可以包括在设备1400中(诸如在存储器1420中)或者可以由设备1400访问的其他存储设备。可以将程序1430从计算机可读介质加载到RAM 1422以供执行。计算机可读介质可以包括任何类型的有形非易失性存储器，例如ROM、EPROM、闪存、硬盘、CD、DVD等。
在一些实施例中，设备1400中的通信模块1440可以被实现为发送器和接收器(或收发器)，其可以被配置为发送/接收诸如压缩配置消息、待训练参数、多个量化组等。另外，设备1400还可以进一步包括调度器、控制器、射频/天线中的一个或多个，本公开不再详细阐述。
示例性地，图14中的设备1400可以被实现为通信装置，或者可以被实现为通信装置中的芯片或芯片系统，本公开的实施例对此不限定。
本公开的实施例还提供了一种芯片，该芯片可以包括输入接口、输出接口和处理电路。在本公开的实施例中，可以由输入接口和输出接口完成信令或数据的交互，由处理电路完成信令或数据信息的生成以及处理。
本公开的实施例还提供了一种芯片系统，包括处理器，用于支持设备以实现上述任一实施例中所涉及的功能。在一种可能的设计中，芯片系统还可以包括存储器，用于存储必要的程序指令和数据，当处理器运行该程序指令时，使得安装该芯片系统的设备实现上述任一实施例中所涉及的方法。示例性地，该芯片系统可以由一个或多个芯片构成，也可以包含芯片和其他分立器件。可选的，存储器和处理器可以集成在一起。
本公开的实施例还提供了一种处理器，用于与存储器耦合，存储器存储有指令，当处理器运行所述指令时，使得处理器执行上述任一实施例中涉及的方法和功能。
本公开的实施例还提供了一种包含指令的计算机程序产品，其在计算机上运行时，使得计算机执行上述各实施例中任一实施例中涉及的方法和功能。
本公开的实施例还提供了一种计算机可读存储介质，其上存储有计算机指令，当处理器运行所述指令时，使得处理器执行上述任一实施例中涉及的方法和功能。
通常，本公开的各种实施例可以以硬件或专用电路、软件、逻辑或其任何组合来实现。一些方面可以用硬件实现，而其他方面可以用固件或软件实现，其可以由控制器，微处理器或其他设备执行。虽然本公开的实施例的各个方面被示出并描述为框图，流程图或使用一些其他图示表示，但是应当理解，本文描述的框，装置、系统、技术或方法可以实现为，如非限制性示例，硬件、软件、固件、专用电路或逻辑、通用硬件或控制器或其他设备，或其某种组合。
本公开还提供有形地存储在非暂时性计算机可读存储介质上的至少一个计算机程序产品。该计算机程序产品包括计算机可执行指令，例如包括在程序模块中的指令，其在目标的真实或虚拟处理器上的设备中执行，以执行如上参考附图的过程/方法。通常，程序模块包括执行特定任务或实现特定抽象数据类型的例程、程序、库、对象、类、组件、数据结构等。在各种实施例中，可以根据需要在程序模块之间组合或分割程序模块的功能。用于程序模块的机器可执行指令可以在本地或分布式设备内执行。在分布式设备中，程序模块可以位于本地和远程存储介质中。
用于实现本公开的方法的计算机程序代码可以用一种或多种编程语言编写。这些计算机程序代码可以提供给通用计算机、专用计算机或其他可编程的数据处理装置的处理器，使得程序代码在被计算机或其他可编程的数据处理装置执行的时候，引起在流程图和/或框图中规定的功能/操作被实施。程序代码可以完全在计算机上、部分在计算机上、作为独立的软件包、部分在计算机上且部分在远程计算机上或完全在远程计算机或服务器上执行。
在本公开的上下文中，计算机程序代码或者相关数据可以由任意适当载体承载，以使得设备、装置或者处理器能够执行上文描述的各种处理和操作。载体的示例包括信号、计算机可读介质、等等。信号的示例可以包括电、光、无线电、声音或其它形式的传播信号，诸如载波、红外信号等。
计算机可读介质可以是包含或存储用于或有关于指令执行系统、装置或设备的程序的任何有形介质。计算机可读介质可以是计算机可读信号介质或计算机可读存储介质。计算机可读介质可以包括但不限于电子的、磁的、光学的、电磁的、红外的或半导体系统、装置或设备，或其任意合适的组合。计算机可读存储介质的更详细示例包括带有一根或多根导线的电气连接、便携式计算机磁盘、硬盘、随机存储存取器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或闪存)、光存储设备、磁存储设备，或其任意合适的组合。
此外，尽管在附图中以特定顺序描述了本公开的方法的操作，但是这并非要求或者暗示必须按照该特定顺序来执行这些操作，或是必须执行全部所示的操作才能实现期望的结果。相反，流程图中描绘的步骤可以改变执行顺序。附加地或备选地，可以省略某些步骤，将多个步骤组合为一个步骤执行，和/或将一个步骤分解为多个步骤执行。还应当注意，根据本公开的两个或更多装置的特征和功能可以在一个装置中具体化。反之，上文描述的一个装置的特征和功能可以进一步划分为由多个装置来具体化。
以上已经描述了本公开的各实现，上述说明是示例性的，并非穷尽的，并且也不限于所公开的各实现。在不偏离所说明的各实现的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在很好地解释各实现的原理、实际应用或对市场中的技术的改进，或者使本技术领域的其他普通技术人员能理解本文公开的各个实现方式。

Claims (40)

  1. 一种数据处理方法，包括：
    第一设备将分布式训练模型的参数划分为多个组，所述多个组中的至少两个组的数值区间不同；
    所述第一设备基于与所述多个组中每个组对应的量化比特数，确定与所述多个组对应的多个量化组，所述量化比特数是基于以下至少一项而被确定的：所述第一设备与第二设备之间的传输资源量、损失函数值、或所述每个组的参数的取值范围；以及
    所述第一设备向所述第二设备发送所述多个量化组。
  2. 根据权利要求1所述的方法，还包括：所述第一设备基于所述多个组中的第一组中的多个第一参数，确定与所述第一组对应的量化边界值。
  3. 根据权利要求2所述的方法，其中所述量化边界值包括以下至少一项：最大值、或最小值，
    所述最大值为所述多个第一参数中的最大值或者所述最大值为所述多个第一参数的多个绝对值的最大值，所述最小值为所述多个第一参数中的最小值或者所述最小值为所述多个第一参数的多个绝对值的最小值。
  4. 根据权利要求2或3所述的方法，其中与所述第一组对应的量化组包括所述量化边界值以及与所述多个第一参数对应的多个量化参数。
  5. 根据权利要求2至4中任一项所述的方法，还包括：
    所述第一设备接收来自所述第二设备的第一指示信息，所述第一指示信息指示是否按照绝对值来确定所述量化边界值；
    并且其中所述第一设备确定与所述第一组对应的量化边界值包括：所述第一设备基于所述第一指示信息，确定所述量化边界值。
  6. 根据权利要求2至4中任一项所述的方法，还包括：
    所述第一设备向所述第二设备发送第二指示信息，所述第二指示信息指示所述量化边界值是否按照绝对值而被确定。
  7. 根据权利要求1至6中任一项所述的方法，其中所述每个组的量化比特数都是相等的。
  8. 根据权利要求1至6中任一项所述的方法，其中所述多个组包括第一组和第二组，并且与所述第一组对应的量化比特数不等于与所述第二组对应的量化比特数。
  9. 根据权利要求8所述的方法，所述方法还包括：
    所述第一设备确定与所述多个组对应的多个量化比特数。
  10. 根据权利要求9所述的方法，其中所述第一设备确定与所述多个组对应的多个量化比特数包括：
    所述第一设备基于来自所述第二设备的量化比特数配置，确定与所述多个组对应的所述多个量化比特数，所述量化比特数配置包括量化阈值和对应的量化比特数。
  11. 根据权利要求10所述的方法，还包括：
    所述第一设备接收来自所述第二设备的所述量化比特数配置。
  12. 根据权利要求9至11中任一项所述的方法，还包括：
    所述第一设备向所述第二设备发送第三指示信息，所述第三指示信息指示所述多个量化比特数。
  13. 根据权利要求1至8中任一项所述的方法，其中所述量化比特数是由所述第二设备确定的，所述方法还包括：
    所述第一设备接收来自所述第二设备的第四指示信息，所述第四指示信息指示与所述多个组中每个组对应的量化比特数。
  14. 根据权利要求1至11中任一项所述的方法，其中所述第一设备将分布式训练模型的参数划分为多个组包括：
    所述第一设备使用模型参数划分方式，将所述分布式训练模型的参数划分为所述多个组。
  15. 根据权利要求14所述的方法，还包括：
    所述第一设备基于所述分布式训练模型的参数的取值大小，确定所述模型参数划分方式；以及
    所述第一设备向所述第二设备发送第五指示信息，所述第五指示信息指示所述模型参数划分方式。
  16. 根据权利要求14所述的方法，还包括：
    所述第一设备接收来自所述第二设备的第六指示信息，所述第六指示信息指示所述模型参数划分方式。
  17. 根据权利要求14至16中任一项所述的方法，其中所述模型参数划分方式指示基于如下至少一项进行划分：参数的类型、参数所在的网络层的类型等。
  18. 根据权利要求1至17中任一项所述的方法，还包括：
    所述第一设备对所述多个量化组进行熵编码；
    并且其中所述第一设备向所述第二设备发送所述多个量化组包括：
    所述第一设备向所述第二设备发送经熵编码的所述多个量化组。
  19. 根据权利要求18所述的方法，还包括：
    所述第一设备向所述第二设备发送第一熵编码指示信息，所述第一熵编码指示信息所述多个量化组已被熵编码。
  20. 根据权利要求18所述的方法，还包括：
    所述第一设备接收来自所述第二设备的第二熵编码指示信息；
    并且其中所述第一设备对所述多个量化组进行熵编码包括：所述第一设备基于所述第二熵编码指示信息，对所述多个量化组进行熵编码。
  21. 一种数据处理方法，包括：
    第二设备接收来自第一设备的多个量化组；
    所述第二设备基于与所述多个量化组中每个量化组对应的量化比特数，确定与所述多个量化组对应的多个组，所述量化比特数是基于以下至少一项而被确定的：所述第一设备与所述第二设备之间的传输资源量、损失函数值、或所述每个组的参数的取值范围；以及
    所述第二设备基于模型参数划分方式和所述多个组，确定分布式训练模型的参数。
  22. 根据权利要求21所述的方法，其中所述多个量化组中第一量化组包括量化边界值以及多个量化参数。
  23. 根据权利要求22所述的方法，其中量化边界值包括以下至少一项：最大值、或最小值，
    所述最大值为与所述第一量化组对应的第一组中多个第一参数中的最大值或者所述最大值为所述多个第一参数的多个绝对值的最大值，所述最小值为所述多个第一参数中的最小值 或者所述最小值为所述多个第一参数的多个绝对值的最小值。
  24. 根据权利要求22或23所述的方法，还包括：
    所述第二设备向所述第一设备发送第一指示信息，所述第一指示信息指示是否按照绝对值来确定所述量化边界值；
    或者，
    所述第二设备接收来自所述第一设备的第二指示信息，所述第二指示信息指示所述量化边界值是否按照绝对值而被确定。
  25. 根据权利要求21至24中任一项所述的方法，其中所述每个量化组的量化比特数都是相等的。
  26. 根据权利要求21至24中任一项所述的方法，其中所述多个量化组包括第一量化组和第二量化组，并且与所述第一量化组对应的量化比特数不等于与所述第二量化组对应的量化比特数。
  27. 根据权利要求26所述的方法，还包括：
    所述第二设备接收来自所述第一设备的第三指示信息，所述第三指示信息指示所述多个量化比特数。
  28. 根据权利要求21至26中任一项所述的方法，还包括：
    所述第二设备向所述第一设备发送量化比特数配置，所述量化比特数配置包括量化阈值和对应的量化比特数。
  29. 根据权利要求21至26中任一项所述的方法，还包括：
    所述第二设备向所述第一设备发送第四指示信息，所述第四指示信息指示与所述多个组中每个组对应的量化比特数。
  30. 根据权利要求21至29中任一项所述的方法，还包括：
    所述第二设备接收来自所述第一设备的第五指示信息，所述第五指示信息指示所述模型参数划分方式；
    或者，
    所述第二设备向所述第一设备发送第六指示信息，所述第六指示信息指示所述模型参数划分方式。
  31. 根据权利要求21至30中任一项所述的方法，其中所述模型参数划分方式指示基于如下至少一项进行划分：参数的类型、参数所在的网络层的类型等。
  32. 根据权利要求21至31中任一项所述的方法，其中所述第二设备接收来自第一设备的多个量化组包括：所述第二设备接收经熵编码的所述多个量化组。
  33. 根据权利要求32所述的方法，还包括：
    所述第二设备接收来自所述第一设备的第一熵编码指示信息，所述第一熵编码指示信息所述多个量化组已被熵编码；
    或者，
    所述第二设备向所述第一设备发送第二熵编码指示信息，以指示所述第一设备对所述多个量化组进行熵编码。
  34. 一种通信装置，包括：
    划分模块，被配置为将分布式训练模型的参数划分为多个组，所述多个组中的至少两个组的数值区间不同；
    确定模块，被配置为基于与所述多个组中每个组对应的量化比特数，确定与所述多个组对应的多个量化组，所述量化比特数是基于以下至少一项而被确定的：所述通信装置与第二设备之间的传输资源量、损失函数值、或所述每个组的参数的取值范围；以及
    发送模块，被配置为向所述第二设备发送所述多个量化组。
  35. 一种通信装置，包括：
    接收模块，被配置为接收来自第一设备的多个量化组；
    第一确定模块，被配置为基于与所述多个量化组中每个量化组对应的量化比特数，确定与所述多个量化组对应的多个组，所述量化比特数是基于以下至少一项而被确定的：所述第一设备与所述通信装置之间的传输资源量、损失函数值、或所述每个组的参数的取值范围；以及
    第二确定模块，被配置为基于模型参数划分方式和所述多个组，确定分布式训练模型的参数。
  36. 一种通信设备，包括存储器和处理器，所述存储器存储有计算机指令，当所述计算机指令被所述处理器执行时，使得所述通信设备实现根据权利要求1至33中任一项所述的方法。
  37. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机可执行指令，所述计算机可执行指令被处理器执行时实现根据权利要求1至33中任一项所述的方法。
  38. 一种芯片，包括处理电路，被配置为执行根据权利要求1至33中任一项所述的方法。
  39. 一种计算机程序产品，所述计算机程序产品上包含计算机可执行指令，所述计算机可执行指令在被执行时实现根据权利要求1至33中任一项所述的方法。
  40. 一种通信系统，包括第一设备和第二设备，其中所述第一设备用于实现根据权利要求1至20中任一项所述的方法，所述第二设备用于实现根据权利要求21至33中任一项所述的方法。
PCT/CN2022/141926 2022-01-29 2022-12-26 信息处理方法及装置 WO2023142828A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210112944.2 2022-01-29
CN202210112944.2A CN116579438A (zh) 2022-01-29 2022-01-29 信息处理方法及装置

Publications (1)

Publication Number Publication Date
WO2023142828A1 true WO2023142828A1 (zh) 2023-08-03

Family

ID=87470407

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/141926 WO2023142828A1 (zh) 2022-01-29 2022-12-26 信息处理方法及装置

Country Status (2)

Country Link
CN (1) CN116579438A (zh)
WO (1) WO2023142828A1 (zh)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110135465A (zh) * 2019-04-22 2019-08-16 腾讯科技（深圳）有限公司 模型参数表示空间大小估计方法及装置、推荐方法
US20200364603A1 (en) * 2019-05-15 2020-11-19 Google Llc Compression of Machine-Learned Models via Entropy Penalized Weight Reparameterization
CN113139663A (zh) * 2021-04-23 2021-07-20 深圳市大数据研究院 联邦边缘学习配置信息获取方法、装置、设备及介质
CN113222179A (zh) * 2021-03-18 2021-08-06 北京邮电大学 一种基于模型稀疏化与权重量化的联邦学习模型压缩方法
CN113258935A (zh) * 2021-05-25 2021-08-13 山东大学 一种联邦学习中基于模型权值分布的通信压缩方法
WO2021189225A1 (zh) * 2020-03-24 2021-09-30 Oppo广东移动通信有限公司 一种机器学习模型训练方法、电子设备及存储介质
CN114611705A (zh) * 2020-11-23 2022-06-10 华为技术有限公司 数据处理方法、机器学习的训练方法及相关装置、设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110135465A (zh) * 2019-04-22 2019-08-16 腾讯科技（深圳）有限公司 模型参数表示空间大小估计方法及装置、推荐方法
US20200364603A1 (en) * 2019-05-15 2020-11-19 Google Llc Compression of Machine-Learned Models via Entropy Penalized Weight Reparameterization
WO2021189225A1 (zh) * 2020-03-24 2021-09-30 Oppo广东移动通信有限公司 一种机器学习模型训练方法、电子设备及存储介质
CN114611705A (zh) * 2020-11-23 2022-06-10 华为技术有限公司 数据处理方法、机器学习的训练方法及相关装置、设备
CN113222179A (zh) * 2021-03-18 2021-08-06 北京邮电大学 一种基于模型稀疏化与权重量化的联邦学习模型压缩方法
CN113139663A (zh) * 2021-04-23 2021-07-20 深圳市大数据研究院 联邦边缘学习配置信息获取方法、装置、设备及介质
CN113258935A (zh) * 2021-05-25 2021-08-13 山东大学 一种联邦学习中基于模型权值分布的通信压缩方法

Also Published As

Publication number Publication date
CN116579438A (zh) 2023-08-11

Similar Documents

Publication Publication Date Title
US11197178B2 (en) Facilitating enhanced beam management in a wireless communication system
WO2021243619A1 (zh) 信息传输方法、装置、通信设备及存储介质
US10314082B2 (en) Facilitating signaling and transmission protocols for enhanced beam management for initial access
WO2021057461A1 (zh) 一种极化码分段编码的方法及通信设备
WO2021258370A1 (zh) 一种通信处理方法、通信处理装置及存储介质
WO2021057564A1 (zh) 信道状态信息的优先级发送、确定方法及装置、存储介质、用户设备
US10826591B1 (en) Irregular grid sub sampling to reduce feedback overhead in a 5G or other next generation wireless network
US11201712B2 (en) Facilitating incremental downlink control information design to support downlink control information scheduling
WO2021189225A1 (zh) 一种机器学习模型训练方法、电子设备及存储介质
WO2022016466A1 (zh) 资源请求信息处理方法及装置、通信设备及存储介质
WO2018133802A1 (zh) 一种资源调度方法和基站及终端
WO2023142828A1 (zh) 信息处理方法及装置
US20200288448A1 (en) Decoding downlink control channels for 5g wireless communication systems
US20170099620A1 (en) Method and apparatus for data transmission
WO2020199552A1 (zh) 一种通信方法及设备
WO2019192442A1 (zh) 确定传输块大小的方法和通信装置
WO2022127702A1 (zh) 信息确定方法、装置及通信设备
WO2019095783A1 (zh) 一种同步块与寻呼调度信令关联方法、指示方法及装置
CN113260032B (zh) 唤醒无线电分组标识符分配
CN108990022B (zh) 一种mtc终端通信方法
WO2020134610A1 (zh) 数据调度方法及装置
WO2018121220A1 (zh) 一种系统信息传输方法、用户终端和传输节点
WO2019052441A1 (zh) 一种用于生成扩展符号的方法及装置
WO2024036526A1 (zh) 一种模型调度方法和装置
WO2023231620A1 (zh) 通信方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22923614

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2022923614

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022923614

Country of ref document: EP

Effective date: 20240813

NENP Non-entry into the national phase

Ref country code: DE