WO2025256563A1 - 注意力模型推理方法、装置及系统 - Google Patents

注意力模型推理方法、装置及系统

Info

Publication number
WO2025256563A1
WO2025256563A1 PCT/CN2025/100461 CN2025100461W WO2025256563A1 WO 2025256563 A1 WO2025256563 A1 WO 2025256563A1 CN 2025100461 W CN2025100461 W CN 2025100461W WO 2025256563 A1 WO2025256563 A1 WO 2025256563A1
Authority
WO
WIPO (PCT)
Prior art keywords
attention
self
data
head
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/100461
Other languages
English (en)
French (fr)
Inventor
唐瀚霖
林菁
林扬
韩庆森
洪世宽
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of WO2025256563A1 publication Critical patent/WO2025256563A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models

Definitions

  • This application relates to the field of artificial intelligence, and in particular to an attention model reasoning method, apparatus and system.
  • attention models are becoming a future trend for reasoning over very long sequences. For example, this has evolved from reasoning over text-based dialogues to reasoning over documents.
  • attention models face the following challenges: the key-value cache (KV cache) increases exponentially, even exceeding the amount of data required for the attention model's weights; the computational cost of the attention model grows dramatically, and its reasoning performance cannot meet commercial demands.
  • KV cache key-value cache
  • discarding tokens contained in the sequence being reasoned over reduces the number of tokens used in the attention model, thereby reducing the required storage resources and computational cost.
  • This application provides an attention model inference method, apparatus, and system, which reduces the computational load and storage resources required for attention model inference while ensuring the accuracy of attention model inference, thereby improving the sequence length and inference speed of attention model inference.
  • an attention model inference method comprising: evaluating the importance level of each of the multiple self-attention heads in the attention model; determining the amount of data to be eliminated by each self-attention head based on the importance level of each self-attention head, wherein the importance level is inversely proportional to the elimination amount; and eliminating the data to be inferred by each self-attention head according to the corresponding elimination amount during the inference process of the attention model.
  • different self-attention heads acquire different amounts of inference data during model inference. For example, important self-attention heads eliminate fewer inference data and retain more, while less important self-attention heads eliminate more inference data and retain fewer. This reduces the computational load and storage resources required for attention model inference while ensuring its accuracy. Because of this reduced computational load and storage resources, the attention model can infer longer sequences and improve its inference speed under the same hardware conditions.
  • evaluating the importance level of each of the multiple self-attention heads in the attention model includes: assessing the importance level of each self-attention head based on its attention score.
  • the method further includes: converting offline data into word segments, converting word segments into key-value data, determining the attention score of the word segments based on the key-value data, and determining the attention score of the word segments output by each self-attention head in the attention model as the attention score of each self-attention head.
  • the highest attention score among the multiple attention scores output by the self-attention head can be determined as the attention score of the self-attention head.
  • Attention score represents the correlation between multiple words processed by a self-attention head. Determining the attention score of the words processed by the self-attention head as the attention score of the self-attention head can accurately determine the importance level of the self-attention head.
  • model inference offline data is used to perform model inference to determine the attention score of the self-attention heads in the model. This allows for the evaluation of the importance level of each self-attention head and the corresponding elimination amount among the multiple self-attention heads in the attention model. This eliminates the need to occupy the storage and computing resources of model inference, alleviates bandwidth and computing bottlenecks, and improves model inference performance.
  • Rope Rotary Position Embedding
  • the elimination quantity corresponding to an important self-attention point is 0; when the importance level of a self-attention point is "unimportant”, the elimination quantity corresponding to an unimportant self-attention point is greater than 0.
  • Setting the elimination threshold for important self-attention heads to 0 ensures that all data inferred by these heads is retained during the inference process of the attention model. This preserves the ability to search for global information about the semantic data input by the user, ensuring the accuracy of the attention model's inference. Setting the elimination threshold for unimportant self-attention heads to greater than 0 allows for the elimination of some data inferred by these heads during the inference process, reducing the computational load and storage resources required for the attention model's inference.
  • the positional encoding method of the attention model is Attention with Linear Biases (Alibi) positional encoding; determining the amount of data to be eliminated by each self-attention head based on the importance level of each self-attention head includes: determining the elimination amount corresponding to each self-attention head based on the model weights and the slope of the positional encoding of each self-attention head, where the model weights include query weights and key weights.
  • Alibi Attention with Linear Biases
  • each self-attention head eliminates the inferred data according to a corresponding elimination amount, including: each self-attention head eliminates the inferred data starting from the first data in the inferred data according to the corresponding elimination amount.
  • the data inferred by the self-attention head pair is key-value data that is transformed from semantic data input by the user into word segmentation and then further transformed by word segmentation.
  • an attention model inference apparatus comprising modules for performing the methods of the first aspect or any possible design of the first aspect.
  • the attention model inference apparatus includes a communication module, an evaluation module, and an inference module.
  • a data processing system which includes a memory and a plurality of processors, the memory being used to store a set of computer instructions; when the processors execute the set of computer instructions, the plurality of processors execute the operational steps of the method in the first aspect or any possible implementation of the first aspect.
  • a computer device including a memory and a plurality of processors, the memory being used to store a set of computer instructions; when the processors execute the set of computer instructions, the plurality of processors jointly execute the operational steps of the method as described in the first aspect or any possible implementation thereof.
  • a chip comprising: a processor and a power supply circuit; wherein the power supply circuit is used to supply power to the processor; and the processor is used to execute the operational steps of the method in the first aspect or any possible implementation of the first aspect.
  • a sixth aspect provides a computer-readable storage medium comprising: computer software instructions; which, when executed in a computing device, cause the computing device to perform operational steps of the method as described in the first aspect or any possible implementation thereof.
  • a computer program product that, when run on a computing device, causes the computing device to perform the operational steps of the method as described in the first aspect or any possible implementation thereof.
  • Figure 1 is a schematic diagram of the architecture of a Transformer model provided in this application.
  • Figure 2 is a schematic diagram illustrating the relationship between sequence length, batch size, and memory size provided in this application;
  • FIG. 3 is a schematic diagram of the architecture of a data processing system provided in this application.
  • Figure 4 is a flowchart illustrating an attention model inference method provided in this application.
  • Figure 5 is a schematic diagram of an attention head for long-range correlation characteristics and an attention head for local correlation characteristics provided in this application;
  • Figure 6 is a schematic diagram of an important level evaluation process for self-attention heads provided in this application.
  • Figure 7 is a schematic diagram of a process for calculating the amount of waste provided in this application.
  • FIG. 8 is a schematic diagram of the elimination of inferred data provided in this application.
  • Figure 9 is a schematic diagram of the structure of an attention model inference device provided in this application.
  • Figure 10 is a schematic diagram of the structure of a computer device provided in this application.
  • the Transformer model is a deep learning model based on a self-attention mechanism. Its core idea is to find relevant information by calculating the similarity between each segmentation and other segments in semantic data (e.g., sequences), thereby generating an output matrix containing more contextual information.
  • This model architecture has achieved significant performance improvements in natural language processing, especially in tasks such as machine translation, text summarization, and question answering systems.
  • the Transformer model is an encoder-decoder architecture.
  • the Transformer model contains multiple encoders and multiple decoders.
  • each encoder contains two sub-layers: a multi-head attention layer and a position-wise-feed forward network (FNN).
  • the multi-head attention layer contains multiple self-attention heads.
  • Each encoder has the same structure but can use different weights.
  • Each encoder also contains input word embeddings, positional embeddings, and add and normalization layers.
  • ⁇ add ⁇ represents a residual connection used to prevent network degradation.
  • ⁇ normalization ⁇ represents layer normalization, used to normalize the activation values of each layer.
  • a decoder contains two multi-head attention layers, the first of which is a masked multi-head attention layer; the decoder also contains a fully connected layer and a normalization layer (SoftMax).
  • Input word embedding This refers to the model embedding each word in the semantic data into a high-dimensional vector representation to obtain a word vector. This embedding process allows the model to capture the semantic similarity between word segments.
  • Position encoding layer Obtains the position vector of the word segmentation, which represents the position of the word segmentation in the semantic data.
  • the word segmentation vector and the word segmentation position vector are added together to obtain the word segmentation representation vector of the input multi-head attention layer.
  • the word segmentation representation vector can be a matrix.
  • Self-attention mechanism This allows the model to compare each word with other words in the semantic data while processing the semantic data to determine the relationships between them. Words input to the self-attention head are converted into three representations: query, key, and value. By calculating the similarity between the query and all keys, the model can decide how much attention should be given to each word in the semantic data when generating each output word. The attention scores of the words are then used to weight the corresponding values to generate the output of the self-attention head.
  • the self-attention head is used to calculate query vectors, key vectors, and value vectors using the segmentation representation vectors and linear transformation matrices (query weight WQ , key weight WK , and value weight WV ), respectively. It also calculates the attention score for each pair of segmentations in the semantic data based on the query and key vectors.
  • the query vector represents the query of the segmentation, i.e., what the model is looking for in the semantic data.
  • the key vector represents the key of the segmentation, i.e., what other segmentations in the semantic data should pay attention to.
  • the value vector represents the value of the segmentation, i.e., the information contributed by the segmentation to the output.
  • the attention score is calculated based on the dot product of the query and key vectors to evaluate the relevance between segmentations.
  • the multi-head attention layer inputs the segmentation representation vector into each self-attention head.
  • Each self-attention head calculates a query vector, a key vector, and a value vector.
  • the query vector, key vector, and value vector output by each self-attention head are concatenated to obtain the query vector, key vector, and value vector output by the multi-head attention layer.
  • the output vector of the multi-head attention layer has the same dimension as the input vector.
  • the attention scores are normalized using the softmax function to obtain attention weights. These weights represent the degree to which each word should pay attention to other words in the semantic data. Words with higher attention weights are considered more critical to the task being performed.
  • attention weights are used to calculate a weighted sum of the value vectors. This produces the self-attention mechanism output of word segmentation in semantic data, capturing contextual information from other word segments.
  • Tokenization In the field of Natural Language Processing (NLP), tokenization refers to the process of breaking down a text string into its smallest semantic units (tokens). These tokens can be words, phrases, or other linguistic units. Below are some common tokenization methods.
  • Space-based word segmentation refers to splitting text into words or phrases based on spaces.
  • Delimiter-based tokenization This method can be used if the text uses specific delimiters (such as commas, periods, etc.) to separate different parts.
  • Syntax-based tokenization Using regular expressions or libraries such as NLTK or SpaCy, more complex tokenization can be performed based on syntax rules.
  • Machine learning-based word segmentation This method uses machine learning models to identify and segment words in text.
  • the attention mechanism is a core component that allows the model to assign different weights to different parts of the context when processing input semantic data.
  • the attention score refers to the weight value calculated in this process, reflecting the degree of attention the model pays to a particular part of the input semantic data.
  • long sequence reasoning refers to the ability to process text data whose length far exceeds conventional limits (such as sequences containing thousands or even tens of thousands of words).
  • KV Cache Key-value caching
  • Model parameters refer to the variables or weights that need to be learned or adjusted in an artificial intelligence model. Model parameters can affect the model's predictive ability and performance. During model training, the model optimizes its performance by trying different combinations of parameters. Common model parameters include weights, biases, learning rate, and regularization coefficients. When using the model for prediction, these model parameters are used to calculate the output results.
  • Large models refer to ultra-large-scale artificial intelligence (AI) models. They are widely used in natural language processing (NLP) and are fundamentally changing the landscape of NLP tasks, giving rise to more powerful and intelligent language technologies. Large models are a key direction in AI development. They also excel in various NLP tasks, such as text classification, sentiment analysis, summarization, and translation. Large models can be used in multiple application areas, including automatic writing, chatbots, virtual assistants, voice assistants, and automatic translation. Examples include Large Language Models (LLMs), Bidirectional Encoder Representations from Transformers (BERT), Generative Pre-Trained Transformers (GPT), GPT3, GPT4, and Mixture of Experts (MoE). Large models possess the following characteristics.
  • Multi-task learning Large models can handle a variety of different Natural Language Processing (NLP) tasks, such as machine translation, text summarization, and question answering systems, enabling the model to learn a broader and more generalized language understanding ability.
  • NLP Natural Language Processing
  • Dedicated processors include, but are not limited to, graphics processing units (GPUs), data processing units (DPUs), neural processing units (NPUs), and neural-network processing units (NPUs).
  • a dedicated processor can refer to a processor other than the central processing unit (CPU) for accelerated processing; it can also be called an accelerator card, training processor, or simply an accelerator.
  • the attention model inference method includes: evaluating the importance level of each self-attention head among multiple self-attention heads in the attention model; determining the amount of data to be eliminated by each self-attention head based on its importance level, wherein the importance level is inversely proportional to the elimination amount; and eliminating the data to be inferred by each self-attention head according to its corresponding elimination amount during the inference process of the attention model.
  • different self-attention heads acquire different amounts of inference data during model inference. For example, important self-attention heads eliminate fewer inference data and retain more, while less important self-attention heads eliminate more inference data and retain fewer. This reduces the computational load and storage resources required for attention model inference while ensuring its accuracy. Because of this reduced computational load and storage resources, the attention model can infer longer sequences and improve its inference speed under the same hardware conditions.
  • Long sequence reasoning has become an important direction for major model vendors to enhance the performance of intelligent agents. Long sequence reasoning has the characteristics of richer semantic information, more accurate prediction, and more coherent text generation.
  • the attention model reasoning method provided in this application can be applied to scenarios such as document summarization, language translation, multi-turn dialogue, code analysis, and code generation.
  • Figure 3 is a schematic diagram of the architecture of a data processing system provided in this application.
  • the data processing system 300 includes a client 310, a computing cluster 320, and a storage cluster 330.
  • Storage cluster 330 comprises multiple storage nodes 331.
  • Each storage node 331 includes one or more controllers, a network interface card (NIC), and multiple hard drives.
  • the hard drives are used to store data.
  • Hard drives can be disks or other types of storage media, such as solid-state drives (SSDs) or shingled magnetic recording (SMR) hard drives.
  • the NICs are used to communicate with the compute nodes 321 included in compute cluster 320.
  • the controllers are used to write data to or read data from the hard drives based on read/write data requests sent by the compute nodes 321. During the read/write process, the controller needs to translate the address carried in the read/write data request into an address that the hard drive can recognize.
  • the computing cluster 320 contains multiple computing nodes 321.
  • Each computing node 321 can be a computing device, such as an accelerator card or a server.
  • the computing cluster 320 can be a heterogeneous computing architecture to provide high-performance computing.
  • computing nodes 321 may include computing units with computing capabilities such as CPUs, graphics processing units (GPUs), data processing units (DPUs), neural processing units (NPUs), and embedded neural-network processing units (NPUs) to provide high-performance computing.
  • computing cluster 320 includes multiple accelerator cards, which perform model training.
  • multiple computing nodes 321 are connected via network devices (such as switches, network interface cards, etc.) based on high-speed interconnect technology, enabling communication between the multiple computing nodes 321.
  • network devices such as switches, network interface cards, etc.
  • Client 310 communicates with computing cluster 320 and storage cluster 330 via network 340. For example, client 310 sends a model inference request to computing cluster 320 via network 340, requesting computing cluster 320 to perform model inference on a sequence.
  • Network 340 can refer to an internal enterprise network (such as a local area network, LAN) or the Internet.
  • Client 310 refers to the computer connected to network 340, also known as a workstation. Different clients can share network resources (such as computing resources and storage resources).
  • the computing cluster 320 further includes a control node 322.
  • the control node and computing nodes can be independent physical devices. Alternatively, the control node and multiple computing nodes can reside on the same physical device.
  • the control node can be a CPU. Multiple computing nodes include computing units such as GPUs, NPUs, and DPUs.
  • the control node 322 manages and allocates tasks, allowing multiple computing nodes to execute multiple tasks in parallel to improve data processing speed. For example, the control node 322 instructs multiple computing nodes to perform model inference in parallel based on a model inference request. For instance, one computing node might perform the functions of an encoder and a decoder.
  • the computing node or control node determines the amount of data to be inferred by multiple self-attention heads based on their importance levels within the attention model.
  • the elimination amount for important self-attention heads is greater than that for less important self-attention heads. These elimination amounts are input into the attention model.
  • the data to be inferred by each self-attention head is eliminated according to its elimination amount, and the remaining data is used for model inference. This reduces the computational load and storage resources required for attention model inference while ensuring its accuracy. Because of this reduced computational load and storage resources, longer sequences can be inferred using the attention model under the same hardware conditions, and the inference speed of the attention model is improved.
  • Storage cluster 330 can be used to store model inference-related data, such as the model and model parameters.
  • model inference-related data such as the model and model parameters.
  • the compute node 321 When the compute node 321 performs model inference, it can retrieve the model and model parameters from the storage node 331 and store them on the local storage medium.
  • client 310 has client program 311 installed.
  • Client 310 runs client program 311 and displays a user interface (UI).
  • User 350 manipulates the user interface to submit a request.
  • user 350 manipulates the user interface to submit a request.
  • control node 322 retrieves the model and model parameters from storage cluster 330, and distributes the model parameters to multiple computing nodes, enabling the multiple computing nodes to perform model inference based on the parallel technology described in the above embodiments.
  • system administrator 360 can use the client 310 to call the application platform interface (API) 312 or the command-line interface (CLI) 313 to configure system information, such as the elimination quantity corresponding to the self-attention head configured for the computing node provided in this application.
  • API application platform interface
  • CLI command-line interface
  • Figure 3 is merely a schematic diagram; the embodiments of this application do not limit the device connection method or the number of devices in the data processing system.
  • the data processing system may include multiple clients.
  • One client can connect to multiple computing nodes. Different clients establish connections with different computing nodes.
  • FIG 4 is a flowchart illustrating an attention model inference method provided in this application.
  • the attention model inference is explained using the computation node shown in Figure 3 as an example.
  • the method includes steps 410 to 430.
  • Step 410 Evaluate the importance level of each self-attention head among the multiple self-attention heads in the attention model.
  • the attention model compares each word segment with other words in the semantic data to determine the relationships between them.
  • Other words include at least one of the following: words that are close to the current word or words that are far away. That is, the attention model includes multiple self-attention heads, including self-attention heads for long-range association and self-attention heads for local association. Self-attention heads for long-range association are used to compare the current word with words that are far away from it to determine the relationships between them. Self-attention heads for local association are used to compare the current word with words that are close to it to determine the relationships between them.
  • the semantic data input by the user contains segments T1 to T8k.
  • the self-attention head (hesd-i) of the long-range association feature can compare the current segment T8k with the segments from T8k to T2k.
  • the self-attention head (hesd-j) of the local association feature can compare the current segment T8k with the segments from T8k to T2k.
  • Important self-attention heads can include self-attention heads with long-range correlation properties.
  • Insignificant self-attention heads can include self-attention heads with local correlation properties.
  • the importance level of each self-attention head is evaluated based on the attention score of each of the plurality of self-attention heads.
  • offline data is input into the attention model, and attention model inference is performed to determine the attention score of each self-attention head in the attention model (step 61).
  • input word embedding and position encoding operations are performed on the offline data to obtain the word segmentation vector and word segmentation position vector.
  • the word segmentation vector and word segmentation position vector are added to obtain the word segmentation representation vector input to the self-attention head.
  • the self-attention head performs attention calculation on the word segmentation representation vector to obtain the attention score.
  • the word segmentation representation vector is multiplied by the query weight WQ to obtain the query vector Q
  • the word segmentation representation vector is multiplied by the key weight WK to obtain the key vector K.
  • the attention score is obtained based on the dot product of the query vector Q and the key vector K.
  • the attention score satisfies formula (1).
  • S MAX(Q* KT ) Formula (1)
  • S represents the attention score
  • Q represents the query vector
  • KT represents the transpose of the key vector K.
  • MAX represents obtaining the maximum value in the attention score matrix.
  • Assess the importance level of the self-attention head (step 62). For example, compare the attention score to a threshold. If the attention score is greater than the threshold, the self-attention head corresponding to the attention score is an important self-attention head. If the attention score is less than the threshold, the self-attention head corresponding to the attention score is an unimportant self-attention head. If the attention score is equal to the threshold, the self-attention head corresponding to the attention score is either an unimportant self-attention head or an important self-attention head.
  • the attention scores of multiple self-attention points can be ranked, and the self-attention point with the highest attention score has the highest importance level, and so on.
  • Step 420 Determine the amount of data to be discarded by each self-attention head based on the importance level of each self-attention head.
  • the elimination rate varies depending on the importance level of the self-attention point. Importance level and elimination rate are inversely proportional. The higher the importance level of the self-attention point, the smaller the elimination rate; conversely, the lower the importance level, the larger the elimination rate.
  • Important self-attention points have a smaller elimination rate than unimportant ones. Important self-attention points retain all or most of the data inferred by the model, while unimportant self-attention points retain only a small portion of the data. Therefore, during the inference process of the attention model, relatively unimportant self-attention points retain more inference data, while important self-attention points retain more inference data.
  • the storage medium stores a correspondence between importance levels and elimination quantities.
  • the processor evaluates the importance level of each of the multiple self-attention heads in the attention model, it queries the correspondence based on the importance level of the self-attention head, determines the importance level in the correspondence that is the same as the importance level of the self-attention head, and determines the elimination quantity corresponding to the importance level that is the same as the importance level of the self-attention head in the correspondence as the elimination quantity of the self-attention head for the inferred data. For example, suppose the correspondence between importance levels and elimination quantities includes 10 corresponding items. The importance level of the first self-attention head is importance level 1. Querying the correspondence determines that the elimination quantity corresponding to importance level 1 is 10%, then the elimination quantity of the first self-attention head is 10%.
  • the elimination quantity corresponding to self-attention heads of different importance levels is determined based on the different positional encodings of the attention model.
  • the positional encoding method of the attention model is Rope positional encoding.
  • the importance level of a self-attention head is important, the elimination amount corresponding to the important self-attention head is 0.
  • the inference process of the attention model there is no need to eliminate the data inferred by the attention model; all the data inferred by the attention model is retained, thus preserving the ability to search for global information about the semantic data input by the user.
  • the elimination amount corresponding to the unimportant self-attention head is greater than 0.
  • the elimination amount can be adjusted based on empirical values.
  • data inferred by the self-attention head is eliminated according to the elimination amount corresponding to the self-attention head.
  • the elimination method is not limited in this application; for example, data at any position in the data inferred by the self-attention head can be eliminated, or data in the middle position in the data inferred by the self-attention head can be eliminated; or, based on the importance level of the inferred data, unimportant data in the inferred data can be eliminated.
  • self-attention heads with long-range correlation characteristics may not require setting a eviction threshold, or may require setting a small eviction threshold.
  • Self-attention heads with local correlation characteristics may require setting a larger eviction threshold.
  • the positional encoding method of the attention model is Alibi positional encoding.
  • the elimination amount for each self-attention head in the attention model is determined based on the model weights and the slope of the positional encoding of each self-attention head. The slope is related to the word segmentation position.
  • the elimination amount corresponding to each self-attention head is approximately calculated based on the query weight WQ and key weight WK , as well as the slope of the position encoding of each self-attention head (step 71).
  • the elimination amount has a functional relationship of f( WQ , WK , m).
  • L represents the elimination quantity
  • Q represents the query vector
  • KT represents the transpose of the key vector K
  • ⁇ represents a constant
  • m represents the slope of the positional encoding.
  • MAX represents obtaining the maximum value in the matrix.
  • the elimination amount corresponding to the self-attention head can be calibrated offline before the attention model inference. This avoids consuming the storage and computational resources of the attention model inference, alleviating bandwidth and computational bottlenecks and improving the inference performance of the attention model.
  • This application does not limit the length of the offline data.
  • the amount of offline data can be less than the amount of semantic data of the user input that the attention model needs to infer.
  • steps 410 and 420 can be executed to determine the elimination amount corresponding to multiple self-attention heads in the attention model, and then the elimination amount corresponding to multiple self-attention heads can be stored. Subsequent times when attention model inference is required, it is not necessary to determine the elimination amount corresponding to multiple self-attention heads again; instead, the elimination amount corresponding to multiple self-attention heads can be retrieved from the storage medium during the attention model inference process, and step 430 can be executed.
  • Step 430 During the inference process of the attention model, each self-attention head eliminates the inferred data according to the corresponding elimination amount.
  • the data inferred by the self-attention head is eliminated according to the elimination amount corresponding to the self-attention head, resulting in the remaining data after elimination.
  • the data inferred by the self-attention head is the key-value data obtained after the user-input semantic data is converted into word segments.
  • the user-input semantic data is converted into word segmentation representation vectors.
  • the self-attention head multiplies these representation vectors with the query weight WQ , key weight WK , and value weight WV respectively to calculate the query vector, key vector, and value vector.
  • the key-value data includes the query vector, key vector, and value vector.
  • the self-attention head Before storing the key-value data corresponding to the word segmentation representation vectors obtained from the user-input semantic data, the self-attention head eliminates the key-value data corresponding to the word segmentation representation vectors according to the elimination amount corresponding to the self-attention head, obtaining the remaining data after elimination. This remaining data is stored so that the encoder can encode the remaining data and the decoder can decode it. Therefore, while ensuring the accuracy of the attention model inference, the computational load and storage resources required for attention model inference are reduced.
  • the positional encoding method of the attention model is Rope positional encoding, where the length of the elimination quantity corresponding to important self-attention heads is 0, and the length of the elimination quantity corresponding to unimportant self-attention heads is greater than 0.
  • the positional encoding method of the attention model is Alibi positional encoding, and the elimination amount corresponding to important self-attention heads is less than the elimination amount corresponding to unimportant self-attention heads.
  • the self-attention head starts from the first data in the inferred data and eliminates the inferred data according to the elimination amount corresponding to the self-attention head.
  • self-attention head 1 discards the key-value data corresponding to the first 4 words and retains the key-value data corresponding to the last 6 words.
  • Self-attention head 2 discards the key-value data corresponding to the first 6 words and retains the key-value data corresponding to the last 4 words.
  • Self-attention head 3 discards and retains the key-value data corresponding to all words.
  • Self-attention head 4 discards the key-value data corresponding to the first 5 words and retains the key-value data corresponding to the last 5 words.
  • the black-filled segments represent discarded data, while the shaded segments represent the remaining data after discarding.
  • This application does not limit the specific value of the discarding amount. For example, 50%-70% of the data inferred by the model can be discarded.
  • the semantic data is "This is a cat”.
  • Word embedding and positional encoding are performed on “this”, “is”, “a”, and “cat” respectively, resulting in key-value data for "this”, “is”, “a”, and “cat”.
  • the self-attention head elimination rate for processing semantic data is 20%, the key-value data for "this” is eliminated, and the key-value data for "is”, “a”, and “cat” are stored.
  • Encoding and decoding are then performed on the key-value data for "is”, "a”, and “cat”.
  • the self-attention head elimination rate for processing semantic data is 40%, the key-value data for "this” and “is” is eliminated, and the key-value data for "a” and “cat” are stored.
  • Encoding and decoding are then performed on the key-value data for "a” and "cat”.
  • Attention model inference is performed on the remaining data after elimination for each self-attention head.
  • the remaining data after elimination can refer to the data remaining after eliminating the inferred data.
  • the remaining data after elimination for each self-attention head is input into the self-attention head, and after calculations involving the self-attention head, addition and layer normalization, a feedforward network, a multi-head attention layer in the decoder, a masked multi-head attention layer, a fully connected layer, and normalization, the attention model inference result is obtained.
  • the specific processing procedure can be found in the working principle of the Transformer attention model, and will not be elaborated further.
  • the attention model inference method provided in this application sets different elimination rates for self-attention heads of different importance levels.
  • self-attention heads of different importance levels acquire different amounts of inference data. For example, important self-attention heads eliminate fewer inference data and retain more, while less important self-attention heads eliminate more inference data and retain fewer. This reduces the computational load and storage resources required for attention model inference while ensuring the accuracy of the inference. Because of the reduced computational load and storage resources, the attention model can infer longer sequences and improve inference speed under the same hardware conditions.
  • the computer device includes hardware structures and/or software modules corresponding to the execution of each function.
  • Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.
  • the apparatus can be the computing node shown in Figure 3, or it can be a module (such as a chip) applied to a computer device.
  • the attention model inference device 900 includes a communication module 901, an evaluation module 902, an inference module 903, and a storage module 904.
  • the attention model inference device 900 is used to implement the functions of the processor in the method embodiment shown in Figure 4 above.
  • Communication module 901 is used to acquire the attention model.
  • Evaluation module 902 is used to evaluate the importance level of each of the multiple self-attention heads in the attention model. For example, evaluation module 902 is used to perform step 410 in Figure 4.
  • Evaluation module 902 is used to determine the amount of data to be discarded by each self-attention head based on the importance level of each self-attention head, wherein the importance level is inversely proportional to the amount of data to be discarded. For example, evaluation module 902 is used to perform step 420 in Figure 4.
  • Evaluation module 902 is specifically used to evaluate the importance level of each self-attention head based on the attention score of each self-attention head.
  • the inference module 903 is used to eliminate the inferred data by each self-attention head according to the corresponding elimination amount during the inference process of the attention model.
  • the inference module 903 is used to execute step 430 in Figure 4.
  • the storage module 904 is used to store the model, the elimination amount corresponding to the self-attention head, etc., so as to facilitate the execution of model inference.
  • the attention model inference device 900 of this application embodiment can be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD).
  • the PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
  • CPLD complex programmable logical device
  • FPGA field-programmable gate array
  • GAL generic array logic
  • the method shown in FIG4 when the method shown in FIG4 is implemented in software, its various modules can also be software modules; the attention model inference device 900 and its various modules can also be software modules.
  • the attention model inference device 900 can be used to execute the methods described in the embodiments of this application.
  • the above and other operations and/or functions of each unit in the attention model inference device 900 are respectively for implementing the corresponding processes of each method in FIG4. For the sake of brevity, they will not be described again here.
  • Figure 10 is a schematic diagram of the structure of a computer device 1000 provided in this application.
  • the computer device 1000 includes a processor 1010, a bus 1020, a memory 1030, a communication interface 1040, a main memory unit 1050 (also referred to as a main memory unit), and a processor 1060.
  • the processor 1010, processor 1060, memory 1030, main memory unit 1050, and communication interface 1040 are connected via the bus 1020.
  • the processor 1010 may be a CPU, but it may also be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • DSPs digital signal processors
  • a general-purpose processor may be a microprocessor or any conventional processor.
  • Computer device 1000 may also include a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of programs according to the present application.
  • GPU graphics processing unit
  • NPU neural network processing unit
  • microprocessor an ASIC
  • processor 1060 may be a GPU or an NPU.
  • the communication interface 1040 is used to enable communication between the computer device 1000 and external devices or components.
  • the communication interface 1040 is used to acquire models, etc. This allows the processor 1010 to evaluate the importance level of each of the multiple self-attention heads in the attention model, and determine the amount of data to be discarded by each self-attention head based on its importance level.
  • the processor 1060 is used to discard the data to be inferred by each self-attention head according to its corresponding discard amount during the inference process of the attention model.
  • Bus 1020 may include a pathway for transferring information between the aforementioned components (such as processor 1010, memory 1050, and storage 1030). In addition to a data bus, bus 1020 may also include a power bus, control bus, and status signal bus. However, for clarity, all buses are labeled as bus 1020 in the figure.
  • Bus 1020 may be a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc.
  • PCIe Peripheral Component Interconnect Express
  • EISA Extended Industry Standard Architecture
  • Ubus or UB Unified Bus
  • CXL Compute Express Link
  • CLIX Cache Coherent Interconnect for Accelerators
  • Bus 1020 can be divided into address bus, data bus, control bus, etc.
  • computer device 1000 may include multiple processors.
  • a processor may be a multi-core (multi-CPU) processor.
  • processor can refer to one or more devices, circuits, and/or computing units used to process data (e.g., computer program instructions).
  • Figure 10 only illustrates a computer device 1000 comprising one processor 1010 and one memory 1030.
  • the processor 1010 and memory 1030 are used to indicate a type of device or equipment.
  • the quantity of each type of device or equipment can be determined according to business requirements.
  • the computer device 1000 may include multiple GPUs or NPUs.
  • Memory 1050 can be a pool of volatile memory or a pool of non-volatile memory, or it can include both volatile and non-volatile memory.
  • the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.
  • the volatile memory can be random access memory (RAM), which is used as an external cache.
  • RAM synchronous dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • DDR SDRAM double data rate synchronous dynamic random access memory
  • ESDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous linked dynamic random access memory
  • DR RAM direct rambus RAM
  • the memory 1030 can correspond to the storage medium used in the above method embodiments for storing information such as the model and the elimination quantity corresponding to the self-attention head, such as a disk, like a mechanical hard disk or a solid-state hard disk.
  • the aforementioned computer device 1000 can be a general-purpose device or a special-purpose device.
  • computer device 1000 can also be a server or other device with computing capabilities.
  • the computer device 1000 can correspond to the attention model inference device 900 in this embodiment, and can correspond to the corresponding subject executing any method in FIG4.
  • the above and other operations and/or functions of each module in the attention model inference device 900 are respectively for implementing the corresponding processes of each method in FIG4. For the sake of brevity, they will not be described in detail here.
  • the method steps in this embodiment can be implemented in hardware or by a processor executing software instructions.
  • the software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art.
  • An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium.
  • the storage medium can also be a component of the processor.
  • the processor and storage medium can reside in an ASIC.
  • the ASIC can reside in a computing device.
  • the processor and storage medium can also exist as discrete components in the computing device.
  • implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof.
  • software When implemented using software, it can be implemented entirely or partially in the form of a computer program product.
  • the computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially.
  • the computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device.
  • the computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.
  • the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.
  • the computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media.
  • the available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).
  • SSD solid-state drive

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Machine Translation (AREA)

Abstract

公开了一种注意力模型推理方法、装置及系统，涉及人工智能领域。评估注意力模型的多个自注意力头中每个自注意力头的重要等级；根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量，其中，重要等级与淘汰量成反比；在注意力模型的推理过程中，每个自注意力头按照对应的淘汰量淘汰所推理的数据。从而，在确保注意力模型推理的精度的情况下，降低注意力模型推理的计算量和所需占用的存储资源。由于降低了注意力模型推理的计算量和所需占用的存储资源，使得在相同的硬件场景下，能够使注意力模型推理更长的序列，以及提升了注意力模型的推理速度。

Description

注意力模型推理方法、装置及系统
本申请要求于2024年06月14日提交国家知识产权局、申请号为202410773401.4，申请名称为“注意力模型推理方法、装置及系统”的中国专利申请的优先权，这些全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，尤其涉及一种注意力模型推理方法、装置及系统。
背景技术
目前，注意力模型对超长序列推理已成为未来趋势。例如，从推理文本对话发展到推理文档。随着序列长度增长，注意力模型推理面临着以下挑战：键值缓存(KV cache)成倍增加，甚至超过注意力模型的权重的数据量；注意力模型的计算量急剧膨胀，注意力模型的推理性能无法满足商用诉求。通常，注意力模型推理过程中，通过丢弃注意力模型所推理的序列包含的分词(token)，来减少注意力模型推理的分词的数量，进而减少所需占用的存储资源和注意力模型推理的计算量，但是，导致注意力模型推理的精度降低。
发明内容
本申请提供了一种注意力模型推理方法、装置及系统，由此在确保注意力模型推理的精度的情况下，降低注意力模型推理的计算量和所需占用的存储资源，从而，提升注意力模型推理的序列长度和推理速度。
第一方面，提供了一种注意力模型推理方法，包括：评估注意力模型的多个自注意力头中每个自注意力头的重要等级；根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量，其中，重要等级与淘汰量成反比；在注意力模型的推理过程中，每个自注意力头按照对应的淘汰量淘汰所推理的数据。
如此，对不同重要等级的自注意力头设置不同的淘汰量，在模型推理的过程中，使不同重要等级的自注意力头获取不同数量的推理数据，例如，重要的自注意力头淘汰较少的推理数据，保留较多的推理数据，不重要的自注意力头淘汰较多的推理数据，保留较少的推理数据，从而，在确保注意力模型推理的精度的情况下，降低注意力模型推理的计算量和所需占用的存储资源。由于降低了注意力模型推理的计算量和所需占用的存储资源，使得在相同的硬件场景下，能够使注意力模型推理更长的序列，以及提升了注意力模型的推理速度。
在一种可能的实现方式中，评估注意力模型的多个自注意力头中每个自注意力头的重要等级包括：根据每个自注意力头的注意力分数评估每个自注意力头的重要等级。
在另一种可能的实现方式中，方法还包括：将离线数据转化为分词，将分词转换为键值(Key-value)数据，根据键值(Key-value)数据确定分词的注意力分数，将注意力模型中每个自注意力头输出的分词的注意力分数确定为每个自注意力头的注意力分数。
例如，将自注意力头输出的多个注意力分数中最大注意力分数确定为自注意力头的注意力分数。
注意力分数表示自注意力头处理的多个分词之间的相关性，将自注意力头处理的分词的注意力分数确定为自注意力头的注意力分数，能够准确地确定自注意力头的重要等级。
另外，在模型推理之前，采用离线数据执行模型推理，来确定模型中自注意力头的注意力分数，进而评估注意力模型的多个自注意力头中每个自注意力头的重要等级和自注意力头对应的淘汰量，无需占用模型推理的存储资源和计算资源，缓解带宽和计算瓶颈，提高模型推理性能。
在另一种可能的实现方式中，注意力模型的位置编码方式为旋转位置编码(Rotary Position Embedding，Rope)位置编码；根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量包括：将重要等级与淘汰量的对应关系中自注意力头的重要等级对应的淘汰量确定为自注意力头对所推理的数据的淘汰量。
例如，当自注意力头的重要等级为重要，确定重要的自注意力头对应的淘汰量为0；当自注意力头的重要等级为不重要，确定不重要的自注意力头对应的淘汰量大于0。
将重要的自注意力头对应的淘汰量设置为0，在注意力模型的推理过程中，保留自注意力头所推理的所有数据，从而保留了对用户输入的语义数据的全局信息的搜索能力，确保注意力模型推理的精度。不重要的自注意力头对应的淘汰量设置为大于0，在注意力模型的推理过程中，可以淘汰自注意力头所推理的数据中部分数据，降低注意力模型推理的计算量和所需占用的存储资源。
在另一种可能的实现方式中，注意力模型的位置编码方式为注意力线性偏置位置编码(Attention with Linear Biases，Alibi)位置编码；根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量包括：根据模型权重和每个自注意力头的位置编码的斜率确定每个自注意力头对应的淘汰量，模型权重包括查询权重和键权重。
在另一种可能的实现方式中，每个自注意力头按照对应的淘汰量淘汰所推理的数据，包括：每个自注意力头从所推理的数据中第一个数据开始按照对应的淘汰量淘汰所推理的数据。
从而，淘汰所推理的数据中开始推理的数据，可以保留所推理的数据相关性较强的数据，确保注意力模型推理的精度。
在另一种可能的实现方式中，自注意力头对所推理的数据为对用户输入的语义数据转化为分词后，再由分词转换的键值数据。
从而，在将用户输入的语义数据转换为键值数据时充分获取了键值数据用户输入的语义数据之间的关系，淘汰键值数据，确保注意力模型推理的精度。
第二方面，提供了一种注意力模型推理装置，注意力模型推理装置包括用于执行第一方面或第一方面的任一种可能设计中的方法的模块。例如，注意力模型推理装置包括通信模块、评估模块和推理模块。
第三方面，提供了一种数据处理系统，该数据处理系统包括存储器和多个处理器，存储器用于存储一组计算机指令；当处理器执行一组计算机指令时，多个处理器执行第一方面或第一方面的任一种可能实现方式中的方法的操作步骤。
第四方面，提供了一种计算机设备，计算机设备包括存储器和多个处理器，存储器用于存储一组计算机指令；当处理器执行一组计算机指令时，多个处理器联合执行如第一方面或第一方面的任一种可能实现方式中的方法的操作步骤。
第五方面，提供一种芯片，包括：处理器和供电电路；其中，供电电路用于为处理器供电；处理器用于执行第一方面或第一方面任一种可能实现方式中的方法的操作步骤。
第六方面，提供了一种计算机可读存储介质，包括：计算机软件指令；当计算机软件指令在计算设备中运行时，使得计算设备执行如第一方面或第一方面任意一种可能的实现方式中所述方法的操作步骤。
第七方面，提供了一种计算机程序产品，当计算机程序产品在计算设备上运行时，使得计算设备执行如第一方面或第一方面任意一种可能的实现方式中所述方法的操作步骤。
第二方面至第七方面中任一种设计方式所带来的技术效果可参见第一方面或第一方面中不同设计方式所带来的技术效果，此处不再赘述。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
图1为本申请提供的一种Transformer模型的架构示意图；
图2为本申请提供的一种序列长度、批大小和内存大小的关系示意图；
图3为本申请提供的一种数据处理系统的架构示意图；
图4为本申请提供的一种注意力模型推理方法的流程示意图；
图5为本申请提供的一种长程关联特性的注意力头和局部关联特性的注意力头的示意图；
图6为本申请提供的一种自注意力头的重要等级评价流程示意图；
图7为本申请提供的一种计算淘汰量的流程示意图；
图8为本申请提供的一种淘汰所推理的数据的示意图；
图9为本申请提供的一种注意力模型推理装置的结构示意图；
图10为本申请提供的一种计算机设备的结构示意图。
具体实施方式
为了便于理解，首先对本申请所涉及的主要术语进行解释。
Transformer模型：是指一种基于自注意力机制的深度学习模型。其核心思想是通过计算语义数据(例如，序列)中每个分词与其他分词之间的相似度来寻找相关信息，从而生成包含更多上下文信息的输出矩阵。这种模型架构在自然语言处理领域取得了显著的性能提升，尤其是在机器翻译、文本摘要、问答系统等多个任务中。
Transformer模型是一个编码器-解码器(encoder-decoder)架构。示例地，如图1中的(a)所示，Transformer模型包含了多个编码器和多个解码器。如图1中的(b)所示，每个编码器包含两个子层：多头注意力层(multi-head attention)和前馈网络(position-wise-feed forward network，FNN)。多头注意力层包含多个自注意力头(self-attention)。每个编码器的结构相同，但是可以使用不同的权重。每个编码器还包含输入词嵌入(input embedding)、位置编码层(positional embedding)和相加和层归一化(add&norm)，add表示残差连接(residual connection)用于防止网络退化。norm表示layer normalization，用于对每一层的激活值进行归一化。解码器与编码器的区别在于：解码器包含两个多头注意力层，第一个多头注意力层为掩码多头注意力层(masked multi-head attention)；解码器还包含全连接层和归一化(SoftMax)。
输入词嵌入：是指模型将语义数据中的每个分词嵌入到一个高维向量表示中，获得分词向量。这个嵌入过程允许模型捕捉分词之间的语义相似性。
位置编码层：获取分词的位置向量，位置向量表示分词出现在语义数据中的位置。
将分词向量和分词位置向量相加得到输入多头注意力层的分词的表示向量。分词的表示向量可以是一个矩阵。
自注意力机制：允许模型在处理语义数据时，将每个分词与语义数据中的其他分词进行比较，以确定它们之间的关系。输入自注意力头的分词都被转换成三种表示：查询、键和值。通过计算查询与所有键之间的相似度，模型可以决策在生成每个输出分词时应该给予语义数据中每个分词多少注意力。然后，分词的注意力分数被用来加权相应的值，生成自注意力头的输出。
如图1中的(c)所示，自注意力头用于使用分词的表示向量分别与线性变化矩阵(查询权重WQ、键权重WK和值权重WV)计算得到查询(queries)向量、键(keys)向量和值(values)向量，以及根据查询向量和键向量计算语义数据中的每一对分词的注意力分数。其中，查询向量表示分词的查询，即模型在语义数据中寻找的内容。键向量表示分词的键，即语义数据中其他分词应该注意的内容。值向量表示分词的值，即分词对输出所贡献的信息。例如，根据查询向量和键向量的点积计算注意力分数，以评估分词之间的相关性。
在一些实施例中，多头注意力层将分词的表示向量输入每个自注意力头，每个自注意力头计算得到查询向量、键向量和值向量，将每个自注意力头输出的查询向量、键向量和值向量分别进行拼接，得到多头注意力层输出的查询向量、键向量和值向量，多头注意力层的输出向量与输入向量的维度一样。
然后，使用softmax函数对注意力分数进行归一化，以获得注意力权重。这些权重表示每个分词应该关注语义数据中其他分词的程度。注意力权重较高的分词被认为对正在执行的任务更为关键。
最后，使用注意力权重计算值向量的加权和。这产生了语义数据中分词的自注意力机制输出，捕获了来自其他分词的上下文信息。
分词(token)：在自然语言处理(Natural Language Processing，NLP)领域，指将文本字符串分解为最小语义单元(tokens)的过程。这些分词可以是单词、短语或其他语言单位。以下是一些常见的token化方法。
基于空格的分词化：指将文本按空格分割成分词或短语。
基于分隔符的分词化：如果文本使用特定的分隔符(如逗号、句号等)分隔不同的部分，可以使用这种方法。
基于语法规则的分词化：使用正则表达式或者库，如NLTK或SpaCy，可以根据语法规则进行更复杂的分词化。
基于机器学习的分词化：这种方法使用机器学习模型来识别和分割文本中的分词。
注意力分数：在Transformer架构中，注意力机制是一种核心组件，允许模型在处理输入语义数据时，根据上下文的不同部分分配不同的权重。注意力分数即是指这一过程中计算出的权重值，反映了模型对于输入语义数据中某一部分的关注程度。
超长序列推理：在自然语言处理任务中，超长序列推理指的是处理长度远超常规限制(如包含几千甚至上万的分词的序列)的文本数据的能力。
键值缓存(KV Cache)：是一种优化策略，主要用于加速长序列的推理过程。该策略利用了自注意力机制中的键和值对，在前向传播时存储并重用自注意力头中计算得到的键和值，从而避免了对历史信息的重复计算，极大降低了计算成本和内存占用。
模型参数:指在人工智能模型中需要进行学习或调整的变量或权重。模型参数可以影响模型的预测能力和性能。在模型训练过程中，模型通过尝试不同的参数组合来优化模型的性能。常见的模型参数包括权重、偏置、学习率和正则化系数等。在使用模型进行预测时，这些模型参数将被用来计算输出结果。
大模型：是指超大规模的人工智能(Artificial Intelligence，AI)模型。大模型在自然语言处理领域得到广泛应用，并正在彻底改变自然语言处理(Natural Language Processing，NLP)任务的状态，催生出更强大、更智能的语言技术。大模型是AI发展的重要方向之一。大模型也具有在各种自然语言处理任务中表现出色的能力，例如文本分类、情感分析、摘要生成、翻译等。大模型可以用于自动写作、聊天机器人、虚拟助手、语音助手、自动翻译等多个应用领域。例如，大语言模型(Large Language Model，LLM)、来自转换的双向编码器表示(Bidirectional Encoder Representations from Transformers，BERT)、生成式预训练转换模型(Generative Pre-Trained Transformer，GPT)、GPT3、GPT4、混合专家(Mixture of Experts，MoE)等。大模型具有以下特征。
1.巨大规模：模型大小可以达到数百吉字节(Gigabyte，GB)甚至更大，大模型包含数十亿、千亿或万亿的模型参数，这种巨大规模的模型提供了强大的表达能力和学习能力。
2.多任务学习：大模型可以处理多种不同的自然语言处理(Natural Language Processing，NLP)任务，如机器翻译、文本摘要、问答系统等，使模型学习到更广泛和泛化的语言理解能力。
3.强大的计算资源：训练大模型通常需要数百甚至上千个专用处理器，而且训练大模型的耗时较长，需要大量的时间。例如，训练大模型需要几周到几个月，强大的计算资源可以加速训练过程而保留大模型的能力。专用处理器包括但不限于图形处理单元(graphic processing unit，GPU)，数据处理单元(data processing unit，DPU)、神经处理单元(neural processing unit，NPU)和嵌入式神经网络处理器(neural-network processing unit，NPU)。专用处理器可以指除中央处理器(central processing unit，CPU)之外的加速处理的处理器，专用处理器也可以称为加速卡、训练处理器或加速处理器等。
4.丰富的数据：采用大量的训练数据训练大模型，发挥大模型的模型参数的规模优势。
注意力模型对超长序列推理已成为未来趋势。为了缓解随着序列长度增长，键值缓存(KV cache)成倍增加以及模型计算量的增加，如图2所示，以ChatGLM2-6B模型为例，模型大小约为12GB，但是随着序列长度N和批大小B增大，键值缓存的容量显著增加。注意力模型推理过程中，通过丢弃注意力模型所推理的序列包含的分词，来减少注意力模型推理的分词的数量，进而减少所需占用的存储资源和注意力模型推理的计算量。但是，在注意力模型处理超长序列场景下，尤其是大海捞针场景，每个分词都可能在某些输入下变得重要，不存在完全不重要的分词，分词的重要等级是相对于当前输入来评价的，所以对分词的丢弃导致注意力模型推理的精度下降。
为了解决丢弃分词导致注意力模型推理的精度下降的问题，本申请提供的注意力模型推理方法包括：评估注意力模型的多个自注意力头中每个自注意力头的重要等级；根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量，其中，重要等级与淘汰量成反比；在注意力模型的推理过程中，每个自注意力头按照对应的淘汰量淘汰所推理的数据。
如此，对不同重要等级的自注意力头设置不同的淘汰量，在模型推理的过程中，使不同重要等级的自注意力头获取不同数量的推理数据，例如，重要的自注意力头淘汰较少的推理数据，保留较多的推理数据，不重要的自注意力头淘汰较多的推理数据，保留较少的推理数据，从而，在确保注意力模型推理的精度的情况下，降低注意力模型推理的计算量和所需占用的存储资源。由于降低了注意力模型推理的计算量和所需占用的存储资源，使得在相同的硬件场景下，能够使注意力模型推理更长的序列，以及提升了注意力模型的推理速度。
超长序列推理已成为各大模型厂商增强智能体水平的重要方向，长序列推理拥有语义信息更加丰富、预测更准确和文本生成更加连贯的特点。本申请提供的注意力模型推理方法可应用于文档摘要、语言翻译、多轮对话和代码分析、代码生成等场景。
下面结合附图对本申请提供的注意力模型推理方法的实施方式进行详细描述。
图3为本申请提供的一种数据处理系统的架构示意图。如图3所示，数据处理系统300包括客户端310、计算集群320和存储集群330。
存储集群330包含多个存储节点331。一个存储节点331包括一个或多个控制器、网卡与多个硬盘。硬盘用于存储数据。硬盘可以是磁盘或者其他类型的存储介质，例如固态硬盘或者叠瓦式磁记录硬盘等。网卡用于与计算集群320包含的计算节点321通信。控制器用于根据计算节点321发送的读/写数据请求，往硬盘中写入数据或者从硬盘中读取数据。在读写数据的过程中，控制器需要将读/写数据请求中携带的地址转换为硬盘能够识别的地址。
计算集群320包含多个计算节点321。计算节点321可以是一种计算设备，如加速卡、服务器等。
在一些实施例中，计算集群320可以是一种异构计算架构，以提供高性能计算。例如计算节点321可以包括CPU、图形处理器(graphics processing unit，GPU)、数据处理单元(data processing unit，DPU)、神经处理单元(neural processing unit，NPU)和嵌入式神经网络处理器(neural-network processing unit，NPU)等具有计算能力的计算单元，以提供高性能计算。
例如，计算集群320包括多个加速卡，多个加速卡执行模型训练。
在另一些实施例中，多个计算节点321基于高速互连技术通过网络设备(如：交换机，网卡等)连接，使多个计算节点321之间相互通信。
客户端310通过网络340与计算集群320和存储集群330进行通信。例如客户端310通过网络340向计算集群320发送模型推理请求，请求计算集群320执行对序列执行模型推理。网络340可以是指企业内部网络(如：局域网(Local Area Network，LAN))或互联网(Internet)。客户端310指连入网络340的计算机，也可称为工作站(workstation)。不同的客户端可以共享网络上的资源(如：计算资源、存储资源)。
在一些实施例中，计算集群320还包括控制节点322。例如，控制节点和计算节点可以是独立的物理设备。又如，控制节点和多个计算节点可以位于同一物理设备。控制节点可以是CPU。多个计算节点包括GPU、NPU、DPU等计算单元。控制节点322用于管理和分配任务，由多个计算节点并行执行多个任务，以提升数据处理速率。例如，控制节点322用于根据模型推理请求指示多个计算节点并行执行模型推理。比如，一个计算节点执行一个编码器和一个解码器的功能。
在本申请中，在注意力模型推理前，计算节点或控制节点根据注意力模型中多个自注意力头的重要等级确定多个自注意力头对所推理的数据的淘汰量，重要的自注意力头对应的淘汰量大于不重要的自注意力头对应的淘汰量。将多个自注意力头对应的淘汰量输入注意力模型，在注意力模型推理过程中，根据自注意力头对应的淘汰量对自注意力头所需要推理的数据进行淘汰，对淘汰后的剩余数据进行模型推理。从而，在确保注意力模型推理的精度的情况下，降低注意力模型推理的计算量和所需占用的存储资源。由于降低了注意力模型推理的计算量和所需占用的存储资源，使得在相同的硬件场景下，能够使注意力模型推理更长的序列，以及提升了注意力模型的推理速度。
存储集群330可以用于存储模型推理相关的数据，例如，模型、模型参数等。在计算节点321执行模型推理时，可以从存储节点331获取模型、模型参数等，将模型、模型参数等存储到本地存储介质。
在另一些实施例中，客户端310安装有客户端程序311，客户端310运行客户端程序311显示一种用户界面(user interface，UI)，用户350操作用户界面提交请求。比如，用户350操作用户界面提交请求。控制节点322获取到请求后，从存储集群330获取模型和模型参数，将模型参数分配给多个计算节点，使多个计算节点基于上述实施例所述的并行技术执行模型推理。
可选地，系统管理员360可以通过客户端310调用应用平台接口(application platform interface，API)312或命令行界面(command-line interface，CLI)接口313配置系统信息等，例如本申请提供的为计算节点配置的自注意力头对应的淘汰量。
图3只是示意图，本申请的实施例对数据处理系统中设备连接方式和设备的数量不做限定。例如，数据处理系统可以包括多个客户端。一个客户端可以与多个计算节点连接。不同的客户端与不同的计算节点建立连接。
接下来，对注意力模型推理的整体过程中对自注意力头对应的淘汰量的设置进行说明。
图4为本申请提供的一种注意力模型推理方法的流程示意图。在这里以图3中所示的计算节点为例进行注意力模型推理进行说明。如图4所示，所述方法包括以下步骤410至步骤430。
步骤410、评估注意力模型的多个自注意力头中每个自注意力头的重要等级。
注意力模型在处理用户输入的语义数据时，将每个分词与语义数据中的其他分词进行比较，以确定它们之间的关系。其他分词包括与当前分词相近的分词或较远的分词中至少一个。也即是，注意力模型中多个自注意力头包括长程关联特性的自注意力头和局部关联特性的自注意力头。长程关联特性的自注意力头用于比较当前分词和与当前分词较远的分词，以确定它们之间的关系。局部关联特性的自注意力头用于比较当前分词和与当前分词相近的分词，以确定它们之间的关系。
示例地，如图5所示，用户输入的语义数据包含分词T1至分词T8k。长程关联特性的自注意力头(hesd-i)可以将当前分词T8k与T8k至分词T2之间的分词进行比较。局部关联特性的自注意力头(hesd-j)可以将当前分词T8k与T8k至分词T2k之间的分词进行比较。
重要的自注意力头可以包括长程关联特性的自注意力头。不重要的自注意力头可以包括局部关联特性的自注意力头。
在一些实施例中，根据多个自注意力头中每个自注意力头的注意力分数评估每个自注意力头的重要等级。
示例地，如图6所示，将离线数据输入注意力模型，执行注意力模型推理，确定注意力模型中每个自注意力头的注意力分数(步骤61)。例如，对离线数据进行输入词嵌入和位置编码操作，得到分词的向量和分词位置向量，将分词的向量和分词位置向量相加得到输入自注意力头的分词的表示向量，自注意力头对分词的表示向量进行注意力计算，得到注意力分数。示例地，将分词的表示向量与查询权重WQ相乘得到查询向量Q，将分词的表示向量与键权重WK相乘，得到键向量K，根据查询向量Q和键向量K的点积，得到注意力分数。注意力分数满足公式(1)。
S＝MAX(Q*KT)    公式(1)
其中，S表示注意力分数，Q表示查询向量，KT表示键向量K的转置。MAX表示获取注意力分数矩阵中的最大值。
评估自注意力头的重要等级(步骤62)。例如，将注意力分数与阈值进行比较，当注意力分数大于阈值，则注意力分数对应的自注意力头为重要的自注意力头。当注意力分数小于阈值，则注意力分数对应的自注意力头为不重要的自注意力头。当注意力分数等于阈值，则注意力分数对应的自注意力头为不重要的自注意力头或重要的自注意力头。
又如，对多个自注意力头的注意力分数进行排序，多个自注意力头中最大的注意力分数的自注意力头的重要等级最高，以此类推。
步骤420、根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量。
对于不同重要等级的自注意力头对应的淘汰量不同。重要等级与淘汰量成反比。自注意力头的重要等级越高，淘汰量越小，自注意力头的重要等级越低，淘汰量越大。重要的自注意力头对应的淘汰量小于不重要的自注意力头对应的淘汰量，重要的自注意力头保留模型所推理的数据中全部或大部分的数据，不重要的自注意力头保留模型所推理的数据中小部分的数据。从而，在注意力模型推理的过程中，相对不重要的自注意力头保留推理数据，重要的自注意力头保留的推理数据较多。
在一些实施例中，存储介质存储有重要等级与淘汰量的对应关系。处理器评估注意力模型的多个自注意力头中每个自注意力头的重要等级后，根据自注意力头的重要等级查询对应关系，确定对应关系中与自注意力头的重要等级相同的重要等级，将对应关系中与自注意力头的重要等级相同的重要等级对应的淘汰量确定为自注意力头对所推理的数据的淘汰量。例如，假设重要等级与淘汰量的对应关系包括10个重要等级与淘汰量的对应项。第一自注意力头的重要等级为重要等级1，查询对应关系，确定重要等级1对应的淘汰量为10％，则第一自注意力头的淘汰量为10％。
在一些实施例中，根据注意力模型的不同的位置编码，确定不同重要等级的自注意力头对应的淘汰量。
在第一种可能的实现方式中，注意力模型的位置编码方式为Rope位置编码。当自注意力头的重要等级为重要，确定重要的自注意力头对应的淘汰量为0。在注意力模型推理的过程中，无需淘汰注意力模型所推理的数据，保留注意力模型所推理的全部数据，从而保留了对用户输入的语义数据的全局信息的搜索能力。
当自注意力头的重要等级为不重要，确定不重要的自注意力头对应的淘汰量大于0。例如，可以根据经验值调节淘汰量。在注意力模型推理过程中，按照自注意力头对应的淘汰量淘汰自注意力头所推理的数据。淘汰方式本申请不予限定，例如，淘汰自注意力头所推理的数据中任意位置的数据，或者，淘汰自注意力头所推理的数据中中间位置的数据；又如，根据所推理的数据的重要等级，淘汰所推理的数据不重要的数据。
例如，对于长程关联特性的自注意力头可以无需设置淘汰量，或者，设置淘汰量较小。对于局部关联特性的自注意力头设置淘汰量较大。
在第二种可能的实现方式中，注意力模型的位置编码方式为Alibi位置编码。根据模型权重和每个自注意力头的位置编码的斜率确定注意力模型中每个自注意力头对应的淘汰量。斜率与分词位置相关。
示例地，如图7所示，根据查询权重WQ和键权重WK，以及每个自注意力头的位置编码的斜率近似计算自注意力头对应的淘汰量(步骤71)。例如，淘汰量具有f(WQ，WK，m)的函数关系。在一些实施例中，淘汰量满足公式(2)。
L＝[MAX(Q*KT)+α]/m  公式(2)
其中，L表示淘汰量，Q表示查询向量，KT表示键向量K的转置，α表示常量，m表示位置编码的斜率。MAX表示获取矩阵中的最大值。
需要说明的是，在注意力模型推理之前，可以离线校准自注意力头对应的淘汰量，从而，无需占用注意力模型推理的存储资源和计算资源，缓解带宽和计算瓶颈，提高注意力模型推理性能。本申请对离线数据的长度不予限定。离线数据的数据量可以小于注意力模型需要推理的用户输入的语义数据的数据量。
另外，可以在初次使用注意力模型进行推理时，执行步骤410和步骤420，确定注意力模型中多个自注意力头对应的淘汰量后，存储多个自注意力头对应的淘汰量。后续再需要执行注意力模型推理时，无需再确定多个自注意力头对应的淘汰量，可以在注意力模型推理过程中，从存储介质中获取多个自注意力头对应的淘汰量，执行步骤430。
步骤430、在注意力模型的推理过程中，每个自注意力头按照对应的淘汰量淘汰所推理的数据。
在注意力模型推理过程中，根据自注意力头对应的淘汰量对自注意力头所推理的数据进行淘汰，得到淘汰后的剩余数据，对淘汰后的剩余数据进行推理。自注意力头对所推理的数据为对用户输入的语义数据转化为分词后，再由分词转换的键值数据。例如，将用户输入的语义数据转化为分词的表示向量，自注意力头使用分词的表示向量分别与查询权重WQ、键权重WK和值权重WV相乘，计算得到查询向量、键向量和值向量。键值数据包括查询向量、键向量和值向量。在存储用户输入的语义数据转换得到分词的表示向量对应的键值数据之前，自注意力头根据自注意力头对应的淘汰量对分词的表示向量对应的键值数据进行淘汰，得到淘汰后的剩余数据，存储淘汰后的剩余数据。以便编码器对淘汰后的剩余数据进行编码操作和解码器对淘汰后的剩余数据进行解码操作。从而，在确保注意力模型推理的精度的情况下，降低注意力模型推理的计算量和所需占用的存储资源。
在第一种可能的实现方式中，注意力模型的位置编码方式为Rope位置编码，重要的自注意力头对应的淘汰量的长度为0，不重要的自注意力头对应的淘汰量的长度大于0。
示例地，如图8中的(a)所示，在注意力模型推理过程中，重要的自注意力头无需淘汰键值数据，无需淘汰注意力模型所推理的数据，保留注意力模型所推理的全部数据，从而保留了对用户输入的语义数据的全局信息的搜索能力。不重要的自注意力头淘汰部分键值数据，保留注意力模型所推理的数据中部分数据。例如，淘汰自注意力头所推理的数据中任意位置的数据，或者，淘汰自注意力头所推理的数据中中间位置的数据；又如，根据所推理的数据的重要等级，淘汰所推理的数据不重要的数据。
在第二种可能的实现方式中，注意力模型的位置编码方式为Alibi位置编码，重要的自注意力头对应的淘汰量小于不重要的自注意力头对应的淘汰量。
例如，自注意力头从所推理的数据中第一个数据开始根据自注意力头对应的淘汰量淘汰所推理的数据。
示例地，如图8中的(b)所示，自注意力头1淘汰前4个分词对应的键值数据，保留后6个分词对应的键值数据。自注意力头2淘汰前6个分词对应的键值数据，保留后4个分词对应的键值数据。自注意力头3淘汰保留所有分词对应的键值数据。自注意力头4淘汰前5个分词对应的键值数据，保留后5个分词对应的键值数据。
图8中的(a)和(b)所示，黑色填充的分词为淘汰的数据，阴影填充的分词为淘汰后的剩余数据。本申请对淘汰量的具体取值不予限定。例如，可以淘汰模型所推理的数据的50％-70％的数据。
例如，语义数据为“这是一只猫”。对“这”、“是”、“一只”、“猫”分别进行词嵌入和位置编码，得到“这”的键值数据、“是”的键值数据、“一只”的键值数据和“猫”的键值数据。假设处理语义数据的自注意力头的淘汰量为20％，则淘汰“这”的键值数据，存储“是”的键值数据、“一只”的键值数据和“猫”的键值数据，对“是”的键值数据、“一只”的键值数据和“猫”的键值数据进行编码和解码。假设处理语义数据的自注意力头的淘汰量为40％，则淘汰“这”的键值数据和“是”的键值数据，存储“一只”的键值数据和“猫”的键值数据，对“一只”的键值数据和“猫”的键值数据进行编码和解码。
对每个自注意力头对应的淘汰后的剩余数据进行注意力模型推理。淘汰后的剩余数据可以指淘汰所推理的数据后剩余的数据。例如，图8中的(a)和(b)所示的阴影填充的分词对应的键值数据。例如，将每个自注意力头对应的淘汰后的剩余数据输入自注意力头，经过自注意力头、相加和层归一化、前馈网络、解码器中的多头注意力层、掩码多头注意力层、全连接层和归一化等计算，得到注意力模型推理结果。具体的处理过程可以参考Transformer注意力模型的工作原理，不予赘述。
本申请提供的注意力模型推理方法，对不同重要等级的自注意力头设置不同的淘汰量，在模型推理的过程中，使不同重要等级的自注意力头获取不同数量的推理数据，例如，重要的自注意力头淘汰较少的推理数据，保留较多的推理数据，不重要的自注意力头淘汰较多的推理数据，保留较少的推理数据，从而，在确保注意力模型推理的精度的情况下，降低注意力模型推理的计算量和所需占用的存储资源。由于降低了注意力模型推理的计算量和所需占用的存储资源，使得在相同的硬件场景下，能够使注意力模型推理更长的序列，以及提升了注意力模型的推理速度。
可以理解的是，为了实现上述实施例中的功能，计算机设备包括了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本申请中所公开的实施例描述的各示例的单元及方法步骤，本申请能够以硬件或硬件和计算机软件相结合的形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用场景和设计约束条件。
上文中结合图1至图8，详细描述了根据本申请所提供的模型推理方法，下面将结合图9，描述根据本申请所提供的装置。这些装置可以用于实现上述方法实施例中专用处理器或通用处理器的功能，因此也能实现上述方法实施例所具备的有益效果。在本实施例中，该装置可以是图3所示的计算节点，还可以是应用于计算机设备的模块(如芯片)。
如图9所示，注意力模型推理装置900包括通信模块901、评估模块902、推理模块903和存储模块904。
注意力模型推理装置900用于实现上述图4中所示的方法实施例中处理器的功能。
通信模块901，用于获取注意力模型。
评估模块902，用于评估注意力模型的多个自注意力头中每个自注意力头的重要等级。例如，评估模块902用于执行图4中步骤410。
评估模块902，用于根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量，其中，重要等级与淘汰量成反比。例如，评估模块902用于执行图4中步骤420。
评估模块902，具体用于根据每个自注意力头的注意力分数评估每个自注意力头的重要等级。
推理模块903，用于在注意力模型的推理过程中，每个自注意力头按照对应的淘汰量淘汰所推理的数据。例如，推理模块903用于执行图4中步骤430。
存储模块904用于存储模型、自注意力头对应的淘汰量等，以便于执行模型推理。
应理解的是，本申请实施例的注意力模型推理装置900可以通过专用集成电路(application-specific integrated circuit，ASIC)实现，或可编程逻辑器件(programmable logic device，PLD)实现，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)，现场可编程门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic，GAL)或其任意组合。也可以通过软件实现图4所示的方法时，及其各个模块也可以为软件模块，注意力模型推理装置900及其各个模块也可以为软件模块。
根据本申请实施例的注意力模型推理装置900可对应于执行本申请实施例中描述的方法，并且注意力模型推理装置900中的各个单元的上述和其它操作和/或功能分别为了实现图4中的各个方法的相应流程，为了简洁，在此不再赘述。
图10为本申请提供的一种计算机设备1000的结构示意图。如图10所示，计算机设备1000包括处理器1010、总线1020、存储器1030、通信接口1040、内存1050(也可以称为主存(main memory)单元)和处理器1060。处理器1010、处理器1060、存储器1030、内存1050和通信接口1040通过总线1020相连。
应理解，在本实施例中，处理器1010可以是CPU，该处理器1010还可以是其他通用处理器、数字信号处理器(digital signal processing，DSP)、ASIC、FPGA或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者是任何常规的处理器等。
计算机设备1000还可以包括图形处理器(graphics processing unit，GPU)、神经网络处理器(neural network processing unit，NPU)、微处理器、ASIC、或一个或多个用于控制本申请方案程序执行的集成电路。例如，处理器1060可以是GPU或NPU。
通信接口1040用于实现计算机设备1000与外部设备或器件的通信。
在本申请中，计算机设备1000用于实现图4所示的处理器的功能时，通信接口1040用于获取模型等。以便于处理器1010用于评估注意力模型的多个自注意力头中每个自注意力头的重要等级，根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量。处理器1060用于在注意力模型的推理过程中，每个自注意力头按照对应的淘汰量淘汰所推理的数据。
总线1020可以包括一通路，用于在上述组件(如处理器1010、内存1050和存储器1030)之间传送信息。总线1020除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都标为总线1020。总线1020可以是快捷外围部件互连标准(Peripheral Component Interconnect Express，PCIe)总线，或扩展工业标准结构(extended industry standard architecture，EISA)总线、统一总线(unified bus，Ubus或UB)、计算机快速链接(compute express link，CXL)、缓存一致互联协议(cache coherent interconnect for accelerators，CCIX)等。总线1020可以分为地址总线、数据总线、控制总线等。
作为一个示例，计算机设备1000可以包括多个处理器。处理器可以是一个多核(multi-CPU)处理器。这里的处理器可以指一个或多个设备、电路、和/或用于处理数据(例如计算机程序指令)的计算单元。
值得说明的是，图10中仅以计算机设备1000包括1个处理器1010和1个存储器1030为例，此处，处理器1010和存储器1030分别用于指示一类器件或设备，具体实施例中，可以根据业务需求确定每种类型的器件或设备的数量。例如，计算机设备1000包括多个GPU或NPU。
内存1050可以是易失性存储器池或非易失性存储器池，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data date SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。内存1050用于存储模型、自注意力头对应的淘汰量等。
存储器1030可以对应上述方法实施例中用于存储模型、自注意力头对应的淘汰量等信息的存储介质，例如，磁盘，如机械硬盘或固态硬盘。
上述计算机设备1000可以是一个通用设备或者是一个专用设备。例如，计算机设备1000也可以是服务器或其他具有计算能力的设备。
应理解，根据本实施例的计算机设备1000可对应于本实施例中的注意力模型推理装置900，并可以对应于执行根据图4中任一方法中的相应主体，并且注意力模型推理装置900中的各个模块的上述和其它操作和/或功能分别为了实现图4中的各个方法的相应流程，为了简洁，在此不再赘述。
本实施例中的方法步骤可以通过硬件的方式来实现，也可以由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、CD-ROM或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于计算设备中。当然，处理器和存储介质也可以作为分立组件存在于计算设备中。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机程序或指令。在计算机上加载和执行所述计算机程序或指令时，全部或部分地执行本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、网络设备、用户设备或者其它可编程装置。所述计算机程序或指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机程序或指令可以从一个网站站点、计算机、服务器或数据中心通过有线或无线方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是集成一个或多个可用介质的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，例如，软盘、硬盘、磁带；也可以是光介质，例如，数字视频光盘(digital video disc，DVD)；还可以是半导体介质，例如，固态硬盘(solid state drive，SSD)。以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。

Claims (15)

  1. 一种注意力模型推理方法，其特征在于，包括：
    评估所述注意力模型的多个自注意力头中每个自注意力头的重要等级；
    根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量，其中，所述重要等级与所述淘汰量成反比；
    在所述注意力模型的推理过程中，每个自注意力头按照对应的淘汰量淘汰所推理的数据。
  2. 根据权利要求1所述的方法，其特征在于，所述评估所述注意力模型的多个自注意力头中每个自注意力头的重要等级包括：
    根据每个自注意力头的注意力分数评估每个自注意力头的重要等级。
  3. 根据权利要求1或2所述的方法，其特征在于，所述根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量包括：
    将重要等级与淘汰量的对应关系中所述自注意力头的重要等级对应的淘汰量确定为自注意力头对所推理的数据的淘汰量。
  4. 根据权利要求1或2所述的方法，其特征在于，所述根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量包括：
    根据模型权重和每个自注意力头的位置编码的斜率确定每个自注意力头对应的淘汰量，所述模型权重包括查询权重和键权重。
  5. 根据权利要求4所述的方法，其特征在于，每个自注意力头按照对应的淘汰量淘汰所推理的数据，包括：
    每个自注意力头从所推理的数据中第一个数据开始按照对应的淘汰量淘汰所推理的数据。
  6. 根据权利要求1-5中任一项所述的方法，其特征在于，自注意力头对所推理的数据为对用户输入的语义数据转化为分词后，再由分词转换的键值Key-value数据。
  7. 一种注意力模型推理装置，其特征在于，包括：
    评估模块，用于评估所述注意力模型的多个自注意力头中每个自注意力头的重要等级；
    所述评估模块，还用于根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量，其中，所述重要等级与所述淘汰量成反比；
    推理模块，用于在所述注意力模型的推理过程中，每个自注意力头按照对应的淘汰量淘汰所推理的数据。
  8. 根据权利要求7所述的装置，其特征在于，所述评估模块评估所述注意力模型的多个自注意力头中每个自注意力头的重要等级时，具体用于：
    根据每个自注意力头的注意力分数评估每个自注意力头的重要等级。
  9. 根据权利要求7或8所述的装置，其特征在于，所述评估模块根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量时，具体用于：
    将重要等级与淘汰量的对应关系中所述自注意力头的重要等级对应的淘汰量确定为自注意力头对所推理的数据的淘汰量。
  10. 根据权利要求7或8所述的装置，其特征在于，所述评估模块根据每个自注意力头的重要等级确定每个自注意力头对所推理的数据的淘汰量时，具体用于：
    根据模型权重和每个自注意力头的位置编码的斜率确定每个自注意力头对应的淘汰量。
  11. 根据权利要求10所述的装置，其特征在于，所述推理模块每个自注意力头按照对应的淘汰量淘汰所推理的数据时，具体用于：
    每个自注意力头从所推理的数据中第一个数据开始按照对应的淘汰量淘汰所推理的数据。
  12. 根据权利要求7-11中任一项所述的装置，其特征在于，自注意力头对所推理的数据为对用户输入的语义数据转化为分词后，再由分词转换的键值Key-value数据。
  13. 一种数据处理系统，其特征在于，所述数据处理系统包括存储器和多个处理器，所述存储器用于存储一组计算机指令；当所述处理器执行所述一组计算机指令时，所述多个处理器联合执行上述权利要求1-6中任一项所述的方法的操作步骤。
  14. 一种包含指令的计算机程序产品，其特征在于，当所述指令被计算设备运行时，使得所述计算设备执行如权利要求的1-6中任一项所述的方法的操作步骤。
  15. 一种计算机可读存储介质，其特征在于，包括计算机程序指令，当所述计算机程序指令由计算设备执行时，所述计算设备执行如权利要求1-6中任一项所述的方法的操作步骤。
PCT/CN2025/100461 2024-06-14 2025-06-11 注意力模型推理方法、装置及系统 Pending WO2025256563A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410773401.4A CN121146037A (zh) 2024-06-14 2024-06-14 注意力模型推理方法、装置及系统
CN202410773401.4 2024-06-14

Publications (1)

Publication Number Publication Date
WO2025256563A1 true WO2025256563A1 (zh) 2025-12-18

Family

ID=97991052

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/100461 Pending WO2025256563A1 (zh) 2024-06-14 2025-06-11 注意力模型推理方法、装置及系统

Country Status (2)

Country Link
CN (1) CN121146037A (zh)
WO (1) WO2025256563A1 (zh)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200134422A1 (en) * 2018-10-29 2020-04-30 International Business Machines Corporation Relation extraction from text using machine learning
CN115310607A (zh) * 2022-10-11 2022-11-08 南京理工大学 一种基于注意力图的视觉Transformer模型剪枝方法
CN118036741A (zh) * 2024-01-19 2024-05-14 Oppo广东移动通信有限公司 推理方法、装置以及电子设备

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200134422A1 (en) * 2018-10-29 2020-04-30 International Business Machines Corporation Relation extraction from text using machine learning
CN115310607A (zh) * 2022-10-11 2022-11-08 南京理工大学 一种基于注意力图的视觉Transformer模型剪枝方法
CN118036741A (zh) * 2024-01-19 2024-05-14 Oppo广东移动通信有限公司 推理方法、装置以及电子设备

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
LI JIAODA, RYAN COTTERELL , MRINMAYA SACHAN: "MasakhaNER: Named entity recognition for African languages", TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, vol. 9, 1 December 2021 (2021-12-01), pages 1442 - 1459, XP093381923, DOI: 10.1162/tacl a 00436 *
WANG HANRUI; ZHANG ZHEKAI; HAN SONG: "SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning", 2021 IEEE INTERNATIONAL SYMPOSIUM ON HIGH-PERFORMANCE COMPUTER ARCHITECTURE (HPCA), 27 February 2021 (2021-02-27), pages 97 - 110, XP033905611, DOI: 10.1109/HPCA51647.2021.00018 *

Also Published As

Publication number Publication date
CN121146037A (zh) 2025-12-16

Similar Documents

Publication Publication Date Title
US20240394477A1 (en) Constructing Prompt Information for Submission to a Language Model by Dynamically Selecting from Context Information
US12265576B2 (en) Systems and methods for multilingual intent prediction
EP3568852A1 (en) Training and/or using an encoder model to determine responsive action(s) for natural language input
JP2023099283A (ja) 情報検索のためのスパース表現を生成するニューラルランキングモデル
WO2024242824A1 (en) Constructing prompt information for submission to a language model by dynamically selecting from context information
CN119067221A (zh) 一种大语言模型推理加速方法、装置、系统及介质
WO2023221370A1 (zh) 批量任务处理的方法、装置及电子设备
CN117933307A (zh) 基于注意力提示的语言模型输出方法、装置和电子设备
CN118396128A (zh) 语言模型推理优化方法、电子设备、存储介质及程序产品
CN116830120A (zh) 具有切换层的神经网络
KR102815186B1 (ko) 이력 현상을 이용한 파라미터 양자화 기반 인공 신경망 연산 방법 및 장치
WO2025222855A1 (zh) 数据压缩方法和电子设备
WO2025256563A1 (zh) 注意力模型推理方法、装置及系统
JP2024519265A (ja) フィードフォワード空間変換ユニットを備えたニューラルネットワーク
WO2026031675A1 (zh) 一种语言模型的推理优化方法、装置、电子设备及存储介质
Hemmat et al. Cap’nn: A class-aware framework for personalized neural network inference
CN116050465B (zh) 文本理解模型的训练方法和文本理解方法、装置
CN119272827A (zh) 神经网络的数据处理方法、神经网络及芯片
CN112183744A (zh) 一种神经网络剪枝方法及装置
WO2024243106A1 (en) Constructing prompt information for submission to a language model by dynamically compressing source
CN114372480A (zh) 机器翻译模型的训练方法和机器翻译方法
US20260119845A1 (en) Low complexity prefix processing in language modeling
CN120745847B (zh) 推理请求处理方法、电子设备、存储介质及计算机程序产品
US20260072920A1 (en) System And Method For Efficient Execution of Large Generative Artificial Intelligence Models on Edge Devices Using State-Space Models
US20230123026A1 (en) Two-phase neural network architecture for user-specific search ranking

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25821269

Country of ref document: EP

Kind code of ref document: A1