WO2022068627A1 - 一种数据处理方法及相关设备 - Google Patents

一种数据处理方法及相关设备 Download PDF

Info

Publication number
WO2022068627A1
WO2022068627A1 PCT/CN2021/119306 CN2021119306W WO2022068627A1 WO 2022068627 A1 WO2022068627 A1 WO 2022068627A1 CN 2021119306 W CN2021119306 W CN 2021119306W WO 2022068627 A1 WO2022068627 A1 WO 2022068627A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
neural network
output
network model
weight value
Prior art date
Application number
PCT/CN2021/119306
Other languages
English (en)
French (fr)
Inventor
李梓超
侯璐
蒋欣
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP21874288.0A priority Critical patent/EP4209965A4/en
Publication of WO2022068627A1 publication Critical patent/WO2022068627A1/zh
Priority to US18/186,942 priority patent/US20230229898A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
    • G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
    • G06F7/544—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices for evaluating functions by calculation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
    • G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
    • G06F7/50—Adding; Subtracting
    • G06F7/501—Half or full adders, i.e. basic adder cells for one denomination
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
    • G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
    • G06F7/52—Multiplying; Dividing
    • G06F7/523—Multiplying only
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions

Definitions

  • the present application relates to the field of artificial intelligence, and in particular, to a data processing method and related equipment.
  • Artificial intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that responds in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.
  • the transformer structure has strong semantic expression ability and can capture long text dependencies. Since it was proposed, it has significantly surpassed the previous models in a series of natural language processing tasks represented by translation. The pre-trained language model based on the transformer structure has also achieved very good results in the fields of question answering systems and voice assistants.
  • the Bert-base model has 12 layers, 768 hidden states, and a total of 1.1 million parameters. Therefore, it is difficult to train a separate model for each natural language processing task, and then store it on the terminal device. It is more feasible to train a multi-task Transformer model.
  • a skeleton network (such as BERT) can be pre-trained first, the parameters of the skeleton network can be shared for all tasks, and then used as the underlying network, and a task-specific neural network can be added on the basis of each task. For training, such methods usually need to add more parameters, which cannot meet the requirements of low computing resources of terminal equipment.
  • the present application provides a data processing method, the method comprising:
  • the target neural network model includes a first transform (transformer) layer, the first transformer layer includes a first residual branch and a second residual branch, the first The residual branch includes the first attention head, and the second residual branch includes the target feedforward layer FFN; wherein, the branch where the multi-head attention layer is located and the branch where the feedforward layer FFN is located can be in the transformer layer
  • the residual branch of ; the data to be processed can be text data, and the target neural network model can be a trained transformer model that can perform multi-task processing.
  • a location includes multiple task layers, each of which is adapted to a different task.
  • the weight value corresponding to the target task where the weight value includes the first weight value corresponding to the first attention head and/or the second weight value corresponding to the target FFN; in order to adapt to different tasks, it can be determined Different weight values are used to operate with the output of the attention head in the transformer layer or the output of the FFN to control the output of the residual branch.
  • different weights can be selected, so that the residual The output of the difference branch is more suitable for the task performed by the current target neural network model;
  • the target task-related processing is performed on the data to be processed according to the target neural network model to obtain a data processing result, wherein the target neural network model is used to associate the output of the first attention head with the The target operation is performed on the first weight value to obtain the output of the first residual branch, and/or the target neural network model is used to perform target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch.
  • a weight value for controlling the output of the residual branch is set, which is equivalent to setting a set of exclusive distributed weight value combination for each task, thereby achieving the goal Multi-task learning of the neural network model, and in this embodiment, compared with adding a neural network adapted to different tasks in the output layer, fewer parameters need to be learned, thereby reducing the computing resource requirements for the terminal device to run the target neural network model.
  • the target neural network model when used to use the output of the first attention head as the output of the first residual branch, the target neural network model is directed to The data processing precision of the target task is smaller than the first processing precision, and the first processing precision is that the target neural network model is used to perform the target operation on the output of the first attention head and the first weight value to obtain In the case of the output of the first residual branch, the data processing accuracy of the target neural network model; or,
  • the data processing accuracy of the target neural network model for the target task is smaller than the second Processing accuracy
  • the second processing accuracy is when the target neural network model is used to perform target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch, the data processing accuracy of the target neural network model needle;
  • the data processing accuracy of the target neural network model for the target task is less than a third processing accuracy, where the third processing accuracy is used in the target neural network model for comparing the output of the target FFN with the second weight value Perform a target operation to obtain the output of the second residual branch, and perform a target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch, the The data processing accuracy of the target neural network model needle.
  • the residual branch of the original transformer layer is added to the The weight control strategy, and different weight values are set for different tasks.
  • the first weight value is obtained by updating the first initial weight value when the target neural network model is trained for the target task, wherein, when the target neural network model is trained for the target task During the training process of the neural network model, the target neural network model is used to perform target operation on the output of the first attention head and the first initial weight value to obtain the output of the first residual branch.
  • the second weight value is obtained by updating the second initial weight value when the target neural network model is trained for the target task.
  • the target neural network model is used to perform target operation on the output of the target FFN and the second initial weight value to obtain the output of the second residual branch.
  • the training device can obtain the data to be processed, the correct data processing result of the data to be processed, and the initial neural network model for performing the target task, the initial neural network model includes a first transformer layer, and the first transformer layer includes the first transformer layer.
  • a residual branch and a second residual branch the first residual branch includes a first attention head, the second residual branch includes a target feedforward layer FFN, and the initial neural network model uses The target operation is performed on the output of the first attention head and the first initial weight value to obtain the output of the first residual branch, and/or the initial neural network model is used to use the target FFN Perform target operation on the output and the second initial weight value to obtain the output of the second residual branch; process the data to be processed according to the initial neural network model to obtain a data processing result; Obtaining the loss from the data processing result and the correct data processing result, and updating the first initial weight value and/or the second initial weight value based on the loss, until the data processing accuracy of the initial neural network model for the target task meets the requirements, In this way, the first weight value
  • different first weight values and second weight values can be obtained for different tasks, and then the preset mapping relationship in the above embodiment can be obtained.
  • the obtaining the weight value corresponding to the target task includes:
  • the weight value corresponding to the target task is obtained according to a preset mapping relationship, and the weight value corresponding to the target task includes the first weight value and/or the second weight value; wherein the preset mapping relationship Including the correspondence between tasks and weight values.
  • Different first weight values and second weight values can be obtained for different tasks, and then the preset mapping relationship in the above embodiment can be obtained.
  • the training device may obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the first initial weight value and/or the first initial weight value based on the loss
  • the second initial weight value is used to obtain the first neural network model, and in the process of the i+1th iteration, the first neural network model is updated based on the loss except for the first initial weight value and/or all
  • the network parameters of the second initial weight value are described to obtain the target neural network model.
  • the skeleton model (the skeleton model can be understood as the part of the network except the weight value in the target neural network model) can be fixed in one iteration (wherein "fixed” can be understood as maintaining the structure and parameter size of the network unchanged), and update the weight value, then in the next iteration, fix the weight value and update the skeleton model (part of the target neural network model except for the weight value).
  • the obtaining the weight value corresponding to the target task includes:
  • the first neural network has the capability of inputting at least one of the data to be processed and the output of the first attention head, as well as the identifier of the target task, and outputting a first weight value.
  • the second neural network has the capability of inputting the data to be processed and/or the output of the target FFN and the identifier of the target task into the second neural network to obtain the second weight value.
  • first neural networks and second neural networks can be obtained by training.
  • the above-mentioned first neural network and second neural network can be, but are not limited to, fully connected networks.
  • the initial gate value of the target can be set, and then the gradient descent method can be used to update the model, and other optimization methods such as reinforcement learning can also be used. Algorithms and Genetic Algorithms for training.
  • the first neural network is obtained by updating a first initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the first attention head, and the identification of the target task into the first an initial neural network, and perform target operation on the output of the first initial neural network and the output of the first attention head to obtain the output of the first residual branch.
  • the second neural network is obtained by updating the second initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the target FFN, and the identification of the target task into the second initial neural network. network, and perform target operation on the output of the second initial neural network and the output of the target FFN to obtain the output of the second residual branch.
  • the first transformer layer includes multiple attention heads, each attention head in the multiple attention heads corresponds to a weight value, and the target neural network model is used to The output of each attention head and the corresponding weight value are subjected to target operation to obtain the output of the first residual branch, wherein the weight values corresponding to different attention heads are different.
  • the target neural network model may include a multi-head attention layer, and for each task t, each attention head n may deploy a weight value of the attention head to control the output of the residual branch where it belongs size.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the present application provides a data processing method, the method comprising:
  • the target neural network model includes a first transformer layer and a second transformer layer, and the first transformer layer includes a first attention head and a target FFN;
  • the weight value includes the first weight value corresponding to the first attention head and/or the second weight value corresponding to the target FFN;
  • the target task-related processing is performed on the data to be processed according to the target neural network model to obtain a data processing result, wherein the target neural network model is used to associate the output of the first attention head with the performing a first operation on the first weight value to obtain a first output, and performing a second operation on the first output and the output of the second transformer layer; and/or, the target neural network model is used to apply the A first operation is performed on the target FFN and the second weight value to obtain a second output, and a second operation is performed on the second output and the output of the second transformer layer.
  • the target neural network model is used to perform the first operation on the output of the first attention head and the first weight value to obtain the first output, which is not used as the output of the first residual branch , but a second operation is performed between the first output and the output of the second transformer layer.
  • the target neural network model is used to perform a first operation on the target FFN and the second weight value to obtain a second output.
  • the second output is not used as the output of the second residual branch, but A second operation is performed between the second output and the output of the second transformer layer.
  • the first operation may include a product operation
  • the second operation may include an addition operation.
  • the target neural network model may, on the one hand, perform a target operation (such as a product operation) on the output of the first attention head and the first weight value, and use the operation result as the output of the first residual branch, and on the other hand.
  • the first output is also summed with the output of the second transformer layer.
  • the target neural network model can perform a target operation (such as a product operation) on the target FFN and the second weight value, and use the operation result as the output of the second residual branch; The second output is summed with the output of the second transformer layer.
  • the target neural network model includes multiple transformer layers and an output layer, and the second transformer layer is the transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes a first residual branch and a second residual branch, and the first residual branch includes the first attention head;
  • the target neural network model When the target neural network model is used to use the output of the first attention head only as the output of the first residual branch, the target neural network model is used for data processing of the target task
  • the accuracy is less than the first processing accuracy
  • the first processing accuracy is that the target neural network model is used for the target neural network model to perform the first operation on the output of the first attention head and the first weight value , the data processing accuracy of the target neural network model when the first output is obtained, and the third output and the output of the second transformer layer are subjected to the second operation; or,
  • the data processing accuracy of the target neural network model for the target task is less than the third Second processing precision
  • the second processing precision is the case where the target neural network model is used to perform the first operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch , the data processing accuracy of the target neural network model needle;
  • the output of the first attention head is only used as the output of the first residual branch and the output of the target FFN is used as the output of the second residual branch
  • the data processing accuracy of the target neural network model for the target task is less than a third processing accuracy
  • the third processing accuracy is when the target neural network model is used for the target neural network model.
  • the first operation is performed on the output of the first attention head and the first weight value to obtain the first output
  • the second operation is performed on the third output and the output of the second transformer layer
  • the The first operation is performed on the output of the target FFN and the second weight value to obtain the output of the second residual branch
  • the first operation is performed on the output of the target FFN and the second weight value to obtain the second
  • the data processing accuracy of the target neural network model needle is the case of the output of the residual branch.
  • the first weight value is obtained by updating the first initial weight value when the target neural network model is trained for the target task, wherein, when the target neural network model is trained for the target task
  • the target neural network model is used to perform a first operation on the output of the first attention head and the first initial weight value, and the operation result is compared with the output of the second transformer layer. Perform the second operation.
  • the second weight value is obtained by updating the second initial weight value when the target neural network model is trained for the target task.
  • the target neural network model is used to perform a first operation on the target FFN and the second initial weight value, and perform a second operation between the operation result and the output of the second transformer layer. operation.
  • the obtaining the weight value corresponding to the target task includes:
  • the weight value corresponding to the target task is obtained according to a preset mapping relationship, and the weight value corresponding to the target task includes the first weight value and/or the second weight value; wherein the preset mapping relationship Including the correspondence between tasks and weight values.
  • the obtaining the weight value corresponding to the target task includes:
  • the first neural network is obtained by updating a first initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the first attention head, and the identification of the target task into the first an initial neural network, performing a first operation on the output of the first initial neural network and the output of the first attention head, and performing a second operation on the operation result and the output of the second transformer layer.
  • the second neural network is obtained by updating the second initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the target FFN, and the identification of the target task into the second initial neural network. network, and perform a first operation between the output of the second initial neural network and the output of the target FFN, and perform a second operation between the operation result and the output of the second transformer layer.
  • the first transformer layer includes multiple attention heads, and each attention head in the multiple attention heads corresponds to a weight value.
  • the target neural network model uses The first operation is performed on the output of each attention head and the corresponding weight value to obtain the third output, and the second operation is performed on the third output and the output of the second transformer layer.
  • the weight values corresponding to the heads are different.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the present application provides a data processing method, comprising:
  • the initial neural network model includes a first transformer layer, and the first transformer layer includes a first residual branch and a second residual branch, the first residual branch includes a first attention head, the second residual branch includes a target feedforward layer FFN, and the initial neural network model is used to The output of the first attention head and the first initial weight value are subjected to target operation to obtain the output of the first residual branch, and/or the initial neural network model is used to combine the output of the target FFN with the first Carry out target operation with two initial weight values to obtain the output of the second residual branch;
  • a loss is obtained, and the first initial weight value and/or the second initial weight value is updated based on the loss to obtain a target neural network model.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the loss is obtained according to the data processing result and the correct data processing result, and the first initial weight value and/or the second initial weight value is updated based on the loss, to Get the target neural network model, including:
  • the loss is obtained, and in the process of the ith iteration, only the first initial weight value and/or the second initial weight value is updated based on the loss, so as to Obtain the first neural network model, and in the process of the i+1th iteration, update the network parameters of the first neural network model except the first initial weight value and/or the second initial weight value based on the loss , to get the target neural network model.
  • the present application provides a data processing method, comprising:
  • the initial neural network model includes a first transformer layer, and the first transformer layer includes a first residual branch and a second residual branch, the first residual branch includes a first attention head, the second residual branch includes a target feedforward layer FFN, and the initial neural network model is used to
  • the data to be processed and/or the output of the first attention head and the identification of the target task are input into the first initial neural network, and the output of the first initial neural network is combined with the first attention
  • the output of the head is subjected to target operation to obtain the output of the first residual branch, and/or the initial neural network model is used to use the data to be processed and/or the output of the target FFN and the target
  • the identification of the task is input into the second initial neural network, and the output of the second initial neural network and the output of the target FFN are subjected to target operation to obtain the output of the second residual branch;
  • a loss is obtained, and the first initial neural network and/or the second initial neural network is updated based on the loss to obtain a target neural network model.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the loss is obtained according to the data processing result and the correct data processing result, and the first initial neural network and/or the second initial neural network is updated based on the loss, to Get the target neural network model, including:
  • the loss is obtained, and in the process of the ith iteration, only the first initial neural network and/or the second initial neural network is updated based on the loss, so as to Obtain a first neural network model, and in the process of the i+1th iteration, update the network parameters of the first neural network model other than the first initial neural network and/or the second initial neural network based on the loss , to get the target neural network model.
  • the application provides a data processing method, the method comprising:
  • the initial neural network model includes a first transformer layer and a second transformer layer.
  • the first transformer layer Including a first attention head and a target FFN; the initial neural network model is used to perform a first operation on the output of the first attention head and a first weight value to obtain a first output, and the first output Perform a second operation with the output of the second transformer layer; and/or, the initial neural network model is used to perform a first operation on the target FFN and the second weight value to obtain a second output, and use the performing a second operation on the second output with the output of the second transformer layer;
  • a loss is obtained, and the first initial weight value and/or the second initial weight value is updated based on the loss to obtain a target neural network model.
  • the initial neural network model includes multiple transformer layers and an output layer
  • the second transformer layer is a transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the loss is obtained according to the data processing result and the correct data processing result, and the first initial weight value and/or the second initial weight value is updated based on the loss, to Get the target neural network model, including:
  • the loss is obtained, and in the process of the ith iteration, only the first initial weight value and/or the second initial weight value is updated based on the loss, so as to Obtain the first neural network model, and in the process of the i+1th iteration, update the network parameters of the first neural network model except the first initial weight value and/or the second initial weight value based on the loss , to get the target neural network model.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the present application provides a data processing method, the method comprising:
  • the initial neural network model includes a first transformer layer and a second transformer layer.
  • the first transformer layer Including a first attention head and a target FFN; the initial neural network model is used to input the data to be processed and/or the output of the first attention head and the identification of the target task into the first initial neural network network, perform a first operation on the output of the first initial neural network and the output of the first attention head, and perform a second operation on the operation result and the output of the second transformer layer; and/or, all
  • the initial neural network model is used to input the data to be processed and/or the output of the target FFN and the identification of the target task into the second initial neural network, and the output of the second initial neural network is combined with the output of the target FFN.
  • the first operation is performed on the output of the target FFN
  • the second operation is performed on the result of the operation and the output of the second transformer layer
  • a loss is obtained, and the first initial neural network and/or the second initial neural network is updated based on the loss to obtain a target neural network model.
  • the initial neural network model includes multiple transformer layers and an output layer
  • the second transformer layer is a transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the loss is obtained according to the data processing result and the correct data processing result, and the first initial neural network and/or the second initial neural network is updated based on the loss, to Get the target neural network model, including:
  • the loss is obtained, and in the process of the ith iteration, only the first initial neural network and/or the second initial neural network is updated based on the loss, so as to Obtain a first neural network model, and in the process of the i+1th iteration, update the network parameters of the first neural network model other than the first initial neural network and/or the second initial neural network based on the loss , to get the target neural network model.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the present application provides a data processing device, the device comprising:
  • the acquisition module is used to acquire the data to be processed and the target neural network model
  • the target neural network model includes a first transform (transformer) layer
  • the first transformer layer includes a first residual branch and a second residual branch
  • the first residual branch includes a first attention head
  • the second residual branch includes a target feedforward layer FFN
  • the weight value corresponding to the target task is obtained, and the weight value includes the first attention the first weight value corresponding to the header and/or the second weight value corresponding to the target FFN;
  • a data processing module is used to process the data to be processed according to the target neural network model to obtain a data processing result, wherein the target neural network model is used to compare the output of the first attention head with the first The weight value is subjected to target operation to obtain the output of the first residual branch, and/or the target neural network model is used to perform target operation on the output of the target FFN and the second weight value to obtain the second The output of the residual branch.
  • the target neural network model when used to use the output of the first attention head as the output of the first residual branch, the target neural network model is directed to The data processing precision of the target task is smaller than the first processing precision, and the first processing precision is that the target neural network model is used to perform the target operation on the output of the first attention head and the first weight value to obtain In the case of the output of the first residual branch, the data processing accuracy of the target neural network model; or,
  • the data processing accuracy of the target neural network model for the target task is smaller than the second Processing accuracy
  • the second processing accuracy is when the target neural network model is used to perform target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch, the data processing accuracy of the target neural network model needle;
  • the data processing accuracy of the target neural network model for the target task is less than a third processing accuracy, where the third processing accuracy is used in the target neural network model for comparing the output of the target FFN with the second weight value Perform a target operation to obtain the output of the second residual branch, and perform a target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch, the The data processing accuracy of the target neural network model needle.
  • the first weight value is obtained by updating the first initial weight value when the target neural network model is trained for the target task, wherein, when the target neural network model is trained for the target task During the training process of the neural network model, the target neural network model is used to perform target operation on the output of the first attention head and the first initial weight value to obtain the output of the first residual branch.
  • the second weight value is obtained by updating the second initial weight value when the target neural network model is trained for the target task.
  • the target neural network model is used to perform target operation on the output of the target FFN and the second initial weight value to obtain the output of the second residual branch.
  • the obtaining module is used to:
  • the weight value corresponding to the target task is obtained according to a preset mapping relationship, and the weight value corresponding to the target task includes the first weight value and/or the second weight value; wherein the preset mapping relationship Including the correspondence between tasks and weight values.
  • the obtaining module is used to:
  • the first neural network is obtained by updating a first initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the first attention head, and the identification of the target task into the first an initial neural network, and perform target operation on the output of the first initial neural network and the output of the first attention head to obtain the output of the first residual branch.
  • the second neural network is obtained by updating the second initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the target FFN, and the identification of the target task into the second initial neural network. network, and perform target operation on the output of the second initial neural network and the output of the target FFN to obtain the output of the second residual branch.
  • the first transformer layer includes multiple attention heads, each attention head in the multiple attention heads corresponds to a weight value, and the target neural network model is used to The output of each attention head and the corresponding weight value are subjected to target operation to obtain the output of the first residual branch, wherein the weight values corresponding to different attention heads are different.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the present application provides a data processing device, the device comprising:
  • the target neural network model includes a first transformer layer and a second transformer layer, and the first transformer layer includes a first attention head and a target FFN; obtain the target The weight value corresponding to the task, the weight value includes the first weight value corresponding to the first attention head and/or the second weight value corresponding to the target FFN;
  • a data processing module is used to process the data to be processed according to the target neural network model to obtain a data processing result, wherein the target neural network model is used to compare the output of the first attention head with the first A first operation is performed on the weight value to obtain a first output, and a second operation is performed on the first output and the output of the second transformer layer; and/or the target neural network model is used to convert the target FFN A first operation is performed with the second weight value to obtain a second output, and a second operation is performed between the second output and the output of the second transformer layer.
  • the target neural network model includes multiple transformer layers and an output layer, and the second transformer layer is the transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes a first residual branch and a second residual branch
  • the first residual branch includes the first attention head
  • the second residual branch includes the target FFN;
  • the target neural network model When the target neural network model is used to use the output of the first attention head only as the output of the first residual branch, the target neural network model is used for data processing of the target task
  • the accuracy is less than the first processing accuracy
  • the first processing accuracy is that the target neural network model is used for the target neural network model to perform the first operation on the output of the first attention head and the first weight value , the data processing accuracy of the target neural network model when the first output is obtained, and the third output and the output of the second transformer layer are subjected to the second operation; or,
  • the data processing accuracy of the target neural network model for the target task is less than the third Second processing precision
  • the second processing precision is the case where the target neural network model is used to perform the first operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch , the data processing accuracy of the target neural network model needle;
  • the output of the first attention head is only used as the output of the first residual branch and the output of the target FFN is used as the output of the second residual branch
  • the data processing accuracy of the target neural network model for the target task is less than a third processing accuracy
  • the third processing accuracy is when the target neural network model is used for the target neural network model.
  • the first operation is performed on the output of the first attention head and the first weight value to obtain the first output
  • the second operation is performed on the third output and the output of the second transformer layer
  • the The first operation is performed on the output of the target FFN and the second weight value to obtain the output of the second residual branch
  • the first operation is performed on the output of the target FFN and the second weight value to obtain the second
  • the data processing accuracy of the target neural network model needle is the case of the output of the residual branch.
  • the first weight value is obtained by updating the first initial weight value when the target neural network model is trained for the target task, wherein, when the target neural network model is trained for the target task
  • the target neural network model is used to perform a first operation on the output of the first attention head and the first initial weight value, and the operation result is compared with the output of the second transformer layer. Perform the second operation.
  • the second weight value is obtained by updating the second initial weight value when the target neural network model is trained for the target task.
  • the target neural network model is used to perform a first operation on the target FFN and the second initial weight value, and perform a second operation between the operation result and the output of the second transformer layer. operation.
  • the obtaining module is used to:
  • the weight value corresponding to the target task is obtained according to a preset mapping relationship, and the weight value corresponding to the target task includes the first weight value and/or the second weight value; wherein the preset mapping relationship Including the correspondence between tasks and weight values.
  • the obtaining module is used to:
  • the first neural network is obtained by updating a first initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the first attention head, and the identification of the target task into the first an initial neural network, performing a first operation on the output of the first initial neural network and the output of the first attention head, and performing a second operation on the operation result and the output of the second transformer layer.
  • the second neural network is obtained by updating the second initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the target FFN, and the identification of the target task into the second initial neural network. network, and perform a first operation between the output of the second initial neural network and the output of the target FFN, and perform a second operation between the operation result and the output of the second transformer layer.
  • the first transformer layer includes multiple attention heads, and each attention head in the multiple attention heads corresponds to a weight value.
  • the target neural network model uses The first operation is performed on the output of each attention head and the corresponding weight value to obtain the third output, and the second operation is performed on the third output and the output of the second transformer layer.
  • the weight values corresponding to the heads are different.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the present application provides a data processing device, comprising:
  • the acquisition module is used to acquire the data to be processed, the correct data processing result of the data to be processed, and the initial neural network model for executing the target task
  • the initial neural network model includes a first transformer layer, and the first transformer layer It includes a first residual branch and a second residual branch
  • the first residual branch includes a first attention head
  • the second residual branch includes a target feedforward layer FFN
  • the model is used to perform target operation on the output of the first attention head and the first initial weight value to obtain the output of the first residual branch
  • the initial neural network model is used to combine the target The output of the FFN and the second initial weight value are subjected to target operation to obtain the output of the second residual branch;
  • a data processing module configured to process the data to be processed according to the initial neural network model to obtain a data processing result
  • a model updating module configured to obtain a loss according to the data processing result and the correct data processing result, and update the first initial weight value and/or the second initial weight value based on the loss to obtain a target neural network Model.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the model updating module is configured to obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the ith based on the loss an initial weight value and/or the second initial weight value to obtain a first neural network model, and in the process of the i+1 th iteration, update the first neural network model based on the loss except for the first neural network model.
  • the initial weight value and/or the network parameters of the second initial weight value to obtain the target neural network model.
  • the present application provides a data processing device, comprising:
  • the acquisition module is used to acquire the data to be processed, the correct data processing result of the data to be processed, and the initial neural network model for executing the target task
  • the initial neural network model includes a first transformer layer, and the first transformer layer It includes a first residual branch and a second residual branch
  • the first residual branch includes a first attention head
  • the second residual branch includes a target feedforward layer FFN
  • the model is used to input the data to be processed and/or the output of the first attention head and the identification of the target task into the first initial neural network, and compare the output of the first initial neural network with the The output of the first attention head is subjected to target operation to obtain the output of the first residual branch, and/or the initial neural network model is used for the output of the data to be processed and/or the target FFN , and the identification of the target task is input to the second initial neural network, and the output of the second initial neural network and the output of the target FFN are carried out target operation to obtain the output of the second residual branch;
  • a data processing module configured to process the data to be processed according to the initial neural network model to obtain a data processing result
  • a model updating module configured to obtain a loss according to the data processing result and the correct data processing result, and update the first initial neural network and/or the second initial neural network based on the loss to obtain a target neural network Model.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the model updating module is configured to obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the ith based on the loss an initial neural network and/or the second initial neural network to obtain a first neural network model, in the process of the i+1th iteration, update the first neural network model based on the loss except the first neural network model The initial neural network and/or the network parameters of the second initial neural network to obtain the target neural network model.
  • the present application provides a data processing device, the device comprising:
  • the acquisition module is used to acquire the data to be processed, the correct data processing result of the data to be processed, and the initial neural network model for performing the target task
  • the initial neural network model includes a first transformer layer and a second transformer layer, so
  • the first transformer layer includes a first attention head and a target FFN;
  • the initial neural network model is used to perform the first operation on the output of the first attention head and the first weight value to obtain the first output, and the A second operation is performed on the first output and the output of the second transformer layer; and/or the initial neural network model is used to perform a first operation on the target FFN and the second weight value to obtain the first operation.
  • a data processing module configured to process the data to be processed according to the target neural network model to obtain a data processing result
  • a model updating module configured to obtain a loss according to the data processing result and the correct data processing result, and update the first initial weight value and/or the second initial weight value based on the loss to obtain a target neural network Model.
  • the initial neural network model includes multiple transformer layers and an output layer
  • the second transformer layer is a transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the model updating module is configured to obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the ith based on the loss an initial weight value and/or the second initial weight value to obtain a first neural network model, and in the process of the i+1 th iteration, update the first neural network model based on the loss except for the first neural network model.
  • the initial weight value and/or the network parameters of the second initial weight value to obtain the target neural network model.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the present application provides a data processing device, the device comprising:
  • the acquisition module is used to acquire the data to be processed, the correct data processing result of the data to be processed, and the initial neural network model for performing the target task
  • the initial neural network model includes a first transformer layer and a second transformer layer, so
  • the first transformer layer includes a first attention head and a target FFN
  • the initial neural network model is used to input the data to be processed and/or the output of the first attention head and the identification of the target task to the first initial neural network, perform a first operation on the output of the first initial neural network and the output of the first attention head, and perform a second operation on the result of the operation and the output of the second transformer layer
  • the initial neural network model is used to input the data to be processed and/or the output of the target FFN and the identification of the target task into the second initial neural network, and the second initial The output of the neural network is subjected to a first operation with the output of the target FFN, and the second operation is performed between the operation result and the output of the second transformer layer;
  • a data processing module configured to process the data to be processed according to the target neural network model to obtain a data processing result
  • a model updating module configured to obtain a loss according to the data processing result and the correct data processing result, and update the first initial neural network and/or the second initial neural network based on the loss to obtain a target neural network Model.
  • the initial neural network model includes multiple transformer layers and an output layer
  • the second transformer layer is a transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the model updating module is configured to obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the ith based on the loss an initial neural network and/or the second initial neural network to obtain a first neural network model, in the process of the i+1th iteration, update the first neural network model based on the loss except the first neural network model The initial neural network and/or the network parameters of the second initial neural network to obtain the target neural network model.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • an embodiment of the present application provides an execution device, which may include a memory, a processor, and a bus system, wherein the memory is used for storing a program, and the processor is used for executing the program in the memory, so as to execute the above-mentioned first aspect and any optional method thereof, the second aspect and any optional method thereof.
  • an embodiment of the present application provides a training device, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory, so as to execute the above-mentioned third to The sixth aspect and any optional method thereof.
  • an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when it runs on a computer, causes the computer to execute the above-mentioned first to sixth aspects and any of its optional methods.
  • an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first to sixth aspects and any optional method thereof.
  • the present application provides a chip system
  • the chip system includes a processor for supporting an execution device or a training device to implement the functions involved in the above aspects, for example, sending or processing the data involved in the above methods ; or, information.
  • the chip system further includes a memory for storing program instructions and data necessary for executing the device or training the device.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • An embodiment of the present application provides a data processing method, including: acquiring data to be processed and a target neural network model, where the target neural network model includes a first transformer layer, and the first transformer layer includes a first residual branch and a second residual branch, the first residual branch includes a first attention head, and the second residual branch includes a target feedforward layer FFN; obtain the weight value corresponding to the target task, the The weight value includes the first weight value corresponding to the first attention head and/or the second weight value corresponding to the target FFN; according to the target neural network model, the data to be processed is performed on the target task-related.
  • the target neural network model is used to perform target operation on the output of the first attention head and the first weight value to obtain the output of the first residual branch
  • the target neural network model is configured to perform target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch.
  • the weight value used to control the output of the residual branch is set, which is equivalent to setting a set of exclusive distributed representation weight value combination for each task, thus realizing the multi-function of the target neural network model.
  • Task learning and in this embodiment, compared with adding neural networks adapted to different tasks in the output layer, fewer parameters need to be learned, thereby reducing the computing resource requirements for the terminal device to run the target neural network model.
  • Fig. 1 is a kind of structural schematic diagram of artificial intelligence main frame
  • Figure 2 is a natural language processing system
  • Fig. 3 is another natural language processing system
  • FIG. 4 is a schematic diagram of a related device for natural language processing provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of the architecture of a transformer layer
  • FIG. 6 is a schematic diagram of an embodiment of a data processing method provided by an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a neural network model in an embodiment of the application.
  • FIG. 8 is a schematic diagram of the structure of a transformer layer
  • FIG. 9 is a schematic diagram of the operation of an attention head
  • FIG. 10 is a schematic structural diagram of a neural network model provided by an embodiment of the application.
  • FIG. 11 is a schematic structural diagram of a neural network model provided by an embodiment of the application.
  • FIG. 12 is a schematic structural diagram of a neural network model provided by an embodiment of the application.
  • FIG. 13 is a schematic diagram of an embodiment of a data processing method provided by an embodiment of the present application.
  • FIG. 14 is a schematic structural diagram of a neural network model provided by an embodiment of the application.
  • 15 is a schematic structural diagram of a data processing device provided by an embodiment of the present application.
  • 16 is a schematic structural diagram of a data processing device provided by an embodiment of the present application.
  • FIG. 17 is a schematic structural diagram of a data processing device provided by an embodiment of the application.
  • FIG. 18 is a schematic structural diagram of a data processing device provided by an embodiment of the application.
  • FIG. 19 is a schematic structural diagram of a data processing device provided by an embodiment of the application.
  • FIG. 20 is a schematic structural diagram of a data processing device provided by an embodiment of the application.
  • FIG. 21 is a schematic structural diagram of an execution device provided by an embodiment of the application.
  • 22 is a schematic structural diagram of a training device provided by an embodiment of the present application.
  • FIG. 23 is a schematic structural diagram of a chip provided by an embodiment of the present application.
  • Figure 1 shows a schematic structural diagram of the main frame of artificial intelligence.
  • the above-mentioned artificial intelligence theme framework is explained in two dimensions (vertical axis).
  • the "intelligent information chain” reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data has gone through the process of "data-information-knowledge-wisdom".
  • the "IT value chain” reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system.
  • the infrastructure provides computing power support for artificial intelligence systems, realizes communication with the outside world, and supports through the basic platform. Communication with the outside world through sensors; computing power is provided by smart chips (hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA); the basic platform includes distributed computing framework and network-related platform guarantee and support, which can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with external parties to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.
  • smart chips hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA
  • the basic platform includes distributed computing framework and network-related platform guarantee and support, which can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with external parties to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.
  • the data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence.
  • the data involves graphics, images, voice, and text, as well as IoT data from traditional devices, including business data from existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.
  • Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.
  • machine learning and deep learning can perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc. on data.
  • Reasoning refers to the process of simulating human's intelligent reasoning method in a computer or intelligent system, using formalized information to carry out machine thinking and solving problems according to the reasoning control strategy, and the typical function is search and matching.
  • Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.
  • some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, image identification, etc.
  • Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall solution of artificial intelligence, and the productization of intelligent information decision-making to achieve landing applications. Its application areas mainly include: intelligent terminals, intelligent transportation, Smart healthcare, autonomous driving, safe city, etc.
  • This application can be applied to the field of natural language processing in the field of artificial intelligence.
  • the following will introduce multiple application scenarios that are applied to products.
  • FIG. 2 shows a natural language processing system, which includes user equipment and data processing equipment.
  • the user equipment includes smart terminals such as mobile phones, personal computers, or information processing centers.
  • the user equipment is the initiator of natural language data processing, and as the initiator of requests such as language question and answer or query, the user usually initiates the request through the user equipment.
  • the above-mentioned data processing device may be a device or server with data processing functions, such as a cloud server, a network server, an application server, and a management server.
  • the data processing equipment receives the query sentences/voice/text and other questions from the intelligent terminal through the interactive interface, and then performs machine learning, deep learning, search, reasoning, decision-making and other languages through the memory for storing data and the processor for data processing. Data processing, and feedback the processing results to the user equipment.
  • the memory in the data processing device may be a general term, including local storage and a database for storing historical data.
  • the database may be on the data processing device or on other network servers.
  • the user equipment can receive an instruction from the user, for example, the user equipment can receive a piece of text input by the user, and then initiate a request to the data processing device, so that the data processing device can target the segment obtained by the user equipment.
  • the text executes natural language processing applications (such as text classification, text reasoning, named entity recognition, translation, etc.), so as to obtain the processing results of the corresponding natural language processing applications (such as classification results, inference results, named entity recognition results for this piece of text) , translation results, etc.).
  • natural language processing applications such as text classification, text reasoning, named entity recognition, translation, etc.
  • the user equipment may receive a segment of Chinese input by the user, and then initiate a request to the data processing device, so that the data processing device performs entity classification on the segment of Chinese, thereby obtaining an entity classification result for the segment of Chinese;
  • the device may receive a segment of Chinese input by the user, and then initiate a request to the data processing device, so that the data processing device translates the segment of Chinese into English, thereby obtaining an English translation for the segment of Chinese.
  • the data processing device may receive, through an interactive interface, a request from a user device to obtain a natural language processing (NLP) related task model and performance upper limit parameters, including but not limited to: accuracy, delay, model At least one of the compression ratio parameters.
  • NLP natural language processing
  • the data processing device can calculate the size of the model suitable for the user equipment according to the already trained scalable transformer model and the performance upper limit parameters uploaded by the user equipment to be satisfied, and then extract the size of the model suitable for the user equipment when the size of the model is satisfied.
  • the sub-network is transmitted back to the user equipment.
  • the data processing device may execute the data processing method of the embodiment of the present application.
  • Fig. 3 shows another natural language processing system.
  • the user equipment is directly used as a data processing device.
  • the user equipment can directly receive the input from the user and process it directly by the hardware of the user equipment itself.
  • the specific process is the same as Similar to FIG. 2 , reference may be made to the above description, which will not be repeated here.
  • the user equipment can receive instructions from the user.
  • the user equipment can receive a piece of text input by the user, and then the user equipment can execute a natural language processing application (such as text classification) for the piece of text. , text reasoning, named entity recognition, translation, etc.), so as to obtain the corresponding natural language processing application processing results (such as classification results, inference results, named entity recognition results, translation results, etc.) for the piece of text.
  • a natural language processing application such as text classification
  • the user equipment may receive a segment of Chinese input by the user, and perform entity classification for the segment of Chinese, thereby obtaining an entity classification result for the segment of Chinese;
  • the Chinese paragraph is translated into English, thereby obtaining an English translation for the Chinese paragraph.
  • the user equipment may store a sub-network model, and before each operating system (operating system, OS) or application (application, APP) calls the model, the user equipment and not limited to at least one of the current power consumption, computing power, and storage parameters of the terminal device), calculate the size of the model that satisfies the current resource situation of the user equipment, and input the size of the suitable model into the stored
  • the sub-network model obtains the current state model after dynamic tailoring, and performs inference tasks.
  • the user equipment itself can execute the data processing method of the embodiment of the present application.
  • FIG. 4 is a schematic diagram of a related device 300 for natural language processing provided by an embodiment of the present application.
  • the user equipment in FIG. 2 and FIG. 3 may be the local device 301 or the local device 302 in FIG. 4 specifically, the data processing device in FIG. 2 may be the execution device 310 in FIG. 4 specifically, and the data storage system 350 may be To store the data to be processed by the execution device 310, the data storage system 350 may be integrated on the execution device 310, or may be set on the cloud or other network servers.
  • the processors in Figures 2 and 3 can perform data training/machine learning/deep learning through a neural network model or other models, and use the data finally trained or learned to perform natural language processing applications (such as text classification, sequence labeling, reading comprehension, text generation, text reasoning, translation, etc.) to obtain corresponding processing results.
  • natural language processing applications such as text classification, sequence labeling, reading comprehension, text generation, text reasoning, translation, etc.
  • a neural network can be composed of neural units, and a neural unit can refer to an operation unit that takes xs and intercept 1 as inputs, and the output of the operation unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • Ws is the weight of xs
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting a plurality of the above single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be an area composed of several neural units.
  • FIG. 5 is a schematic diagram of the architecture of a transformer layer.
  • the neural network includes an embedding layer and at least one transformer layer, and at least one transformer layer can be N transformer layers (N is an integer greater than 0), Among them, each transformer layer includes successively adjacent attention layers, summation and normalization (add&norm) layers, feedforward (feed forward) layers, and summation and normalization layers.
  • the current input is embedded to obtain multiple feature vectors; in the attention layer, P input vectors are obtained from the upper layer of the first transformer layer, and any one of the P input vectors is obtained.
  • the first input vector is the center, and based on the degree of association between each input vector within the preset attention window and the first input vector, an intermediate vector corresponding to the first input vector is obtained, and P input vectors are determined in this way.
  • the corresponding P intermediate vectors; in the pooling layer, the P intermediate vectors are combined into Q output vectors, wherein the multiple output vectors obtained by the last transformer layer in the transformer layer are used as the features of the current input express.
  • the current input is embedded to obtain multiple feature vectors.
  • the embedding layer can be referred to as the input embedding layer.
  • the current input can be a text input, such as a piece of text or a sentence.
  • the text can be Chinese text, English text, or other language text.
  • the embedding layer After the embedding layer obtains the current input, it can perform embedding processing on each word in the current input to obtain the feature vector of each word.
  • the embedding layer includes an input embedding layer and a positional encoding layer. In the input embedding layer, word embedding processing can be performed on each word in the current input, so as to obtain the word embedding vector of each word.
  • the position of each word in the current input can be obtained, and then a position vector can be generated for the position of each word.
  • the position of each word may be the absolute position of each word in the current input. Taking the current input as "a few numbers should be paid back" as an example, the position of "a few” can be expressed as the first place, the position of "number” can be expressed as the second place, . . . In some examples, the positions of the respective words may be relative positions between the respective words. Still taking the current input as "a few days should be paid back" as an example, the position of "a few” can be expressed as before “number”, and the position of "number” can be expressed as after "a few", before “should”, ...
  • each word feature vector that is, multiple feature vectors corresponding to the current input are obtained.
  • Multiple eigenvectors can be represented as embedding matrices with preset dimensions. The number of eigenvectors in the plurality of eigenvectors may be set to M, and the preset dimension is H, then the plurality of eigenvectors may be represented as an M ⁇ H embedding matrix.
  • the attention layer can also be referred to as a multi-head attention layer.
  • the attention layer can be a fixed window multi-head attention layer.
  • the first transformer layer may be the next layer of the above-mentioned embedding layer, and the P input vectors are the plurality of feature vectors obtained from the embedding layer.
  • at least one transformer layer in the neural network provided by the embodiments of this specification further includes a second transformer layer.
  • the second transformer layer is the upper layer of the first self-attention, and the P input vectors are the P output vectors output by the second transformer layer.
  • multiple output vectors from the above steps can be used as feature representations for the current input.
  • the feature representation is a feature representation suitable for computer processing for the current input, and can be used for tasks such as text similarity, text classification, reading comprehension, and machine translation.
  • the attention mechanism imitates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external sense to increase the fineness of observation in some areas, and can use limited attention resources to quickly screen out high-value information from a large amount of information. .
  • Attention mechanism can quickly extract important features from sparse data, so it is widely used in natural language processing tasks, especially machine translation.
  • the self-attention mechanism is an improvement of the attention mechanism, which reduces the dependence on external information and is better at capturing the internal correlation of data or features.
  • the essential idea of the attention mechanism can be rewritten as the following formula:
  • Lx
  • represents the length of Source.
  • the meaning of the formula is to imagine that the constituent elements in Source are composed of a series of data pairs. At this time, given an element Query in the target Target, by calculating the Query and The similarity or correlation of each Key, the weight coefficient of the Value corresponding to each Key is obtained, and then the weighted sum of the Value is obtained, that is, the final Attention value is obtained. So in essence, the Attention mechanism is to weight and sum the Value values of the elements in the Source, and Query and Key are used to calculate the weight coefficient of the corresponding Value.
  • Attention can be understood as selectively screening out a small amount of important information from a large amount of information and focusing on these important information, ignoring most of the unimportant information.
  • the process of focusing is reflected in the calculation of the weight coefficient.
  • the self-attention mechanism can be understood as internal Attention (intra attention).
  • the Attention mechanism occurs between the element Query of the Target and all elements in the Source.
  • the self-attention mechanism refers to the internal elements of the Source or between the internal elements of the Target.
  • NLP Natural language processing
  • Natural language is human language
  • natural language processing is the processing of human language.
  • Natural language processing is the process of systematically analyzing, understanding, and extracting information from text data in an intelligent and efficient manner.
  • NLP natural language processing
  • NER Named entity recognition
  • RE relation extraction
  • IE information extraction
  • sentiment analysis speech recognition
  • speech recognition question answering
  • topic segmentation etc.
  • natural language processing tasks can fall into the following categories.
  • Sequence tagging Each word in a sentence requires the model to give a categorical category based on the context. Such as Chinese word segmentation, part-of-speech tagging, named entity recognition, semantic role tagging.
  • Classification tasks output a classification value for the entire sentence, such as text classification.
  • Sentence relationship inference Given two sentences, determine whether the two sentences have a nominal relationship. For example, enlightenment, QA, semantic rewriting, natural language inference.
  • Generative task output a piece of text, generate another piece of text.
  • Word segmentation (word segmentation or word breaker, WB): The continuous natural language text is divided into lexical sequences with semantic rationality and completeness, which can solve the problem of cross ambiguity. For example: to graduates and students who have not yet graduated; participle 1: to graduates, and, students who have not graduated; participle 2: to graduates, monks, students who have not graduated.
  • NER Named Entity Recognition
  • Part-speech tagging assigns a part of speech (noun, verb, adjective, etc.) to each word in natural language text; dependency parsing: automatically analyzes the syntactic components (subject, predicate, object, attributive, adverbial and complement, etc.), can solve the problem of structural ambiguity. Comment: You can enjoy the sunrise in the room; Ambiguity 1: The room is okay; Ambiguity 2: You can enjoy the sunrise; Part of speech: In the room (subject), you can (predicate), enjoy the sunrise (verb-object phrase).
  • Word embedding&semantic similarity vectorized representation of vocabulary, and the calculation of semantic similarity of vocabulary based on this, which can solve the similarity of vocabulary and language. For example: watermelon and (dumb melon/strawberry), which is closer?
  • Vectorized representation watermelon (0.1222,0.22333,..); similarity calculation: dummy (0.115) strawberry (0.325); vectorized representation: (-0.333,0.1223..)(0.333,0.3333,..).
  • Text semantic similarity Relying on the massive data of the whole network and deep neural network technology, the ability to realize the calculation of semantic similarity between texts can solve the problem of text semantic similarity. For example: how to prevent the license plate from the front of the car and (how to install the front license plate/how to apply for the Beijing license plate), which is closer?
  • Vectorized representation how to prevent the front of the car from the license plate (0.1222,0.22333,..); similarity calculation: how to install the front license plate (0.762), how to apply for the Beijing license plate (0.486), vectorized representation: (-0.333,0.1223..)( 0.333, 0.3333, .. ).
  • the data processing methods provided by the embodiments of the present application relate to the processing of natural language texts, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning, and perform symbolic and formalized intelligent information modeling and extraction for training data. , preprocessing, training, etc., to finally obtain a trained target neural network model; and, the data processing method provided by the embodiment of the present application can use the above-mentioned trained target neural network model, input data (such as language information to be processed) input into the trained target neural network model to obtain output data (eg, processing results corresponding to the target task).
  • target neural network-related model training method and data processing method provided in the embodiments of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two parts of an overall process. Stages: such as model training stage and model application stage.
  • FIG. 6 is a schematic diagram of an embodiment of a data processing method provided by an embodiment of the present application.
  • a data processing method provided by the embodiment can be applied to mobile phones, tablets, notebook computers, smart wearable devices, etc.
  • a data processing method provided by an embodiment of the present application includes:
  • the target neural network model includes a first transformer layer
  • the first transformer layer includes a first residual branch and a second residual branch
  • the The first residual branch includes a first attention head
  • the second residual branch includes a target feedforward layer FFN.
  • the terminal device may acquire the data to be processed and the target neural network model, wherein the data to be processed may be text data, and the target neural network model may be a trained transformer model capable of multitasking.
  • the target neural network model may include the first transformer layer.
  • the target neural network model may be a neural network model based on the transformer layer, or alternatively, the target neural network model may be based on the transformer layer. the NLP model.
  • FIG. 7 is a schematic structural diagram of a neural network model in an embodiment of the application, and the neural network model shown in FIG. 7 may be a target neural network model in an embodiment of the application.
  • the target neural network model may include sequentially connected embedding layers and multiple transformer layers.
  • transformer models are mostly used to perform natural language processing NLP tasks. It should be understood that the structure of FIG. 7 is only an example, and the number of transformer layers can be set as required. For example, only one transformer layer can be set, or more transformer layers can be set.
  • the neural network model determines the feature vector corresponding to the current node based on the N output vectors obtained by each transformer layer.
  • the current input is embedded to obtain multiple feature vectors.
  • the core feature of the transformer model is the unique attention mechanism it adopts. When processing natural language, such as a sentence, the transformer model uses this attention mechanism to assign different attention coefficients to each word vector in the sentence, so as to more comprehensively consider the impact of the context in the sentence on each word.
  • the embedding layer obtains N embedding vectors X l based on the node features and position codes of each node in the current sequence.
  • the attention layer is connected to the embedding layer, and N embedding vectors are obtained from the embedding layer as input vectors.
  • each input vector is synthesized to obtain N output vectors, which are output to Subsequent transformer layers.
  • the transformer layer takes the output of the previous layer as an input vector and performs similar operations as the previous transformer layer.
  • FIG. 8 is a schematic diagram of the structure of a transformer layer.
  • the transformer layers of each neural network in the embodiment of the present application (for example, the first transformer layer and the second transformer layer in the embodiment) can be referred to as shown in FIG. 8 .
  • the resulting structure, as shown in Figure 8, the transformer layer includes successively adjacent multi-head attention layers, summation and normalization (add&norm) layers, feedforward (feed forward) layers, summation and normalization layers layer.
  • the multi-head attention layer obtains N input vectors X l from the upper layer, which can be expressed as a matrix X, and uses the self-attention mechanism to transform each vector based on the correlation between the vectors to obtain N output vectors, It can also be represented as a matrix Y.
  • the multi-head attention layer is a layer directly connected to the embedding layer, such as the transformer layer directly connected to the embedding layer in Figure 7, the obtained input vector is the embedding vector output by the embedding layer; when the multi-head attention layer is directly connected to the embedding layer
  • the layer is the multi-head attention layer included in the subsequent transformer layer.
  • the multi-head attention layer included in the transformer layer directly connected to the previous transformer layer in Figure 7 the input vector obtained is the output vector of the previous transformer layer.
  • the multi-head attention (MHA) based MHA layer includes multiple attention heads (Head 1, Head 2, ..., Head N as shown in Fig. 8).
  • Figure 9 is a schematic diagram of the operation of an attention head, which shows how the attention head transforms the input matrix X into the output matrix Y.
  • the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V are respectively used to transform each input vector Xi in the N input vectors ⁇ X1, X2,..., XN> to obtain each input vector
  • the vectors correspond to the first intermediate vector (q vector), the second intermediate vector (k vector) and the third intermediate vector (v vector).
  • the first transformation matrix Q, the second transformation matrix K and the third transformation matrix V can be used to linearly transform the input matrix X composed of N input vectors to obtain the Q matrix, K matrix and V of the input matrix respectively.
  • the dot product result of qi and kj can also be directly determined as the correlation degree, more classically, the dot multiplication result is divided by a constant, and then the softmax operation is performed, and the operation result is used as the correlation degree between the input vectors Xi and Xj, which is:
  • each correlation degree ⁇ i,j of the ith input vector Xi and each input vector Xj can be used as a weighting factor, and the third intermediate vector (v vector, vj) corresponding to each input vector Xj can be weighted and combined to obtain the ith
  • the vector sequence ⁇ C1, C2, . . . , CN> of N combined vectors corresponding to the N input vectors, or the matrix C can be obtained.
  • N output vectors can be obtained.
  • the output matrix Y is the combined vector matrix C, which can be written as:
  • the MHA layer maintains m sets of transformation matrices, and each set of transformation matrices includes the aforementioned first transformation matrix Q, second transformation matrix K and third transformation matrix V, thus
  • the above operations can be performed in parallel to obtain m combined vector sequences (ie, m matrices C), each vector sequence including N combined vectors obtained based on a set of transformation matrices.
  • the MHA layer splices the obtained m combination vector sequences to obtain a splicing matrix; and then transforms the splicing matrix through the fourth transformation matrix W to obtain the final output matrix Y.
  • Splitting the output matrix Y corresponds to N output vectors ⁇ Y1, Y2,..., YN>.
  • the MHA layer performs a transformation operation based on the correlation between N input vectors to obtain N output vectors.
  • the branch where the multi-head attention layer is located and the branch where the feedforward layer FFN is located may be the residual branch in the transformer layer.
  • the first transformer layer in this embodiment includes a first residual branch and a second residual branch, the first residual branch includes a first attention head, and the second residual branch includes Target feedforward layer FFN.
  • the first transformer layer may include a first residual branch and a second residual branch, The first residual branch is the branch where the first attention head is located, and the second residual branch is the branch where the target FFN is located.
  • the target neural network model may include multiple task layers at the output position, and each task layer is adapted to different tasks.
  • weight value corresponding to the target task where the weight value includes a first weight value corresponding to the first attention head and/or a second weight value corresponding to the target FFN.
  • different weight values can be determined to operate with the output of the attention head in the transformer layer or the output of the FFN to control the output of the residual branch, the target neural network
  • different weights can be selected, so that the output of the residual branch is more suitable for the task performed by the current target neural network model.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the weight value may include a first weight value corresponding to the first attention head
  • the weight value may include a second weight value corresponding to the target FFN
  • the weight value may include a first weight value corresponding to the first attention head and a second weight value corresponding to the target FFN.
  • the weight value corresponding to the target task may be obtained according to a preset mapping relationship, and the target weight value includes the first weight value and/or the second weight value; wherein the predetermined weight value is
  • the set mapping relationship includes a corresponding relationship between tasks and weight values, and in the preset mapping relationship, the target task corresponds to the target weight value.
  • the above-mentioned preset mapping relationship is obtained when the target neural network model is trained for the target task.
  • the target task corresponds to the first weight value.
  • the target task corresponds to the second weight value.
  • the target task corresponds to the first weight value and the second weight value.
  • the first weight value may be obtained by updating the first initial weight value when the target neural network model is trained for the target task.
  • the target neural network model is used to perform target operation on the output of the first attention head and the first initial weight value to obtain the output of the first residual branch.
  • the second weight value may be obtained by updating the second initial weight value when the target neural network model is trained for the target task.
  • the target neural network model is used to perform target operation on the output of the target FFN and the second initial weight value to obtain the output of the second residual branch.
  • the training device can obtain the data to be processed, the correct data processing result of the data to be processed, and the initial neural network model for performing the target task, the initial neural network model includes the first transformer layer, and the first transformer layer includes the first transformer layer.
  • a residual branch and a second residual branch the first residual branch includes a first attention head, the second residual branch includes a target feedforward layer FFN, and the initial neural network model uses The target operation is performed on the output of the first attention head and the first initial weight value to obtain the output of the first residual branch, and/or the initial neural network model is used to use the target FFN Perform target operation on the output and the second initial weight value to obtain the output of the second residual branch; process the data to be processed according to the initial neural network model to obtain a data processing result; Obtaining the loss from the data processing result and the correct data processing result, and updating the first initial weight value and/or the second initial weight value based on the loss, until the data processing accuracy of the initial neural network model for the target task meets the requirements, In this way, the first weight value and
  • different first weight values and second weight values can be obtained for different tasks, and then the preset mapping relationship in the above embodiment can be obtained.
  • the training device may obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the first initial weight value and/or the first initial weight value based on the loss
  • the second initial weight value is used to obtain the first neural network model, and in the process of the i+1th iteration, the first neural network model is updated based on the loss except for the first initial weight value and/or all
  • the network parameters of the second initial weight value are described to obtain the target neural network model.
  • the skeleton model (the skeleton model can be understood as the part of the network except the weight value in the target neural network model) can be fixed in one iteration (wherein "fixed” can be understood as maintaining the structure and parameter size of the network unchanged), and update the weight value, then in the next iteration, fix the weight value and update the skeleton model (part of the target neural network model except for the weight value).
  • training sets of multiple tasks can be obtained, wherein the training set of the t-th task is D t , the pre-trained BERT model (the parameter is w) is completed, the first initial weight value and/or the second initial weight value are initialized The initial weight value (the parameter is the gating module parameter ⁇ ).
  • loss function for each task t The specific iterative steps are as follows: first randomly select the task id t, and randomly sample a batch of training data from D t ; then fix the skeleton model, according to the loss function Update the gating module parameter ⁇ ; then fix the gating module, according to the loss function Update the skeleton network parameter w.
  • the first initial weight value and the second initial weight value in the above embodiment may be values obtained after the learnable scalar is processed by sigmoid or other functions.
  • the first initial weight value may be Among them, ⁇ t, l is a learnable scalar, t represents the target task, and l can represent the first attention head. That is to say, in the training process, what is updated is a learnable scalar, and then the first initial weight value and the second initial weight value are also updated accordingly.
  • the learnable scalar can be initialized to some constant, such as 0.5.
  • the above preset mapping relationship may be a discrete correspondence (one task corresponds to one weight value), or may be represented by some linear or nonlinear models, which is not limited in this application.
  • At least one of the data to be processed and the output of the first attention head, and the identifier of the target task may be input into the first neural network to obtain the first weight value and/or, input at least one of the data to be processed and the output of the target FFN, and the identifier of the target task into the second neural network to obtain the second weight value.
  • the data to be processed, the correct data processing result of the data to be processed, and the initial neural network model for executing the target task can be obtained, where the initial neural network model includes a first transformer layer, the first The transformer layer includes a first residual branch and a second residual branch, the first residual branch includes a first attention head, the second residual branch includes a target feedforward layer FFN, the initial The neural network model is used to input at least one of the data to be processed and the output of the first attention head, and the identification of the target task into the first initial neural network, and the first initial neural network The output of the network is subjected to target operation with the output of the first attention head to obtain the output of the first residual branch, and/or the initial neural network model is used to combine the data to be processed and the target At least one of the outputs of the FFN and the identifier of the target task are input into the second initial neural network, and the output of the second initial neural network and the output of the target FFN are subjected to target operation to obtain the first The output
  • the first neural network has the capability of inputting at least one of the data to be processed and the output of the first attention head, as well as the identifier of the target task, and outputting a first weight value.
  • the second neural network has the capability of inputting the data to be processed and/or the output of the target FFN and the identifier of the target task into the second neural network to obtain the second weight value.
  • different first neural networks and second neural networks can be obtained by training.
  • the above-mentioned first neural network and second neural network can be, but are not limited to, fully connected networks.
  • the initial gate value of the target can be set, and then the gradient descent method can be used to update the model, and other optimization methods such as reinforcement learning can also be used. Algorithms and Genetic Algorithms for training.
  • the target neural network model is used to target the output of the first attention head and the first weight value. operation to obtain the output of the first residual branch, and/or the target neural network model is used to perform target operation on the output of the target FFN and the second weight value to obtain the second residual branch Output.
  • the terminal device may perform the target task-related processing on the data to be processed according to the target neural network model, so as to obtain a data processing result, wherein the target neural network model is used to The output of an attention head is subjected to target operation with the first weight value to obtain the output of the first residual branch, and/or the target neural network model is used to combine the output of the target FFN with the A target operation is performed on the second weight value to obtain the output of the second residual branch.
  • the target operation may be a product operation.
  • the target neural network model can perform a product operation on the output of the first attention head and the first weight value (*first weight value) to obtain the output of the first residual branch, and
  • the output of the first residual branch is input to the summation and normalization layer
  • the target neural network model can perform a product operation on the output of the target FFN and the second weight value (*second weight value) , so as to obtain the output of the second residual branch, and input the output of the second residual branch to the summation and normalization layer.
  • the combination of the weight values of each residual branch can make the target neural network model adapt to different tasks.
  • a mechanism equivalent to a gating module is added to the output of the first attention head and/or FFN to control the output size of the residual branch in the transformer layer, and the gating mechanism can learn for each task A unique combination of weight values (either the first neural network and/or the second neural network).
  • the first weight value and the second weight value may be 1 or 0.
  • the model can only open the bottom gate (that is, set the weight value of the residual branch in the transformer layer close to the embedding layer to 1, and set the weight value of the residual branch close to the output layer to 1.
  • the weight value of the residual branch in the transformer layer is set to 0); for tasks that rely on grammatical features, such as syntactic analysis, the model can open the door of the middle layer (that is, the transformer that is neither close to the embedding layer nor the output layer).
  • the weight value of the residual branch in the layer is set to 1, and the weight value of the residual branch in the transformer layer near the embedding layer and near the output layer is set to 0); for more complex tasks, such as reading comprehension, the model Doors on all floors can be opened.
  • the first transformer layer includes a plurality of attention heads, each attention head in the plurality of attention heads corresponds to a weight value, and the target neural network model is used to convert the each attention head The output of one attention head and the corresponding weight value are subjected to target operation to obtain the output of the first residual branch, wherein the weight values corresponding to different attention heads are different.
  • the plurality of attention heads may include the first attention head and the second attention head, and the first attention head and the second attention head are any of the plurality of attention heads.
  • the target neural network model is further configured to perform target operation on the output of the second attention head and a third weight value, and the first weight value is different from the third weight value.
  • the target neural network model may include a multi-head attention layer, and for each task t, each attention head n may deploy a weight value of the attention head to control the output of the residual branch where it belongs size.
  • the plurality of attention heads include the first attention head and the second attention head, and the target neural network model is further configured to combine the output of the second attention head with a third weight Values are multiplied.
  • the target neural network model when the target neural network model is used to use the output of the first attention head as the output of the first residual branch, the target neural network model is used for the The data processing accuracy of the target task is less than the first processing accuracy, and the first processing accuracy is that the target neural network model is used to perform target operation on the output of the first attention head and the first weight value, to obtain the In the case of the output of the first residual branch, the data processing accuracy of the target neural network model; or,
  • the data processing accuracy of the target neural network model for the target task is smaller than the second Processing accuracy
  • the second processing accuracy is when the target neural network model is used to perform target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch, the data processing accuracy of the target neural network model needle;
  • the data processing accuracy of the target neural network model for the target task is less than a third processing accuracy, where the third processing accuracy is used in the target neural network model for comparing the output of the target FFN with the second weight value Perform a target operation to obtain the output of the second residual branch, and perform a target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch, the The data processing accuracy of the target neural network model needle.
  • the residual branch of the original transformer layer is added to the The weight control strategy, and different weight values are set for different tasks.
  • the target neural network model includes a first transformer layer, and the first transformer layer includes a first residual branch and a second residual branch
  • the first residual branch includes the first attention head
  • the second residual branch includes the target feedforward layer FFN
  • the weight value corresponding to the target task is obtained, and the weight value includes the first attention
  • the data to be processed is processed related to the target task to obtain a data processing result
  • the target neural network model is used to perform target operation on the output of the first attention head and the first weight value to obtain the output of the first residual branch, and/or the target neural network
  • the model is used to perform target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch.
  • the weight value used to control the output of the residual branch is set, which is equivalent to setting a set of exclusive distributed representation weight value combination for each task, thus realizing the multi-function of the target neural network model.
  • Task learning and in this embodiment, compared with adding neural networks adapted to different tasks in the output layer, fewer parameters need to be learned, thereby reducing the computing resource requirements for the terminal device to run the target neural network model.
  • an embodiment of the present application further provides a data processing method, the method includes:
  • Acquire data to be processed and a target neural network model where the target neural network model includes a first transformer layer and a second transformer layer, and the first transformer layer includes a first attention head and a target FFN.
  • the target neural network model includes multiple transformer layers and an output layer
  • the second transformer layer is the transformer layer closest to the output layer among the multiple transformer layers. That is to say, the second transformer layer is the transformer layer farthest from the embedding layer among the multiple transformer layers.
  • the weight value includes a first weight value corresponding to the first attention head and/or a second weight value corresponding to the target FFN.
  • step 1302 may refer to the description of step 602, and the similarities will not be repeated.
  • the target neural network model Process the to-be-processed data according to the target neural network model to obtain a data processing result, wherein the target neural network model is used to perform a first step between the output of the first attention head and the first weight value. performing an operation to obtain a first output, and performing a second operation on the first output and the output of the second transformer layer; and/or, the target neural network model is used to combine the target FFN with the first output Perform a first operation on two weight values to obtain a second output, and perform a second operation on the second output and the output of the second transformer layer.
  • the target neural network model is used to perform the first operation on the output of the first attention head and the first weight value to obtain the first output, and the first output is not used as the first output.
  • the output of the first residual branch but a second operation is performed between the first output and the output of the second transformer layer.
  • the target neural network model is used to perform a first operation on the target FFN and the second weight value to obtain a second output.
  • the second output is not used as the output of the second residual branch, but A second operation is performed between the second output and the output of the second transformer layer.
  • the first operation may include a product operation, and/or the second operation may include an addition operation.
  • the target neural network model is used to multiply the output of the first attention head with the first weight value (*first weight value) to obtain the first output, and the first output Perform a sum operation with the output of the second transformer layer; and/or, the target neural network model is used to perform a product operation (*second weight value) on the target FFN and the second weight value to obtain a second output, and performing a sum operation on the second output and the output of the second transformer layer.
  • the target neural network model can be as in the embodiment corresponding to FIG. 6 .
  • the output of the first attention head and the first weight value are subjected to target operation (for example, a product operation), and the operation result is calculated.
  • target operation for example, a product operation
  • the first output and the output of the second transformer layer are also added.
  • the target neural network model can be as in the embodiment corresponding to FIG. 6 .
  • the target FFN and the second weight value are subjected to a target operation (for example, a product operation), and the operation result is used as the second residual.
  • the output of the branch on the other hand, also adds the second output to the output of the second transformer layer.
  • the first transformer layer includes a first residual branch and a second residual branch, and the first residual branch includes the first attention head;
  • the target neural network model When the target neural network model is used to use the output of the first attention head only as the output of the first residual branch, the target neural network model is used for data processing of the target task
  • the accuracy is less than the first processing accuracy
  • the first processing accuracy is that the target neural network model is used for the target neural network model to perform the first operation on the output of the first attention head and the first weight value , the data processing accuracy of the target neural network model when the first output is obtained, and the third output and the output of the second transformer layer are subjected to the second operation; or,
  • the data processing accuracy of the target neural network model for the target task is less than the third Second processing precision
  • the second processing precision is the case where the target neural network model is used to perform the first operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch , the data processing accuracy of the target neural network model needle;
  • the output of the first attention head is only used as the output of the first residual branch and the output of the target FFN is used as the output of the second residual branch
  • the data processing accuracy of the target neural network model for the target task is less than a third processing accuracy
  • the third processing accuracy is when the target neural network model is used for the target neural network model.
  • the first operation is performed on the output of the first attention head and the first weight value to obtain the first output
  • the second operation is performed on the third output and the output of the second transformer layer
  • the The first operation is performed on the output of the target FFN and the second weight value to obtain the output of the second residual branch
  • the first operation is performed on the output of the target FFN and the second weight value to obtain the second
  • the data processing accuracy of the target neural network model needle is the case of the output of the residual branch.
  • the first weight value is obtained by updating the first initial weight value when the target neural network model is trained for the target task, wherein, when the target neural network model is trained for the target task
  • the target neural network model is used to perform a first operation on the output of the first attention head and the first initial weight value, and the operation result is compared with the output of the second transformer layer. Perform the second operation.
  • the second weight value is obtained by updating the second initial weight value when the target neural network model is trained for the target task.
  • the target neural network model is used to perform a first operation on the target FFN and the second initial weight value, and perform a second operation between the operation result and the output of the second transformer layer. operation.
  • the weight value corresponding to the target task is obtained according to a preset mapping relationship, and the target weight value includes the first weight value and/or the second weight value; wherein, the The preset mapping relationship includes a corresponding relationship between tasks and weight values, and in the preset mapping relationship, the target task corresponds to the target weight value.
  • the data to be processed, the correct data processing result of the data to be processed, and an initial neural network model for executing the target task are obtained, where the initial neural network model includes a first transformer layer and a second transformer layer, the first transformer layer includes the first attention head and the target FFN; the initial neural network model is used to perform the first operation on the output of the first attention head and the first weight value to obtain the first output , and perform a second operation on the first output and the output of the second transformer layer; and/or, the initial neural network model is used to perform a first operation on the target FFN and the second weight value , obtain a second output, and perform a second operation on the second output and the output of the second transformer layer; process the data to be processed according to the target neural network model to obtain a data processing result; From the data processing result and the correct data processing result, a loss is obtained, and the first initial weight value and/or the second initial weight value is updated based on the loss to obtain a target neural network model.
  • the training device may obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, only update the first initial weight value and the loss based on the loss /or the second initial weight value to obtain the first neural network model, in the process of the i+1th iteration, update the first neural network model based on the loss divided by the first initial weight value and/ or the network parameters of the second initial weight value to obtain the target neural network model.
  • the data to be processed and/or the output of the first attention head, and the identifier of the target task may be input into the first neural network to obtain the first weight value; And/or, the data to be processed and/or the output of the target FFN and the identifier of the target task are input into the second neural network to obtain the second weight value.
  • the first neural network is obtained by updating a first initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input the data to be processed and/or the output of the first attention head and the identification of the target task into the first initial neural network. network, performing a first operation on the output of the first initial neural network and the output of the first attention head, and performing a second operation on the operation result and the output of the second transformer layer.
  • the second neural network is obtained by updating the second initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input the data to be processed and/or the output of the target FFN and the identification of the target task into the second initial neural network, and A first operation is performed on the output of the second initial neural network and the output of the target FFN, and a second operation is performed on the operation result and the output of the second transformer layer.
  • the training device can acquire the data to be processed, the correct data processing result of the data to be processed, and an initial neural network model for performing the target task, the initial neural network model including the first transformer layer and The second transformer layer, the first transformer layer includes a first attention head and a target FFN; the initial neural network model is used to combine the data to be processed and/or the output of the first attention head, and all The identification of the target task is input into the first initial neural network, the output of the first initial neural network and the output of the first attention head are subjected to a first operation, and the operation result is compared with the second transformer layer.
  • the initial neural network model is used to input the data to be processed and/or the output of the target FFN and the identification of the target task into the second initial neural network, and Perform a first operation on the output of the second initial neural network and the output of the target FFN, and perform a second operation on the result of the operation and the output of the second transformer layer; processing the data to obtain a data processing result; obtaining a loss according to the data processing result and the correct data processing result, and updating the first initial neural network and/or the second initial neural network based on the loss, to get the target neural network model.
  • the training device may obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, only update the first initial neural network and the first initial neural network based on the loss. /or the second initial neural network to obtain a first neural network model, and in the process of the i+1th iteration, update the first neural network model based on the loss except for the first initial neural network and/or or the network parameters of the second initial neural network to obtain the target neural network model.
  • the first transformer layer includes multiple attention heads, the multiple attention heads include the first attention head and the second attention head, and correspondingly, the target neural
  • the network model is also used to perform a first operation on the output of the second attention head and the third weight value to obtain a third output, and perform the first operation on the third output and the output of the second transformer layer. A two operation, wherein the first weight value is different from the third weight value.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the embodiment of the present application also provides a data processing method, including:
  • the initial neural network model includes a first transformer layer, and the first transformer layer includes a first residual branch and a second residual branch, the first residual branch includes a first attention head, the second residual branch includes a target feedforward layer FFN, and the initial neural network model is used to The output of the first attention head and the first initial weight value are subjected to target operation to obtain the output of the first residual branch, and/or the initial neural network model is used to combine the output of the target FFN with the first Carry out target operation with two initial weight values to obtain the output of the second residual branch;
  • a loss is obtained, and the first initial weight value and/or the second initial weight value is updated based on the loss to obtain a target neural network model.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the loss is obtained according to the data processing result and the correct data processing result, and the first initial weight value and/or the second initial weight value is updated based on the loss, to Get the target neural network model, including:
  • the loss is obtained, and in the process of the ith iteration, only the first initial weight value and/or the second initial weight value is updated based on the loss, so as to Obtain the first neural network model, and in the process of the i+1th iteration, update the network parameters of the first neural network model except the first initial weight value and/or the second initial weight value based on the loss , to get the target neural network model.
  • the embodiment of the present application also provides a data processing method, including:
  • the initial neural network model includes a first transformer layer, and the first transformer layer includes a first residual branch and a second residual branch, the first residual branch includes a first attention head, the second residual branch includes a target feedforward layer FFN, and the initial neural network model is used to
  • the data to be processed and/or the output of the first attention head and the identification of the target task are input into the first initial neural network, and the output of the first initial neural network is combined with the first attention
  • the output of the head is subjected to target operation to obtain the output of the first residual branch, and/or the initial neural network model is used to use the data to be processed and/or the output of the target FFN and the target
  • the identification of the task is input into the second initial neural network, and the output of the second initial neural network and the output of the target FFN are subjected to target operation to obtain the output of the second residual branch;
  • a loss is obtained, and the first initial neural network and/or the second initial neural network is updated based on the loss to obtain a target neural network model.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the loss is obtained according to the data processing result and the correct data processing result, and the first initial neural network and/or the second initial neural network is updated based on the loss, to Get the target neural network model, including:
  • the loss is obtained, and in the process of the ith iteration, only the first initial neural network and/or the second initial neural network is updated based on the loss, so as to Obtain a first neural network model, and in the process of the i+1th iteration, update the network parameters of the first neural network model other than the first initial neural network and/or the second initial neural network based on the loss , to get the target neural network model.
  • the embodiment of the present application also provides a data processing method, including:
  • the initial neural network model includes a first transformer layer and a second transformer layer, the first transformer layer Including a first attention head and a target FFN; the initial neural network model is used to perform a first operation on the output of the first attention head and a first weight value to obtain a first output, and the first output Perform a second operation with the output of the second transformer layer; and/or, the initial neural network model is used to perform a first operation on the target FFN and the second weight value to obtain a second output, and use the performing a second operation on the second output with the output of the second transformer layer;
  • a loss is obtained, and the first initial weight value and/or the second initial weight value is updated based on the loss to obtain a target neural network model.
  • the initial neural network model includes multiple transformer layers and an output layer
  • the second transformer layer is a transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the loss is obtained according to the data processing result and the correct data processing result, and the first initial weight value and/or the second initial weight value is updated based on the loss, to Get the target neural network model, including:
  • the loss is obtained, and in the process of the ith iteration, only the first initial weight value and/or the second initial weight value is updated based on the loss, so as to Obtain the first neural network model, and in the process of the i+1th iteration, update the network parameters of the first neural network model except the first initial weight value and/or the second initial weight value based on the loss , to get the target neural network model.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the embodiment of the present application also provides a data processing method, including:
  • the initial neural network model includes a first transformer layer and a second transformer layer, the first transformer layer Including a first attention head and a target FFN; the initial neural network model is used to input the data to be processed and/or the output of the first attention head and the identification of the target task into the first initial neural network network, perform a first operation on the output of the first initial neural network and the output of the first attention head, and perform a second operation on the operation result and the output of the second transformer layer; and/or, all The initial neural network model is used to input the data to be processed and/or the output of the target FFN and the identification of the target task into the second initial neural network, and the output of the second initial neural network is combined with the output of the target FFN.
  • the first operation is performed on the output of the target FFN
  • the second operation is performed on the result of the operation and the output of the second transformer layer;
  • a loss is obtained, and the first initial neural network and/or the second initial neural network is updated based on the loss to obtain a target neural network model.
  • the initial neural network model includes multiple transformer layers and an output layer
  • the second transformer layer is a transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the loss is obtained according to the data processing result and the correct data processing result, and the first initial neural network and/or the second initial neural network is updated based on the loss, to Get the target neural network model, including:
  • the loss is obtained, and in the process of the ith iteration, only the first initial neural network and/or the second initial neural network is updated based on the loss, so as to Obtain a first neural network model, and in the process of the i+1th iteration, update the network parameters of the first neural network model other than the first initial neural network and/or the second initial neural network based on the loss , to get the target neural network model.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • FIG. 15 is a schematic structural diagram of a data processing device 1500 provided by an embodiment of the application.
  • the data processing device 1500 may be a terminal device or a server, and the data processing device 1500 includes:
  • the acquisition module 1501 is used to acquire the data to be processed and a target neural network model, the target neural network model includes a first transformer layer, and the first transformer layer includes a first residual branch and a second residual branch
  • the first residual branch includes the first attention head
  • the second residual branch includes the target feedforward layer FFN;
  • the weight value corresponding to the target task is obtained, and the weight value includes the first attention the first weight value corresponding to the force head and/or the second weight value corresponding to the target FFN;
  • the data processing module 1502 is used to process the data to be processed according to the target neural network model to obtain a data processing result, wherein the target neural network model is used to combine the output of the first attention head with the first A weight value is used to perform target operation to obtain the output of the first residual branch, and/or the target neural network model is used to perform target operation on the output of the target FFN and the second weight value to obtain the first The output of the two residual branches.
  • the target neural network model when used to use the output of the first attention head as the output of the first residual branch, the target neural network model is directed to The data processing precision of the target task is smaller than the first processing precision, and the first processing precision is that the target neural network model is used to perform the target operation on the output of the first attention head and the first weight value to obtain In the case of the output of the first residual branch, the data processing accuracy of the target neural network model; or,
  • the data processing accuracy of the target neural network model for the target task is smaller than the second Processing accuracy
  • the second processing accuracy is when the target neural network model is used to perform target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch, the data processing accuracy of the target neural network model needle;
  • the data processing accuracy of the target neural network model for the target task is less than a third processing accuracy, where the third processing accuracy is used in the target neural network model for comparing the output of the target FFN with the second weight value Perform a target operation to obtain the output of the second residual branch, and perform a target operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch, the The data processing accuracy of the target neural network model needle.
  • the first weight value is obtained by updating the first initial weight value when the target neural network model is trained for the target task, wherein, when the target neural network model is trained for the target task During the training process of the neural network model, the target neural network model is used to perform target operation on the output of the first attention head and the first initial weight value to obtain the output of the first residual branch.
  • the second weight value is obtained by updating the second initial weight value when the target neural network model is trained for the target task.
  • the target neural network model is used to perform target operation on the output of the target FFN and the second initial weight value to obtain the output of the second residual branch.
  • the obtaining module is used to:
  • the weight value corresponding to the target task is obtained according to a preset mapping relationship, and the weight value corresponding to the target task includes the first weight value and/or the second weight value; wherein the preset mapping relationship Including the correspondence between tasks and weight values.
  • the obtaining module is used to:
  • the first neural network is obtained by updating a first initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the first attention head, and the identification of the target task into the first an initial neural network, and perform target operation on the output of the first initial neural network and the output of the first attention head to obtain the output of the first residual branch.
  • the second neural network is obtained by updating the second initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the target FFN, and the identification of the target task into the second initial neural network. network, and perform target operation on the output of the second initial neural network and the output of the target FFN to obtain the output of the second residual branch.
  • the first transformer layer includes multiple attention heads, each attention head in the multiple attention heads corresponds to a weight value, and the target neural network model is used to The output of each attention head and the corresponding weight value are subjected to target operation to obtain the output of the first residual branch, wherein the weight values corresponding to different attention heads are different.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the specific description of the data processing device 1500 can refer to the description of the corresponding embodiment in FIG. 6 , wherein, the acquisition module 1501 can perform steps 601 and 602 and the descriptions in the corresponding embodiments, and the data processing module 1502 can perform step 603 and corresponding described in the examples.
  • FIG. 16 is a schematic structural diagram of a data processing device 1600 provided by an embodiment of the application.
  • the data processing device 1600 may be a terminal device or a server, and the data processing device 1600 includes:
  • the acquisition module 1601 is used to acquire the data to be processed and a target neural network model, the target neural network model includes a first transformer layer and a second transformer layer, and the first transformer layer includes a first attention head and a target FFN; obtain The weight value corresponding to the target task, the weight value includes the first weight value corresponding to the first attention head and/or the second weight value corresponding to the target FFN;
  • a data processing module 1602 configured to process the data to be processed according to the target neural network model to obtain a data processing result, wherein the target neural network model is used to compare the output of the first attention head with the first A first operation is performed on a weight value to obtain a first output, and a second operation is performed between the first output and the output of the second transformer layer; and/or, the target neural network model is used to convert the target A first operation is performed on the FFN and the second weight value to obtain a second output, and a second operation is performed on the second output and the output of the second transformer layer.
  • the target neural network model includes multiple transformer layers and an output layer, and the second transformer layer is the transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes a first residual branch and a second residual branch
  • the first residual branch includes the first attention head
  • the second residual branch includes the target FFN;
  • the target neural network model When the target neural network model is used to use the output of the first attention head only as the output of the first residual branch, the target neural network model is used for data processing of the target task
  • the accuracy is less than the first processing accuracy
  • the first processing accuracy is that the target neural network model is used for the target neural network model to perform the first operation on the output of the first attention head and the first weight value , the data processing accuracy of the target neural network model when the first output is obtained, and the third output and the output of the second transformer layer are subjected to the second operation; or,
  • the data processing accuracy of the target neural network model for the target task is less than the third Second processing precision
  • the second processing precision is the case where the target neural network model is used to perform the first operation on the output of the target FFN and the second weight value to obtain the output of the second residual branch , the data processing accuracy of the target neural network model needle;
  • the output of the first attention head is only used as the output of the first residual branch and the output of the target FFN is used as the output of the second residual branch
  • the data processing accuracy of the target neural network model for the target task is less than a third processing accuracy
  • the third processing accuracy is when the target neural network model is used for the target neural network model.
  • the first operation is performed on the output of the first attention head and the first weight value to obtain the first output
  • the second operation is performed on the third output and the output of the second transformer layer
  • the The first operation is performed on the output of the target FFN and the second weight value to obtain the output of the second residual branch
  • the first operation is performed on the output of the target FFN and the second weight value to obtain the second
  • the data processing accuracy of the target neural network model needle is the case of the output of the residual branch.
  • the first weight value is obtained by updating the first initial weight value when the target neural network model is trained for the target task, wherein, when the target neural network model is trained for the target task
  • the target neural network model is used to perform a first operation on the output of the first attention head and the first initial weight value, and the operation result is compared with the output of the second transformer layer. Perform the second operation.
  • the second weight value is obtained by updating the second initial weight value when the target neural network model is trained for the target task.
  • the target neural network model is used to perform a first operation on the target FFN and the second initial weight value, and perform a second operation between the operation result and the output of the second transformer layer. operation.
  • the obtaining module is used to:
  • the weight value corresponding to the target task is obtained according to a preset mapping relationship, and the weight value corresponding to the target task includes the first weight value and/or the second weight value; wherein the preset mapping relationship Including the correspondence between tasks and weight values.
  • the obtaining module is used to:
  • the first neural network is obtained by updating a first initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the first attention head, and the identification of the target task into the first an initial neural network, performing a first operation on the output of the first initial neural network and the output of the first attention head, and performing a second operation on the operation result and the output of the second transformer layer.
  • the second neural network is obtained by updating the second initial neural network when the target neural network model is trained for the target task, wherein In the training process of the neural network model, the target neural network model is used to input at least one of the data to be processed and the output of the target FFN, and the identification of the target task into the second initial neural network. network, and perform a first operation between the output of the second initial neural network and the output of the target FFN, and perform a second operation between the operation result and the output of the second transformer layer.
  • the first transformer layer includes multiple attention heads, and each attention head in the multiple attention heads corresponds to a weight value.
  • the target neural network model uses The first operation is performed on the output of each attention head and the corresponding weight value to obtain the third output, and the second operation is performed on the third output and the output of the second transformer layer.
  • the weight values corresponding to the heads are different.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the specific description of the data processing device 1600 can refer to the description of the corresponding embodiment in FIG. 13 , wherein, the acquisition module 1601 can perform steps 1301 and 1302 and the descriptions in the corresponding embodiments, and the data processing module 1602 can perform step 1303 and corresponding described in the examples.
  • FIG. 17 is a schematic structural diagram of a data processing device 1700 provided by an embodiment of the application.
  • the data processing device 1700 may be a terminal device or a server, and the data processing device 1700 includes:
  • the acquisition module 1701 is used to acquire the data to be processed, the correct data processing result of the data to be processed, and the initial neural network model for executing the target task
  • the initial neural network model includes a first transformer layer, the first transformer The layer includes a first residual branch and a second residual branch, the first residual branch includes a first attention head, the second residual branch includes a target feedforward layer FFN, the initial neural The network model is used to perform target operation on the output of the first attention head and the first initial weight value to obtain the output of the first residual branch, and/or the initial neural network model is used to combine the The output of the target FFN and the second initial weight value are subjected to target operation to obtain the output of the second residual branch;
  • a data processing module 1702 configured to process the data to be processed according to the initial neural network model to obtain a data processing result
  • the model updating module 1703 is configured to obtain the loss according to the data processing result and the correct data processing result, and update the first initial weight value and/or the second initial weight value based on the loss to obtain the target neural network model.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the model updating module is configured to obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the ith based on the loss an initial weight value and/or the second initial weight value to obtain a first neural network model, and in the process of the i+1 th iteration, update the first neural network model based on the loss except for the first neural network model.
  • the initial weight value and/or the network parameters of the second initial weight value to obtain the target neural network model.
  • FIG. 18 is a schematic structural diagram of a data processing device 1800 provided by an embodiment of the application.
  • the data processing device 1800 may be a terminal device or a server, and the data processing device 1800 includes:
  • the acquisition module 1801 is used to acquire the data to be processed, the correct data processing result of the data to be processed, and an initial neural network model for executing the target task
  • the initial neural network model includes a first transformer layer, the first transformer The layer includes a first residual branch and a second residual branch, the first residual branch includes a first attention head, the second residual branch includes a target feedforward layer FFN, the initial neural
  • the network model is used to input the data to be processed and/or the output of the first attention head and the identification of the target task into the first initial neural network, and the output of the first initial neural network is combined with the output of the first initial neural network.
  • the output of the first attention head is subjected to target operation to obtain the output of the first residual branch, and/or the initial neural network model is used for the data to be processed and/or the target FFN.
  • the output and the identifier of the target task are input into the second initial neural network, and the output of the second initial neural network and the output of the target FFN are subjected to target operation to obtain the output of the second residual branch. ;
  • Data processing module 1802 for processing the data to be processed according to the initial neural network model to obtain a data processing result
  • a model updating module 1803 configured to obtain a loss according to the data processing result and the correct data processing result, and update the first initial neural network and/or the second initial neural network based on the loss to obtain a target neural network network model.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the target operation includes a product operation.
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • the model updating module is configured to obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the ith based on the loss an initial neural network and/or the second initial neural network to obtain a first neural network model, in the process of the i+1th iteration, update the first neural network model based on the loss except the first neural network model The initial neural network and/or the network parameters of the second initial neural network to obtain the target neural network model.
  • FIG. 19 is a schematic structural diagram of a data processing device 1900 provided by an embodiment of the application.
  • the data processing device 1900 may be a terminal device or a server, and the data processing device 1900 includes:
  • the acquisition module 1901 is used to acquire the data to be processed, the correct data processing result of the data to be processed, and an initial neural network model for performing the target task, where the initial neural network model includes a first transformer layer and a second transformer layer,
  • the first transformer layer includes a first attention head and a target FFN;
  • the initial neural network model is used to perform a first operation on the output of the first attention head and a first weight value to obtain a first output, and performing a second operation on the first output and the output of the second transformer layer;
  • the initial neural network model is used to perform a first operation on the target FFN and the second weight value to obtain a second output, and perform a second operation on the second output and the output of the second transformer layer;
  • a data processing module 1902 configured to process the data to be processed according to the target neural network model to obtain a data processing result
  • a model updating module 1903 configured to obtain a loss according to the data processing result and the correct data processing result, and update the first initial weight value and/or the second initial weight value based on the loss to obtain the target neural network network model.
  • the initial neural network model includes multiple transformer layers and an output layer
  • the second transformer layer is a transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the model updating module is configured to obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the ith based on the loss an initial weight value and/or the second initial weight value to obtain a first neural network model, and in the process of the i+1 th iteration, update the first neural network model based on the loss except for the first neural network model.
  • the initial weight value and/or the network parameters of the second initial weight value to obtain the target neural network model.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • FIG. 20 is a schematic structural diagram of a data processing device 2000 provided by an embodiment of the application.
  • the data processing device 2000 may be a terminal device or a server, and the data processing device 2000 includes:
  • the acquisition module 2001 is used to acquire the data to be processed, the correct data processing result of the data to be processed, and an initial neural network model for performing the target task
  • the initial neural network model includes a first transformer layer and a second transformer layer,
  • the first transformer layer includes a first attention head and a target FFN;
  • the initial neural network model is used to convert the data to be processed and/or the output of the first attention head and the identification of the target task Input to the first initial neural network, perform a first operation on the output of the first initial neural network and the output of the first attention head, and perform a second operation on the operation result and the output of the second transformer layer and/or
  • the initial neural network model is used to input the output of the data to be processed and/or the target FFN and the identification of the target task into the second initial neural network, and the second The output of the initial neural network is subjected to a first operation with the output of the target FFN, and the second operation is performed between the operation result and the output of the second transformer layer;
  • a data processing module 2002 configured to process the data to be processed according to the target neural network model to obtain a data processing result
  • a model updating module 2003 configured to obtain a loss according to the data processing result and the correct data processing result, and update the first initial neural network and/or the second initial neural network based on the loss to obtain a target neural network network model.
  • the initial neural network model includes multiple transformer layers and an output layer
  • the second transformer layer is a transformer layer closest to the output layer among the multiple transformer layers.
  • the first transformer layer includes multiple attention heads, and the multiple attention heads include the first attention head and the second attention head; correspondingly, the initial neural
  • the network model is further configured to perform a target operation on the output of the second attention head and the third weight value, wherein the first weight value is different from the third weight value.
  • the model updating module is configured to obtain a loss according to the data processing result and the correct data processing result, and in the process of the ith iteration, update only the ith based on the loss an initial neural network and/or the second initial neural network to obtain a first neural network model, in the process of the i+1th iteration, update the first neural network model based on the loss except the first neural network model The initial neural network and/or the network parameters of the second initial neural network to obtain the target neural network model.
  • the first operation includes a product operation
  • the second operation includes an addition operation
  • the target task includes one of the following: reading comprehension, text translation, paraphrase recognition, named entity recognition, text sentiment analysis, natural language inference, text automatic question answering, text intent recognition, text classification, Text simplification and text story generation.
  • FIG. 21 is a schematic structural diagram of the execution device provided by the embodiment of the present application.
  • the execution device 2100 may specifically be represented as a virtual reality VR device, a mobile phone, Tablets, laptops, smart wearable devices, monitoring data processing devices or servers, etc., are not limited here.
  • the execution device 2100 includes: a receiver 2101, a transmitter 2102, a processor 2103, and a memory 2104 (wherein the number of processors 2103 in the execution device 2100 may be one or more, and one processor is taken as an example in FIG. 21 ) , wherein the processor 2103 may include an application processor 21031 and a communication processor 21032.
  • the receiver 2101, the transmitter 2102, the processor 2103, and the memory 2104 may be connected by a bus or otherwise.
  • Memory 2104 may include read-only memory and random access memory, and provides instructions and data to processor 2103 . A portion of memory 2104 may also include non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • the memory 2104 stores processors and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations.
  • the processor 2103 controls the operation of the execution device.
  • various components of the execution device are coupled together through a bus system, where the bus system may include a power bus, a control bus, a status signal bus, and the like in addition to a data bus.
  • the various buses are referred to as bus systems in the figures.
  • the methods disclosed in the above embodiments of the present application may be applied to the processor 2103 or implemented by the processor 2103 .
  • the processor 2103 may be an integrated circuit chip, which has signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 2103 or an instruction in the form of software.
  • the above-mentioned processor 2103 may be a general-purpose processor, a digital signal processing (DSP), a microprocessor or a microcontroller, and may further include an application specific integrated circuit (ASIC), a field programmable Field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • DSP digital signal processing
  • ASIC application specific integrated circuit
  • FPGA field programmable Field-programmable gate array
  • the processor 2103 may implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of this application.
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.
  • the software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art.
  • the storage medium is located in the memory 2104, and the processor 2103 reads the information in the memory 2104, and completes the steps of the above method in combination with its hardware.
  • the receiver 2101 can be used to receive input numerical or character information, and generate signal input related to the relevant settings and function control of the execution device.
  • the transmitter 2102 can be used to output digital or character information through the first interface; the transmitter 2102 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 2102 can also include display devices such as a display screen .
  • the processor 2103 is configured to execute the data processing method executed by the device in the embodiment corresponding to FIG. 6 and FIG. 13 .
  • FIG. 22 is a schematic structural diagram of the training device provided by the embodiment of the present application.
  • the training device 2200 may be deployed with all of the training devices in the corresponding embodiments of FIGS. 17 to 20 .
  • the described data processing apparatus, specifically, the training device 2200 is implemented by one or more servers, and the training device 2200 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (central processing units, CPU) 2222 (eg, one or more processors) and memory 2232, one or more storage media 2230 (eg, one or more mass storage devices) storing applications 2242 or data 2244.
  • CPU central processing units
  • storage media 2230 eg, one or more mass storage devices
  • the memory 2232 and the storage medium 2230 may be short-term storage or persistent storage.
  • the program stored in the storage medium 2230 may include one or more modules (not shown in the figure), and each module may include a series of instructions to operate on the training device.
  • the central processing unit 2222 may be configured to communicate with the storage medium 2230 to execute a series of instruction operations in the storage medium 2230 on the training device 2200.
  • the training device 2200 may also include one or more power supplies 2226, one or more wired or wireless network interfaces 2250, one or more input and output interfaces 2258; or, one or more operating systems 2241, such as Windows ServerTM, Mac OS XTM , UnixTM, LinuxTM, FreeBSDTM and so on.
  • operating systems 2241 such as Windows ServerTM, Mac OS XTM , UnixTM, LinuxTM, FreeBSDTM and so on.
  • the central processing unit 2222 is configured to execute the data processing method executed by the data processing apparatus in the embodiment corresponding to FIG. 18 .
  • Embodiments of the present application also provide a computer program product that, when running on a computer, causes the computer to perform the steps performed by the aforementioned execution device, or causes the computer to perform the steps performed by the aforementioned training device.
  • Embodiments of the present application further provide a computer-readable storage medium, where a program for performing signal processing is stored in the computer-readable storage medium, and when it runs on a computer, the computer executes the steps performed by the aforementioned execution device. , or, causing the computer to perform the steps as performed by the aforementioned training device.
  • the execution device, training device, or terminal device provided in this embodiment of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, pins or circuits, etc.
  • the processing unit can execute the computer executable instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiments, or the chip in the training device executes the data processing method described in the above embodiment.
  • the storage unit is a storage unit in the chip, such as a register, a cache, etc.
  • the storage unit may also be a storage unit located outside the chip in the wireless access device, such as only Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), etc.
  • ROM Read-only memory
  • RAM random access memory
  • FIG. 23 is a schematic structural diagram of a chip provided by an embodiment of the application.
  • the chip may be represented as a neural network processor NPU 2300, and the NPU 2300 is mounted as a co-processor to the main CPU (Host CPU), tasks are allocated by the Host CPU.
  • the core part of the NPU is the arithmetic circuit 2303, which is controlled by the controller 2304 to extract the matrix data in the memory and perform multiplication operations.
  • the arithmetic circuit 2303 includes multiple processing units (Process Engine, PE). In some implementations, the arithmetic circuit 2303 is a two-dimensional systolic array. The arithmetic circuit 2303 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2303 is a general-purpose matrix processor.
  • the arithmetic circuit fetches the data corresponding to the matrix B from the weight memory 2302 and buffers it on each PE in the arithmetic circuit.
  • the arithmetic circuit fetches the data of matrix A and matrix B from the input memory 2301 to perform matrix operation, and stores the partial result or final result of the matrix in the accumulator 2308 .
  • Unified memory 2306 is used to store input data and output data.
  • the weight data directly passes through the storage unit access controller (Direct Memory Access Controller, DMAC) 2305, and the DMAC is transferred to the weight memory 2302.
  • Input data is also moved to unified memory 2306 via the DMAC.
  • DMAC Direct Memory Access Controller
  • the BIU is the Bus Interface Unit, that is, the bus interface unit 2310, which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (Instruction Fetch Buffer, IFB) 2309.
  • IFB Instruction Fetch Buffer
  • the bus interface unit 2310 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2309 to obtain instructions from the external memory, and also for the storage unit access controller 2305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
  • the DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 2306 , the weight data to the weight memory 2302 , or the input data to the input memory 2301 .
  • the vector calculation unit 2307 includes a plurality of operation processing units, and further processes the output of the operation circuit if necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on. It is mainly used for non-convolutional/fully connected layer network computation in neural networks, such as Batch Normalization, pixel-level summation, and upsampling of feature planes.
  • the vector computation unit 2307 can store the processed output vectors to the unified memory 2306 .
  • the vector calculation unit 2307 can apply a linear function; or a nonlinear function to the output of the operation circuit 2303, such as performing linear interpolation on the feature plane extracted by the convolution layer, such as a vector of accumulated values, to generate activation values.
  • the vector computation unit 2307 generates normalized values, pixel-level summed values, or both.
  • the vector of processed outputs can be used as activation input to the arithmetic circuit 2303, eg, for use in subsequent layers in a neural network.
  • the instruction fetch memory (instruction fetch buffer) 2309 connected to the controller 2304 is used to store the instructions used by the controller 2304;
  • the unified memory 2306, the input memory 2301, the weight memory 2302 and the instruction fetch memory 2309 are all On-Chip memories. External memory is private to the NPU hardware architecture.
  • the processor mentioned in any one of the above may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.
  • the device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be A physical unit, which can be located in one place or distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general purpose computer, special purpose computer, computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be retrieved from a website, computer, training device, or data Transmission from the center to another website site, computer, training facility or data center via wired (eg coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) means.
  • wired eg coaxial cable, fiber optic, digital subscriber line (DSL)
  • wireless eg infrared, wireless, microwave, etc.
  • the computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a training device, a data center, or the like that includes an integration of one or more available media.
  • the usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Pure & Applied Mathematics (AREA)
  • Mathematical Optimization (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Image Analysis (AREA)

Abstract

一种数据处理方法，包括：获取待处理数据以及目标神经网络模型，目标神经网络模型包括第一transformer层，第一transformer层包括第一残差支路和第二残差支路，第一残差支路包括第一注意力头，第二残差支路包括目标前馈层FFN（601）；根据目标神经网络模型对待处理数据进行目标任务相关的处理，以得到数据处理结果，其中目标神经网络模型用于将第一注意力头的输出与第一权重值进行目标运算，得到第一残差支路的输出，和/或目标神经网络模型用于将目标FFN的输出与第二权重值进行目标运算，得到第二残差支路的输出（603）。该方法针对于不同的任务，设置了用于控制残差支路的输出的权重值，降低了终端设备运行目标神经网络模型的计算资源需求。

Description

一种数据处理方法及相关设备
本申请要求于2020年9月29日提交中国国家知识产权局、申请号为202011052624.X、申请名称为“一种数据处理方法及相关设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，尤其涉及一种数据处理方法及相关设备。
背景技术
人工智能(artificial intelligence，AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个分支，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式作出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。
随着人工智能技术的不断发展，让人机之间能够通过自然语言进行交互的自然语言人机交互系统变的越来越重要。人机之间能够通过自然语言进行交互，就需要系统能够识别出人类自然语言的具体含义。通常，系统通过采用对自然语言的句子进行关键信息提取来识别句子的具体含义。
transformer结构具有强大的语义表达能力，能捕捉文本长依赖关系。自被提出以来在以翻译为代表的一系列自然语言处理的任务上显著超越了之前的模型，基于transformer结构的预训练语言模型在问答系统,语音助手等领域也取得了非常好的效果。
transformer模型参数很多，对计算、功耗的需求高。例如Bert-base模型有12层，768个隐状态，共有110万参数量。因此难以给每一个自然语言处理的任务训练一个单独的模型，再存放在终端设备上，更加可行的做法是训练一个多任务的Transformer模型。
在现有的实现中，可以先预训练一个骨架网络(例如BERT)，给所有任务共享骨架网络的参数，再将其作为底层网络，在其基础上新增一个任务专属的神经网络，并各自训练，此类方法通常需要增加较多的参数，无法满足终端设备低计算资源的需求。
发明内容
第一方面，本申请提供了一种数据处理方法，所述方法包括：
获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一转换(transformer)层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN；其中，多头注意力层所在的支路以及前馈层FFN所在的支路可以为transformer层中的残差支路；待处理数据可以为文本数据，目标神经网络模型可以为训练好的，且可以进行多任务处理的transformer模型，应理解，为了实现不同的任务，目标神经网络模型可以在输出位置包括多个任务层， 各任务层适配于不同的任务。获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值；为了适配于不同的任务，可以确定不同的权重值来与transformer层中的注意力头的输出或者FFN的输出进行运算，来控制残差支路的输出，目标神经网络模型在执行不同的任务时，可以选择不同的权重，使得残差支路的输出更适配于当前目标神经网络模型执行的任务；
根据所述目标神经网络模型对所述待处理数据进行所述目标任务相关的处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与所述第一权重值进行目标运算，得到所述第一残差支路的输出，和/或所述目标神经网络模型用于将所述目标FFN的输出与所述第二权重值进行目标运算，得到所述第二残差支路的输出。
本实施例中，针对于不同的任务，设置了用于控制残差支路的输出的权重值，相当于对每个任务可以设置一套专属的分布式表示的权重值组合，从而实现了目标神经网络模型的多任务学习，且本实施例中相比于在输出层增加适配不同任务的神经网络，需要学习的参数较少，进而可以降低终端设备运行目标神经网络模型的计算资源需求。
在一种可能的实现中，在所述目标神经网络模型用于将所述第一注意力头的输出作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行目标运算，得到所述第一残差支路的输出的情况下，所述目标神经网络模型的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度，所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出，且将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
本申请实施例中，为了使得目标神经网络模型能够适配于不同的任务(也就是说针对于不同的任务都有较高的数据处理能力)，在原有的transformer层的残差支路中增加了权重控制策略，针对于不同的任务设置不同的权重值。
在一种可能的实现中，所述第一权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与所述第一初始权重 值进行目标运算，得到所述第一残差支路的输出。
在一种可能的实现中，所述第二权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN的输出与所述第二初始权重值进行目标运算，得到所述第二残差支路的输出。
训练设备可以获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述目标FFN的输出与第二初始权重值进行目标运算，得到所述第二残差支路的输出；根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；之后可以根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或第二初始权重值，直至初始神经网络模型针对于目标任务的数据处理精度满足要求，以此得到第一权重值和/或第二权重值。
本申请实施例中，针对于不同的任务可以得到不同的第一权重值以及第二权重值，进而可以得到上述实施例中预设的映射关系。
在一种可能的实现中，所述获取目标任务对应的权重值，包括：
根据预设的映射关系获取所述目标任务对应的权重值，所述目标任务对应的权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系。针对于不同的任务可以得到不同的第一权重值以及第二权重值，进而可以得到上述实施例中预设的映射关系。
在一种实现中，训练设备可以根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。本实施例中，可以在一次迭代时将骨架模型(骨架模型可以理解为目标神经网络模型中除权重值之外的部分网络)固定住(其中“固定”可以理解为保持网络的结构和参数大小不变)，并更新权重值，然后在下一次迭代时，将权重值固定住，并更新骨架模型(目标神经网络模型中除权重值之外的部分网络)。
在一种可能的实现中，所述获取目标任务对应的权重值，包括：
将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，
将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输 入到第二神经网络，得到所述第二权重值。
也就是说，第一神经网络具有输入待处理数据和所述第一注意力头的输出中的至少一项、以及所述目标任务的标识，输出第一权重值的能力。第二神经网络具有输入待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值的能力。
本申请实施例中，针对于不同的任务，可以训练得到不同的第一神经网络以及第二神经网络。上述第一神经网络以及第二神经网络可以但不限于为全连接网络，在训练时，可以设定目标的初始门控值，然后利用梯度下降方法更新模型，也可以利用其它优化方法例如强化学习算法、遗传算法进行训练。
在一种可能的实现中，所述第一神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到所述第一初始神经网络，并将所述第一初始神经网络的输出与所述第一注意力头的输出进行目标运算，得到所述第一残差支路的输出。
在一种可能的实现中，所述第二神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到所述第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行目标运算，得到所述第二残差支路的输出。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头中的每一个注意力头对应一个权重值，所述目标神经网络模型用于将所述每一个注意力头的输出与对应的权重值进行目标运算得到所述第一残差支路的输出，其中，不同注意力头对应的权重值不同。本申请实施例中，目标神经网络模型可以包括多头注意力层，针对于每个任务t，每个注意力头n可以都部署一个注意力头的权重值来控制所在的残差支路的输出大小。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
第二方面，本申请提供了一种数据处理方法，所述方法包括：
获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；
获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/ 或所述目标FFN对应的第二权重值；
根据所述目标神经网络模型对所述待处理数据进行所述目标任务相关的处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与所述第一权重值进行第一运算，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述目标神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算。本实施例中，目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，第一输出并不作为第一残差支路的输出，而是将所述第一输出与所述第二transformer层的输出进行第二运算。本实施例中，目标神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，第二输出并不作为第二残差支路的输出，而是将所述第二输出与所述第二transformer层的输出进行第二运算。所述第一运算可以包括乘积运算，所述第二运算可以包括加和运算。
应理解，目标神经网络模型可以一方面，将所述第一注意力头的输出与第一权重值进行目标运算(例如乘积运算)，并将运算结果作为第一残差支路的输出，另一方面，也将所述第一输出与所述第二transformer层的输出进行加和运算。类似的，目标神经网络模型可以一方面将目标FFN与所述第二权重值进行目标运算(例如乘积运算)，并将运算结果作为第二残差支路的输出，另一方面，也将所述第二输出与所述第二transformer层的输出进行加和运算。
在一种可能的实现中，所述目标神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括所述第一注意力头；其中，
在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算的情况下，所述目标神经网络模型的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出仅作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度，所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述 目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，且在将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出，且将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
在一种可能的实现中，所述第一权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第二权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN与所述第二初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述获取目标任务对应的权重值，包括：
根据预设的映射关系获取所述目标任务对应的权重值，所述目标任务对应的权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系。
在一种可能的实现中，所述获取目标任务对应的权重值，包括：
将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，
将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值。
在一种可能的实现中，所述第一神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到所述第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第二神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述目标FFN的输出中 的至少一项，以及所述目标任务的标识输入到所述第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头中的每一个注意力头对应一个权重值，相应的，所述目标神经网络模型用于将每一个注意力头的输出与对应的权重值进行第一运算，得到第三输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，其中，不同注意力头对应的权重值不同。
在一种可能的实现中，所述第一运算包括乘积运算，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
第三方面，本申请提供了一种数据处理方法，包括：
获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述目标FFN的输出与第二初始权重值进行目标运算，得到所述第二残差支路的输出；
根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
在一种可能的实现中，所述根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型，包括：
根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。
第四方面，本申请提供了一种数据处理方法，包括：
获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一初始神经网络，并将所述第一初始神经网络的输出与所述第一注意力头的输出进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行目标运算，得到所述第二残差支路的输出；
根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
在一种可能的实现中，所述根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型，包括：
根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程 中，基于损失仅更新所述第一初始神经网络和/或所述第二初始神经网络，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始神经网络和/或所述第二初始神经网络的网络参数，以得到目标神经网络模型。
第五方面，本申请提供了一种数据处理方法，所述方法包括：
获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算；
根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型。
在一种可能的实现中，所述初始神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型，包括：
根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。
在一种可能的实现中，所述第一运算包括乘积运算，和/或，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
第六方面，本申请提供了一种数据处理方法，所述方法包括：
获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算；
根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
在一种可能的实现中，所述初始神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型，包括：
根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始神经网络和/或所述第二初始神经网络，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始神经网络和/或所述第二初始神经网络的网络参数，以得到目标神经网络模型。
在一种可能的实现中，所述第一运算包括乘积运算，和/或，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
第七方面，本申请提供了一种数据处理装置，所述装置包括：
获取模块，用于获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一转换(transformer)层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN；获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值；
数据处理模块，用于根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行目标运算，得到所述第一残差支路的输出，和/或所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出。
在一种可能的实现中，在所述目标神经网络模型用于将所述第一注意力头的输出作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行目标运算，得到所述第一残差支路的输出的情况下，所述目标神经网络模型的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度，所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出，且将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
在一种可能的实现中，所述第一权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与所述第一初始权重值进行目标运算，得到所述第一残差支路的输出。
在一种可能的实现中，所述第二权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN的输出与所述第二初始权重值进 行目标运算，得到所述第二残差支路的输出。
在一种可能的实现中，所述获取模块用于：
根据预设的映射关系获取所述目标任务对应的权重值，所述目标任务对应的权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系。
在一种可能的实现中，所述获取模块用于：
将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，
将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值。
在一种可能的实现中，所述第一神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到所述第一初始神经网络，并将所述第一初始神经网络的输出与所述第一注意力头的输出进行目标运算，得到所述第一残差支路的输出。
在一种可能的实现中，所述第二神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到所述第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行目标运算，得到所述第二残差支路的输出。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头中的每一个注意力头对应一个权重值，所述目标神经网络模型用于将所述每一个注意力头的输出与对应的权重值进行目标运算得到所述第一残差支路的输出，其中，不同注意力头对应的权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
第八方面，本申请提供了一种数据处理装置，所述装置包括：
获取模块，用于获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值；
数据处理模块，用于根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述目标神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述目标神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括所述第一注意力头，所述第二残差支路包括所述目标FFN；其中，
在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算的情况下，所述目标神经网络模型的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出仅作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度，所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，且在将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出，且将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
在一种可能的实现中，所述第一权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的 训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第二权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN与所述第二初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述获取模块用于：
根据预设的映射关系获取所述目标任务对应的权重值，所述目标任务对应的权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系。
在一种可能的实现中，所述获取模块用于：
将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，
将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值。
在一种可能的实现中，所述第一神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到所述第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第二神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到所述第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头中的每一个注意力头对应一个权重值，相应的，所述目标神经网络模型用于将每一个注意力头的输出与对应的权重值进行第一运算，得到第三输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，其中，不同注意力头对应的权重值不同。
在一种可能的实现中，所述第一运算包括乘积运算，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
第九方面，本申请提供了一种数据处理装置，包括：
获取模块，用于获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述目标FFN的输出与第二初始权重值进行目标运算，得到所述第二残差支路的输出；
数据处理模块，用于根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
在一种可能的实现中，所述模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。
第十方面，本申请提供了一种数据处理装置，包括：
获取模块，用于获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一初始神经网络，并将所述第一初始神经网络的输出与所述第一注意力头的输出进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行目标运算，得到所述第二残差支路的输出；
数据处理模块，用于根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
在一种可能的实现中，所述模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始神经网络和/或所述第二初始神经网络，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始神经网络和/或所述第二初始神经网络的网络参数，以得到目标神经网络模型。
第十一方面，本申请提供了一种数据处理装置，所述装置包括：
获取模块，用于获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所 述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算；
数据处理模块，用于根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型。
在一种可能的实现中，所述初始神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。
在一种可能的实现中，所述第一运算包括乘积运算，和/或，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
第十二方面，本申请提供了一种数据处理装置，所述装置包括：
获取模块，用于获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输 入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算；
数据处理模块，用于根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
在一种可能的实现中，所述初始神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始神经网络和/或所述第二初始神经网络，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始神经网络和/或所述第二初始神经网络的网络参数，以得到目标神经网络模型。
在一种可能的实现中，所述第一运算包括乘积运算，和/或，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
第十三方面，本申请实施例提供了一种执行设备，可以包括存储器、处理器以及总线系统，其中，存储器用于存储程序，处理器用于执行存储器中的程序，以执行如上述第一方面及其任一可选的方法、第二方面及其任一可选的方法。
第十四方面，本申请实施例提供了一种训练设备，可以包括存储器、处理器以及总线系统，其中，存储器用于存储程序，处理器用于执行存储器中的程序，以执行如上述第三至第六方面及其任一可选的方法。
第十五方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序，当其在计算机上运行时，使得计算机执行上述第一至第六方面及 其任一可选的方法。
第十六方面，本申请实施例提供了一种计算机程序，当其在计算机上运行时，使得计算机执行上述第一至第六方面及其任一可选的方法。
第十七方面，本申请提供了一种芯片系统，该芯片系统包括处理器，用于支持执行设备或训练设备实现上述方面中所涉及的功能，例如，发送或处理上述方法中所涉及的数据；或，信息。在一种可能的设计中，所述芯片系统还包括存储器，所述存储器，用于保存执行设备或训练设备必要的程序指令和数据。该芯片系统，可以由芯片构成，也可以包括芯片和其他分立器件。
本申请实施例提供了一种数据处理方法，包括：获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一转换(transformer)层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN；获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值；根据所述目标神经网络模型对所述待处理数据进行所述目标任务相关的处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与所述第一权重值进行目标运算，得到所述第一残差支路的输出，和/或所述目标神经网络模型用于将所述目标FFN的输出与所述第二权重值进行目标运算，得到所述第二残差支路的输出。针对于不同的任务，设置了用于控制残差支路的输出的权重值，相当于对每个任务可以设置一套专属的分布式表示的权重值组合，从而实现了目标神经网络模型的多任务学习，且本实施例中相比于在输出层增加适配不同任务的神经网络，需要学习的参数较少，进而可以降低终端设备运行目标神经网络模型的计算资源需求。
附图说明
图1为人工智能主体框架的一种结构示意图；
图2为一种自然语言处理系统；
图3为另一种自然语言处理系统；
图4为本申请实施例提供的自然语言处理的相关设备的示意图；
图5为一种transformer层的架构示意；
图6为本申请实施例提供的一种数据处理方法的实施例示意；
图7为本申请实施例中的一种神经网络模型的结构示意；
图8为一种transformer层的结构示意；
图9为一个注意力头head的操作示意图；
图10为本申请实施例提供的一种神经网络模型的结构示意；
图11为本申请实施例提供的一种神经网络模型的结构示意；
图12为本申请实施例提供的一种神经网络模型的结构示意；
图13为本申请实施例提供的一种数据处理方法的实施例示意；
图14为本申请实施例提供的一种神经网络模型的结构示意；
图15为本申请实施例提供的数据处理设备的结构示意图；
图16为本申请实施例提供的数据处理设备的一种结构示意图；
图17为本申请实施例提供的数据处理设备的一种结构示意图；
图18为本申请实施例提供的数据处理设备的一种结构示意图；
图19为本申请实施例提供的数据处理设备的一种结构示意图；
图20为本申请实施例提供的数据处理设备的一种结构示意图；
图21为本申请实施例提供的执行设备的一种结构示意图；
图22是本申请实施例提供的训练设备一种结构示意图；
图23为本申请实施例提供的芯片的一种结构示意图。
具体实施方式
下面结合本发明实施例中的附图对本发明实施例进行描述。本发明的实施方式部分使用的术语仅用于对本发明的具体实施例进行解释，而非旨在限定本发明。
下面结合附图，对本申请的实施例进行描述。本领域普通技术人员可知，随着技术的发展和新场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的术语在适当情况下可以互换，这仅仅是描述本申请的实施例中对相同属性的对象在描述时所采用的区分方式。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，以便包含一系列单元的过程、方法、系统、产品或设备不必限于那些单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它单元。
首先对人工智能系统总体工作流程进行描述，请参见图1，图1示出的为人工智能主体框架的一种结构示意图，下面从“智能信息链”(水平轴)和“IT价值链”(垂直轴)两个维度对上述人工智能主题框架进行阐述。其中，“智能信息链”反映从数据的获取到处理的一列过程。举例来说，可以是智能信息感知、智能信息表示与形成、智能推理、智能决策、智能执行与输出的一般过程。在这个过程中，数据经历了“数据—信息—知识—智慧”的凝练过程。“IT价值链”从人智能的底层基础设施、信息(提供和处理技术实现)到系统的产业生态过程，反映人工智能为信息技术产业带来的价值。
(1)基础设施
基础设施为人工智能系统提供计算能力支持，实现与外部世界的沟通，并通过基础平台实现支撑。通过传感器与外部沟通；计算能力由智能芯片(CPU、NPU、GPU、ASIC、FPGA等硬件加速芯片)提供；基础平台包括分布式计算框架及网络等相关的平台保障和支持，可以包括云存储和计算、互联互通网络等。举例来说，传感器和外部沟通获取数据，这些数据提供给基础平台提供的分布式计算系统中的智能芯片进行计算。
(2)数据
基础设施的上一层的数据用于表示人工智能领域的数据来源。数据涉及到图形、图像、语音、文本，还涉及到传统设备的物联网数据，包括已有系统的业务数据以及力、位移、 液位、温度、湿度等感知数据。
(3)数据处理
数据处理通常包括数据训练，机器学习，深度学习，搜索，推理，决策等方式。
其中，机器学习和深度学习可以对数据进行符号化和形式化的智能信息建模、抽取、预处理、训练等。
推理是指在计算机或智能系统中，模拟人类的智能推理方式，依据推理控制策略，利用形式化的信息进行机器思维和求解问题的过程，典型的功能是搜索与匹配。
决策是指智能信息经过推理后进行决策的过程，通常提供分类、排序、预测等功能。
(4)通用能力
对数据经过上面提到的数据处理后，进一步基于数据处理的结果可以形成一些通用的能力，比如可以是算法或者一个通用系统，例如，翻译，文本的分析，计算机视觉的处理，语音识别，图像的识别等等。
(5)智能产品及行业应用
智能产品及行业应用指人工智能系统在各领域的产品和应用，是对人工智能整体解决方案的封装，将智能信息决策产品化、实现落地应用，其应用领域主要包括：智能终端、智能交通、智能医疗、自动驾驶、平安城市等。
本申请可以应用于人工智能领域的自然语言处理领域中，下面将对多个落地到产品的多个应用场景进行介绍。
为了更好地理解本申请实施例的方案，下面先结合图1至图3对本申请实施例可能的应用场景进行简单的介绍。
图2示出了一种自然语言处理系统，该自然语言处理系统包括用户设备以及数据处理设备。其中，用户设备包括手机、个人电脑或者信息处理中心等智能终端。用户设备为自然语言数据处理的发起端，作为语言问答或者查询等请求的发起方，通常用户通过用户设备发起请求。
上述数据处理设备可以是云服务器、网络服务器、应用服务器以及管理服务器等具有数据处理功能的设备或服务器。数据处理设备通过交互接口接收来自智能终端的查询语句/语音/文本等问句，再通过存储数据的存储器以及数据处理的处理器环节进行机器学习，深度学习，搜索，推理，决策等方式的语言数据处理，并将处理结果反馈至用户设备。数据处理设备中的存储器可以是一个统称，包括本地存储以及存储历史数据的数据库，数据库可以在数据处理设备上，也可以在其它网络服务器上。
在图2所示的自然语言处理系统中，用户设备可以接收用户的指令，例如用户设备可以接收用户输入的一段文本，然后向数据处理设备发起请求，使得数据处理设备针对用户设备得到的该一段文本执行自然语言处理应用(例如文本分类、文本推理、命名实体识别、翻译等)，从而得到针对该一段文本的对应的自然语言处理应用的处理结果(例如分类结果、推理结果、命名实体识别结果、翻译结果等)。示例性的，用户设备可以接收用户输入的一段中文，然后向数据处理设备发起请求，使得数据处理设备对该一段中文进行实体分类，从而得到针对该一段中文的实体分类结果；示例性的，用户设备可以接收用户输入的一段 中文，然后向数据处理设备发起请求，使得数据处理设备将该一段中文翻译成英文，从而得到针对该一段中文的英文译文。
在本申请实施例中，数据处理设备可以通过交互接口接收来自用户设备获取自然语言处理(natural language processing，NLP)相关任务模型的请求以及性能上限参数，包括且不限于：精度、时延，模型压缩比参数中的至少一个。数据处理设备可以根据已经训练好的可伸缩transformer模型以及用户设备上传的需满足的性能上限参数，在满足模型大小的情况下计算出适合该用户设备的模型的尺寸大小，然后抽取出该尺寸大小的子网络，传回用户设备。
在图2中，数据处理设备可以执行本申请实施例的数据处理方法。
图3示出了另一种自然语言处理系统，在图3中，用户设备直接作为数据处理设备，该用户设备能够直接接收来自用户的输入并直接由用户设备本身的硬件进行处理，具体过程与图2相似，可参考上面的描述，在此不再赘述。
在图3所示的自然语言处理系统中，用户设备可以接收用户的指令，例如用户设备可以接收用户输入的一段文本，然后再由用户设备自身针对该一段文本执行自然语言处理应用(例如文本分类、文本推理、命名实体识别、翻译等)，从而得到针对该一段文本的对应的自然语言处理应用的处理结果(例如分类结果、推理结果、命名实体识别结果、翻译结果等)。示例性的，用户设备可以接收用户输入的一段中文，并针对该一段中文进行实体分类，从而得到针对该一段中文的实体分类结果；示例性的，用户设备可以接收用户输入的一段中文，并将该一段中文翻译成英文，从而得到针对该一段中文的英文译文。
在本申请实施例中，用户设备可以存储有子网络模型，并在每次操作系统(operating system，OS)或应用程序(application，APP)调用该模型之前，根据用户设备当前的资源情况(包括且不限于端侧设备当前的功耗、计算能力、存储参数中的至少一个)，计算出满足用户设备当前资源情况的模型的尺寸大小，将计算得到适合的模型的尺寸大小，输入到存储的子网络模型，得到动态裁剪后的当前状态模型，执行推理任务。
在图3中，用户设备自身就可以执行本申请实施例的数据处理方法。
图4是本申请实施例提供的自然语言处理的相关设备300的示意图。
上述图2和图3中的用户设备具体可以是图4中的本地设备301或者本地设备302，图2中的数据处理设备具体可以是图4中的执行设备310，其中，数据存储系统350可以存储执行设备310的待处理数据，数据存储系统350可以集成在执行设备310上，也可以设置在云上或其它网络服务器上。
图2和图3中的处理器可以通过神经网络模型或者其它模型进行数据训练/机器学习/深度学习，并利用数据最终训练或者学习得到的模型针对文本序列执行自然语言处理应用(例如文本分类、序列标注、阅读理解、文本生成、文本推理、翻译等)，从而得到相应的处理结果。
由于本申请实施例涉及大量神经网络的应用，为了便于理解，下面先对本申请实施例涉及的相关术语及神经网络等相关概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以xs和截距1为输入的运算单元，该运算单元的输出可以为：
其中，s＝1、2、……n，n为大于1的自然数，Ws为xs的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)transformer层
参照图5，图5为一种transformer层的架构示意，如图5所示，神经网络包括嵌入层和至少一个transformer层，至少一个transformer层可以为N个transformer层(N大于0的整数)，其中，每个transformer层包括依次相邻的注意力层、加和与归一化(add&norm)层、前馈(feed forward)层和加和与归一化层。在嵌入层，对当前输入进行嵌入处理，得到多个特征向量；在所述注意力层，从所述第一transformer层的上一层获取P个输入向量，以P个输入向量中的任意的第一输入向量为中心，基于预设的注意力窗口范围内的各个输入向量与该第一输入向量之间的关联度，得到该第一输入向量对应的中间向量，如此确定出P个输入向量对应的P个中间向量；在所述池化层，将所述P个中间向量合并为Q个输出向量，其中transformer层中最后一个transformer层得到的多个输出向量用作所述当前输入的特征表示。
接下来，结合具体例子对上述各步骤进行具体介绍。
首先，在所述嵌入层，对当前输入进行嵌入处理，得到多个特征向量。
嵌入层可以称为输入嵌入(input embedding)层。当前输入可以为文本输入，例如可以为一段文本，也可以为一个句子。文本可以为中文文本，也可以为英文文本，还可以为其他语言文本。嵌入层在获取当前输入后，可以对该当前输入中各个词进行嵌入处理，可得到各个词的特征向量。在一些实施例中，如图1所示，所述嵌入层包括输入嵌入层和位置编码(positional encoding)层。在输入嵌入层，可以对当前输入中的各个词进行词嵌入处理，从而得到各个词的词嵌入向量。在位置编码层，可以获取各个词在该当前输入中的位置，进而对各个词的位置生成位置向量。在一些示例中，各个词的位置可以为各个词在该当前输入中的绝对位置。以当前输入为“几号应还花呗”为例，其中的“几”的位置可以表示为第一位，“号”的位置可以表示为第二位，……。在一些示例中，各个词的位置可以为各个词之间的相对位置。仍以当前输入为“几号应还花呗”为例，其中的“几”的位置可以表示为“号”之前，“号”的位置可以表示为“几”之后、“应”之前，……。当得到当前输入中各个词的词嵌入向量和位置向量时，可以将各个词的位置向量和对应的词嵌入向量进行组合，得到各个词特征向量，即得到该当前输入对应的多个特征向量。多个特征向量可以表示为具有预设维度的嵌入矩阵。可以设定该多个特征向量中的特征向量个数为M，预设维度为H维，则该多个特征向量可以表示为M×H的嵌入矩阵。
其次，从所述第一transformer层的上一层获取P个输入向量，以P个输入向量中的任意的第一输入向量为中心，基于预设的注意力窗口范围内的各个输入向量与该第一输入向量之间的关联度，得到该第一输入向量对应的中间向量，如此确定出P个输入向量对应的P个中间向量。注意力层也可以称为多头注意力(multi-head attention)层。在一个例子中，注意力层可以为固定窗口多头注意力(fixed window multi-head attention)层。
在一些实施例中，第一transformer层可以为上述嵌入层的下一层，P个输入向量为从嵌入层得到的所述多个特征向量。在一些实施例中，本说明书实施例提供的神经网络中的至少一个transformer层还包括第二transformer层。该第二transformer层为第一自注意力的上一层，则P个输入向量为第二transformer层输出的P个输出向量。在该神经网络中的最后一个transformer层，通过上述步骤的多个输出向量可用作当前输入的特征表示。该特征表示为为当前输入的一种适合计算机处理的特征表示，可用于文本相似度、文本分类、阅读理解、机器翻译等任务。
(3)注意力机制(attention mechanism)
注意力机制模仿了生物观察行为的内部过程，即一种将内部经验和外部感觉对齐从而增加部分区域的观察精细度的机制，能够利用有限的注意力资源从大量信息中快速筛选出高价值信息。注意力机制可以快速提取稀疏数据的重要特征，因而被广泛用于自然语言处理任务，特别是机器翻译。而自注意力机制(self-attention mechanism)是注意力机制的改进，其减少了对外部信息的依赖，更擅长捕捉数据或特征的内部相关性。注意力机制的本质思想可以改写为如下公式：
其中，Lx＝||Source||代表Source的长度，公式含义即将Source中的构成元素想象成是由一系列的数据对构成，此时给定目标Target中的某个元素Query，通过计算Query和各个Key的相似性或者相关性，得到每个Key对应Value的权重系数，然后对Value进行加权求和，即得到了最终的Attention数值。所以本质上Attention机制是对Source中元素的Value值进行加权求和，而Query和Key用来计算对应Value的权重系数。从概念上理解，把Attention可以理解为从大量信息中有选择地筛选出少量重要信息并聚焦到这些重要信息上，忽略大多不重要的信息。聚焦的过程体现在权重系数的计算上，权重越大越聚焦于其对应的Value值上，即权重代表了信息的重要性，而Value是其对应的信息。自注意力机制可以理解为内部Attention(intra attention)，Attention机制发生在Target的元素Query和Source中的所有元素之间，自注意力机制指的是在Source内部元素之间或者Target内部元素之间发生的Attention机制，也可以理解为Target＝Source这种特殊情况下的注意力计算机制，其具体计算过程是一样的，只是计算对象发生了变化而已。
(4)自然语言处理(natural language processing，NLP)
自然语言(natural language)即人类语言，自然语言处理(NLP)就是对人类语言的处理。自然语言处理是以一种智能与高效的方式，对文本数据进行系统化分析、理解与信息提取的过程。通过使用NLP及其组件，我们可以管理非常大块的文本数据，或者执行大量的自动化任务，并且解决各式各样的问题，如自动摘要(automatic summarization)，机器翻译(machine translation，MT)，命名实体识别(named entity recognition，NER)，关系提取(relation  extraction，RE)，信息抽取(information extraction，IE)，情感分析，语音识别(speech recognition)，问答系统(question answering)以及主题分割等等。
示例性的，自然语言处理任务可以有以下几类。
序列标注：句子中每一个单词要求模型根据上下文给出一个分类类别。如中文分词、词性标注、命名实体识别、语义角色标注。
分类任务：整个句子输出一个分类值，如文本分类。
句子关系推断：给定两个句子，判断这两个句子是否具备某种名义关系。例如entilment、QA、语义改写、自然语言推断。
生成式任务：输出一段文本，生成另一段文本。如机器翻译、文本摘要、写诗造句、看图说话。
下面示例性的列举一些自然语言处理案例。
分词(word segmentation或word breaker，WB)：将连续的自然语言文本，切分成具有语义合理性和完整性的词汇序列，可以解决交叉歧义问题。例句：致毕业和尚未毕业的同学；分词1：致毕业，和，尚未毕业的同学；分词2：致毕业，和尚，未毕业的同学。
命名实体识别(named entity recognition，NER)：识别自然语言文本中具有特定意义的实体(人、地、机构、时间、作品等)，可以从粒度整合未登录体词。例句：天使爱美丽在线观看；分词：天使爱美丽，在线，观看；实体：天使爱美丽->电影。
词性标注(part-speech tagging)：为自然语言文本中的每个词汇赋予一个词性(名词、动词、形容词等)；依存句法分析(dependency parsing)：自动分析句子中的句法成分(主语、谓语、宾语、定语、状语和补语等成分)，可以解决结构歧义问题。评论：房间里还可以欣赏日出；歧义1：房间还可以；歧义2：可以欣赏日出；词性：房间里(主语)，还可以(谓语)，欣赏日出(动宾短语)。
词向量与语义相似度(word embedding&semantic similarity)：对词汇进行向量化表示，并据此实现词汇的语义相似度计算，可以解决词汇语言相似度。例如：西瓜与(呆瓜/草莓)，哪个更接近？向量化表示：西瓜(0.1222,0.22333,..)；相似度计算：呆瓜(0.115)草莓(0.325)；向量化表示：(-0.333,0.1223..)(0.333，0.3333,..)。
文本语义相似度(text semantic similarity)：依托全网海量数据和深度神经网络技术，实现文本间的语义相似度计算的能力，可以解决文本语义相似度问题。例如：车头如何防止车牌与(前牌照怎么装/如何办理北京牌照)，哪个更接近？向量化表示：车头如何防止车牌(0.1222,0.22333,..)；相似度计算：前牌照怎么装(0.762)，如何办理北京牌照(0.486)，向量化表示：(-0.333,0.1223..)(0.333，0.3333,..)。
本申请实施例提供的数据处理方法，涉及自然语言文本的处理，具体可以应用于数据训练、机器学习、深度学习等数据处理方法，对训练数据进行符号化和形式化的智能信息建模、抽取、预处理、训练等，最终得到训练好的目标神经网络模型；并且，本申请实施例提供的数据处理方法可以运用上述训练好的目标神经网络模型，将输入数据(如待处理语言信息)输入到所述训练好的目标神经网络模型中，得到输出数据(如与目标任务对应的处理结果)。需要说明的是，本申请实施例提供的目标神经网络相关的模型训练方法和数据处理 方法是基于同一个构思产生的发明，也可以理解为一个系统中的两个部分，或一个整体流程的两个阶段：如模型训练阶段和模型应用阶段。
首先以模型训练阶段为例对本申请实施例提供的数据处理方法进行说明。
参照图6，图6为本申请实施例提供的一种数据处理方法的实施例示意，本申请实
施例提供的一种数据处理方法可以应用在手机、平板、笔记本电脑、智能穿戴设备等
终端设备上，如图6示出的那样，本申请实施例提供的一种数据处理方法包括：
601、获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一转换(transformer)层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN。
本申请实施例中，终端设备可以获取待处理数据以及目标神经网络模型，其中，待处理数据可以为文本数据，目标神经网络模型可以为训练好的，且可以进行多任务处理的transformer模型。
本申请实施例中，目标神经网络模型可以包括第一transformer层，换一种表述方式，目标神经网络模型可以为基于transformer层的神经网络模型，可选地，目标神经网络模型可以为基于transformer层的NLP模型。
接下来，对目标神经网络模型的一种示例结构进行描述：
参照图7，图7为本申请实施例中的一种神经网络模型的结构示意，图7所示的神经网络模型可以为本申请实施例中的目标神经网络模型。如图7中示出的那样，目标神经网络模型可以包括依次连接的嵌入层以及多个transformer层。如本领域技术人员所了解，transformer模型多用于执行自然语言处理NLP任务。需要理解，图7的结构仅仅是一个示例，transformer层的数目可以根据需要而设置。例如，可以仅设置一个transformer层，也可以设置更多的transformer层。神经网络模型基于各transformer层得到的N个输出向量，确定当前节点对应的特征向量。
下面描述各个层的具体工作过程。
在嵌入层，对当前输入进行嵌入处理，得到多个特征向量。transformer模型的核心特点在于其采用的独特的注意力机制。在处理自然语言，例如一个句子时，transformer模型利用该注意力机制，为句子中各个词向量赋予不同的注意力系数，从而更全面地考虑句子中上下文对各个词的影响。嵌入层基于当前序列中各个节点的节点特征及其位置编码，得到N个嵌入向量X l。注意力层与嵌入层相连，从嵌入层获取N个嵌入向量作为输入向量，基于N个输入向量中各个输入向量之间的关联度，对各个输入向量进行综合，得到N个输出向量，输出给后续的transformer层。transformer层获取前一层的输出作为输入向量，执行与前一级transformer层类似的操作。
参照图8，图8为一种transformer层的结构示意，本申请实施例中的各个神经网络的transformer层(例如实施例中的第一transformer层、第二transformer层)都可以参照图8中示出的结构，如图8中示出的那样，transformer层包括依次相邻的多头注意力层、加和与归一化(add&norm)层、前馈(feed forward)层、加和与归一化层。
其中，多头注意力层从其上一层获取N个输入向量X l，又可以表示为矩阵X，采用自 注意力机制，基于向量间的关联度对各个向量进行变换，得到N个输出向量，又可以表示为矩阵Y。可以理解，当该多头注意力层是与嵌入层直接相连的层，例如图7中与嵌入层直连的transformer层，其获取的输入向量即为嵌入层输出的嵌入向量；当该多头注意力层是后续的transformer层包括的多头注意力层，例如图7中与上一级transformer层直连的transformer层包括的多头注意力层，其获取的输入向量即为前一级transformer层的输出向量。在多头注意力层，基于多头注意力(multi-head attention，MHA)的MHA层包括多个注意力头head(如图8中示出的Head 1、Head 2、…、Head N)。
图9为一个注意力头head的操作示意图，该示意图示出注意力头head如何将输入矩阵X变换为输出矩阵Y。如图9所示，分别采用第一变换矩阵Q，第二变换矩阵K和第三变换矩阵V对N个输入向量<X1,X2,…,XN>中各个输入向量Xi进行变换，得到各个输入向量对应的第一中间向量(q向量)，第二中间向量(k向量)和第三中间向量(v向量)。操作上，可以分别用第一变换矩阵Q，第二变换矩阵K和第三变换矩阵V，对N个输入向量构成的输入矩阵X进行线性变换，分别得到输入矩阵的Q矩阵，K矩阵和V矩阵，再分别对矩阵进行拆分，即可得到各个输入向量对应的q向量，k向量和v向量。对于N个输入向量中任意的第i输入向量Xi，基于该第i输入向量对应的第一中间向量(q向量，qi)与各个输入向量Xj对应的各个第二中间向量(k向量，kj)的点乘操作，确定该第i输入向量Xi与各个输入向量Xj的各个关联度。尽管也可以直接将qi与kj的点乘结果确定为关联度，但是更经典地，先将点乘结果除以一常数，然后进行softmax运算，将运算结果作为输入向量Xi与Xj的关联度，即：
Figure PCTCN2021119306-appb-000001
于是，可以以该第i输入向量Xi与各个输入向量Xj的各个关联度αi,j作为权重因子，对各个输入向量Xj对应的第三中间向量(v向量，vj)进行加权组合，得到该第i输入向量Xi对应的第i组合向量Ci：
Figure PCTCN2021119306-appb-000002
于是，可以得到N个输入向量对应的N个组合向量的向量序列<C1,C2,…,CN>，或矩阵C。基于该组合向量序列，可以得到N个输出向量。具体地，在一个实施例中，可以直接将N个组合向量的向量序列作为N个输出向量，即Yi＝Ci。此时，输出矩阵Y即为组合向量矩阵C，又可以写成：
Figure PCTCN2021119306-appb-000003
以上为一个注意力头head的处理过程描述，在MHA架构中，MHA层维护m套变换矩阵，每套变换矩阵包括前述第一变换矩阵Q、第二变换矩阵K和第三变换矩阵V，从而可以并行地进行上述操作，得到m个组合向量序列(即m个矩阵C)，每个向量序列包括基于一套变换矩阵得到的N个组合向量。在这样的情况下，MHA层将得到的m个组合向量序列进行拼接，得到拼接矩阵；再通过第四变换矩阵W对该拼接矩阵进行变换，得到最终的输出矩阵Y。将该输出矩阵Y拆分即对应于N个输出向量<Y1,Y2,…,YN>。通过以上的操作过程，MHA层 基于N个输入向量之间的关联度进行变换操作，得到N个输出向量。
本申请实施例中，多头注意力层所在的支路以及前馈层FFN所在的支路可以为transformer层中的残差支路。本实施例中的所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN。示例性的，可以参照图10，图10示出了一种第一transformer层的结构示意，如图10所示，第一transformer层可以包括第一残差支路和第二残差支路，第一残差支路为第一注意力头所在的支路，第二残差支路为目标FFN所在的支路。
应理解，为了实现不同的任务，目标神经网络模型可以在输出位置包括多个任务层，各任务层适配于不同的任务。
602、获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值。
本申请实施例中，为了适配于不同的任务，可以确定不同的权重值来与transformer层中的注意力头的输出或者FFN的输出进行运算，来控制残差支路的输出，目标神经网络模型在执行不同的任务时，可以选择不同的权重，使得残差支路的输出更适配于当前目标神经网络模型执行的任务。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
接下来以目标神经网络模型需要执行目标任务为例，描述如何获取目标任务对应的权重值。
本申请实施例中，权重值可以包括所述第一注意力头对应的第一权重值；
本申请实施例中，权重值可以包括所述目标FFN对应的第二权重值；
本申请实施例中，权重值可以包括所述第一注意力头对应的第一权重值以及所述目标FFN对应的第二权重值。
在一种实现中，可以根据预设的映射关系获取所述目标任务对应的权重值，所述目标权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系，且在所述预设的映射关系中，所述目标任务对应于所述目标权重值。其中，上述预设的映射关系为对所述目标神经网络模型进行针对于所述目标任务的训练时得到的。
其中，在预设的映射关系中，目标任务对应于第一权重值。
其中，在预设的映射关系中，目标任务对应于第二权重值。
其中，在预设的映射关系中，目标任务对应于第一权重值和第二权重值。
本申请实施例中，所述第一权重值可以为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与所述第一初始权重值进行目标运算，得到所述第一残差支路的输出。
本申请实施例中，所述第二权重值可以为对所述目标神经网络模型进行针对于所述目 标任务的训练时对第二初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN的输出与所述第二初始权重值进行目标运算，得到所述第二残差支路的输出。
训练设备可以获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述目标FFN的输出与第二初始权重值进行目标运算，得到所述第二残差支路的输出；根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；之后可以根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或第二初始权重值，直至初始神经网络模型针对于目标任务的数据处理精度满足要求，以此得到第一权重值和/或第二权重值。
本申请实施例中，针对于不同的任务可以得到不同的第一权重值以及第二权重值，进而可以得到上述实施例中预设的映射关系。
在一种实现中，训练设备可以根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。本实施例中，可以在一次迭代时将骨架模型(骨架模型可以理解为目标神经网络模型中除权重值之外的部分网络)固定住(其中“固定”可以理解为保持网络的结构和参数大小不变)，并更新权重值，然后在下一次迭代时，将权重值固定住，并更新骨架模型(目标神经网络模型中除权重值之外的部分网络)。
具体的，可以获取多个任务的训练集，其中，第t个任务的训练集为D t，完成预训练的BERT模型(参数为w)，初始化第一初始权重值和/或所述第二初始权重值(参数为门控模块参数θ)。每个任务t的损失函数
Figure PCTCN2021119306-appb-000004
具体迭代步骤如下：首先随机抽取任务id t，从D t中随机取样一批训练数据；之后固定住骨架模型，根据损失函数
Figure PCTCN2021119306-appb-000005
更新门控模块参数θ；之后固定住门控模块，根据损失函数
Figure PCTCN2021119306-appb-000006
更新骨架网络参数w。
在一种实现中，上述实施例中的第一初始权重值以及第二初始权重值可以为可学习标量经过sigmoid或其他函数处理之后得到的值，例如，第一初始权重值可以为
Figure PCTCN2021119306-appb-000007
其中，θ t，l为可学习标量，t表示目标任务，l可以表示第一注意力头。也就是说，在训练过程中，更新的为可学习标量，进而第一初始权重值以及第二初始 权重值也相应的进行了更新。在一种实现中，可学习标量可以初始化为某个常量，例如可以为0.5。
应理解，上述预设的映射关系可以为离散的对应关系(一个任务对应于一个权重值)，也可以通过一些线性或者非线性模型来表示，本申请并不限定。
在一种实现中，可以将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值。
本申请实施例中，可以获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一初始神经网络，并将所述第一初始神经网络的输出与所述第一注意力头的输出进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行目标运算，得到所述第二残差支路的输出；根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
也就是说，第一神经网络具有输入待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识，输出第一权重值的能力。第二神经网络具有输入待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值的能力。
本申请实施例中，针对于不同的任务，可以训练得到不同的第一神经网络以及第二神经网络。
上述第一神经网络以及第二神经网络可以但不限于为全连接网络，在训练时，可以设定目标的初始门控值，然后利用梯度下降方法更新模型，也可以利用其它优化方法例如强化学习算法、遗传算法进行训练。
603、根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行目标运算，得到所述第一残差支路的输出，和/或所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出。
本申请实施例中，终端设备可以根据所述目标神经网络模型对所述待处理数据进行所述目标任务相关的处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与所述第一权重值进行目标运算，得到所述第一残差支路的输出，和/或所述目标神经网络模型用于将所述目标FFN的输出与所述第二权重值进行目标运算，得到所述第二残差支路的输出。其中，目标运算可以为乘积运算。
参照图11，目标神经网络模型可以将所述第一注意力头的输出与所述第一权重值进行乘积运算(*第一权重值)，得到所述第一残差支路的输出，并将第一残差支路的输出输入至加和与归一化层，所述目标神经网络模型可以将所述目标FFN的输出与所述第二权重值进行乘积运算(*第二权重值)，以此得到所述第二残差支路的输出，并将第二残差支路的输出输入至加和与归一化层。
应理解，目标神经网络模型中其他的transformer层都可以参照第一transformer层的描述，假设第l个transformer层的残差支路输出为a l，当前正在处理任务t，相关联的权重值为g t，l，则残差支路经过调整后的输出为a′ l＝g t，l*a l。各个残差支路的权重值之间的组合，可以使得目标神经网络模型适配于不同的任务。
本申请实施例中，在第一注意力头和/或FFN的输出增加了相当于门控模块的机制，来控制transformer层中残差支路的输出大小，门控机制能够对每一个任务学习一个专属的权重值(或者是第一神经网络和/或第二神经网络)的组合。
在一些情况下，第一权重值和第二权重值可以为1或者0。例如，对于比较依赖字符特征的任务，例如文本改错，模型可以只打开底层的门(也就是将靠近嵌入层的transformer层中的残差支路的权重值设置为1，将靠近输出层的transformer层中的残差支路的权重值设置为0)；对于比较依赖语法特征的任务，例如句法分析，模型可以打开中层的门(也就是将既不靠近嵌入层也不靠近输出层的transformer层中的残差支路的权重值设置为1，将靠近嵌入层以及靠近输出层的transformer层中的残差支路的权重值设置为0)；对于比较复杂的任务，例如阅读理解，模型可以打开所有层的门。
本申请实施例中，所述第一transformer层包括多个注意力头，所述多个注意力头中的每一个注意力头对应一个权重值，所述目标神经网络模型用于将所述每一个注意力头的输出与对应的权重值进行目标运算得到所述第一残差支路的输出，其中，不同注意力头对应的权重值不同。示例性的，所述多个注意力头可以包括所述第一注意力头和第二注意力头，所述第一注意力头和第二注意力头为所述多个注意力头中任意两个注意力头，所述目标神经网络模型还用于将所述第二注意力头的输出与第三权重值进行目标运算，所述第一权重值与所述第三权重值不同。本申请实施例中，目标神经网络模型可以包括多头注意力层，针对于每个任务t，每个注意力头n可以都部署一个注意力头的权重值来控制所在的残差支路的输出大小。
参照图12，所述多个注意力头包括所述第一注意力头和第二注意力头，所述目标神经网络模型还用于将所述第二注意力头的输出可以与第三权重值进行乘积运算。
本申请实施例中，在所述目标神经网络模型用于将所述第一注意力头的输出作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行目标运算，得到所述第一残差支路的输出的情况下，所述目标神经网络模型的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度， 所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出，且将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
本申请实施例中，为了使得目标神经网络模型能够适配于不同的任务(也就是说针对于不同的任务都有较高的数据处理能力)，在原有的transformer层的残差支路中增加了权重控制策略，针对于不同的任务设置不同的权重值。
本申请实施例中，获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一转换(transformer)层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN；获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值；根据所述目标神经网络模型对所述待处理数据进行所述目标任务相关的处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与所述第一权重值进行目标运算，得到所述第一残差支路的输出，和/或所述目标神经网络模型用于将所述目标FFN的输出与所述第二权重值进行目标运算，得到所述第二残差支路的输出。针对于不同的任务，设置了用于控制残差支路的输出的权重值，相当于对每个任务可以设置一套专属的分布式表示的权重值组合，从而实现了目标神经网络模型的多任务学习，且本实施例中相比于在输出层增加适配不同任务的神经网络，需要学习的参数较少，进而可以降低终端设备运行目标神经网络模型的计算资源需求。
参照图13，本申请实施例还提供了一种数据处理方法，所述方法包括：
1301、获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN。
关于第一transformer层的描述可以参照步骤601的描述，相似之处不再赘述。
本申请实施例中，所述目标神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。也就是说第二transformer层为多个transformer层中距离嵌入层最远的transformer层。
1302、获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值。
步骤1302的描述可以参照步骤602的描述，相似之处不再赘述。
1303、根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果， 其中所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述目标神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算。
和上述步骤603不同的是，本实施例中，目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，第一输出并不作为第一残差支路的输出，而是将所述第一输出与所述第二transformer层的输出进行第二运算。本实施例中，目标神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，第二输出并不作为第二残差支路的输出，而是将所述第二输出与所述第二transformer层的输出进行第二运算。所述第一运算可以包括乘积运算，和/或，所述第二运算可以包括加和运算。
参照图14，所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行乘积运算(*第一权重值)，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行加和运算；和/或，所述目标神经网络模型用于将所述目标FFN与所述第二权重值进行乘积运算(*第二权重值)，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行加和运算。
应理解，目标神经网络模型可以按照图6对应的实施例中的那样，一方面，将所述第一注意力头的输出与第一权重值进行目标运算(例如乘积运算)，并将运算结果作为第一残差支路的输出，另一方面，也将所述第一输出与所述第二transformer层的输出进行加和运算。类似的，目标神经网络模型可以按照图6对应的实施例中的那样，一方面，将目标FFN与所述第二权重值进行目标运算(例如乘积运算)，并将运算结果作为第二残差支路的输出，另一方面，也将所述第二输出与所述第二transformer层的输出进行加和运算。
在一种可能的实现中，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括所述第一注意力头；其中，
在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算的情况下，所述目标神经网络模型的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出仅作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度，所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络 模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，且在将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出，且将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
在一种可能的实现中，所述第一权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第二权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN与所述第二初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，根据预设的映射关系获取所述目标任务对应的权重值，所述目标权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系，且在所述预设的映射关系中，所述目标任务对应于所述目标权重值。
在一种可能的实现中，获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算；根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型。
在一种可能的实现中，训练设备可以根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。
在一种可能的实现中，可以将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值。
在一种可能的实现中，所述第一神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到所述第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第二神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到所述第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，训练设备可以获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算；根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
在一种可能的实现中，训练设备可以根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始神经网络和/或所述第二初始神经网络，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始神经网络和/或所述第二初始神经网络的网络参数，以得到目标神经网络模型。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头，相应的，所述目标神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行第一运算，得到第三输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
应理解，图13对应的实施例可以参照与图6对应的实施例的描述，相似之处这里不再 赘述。
从模型训练的角度，本申请实施例还提供了一种数据处理方法，包括：
获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述目标FFN的输出与第二初始权重值进行目标运算，得到所述第二残差支路的输出；
根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
在一种可能的实现中，所述根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型，包括：
根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。
本申请实施例还提供了一种数据处理方法，包括：
获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一初始神经网络，并将所述第一初始神经网络的输出与所述第一注意力头的输出进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行目标运算，得到所述第二残差支路的输出；
根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
在一种可能的实现中，所述根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型，包括：
根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始神经网络和/或所述第二初始神经网络，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始神经网络和/或所述第二初始神经网络的网络参数，以得到目标神经网络模型。
本申请实施例还提供了一种数据处理方法，包括：
获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算；
根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型。
在一种可能的实现中，所述初始神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型，包括：
根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。
在一种可能的实现中，所述第一运算包括乘积运算，和/或，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
本申请实施例还提供了一种数据处理方法，包括：
获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算；
根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
在一种可能的实现中，所述初始神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型，包括：
根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始神经网络和/或所述第二初始神经网络，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始神经网络和/或所述第二初始神经网络的网络参数，以得到目标神经网络模型。
在一种可能的实现中，所述第一运算包括乘积运算，和/或，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
在图1至图14所对应的实施例的基础上，为了更好的实施本申请实施例的上述方案，下面还提供用于实施上述方案的相关设备。具体参阅图15，图15为本申请实施例提供的数据处理设备1500的一种结构示意图，数据处理设备1500可以是终端设备或服务器，数据处理设备1500包括：
获取模块1501，用于获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一转换(transformer)层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN；获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值；
数据处理模块1502，用于根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行目标运算，得到所述第一残差支路的输出，和/或所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出。
在一种可能的实现中，在所述目标神经网络模型用于将所述第一注意力头的输出作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行目标运算，得到所述第一残差支路的输出的情况下，所述目标神经网络模型的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度，所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出，且将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
在一种可能的实现中，所述第一权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与所述第一初始权重值进行目标运算，得到所述第一残差支路的输出。
在一种可能的实现中，所述第二权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN的输出与所述第二初始权重值进行目标运算，得到所述第二残差支路的输出。
在一种可能的实现中，所述获取模块用于：
根据预设的映射关系获取所述目标任务对应的权重值，所述目标任务对应的权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系。
在一种可能的实现中，所述获取模块用于：
将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，
将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值。
在一种可能的实现中，所述第一神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到所述第一初始神经网络，并将所述第一初始神经网络的输出与所述第一注意力头的输出进行目标运算，得到所述第一残差支路的输出。
在一种可能的实现中，所述第二神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到所述第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行目标运算，得到所述第二残差支路的输出。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头中的每一个注意力头对应一个权重值，所述目标神经网络模型用于将所述每一个注意力头的输出与对应的权重值进行目标运算得到所述第一残差支路的输出，其中，不同注意力头对应的权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
数据处理设备1500的具体描述可以参照图6对应的实施例的描述，其中，获取模块1501可以执行步骤601、步骤602以及对应的实施例中的描述，数据处理模块1502可以执行步骤603以及对应的实施例中的描述。
参阅图16，图16为本申请实施例提供的数据处理设备1600的一种结构示意图，数据处理设备1600可以是终端设备或服务器，数据处理设备1600包括：
获取模块1601，用于获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值；
数据处理模块1602，用于根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述目标神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述目标神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括所述第一注意力头，所述第二残差支路包括所述目标FFN；其中，
在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算的情况下，所述目标神经网络模型的数据处理精度；或，
在所述目标神经网络模型用于将所述目标FFN的输出仅作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度，所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，且在将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出，且将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
在一种可能的实现中，所述第一权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第二权重值为对所述目标神经网络模型进行针对于所述目 标任务的训练时对第二初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN与所述第二初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述获取模块用于：
根据预设的映射关系获取所述目标任务对应的权重值，所述目标任务对应的权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系。
在一种可能的实现中，所述获取模块用于：
将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，
将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值。
在一种可能的实现中，所述第一神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到所述第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第二神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到所述第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头中的每一个注意力头对应一个权重值，相应的，所述目标神经网络模型用于将每一个注意力头的输出与对应的权重值进行第一运算，得到第三输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，其中，不同注意力头对应的权重值不同。
在一种可能的实现中，所述第一运算包括乘积运算，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
数据处理设备1600的具体描述可以参照图13对应的实施例的描述，其中，获取模块1601可以执行步骤1301、步骤1302以及对应的实施例中的描述，数据处理模块1602可以执行步骤1303以及对应的实施例中的描述。
参阅图17，图17为本申请实施例提供的数据处理设备1700的一种结构示意图，数据处理设备1700可以是终端设备或服务器，数据处理设备1700包括：
获取模块1701，用于获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述目标FFN的输出与第二初始权重值进行目标运算，得到所述第二残差支路的输出；
数据处理模块1702，用于根据所述初始神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
模型更新模块1703，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
在一种可能的实现中，所述模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。
参阅图18，图18为本申请实施例提供的数据处理设备1800的一种结构示意图，数据处理设备1800可以是终端设备或服务器，数据处理设备1800包括：
获取模块1801，用于获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN，所述初始神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一初始神经网络，并将所述第一初始神经网络的输出与所述第一注意力头的输出进行目标运算，得到所述第一残差支路的输出，和/或所述初始神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行目标运算，得到所述第二残差支路的输出；
数据处理模块1802，用于根据所述初始神经网络模型对所述待处理数据进行处理，以 得到数据处理结果；
模型更新模块1803，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述目标运算包括乘积运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
在一种可能的实现中，所述模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始神经网络和/或所述第二初始神经网络，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始神经网络和/或所述第二初始神经网络的网络参数，以得到目标神经网络模型。
参阅图19，图19为本申请实施例提供的数据处理设备1900的一种结构示意图，数据处理设备1900可以是终端设备或服务器，数据处理设备1900包括：
获取模块1901，用于获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算；
数据处理模块1902，用于根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
模型更新模块1903，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始权重值和/或所述第二初始权重值，以得到目标神经网络模型。
在一种可能的实现中，所述初始神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始权重值和/或所述第二初始权重值，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始权重值和/或所述第二初始权重值的网络参数，以得到目标神经网络模型。
在一种可能的实现中，所述第一运算包括乘积运算，和/或，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
参阅图20，图20为本申请实施例提供的数据处理设备2000的一种结构示意图，数据处理设备2000可以是终端设备或服务器，数据处理设备2000包括：
获取模块2001，用于获取待处理数据、所述待处理数据的正确数据处理结果以及用于执行目标任务的初始神经网络模型，所述初始神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；所述初始神经网络模型用于将所述待处理数据和/或所述第一注意力头的输出、以及所述目标任务的标识输入到第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算；和/或，所述初始神经网络模型用于将所述待处理数据和/或所述目标FFN的输出、以及所述目标任务的标识输入到第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算；
数据处理模块2002，用于根据所述目标神经网络模型对所述待处理数据进行处理，以得到数据处理结果；
模型更新模块2003，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并基于损失更新所述第一初始神经网络和/或所述第二初始神经网络，以得到目标神经网络模型。
在一种可能的实现中，所述初始神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
在一种可能的实现中，所述第一transformer层包括多个注意力头，所述多个注意力头包括所述第一注意力头和第二注意力头；相应的，所述初始神经网络模型还用于将所述第二注意力头的输出与所述第三权重值进行目标运算，其中所述第一权重值与所述第三权重值不同。
在一种可能的实现中，所述模型更新模块，用于根据所述数据处理结果和所述正确数据处理结果，获取损失，并在第i次迭代的过程中，基于损失仅更新所述第一初始神经网络和/或所述第二初始神经网络，以得到第一神经网络模型，在第i+1次迭代的过程中，基于损失更新所述第一神经网络模型中除所述第一初始神经网络和/或所述第二初始神经网络的网络参数，以得到目标神经网络模型。
在一种可能的实现中，所述第一运算包括乘积运算，和/或，所述第二运算包括加和运算。
在一种可能的实现中，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
接下来介绍本申请实施例提供的一种执行设备，请参阅图21，图21为本申请实施例提供的执行设备的一种结构示意图，执行设备2100具体可以表现为虚拟现实VR设备、手机、平板、笔记本电脑、智能穿戴设备、监控数据处理设备或服务器等，此处不做限定。具体的，执行设备2100包括：接收器2101、发射器2102、处理器2103和存储器2104(其中执行设备2100中的处理器2103的数量可以一个或多个，图21中以一个处理器为例)，其中，处理器2103可以包括应用处理器21031和通信处理器21032。在本申请的一些实施例中，接收器2101、发射器2102、处理器2103和存储器2104可通过总线或其它方式连接。
存储器2104可以包括只读存储器和随机存取存储器，并向处理器2103提供指令和数据。存储器2104的一部分还可以包括非易失性随机存取存储器(non-volatile random access memory，NVRAM)。存储器2104存储有处理器和操作指令、可执行模块或者数据结构，或者它们的子集，或者它们的扩展集，其中，操作指令可包括各种操作指令，用于实现各种操作。
处理器2103控制执行设备的操作。具体的应用中，执行设备的各个组件通过总线系统耦合在一起，其中总线系统除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都称为总线系统。
上述本申请实施例揭示的方法可以应用于处理器2103中，或者由处理器2103实现。处理器2103可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器2103中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器2103可以是通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器，还可进一步包括专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。该处理器2103可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器2104，处理器2103读取存储器2104中的信息，结合其硬件完成上述方法的步骤。
接收器2101可用于接收输入的数字或字符信息，以及产生与执行设备的相关设置以及功能控制有关的信号输入。发射器2102可用于通过第一接口输出数字或字符信息；发射器2102还可用于通过第一接口向磁盘组发送指令，以修改磁盘组中的数据；发射器2102还可以包括显示屏等显示设备。
本申请实施例中，在一种情况下，处理器2103，用于执行图6、图13对应实施例中的设备执行的数据处理方法。
本申请实施例还提供了一种训练设备，请参阅图22，图22是本申请实施例提供的训练设备一种结构示意图，训练设备2200上可以部署有图17至图20对应实施例中所描述的数据处理装置，具体的，训练设备2200由一个或多个服务器实现，训练设备2200可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上中央处理器(central processing units，CPU)2222(例如，一个或一个以上处理器)和存储器2232，一个或一个以上存储应用程序2242或数据2244的存储介质2230(例如一个或一个以上海量存储设备)。其中，存储器2232和存储介质2230可以是短暂存储或持久存储。存储在存储介质2230的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对训练设备中的一系列指令操作。更进一步地，中央处理器2222可以设置为与存储介质2230通信，在训练设备2200上执行存储介质2230中的一系列指令操作。
训练设备2200还可以包括一个或一个以上电源2226，一个或一个以上有线或无线网络接口2250，一个或一个以上输入输出接口2258；或，一个或一个以上操作系统2241，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM等等。
本申请实施例中，中央处理器2222，用于执行图18对应实施例中的数据处理装置执行的数据处理方法。
本申请实施例中还提供一种包括计算机程序产品，当其在计算机上运行时，使得计算机执行如前述执行设备所执行的步骤，或者，使得计算机执行如前述训练设备所执行的步骤。
本申请实施例中还提供一种计算机可读存储介质，该计算机可读存储介质中存储有用于进行信号处理的程序，当其在计算机上运行时，使得计算机执行如前述执行设备所执行的步骤，或者，使得计算机执行如前述训练设备所执行的步骤。
本申请实施例提供的执行设备、训练设备或终端设备具体可以为芯片，芯片包括：处理单元和通信单元，所述处理单元例如可以是处理器，所述通信单元例如可以是输入/输出接口、管脚或电路等。该处理单元可执行存储单元存储的计算机执行指令，以使执行设备内的芯片执行上述实施例描述的数据处理方法，或者，以使训练设备内的芯片执行上述实施例描述的数据处理方法。可选地，所述存储单元为所述芯片内的存储单元，如寄存器、缓存等，所述存储单元还可以是所述无线接入设备端内的位于所述芯片外部的存储单元，如只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)等。
具体的，请参阅图23，图23为本申请实施例提供的芯片的一种结构示意图，所述芯片可以表现为神经网络处理器NPU 2300，NPU 2300作为协处理器挂载到主CPU(Host CPU)上，由Host CPU分配任务。NPU的核心部分为运算电路2303，通过控制器2304控制运算电路2303提取存储器中的矩阵数据并进行乘法运算。
在一些实现中，运算电路2303内部包括多个处理单元(Process Engine,PE)。在一些实现中，运算电路2303是二维脉动阵列。运算电路2303还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路2303是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路从权重存储器2302中取矩阵B相应的数据，并缓存在运算电路中每一个PE上。运算电路从输入存储器2301中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)2308中。
统一存储器2306用于存放输入数据以及输出数据。权重数据直接通过存储单元访问控制器(Direct Memory Access Controller，DMAC)2305，DMAC被搬运到权重存储器2302中。输入数据也通过DMAC被搬运到统一存储器2306中。
BIU为Bus Interface Unit即，总线接口单元2310，用于AXI总线与DMAC和取指存储器(Instruction Fetch Buffer，IFB)2309的交互。
总线接口单元2310(Bus Interface Unit，简称BIU)，用于取指存储器2309从外部存储器获取指令，还用于存储单元访问控制器2305从外部存储器获取输入矩阵A或者权重矩阵B的原数据。
DMAC主要用于将外部存储器DDR中的输入数据搬运到统一存储器2306或将权重数据搬运到权重存储器2302中或将输入数据数据搬运到输入存储器2301中。
向量计算单元2307包括多个运算处理单元，在需要的情况下，对运算电路的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。主要用于神经网络中非卷积/全连接层网络计算，如Batch Normalization(批归一化)，像素级求和，对特征平面进行上采样等。
在一些实现中，向量计算单元2307能将经处理的输出的向量存储到统一存储器2306。例如，向量计算单元2307可以将线性函数；或，非线性函数应用到运算电路2303的输出，例如对卷积层提取的特征平面进行线性插值，再例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元2307生成归一化的值、像素级求和的值，或二者均有。在一些实现中，处理过的输出的向量能够用作到运算电路2303的激活输入，例如用于在神经网络中的后续层中的使用。
控制器2304连接的取指存储器(instruction fetch buffer)2309，用于存储控制器2304使用的指令；
统一存储器2306，输入存储器2301，权重存储器2302以及取指存储器2309均为On-Chip存储器。外部存储器私有于该NPU硬件架构。
其中，上述任一处提到的处理器，可以是一个通用中央处理器，微处理器，ASIC，或一个或多个用于控制上述程序执行的集成电路。
另外需说明的是，以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际 的需要选择其中的部分或者全部模块来实现本实施例方案的目的。另外，本申请提供的装置实施例附图中，模块之间的连接关系表示它们之间具有通信连接，具体可以实现为一条或多条通信总线或信号线。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到本申请可借助软件加必需的通用硬件的方式来实现，当然也可以通过专用硬件包括专用集成电路、专用CPU、专用存储器、专用元器件等来实现。一般情况下，凡由计算机程序完成的功能都可以很容易地用相应的硬件来实现，而且，用来实现同一功能的具体硬件结构也可以是多种多样的，例如模拟电路、数字电路或专用电路等。但是，对本申请而言更多情况下软件程序实现是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在可读取的存储介质中，如计算机的软盘、U盘、移动硬盘、ROM、RAM、磁碟或者光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，训练设备，或者网络设备等)执行本申请各个实施例所述的方法。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。
所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、训练设备或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、训练设备或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存储的任何可用介质或者是包含一个或多个可用介质集成的训练设备、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘(Solid State Disk，SSD))等。

Claims (25)

  1. 一种数据处理方法，其特征在于，所述方法包括：
    获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一转换(transformer)层，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括第一注意力头，所述第二残差支路包括目标前馈层FFN；
    获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值；
    根据所述目标神经网络模型对所述待处理数据进行所述目标任务相关的处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与所述第一权重值进行目标运算，得到所述第一残差支路的输出，和/或所述目标神经网络模型用于将所述目标FFN的输出与所述第二权重值进行目标运算，得到所述第二残差支路的输出。
  2. 根据权利要求1所述的方法，其特征在于，
    在所述目标神经网络模型用于将所述第一注意力头的输出作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行目标运算，得到所述第一残差支路的输出的情况下，所述目标神经网络模型的数据处理精度；或，
    在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度，所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
    在所述目标神经网络模型用于将所述目标FFN的输出作为所述第二残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出，且将所述目标FFN的输出与第二权重值进行目标运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
  3. 根据权利要求2所述的方法，其特征在于，所述第一权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与所述第一初始权重值进行目标运算，得到所述第一残差支路的输出。
  4. 根据权利要求2或3所述的方法，其特征在于，所述第二权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始权重值进行更新得到的，其中，在 对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN的输出与所述第二初始权重值进行目标运算，得到所述第二残差支路的输出。
  5. 根据权利要求2至4任一所述的方法，其特征在于，所述获取目标任务对应的权重值，包括：
    根据预设的映射关系获取所述目标任务对应的权重值，所述目标任务对应的权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系。
  6. 根据权利要求2所述的方法，其特征在于，所述获取目标任务对应的权重值，包括：
    将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，
    将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值。
  7. 根据权利要求6所述的方法，其特征在于，所述第一神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到所述第一初始神经网络，并将所述第一初始神经网络的输出与所述第一注意力头的输出进行目标运算，得到所述第一残差支路的输出。
  8. 根据权利要求6或7所述的方法，其特征在于，所述第二神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到所述第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行目标运算，得到所述第二残差支路的输出。
  9. 根据权利要求1至8任一所述的方法，其特征在于，所述第一transformer层包括多个注意力头，所述多个注意力头中的每一个注意力头对应一个权重值，所述目标神经网络模型用于将所述每一个注意力头的输出与对应的权重值进行目标运算得到所述第一残差支路的输出，其中，不同注意力头对应的权重值不同。
  10. 根据权利要求1至9任一所述的方法，其特征在于，所述目标运算包括乘积运算。
  11. 根据权利要求2至10任一所述的方法，其特征在于，所述目标任务包括如下的一 种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
  12. 一种数据处理方法，其特征在于，所述方法包括：
    获取待处理数据以及目标神经网络模型，所述目标神经网络模型包括第一transformer层以及第二transformer层，所述第一transformer层包括第一注意力头和目标FFN；
    获取目标任务对应的权重值，所述权重值包括所述第一注意力头对应的第一权重值和/或所述目标FFN对应的第二权重值；
    根据所述目标神经网络模型对所述待处理数据进行所述目标任务相关的处理，以得到数据处理结果，其中所述目标神经网络模型用于将所述第一注意力头的输出与所述第一权重值进行第一运算，得到第一输出，并将所述第一输出与所述第二transformer层的输出进行第二运算；和/或，所述目标神经网络模型用于将所述目标FFN与所述第二权重值进行第一运算，得到第二输出，并将所述第二输出与所述第二transformer层的输出进行第二运算。
  13. 根据权利要求12所述的方法，其特征在于，所述目标神经网络模型包括多个transformer层以及输出层，所述第二transformer层为所述多个transformer层中距离所述输出层最近的transformer层。
  14. 根据权利要求12或13所述的方法，其特征在于，所述第一transformer层包括第一残差支路和第二残差支路，所述第一残差支路包括所述第一注意力头；其中，
    在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第一处理精度，所述第一处理精度为在所述目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算的情况下，所述目标神经网络模型的数据处理精度；或，
    在所述目标神经网络模型用于将所述目标FFN的输出仅作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第二处理精度，所述第二处理精度为在所述目标神经网络模型用于将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度；或，
    在所述目标神经网络模型用于将所述第一注意力头的输出仅作为所述第一残差支路的输出且将所述目标FFN的输出作为所述第二残差支路的输出的情况下，所述目标神经网络模型针对于所述目标任务的数据处理精度小于第三处理精度，所述第三处理精度为在所述目标神经网络模型用于所述目标神经网络模型用于将所述第一注意力头的输出与第一权重值进行第一运算，得到第一输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，且在将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支 路的输出，且将所述目标FFN的输出与第二权重值进行第一运算，得到所述第二残差支路的输出的情况下，所述目标神经网络模型针的数据处理精度。
  15. 根据权利要求12至14任一所述的方法，其特征在于，所述第一权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述第一注意力头的输出与第一初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
  16. 根据权利要求12至15任一所述的方法，其特征在于，所述第二权重值为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始权重值进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述目标FFN与所述第二初始权重值进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
  17. 根据权利要求13至16任一所述的方法，其特征在于，所述获取目标任务对应的权重值，包括：
    根据预设的映射关系获取所述目标任务对应的权重值，所述目标任务对应的权重值包括所述第一权重值和/或所述第二权重值；其中，所述预设的映射关系包括任务与权重值之间的对应关系。
  18. 根据权利要求13所述的方法，其特征在于，所述获取目标任务对应的权重值，包括：
    将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到第一神经网络，得到所述第一权重值；和/或，
    将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到第二神经网络，得到所述第二权重值。
  19. 根据权利要求18所述的方法，其特征在于，所述第一神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第一初始神经网络进行更新得到的，其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述第一注意力头的输出中的至少一项，以及所述目标任务的标识输入到所述第一初始神经网络，将所述第一初始神经网络的输出与所述第一注意力头的输出进行第一运算，并将运算结果与所述第二transformer层的输出进行第二运算。
  20. 根据权利要求18或19所述的方法，其特征在于，所述第二神经网络为对所述目标神经网络模型进行针对于所述目标任务的训练时对第二初始神经网络进行更新得到的， 其中，在对所述目标神经网络模型的训练过程中，所述目标神经网络模型用于将所述待处理数据和所述目标FFN的输出中的至少一项，以及所述目标任务的标识输入到所述第二初始神经网络，并将所述第二初始神经网络的输出与所述目标FFN的输出进行第一运算，将运算结果与所述第二transformer层的输出进行第二运算。
  21. 根据权利要求12至20任一所述的方法，其特征在于，所述第一transformer层包括多个注意力头，所述多个注意力头中的每一个注意力头对应一个权重值，相应的，所述目标神经网络模型用于将每一个注意力头的输出与对应的权重值进行第一运算，得到第三输出，并将所述第三输出与所述第二transformer层的输出进行第二运算，其中，不同注意力头对应的权重值不同。
  22. 根据权利要求12至21任一所述的方法，其特征在于，所述第一运算包括乘积运算，所述第二运算包括加和运算。
  23. 根据权利要求12至22任一所述的方法，其特征在于，所述目标任务包括如下的一种：阅读理解、文本翻译、复述识别、命名实体识别、文本情感分析、自然语言推理、文本自动问答、文本意图识别、文本分类、文本简化以及文本故事生成。
  24. 一种数据处理装置，其特征在于，所述装置包括存储器和处理器；所述存储器存储有代码，所述处理器被配置为获取所述代码，并执行如权利要求1至23任一所述的方法。
  25. 一种计算机存储介质，其特征在于，所述计算机存储介质存储有一个或多个指令，所述指令在由一个或多个计算机执行时使得所述一个或多个计算机实施权利要求1至23任一所述的方法。
PCT/CN2021/119306 2020-09-29 2021-09-18 一种数据处理方法及相关设备 WO2022068627A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP21874288.0A EP4209965A4 (en) 2020-09-29 2021-09-18 DATA PROCESSING METHOD AND ASSOCIATED DEVICE
US18/186,942 US20230229898A1 (en) 2020-09-29 2023-03-20 Data processing method and related device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011052624.X 2020-09-29
CN202011052624.XA CN112288075B (zh) 2020-09-29 2020-09-29 一种数据处理方法及相关设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/186,942 Continuation US20230229898A1 (en) 2020-09-29 2023-03-20 Data processing method and related device

Publications (1)

Publication Number Publication Date
WO2022068627A1 true WO2022068627A1 (zh) 2022-04-07

Family

ID=74422575

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/119306 WO2022068627A1 (zh) 2020-09-29 2021-09-18 一种数据处理方法及相关设备

Country Status (4)

Country Link
US (1) US20230229898A1 (zh)
EP (1) EP4209965A4 (zh)
CN (1) CN112288075B (zh)
WO (1) WO2022068627A1 (zh)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112288075B (zh) * 2020-09-29 2024-02-02 华为技术有限公司 一种数据处理方法及相关设备
CN113029155A (zh) * 2021-04-02 2021-06-25 杭州申昊科技股份有限公司 机器人自动导航方法及装置、电子设备、存储介质
CN113051938B (zh) * 2021-04-02 2022-03-01 四川大学 一种基于Transformer模型的机器翻译模型优化方法
CN113569055B (zh) * 2021-07-26 2023-09-22 东北大学 基于遗传算法优化神经网络的露天矿知识图谱构建方法
CN113704511B (zh) * 2021-07-30 2022-11-22 北京达佳互联信息技术有限公司 多媒体资源的推荐方法、装置、电子设备及存储介质
CN115713079A (zh) * 2021-08-18 2023-02-24 北京京东方技术开发有限公司 用于自然语言处理、训练自然语言处理模型的方法及设备
CN116579403A (zh) * 2022-01-29 2023-08-11 华为技术有限公司 一种数据处理方法及相关设备
CN114626517B (zh) * 2022-05-17 2022-07-29 中科南京智能技术研究院 一种存储双层权重数据的脉动阵列
CN114925668B (zh) * 2022-05-20 2023-04-07 电子科技大学 自动生成新闻的系统、方法及存储介质
CN115392218B (zh) * 2022-07-15 2023-06-20 哈尔滨工业大学 一种预训练语言模型的构建方法及系统
CN116759077A (zh) * 2023-08-18 2023-09-15 北方健康医疗大数据科技有限公司 一种基于智能体的医疗对话意图识别方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108665065A (zh) * 2018-04-25 2018-10-16 清华大学 任务数据的处理方法、装置、设备和存储介质
CN110349676A (zh) * 2019-06-14 2019-10-18 华南师范大学 时序生理数据分类方法、装置、存储介质和处理器
CN110807517A (zh) * 2019-10-30 2020-02-18 智者四海（北京）技术有限公司 用于多任务识别的神经网络系统
US20200134454A1 (en) * 2018-10-30 2020-04-30 Samsung Sds Co., Ltd. Apparatus and method for training deep learning model
CN111368993A (zh) * 2020-02-12 2020-07-03 华为技术有限公司 一种数据处理方法及相关设备
CN112288075A (zh) * 2020-09-29 2021-01-29 华为技术有限公司 一种数据处理方法及相关设备

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108549646B (zh) * 2018-04-24 2022-04-15 中译语通科技股份有限公司 一种基于胶囊的神经网络机器翻译系统、信息数据处理终端
CN109543824B (zh) * 2018-11-30 2023-05-23 腾讯科技（深圳）有限公司 一种序列模型的处理方法和装置
CN110263324B (zh) * 2019-05-16 2021-02-12 华为技术有限公司 文本处理方法、模型训练方法和装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108665065A (zh) * 2018-04-25 2018-10-16 清华大学 任务数据的处理方法、装置、设备和存储介质
US20200134454A1 (en) * 2018-10-30 2020-04-30 Samsung Sds Co., Ltd. Apparatus and method for training deep learning model
CN110349676A (zh) * 2019-06-14 2019-10-18 华南师范大学 时序生理数据分类方法、装置、存储介质和处理器
CN110807517A (zh) * 2019-10-30 2020-02-18 智者四海（北京）技术有限公司 用于多任务识别的神经网络系统
CN111368993A (zh) * 2020-02-12 2020-07-03 华为技术有限公司 一种数据处理方法及相关设备
CN112288075A (zh) * 2020-09-29 2021-01-29 华为技术有限公司 一种数据处理方法及相关设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4209965A4

Also Published As

Publication number Publication date
CN112288075B (zh) 2024-02-02
CN112288075A (zh) 2021-01-29
EP4209965A4 (en) 2024-02-28
US20230229898A1 (en) 2023-07-20
EP4209965A1 (en) 2023-07-12

Similar Documents

Publication Publication Date Title
WO2022068627A1 (zh) 一种数据处理方法及相关设备
WO2022057776A1 (zh) 一种模型压缩方法及装置
WO2022007823A1 (zh) 一种文本数据处理方法及装置
WO2021047286A1 (zh) 文本处理模型的训练方法、文本处理方法及装置
WO2021159714A1 (zh) 一种数据处理方法及相关设备
WO2020228376A1 (zh) 文本处理方法、模型训练方法和装置
WO2022068314A1 (zh) 神经网络训练的方法、神经网络的压缩方法以及相关设备
WO2023160472A1 (zh) 一种模型训练方法及相关设备
WO2022001724A1 (zh) 一种数据处理方法及装置
WO2023236977A1 (zh) 一种数据处理方法及相关设备
WO2022253074A1 (zh) 一种数据处理方法及相关设备
WO2023284716A1 (zh) 一种神经网络搜索方法及相关设备
EP4379603A1 (en) Model distillation method and related device
CN116432019A (zh) 一种数据处理方法及相关设备
US20240046067A1 (en) Data processing method and related device
WO2020192523A1 (zh) 译文质量检测方法、装置、机器翻译系统和存储介质
WO2021129411A1 (zh) 文本处理方法及装置
WO2023143262A1 (zh) 一种数据处理方法及相关设备
US20240185086A1 (en) Model distillation method and related device
WO2023207665A1 (zh) 一种数据处理方法及相关设备
CN115795025A (zh) 一种摘要生成方法及其相关设备
CN117035019A (zh) 一种数据处理方法及相关设备
CN114492447A (zh) 一种文本数据处理方法和智能设备
CN115525263A (zh) 代码补全模型的训练方法、代码补全方法和装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21874288

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021874288

Country of ref document: EP

Effective date: 20230403

NENP Non-entry into the national phase

Ref country code: DE