CN117708307B - Method and device for fusing micro-tuning and Adapter of large language model - Google Patents

Abstract

The invention discloses a large language model tuning and Adapter fusion method and device, and relates to the field of deep learning. The method is used for solving the problems of large consumption and poor data quality caused by the fact that the data collection is required to be manually carried out in the construction of the existing multi-mode data set. The method comprises the following steps: collecting a plurality of question-answer data sets and dialogue data sets from a set network platform; performing LoRA-adapter fine tuning on the question-answer data set and the dialogue data set respectively to sequentially obtain a question-answer large language model, a question-answer negative log-likelihood loss function, a dialogue large language model and a dialogue negative log-likelihood loss function; obtaining an ideal loss function, ideal fusion weight and a first ideal parameter of the question-answer data set and the dialogue data set in an ideal state; obtaining optimal parameters of the questioning and answering LoRA-adapter, optimal parameters of the dialogue LoRA-adapter and optimal fusion parameters; and obtaining the general LORA-adapter according to the optimal parameters of the questioning and answering LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.

Description

A large language model fine-tuning and adapter fusion method and device

Technical Field

The present invention relates to the field of deep learning, and more specifically to a large language model fine-tuning and adapter fusion method and device.

Background technique

Training large language models has important scientific research and application value. It can improve the performance of natural language processing tasks, improve the interactive experience of dialogue systems, and promote the inclusiveness of scientific research, technological innovation, and artificial intelligence development. Large language models can learn rich language knowledge and grammatical rules by training massive corpora, so as to show better performance in natural language processing tasks such as machine translation, text generation, and text classification. These models can understand and generate more accurate and fluent natural language. Large language models can be used to build intelligent dialogue systems to provide more natural, accurate, and personalized responses through dialogues with users. The trained models can understand and generate human language, so as to better meet the needs of users and improve the interactive experience of dialogue systems. Training large language models requires processing massive amounts of data and huge computing resources, which is of great significance for promoting scientific research and technological innovation. In the process of training large language models, many technical challenges need to be solved, such as data processing, model design, training algorithms, etc. The solution of these challenges has a positive role in promoting research and development in related fields.

The conventional approach currently used for large language model training is to collect a large amount of instruction fine-tuning data, fuse it to build a large-scale dataset, and fine-tune the open source large language model on this dataset. However, it seems impossible to fuse multiple datasets to build a multi-functional dataset. On the one hand, there may be contradictions between different datasets, and it is difficult to evaluate the quality of the data; on the other hand, these datasets are composed of instances of various specific tasks, such as mathematics, coding, role-playing, writing, etc. If these datasets are mixed and fine-tuned on this fused dataset, the performance of the large language model may decline, or even seriously decline.

Summary of the invention

The embodiment of the present invention provides a large language model fine-tuning and adapter fusion method and device, which can prevent the problem of performance degradation caused by conflicts between different data sets in the semantic space.

The embodiment of the present invention provides a large language model fine-tuning and adapter fusion method, including:

Collect multiple question-answering datasets and dialogue datasets from a set network platform, perform LoRA-adapter fine-tuning on the question-answering datasets and the dialogue datasets respectively, and obtain a question-answering large language model, a question-answering negative log-likelihood loss function, a dialogue large language model, and a dialogue negative log-likelihood loss function in turn;

According to the question-answering negative log-likelihood loss function, the dialogue negative log-likelihood loss function and the initial fusion weight included in the fine-tuning based on each LoRA-adapter, the ideal loss function of the question-answering dataset and the dialogue dataset in an ideal state is obtained, and the ideal fusion weight and the first ideal parameter corresponding to the ideal loss function are obtained according to the minimum value of the ideal loss function; wherein the first ideal parameter represents all LoRA-adapters added to the question-answering large language model and the dialogue large language model respectively;

According to the ideal loss function, fine-tune the question-answer LoRA-adapter corresponding to each question-answer data set and the conversation LoRA-adapter corresponding to each conversation data set to obtain the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the conversation LoRA-adapter and the optimal fusion parameters respectively;

A universal LoRA-adapter is obtained according to the optimal parameters of the question-and-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.

Preferably, the LoRA-adapter is fine-tuned on the question-answering dataset to obtain a question-answering large language model and a question-answering negative log-likelihood loss function in sequence, specifically including:

The question-answering data set is trained to obtain a question-answering LoRA-adapter, and the question-answering large language model is obtained according to the question-answering LoRA-adapter and the question-answering data set;

Obtaining the question-answering negative log-likelihood loss function according to the question-answering large language model and the token of the question-answering large language model;

The question-answering dataset, the question-answering large language model, and the question-answering negative log-likelihood loss function are as follows:

Where _Qi represents the i-th question-answering dataset, si _,j represents the j-th system information of the i-th question-answering dataset, qi _,j represents the j-th question of the i-th question-answering dataset, ri _,j represents the j-th reply of the i-th question-answering dataset, | _Qi | represents the length of the question-answering dataset _Qi , represents the question-answering LoRA-adapter trained on the question-answering dataset _Qi , | _ri,j | represents the length of ri _,j , _rk represents the kth token generated by the large language model, _pθ represents the large language model, θ represents the frozen parameters of the large language model,/> represents the question-answering negative log-likelihood loss function.

Preferably, the LoRA-adapter fine-tuning of the conversation dataset to obtain a large conversation language model and a conversation negative log-likelihood loss function in sequence specifically includes:

Training the conversation data set to obtain a conversation LoRA-adapter, and obtaining a conversation large language model according to the conversation LoRA-adapter and the conversation data set;

Obtaining the conversation negative log-likelihood loss function according to the conversation large language model and the token of the conversation large language model;

The dialogue dataset, the dialogue large language model and the dialogue negative log-likelihood loss function are as follows:

Among them, _Ci represents the i-th dialogue dataset, represents the jth query of the i-th dialogue dataset in the T-th round, represents the jth reply of the ith dialogue dataset in the Tth round, |C _i | represents the length of the dialogue dataset C _i , /> represents the dialog LoRA-adapter trained on the dialog dataset _Ci , _Qj represents all tags belonging to the user query, _Rj represents the target tag, /> Indicates the number of tokens contained in the jth data in the conversation dataset _Ci ,/> represents the dialogue negative log-likelihood loss function, p _θ represents the large language model, and θ represents the frozen parameters of the large language model.

Preferably, the ideal loss function is as follows:

The minimum value of the ideal loss function is as follows:

Where L represents the ideal loss function, Denotes fine-tuning on the question-answering dataset _Qi /> The initial fusion weight of Represents fine-tuning on the dialogue dataset _Ci to obtain/> The initial fusion weight of , A ^* represents all the first ideal parameters, ω ^* represents all the ideal fusion weights, A represents the first ideal parameter, and ω represents the ideal fusion weight.

Preferably, the first ideal parameter is as follows:

The ideal fusion weights are as follows:

Where A represents the first ideal parameter, represents the first ideal parameter obtained by fine-tuning on the question-answering dataset Q _M , represents the first ideal parameter obtained by fine-tuning on the dialogue dataset _CN , M represents the number of question-answering datasets, N represents the number of dialogue datasets, ω represents the ideal fusion weight,/> Indicates/> The ideal fusion weight of Indicates/> The ideal fusion weights.

Preferably, the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters are as follows:

in, represents the optimal parameters of the question-answering LoRA-adapter,/> represents the best parameter of the dialogue LoRA-adapter, ω ^** represents the best fusion parameter, /> represents the question-answering negative log-likelihood loss function,/> represents the negative log-likelihood loss function of the conversation,/> represents the initial fusion weight obtained by fine-tuning on the question-answering dataset _Qi ,/> represents the initial fusion weight obtained by fine-tuning on the dialogue dataset _Ci ,/> Represents the conversation LoRA-adapter trained on the conversation dataset _Ci .

The embodiment of the present invention provides a large language model fine-tuning and adapter fusion device, including:

The first obtaining unit is used to collect multiple question-answering datasets and dialogue datasets from a set network platform; the LoRA-adapter is fine-tuned on the question-answering dataset and the dialogue dataset respectively, and the question-answering large language model, the question-answering negative log-likelihood loss function, the dialogue large language model and the dialogue negative log-likelihood loss function are obtained in turn;

The second obtaining unit is used to obtain the ideal loss function of the question and answer dataset and the dialogue dataset under the ideal state according to the question and answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and the initial fusion weight included in the fine-tuning based on each LoRA-adapter, and obtain the ideal fusion weight and the first ideal parameter corresponding to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parameter represents all LoRA-adapters added to the question and answer large language model and the dialogue large language model respectively;

The third obtaining unit is used to fine-tune the question and answer LoRA-adapter corresponding to each question and answer data set and the conversation LoRA-adapter corresponding to each conversation data set according to the ideal loss function, and respectively obtain the optimal parameters of the question and answer LoRA-adapter, the optimal parameters of the conversation LoRA-adapter and the optimal fusion parameters;

The fourth obtaining unit is used to obtain a universal LoRA-adapter according to the optimal parameters of the question-and-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.

An embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes any one of the large language model fine-tuning and adapter fusion methods described above.

An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes any one of the large language model fine-tuning and adapter fusion methods described above.

The embodiment of the present invention provides a large language model fine-tuning and adapter fusion method and device, the method comprising: collecting multiple question-answering data sets and dialogue data sets from a set network platform; performing LoRA-adapter fine-tuning on the question-answering data set and the dialogue data set respectively, and obtaining a large question-answering language model, a question-answering negative log-likelihood loss function, a dialogue large language model and a dialogue negative log-likelihood loss function in turn; according to the question-answering negative log-likelihood loss function, the dialogue negative log-likelihood loss function and the initial fusion weights included in each LoRA-adapter fine-tuning, obtaining the ideal loss function of the question-answering data set and the dialogue data set in an ideal state, and obtaining the ideal fusion corresponding to the ideal loss function according to the minimum value of the ideal loss function Weight and first ideal parameter; wherein the first ideal parameter represents all LoRA-adapters added to the question-answering large language model and the dialogue large language model respectively; according to the ideal loss function, the question-answering LoRA-adapter corresponding to each question-answering data set and the dialogue LoRA-adapter corresponding to each dialogue data set are fine-tuned to obtain the optimal parameters of the question-answering LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters respectively; the universal LoRA-adapter is obtained according to the optimal parameters of the question-answering LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters. The method constructs multiple instruction fine-tuning data sets and uses the quantization technology of QLoRA to save the consumption of GPU (Graphic Process Unit in English, Graphics Processing Unit in Chinese), providing a large language model training method that saves computing resource costs and is high-quality, and at the same time designs a multi-LoRA-adapter fusion method based on Grid-Search (parameter adjustment means in Chinese) to fuse the trained LoRA-adapter. This method can effectively avoid the semantic space conflict caused by data set fusion by fusing LoRA-adapter, and improve the generalization performance of large language models on multiple tasks. It solves the problem of performance degradation caused by conflicts in the semantic space of different data sets in the existing technology.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of a large language model fine-tuning and adapter fusion method provided by an embodiment of the present invention;

FIG2 is a schematic diagram of the structure of a large language model fine-tuning and Adapter fusion device provided by an embodiment of the present invention.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Training large language models is not only for improving the performance of natural language processing tasks and improving the interactive experience of dialogue systems, but also has richer value and significance in scientific research and application fields.

First, large language models can learn rich language knowledge and grammatical rules by training massive corpora. These models can understand and generate more accurate and fluent natural language, providing better performance for natural language processing tasks such as machine translation, text generation, and text classification. In machine translation tasks, large language models can more accurately understand the meaning of the source language and generate more natural translation results in the target language. In text generation tasks, the model can generate more logical and coherent text content. In text classification tasks, the model can more accurately determine the category of the text and improve the accuracy of classification.

Secondly, large language models can be used to build intelligent dialogue systems, which can provide more natural, accurate and personalized responses by communicating with users. This capability is very useful for scenarios such as chatbots and intelligent customer service in daily life. The trained model can understand and generate human language, so that it can better meet user needs and improve the interactive experience of the dialogue system. The dialogue system can generate personalized responses through the model, so that users can experience the same interactive experience as human dialogue, thereby enhancing user satisfaction.

In addition, training large language models requires processing massive amounts of data and huge computing resources, which is of great significance for promoting scientific research and technological innovation. In the process of training large language models, many technical challenges need to be addressed, such as data processing, model design, training algorithms, etc. Solving these challenges can not only promote the development of language models, but also contribute to the research and development of related fields. For example, by improving and optimizing the model, the efficiency and performance of the model can be improved, providing technical support for the development and application of other natural language processing tasks.

Finally, training large language models can provide intelligent natural language processing services and promote the inclusive development of artificial intelligence technology. These models can be applied to various fields, such as education, medical care, and finance. In the field of education, models can be used to assist learning, intelligent question answering, etc., to provide personalized learning resources and communication platforms. In the medical field, models can be used to assist doctors in diagnosis, intelligent medical record recording, etc., to improve the quality and efficiency of medical services. In the financial field, models can be used for intelligent customer service, risk management, etc., to provide more personalized and efficient financial services.

In summary, training large language models has important scientific research and application value. It can not only improve the performance of natural language processing tasks and improve the interactive experience of dialogue systems, but also promote the inclusiveness of scientific research, technological innovation and artificial intelligence development. By training large language models, intelligent natural language processing technology can be applied in various fields to provide better intelligent services and solutions for society.

Due to the conventional scheme currently used for large-scale language model training, it seems impossible to fuse multiple data sets to construct a multifunctional data set. There may be contradictions between different data sets, and it is difficult to evaluate the quality of the data. Based on this, in order to be able to build a high-quality and powerful large language model, an embodiment of the present invention proposes an efficient training method. By cleaning and sorting multiple open source data on the Huggingface platform, multiple different knowledge question and answer data sets and dialogue data sets are obtained, and then QLoRA (Chinese: low-rank adaptation) is used to train a LoRA-adapter separately on each data set, and finally Grid-Search is used to dynamically optimize the fusion weights of these LoRA-adapters.

FIG1 is a flow chart of a method for fine-tuning a large language model and fusing an adapter according to an embodiment of the present invention; as shown in FIG1 , the method includes the following steps:

Step 101, collecting multiple question-answering datasets and dialogue datasets from a set network platform; performing LoRA-adapter fine-tuning on the question-answering dataset and the dialogue dataset, respectively, to obtain a question-answering large language model, a question-answering negative log-likelihood loss function, a dialogue large language model, and a dialogue negative log-likelihood loss function;

Step 102, according to the question-answering negative log-likelihood loss function, the dialogue negative log-likelihood loss function and the initial fusion weight included in the fine-tuning based on each LoRA-adapter, obtain the ideal loss function of the question-answering dataset and the dialogue dataset in an ideal state, and obtain the ideal fusion weight and the first ideal parameter corresponding to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parameter represents all LoRA-adapters added to the question-answering large language model and the dialogue large language model respectively;

Step 103, according to the ideal loss function, fine-tune the question-answer LoRA-adapter corresponding to each question-answer data set and the conversation LoRA-adapter corresponding to each conversation data set to obtain the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the conversation LoRA-adapter and the optimal fusion parameters respectively;

Step 104, obtaining a universal LoRA-adapter according to the optimal parameters of the question-and-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.

It should be noted that the execution subject of the large language model fine-tuning and Adapter fusion method provided in the embodiment of the present invention is a processor.

In step 101, a plurality of question-answer data sets and conversation data sets are first collected from a set network platform. The set network platform here may be the Huggingface community. In the embodiment of the present invention, the set network platform is not specifically limited.

Specifically, after collecting multiple data sets from a set network platform, the multiple data sets need to be cleaned. After cleaning, multiple question-answer data sets and multiple dialogue data sets are finally obtained. In an embodiment of the present invention, the cleaning rules for the data sets are as follows:

1) Delete the dialogue instances with ChatGPT (Chat Generative Pre-trained Transformer)-3.5-Turbo, and only keep the dialogue instances with GPT-4; 2) Delete the dialogues that GPT-4 refuses to answer or directly explains; 3) Delete the dialogues where the answer is GPT-4 empty or GPT-4 omitted to answer; 4) Delete the dialogues containing toxic or illegal information; 5) Delete the dialogues containing the words OpenAI or ChatGPT, or replace the dialogues containing the words OpenAI or ChatGPT with information that ensures that the model has the correct identity; 6) Delete user questions with a similarity greater than 85% with the benchmark questions; 7) Divide the dialogue instances that are too long into dialogues that match the maximum context length of the model.

Through the above situation rules, multiple question and answer data sets and multiple dialogue data sets required by the embodiment of the present invention can be finally obtained.

In the embodiment of the present invention, the question-answering dataset may be represented as {Q ₁ , Q ₂ , …, Q _M }, and the conversation dataset may be represented as {C ₁ , C ₂ , …, C _N }.

Specifically, the question-answering dataset can be expressed by formula (1):

Among them, s represents system message, q represents the user's query, r represents the artificial intelligence's response, _Qi represents the i-th question and answer dataset, si _,j represents the j-th system message of the i-th question and answer dataset, qi _,j represents the j-th question of the i-th question and answer dataset, ri _,j represents the j-th reply of the i-th question and answer dataset, and | _Qi | represents the length of the question and answer dataset _Qi .

In the embodiment of the present invention, when the obtained question-answering dataset is fine-tuned by LoRA-adapter, given a specific instance Given a system message si _,j and a query qi _,j , the large language model should learn to generate the corresponding reply _ri,j . This process can get the question-answering large language model, which is as follows:

in, represents the question-answering LoRA-adapter trained on the question-answering dataset _Qi , _pθ represents the large language model, | _ri,j | represents the length of ri _,j , _rk represents the kth token generated by the large language model, θ represents the frozen parameter of the large language model, and r _<k represents all r whose sub-scalar is less than k.

Furthermore, a question-answering negative log-likelihood loss function is obtained according to the question-answering large language model and the token of the question-answering large language model.

Among them, the question-answering negative log-likelihood loss function is as follows:

in, represents the negative log-likelihood loss function of question and answer, si _,j represents the j-th system information of the ith question and answer dataset, qi _,j represents the j-th question of the ith question and answer dataset, and _ri,j represents the j-th reply of the ith question and answer dataset.

Accordingly, the dialogue dataset can be expressed by formula (4):

The dialogue dataset contains multiple dialogue instances with T rounds, and _Ci represents the i-th dialogue dataset. represents the jth query of the i-th dialogue dataset in the T-th round,/> represents the j-th reply of the i-th dialogue dataset in the T-th round, and | _Ci | represents the length of the dialogue dataset _Ci .

In the embodiment of the present invention, when the LoRA-adapter is fine-tuned on the conversation dataset, the large language model will learn the conversation history and query before the given Tth round. Predict each response in the case of/> This process can obtain a large conversation language model, which is as follows:

in, represents the dialog LoRA-adapter trained on the dialog dataset _Ci , _Qj represents all tags belonging to the user query, _Rj represents the target tag, /> Indicates the number of tokens contained in the j-th data in the conversation dataset _Ci .

Furthermore, the conversation negative log likelihood loss function is obtained based on the conversation large language model and the token of the conversation large language model. The conversation negative log likelihood loss function is as follows:

in, represents the negative log-likelihood loss function of the conversation,/> represents the number of tokens contained in the jth data in the dialogue dataset _Ci , _pθ represents the large language model, θ represents the frozen parameters of the large language model,/> Represents the conversation LoRA-adapter trained on the conversation dataset _Ci .

It should be noted that in an embodiment of the present invention, for the fusion of LoRA-adapter, the loss function of each fine-tuned LoRA-adapter will be assigned a trainable weight, and the loss function of all LoRA-adapters with fusion weights will be fine-tuned, where the fusion weight can be expressed as:

In step 102, according to the question-answering negative log-likelihood loss function, the dialogue negative log-likelihood loss function and the initial fusion weights included in the fine-tuning based on each LoRA-adapter, the ideal loss function of the question-answering dataset and the dialogue dataset under ideal conditions can be obtained, wherein the ideal loss function is as follows:

Where L represents the ideal loss function, represents the question-answering negative log-likelihood loss function,/> represents the conversation negative log-likelihood loss function,/> Denotes fine-tuning on the question-answering dataset _Qi /> The initial fusion weight of Represents fine-tuning on the dialogue dataset _Ci to obtain/> The initial fusion weights.

Furthermore, the ideal fusion weight and the first ideal parameter corresponding to the ideal loss function are obtained according to the minimum value of the ideal loss function, and the minimum value of the ideal loss function is expressed by the following formula:

Among them, A ^* represents all ideal LORA-adapers, ω ^* represents all ideal fusion weights, and argmin represents the values of the corresponding parameters A and ω when the following formula reaches the minimum value.

In an embodiment of the present invention, all ideal LoRA-adapers may also be referred to as all first ideal parameters, where the first ideal parameters represent all LoRA-adapters added to the question-answering large language model and the dialogue large language model, respectively, wherein the first ideal parameter is represented by the following formula:

Where A represents the first ideal parameter, represents the first ideal parameter obtained by fine-tuning on the question-answering dataset Q _M , represents the first ideal parameter obtained by fine-tuning on the dialogue dataset _CN , M represents the number of question-answering datasets, and N represents the number of dialogue datasets.

Specifically, ω represents the ideal fusion weight of all LoRA-adapters, that is, the ideal fusion weight of the question-answer LoRA-adapter and the conversation LoRA-adapter, which can be expressed by the following formula:

Among them, ω represents the ideal fusion weight, Indicates/> The ideal fusion weight of Indicates/> The ideal fusion weights.

In step 103, according to the ideal loss function, the question-answer LoRA-adapter corresponding to each question-answer data set and the conversation LoRA-adapter corresponding to each conversation data set are fine-tuned to obtain the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the conversation LoRA-adapter and the optimal fusion parameters respectively.

In practical applications, in order to improve efficiency and simplicity, the first ideal parameters and the ideal fusion weights are fine-tuned sequentially. In the first stage, each LoRA-adapter (all first ideal parameters) on each question-answering dataset or dialogue dataset is fine-tuned, that is, for formula (8), first split formula (8) into two formulas as shown below, and then the optimal parameters of the question-answering LoRA-adapter and the optimal parameters of the dialogue LoRA-adapter can be calculated, as shown below:

in, represents the optimal parameters of the question-answering LoRA-adapter,/> Indicates the optimal parameters for the LoRA-adapter dialogue, /> represents the conversation LoRA-adapter trained on the conversation dataset _Ci ,/> represents the question-answering LoRA-adapter trained on the question-answering dataset _Qi ,/> represents the question-answering negative log-likelihood loss function,/> represents the dialogue negative log-likelihood loss function.

Furthermore, in the second stage, only the ideal fusion weights are fine-tuned, while the basic large language model and the first ideal parameters are frozen to obtain the optimal fusion parameters. The optimal fusion parameters are specifically as follows:

Among them, ω ^** represents the optimal fusion parameter, Denotes fine-tuning on the question-answering dataset _Qi /> The initial fusion weight of Represents fine-tuning on the dialogue dataset _Ci to obtain/> The initial fusion weight of represents the question-answering negative log-likelihood loss function,/> represents the dialogue negative log-likelihood loss function, and ω represents the ideal fusion weight.

It should be noted that in practical applications, when the number of question-answering datasets and dialogue datasets is small, some simple and fast algorithms can be used to optimize the ideal fusion weights.

Step 104, obtaining a universal LoRA-adapter according to the optimal parameters of the question-and-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.

In summary, the embodiment of the present invention provides a large language model fine-tuning and adapter fusion method and device, which can effectively avoid the semantic space conflict caused by data set fusion by fusing LoRA-adapter, and improve the generalization performance of the large language model on multiple tasks. It solves the problem of performance degradation caused by conflicts between different data sets in the semantic space in the prior art.

Based on the same inventive concept, an embodiment of the present invention provides a large language model fine-tuning and Adapter fusion device. Since the principle of solving the technical problem of the device is similar to that of a large language model fine-tuning and Adapter fusion method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

As shown in FIG. 2 , the device mainly includes a first obtaining unit 201 , a second obtaining unit 202 , a third obtaining unit 203 and a fourth obtaining unit 204 .

The first obtaining unit 201 is used to collect multiple question-answering datasets and dialogue datasets from a set network platform, perform LoRA-adapter fine-tuning on the question-answering dataset and the dialogue dataset respectively, and obtain a question-answering large language model, a question-answering negative log-likelihood loss function, a dialogue large language model, and a dialogue negative log-likelihood loss function in turn;

The second obtaining unit 202 is used to obtain the ideal loss function of the question and answer dataset and the dialogue dataset in an ideal state according to the question and answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and the initial fusion weight included in the fine-tuning based on each LoRA-adapter, and obtain the ideal fusion weight and the first ideal parameter corresponding to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parameter represents all LoRA-adapters added to the question and answer large language model and the dialogue large language model respectively;

The third obtaining unit 203 is used to fine-tune the question and answer LoRA-adapter corresponding to each question and answer data set and the conversation LoRA-adapter corresponding to each conversation data set according to the ideal loss function, and obtain the optimal parameters of the question and answer LoRA-adapter, the optimal parameters of the conversation LoRA-adapter and the optimal fusion parameters respectively;

The fourth obtaining unit 204 is used to obtain a universal LoRA-adapter according to the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters.

It should be understood that the units included in the above large language model fine-tuning and Adapter fusion device are logically divided only according to the functions implemented by the device. In practical applications, the above units can be superimposed or split. In addition, the functions implemented by the large language model fine-tuning and Adapter fusion device provided in this embodiment correspond one-to-one to the large language model fine-tuning and Adapter fusion method provided in the above embodiment. The more detailed processing flow implemented by the device has been described in detail in the above method embodiment 1 and will not be described in detail here.

Another embodiment of the present invention further provides a computer device, comprising: a processor and a memory; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes each step of the large language model fine-tuning and Adapter fusion method in the method flow shown in the above method embodiment.

Another embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer device, the computer device executes each step of the large language model fine-tuning and Adapter fusion method in the method flow shown in the above method embodiment.

Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (8)

1. A large language model micro-tuning and Adapter fusion method is characterized by comprising the following steps:

Collecting a plurality of question-answer data sets and dialogue data sets from a set network platform, and respectively performing LoRA-adapter fine tuning on the question-answer data sets and the dialogue data sets to sequentially obtain a question-answer large language model, a question-answer negative log-likelihood loss function, a dialogue large language model and a dialogue negative log-likelihood loss function;

Obtaining an ideal loss function of the question-answer dataset and the dialogue dataset in an ideal state according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and initial fusion weights which are included based on fine tuning of each LoRA-adapter, and obtaining an ideal fusion weight and a first ideal parameter which correspond to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parametric representation is added to all LoRA-adapters of the question-answer large language model and the dialogue large language model, respectively;

according to the ideal loss function, fine tuning is carried out on the question-answer LoRA-adapter corresponding to each question-answer data set and the dialogue LoRA-adapter corresponding to each dialogue data set, so as to respectively obtain the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters;

obtaining a general LoRA-adapter according to the optimal parameters of the question-answer LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter and the optimal fusion parameters;

The ideal loss function, the optimal parameters of the question LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter, and the optimal fusion parameters are as follows:

Wherein L represents an ideal loss function, Representing fine-tuning the acquisition/>, on the question-answer dataset Q _i Is used to determine the initial fusion weights of (1),Representing fine-tuning of acquisition/>, on dialog dataset C _i Initial fusion weights of,/>Optimal parameters representing question-answer LoRA-adapter,/>The best parameters for dialog LoRA-adapter, ω ^** the best fusion parameters,/>Representing a question-answer negative log-likelihood loss function,/>Representing a dialogue negative log likelihood loss function,/>Representing a question LoRA-adapter, i.e./>, trained on a question dataset Q _i Representing the resulting dialog LoRA-adapter trained on dialog data set C _i.

2. The method of claim 1, wherein said performing LoRA-adapter fine-tuning on said question-answer dataset sequentially results in a question-answer large language model, a question-answer negative log-likelihood loss function, comprising:

Training the question-answer data set to obtain a question-answer LoRA-adapter, and obtaining the question-answer large language model according to the question-answer LoRA-adapter and the question-answer data set;

obtaining the question-answer negative log-likelihood loss function according to the question-answer large language model and the token of the question-answer large language model;

the question-answer dataset, the question-answer large language model, and the question-answer negative log-likelihood loss function are as follows:

where Q _i represents the ith question-answer dataset, s _i,j represents the jth system information of the ith question-answer dataset, Q _i,j represents the jth question of the ith question-answer dataset, r _i,j represents the jth reply of the ith question-answer dataset, |Q _i | represents the length of the question-answer dataset Q _i, Representing the question and answer LoRA-adapter trained on the question and answer dataset Q _i, p _θ representing a large language model, |r _i,j | representing the length of r _i,j, r _k representing the kth token generated by the large language model, θ representing the freezing parameters of the large language model,Representing a question-answer negative log-likelihood loss function.

3. The method of claim 1, wherein performing LoRA-adapter fine-tuning on the dialogue dataset sequentially obtains a dialogue large language model and a dialogue negative log likelihood loss function, specifically comprising:

Training the dialogue data set to obtain a dialogue LoRA-adapter, and obtaining a dialogue large language model according to the dialogue LoRA-adapter and the dialogue data set;

obtaining the dialogue negative log likelihood loss function according to the dialogue large language model and the token of the dialogue large language model;

The dialogue dataset, dialogue large language model, and the dialogue negative log likelihood loss function are as follows:

Wherein C _i denotes the ith session dataset, Jth query representing ith session dataset in the T-th round,/>The j-th reply representing the i-th dialog data set in the T-th round, |c _i | represents the length of dialog data set C _i,/>Representing a dialog LoRA-adapter trained on dialog dataset C _i, Q _j representing all tags belonging to a user query, R _j representing a target tag,/>Representing the number of j-th data containing tokens in the dialog data set C _i,/>Representing a dialogue negative log-likelihood loss function, p _θ representing a large language model, θ representing a freeze parameter of the large language model.

4. The method of claim 1, wherein the minimum value of the ideal loss function is as follows:

Wherein, Representing fine-tuning the acquisition/>, on the question-answer dataset Q _i Initial fusion weights of,/>Representing fine-tuning of acquisition/>, on dialog dataset C _i A ^* denotes all the first ideal parameters, ω ^* denotes all the ideal fusion weights, a denotes the first ideal parameters, ω denotes the ideal fusion weights.

5. The method of claim 1, wherein the first desired parameter is as follows:

the ideal fusion weights are as follows:

Wherein A represents a first ideal parameter, Representing the first ideal parameter obtained by fine tuning on the question-answer dataset Q _M,/>Represents the first ideal parameter obtained by fine tuning on the dialogue dataset C _N, M represents the number of question-answer datasets, N represents the number of dialogue datasets, ω represents the ideal fusion weight,/>Representation/>Is/are the ideal fusion weights of (1)Representation/>Is used for the optimal fusion weight of the (c).

6. A large language model refinement and adaptation fusion apparatus, comprising:

The first obtaining unit is used for collecting a plurality of question-answer data sets and dialogue data sets from a set network platform, respectively performing LoRA-adapter fine tuning on the question-answer data sets and the dialogue data sets, and sequentially obtaining a question-answer large language model, a question-answer negative log likelihood loss function, a dialogue large language model and a dialogue negative log likelihood loss function;

the second obtaining unit is used for obtaining an ideal loss function of the question-answer dataset and the dialogue dataset in an ideal state according to the question-answer negative log-likelihood loss function, the dialogue negative log-likelihood loss function and initial fusion weights which are included based on fine adjustment of each LoRA-adapter, and obtaining an ideal fusion weight and a first ideal parameter which correspond to the ideal loss function according to the minimum value of the ideal loss function; wherein the first ideal parametric representation is added to all LoRA-adapters of the question-answer large language model and the dialogue large language model, respectively;

The third obtaining unit is configured to fine tune the question answer LoRA-adapter corresponding to each question-answer dataset and the dialogue LoRA-adapter corresponding to each dialogue dataset according to the ideal loss function, so as to obtain an optimal parameter of the question answer LoRA-adapter, an optimal parameter of the dialogue LoRA-adapter and an optimal fusion parameter respectively;

A fourth obtaining unit, configured to obtain a general LoRA-adapter according to the optimal parameter of the question-answer LoRA-adapter, the optimal parameter of the dialogue LoRA-adapter, and the optimal fusion parameter;

The ideal loss function, the optimal parameters of the question LoRA-adapter, the optimal parameters of the dialogue LoRA-adapter, and the optimal fusion parameters are as follows:

Wherein L represents an ideal loss function, Representing fine-tuning the acquisition/>, on the question-answer dataset Q _i Is used to determine the initial fusion weights of (1),Representing fine-tuning of acquisition/>, on dialog dataset C _i Initial fusion weights of,/>Optimal parameters representing question-answer LoRA-adapter,/>The best parameters for dialog LoRA-adapter, ω ^** the best fusion parameters,/>Representing a question-answer negative log-likelihood loss function,/>Representing a dialogue negative log likelihood loss function,/>Representing a question LoRA-adapter, i.e./>, trained on a question dataset Q _i Representing the resulting dialog LoRA-adapter trained on dialog data set C _i.

7. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the large language model refinement and adaptation fusion method of any one of claims 1-5.

8. A computer readable storage medium, storing a computer program which, when executed by a processor, causes the processor to perform the large language model refinement and adaptation fusion method of any one of claims 1-5.