CN117743950B - Correlation judgment method and LLM-based correlation judgment model construction method - Google Patents
Correlation judgment method and LLM-based correlation judgment model construction method Download PDFInfo
- Publication number
- CN117743950B CN117743950B CN202410190721.7A CN202410190721A CN117743950B CN 117743950 B CN117743950 B CN 117743950B CN 202410190721 A CN202410190721 A CN 202410190721A CN 117743950 B CN117743950 B CN 117743950B
- Authority
- CN
- China
- Prior art keywords
- relevance
- word
- search
- recall
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 114
- 238000010276 construction Methods 0.000 title claims abstract description 8
- 238000012549 training Methods 0.000 claims abstract description 100
- 230000008569 process Effects 0.000 claims description 65
- 230000015654 memory Effects 0.000 claims description 21
- 238000013140 knowledge distillation Methods 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 4
- 230000000875 corresponding effect Effects 0.000 description 16
- 241001122767 Theaceae Species 0.000 description 13
- 239000008267 milk Substances 0.000 description 13
- 235000013336 milk Nutrition 0.000 description 13
- 210000004080 milk Anatomy 0.000 description 13
- 239000003607 modifier Substances 0.000 description 12
- 150000001875 compounds Chemical class 0.000 description 9
- 238000004590 computer program Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000003058 natural language processing Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 235000013305 food Nutrition 0.000 description 2
- 241000272525 Anas platyrhynchos Species 0.000 description 1
- 241000272517 Anseriformes Species 0.000 description 1
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 235000013601 eggs Nutrition 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000007787 long-term memory Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007967 peppermint flavor Substances 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
One or more embodiments of the present disclosure provide a correlation determination method and a LLM-based correlation determination model construction method. The construction method of the LLM-based correlation judgment model comprises the following steps: acquiring a data sample comprising search words, recall words and correlation tags; generating a relevance judgment prompt according to the search word, the recall word and a preset relevance scoring rule in the data sample, and inputting the relevance judgment prompt into the pre-training language model, so that the pre-training language model executes the following steps according to the relevance judgment prompt: performing relevance scoring on the search word and the recall word by referring to a relevance scoring rule, and generating and outputting relevance scores between the search word and the recall word and scoring interpretation instructions of the relevance scores; constructing a training sample based on each data sample, the relevance score corresponding to the data sample and the scoring interpretation; and performing supervised training based on the training samples to obtain a correlation judgment model. And a correlation judgment model with high prediction accuracy is obtained.
Description
Technical Field
One or more embodiments of the present disclosure relate to the field of computer software technologies, and in particular, to a search relevance determining method, a method for constructing a relevance determining model based on LLM, and an electronic device.
Background
In the search scenario, after a user enters a search term, the search engine retrieves according to the search term provided by the user, recalls recall terms associated with the search term. The relevance judgment between the search word and the recall word is a key link for ensuring the quality of the search result, so that recall words which are strongly related to the search word can be arranged at a front position and recall words which are strongly or poorly related to the search word or are less related to the search word can be arranged at a rear position based on the relevance judgment result when the search result is displayed, and the search experience of a user is improved.
However, the correlation judgment method in the related art has a problem that the correlation judgment is inaccurate when the correlation judgment is performed on some search words (such as long-tail search words or other search words with a small number of searches) and recall words thereof.
Disclosure of Invention
In view of this, one or more embodiments of the present specification provide a search relevance determination method, a LLM-based relevance determination model construction method, an electronic device, a computer-readable storage medium, and a computer program product.
In order to achieve the above object, one or more embodiments of the present disclosure provide the following technical solutions:
According to a first aspect of one or more embodiments of the present disclosure, a method for constructing a LLM-based relevance determining model is provided, including:
acquiring a plurality of data samples, wherein each data sample comprises a search word, a recall word and a correlation tag between the search word and the recall word;
For each data sample, generating a relevance judgment prompt according to search words, recall words and a preset relevance scoring rule in the data sample, and inputting the relevance judgment prompt into a pre-training language model so that the pre-training language model executes the following steps according to the relevance judgment prompt: performing relevance scoring on the search word and the recall word by referring to the relevance scoring rule, and generating and outputting a relevance score between the search word and the recall word and a scoring interpretation for the relevance score;
Constructing a training sample based on each data sample, the relevance scores between the search words and the recall words corresponding to the data samples and scoring explanation thereof;
Performing supervised training based on the training sample to obtain a correlation judgment model; the relevance judgment model is used for carrying out relevance judgment based on the input search word and recall word and outputting a relevance judgment result.
According to a second aspect of one or more embodiments of the present disclosure, there is provided a correlation determination method, including:
Recall at least one candidate recall word based on the search term entered by the user;
Inputting the search word and each candidate recall word into a pre-constructed relevance judgment model, judging the relevance between the search word and each candidate recall word by the relevance judgment model, and outputting a relevance judgment result; the correlation judgment model is constructed based on the LLM-based correlation judgment model construction method in the first aspect.
According to a third aspect of one or more embodiments of the present specification, there is provided a correlation determination method, including:
Recall at least one candidate recall word based on the search term entered by the user;
generating a search relevance judgment prompt according to the search word, the at least one candidate recall word and a preset relevance scoring rule, and inputting the search relevance judgment prompt into a pre-training language model, so that the pre-training language model executes the following operation steps according to the relevance judgment prompt: performing relevance scoring on the search word and each candidate recall word by referring to the relevance scoring rule, and generating and outputting a relevance score between the search word and each candidate recall word and a scoring interpretation for the relevance score;
and obtaining a correlation judgment result between the search word and each candidate recall word according to the correlation score between the search word and each candidate recall word and the scoring explanation aiming at the correlation score.
According to a fourth aspect of embodiments of the present disclosure, there is provided an electronic device, comprising:
A processor;
A memory for storing processor-executable instructions;
Wherein the processor, when executing the executable instructions, is configured to implement the method of the first aspect, the second aspect, or the third aspect.
According to a fifth aspect of embodiments of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of any of the methods described above.
According to a sixth aspect of embodiments of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of any of the above.
The technical scheme provided by the embodiment of the disclosure can comprise the following beneficial effects:
in the embodiment of the disclosure, a pre-training language model with larger parameter quantity is introduced, the strong semantic understanding capability is utilized to obtain the relevance score between the search word and the recall word and the scoring explanation thereof, a training sample is constructed, and the supervised training is performed based on the training sample to obtain a relevance judgment model.
On the one hand, the relevance score in the training sample and the scoring interpretation can enable the relevance judgment model to have stronger capability when understanding the semantic relationship between the search word and the recall word, so that more relevance features can be captured, and the accuracy of relevance judgment is improved.
On the other hand, the relevance score and the scoring interpretation of the relevance score in the training sample are obtained through a pre-training language model, the pre-training language model has learned a great deal of language knowledge and semantic representation, and can provide useful intermediate representation for the relevance judgment model, so that the relevance judgment model benefits from the pre-training language model, and the generalization capability of the relevance judgment model is improved.
In yet another aspect, scoring interpretation of the relevance scores may help understand how the model derives a relevance assessment, improving the interpretability of the relevance determination model, making the determination of the relevance determination model more reliable and understandable.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
Fig. 1 is a flowchart of a method for constructing a LLM-based correlation determination model according to an exemplary embodiment.
Fig. 2 is another flowchart of a method for constructing a LLM-based correlation determination model according to an exemplary embodiment.
FIG. 3 is a schematic diagram of a training relevance determination model provided by an exemplary embodiment.
FIG. 4 is a schematic diagram of knowledge distillation training as provided by an exemplary embodiment.
Fig. 5 is a schematic structural diagram of a correlation determination model according to an exemplary embodiment.
Fig. 6 is a flowchart of a correlation determination method according to an exemplary embodiment.
Fig. 7 is a flowchart of another correlation determination method provided in an exemplary embodiment.
Fig. 8 is a schematic structural diagram of an electronic device according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of the present specification. Rather, they are merely examples of apparatus and methods consistent with aspects of one or more embodiments of the present description as detailed in the accompanying claims.
It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than described in this specification. Furthermore, individual steps described in this specification, in other embodiments, may be described as being split into multiple steps; while various steps described in this specification may be combined into a single step in other embodiments.
Related terms are explained herein:
LLM: the large language model (Large Language Model) is a modeling mode based on statistics and natural language processing technology, and is trained on massive texts by utilizing a high-capacity model architecture (such as a transducer), so that a large amount of priori knowledge can be obtained by means of the pre-training, and model questions and answers are processes of giving related knowledge by instructions. Common LLMs include ChatGPT3.5, chatGPT4, chatGLM, chinese-Alpaca, ziya-Llama, baichuan, and the like.
Prompt (hint): prompt can be understood as a way to specify the direction in which LLM generates emphasis, which is a piece of text or sentence that directs LLM to generate an output of a particular type, topic or format. Guiding a base model towards the desired a priori direction by means of elaborate hints (promts) is the method of the lowest current threshold, sometimes also referred to as gradient-free tuning. The promt uses the association and prior information seen during training, for example, by a promt approach like "you are a food expert …" to let the LLM output answers more biased towards the food dimension.
Thinking Chain (Chain-of-Thought, coT): by letting a Large Language Model (LLM) disassemble a question into multiple steps, a step-by-step analysis is performed to gradually get the correct answer.
In a search scenario, relevance determination is a key element to ensure the quality of search results. In order to improve the user's search experience, search engines employ various methods to determine the relevance between the search terms and recall terms and rank the recall terms.
One relevance determination method in the related art is to perform relevance determination on the search term and the recalled recall term based on a tree model (e.g., GBDT) and provide two classification results (e.g., determine whether the recalled recall term is semantically relevant to the search term, the relevance is 1, and the non-relevance is 0). Tree models have been popular in the industry because of their low number of parameters and their ability to utilize custom statistical class features. However, under the condition that the flow of the search words in the search scene is uneven, for the search words with more search quantity, the search behaviors of the user are rich, the statistical features determined by the tree model are relatively more confidence, and for the search words with less search quantity (such as long tail search words or other search words with less search times), the search behaviors of the user are extremely sparse, so that the statistical features determined by the tree model are not confidence, and the understanding capability of the tree model on the search words with less search quantity and recall words is insufficient, thereby influencing the accuracy of the relevance judgment.
Based on the above, the embodiment of the specification better understands the relationship between the search word and the recall word by introducing a pre-trained language model (such as the LLM model) with larger parameter number and utilizing the strong semantic understanding capability, thereby improving the accuracy of the relevance judgment. For long-tail search words or other search words with fewer searches, the pre-trained language model has stronger generalization capability, can better understand the search words by learning knowledge and semantic relationships in a large corpus, and provides more accurate relevance judgment.
The following is an exemplary description of the solution provided by the embodiments of the present specification:
Referring to fig. 1 and fig. 2, fig. 1 is a flow chart illustrating a method for constructing a LLM-based correlation determination model, which may be performed by an electronic device, and includes:
In S101, a plurality of data samples are acquired, each data sample including a search term, a recall term, and a correlation tag therebetween.
Illustratively, in a search scenario, multiple data samples need to be acquired in order to construct a relevance determination model.
Wherein the search term is a keyword or phrase input by a user and is used for triggering the search function of the search engine. Recall words are a set of text information, such as commodities, dishes, character introduction, science popularization content and the like, which are returned by a search engine and are related to the search words; or may be other forms of recall content such as web pages, pictures, video, audio, etc. Relevance labels are a sign for measuring the degree of relevance between search words and recall words, and generally, relevance labels are labeled manually or semi-automatically, such as manual labeling, machine learning algorithms, and the like. The specific form of the correlation tag may be a two-class (correlated/uncorrelated), a multi-class (high correlation/medium correlation/low correlation), or a continuous value (correlation score), etc.
In the process of constructing a relevance judgment model, it is important to acquire a plurality of data samples including search words, recall words and relevance labels thereof. These data samples may be used to train a relevance determination model, thereby enabling a search engine to more accurately predict relevance between search terms and recall terms, providing recall terms that more meet user needs.
In S102, for each data sample, generating a relevance judgment prompt according to the search word, the recall word and the preset relevance scoring rule in the data sample, and inputting the relevance judgment prompt into the pre-training language model, so that the pre-training language model performs the following steps according to the relevance judgment prompt: and carrying out relevance scoring on the search word and the recall word by referring to a relevance scoring rule, and generating and outputting relevance scores between the search word and the recall word and scoring interpretation descriptions aiming at the relevance scores.
Illustratively, the relevance scoring rules are a set of rules designed in advance for guiding the pre-trained language model in evaluating relevance between search terms and recall terms. The relevance judgment prompt is used for prompting the pre-training language model to perform relevance scoring on the search word and the recall word by referring to the relevance scoring rule and giving scoring interpretation.
Referring to fig. 2, after the pre-training language model is input for the relevance judgment prompt, the pre-training language model may utilize its strong natural language processing capability, and based on the relevance scoring rule, the pre-training language model may comprehensively consider various factors between the search word and the recall word, and assign a relevance score to the pre-training language model, where the relevance score may be a continuous value, or may be a discrete value or a classification label.
In order to enable the user to better understand the basis of the relevance evaluation of the recall words, the pre-training language model can also generate a scoring interpretation, and the relevance score is interpreted and interpreted.
In S103, a training sample is constructed based on each data sample and the relevance scores between the search words and recall words corresponding to the data sample and the scoring interpretation.
Illustratively, each training sample contains a search term, a recall term, a relevance score between the search term and the recall term, a scoring interpretation of the relevance score, and a relevance label.
In S104, performing supervised training based on the training sample to obtain a correlation judgment model; the relevance judgment model is used for carrying out relevance judgment based on the input search word and recall word and outputting a relevance judgment result.
In this embodiment, by introducing a pre-training language model (such as the LLM model described above) with a large parameter amount, the correlation score between the search word and the recall word and the scoring interpretation thereof are obtained by using the strong semantic understanding capability of the pre-training language model, and a training sample is constructed according to the correlation score, and the supervised training is performed based on the training sample to obtain the correlation judgment model.
On the one hand, the relevance score in the training sample and the scoring interpretation can enable the relevance judgment model to have stronger capability when understanding the semantic relationship between the search word and the recall word, so that more relevance features can be captured, and the accuracy of relevance judgment is improved.
On the other hand, the relevance score and the scoring interpretation of the relevance score in the training sample are obtained through a pre-training language model, the pre-training language model has learned a great deal of language knowledge and semantic representation, and can provide useful intermediate representation for the relevance judgment model, so that the relevance judgment model benefits from the pre-training language model, and the generalization capability of the relevance judgment model is improved.
In yet another aspect, scoring interpretation of the relevance scores may help understand how the model derives a relevance assessment, improving the interpretability of the relevance determination model, making the determination of the relevance determination model more reliable and understandable.
In some embodiments, the pre-trained language model generates relevance scores between search words and recall words corresponding to the data samples and reasoning processes of scoring interpretations thereof, including a chain of thought consisting of a plurality of intermediate reasoning processes. The preset relevance scoring rules include scoring rules for different types of search terms.
The pre-training language model performs a first intermediate reasoning process to determine the type to which the search term belongs to determine scoring rules corresponding to the type to which the search term belongs.
Illustratively, the first intermediate reasoning process includes a plurality of sub-reasoning processes, each for determining whether the search term is of one of a plurality of types.
For example, if the preset relevance scoring rules include scoring rules of 2 types of search terms, the first intermediate reasoning process includes 2 sub-reasoning processes, where the first sub-reasoning process is used to determine whether the search term belongs to one of the 2 types, and if not, continuing to perform the second sub-reasoning process; the second sub-reasoning process is used for judging whether the search word belongs to another type of the 2 types, if so, the second intermediate reasoning process is carried out, if not, the reasoning process is ended, and a prompt that the search word does not belong to any type and the relevance judgment cannot be carried out is output.
In one possible implementation, the relevance judgment prompt further includes an explanation for a different type to which the search term belongs; the pre-training language model performs a first intermediate reasoning process to determine a type to which the search term belongs according to the interpretation of the different types to which the search term belongs, to determine a scoring rule corresponding to the type to which the search term belongs. And providing explanation for different types of search words, and combining a pre-training language model to perform correlation judgment, so that the accuracy and reliability of the correlation judgment are improved.
After determining scoring rules corresponding to the types to which the search words belong, the pre-training language model executes a second intermediate reasoning process to analyze the semantics of the search words and the semantics of the recall words so as to judge the relevance between the search words and the recall words based on the analyzed results and obtain a relevance judgment result.
Finally, the pre-training language model performs a third intermediate reasoning process to determine a relevance score between the search word and the recall word according to a scoring rule corresponding to the type to which the search word belongs and a relevance determination result, and generates a scoring interpretation based on the plurality of intermediate reasoning processes.
The embodiment takes the relevance score and scoring explanation thereof obtained by utilizing the thinking chain capability of the pre-training language model as the input of the relevance judgment model, and expands the thinking chain knowledge of the pre-training language model to the relevance judgment model, so that the relevance judgment model has better interpretability and accuracy.
For example, the relevance judgment hint is designed as "you now have the role of query-item relevance judgment expert in the search field. You need to score the relevance of a given query and item according to the following relevance scoring rules. The scoring rules are as follows:
0 point: the query is irrelevant to the item, the query is a product intention word, and the query is a compound word.
1, The method comprises the following steps: query is weakly related to item, query is a term of interest, and query is a compound word.
2, The method comprises the following steps: the query is strongly related to the item, the query is a product intention word, and the query is a compound word.
3, The method comprises the following steps: query is weakly related to item, query is a term of intended purpose, and query is a single word.
4, The following steps: the query is strongly related to item, the query is a term intended word, and the query is a single word.
5, The method comprises the following steps: query is a store intention word.
And 6, dividing: the query is neither a good intent word nor a store intent word.
7, The following steps: the query contains a plurality of commodity bodies (i.e., commodities or commodity categories).
8, The following steps: the query is not related to the item, and the query is a special article intention word.
9, The method comprises the following steps: the query is strongly related to item, and is a special article intention word.
The nouns involved in the scoring rules are explained as follows:
Commodity body word: refers to specific retail trade names (e.g., batteries, cups, etc.), restaurant trade names (e.g., roast ducks, stewed eggs, etc.), or abstract types of merchandise (e.g., duck goods, fruit, etc.).
Modifier: specific modifier words (e.g., specification, taste, color, size, region, unit of measure, brand of merchandise, etc., words that are all capable of modifying merchandise) that may be used to modify the subject word of the merchandise.
Compound words: the query contains at least 1 modifier and at least 1 commodity subject word.
The single word: it is meant that the query does not contain any modifier (in fact allows punctuation marks or nonsensical non-modifier) and contains only 1 commodity body word.
Special article intention word: refers to a single word (e.g., peppermint flavor, etc.) that contains only modifier words.
Please do you score the relevance of $ { query } and $ { item } according to the rules above, and give an explanation of the scoring;
Wherein, $ { query = ' large cup milk tea ', = ', $ { item = '/large cup milk tea in iceway ' "".
Query in the relevance judgment prompt represents a search term, and item represents a recall term. After inputting the correlation judgment prompt into the pre-training language model, the pre-training language model performs the following intermediate reasoning process:
(1) A first intermediate reasoning process comprising: a. judging whether the query contains a special product intention word, and determining that 'big cup milk tea' is not the special product intention word based on the explanation of the special product intention word; b. judging whether the query is a shop intention word, and finding that 'big cup milk tea' is not the shop intention word according to the rule; c. judging whether the query is a product intention word or not, and further judging whether the { query } is a compound word or a single word or not; and checking whether the query contains the modifier word and the commodity body word and whether only one commodity body word exists, so as to obtain that the 'big cup milk tea' is a commodity intention word and a compound word.
(2) In the second intermediate reasoning process, the semantics of 'large cup milk tea' and the semantics of 'great cup package of ice milk tea' are analyzed, and the core commodity main words in the item are determined to be milk tea and are the same as the commodity main words 'milk tea' in the query, and the modifier words in the item are large cup packages and are the same as the modifier words 'large cup' in the query.
(3) And a third intermediate reasoning process, wherein the score is 0 according to a scoring rule: the query is irrelevant to the item, the query is a product intention word, and the query is a compound word; 1, the method comprises the following steps: the query is weakly related to the item, the query is a product intention word, the query is a compound word, and the correlation judgment result obtained in the second intermediate reasoning process determines the semantic meaning of 'large cup milk tea' and the correlation score between 'large cup milk tea' and generates scoring explanation based on the plurality of intermediate reasoning processes.
Finally, the pre-trained language model outputs the following information: because modifier words are present in the query: large cup, and there are commodity body words: milk tea, so query is a compound word and is a product intention word; the core commodity body words in the item are milk tea, which are the same as the commodity body words in the query, and the modifier words in the item are big cup-shaped, which have the same meaning as the modifier words in the query, so the score is 2.
In some embodiments, referring to fig. 3, when performing supervised training based on a training sample to obtain a correlation determination model, the electronic device may input the training sample into the correlation determination model to be trained, determine, by the correlation determination model, a correlation between the search word and the recall word based on the search word, the recall word, a correlation score between the search word and the recall word, and a scoring interpretation thereof in the training sample, and obtain a correlation prediction result, and train the correlation determination model with a minimized difference between the correlation prediction result and the correlation label as an optimization objective. In this way, the model can learn the association between the search term, recall term, and the correlation therebetween, thereby making the correlation determination more accurate.
In other embodiments, referring to fig. 4, in order to meet the online performance requirements of some real application scenarios, knowledge distillation training may be performed based on training samples to obtain a correlation judgment model, so that online actual deployment may be implemented.
For example, when knowledge distillation training is performed based on a training sample to obtain a correlation judgment model, search words and recall words in the training sample and correlation labels between the search words and recall words can be input into a student model, so that the student model performs correlation judgment based on the search words and recall words and obtains a first correlation prediction result; inputting the training sample into a teacher model to judge the correlation between the search word and the recall word and obtain a second correlation prediction result based on the search word, the recall word, the correlation score between the search word and the recall word and the scoring interpretation description thereof in the training sample by the teacher model; wherein the scale of the student model is smaller than the scale of the teacher model; then, knowledge distillation training is carried out on the student model by taking the minimized difference between the first correlation prediction result and the second correlation prediction result and the minimized difference between the first correlation prediction result and the correlation label as optimization targets; and finally, taking the trained student model as a correlation judgment model.
For example, a first loss function value (e.g., KL divergence loss) may be calculated based on the first and second correlation prediction results, a second loss function value (e.g., cross entropy loss) may be calculated based on the difference between the first correlation prediction result and the correlation tag, and the first and second loss function values may be weighted and summed as the final loss of the student model.
In this embodiment, the student model can be inferred faster and occupies less memory resources due to its smaller scale, which is very advantageous for practical online deployment. In the knowledge distillation process, the teacher model carries out relevance judgment based on the search word, the recall word, the relevance scores among the search word and the recall word and scoring explanation thereof, so that the judgment process of the model can be understood, and the interpretability of the model can be improved. More correlation rules and modes learned by a teacher model during training are transmitted to a student model in a knowledge distillation mode, so that generalization capability and robustness of the student model can be improved, and the student model is more accurate and stable in an actual scene.
It is to be understood that the present embodiment does not limit the model structure of the correlation determination model, and the correlation determination model includes, but is not limited to, convolutional Neural Network (CNN), long-term memory network (LSTM), attention mechanism (Attention) model, and transducer model.
Convolutional neural networks (CNN is a model structure commonly used for image processing and text classification tasks in a relevance judgment model, search and recall words can be represented as text sequences, and one-dimensional convolutional layers are used to extract local relationships and features between them, and then the final relevance judgment is performed through the full-join layer.
Long short term memory networks (LSTM) are one type of recurrent neural network architecture suitable for processing sequence data. In the relevance judgment model, search words and recall words can be expressed as sequences, context information and long-term dependency relationship between the search words and the recall words are captured through an LSTM layer, and finally relevance prediction is performed through an output layer.
An Attention mechanism (Attention) model allows the model to selectively focus on information at different locations as the input sequence is processed. In the relevance judgment model, the attention mechanism can be used for weighting the search word and the recall word so as to better capture the relevance degree between the search word and the recall word and finally carry out relevance judgment.
The transducer model is a model structure based on a self-attention mechanism, and has been successful in the field of natural language processing. In the relevance judgment model, a transducer model can be used to process search words and recall words simultaneously, and relevance prediction can be performed through a multi-layer self-attention and feed-forward neural network.
For example, referring to FIG. 5, the relevance determination model includes a pre-trained BERT model and a multi-layer perceptron coupled to the BERT model. The BERT (Bidirectional Encoder Representations from Transformers) model is a pre-training model based on a transducer architecture.
The BERT model is used for processing the input search words and recall words and obtaining semantic representations corresponding to the search words and recall words; the multi-layer perceptron is used for carrying out relevance classification according to semantic representations corresponding to the search word and the recall word, and outputting a relevance judgment result. For example, the correlation determination result is a classification result including { correlation, uncorrelation }, or the correlation determination result is a multi-classification result, which is not limited in this embodiment.
Wherein the BERT model is pre-trained based on MLM tasks and fine-tuned based on classification tasks.
The MLM (Masked Language Modeling) task refers to masking any word in the sentence entered into the BERT model, and then predicting the masked word through the BERT model. Through the MLM task, the BERT model can learn the context information and can better understand the relationships between words.
The second classification task is to input the search word and the recall word into the BERT model, so that semantic representations corresponding to the search word and the recall word are obtained by the BERT model, and the semantic representations are classified by the multi-layer perceptron. For example, a classification task may be used to determine whether a recall word is related to a search word or predict whether a user will click on a presented recall word. The BERT model can learn the related information of the search word and the recall word in advance through the classification task, so that the training efficiency of the subsequent correlation judgment model is improved.
In some embodiments, after the relevance judgment model is trained, it may be deployed into the relevant electronic device. Referring to fig. 6, a flowchart of a correlation determination method is shown, which may be performed by an electronic device, and includes:
In S201, at least one candidate recall word is recalled based on the search word input by the user.
In S202, inputting the search word and each candidate recall word into a pre-constructed relevance judgment model, so as to judge the relevance between the search word and each candidate recall word by the relevance judgment model and output a relevance judgment result; the correlation judgment model is constructed based on the LLM-based correlation judgment model construction method of any one of the above.
In this embodiment, the training process of the relevance judgment model introduces a pre-training language model with a larger parameter, and uses the strong semantic understanding capability of the pre-training language model to enable the trained relevance judgment model to better understand the relationship between the search word and the recall word, thereby having better accuracy, resolvability and generalization capability. Therefore, based on the correlation judgment model, the correlation between the search word and each candidate recall word is judged, and an accurate correlation judgment result can be obtained.
In some embodiments, where the pre-trained language model can be deployed online, the pre-trained language model can be directly utilized to make relevance determinations between search terms and recall terms. Referring to fig. 7, another correlation determination method provided in the embodiment of the present disclosure may be applied to an electronic device, including:
in S301, at least one candidate recall word is recalled based on the search word entered by the user.
In S302, a search relevance discrimination prompt is generated according to the search word, at least one candidate recall word and a preset relevance scoring rule, and the search relevance discrimination prompt is input into a pre-training language model, so that the pre-training language model performs the following operation steps according to the relevance discrimination prompt: relevance scoring is performed on the search word and each candidate recall word with reference to a relevance scoring rule, and relevance scores between the search word and each candidate recall word and scoring interpretation descriptions for the relevance scores are generated and output.
In S303, a result of the relevance judgment between the search word and each candidate recall word is obtained from the relevance score between the search word and each candidate recall word and a scoring interpretation for the relevance score.
In this embodiment, the search word input by the user is recalled into at least one candidate recall word, and the search relevance judgment prompt is generated according to the preset relevance scoring rule, so that the pre-training language model can score the relevance of the search word and the candidate recall word according to the prompt, the accuracy of the recall word can be improved, and the user can find the required information more easily. And by generating scoring interpretation for the relevance score, the user can know why the recall is evaluated as relevant or irrelevant, which helps the user understand the basis of the relevance judgment result and increases the trust degree of the recall.
In some embodiments, the pre-trained language model generates relevance scores between the search term and each candidate recall term, and an inference process for scoring an explanation of the relevance scores, including a chain of thought consisting of a plurality of intermediate inference processes. The preset relevance scoring rules include scoring rules for different types of search terms.
The plurality of intermediate reasoning processes includes a first intermediate reasoning process, a second intermediate reasoning process, and a third intermediate reasoning process.
Wherein the first intermediate reasoning process comprises: and judging the type of the search word to determine a scoring rule corresponding to the type of the search word.
The second intermediate reasoning process includes: analyzing the semantics of the search word and the semantics of each candidate recall word to judge the relevance between the search word and each candidate recall word based on the analyzed result, and obtaining a relevance judgment result.
The third intermediate reasoning process includes: and determining the relevance scores between the search words and the candidate recall words according to the scoring rules corresponding to the types of the search words and the relevance judgment results, and generating scoring interpretation based on a plurality of intermediate reasoning processes.
In some embodiments, the relevance judgment prompt further includes an explanation for a different type to which the search term belongs; the first intermediate reasoning process specifically comprises: according to the explanation aiming at different types of the search word, judging the type of the search word to determine a scoring rule corresponding to the type of the search word.
The various technical features of the above embodiments may be arbitrarily combined as long as there is no conflict or contradiction between the features, but are not described in detail, and therefore, the arbitrary combination of the various technical features of the above embodiments is also within the scope of the disclosure of the present specification.
In some embodiments, embodiments of the present disclosure further provide an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor implements the method of any of the above by executing the executable instructions.
Illustratively, fig. 8 is a schematic block diagram of an electronic device according to an exemplary embodiment. Referring to fig. 8, at the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile storage 810, although other scenarios may also include the hardware required. One or more embodiments of the present description may be implemented in a software-based manner, such as by the processor 802 reading a corresponding computer program from the non-volatile memory 810 into the memory 808 and then running. Of course, in addition to software implementation, one or more embodiments of the present disclosure do not exclude other implementation manners, such as a logic device or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or a logic device.
In some embodiments, the present description further provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of any of the methods described above.
In some embodiments, the present description embodiments also provide a computer program product comprising a computer program which, when executed by a processor, implements the steps of any of the methods described above.
It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) related to the present disclosure are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and be provided with corresponding operation entries for the user to select authorization or rejection.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or a combination of any of these devices.
In a typical configuration, a computer includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, read only compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by the computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
The terminology used in the one or more embodiments of the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the specification. As used in this specification, one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in one or more embodiments of the present description to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "in response to a determination" depending on the context.
The foregoing description of the preferred embodiment(s) is (are) merely intended to illustrate the embodiment(s) of the present invention, and it is not intended to limit the embodiment(s) of the present invention to the particular embodiment(s) described.
Claims (10)
1. A method for constructing a correlation judgment model based on LLM is characterized by comprising the following steps:
acquiring a plurality of data samples, wherein each data sample comprises a search word, a recall word and a correlation tag between the search word and the recall word;
For each data sample, generating a relevance judgment prompt according to search words, recall words and a preset relevance scoring rule in the data sample, and inputting the relevance judgment prompt into a pre-training language model so that the pre-training language model executes the following steps according to the relevance judgment prompt: performing relevance scoring on the search word and the recall word by referring to the relevance scoring rule, and generating and outputting a relevance score between the search word and the recall word and a scoring interpretation for the relevance score;
Constructing a training sample based on each data sample, the relevance scores between the search words and the recall words corresponding to the data samples and scoring explanation thereof;
Performing supervised training based on the training sample to obtain a correlation judgment model; the relevance judgment model is used for carrying out relevance judgment based on the input search word and recall word and outputting a relevance judgment result;
The pre-training language model generates relevance scores between the search words and the recall words corresponding to the data samples and the reasoning process of scoring explanation, wherein the reasoning process comprises a thinking chain formed by a plurality of intermediate reasoning processes;
The preset relevance scoring rules comprise scoring rules of different types of search words;
The plurality of intermediate reasoning processes includes a first intermediate reasoning process, a second intermediate reasoning process, and a third intermediate reasoning process;
the first intermediate reasoning process includes: judging the type of the search word to determine a scoring rule corresponding to the type of the search word;
The second intermediate reasoning process includes: analyzing the semantics of the search word and the semantics of the recall word to judge the correlation between the search word and the recall word based on the analysis result, so as to obtain a correlation judgment result;
the third intermediate reasoning process includes: and determining the relevance scores between the search words and the recall words according to scoring rules corresponding to the types of the search words and the relevance judgment results, and generating scoring interpretation based on the plurality of intermediate reasoning processes.
2. The method of claim 1, wherein the relevance judgment prompt further includes an explanation for a different type to which the search term belongs;
The first intermediate reasoning process specifically comprises: and judging the type of the search word according to the explanation aiming at the different types of the search word so as to determine a scoring rule corresponding to the type of the search word.
3. The method of claim 1, wherein the performing supervised training based on the training samples results in a correlation judgment model, comprising:
Inputting the training sample into a correlation judgment model to be trained, judging the correlation between the search word and the recall word by the correlation judgment model based on the search word, the recall word, the correlation score between the search word and the recall word and the scoring interpretation explanation thereof in the training sample, and obtaining a correlation prediction result, and training the correlation judgment model by taking the minimized difference between the correlation prediction result and the correlation label as an optimization target.
4. The method of claim 1, wherein the performing supervised training based on the training samples results in a correlation judgment model, comprising:
And carrying out knowledge distillation training based on the training sample to obtain a correlation judgment model.
5. The method of claim 4, wherein performing knowledge distillation training based on the training samples to obtain a correlation judgment model comprises:
Inputting search words and recall words in the training samples and correlation labels between the search words and the recall words into a student model, so that the student model can judge the correlation based on the search words and the recall words and obtain a first correlation prediction result;
Inputting the training sample into a teacher model, and judging the relevance between the search word and the recall word by the teacher model based on the relevance scores and scoring explanation of the search word, the recall word, the search word and the recall word in the training sample and obtaining a second relevance prediction result; wherein the scale of the student model is smaller than the scale of the teacher model;
Performing knowledge distillation training on the student model with a minimum difference between the first correlation prediction result and the second correlation prediction result and a minimum difference between the first correlation prediction result and the correlation label as optimization targets;
And taking the trained student model as the correlation judgment model.
6. The method of claim 1, wherein the relevance determination model includes a pre-trained BERT model and a multi-layer perceptron coupled to the BERT model.
7. The method of claim 6, wherein the BERT model is pre-trained based on MLM tasks and fine-tuned based on bi-classification tasks; the MLM task is to mask any word in a sentence input into the BERT model, and then predict the masked word through the BERT model;
the classification task refers to inputting search words and recall words into the BERT model, so that semantic representations corresponding to the search words and recall words are obtained by the BERT model, and then the semantic representations are classified by the multi-layer perceptrons.
8. A correlation determination method, comprising:
Recall at least one candidate recall word based on the search term entered by the user;
Inputting the search word and each candidate recall word into a pre-constructed relevance judgment model, judging the relevance between the search word and each candidate recall word by the relevance judgment model, and outputting a relevance judgment result;
Wherein the correlation judgment model is constructed based on the LLM-based correlation judgment model construction method according to any one of claims 1 to 7.
9. A correlation determination method, comprising:
Recall at least one candidate recall word based on the search term entered by the user;
generating a search relevance judgment prompt according to the search word, the at least one candidate recall word and a preset relevance scoring rule, and inputting the search relevance judgment prompt into a pre-training language model, so that the pre-training language model executes the following operation steps according to the relevance judgment prompt: performing relevance scoring on the search word and each candidate recall word by referring to the relevance scoring rule, and generating and outputting a relevance score between the search word and each candidate recall word and a scoring interpretation for the relevance score;
obtaining a correlation judgment result between the search word and each candidate recall word according to the correlation score between the search word and each candidate recall word and a scoring explanation aiming at the correlation score;
Wherein the pre-trained language model generates relevance scores between the search term and each of the candidate recall terms and an inference process of a scoring interpretation for the relevance scores, including a chain of thought consisting of a plurality of intermediate inference processes;
The preset relevance scoring rules comprise scoring rules of different types of search words;
The plurality of intermediate reasoning processes includes a first intermediate reasoning process, a second intermediate reasoning process, and a third intermediate reasoning process;
the first intermediate reasoning process includes: judging the type of the search word to determine a scoring rule corresponding to the type of the search word;
The second intermediate reasoning process includes: analyzing the semantics of the search word and the semantics of each candidate recall word to judge the relevance between the search word and each candidate recall word based on the analysis result, so as to obtain a relevance judgment result;
The third intermediate reasoning process includes: and determining the relevance scores between the search words and the candidate recall words according to scoring rules corresponding to the types of the search words and the relevance judgment results, and generating scoring interpretation based on the plurality of intermediate reasoning processes.
10. An electronic device, comprising:
A processor;
A memory for storing processor-executable instructions;
Wherein the processor is configured to implement the method of any one of claims 1 to 9 by executing the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410190721.7A CN117743950B (en) | 2024-02-20 | 2024-02-20 | Correlation judgment method and LLM-based correlation judgment model construction method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410190721.7A CN117743950B (en) | 2024-02-20 | 2024-02-20 | Correlation judgment method and LLM-based correlation judgment model construction method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117743950A CN117743950A (en) | 2024-03-22 |
| CN117743950B true CN117743950B (en) | 2024-05-28 |
Family
ID=90251254
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410190721.7A Active CN117743950B (en) | 2024-02-20 | 2024-02-20 | Correlation judgment method and LLM-based correlation judgment model construction method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117743950B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117972097B (en) * | 2024-03-29 | 2024-08-06 | 长城汽车股份有限公司 | Text classification method, classification device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019889A (en) * | 2017-12-01 | 2019-07-16 | 北京搜狗科技发展有限公司 | Training characteristics extract model and calculate the method and relevant apparatus of picture and query word relative coefficient |
| CN115374785A (en) * | 2022-08-24 | 2022-11-22 | 北京智慧星光信息技术有限公司 | Method, system and storage medium for tuning text relevance identification model |
| CN116340383A (en) * | 2022-12-20 | 2023-06-27 | 拉扎斯网络科技（上海）有限公司 | Searching method, device, medium and equipment based on correlation of query and result |
| CN116501950A (en) * | 2022-01-18 | 2023-07-28 | 北京奇虎科技有限公司 | Recall model optimization method, recall model optimization device, recall model optimization equipment and recall model storage medium |
| CN116956931A (en) * | 2022-12-29 | 2023-10-27 | 腾讯科技（深圳）有限公司 | Training method and device of semantic recognition model, storage medium and electronic equipment |
| CN117217278A (en) * | 2023-04-25 | 2023-12-12 | 腾讯科技（深圳）有限公司 | Model training method, text recall method, device, medium and program product |
| CN117391824A (en) * | 2023-12-11 | 2024-01-12 | 深圳须弥云图空间科技有限公司 | Method and device for recommending articles based on large language model and search engine |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10474689B1 (en) * | 2018-04-26 | 2019-11-12 | Quickpath, Inc. | Model score recall for machine learning systems and applications |
-
2024
- 2024-02-20 CN CN202410190721.7A patent/CN117743950B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019889A (en) * | 2017-12-01 | 2019-07-16 | 北京搜狗科技发展有限公司 | Training characteristics extract model and calculate the method and relevant apparatus of picture and query word relative coefficient |
| CN116501950A (en) * | 2022-01-18 | 2023-07-28 | 北京奇虎科技有限公司 | Recall model optimization method, recall model optimization device, recall model optimization equipment and recall model storage medium |
| CN115374785A (en) * | 2022-08-24 | 2022-11-22 | 北京智慧星光信息技术有限公司 | Method, system and storage medium for tuning text relevance identification model |
| CN116340383A (en) * | 2022-12-20 | 2023-06-27 | 拉扎斯网络科技（上海）有限公司 | Searching method, device, medium and equipment based on correlation of query and result |
| CN116956931A (en) * | 2022-12-29 | 2023-10-27 | 腾讯科技（深圳）有限公司 | Training method and device of semantic recognition model, storage medium and electronic equipment |
| CN117217278A (en) * | 2023-04-25 | 2023-12-12 | 腾讯科技（深圳）有限公司 | Model training method, text recall method, device, medium and program product |
| CN117391824A (en) * | 2023-12-11 | 2024-01-12 | 深圳须弥云图空间科技有限公司 | Method and device for recommending articles based on large language model and search engine |
Non-Patent Citations (2)
| Title |
|---|
| Evaluation of undertaking engineering project capabilities of construction companies based on grey relational analysis;Xun zhi yuan et al;《Journal of Qingdao Technological University 》;20150228;第36卷(第1期);9-14 * |
| 基于机器学习的搜索排序算法的研究;程睿;《中国优秀硕士学位论文全文数据库（电子期刊）》;20210215;第2021卷(第02期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117743950A (en) | 2024-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN117033608B (en) | Knowledge graph generation type question-answering method and system based on large language model | |
| CN111324769B (en) | Training method of video information processing model, video information processing method and device | |
| CN117743950B (en) | Correlation judgment method and LLM-based correlation judgment model construction method | |
| Perez et al. | Dialog state tracking, a machine reading approach using memory network | |
| US11531824B2 (en) | Cross-lingual information retrieval and information extraction | |
| CN116821308B (en) | Generation method, training method and device of model and storage medium | |
| CN111460264B (en) | Training method and device for semantic similarity matching model | |
| CN110209824A (en) | Text emotion analysis method based on built-up pattern, system, device | |
| CN105989040A (en) | Intelligent question-answer method, device and system | |
| CN111382573A (en) | Method, apparatus, device and storage medium for answer quality assessment | |
| US20220114644A1 (en) | Recommendation system with sparse feature encoding | |
| CN109189922B (en) | Comment evaluation model training method and device | |
| CN110852071B (en) | Knowledge point detection method, device, equipment and readable storage medium | |
| CN118627511A (en) | Assessment method and device for large language model, storage medium and computer equipment | |
| CN117972118A (en) | Target retrieval method, target retrieval device, electronic equipment and storage medium | |
| CN115374259A (en) | Question and answer data mining method and device and electronic equipment | |
| Jousselme et al. | Uncertain about ChatGPT: enabling the uncertainty evaluation of large language models | |
| CN115836288A (en) | Method and apparatus for generating training data | |
| CN115658845A (en) | Intelligent question-answering method and device suitable for open-source software supply chain | |
| Koller et al. | Who wants to be a Click-Millionaire? On the Influence of Thumbnails and Captions | |
| Narayan et al. | Fake news detection using hybrid of deep neural network and stacked lstm | |
| CN109740162A (en) | Text representation method, device and medium | |
| CN113987360B (en) | Object recommendation method and device, electronic equipment and storage medium | |
| CN116028617B (en) | Information recommendation method, apparatus, device, readable storage medium and program product | |
| AGUH et al. | A RESEARCH ON DETECTION OF FAKE NEWS ON SOCIAL MEDIA USING MACHINE LEARNING ALGORITHMS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |