CN117493588A - Search result determining method and device, storage medium and electronic device - Google Patents
Search result determining method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN117493588A CN117493588A CN202311839056.1A CN202311839056A CN117493588A CN 117493588 A CN117493588 A CN 117493588A CN 202311839056 A CN202311839056 A CN 202311839056A CN 117493588 A CN117493588 A CN 117493588A
- Authority
- CN
- China
- Prior art keywords
- search
- target
- result
- retrieval
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000011156 evaluation Methods 0.000 claims abstract description 70
- 238000013210 evaluation model Methods 0.000 claims description 53
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012549 training Methods 0.000 claims description 11
- 238000012216 screening Methods 0.000 claims description 6
- 230000000694 effects Effects 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 16
- 206010028470 Mycoplasma infections Diseases 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 210000002569 neuron Anatomy 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003058 natural language processing Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 208000024891 symptom Diseases 0.000 description 2
- 101100481876 Danio rerio pbk gene Proteins 0.000 description 1
- 101100481878 Mus musculus Pbk gene Proteins 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000013399 early diagnosis Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 210000002345 respiratory system Anatomy 0.000 description 1
- 230000009385 viral infection Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/38—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/383—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/194—Calculation of difference between files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Library & Information Science (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a method and a device for determining a search result, a storage medium and an electronic device, wherein the method comprises the following steps: the method for determining the search result is characterized by comprising the following steps: retrieving N first retrieval results corresponding to the target problem; determining the evaluation score of each first search result by utilizing the availability score of each first search result and the text characteristic of each first search result to obtain N evaluation scores; determining M second search results from the N first search results based on the N evaluation scores; and retrieving by using each spliced sentence consisting of the second retrieval result and the target problem to obtain M item target retrieval results. By the method and the device, the problem of low retrieval accuracy of the problems in the related technology is solved, and the effect of increasing the retrieval accuracy is achieved.
Description
Technical Field
The embodiment of the application relates to the field of computers, in particular to a method and a device for determining a search result, a storage medium and an electronic device.
Background
At present, the generation result of a large language model (Large Language Model, abbreviated as LLM) is enhanced by retrieving external related information as a Prompt to perform context learning, so as to solve the problem that LLM cannot give or give out outdated answers to some problems with relatively strong timeliness. However, search enhancement is not always effective, and sometimes may even work against, and accurate results may not be accurately retrieved.
Disclosure of Invention
The embodiment of the application provides a method and a device for determining a search result, a storage medium and an electronic device, so as to at least solve the problem of low search accuracy of the problem in the related technology.
According to an embodiment of the present application, there is provided a method for determining a search result, including: retrieving N first retrieval results corresponding to a target question, wherein N first retrieval results comprise answers to the target question, and N is a natural number greater than or equal to 1; determining an evaluation score of each first search result by using the availability score of each first search result and the text characteristic of each first search result to obtain N evaluation scores, wherein the availability score is used for representing the relevance between the first search result and the target problem; determining M second search results from the N first search results based on the N evaluation scores, wherein the evaluation score of each second search result is larger than a first preset threshold value, and M is a natural number smaller than or equal to N; and retrieving by using each spliced sentence formed by the second retrieval result and the target problem to obtain an M-item target retrieval result.
According to another embodiment of the present application, there is provided a search result determining apparatus including: the first acquisition module is used for searching N first search results corresponding to the target problem, wherein the N first search results comprise answers to the target problem, and N is a natural number greater than or equal to 1; the first determining module is used for determining an evaluation score of each first search result by using the availability score of each first search result and the text characteristic of each first search result to obtain N evaluation scores, wherein the availability score is used for representing the relevance between the first search result and the target problem; a second determining module, configured to determine M second search results from the N first search results based on the N evaluation scores, where an evaluation score of each of the second search results is greater than a first preset threshold, and M is a natural number less than or equal to N; and the first input module is used for retrieving M item mark retrieval results by utilizing each spliced statement consisting of the second retrieval result and the target problem.
In an exemplary embodiment, the first obtaining module includes: a first acquiring unit configured to acquire text information having a correlation with the target problem; and a first input unit configured to input a combined text between the target question and the text information into a search enhancement model, to obtain N pieces of the first search results output by the search enhancement model, where the search enhancement model is a model constructed based on a self-attention model, and the combined text is a text set generated by randomly combining a plurality of texts in the target question and the text information.
In an exemplary embodiment, the first determining module includes: the first processing unit is configured to perform the following steps on each of the first search results, to obtain N evaluation scores: inputting the first search result into a target evaluation model to obtain the availability score output by the target evaluation model, wherein the target evaluation model is constructed based on a BERT model; determining a relevance score, a text quality score and a data freshness score of the first search result to obtain the text feature, wherein the relevance score is used for representing semantic relevance between the first search result and the target problem, the text quality score is used for representing accuracy, definition, consistency and logic of the first search result, and the data freshness score is used for representing a time interval between the time for obtaining the first search result and the current time; and calculating a weighted average of the usability score, the relevance score, the text quality score, and the data freshness score to obtain the evaluation score.
In an exemplary embodiment, the above apparatus further includes: a first construction module, configured to input the first search result to a target evaluation model, and construct a sample data set before obtaining the availability score output by the target evaluation model, where the sample data set includes a plurality of sample questions and a plurality of sample search results corresponding to the plurality of sample questions, the plurality of sample search results include a positive sample search result and a negative sample search result in a preset proportion, the positive sample search result is a sample with a semantic similarity greater than or equal to a second preset threshold, and the negative sample search result is a sample with the semantic similarity less than the second preset threshold; and the first training module is used for training the original evaluation model by using the sample data set to obtain the target evaluation model.
In an exemplary embodiment, the first building block includes: a second input unit, configured to input each sample question into the self-attention model, and obtain a first sample result corresponding to each sample question output by the self-attention model; a third input unit, configured to input each of the sample questions and the corresponding first sample result to a search enhancement model, and obtain a second sample result output by the search enhancement model; and a first determining unit configured to determine the first sample result, the second sample result, and a third sample result corresponding to each sample question as the sample data set, where the third sample result is a sample answer corresponding to each sample question.
In an exemplary embodiment, the above apparatus further includes: a third determining module, configured to determine a first semantic similarity between the first sample result and the third sample result, and a second semantic similarity between the second sample result and the third sample result; a fourth determining module, configured to determine, as the negative sample search result, a sample result with the first semantic similarity being greater than the second semantic similarity; and a fifth determining module, configured to determine, as the positive sample search result, a sample result with the first semantic similarity being less than or equal to the second semantic similarity.
In an exemplary embodiment, the first processing unit inputs the first search result to a target evaluation model to obtain the availability score output by the target evaluation model by: and inputting the first search result and the target problem into a target evaluation model, extracting a plurality of search keywords from the first search result through the target evaluation model, calculating correlation scores between the plurality of search keywords and other search results, and calculating the availability score according to the correlation scores between the plurality of search keywords and the other search results, wherein the other search results are the search results in the N first search results.
In an exemplary embodiment, the first processing unit determines the relevance score of the first search result by: extracting a plurality of search keywords in the first search result; and calculating the correlation degree between a plurality of search keywords in the first search result and a question keyword in the target question to obtain the correlation score.
In an exemplary embodiment, the first processing unit determines the text quality score of the first search result by: and inputting the first search result into a text quality scoring model to obtain the text quality score output by the text quality scoring model, wherein the text quality scoring model is used for randomly combining a plurality of search keywords in the first search result so as to determine the accuracy, definition, consistency and logic of the semantics of the first search result from a plurality of combined sentences, and the text quality scoring model is constructed based on the BERT model.
In one exemplary embodiment, the first processing unit determines the data freshness score of the first search result by: determining the time of outputting the first search result from the search enhancement model, and obtaining the time of obtaining the first search result; and normalizing the time interval between the time for obtaining the first search result and the current time to obtain the data freshness score.
In an exemplary embodiment, the second determining module includes: the first sequencing unit is used for carrying out reordering operation on the N first retrieval results according to the N evaluation scores to obtain a target sequence; and a second determining unit configured to determine, as the second search result, search results in the target sequence for which the evaluation scores are all greater than the first preset threshold, and obtain M second search results.
In an exemplary embodiment, the first input module includes: the first execution module is used for executing the following steps on each second search result to obtain M target search results: extracting a plurality of search keywords in the second search result; splicing a plurality of search keywords in the second search result and the target problem according to a preset splicing sequence to obtain a plurality of spliced sentences; and inputting the spliced sentences into a retrieval enhancement model to obtain a target retrieval result output by the retrieval enhancement model, wherein the retrieval enhancement model is a model constructed based on a self-attention model.
In an exemplary embodiment, the above apparatus further includes: the first combination module is used for combining M target search results under the condition that M is greater than 1 after M target search results are obtained by using the spliced sentences formed by each second search result and the target problems; or the first screening module is used for screening out the target retrieval results with highest evaluation scores from the M target retrieval results.
In an exemplary embodiment, the above apparatus further includes: and the first updating module is used for updating the M target search results according to a preset time period after retrieving the M target search results by using the spliced sentences formed by each second search result and the target problems, and displaying the updated search results through the target client.
According to a further embodiment of the present application, there is also provided a computer readable storage medium having stored therein a computer program, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.
According to a further embodiment of the present application, there is also provided an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.
According to the method and the device, the availability score of each first search result corresponding to the target problem output by the search enhancement model is calculated, the availability score and the text characteristics of the first search result are used for calculating the evaluation score of the first search result, the second search result is selected based on the evaluation score, and the M-item target search result is obtained by searching by using the spliced sentences formed by each second search result and the target problem. Namely, the retrieval result is reintroduced into the retrieval enhancement model, so that the retrieval enhancement effect is ensured. Therefore, the problem of low retrieval accuracy of the problems in the related art can be solved, and the effect of increasing the retrieval accuracy is achieved.
Drawings
Fig. 1 is a hardware block diagram of a mobile terminal according to a method for determining a search result according to an embodiment of the present application;
FIG. 2 is a flow chart of a method of determining a search result according to an embodiment of the present application;
FIG. 3 is a flow chart of search result evaluation and reordering according to an embodiment of the present application;
FIG. 4 is a flow chart of calculating an availability score according to an embodiment of the present application;
FIG. 5 is a flow chart one of constructing a sample dataset according to an embodiment of the present application;
FIG. 6 is a second flowchart of constructing a sample dataset according to an embodiment of the present application;
FIG. 7 is a flow chart of constructing a target evaluation model according to an embodiment of the present application;
fig. 8 is a block diagram of the configuration of the determination device of the search result according to the embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in detail below with reference to the accompanying drawings in conjunction with the embodiments.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the embodiments of the present application may be performed in a mobile terminal, a computer terminal or similar computing device. Taking the mobile terminal as an example, fig. 1 is a block diagram of a hardware structure of the mobile terminal according to an embodiment of the present application. As shown in fig. 1, a mobile terminal may include one or more (only one is shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a microprocessor MCU or a processing device such as a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the structure shown in fig. 1 is merely illustrative and not limiting of the structure of the mobile terminal described above. For example, the mobile terminal may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1.
The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a method for determining a search result in the embodiment of the present application, and the processor 102 executes the computer program stored in the memory 104, thereby performing various functional applications and data processing, that is, implementing the method described above. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, which may be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is configured to communicate with the internet wirelessly.
In this embodiment, a method for determining a search result is provided, fig. 2 is a flowchart of a method for determining a search result according to an embodiment of the present application, and as shown in fig. 2, the flowchart includes the following steps:
step S202, searching N first search results corresponding to a target question, wherein N first search results comprise answers to the target question, and N is a natural number greater than or equal to 1;
step S204, determining an evaluation score of each first search result by using the availability score of each first search result and the text characteristic of each first search result to obtain N evaluation scores, wherein the availability scores are used for representing the relevance between the first search result and the target problem;
step S206, determining M second search results from the N first search results based on the N evaluation scores, wherein the evaluation score of each second search result is greater than a first preset threshold value, and M is a natural number less than or equal to N;
step S208, retrieving the M item target retrieval results by using the spliced sentences formed by each second retrieval result and the target problem.
The main body of execution of the above steps may be a specific processor set in a terminal, a server, a terminal or a server, or a processor or a processing device set relatively independently from the terminal or the server, but is not limited thereto.
Alternatively, the target question may be any question issued by the user, and the first search result may be multiple or one.
Optionally, in this embodiment, evaluation of the search result is mainly performed on N first search results corresponding to the target problem output by the search enhancement model, and reordering is performed, so as to obtain an optimal target search result. For example, as shown in fig. 3, includes: 1) The retrieval result evaluation method is to acquire the evaluation result of each retrieval result of the current LLM (LLM) according to the user problem (namely the target problem) and the retrieval result (namely the first retrieval result), namely the availability score of the retrieval result; 2) And the reordering method is used for setting a reasonable threshold according to the availability score, and reordering the retrieval result by combining the text feature score of the result. A score of [0,1] is evaluated, the larger the score is, the more the retrieval enhancement model is needed, the higher the availability is, the lower the score is, the lower the availability of the result to the retrieval enhancement model is, and if the score is 0, the result is not needed by the retrieval enhancement model. For example, the objective problem is "what the doctor has a relation to artificial intelligence", and one search result is "with the advent of industry 4.0, intelligent robots are an important part of future productivity as an important ring. In the intelligent robot industry, ai Bobi robot companies are always at the front of the industry, and continuously advance the application of the AI technology to bring the intelligent robot into a new field, wherein the evaluation score of the search result is 0 score, and the search result is not useful for the search enhancement model. Another search result is "Aidoctor say: advantageously, we now say another form of neural network: convolutional neural networks. First we look at what is deficient in fully-connected neural networks. Just like the name, the neural network is fully connected, and the neurons of two adjacent layers are connected, so that when the number of the neurons is relatively large, the connection weight is very large, on one hand, the training speed of the neural network can be influenced, and on the other hand, the calculation speed can be influenced when the neural network is used. In fact, in some cases, the neuron is sharable ", and the evaluation score of the search result is 0.8, which is a search result useful for the search enhancement model. The retrieval result and the target question forming a spliced sentence can be input into the retrieval enhancement model, the target retrieval result is regenerated, and the target retrieval result is the result closest to the correct answer.
Through the steps, the availability score of each first search result corresponding to the target problem output by the search enhancement model is calculated, the availability score and the text characteristics of the first search result are used for calculating the evaluation score of the first search result, the second search result is selected based on the evaluation score, and the M-item target search result is obtained by searching the spliced sentences formed by each second search result and the target problem. Namely, the retrieval result is reintroduced into the retrieval enhancement model, so that the retrieval enhancement effect is ensured. Therefore, the problem of low retrieval accuracy of the problems in the related art can be solved, and the effect of increasing the retrieval accuracy is achieved.
In one exemplary embodiment, retrieving N first retrieval results corresponding to a target problem includes: acquiring text information with relevance to a target problem; and inputting a combined text between the target problem and the text information into a retrieval enhancement model to obtain N pieces of first retrieval results output by the retrieval enhancement model, wherein the retrieval enhancement model is a model constructed based on a self-attention model, and the combined text is a text set generated by randomly combining a plurality of texts in the target problem and the text information.
Alternatively, the self-attention model is a model for understanding and simulating human thought and cognitive processes. It processes information by focusing on and weighting different inputs based on a self-attention mechanism. Used in the fields of natural language processing, speech recognition, computer vision, etc., to assist machines in understanding and processing complex information. The core idea of the self-attention model is to determine the focus of the model when processing the input by calculating the correlation of different positions in the input sequence. This mechanism enables the model to more effectively capture correlations between the input data, thereby increasing its ability to process complex information.
Alternatively, the self-attention model may be a large language model LLM with multiple aspects of language and code understanding, human instruction compliance, logical reasoning, and the like. LLM may not be able to give or give outdated answers to some more time-efficient questions. The enhancement of LLM by retrieving external relevant information as a template for contextual learning results in a retrieval enhancement model. That is, an external database is provided for LLM, and for user problems (Query), information related to the user problems, also referred to as search results, is first searched from the external database by means of information search (Information Retrieval, abbreviated as IR), and then the LLM is allowed to combine the related information to generate results.
Optionally, the text information that is relevant to the target problem includes, but is not limited to, text information that is directly related and indirectly related. For example, the objective problem is "how long is mycoplasma infection latency? The text information related to the target problem includes, but is not limited to, "symptoms of mycoplasma infection", "diagnosis of mycoplasma infection", "season of mycoplasma infection", etc. After the target problem and the text information are randomly spliced, the target problem and the text information are input into a retrieval enhancement model as the prompt, and the retrieval enhancement model outputs 'the symptoms caused by mycoplasma infection are various and similar to those caused by other respiratory tract virus infection', so that early diagnosis has a certain difficulty. According to the embodiment, the target problems and the related text information are randomly combined and input into the retrieval enhancement model, so that relatively wide retrieval results can be output, the diversity of the retrieval results is increased, and more text features of the retrieval results are obtained.
In one exemplary embodiment, determining the evaluation score of each first search result using the availability score of each first search result and the text feature of each first search result, to obtain N evaluation scores, includes: the following steps are executed for each first search result to obtain N evaluation scores: inputting the first search result into a target evaluation model to obtain an availability score output by the target evaluation model, wherein the target evaluation model is a model constructed based on the BERT model; determining a relevance score, a text quality score and a data freshness score of the first search result to obtain text characteristics, wherein the relevance score is used for representing semantic relevance between the first search result and a target problem, the text quality score is used for representing accuracy, definition, continuity and logic of the first search result, and the data freshness score is used for representing a time interval between the time for obtaining the first search result and the current time; and calculating a weighted average of the usability score, the relevance score, the text quality score and the data freshness score to obtain an evaluation score.
Alternatively, BERT (Bidirectional Encoder Representations from Transformers) is a natural language processing pre-training model developed by Google. The model is based on a transducer architecture, can perform unsupervised pre-training on a large-scale text corpus, and can perform fine tuning in various downstream tasks. The BERT model is characterized by being capable of understanding the two-way context information of the text and capturing the semantic and context information in the language more accurately. For example, as shown in fig. 4, the package objective evaluation model output availability score includes: the user question and a search result are input to a search result evaluation model (i.e., a target evaluation model), and an availability score corresponding to the search result is output.
Alternatively, text quality refers to the accuracy, clarity, consistency, and logic of text. The text should have the correct information, clear and understandable language, clear structured and logical rigorous demonstration.
Alternatively, the results retrieved at different time periods are different, and the freshness of the data determines whether the retrieved result is the most current result.
Alternatively, for example, the availability score is q 1 Correlation score q 2 The text quality score is q 3 Data freshness score q 4 And (3) calculating the final comprehensive evaluation score of the search result by weighted average:wherein, the method comprises the steps of, wherein,。
in the embodiment, the evaluation score is obtained by calculating the weighted average of the usability score, the relevance score, the text quality score and the data freshness score, so that the relatively accurate evaluation score can be obtained.
In an exemplary embodiment, before inputting the first search result to the target evaluation model to obtain the availability score output by the target evaluation model, the method further includes: constructing a sample data set, wherein the sample data set comprises a plurality of sample problems and a plurality of sample retrieval results corresponding to the plurality of sample problems, the plurality of sample retrieval results comprise positive sample retrieval results and negative sample retrieval results in preset proportions, the positive sample retrieval results are samples with semantic similarity being greater than or equal to a second preset threshold, and the negative sample retrieval results are samples with semantic similarity being less than the second preset threshold; and training the original evaluation model by using the sample data set to obtain a target evaluation model.
Optionally, the sample data set includes a preset number of sample questions, and one or more sample retrieval results corresponding to each sample question.
Optionally, constructing the sample dataset comprises: inputting each sample problem into the self-attention model to obtain a first sample result corresponding to each sample problem output by the self-attention model; inputting each sample problem and the corresponding first sample result into a retrieval enhancement model to obtain a second sample result output by the retrieval enhancement model; and determining a first sample result, a second sample result and a third sample result corresponding to each sample question as a sample data set, wherein the third sample result is a sample answer corresponding to each sample question. For example, as shown in fig. 5, according to the user problem, the large language model generates a direct result (corresponding to the first sample result in the above), denoted as (query, answer); the retrieval enhancement model generates a retrieval enhancement result (corresponding to the second sample result in the above) based on the user question and the first sample result, and is noted as (query, ir_answer); the user questions and the correct answers written by the human (corresponding to the third sample result) are recorded as (query, true_answer), and the user questions and the search results are subjected to sample classification according to the answer, the true_answer and the true_answer to obtain a sample data set.
Optionally, the method further comprises: determining a first semantic similarity between the first sample result and the third sample result, and a second semantic similarity between the second sample result and the third sample result; determining a sample result with the first semantic similarity being greater than the second semantic similarity as a negative sample retrieval result; and determining the sample result with the first semantic similarity smaller than or equal to the second semantic similarity as a positive sample retrieval result. For example, as shown in fig. 6, the result similarity is calculated according to answer, ir_answer, and true_answer, and if the semantic similarity of (answer, true_answer) is higher than the semantic similarity of (ir_answer, true_answer) (corresponding to the first semantic similarity being greater than the second semantic similarity), it is indicated that the search result does not substantially improve the quality of the model reasoning result, then it is a negative sample (corresponding to the negative sample search result), otherwise it is a positive sample (corresponding to the positive sample search result). The number of sample data sets is typically tens of thousands of samples, and in this embodiment, thirty thousands of samples are used, and the ratio of the number of positive and negative samples is 1:1. According to the method and the device, the accuracy of the training target evaluation model is improved by classifying the negative sample retrieval result and the positive sample retrieval result.
Alternatively, as shown in fig. 7, the present embodiment uses a BERT model, and performs training optimization based on a sample data set to obtain a target evaluation model.
Optionally, inputting the first search result to the target evaluation model to obtain an availability score output by the target evaluation model, including: the first search result and the target problem are input into a target evaluation model, so that a plurality of search keywords are extracted from the first search result through the target evaluation model, correlation scores between the plurality of search keywords and other search results are calculated, availability scores are calculated according to the correlation scores between the plurality of search keywords and other search results, and the other search results are search results in N pieces of first search results. In this embodiment, the search keyword is an effective word extracted from the first search result, for example, when the first search result is "mycoplasma infection latency is generally 1 to 3 weeks unequal", the extracted search keyword includes "latency", "1 to 3 weeks", and the like. The relevance score between the search keyword and other search results is calculated based on text feature matching. According to the embodiment, the search keywords and the keywords of other search results are extracted, so that the correlation degree between the search results and other search results can be accurately calculated.
Alternatively, the present embodiment may determine the relevance score of the first search result by: extracting a plurality of search keywords in a first search result; and calculating the correlation degree between the plurality of search keywords in the first search result and the problem keywords in the target problem to obtain a correlation score. In this embodiment, the correlation degree between the search keyword and the question keyword in the target question is also calculated based on text feature matching. According to the method and the device, the relevance between the search result and the target problem can be accurately calculated by extracting the search keywords and the problem keywords.
Optionally, the present embodiment determines the text quality score of the first search result by: and inputting the first search result into a text quality scoring model to obtain a text quality score output by the text quality scoring model, wherein the text quality scoring model is used for randomly combining a plurality of search keywords in the first search result so as to determine the accuracy, definition, consistency and logic of the semantics of the first search result from a plurality of combined sentences, and the text quality scoring model is a model constructed based on the BERT model. In this embodiment, the text quality score is a combination of weighted semantic accuracy, sharpness, consistency, and logic.
Alternatively, the present embodiment determines the data freshness score of the first search result by: determining the time of outputting the first retrieval result from the retrieval enhancement model, and obtaining the time of obtaining the first retrieval result; and carrying out normalization processing on the time interval between the time for obtaining the first retrieval result and the current time to obtain a data freshness score. In this embodiment, the shorter the time interval, the more fresh the data.
In one exemplary embodiment, determining M second search results from the N first search results based on the N evaluation scores includes: re-ordering the N first retrieval results according to the N evaluation scores to obtain a target sequence; and determining the retrieval results with the evaluation scores larger than the first preset threshold value in the target sequence as second retrieval results to obtain M second retrieval results. Alternatively, the higher the score in the target sequence, the more forward, the closer to the target problem. And selecting a result of the previous TopK with the score larger than a threshold value and the target problem to be spliced together as a Prompt input target evaluation model according to the promt length limit supported by the target evaluation model and the reasoning efficiency requirement of the target evaluation model. The first preset threshold is preset, and in this embodiment is set to 0.6. According to the embodiment, the retrieval result approaching the target problem can be rapidly determined by carrying out the reordering operation on the retrieval result.
In an exemplary embodiment, retrieving the M-item target retrieval result by using the spliced statement composed of each second retrieval result and the target question includes: the following steps are executed for each second search result to obtain M item mark search results: extracting a plurality of search keywords in the second search result; splicing a plurality of search keywords and target problems in the second search result according to a preset splicing sequence to obtain a plurality of spliced sentences; and inputting the plurality of spliced sentences into a retrieval enhancement model to obtain a target retrieval result output by the retrieval enhancement model, wherein the retrieval enhancement model is a model constructed based on the self-attention model. Alternatively, a plurality of search keywords and target questions may be randomly spliced.
In an exemplary embodiment, after retrieving the M-item target retrieval result by using the spliced statement composed of each second retrieval result and the target question, the method further includes: under the condition that M is larger than 1, combining M item mark retrieval results; or screening out the target retrieval result with the highest evaluation score from the M-item target retrieval results. Alternatively, the manner of combining the M-item target search results is not limited, and the M-item target search results may be sequentially arranged into the search result list. Or may be randomly combined. According to the embodiment, the target search result is processed again, so that the search result which is closer to the target problem can be determined.
In an exemplary embodiment, after retrieving the M-item target retrieval result by using the spliced statement composed of each second retrieval result and the target question, the method further includes: and updating the M item target retrieval results according to a preset time period, and displaying the updated retrieval results through the target client. Alternatively, the preset time period may be one week or one month. The method for updating the search result is the same as the method for determining the target search result, and will not be described in detail here. The embodiment can ensure the real-time performance of the search result by updating the search result.
The present application is described below in connection with specific embodiments:
the embodiment provides a method for evaluating and reordering the retrieval results in a retrieval enhancement model, which can evaluate and reorder the retrieval results and answer the question of whether the retrieval results are used and how to use the retrieval results, so that the retrieved results are reasonably used, and the purpose of improving the quality of model reasoning results is realized. The method specifically comprises the following steps:
s1, based on the user problem "the talent is a new stock of the life has undergone several marital Will (query, ir_answer i ) The retrieval result evaluation module evaluates the piece-by-piece related information to obtain availability score q 1i 0.35,0.67,0.89,0.01,0.12, respectively.
S2, respectively calculating text quality scores q in a retrieval result reordering module 3i And data freshness score q 4i Combining the obtained correlation scores q 2i Availability score q 1i The final comprehensive evaluation score of the search result can be calculated by weighted average. The threshold value is preset to 0.6, wherein only +.>And->Is above a threshold value and ∈>。
S3 because only ir_answer 2 And ir_answer 3 Is preserved and the most important information is spliced near the problem, and the final prompt is ir_answer 2 +ir_answer 3 The +query is fed into LLM together, and the model reasoning results are "Liqing life experienced two marital events". Her first husband was Zhao Mingcheng, and Zhao Mingcheng later was removed by illness. The second marital is coupled to Zhang Ruzhou, but this is not happy because Zhang Ruzhou is a potential adult, exposing nature after wedding, and then the clearing of the plum selects divorce.
In summary, the present embodiment comprehensively considers the characteristics of multiple aspects of the search result, including four aspects of availability, correlation, text quality score and data freshness, so as to ensure that the search result with high availability, high correlation, high text quality and newer data is closer to the user problem, thereby obtaining greater attention of LLM and obtaining the reasoning result with higher quality.
It should be noted that, the retrieval result evaluation and reordering device based on the retrieval enhancement model provided in this embodiment and the retrieval result evaluation and reordering method in the retrieval enhancement model described above may be referred to correspondingly, which is not described herein.
From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method described in the embodiments of the present application.
The embodiment also provides a device for determining a search result, which is used for implementing the foregoing embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
Fig. 8 is a block diagram of a search result determining apparatus according to an embodiment of the present application, as shown in fig. 8, including:
a first obtaining module 82, configured to retrieve N first retrieval results corresponding to a target question, where N first retrieval results each include an answer to the target question, and N is a natural number greater than or equal to 1;
a first determining module 84, configured to determine an evaluation score of each of the first search results by using an availability score of each of the first search results and a text feature of each of the first search results, to obtain N evaluation scores, where the availability score is used to represent a relevance between the first search result and the target problem;
a second determining module 86, configured to determine M second search results from the N first search results based on the N evaluation scores, where an evaluation score of each of the second search results is greater than a first preset threshold, and M is a natural number less than or equal to N;
and a first input module 88, configured to retrieve M target retrieval results by using each of the second retrieval results and the spliced sentences formed by the target questions.
In an exemplary embodiment, the first obtaining module includes: a first acquiring unit configured to acquire text information having a correlation with the target problem; and a first input unit configured to input a combined text between the target question and the text information into a search enhancement model, to obtain N pieces of the first search results output by the search enhancement model, where the search enhancement model is a model constructed based on a self-attention model, and the combined text is a text set generated by randomly combining a plurality of texts in the target question and the text information.
In an exemplary embodiment, the first determining module includes: the first processing unit is configured to perform the following steps on each of the first search results, to obtain N evaluation scores: inputting the first search result into a target evaluation model to obtain the availability score output by the target evaluation model, wherein the target evaluation model is constructed based on a BERT model; determining a relevance score, a text quality score and a data freshness score of the first search result to obtain the text feature, wherein the relevance score is used for representing semantic relevance between the first search result and the target problem, the text quality score is used for representing accuracy, definition, consistency and logic of the first search result, and the data freshness score is used for representing a time interval between the time for obtaining the first search result and the current time; and calculating a weighted average of the usability score, the relevance score, the text quality score, and the data freshness score to obtain the evaluation score.
In an exemplary embodiment, the above apparatus further includes: a first construction module, configured to input the first search result to a target evaluation model, and construct a sample data set before obtaining the availability score output by the target evaluation model, where the sample data set includes a plurality of sample questions and a plurality of sample search results corresponding to the plurality of sample questions, the plurality of sample search results include a positive sample search result and a negative sample search result in a preset proportion, the positive sample search result is a sample with a semantic similarity greater than or equal to a second preset threshold, and the negative sample search result is a sample with the semantic similarity less than the second preset threshold; and the first training module is used for training the original evaluation model by using the sample data set to obtain the target evaluation model.
In an exemplary embodiment, the first building block includes: a second input unit, configured to input each sample question into the self-attention model, and obtain a first sample result corresponding to each sample question output by the self-attention model; a third input unit, configured to input each of the sample questions and the corresponding first sample result to a search enhancement model, and obtain a second sample result output by the search enhancement model; and a first determining unit configured to determine the first sample result, the second sample result, and a third sample result corresponding to each sample question as the sample data set, where the third sample result is a sample answer corresponding to each sample question.
In an exemplary embodiment, the above apparatus further includes: a third determining module, configured to determine a first semantic similarity between the first sample result and the third sample result, and a second semantic similarity between the second sample result and the third sample result; a fourth determining module, configured to determine, as the negative sample search result, a sample result with the first semantic similarity being greater than the second semantic similarity; and a fifth determining module, configured to determine, as the positive sample search result, a sample result with the first semantic similarity being less than or equal to the second semantic similarity.
In an exemplary embodiment, the first processing unit inputs the first search result to a target evaluation model to obtain the availability score output by the target evaluation model by: and inputting the first search result and the target problem into a target evaluation model, extracting a plurality of search keywords from the first search result through the target evaluation model, calculating correlation scores between the plurality of search keywords and other search results, and calculating the availability score according to the correlation scores between the plurality of search keywords and the other search results, wherein the other search results are the search results in the N first search results.
In an exemplary embodiment, the first processing unit determines the relevance score of the first search result by: extracting a plurality of search keywords in the first search result; and calculating the correlation degree between a plurality of search keywords in the first search result and a question keyword in the target question to obtain the correlation score.
In an exemplary embodiment, the first processing unit determines the text quality score of the first search result by: and inputting the first search result into a text quality scoring model to obtain the text quality score output by the text quality scoring model, wherein the text quality scoring model is used for randomly combining a plurality of search keywords in the first search result so as to determine the accuracy, definition, consistency and logic of the semantics of the first search result from a plurality of combined sentences, and the text quality scoring model is constructed based on the BERT model.
In one exemplary embodiment, the first processing unit determines the data freshness score of the first search result by: determining the time of outputting the first search result from the search enhancement model, and obtaining the time of obtaining the first search result; and normalizing the time interval between the time for obtaining the first search result and the current time to obtain the data freshness score.
In an exemplary embodiment, the second determining module includes: the first sequencing unit is used for carrying out reordering operation on the N first retrieval results according to the N evaluation scores to obtain a target sequence; and a second determining unit configured to determine, as the second search result, search results in the target sequence for which the evaluation scores are all greater than the first preset threshold, and obtain M second search results.
In an exemplary embodiment, the first input module includes: the first execution module is used for executing the following steps on each second search result to obtain M target search results: extracting a plurality of search keywords in the second search result; splicing a plurality of search keywords in the second search result and the target problem according to a preset splicing sequence to obtain a plurality of spliced sentences; and inputting the spliced sentences into a retrieval enhancement model to obtain a target retrieval result output by the retrieval enhancement model, wherein the retrieval enhancement model is a model constructed based on a self-attention model.
In an exemplary embodiment, the above apparatus further includes: the first combination module is used for combining M target search results under the condition that M is greater than 1 after M target search results are obtained by using the spliced sentences formed by each second search result and the target problems; or the first screening module is used for screening out the target retrieval results with highest evaluation scores from the M target retrieval results.
In an exemplary embodiment, the above apparatus further includes: and the first updating module is used for updating the M target search results according to a preset time period after retrieving the M target search results by using the spliced sentences formed by each second search result and the target problems, and displaying the updated search results through the target client.
It should be noted that each of the above modules may be implemented by software or hardware, and for the latter, it may be implemented by, but not limited to: the modules are all located in the same processor; alternatively, the above modules may be located in different processors in any combination.
Embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.
In one exemplary embodiment, the computer readable storage medium may include, but is not limited to: a usb disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing a computer program.
Embodiments of the present application also provide an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.
In an exemplary embodiment, the electronic device may further include a transmission device connected to the processor, and an input/output device connected to the processor.
Specific examples in this embodiment may refer to the examples described in the foregoing embodiments and the exemplary implementation, and this embodiment is not described herein.
It will be appreciated by those skilled in the art that the modules or steps of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.
The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the same, but rather, various modifications and variations may be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application should be included in the protection scope of the present application.
Claims (17)
1. A method for determining a search result, comprising:
retrieving N first retrieval results corresponding to a target question, wherein N first retrieval results comprise answers to the target question, and N is a natural number greater than or equal to 1;
determining an evaluation score of each first search result by using the availability score of each first search result and the text characteristic of each first search result to obtain N evaluation scores, wherein the availability score is used for representing the relevance between the first search result and the target problem;
determining M second search results from the N first search results based on the N evaluation scores, wherein the evaluation score of each second search result is larger than a first preset threshold value, and M is a natural number smaller than or equal to N;
And retrieving by using each spliced statement consisting of the second retrieval result and the target problem to obtain M item target retrieval results.
2. The method of claim 1, wherein retrieving N first retrieval results corresponding to the target problem comprises:
acquiring text information related to the target problem;
and inputting a combined text between the target problem and the text information into a retrieval enhancement model to obtain N first retrieval results output by the retrieval enhancement model, wherein the retrieval enhancement model is a model constructed based on a self-attention model, and the combined text is a text set generated by randomly combining a plurality of texts in the target problem and the text information.
3. The method of claim 1, wherein determining an evaluation score for each of the first search results using the availability score for each of the first search results and the text feature of each of the first search results, and obtaining N of the evaluation scores, comprises:
the following steps are executed for each first search result to obtain N evaluation scores:
inputting the first search result into a target evaluation model to obtain the availability score output by the target evaluation model, wherein the target evaluation model is a model constructed based on a BERT model;
Determining a relevance score, a text quality score and a data freshness score of the first search result to obtain the text feature, wherein the relevance score is used for representing semantic relevance between the first search result and the target problem, the text quality score is used for representing accuracy, definition, consistency and logic of the first search result, and the data freshness score is used for representing a time interval between the time for obtaining the first search result and the current time;
and calculating a weighted average of the availability score, the relevance score, the text quality score and the data freshness score to obtain the evaluation score.
4. A method according to claim 3, wherein before inputting the first search result to a target evaluation model to obtain the availability score output by the target evaluation model, the method further comprises:
constructing a sample data set, wherein the sample data set comprises a plurality of sample problems and a plurality of sample retrieval results corresponding to the sample problems, the plurality of sample retrieval results comprise positive sample retrieval results and negative sample retrieval results with preset proportions, the positive sample retrieval results are samples with semantic similarity being greater than or equal to a second preset threshold, and the negative sample retrieval results are samples with the semantic similarity being smaller than the second preset threshold;
And training an original evaluation model by using the sample data set to obtain the target evaluation model.
5. The method of claim 4, wherein constructing the sample dataset comprises:
inputting each sample problem into a self-attention model to obtain a first sample result corresponding to each sample problem output by the self-attention model;
inputting each sample problem and the corresponding first sample result into a retrieval enhancement model to obtain a second sample result output by the retrieval enhancement model;
and determining the first sample result, the second sample result and a third sample result corresponding to each sample question as the sample data set, wherein the third sample result is a sample answer corresponding to each sample question.
6. The method of claim 5, wherein the method further comprises:
determining a first semantic similarity between the first sample result and the third sample result, and a second semantic similarity between the second sample result and the third sample result;
determining a sample result with the first semantic similarity being greater than the second semantic similarity as the negative sample retrieval result;
And determining a sample result with the first semantic similarity smaller than or equal to the second semantic similarity as the positive sample retrieval result.
7. A method according to claim 3, wherein inputting the first search result into a target evaluation model to obtain the availability score output by the target evaluation model comprises:
and inputting the first search result and the target problem into a target evaluation model, extracting a plurality of search keywords from the first search result through the target evaluation model, calculating correlation scores among the plurality of search keywords and other search results, and calculating the availability score according to the correlation scores among the plurality of search keywords and the other search results, wherein the other search results are the search results in the N first search results.
8. A method according to claim 3, wherein the relevance score of the first search result is determined by:
extracting a plurality of search keywords in the first search result;
and calculating the correlation degree between a plurality of search keywords in the first search result and the problem keywords in the target problem to obtain the correlation score.
9. A method according to claim 3, characterized in that the text quality score of the first search result is determined by:
inputting the first search result into a text quality scoring model to obtain the text quality score output by the text quality scoring model, wherein the text quality scoring model is used for randomly combining a plurality of search keywords in the first search result so as to determine the accuracy, definition, consistency and logic of the semantics of the first search result from a plurality of combined sentences, and the text quality scoring model is a model constructed based on the BERT model.
10. A method according to claim 3, wherein the data freshness score of the first search result is determined by:
determining the time of outputting the first retrieval result from the retrieval enhancement model, and obtaining the time of obtaining the first retrieval result;
and normalizing the time interval between the time for obtaining the first retrieval result and the current time to obtain the data freshness score.
11. The method of claim 1, wherein determining M second search results from the N first search results based on the N evaluation scores comprises:
Performing reordering operation on the N first retrieval results according to the N evaluation scores to obtain a target sequence;
and determining the retrieval results with the evaluation scores larger than the first preset threshold value in the target sequence as the second retrieval results to obtain M second retrieval results.
12. The method according to claim 1, wherein retrieving the M-gram retrieval result using each of the second retrieval result and the spliced statement composed of the target question comprises:
the following steps are executed for each second search result to obtain M target search results:
extracting a plurality of search keywords in the second search result;
splicing a plurality of search keywords in the second search result and the target problem according to a preset splicing sequence to obtain a plurality of spliced sentences;
and inputting a plurality of spliced sentences into a retrieval enhancement model to obtain a target retrieval result output by the retrieval enhancement model, wherein the retrieval enhancement model is a model constructed based on a self-attention model.
13. The method according to claim 1, wherein after retrieving M target retrieval results using each of the second retrieval results and the spliced sentences of target questions, the method further comprises:
Combining M target search results under the condition that M is greater than 1; or,
and screening out target retrieval results with highest evaluation scores from the M target retrieval results.
14. The method according to claim 1, wherein after retrieving M target retrieval results using each of the second retrieval results and the spliced sentences of target questions, the method further comprises:
and updating M target search results according to a preset time period, and displaying the updated search results through a target client.
15. A search result determining apparatus, comprising:
the first acquisition module is used for searching N first search results corresponding to the target problem, wherein N first search results comprise answers to the target problem, and N is a natural number greater than or equal to 1;
the first determining module is used for determining an evaluation score of each first search result by using the availability score of each first search result and the text characteristic of each first search result to obtain N evaluation scores, wherein the availability score is used for representing the relevance between the first search result and the target problem;
The second determining module is used for determining M second search results from N first search results based on N evaluation scores, wherein the evaluation score of each second search result is larger than a first preset threshold value, and M is a natural number smaller than or equal to N;
and the first input module is used for retrieving M item mark retrieval results by utilizing the spliced sentences formed by each second retrieval result and the target problem.
16. A computer readable storage medium, characterized in that a computer program is stored in the computer readable storage medium, wherein the computer program, when being executed by a processor, implements the steps of the method according to any of the claims 1 to 14.
17. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any one of claims 1 to 14 when the computer program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311839056.1A CN117493588B (en) | 2023-12-28 | 2023-12-28 | Search result determining method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311839056.1A CN117493588B (en) | 2023-12-28 | 2023-12-28 | Search result determining method and device, storage medium and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117493588A true CN117493588A (en) | 2024-02-02 |
| CN117493588B CN117493588B (en) | 2024-03-22 |
Family
ID=89671209
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311839056.1A Active CN117493588B (en) | 2023-12-28 | 2023-12-28 | Search result determining method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117493588B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009295101A (en) * | 2008-06-09 | 2009-12-17 | Hitachi Ltd | Speech data retrieval system |
| CN113806510A (en) * | 2021-09-22 | 2021-12-17 | 中国科学院深圳先进技术研究院 | Legal provision retrieval method, terminal device and computer storage medium |
-
2023
- 2023-12-28 CN CN202311839056.1A patent/CN117493588B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2009295101A (en) * | 2008-06-09 | 2009-12-17 | Hitachi Ltd | Speech data retrieval system |
| CN113806510A (en) * | 2021-09-22 | 2021-12-17 | 中国科学院深圳先进技术研究院 | Legal provision retrieval method, terminal device and computer storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117493588B (en) | 2024-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111353310B (en) | Named entity identification method and device based on artificial intelligence and electronic equipment | |
| CN108363790B (en) | Method, device, equipment and storage medium for evaluating comments | |
| KR102288249B1 (en) | Information processing method, terminal, and computer storage medium | |
| CN109840287A (en) | A kind of cross-module state information retrieval method neural network based and device | |
| CN112765312B (en) | Knowledge graph question-answering method and system based on graph neural network embedded matching | |
| CN111159485B (en) | Tail entity linking method, device, server and storage medium | |
| CN110765247A (en) | Input prompting method and device for question-answering robot | |
| CN110795544B (en) | Content searching method, device, equipment and storage medium | |
| CN112307048B (en) | Semantic matching model training method, matching method, device, equipment and storage medium | |
| CN116992007B (en) | Limiting question-answering system based on question intention understanding | |
| CN112101042A (en) | Text emotion recognition method and device, terminal device and storage medium | |
| CN111241248A (en) | Synonymy question generation model training method and system and synonymy question generation method | |
| Xue et al. | Automatic generation and recommendation for API mashups | |
| CN112925904A (en) | Lightweight text classification method based on Tucker decomposition | |
| CN114647713A (en) | Knowledge graph question-answering method, device and storage medium based on virtual confrontation | |
| CN117290488A (en) | Man-machine interaction method and device based on large model, electronic equipment and storage medium | |
| CN116821307B (en) | Content interaction method, device, electronic equipment and storage medium | |
| CN111783473B (en) | Method and device for identifying best answer in medical question and answer and computer equipment | |
| CN112949293A (en) | Similar text generation method, similar text generation device and intelligent equipment | |
| CN117493588B (en) | Search result determining method and device, storage medium and electronic device | |
| CN116680401A (en) | Document processing method, document processing device, apparatus and storage medium | |
| Novais et al. | Facial emotions classification supported in an ensemble strategy | |
| Ribeiro et al. | UA. PT Bioinformatics at ImageCLEF 2019: Lifelog Moment Retrieval based on Image Annotation and Natural Language Processing. | |
| CN114238798A (en) | Search ranking method, system, device and storage medium based on neural network | |
| Jeong et al. | Label and context augmentation for response selection at DSTC8 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |