CN117725423B

CN117725423B - Method and device for generating feedback information based on large model

Info

Publication number: CN117725423B
Application number: CN202410179457.7A
Authority: CN
Inventors: 邓邱伟; 田云龙; 刘广通; 尹飞; 王淼
Original assignee: Qingdao Haier Technology Co Ltd; Qingdao Haier Intelligent Home Appliance Technology Co Ltd; Haier Uplus Intelligent Technology Beijing Co Ltd
Current assignee: Qingdao Haier Technology Co Ltd; Qingdao Haier Intelligent Home Appliance Technology Co Ltd; Haier Uplus Intelligent Technology Beijing Co Ltd
Priority date: 2024-02-18
Filing date: 2024-02-18
Publication date: 2024-05-24
Anticipated expiration: 2044-02-18
Also published as: CN117725423A

Abstract

The application discloses a method and a device for generating feedback information based on a large model, and relates to the technical field of smart families, wherein the method comprises the following steps: classifying input information input into the large model by a knowledge classifier in the large model to obtain a classification result, and searching a first knowledge segment corresponding to the input information in a first knowledge base indicated by the classification result; determining the similarity between M second knowledge pieces included in the first knowledge pieces and input information, and determining second knowledge pieces corresponding to the N similarities respectively; determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model; the K third knowledge pieces are input to the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge pieces and the input information. The effect of generating high-quality feedback information aiming at the input information of the user is achieved.

Description

Method and device for generating feedback information based on large model

Technical Field

The application relates to the technical field of smart families, in particular to a method and a device for generating feedback information based on a large model.

Background

The wave tide of the large model (Large Language Model, abbreviated as LLM) has already rolled up almost every industry, but when related to professional scenes or industry subdivision areas, the general large model faces the problem of insufficient professional knowledge.

Specifically, 1) large models have knowledge limitations: the knowledge of the model is completely derived from training data of the model, and the training set of the existing mainstream large model is basically constructed on the data disclosed by the network, and cannot be obtained for some real-time, non-disclosed or offline data, so that the knowledge is not available. 2) Large models have the illusion problem: all the underlying principles of artificial intelligence (ARTIFICIAL INTELLIGENCE, AI for short) models are based on mathematical probabilities, the model output of which is essentially a series of numerical operations, and large models are no exception, so that large models sometimes give answers that are irrelevant to the problem or answers that are inconsistent with the actual situation, etc., especially when faced with a scene where the large model itself does not have knowledge or is not good at. The distinction of such illusion problems requires knowledge of the user itself in the relevant field and is therefore often difficult. 3) Data security: the data security is critical, no mechanism is willing to bear the risk of data leakage, and private domain data of the mechanism is uploaded to a third party platform for training. This also results in application schemes that rely entirely on the capabilities of the generic large model itself having to trade off data security and effectiveness.

Aiming at the problems of knowledge limitation, illusion problem, data security and the like in the related technology, a large model cannot generate targeted feedback information for user input information belonging to different knowledge fields, and an effective solution is not proposed.

Accordingly, there is a need for improvements in the related art to overcome the drawbacks of the related art.

Disclosure of Invention

The embodiment of the application provides a method and a device for generating feedback information based on a large model, which are used for at least solving the problems of knowledge limitation, illusion, data security and the like in the related technology, wherein the large model cannot generate targeted feedback information for user input information belonging to different knowledge fields.

According to an aspect of the embodiment of the present application, there is provided a method for generating feedback information based on a large model, including: classifying input information input into the large model by a knowledge classifier in the large model to obtain a classification result, and searching a first knowledge segment corresponding to the input information in a first knowledge base indicated by the classification result; determining the similarity between M second knowledge pieces included in the first knowledge pieces and the input information, and determining second knowledge pieces corresponding to N similarities respectively, wherein the minimum similarity of the N similarities is larger than the maximum similarity of other similarities, and the other similarities are similarities except the N similarities; determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model; inputting K third knowledge pieces into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge pieces and the input information; wherein,

M, N and K are positive integers, M is greater than N, and N is greater than or equal to K; wherein, in the case that the first knowledge base includes a plurality of second knowledge bases, the value of M is determined by the following formula: ; wherein n represents the number of a plurality of second knowledge bases included in the first knowledge base, W is a first number of second knowledge segments which are preset for each second knowledge base and are allowed to be screened out of each second knowledge base, and the first numbers corresponding to the plurality of second knowledge bases are different; wherein, in the case that the first knowledge base includes a plurality of second knowledge bases and third knowledge bases, the value of M is determined by the following formula: /(I) ; Wherein S represents a second number of second knowledge pieces that are searched from the third knowledge base and allowed to be screened by the piece screener after being screened by the piece screener, and the large model includes the piece screener; and the second knowledge segments searched from the second knowledge base are allowed to calculate the similarity, and S second knowledge segments allowed to calculate the similarity are required to be screened out by the segment screener.

In an exemplary embodiment, before classifying, by the knowledge classifier in the large model, input information of the target object input into the large model, and obtaining a classification result, the method further includes: constructing first prompt information of the knowledge classifier based on a large model, wherein the first prompt information is used for indicating an input information label which is allowed to be classified by the knowledge classifier by the input information, and the input information label comprises at least one of the following: home class labels, general class labels, chat class labels, other class labels, wherein the large model is a deep learning model for performing natural language processing tasks.

In an exemplary embodiment, classifying, by a knowledge classifier in the large model, input information of the target object input into the large model, to obtain a classification result, includes: converting the input information into a first vector by a large model; inputting the first vector into a first target neural network model corresponding to the knowledge classifier, so as to determine a target label corresponding to the first vector in the input information label through the first target neural network model and the first prompt information; and determining the target label as the classification result.

In an exemplary embodiment, searching the first knowledge base indicated by the classification result for a first knowledge segment corresponding to the input information includes: determining a first vector into which the input information is converted; under the condition that a plurality of second knowledge bases are included in the first knowledge base, L second knowledge segments are screened out from each second knowledge base according to second vectors included in the first vectors, wherein the second vectors are vectors obtained by converting metadata information included in the input information, and the metadata information is indication information used for screening the second knowledge segments in the input information; determining semantic similarity of the L second knowledge pieces in each second knowledge base with the first vector under the condition that L is higher than a first quantity W; determining W second knowledge segments corresponding to the semantic similarities respectively, so as to determine W second knowledge segments corresponding to each second knowledge base respectively, and determining a plurality of the W second knowledge segments as the first knowledge segments, wherein the minimum similarity in the W semantic similarities is larger than the maximum similarity in the rest similarity, and the rest similarity is the similarity except the W semantic similarities in the L semantic similarities; wherein L, W is a positive integer, L, W is less than M.

In an exemplary embodiment, searching the first knowledge base indicated by the classification result for a first knowledge segment corresponding to the input information includes: determining a first vector into which the input information is converted; under the condition that the first knowledge base comprises a plurality of second knowledge bases, determining L second knowledge segments with highest semantic similarity with the first vector in each second knowledge base through an information retrieval technology; screening P second knowledge segments from the L second knowledge segments according to a second vector included in the first vector when L is higher than the first number W, wherein the second vector is a vector obtained by converting metadata information included in the input information, and the metadata information is indication information used for screening the second knowledge segments in the input information; under the condition that W is equal to P, determining W second knowledge segments corresponding to a plurality of second knowledge bases as the first knowledge segments; when W is larger than P, P second knowledge segments corresponding to the second knowledge bases and Q second knowledge segments corresponding to the second knowledge bases are determined to be the first knowledge segments, wherein the Q second knowledge segments are Q second knowledge segments with highest semantic similarity except the P second knowledge segments in the L second knowledge segments; wherein L, W, P, Q is a positive integer, L, W, P, Q is less than M.

In an exemplary embodiment, before classifying, by the knowledge classifier in the large model, input information of the target object input into the large model, and obtaining a classification result, the method further includes: constructing second prompt information of a segment screener based on a large model, wherein the segment screener is used for screening R second knowledge segments searched out from a third knowledge base under the condition that the third knowledge base is included in the first knowledge base, and the first knowledge segments comprise the R second knowledge segments; the second prompt information is used for indicating scoring standards of the segment screener on the R second knowledge segments.

In an exemplary embodiment, before determining the similarity between the M second knowledge pieces included in the first knowledge piece and the input information, the method further includes: under the condition that the first knowledge base comprises the third knowledge base, inputting the R second knowledge segments into a second target neural network model corresponding to the segment screener, so as to determine similarity scores of knowledge segment titles respectively included in the R second knowledge segments and the input information through the second target neural network model and the second prompt information; determining S second knowledge segments and T second knowledge segments as the M second knowledge segments, wherein the S second knowledge segments are S second knowledge segments with highest similarity scores in the R second knowledge segments, the T second knowledge segments are knowledge segments except the R second knowledge segments in the first knowledge segments, R, S, T are positive integers, and R is larger than S.

In an exemplary embodiment, determining the similarity of the M second knowledge pieces included in the first knowledge piece and the input information includes: determining a source knowledge base of a fourth knowledge segment in the case that a plurality of second knowledge bases and/or third knowledge bases are included in the first knowledge base, wherein the fourth knowledge segment is any knowledge segment in the M second knowledge segments, and the plurality of second knowledge bases and/or third knowledge bases include the source knowledge base; determining semantic similarity of the fourth knowledge segment and the input information, and determining preset weights of the source knowledge base; and determining the product of the semantic similarity and the preset weight as the similarity of the fourth knowledge segment.

In an exemplary embodiment, determining the similarity of the M second knowledge pieces included in the first knowledge piece and the input information includes: the similarity score is calculated by the following formula:

score = a*sim*value；

wherein a represents a preset weight of a source knowledge base, sim represents semantic similarity between a fourth knowledge segment and the input information, and value represents importance degree of the fourth knowledge segment; the fourth knowledge segment is any knowledge segment in the M second knowledge segments, the source knowledge base is a source knowledge base of the fourth knowledge segment, the first knowledge base comprises a plurality of second knowledge bases and/or third knowledge bases, and the plurality of second knowledge bases and/or third knowledge bases comprise the source knowledge base.

In one exemplary embodiment, the method includes: calculating the importance value of the fourth knowledge piece according to the following formula:

value=num（B）/len(B)+s_bm25；

Wherein num (B) is the number of keywords included in the fourth knowledge piece, len (B) is the length of the number of characters included in the fourth knowledge piece, and s_bm25 is the score obtained by calculating the fourth knowledge piece through a target algorithm.

In one exemplary embodiment, the method includes: calculating the semantic similarity sim of the fourth knowledge piece according to the following formula:

sim=similarity(A,B)*X+Y*min(1,n_B/n_A);

The similarity (a, B) is used for representing cosine similarity between the input information and the fourth knowledge segment, n_b is the same number of keywords included in the input information and the fourth knowledge segment, n_a is the number of keywords of the input information, X represents a first weight of the cosine similarity, and Y represents a second weight of min (1, n_b/n_a).

In an exemplary embodiment, determining, by the knowledge filter in the large model, K third knowledge segments of the N second knowledge segments that have a response relationship with the input information includes: splicing N second knowledge segments according to the sequence from the big similarity to the small similarity to obtain a first context; and screening K third knowledge segments with response relation with the input information from the first context through a knowledge screening device.

In one exemplary embodiment, inputting K third knowledge pieces into the large model includes: determining the extraction sequence of the K third knowledge pieces from N second knowledge pieces; and splicing the K third knowledge segments according to the extraction sequence to obtain a second context, and inputting the second context into the large model.

In one exemplary embodiment, the method includes: directly generating feedback information of the input information through a knowledge generator of a large model in case that a target condition is satisfied, wherein the target condition comprises one of the following: the first knowledge segments are not searched from the first knowledge base; determining that no second knowledge segments with response relation with the input information exist in the N second knowledge segments through the knowledge screener; the knowledge generator is arranged behind a knowledge filter of the large model or behind an input information interface of the large model, and the input information interface is used for inputting the input information.

In an exemplary embodiment, after inputting K third knowledge pieces into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge pieces and the input information, the method further includes: determining the matching degree of the feedback information and the input information through a knowledge evaluator, and grading the feedback information according to the matching degree to obtain a grading value, wherein the large model comprises the knowledge evaluator, and the knowledge evaluator is arranged behind a knowledge generator of the large model and is connected with the knowledge generator; under the condition that the grading value is higher than a preset grading value, determining the feedback information as first feedback information, and sampling and auditing the first feedback information through a target object; and under the condition that the scoring value is lower than or equal to a preset scoring value, determining the feedback information as second feedback information, and modifying the second feedback information through the target object.

According to another aspect of the embodiment of the present application, there is also provided a device for generating feedback information based on a large model, including: the classification module is used for classifying input information of a target object input into the large model through a knowledge classifier in the large model to obtain a classification result, and searching a first knowledge segment corresponding to the input information in a first knowledge base indicated by the classification result; a first determining module, configured to determine similarities between M second knowledge pieces included in the first knowledge pieces and the input information, and determine second knowledge pieces corresponding to N similarities respectively, where a minimum similarity of the N similarities is greater than a maximum similarity of other similarities, where the other similarities are similarities of the M similarities except the N similarities; the second determining module is used for determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model; the generation module is used for inputting K third knowledge segments into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge segments and the input information;

Wherein M, N and K are positive integers, M is greater than N, and N is greater than or equal to K; wherein, in the case that the first knowledge base includes a plurality of second knowledge bases, the value of M is determined by the following formula: ; wherein n represents the number of a plurality of second knowledge bases included in the first knowledge base, W is a first number of second knowledge segments which are preset for each second knowledge base and are allowed to be screened out of each second knowledge base, and the first numbers corresponding to the plurality of second knowledge bases are different; wherein, in the case that the first knowledge base includes a plurality of second knowledge bases and third knowledge bases, the value of M is determined by the following formula: /(I) ; Wherein S represents a second number of second knowledge pieces that are searched from the third knowledge base and allowed to be screened by the piece screener after being screened by the piece screener, and the large model includes the piece screener; and the second knowledge segments searched from the second knowledge base are allowed to calculate the similarity, and S second knowledge segments allowed to calculate the similarity are required to be screened out by the segment screener.

According to a further aspect of embodiments of the present application, there is also provided a computer readable storage medium having a computer program stored therein, wherein the computer program is configured to perform the above-described method of generating large model-based feedback information at run-time.

According to still another aspect of the embodiment of the present application, there is further provided an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the method for generating feedback information based on a large model through the computer program.

According to the application, the input information of the target object input into the large model is classified through the knowledge classifier in the large model to obtain a classification result, and a first knowledge segment corresponding to the input information is searched in a first knowledge base indicated by the classification result; determining the similarity between M second knowledge pieces included in the first knowledge pieces and the input information, and determining second knowledge pieces corresponding to N similarities respectively, wherein the minimum similarity of the N similarities is larger than the maximum similarity of other similarities, and the other similarities are similarities except the N similarities; determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model; inputting K third knowledge pieces into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge pieces and the input information; wherein M, N and K are positive integers, M is greater than N, and N is greater than or equal to K; wherein, in the case that the first knowledge base includes a plurality of second knowledge bases, the value of M is determined by the following formula: ; wherein n represents the number of a plurality of second knowledge bases included in the first knowledge base, W is a first number of second knowledge segments which are preset for each second knowledge base and are allowed to be screened out of each second knowledge base, and the first numbers corresponding to the plurality of second knowledge bases are different; wherein, in the case that the first knowledge base includes a plurality of second knowledge bases and third knowledge bases, the value of M is determined by the following formula: /(I) ; Wherein S represents a second number of second knowledge pieces that are searched from the third knowledge base and allowed to be screened by the piece screener after being screened by the piece screener, and the large model includes the piece screener; and the second knowledge segments searched from the second knowledge base are allowed to calculate the similarity, and S second knowledge segments allowed to calculate the similarity are required to be screened out by the segment screener. That is, by searching a plurality of second knowledge bases and/or third knowledge bases, searching out first knowledge segments, and screening out effective and high-quality K third knowledge segments by a segment screener, a knowledge screener and the like, the finally obtained feedback information can be generated by combining the K third knowledge segments and knowledge of a large model; therefore, by adopting the technical scheme, the problems that the large model cannot generate targeted feedback information for user input information belonging to different knowledge fields due to knowledge limitation, illusion problem, data security and the like in the related technology are solved; the technical effect of generating high-quality feedback information aiming at the input information of the user is achieved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

In order to more clearly illustrate the embodiments of the application or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to a person skilled in the art that other drawings can be obtained from these drawings without inventive effort.

FIG. 1 is a schematic diagram of a hardware environment of an alternative method of generating large model-based feedback information according to an embodiment of the present application;

FIG. 2 is a flow chart of an alternative method of generating large model-based feedback information in accordance with an embodiment of the present application;

FIG. 3 is an architecture diagram (one) of an alternative method of generating large model-based feedback information according to an embodiment of the present application;

FIG. 4 is a block diagram (II) of an alternative apparatus for generating feedback information based on a large model according to an embodiment of the present application;

fig. 5 is a block diagram of an alternative apparatus for generating feedback information based on a large model according to an embodiment of the present application.

Detailed Description

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description of the present application and the above-described drawings are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to an aspect of an embodiment of the present application, there is provided a method for generating feedback information based on a large model. The method for generating the feedback information based on the large model is widely applied to full-house intelligent digital control application scenes such as intelligent Home (Smart Home), intelligent Home equipment ecology, intelligent Home (IntelligenceHouse) ecology and the like. Alternatively, in the present embodiment, the above-described method of generating feedback information based on a large model may be applied to a hardware environment constituted by a plurality of terminal devices 102 and servers 104 as shown in fig. 1. As shown in fig. 1, the server 104 is connected to a plurality of terminal devices 102 through a network, and may be used to provide services (such as application services, etc.) for a terminal or a client installed on the terminal, a database may be set on the server or independent of the server, for providing data storage services for the server 104, and cloud computing and/or edge computing services may be configured on the server or independent of the server, for providing data operation services for the server 104.

The network may include, but is not limited to, at least one of: wired network, wireless network. The wired network may include, but is not limited to, at least one of: a wide area network, a metropolitan area network, a local area network, and the wireless network may include, but is not limited to, at least one of: WIFI (WIRELESS FIDELITY ), bluetooth. The terminal device 102 may not be limited to a PC, a mobile phone, a tablet computer, an intelligent air conditioner, an intelligent smoke machine, an intelligent refrigerator, an intelligent oven, an intelligent cooking range, an intelligent washing machine, an intelligent water heater, an intelligent washing device, an intelligent dish washer, an intelligent projection device, an intelligent television, an intelligent clothes hanger, an intelligent curtain, an intelligent video, an intelligent socket, an intelligent sound box, an intelligent fresh air device, an intelligent kitchen and toilet device, an intelligent bathroom device, an intelligent sweeping robot, an intelligent window cleaning robot, an intelligent mopping robot, an intelligent air purifying device, an intelligent steam box, an intelligent microwave oven, an intelligent kitchen appliance, an intelligent purifier, an intelligent water dispenser, an intelligent door lock, and the like.

In the related art, information is retrieved from a data source by a retrieval enhancement generation (RETRIEVAL AUGMENTED GENERATION, abbreviated as RAG) technique to assist a large language model (Large Language Model, abbreviated as LLM) in generating answers.

However, since the initial RAG component is simple, there is a high probability that there is no way to achieve a good effect immediately in the actual application function. The following problems are likely to occur with large models aided by RAG: 1) Search quality problems: the results of the query do not necessarily fit into a given input message (query); 2) Recovery quality problem: in general, the method can be said to be a hallucination problem, but is not limited to the hallucination problem, and although RAG can greatly restrict random generation of a large model, the constraint cannot be absolutely guaranteed, and the large model can be subjected to conditions of braiding, against reality, refusal, no reference to a search result and the like; 3) Enhancement process problem: that is, the problem of the synergy of the retrieval result and the reply of the large model, the failure of the block may be caused by an improper document or an insufficiently strong model, and the large model may be misled by the retrieval result or affected by repeated information, so that the feedback information of disordering, repeated and non-questionable answer is generated. In addition, if the generation of the large model excessively depends on the search result, the reply content (i.e., feedback information) may be limited.

Accordingly, in this embodiment, there is provided a method for generating feedback information based on a large model, including but not limited to being applied to a computer terminal, and fig. 2 is a flowchart of an alternative method for generating feedback information based on a large model according to an embodiment of the present application, where the flowchart includes the following steps:

step S202: classifying input information input into the large model by a knowledge classifier in the large model to obtain a classification result, and searching a first knowledge segment corresponding to the input information in a first knowledge base indicated by the classification result;

that is, the first knowledge segments are searched by classifying the input information into the corresponding first knowledge base by the knowledge classifier.

It should be noted that, the large model LLM adopted in the embodiment of the present application is a language model based on deep learning technology, and can process and generate natural language text. Large models typically use large-scale neural network structures, with billions or even billions of parameters, capable of learning and understanding more complex language structures and semantic relationships. The large model has great breakthrough in the fields of natural language processing, machine translation, text generation and the like, and is widely applied to language understanding and generating tasks

Step S204: determining the similarity between M second knowledge pieces included in the first knowledge pieces and the input information, and determining second knowledge pieces corresponding to N similarities respectively, wherein the minimum similarity of the N similarities is larger than the maximum similarity of other similarities, and the other similarities are similarities except the N similarities;

Namely, the similarity between the M second knowledge segments and the input information is calculated, and N second knowledge segments with the highest similarity between the M second knowledge segments and the input information are determined.

Step S206: determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model;

Step S208: inputting K third knowledge pieces into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge pieces and the input information; wherein M, N and K are positive integers, M is greater than N, and N is greater than or equal to K;

optionally, the value of M is different according to whether the first knowledge base includes the third knowledge base.

In the case that the first knowledge base includes a plurality of second knowledge bases, the value of M is determined by the following formula: ; wherein n represents the number of a plurality of second knowledge bases included in the first knowledge base, W is a first number of second knowledge segments which are preset for each second knowledge base and are allowed to be screened out of each second knowledge base, and the first numbers corresponding to the plurality of second knowledge bases are different; i.e./> A first number of second knowledge pieces preset for an ith second knowledge base and allowed to be screened from the ith second knowledge base

For example, the first knowledge base pointed to by the classification result includes 3 second knowledge bases, and the first numbers corresponding to the 3 second knowledge bases are 5, 5 and 7 respectively, so that m=5+5+7=17.

In the case that the first knowledge base includes a plurality of second knowledge bases and third knowledge bases, the value of M is determined by the following formula: ; wherein S represents a second number of second knowledge pieces that are searched from the third knowledge base and allowed to be screened by the piece screener after being screened by the piece screener, and the large model includes the piece screener; and the second knowledge segments searched from the second knowledge base are allowed to calculate the similarity, and S second knowledge segments allowed to calculate the similarity are required to be screened out by the segment screener.

For example, the first knowledge base to which the classification result points includes 3 second knowledge bases, the first numbers corresponding to the 3 second knowledge bases are 5, and 7, respectively, and further includes a third knowledge base, and the second number s=6 corresponding to the third knowledge base, where m=5+5+7+6=23.

Optionally, the number of third knowledge bases included in the first knowledge base may be plural, thenM represents the number of the plurality of third knowledge bases included in the first knowledge base; /(I)Representing a second number of second knowledge segments from an ith third knowledge base of the m third knowledge bases, which are allowed to be screened by the segment screener after being screened by the segment screener.

Through the steps, the input information of the target object input into the large model is classified through the knowledge classifier in the large model, a classification result is obtained, and a first knowledge segment corresponding to the input information is searched in a first knowledge base indicated by the classification result; determining the similarity between M second knowledge pieces included in the first knowledge pieces and the input information, and determining second knowledge pieces corresponding to N similarities respectively, wherein the minimum similarity of the N similarities is larger than the maximum similarity of other similarities, and the other similarities are similarities except the N similarities; determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model; inputting K third knowledge pieces into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge pieces and the input information; wherein M, N and K are positive integers, M is greater than N, and N is greater than or equal to K; wherein, in the case that the first knowledge base includes a plurality of second knowledge bases, the value of M is determined by the following formula: ; wherein n represents the number of a plurality of second knowledge bases included in the first knowledge base, W is a first number of second knowledge segments which are preset for each second knowledge base and are allowed to be screened out of each second knowledge base, and the first numbers corresponding to the plurality of second knowledge bases are different; wherein, in the case that the first knowledge base comprises a plurality of second knowledge bases and third knowledge bases, the value/>, of the M is determined by the following formula ; Wherein S represents a second number of second knowledge pieces that are searched from the third knowledge base and allowed to be screened by the piece screener after being screened by the piece screener, and the large model includes the piece screener; and the second knowledge segments searched from the second knowledge base are allowed to calculate the similarity, and S second knowledge segments allowed to calculate the similarity are required to be screened out by the segment screener. That is, by searching a plurality of second knowledge bases and/or third knowledge bases, searching out first knowledge segments, and screening out effective and high-quality K third knowledge segments by a segment screener, a knowledge screener and the like, the finally obtained feedback information can be generated by combining the K third knowledge segments and knowledge of a large model; therefore, by adopting the technical scheme, the problems that the large model cannot generate targeted feedback information for user input information belonging to different knowledge fields due to knowledge limitation, illusion problem, data security and the like in the related technology are solved; the technical effect of generating high-quality feedback information aiming at the input information of the user is achieved.

As shown in fig. 3 and fig. 4, the large model of the embodiment of the present application adopts the advanced search enhancement generation RAG technology, and the main architecture includes: a knowledge classifier 31 (equivalent to the knowledge class classifier in fig. 3), a knowledge enhancement retriever 32, a knowledge filter 33, a knowledge generator 34, a knowledge evaluator 35, and a knowledge audit store 36. As shown in fig. 3, a user (corresponding to a target object) query (corresponding to input information of the user) inputs a knowledge classifier, and the knowledge classifier is classified into knowledge bases of different fields to search household appliance knowledge segments, encyclopedia knowledge segments, chat knowledge segments, and other knowledge segments (the knowledge segments of household appliances, encyclopedia, and the like are corresponding to the first knowledge segments); the searched first knowledge segments are input into a knowledge enhancement retriever, and household appliance knowledge segments, general knowledge segments, boring knowledge segments, other knowledge segments and the like (equivalent to N second knowledge segments) are obtained through enhancement retrieval; the obtained household appliance knowledge segments, general knowledge segments, boring knowledge segments, other knowledge segments and the like are continuously input into the knowledge filter 33 in fig. 4 for filtering to obtain effective knowledge segments (equivalent to K third knowledge segments), and are input into the knowledge generator for generating feedback information.

Based on the architecture described in fig. 3 and 4, before performing the above step S202, the method further includes: constructing first prompt information of the knowledge classifier based on a large model, wherein the first prompt information is used for indicating an input information label which is allowed to be classified by the knowledge classifier by the input information, and the input information label comprises at least one of the following: home class labels, general class labels, chat class labels, other class labels, wherein the large model is a deep learning model for performing natural language processing tasks.

Optionally, the first hint information of the knowledge classifier of the large model should have been constructed before allowing the input information to be entered. Therefore, the input information of the target object input into the large model is classified by the knowledge classifier in the large model, and a classification result is obtained, which comprises the following steps: converting the input information into a first vector by a large model; inputting the first vector into a first target neural network model corresponding to the knowledge classifier, so as to determine a target label corresponding to the first vector in the input information label through the first target neural network model and the first prompt information; and determining the target label as the classification result.

It can be appreciated that based on the large model LLM, a prompt (corresponding to the first prompt information) of the knowledge base classification is constructed, then the user query is transmitted to the prompt, and the category (corresponding to the target tag) of the query is obtained through the large model.

Illustratively, the Prompt example may be: is a user question classifier, please classify the user question query into labels (corresponding to the input information labels): household appliance related questions (corresponding to household category labels), conventional questions (corresponding to general category labels), chat questions (corresponding to chat category labels), and others (corresponding to other category labels).

So that according to the user problem query: { query }, get the category. For example: 1) If the user question query is: how does a refrigerator store vegetables? The corresponding category is: problems associated with home appliances; 2) If the user question query is: what is octman? The corresponding category: conventional problems.

Further, the input information of the user passes through the knowledge classifier, and it can be determined which problem or problems of household appliances, conventional problems, chat and chat problems, other problems, or which problem the input information belongs to. The four problem categories correspond to the home appliance query, the general query, the boring query and other queries in fig. 3, so that the input information corresponding to the home appliance query can be classified into the home appliance domain knowledge base, the input information corresponding to the general query can be classified into the encyclopedia knowledge base, and so on. It should be noted that, the home appliance domain knowledge base, encyclopedia knowledge base, chat knowledge base, other knowledge bases, and the like in fig. 3 are knowledge segment vectorization knowledge bases for storing knowledge segment vector data.

Optionally, after determining the classification result of the input information, the knowledge enhancement retriever in fig. 3 may search for the first knowledge segment corresponding to the input information. Knowledge pieces related to the user query are recalled, for example, based on semantics (keywords) or some metadata fields.

It will be appreciated that prior to searching for the knowledge segments corresponding to the input information, the input information needs to be converted into a vector form representation, i.e. a first vector. In the embodiment of the application, the query is processed into the semantic vector (namely the first vector) by a knowledge base based on a large model vectorization technology, such as openai text-embedding-ada-002 model, so that the subsequent calculation of the semantic similarity is facilitated.

In this embodiment, during searching the knowledge segments corresponding to the input information from each second knowledge base, L second knowledge segments may be searched by first using the second vectors included in the first vectors, if L is greater than the first number W of knowledge segments allowed to be searched by each second knowledge base, and then, by calculating semantic similarity between the L second knowledge segments and the input information, the W second knowledge segments are screened out from the L second knowledge segments. Finally, the input information may be classified into a plurality of second knowledge bases at the same time to search for knowledge segments, and then the plurality of knowledge bases respectively correspond to W second knowledge segments (i.e., a plurality of the W second knowledge segments), that is, the first knowledge segments searched out from the first knowledge base.

Optionally, when l=w, determining a plurality of the L second knowledge pieces as the first knowledge pieces; optionally, if L is smaller than W, determining a plurality of L second knowledge pieces and a plurality of H second knowledge pieces as the first knowledge pieces, where the H second knowledge pieces are second knowledge pieces that are screened from each second knowledge base and have the highest semantic similarity and do not overlap with the L second knowledge pieces, where w=l+h.

In the above embodiment, when L is smaller than W, w=p+q.

It can be understood that in this embodiment, in the process of searching the knowledge segments corresponding to the input information from each second knowledge base, first, L second knowledge segments with the highest semantic similarity with the input information in each second knowledge base may be determined;

second, if L is greater than the first number W of knowledge segments that each second knowledge base is allowed to search, P second knowledge segments in the L second knowledge segments are screened out through the second vector. If W is equal to P, determining W second knowledge segments corresponding to the second knowledge bases as the first knowledge segments, and if W is greater than P, determining P second knowledge segments corresponding to the second knowledge bases and Q second knowledge segments corresponding to the second knowledge bases as the first knowledge segments; if W is smaller than P, screening W second index fragments with highest semantic similarity from the P second knowledge fragments, and determining W second knowledge fragments corresponding to the second knowledge bases as first knowledge fragments.

Optionally, if l=w, a plurality of L second knowledge pieces are determined as the first knowledge pieces. If L is smaller than W, determining a plurality of L second knowledge pieces and a plurality of J second knowledge pieces as first knowledge pieces; the J second knowledge segments are second knowledge segments which are searched from each second knowledge base through the second vector and are not overlapped with the L second knowledge segments, and at this time, w=l+j.

It should be noted that, the process of screening R second knowledge segments (knowledge segments not yet screened by the segment screener) from each third knowledge base is similar to the process of screening W second knowledge segments from each second knowledge base, and this embodiment is not repeated here.

Optionally, a q-q (question-question) filter (corresponding to a segment filter) is further included in the knowledge enhancement retriever in fig. 3, for further filtering knowledge segments recalled from the encyclopedia knowledge base or the like (corresponding to the third knowledge base). Wherein the encyclopedia knowledge base and the like are knowledge bases with poor knowledge recall effect by searching knowledge sources through an open source search engine (included in a knowledge enhancement retriever).

Specifically, the method classifies input information of the target object input into the large model through a knowledge classifier in the large model, and before the classification result is obtained, the method further comprises: constructing second prompt information of a segment screener based on a large model, wherein the segment screener is used for screening R second knowledge segments searched out from a third knowledge base under the condition that the third knowledge base is included in the first knowledge base, and the first knowledge segments comprise the R second knowledge segments; the second prompt information is used for indicating scoring standards of the segment screener on the R second knowledge segments.

That is, before the input information is input to the knowledge classifier of the large model, a second hint information of the segment filter is also required to be constructed, where the second hint information is used to instruct the segment filter how to calculate a scoring criterion for the similarity score for the second knowledge segment.

Further, before determining the similarity between the M second knowledge pieces included in the first knowledge piece and the input information, the method further includes: under the condition that the first knowledge base comprises the third knowledge base, inputting the R second knowledge segments into a second target neural network model corresponding to the segment screener, so as to determine similarity scores of knowledge segment titles respectively included in the R second knowledge segments and the input information through the second target neural network model and the second prompt information; determining S second knowledge segments and T second knowledge segments as the M second knowledge segments, wherein the S second knowledge segments are S second knowledge segments with highest similarity scores in the R second knowledge segments, the T second knowledge segments are knowledge segments except the R second knowledge segments in the first knowledge segments, R, S, T are positive integers, and R is larger than S.

It is understood that the segment filter is configured to calculate similarity scores of knowledge segment titles and input information included in R second knowledge segments screened from the third knowledge base, respectively. For example, 10 (corresponding to R) second knowledge pieces may be searched out from the third knowledge base, and 5 (corresponding to S) second knowledge pieces are allowed to be finally screened out from the third knowledge base, and similarity scores of knowledge piece titles of the 10 second knowledge pieces and the input information are respectively determined by a piece screener, so that 5 second knowledge pieces with highest similarity scores are screened out.

In other words, the open source search engine will return the title of the relevant knowledge piece and the answer context to which the title corresponds. And q-q screening is carried out on the knowledge fragment titles returned by the user query and the search engine, so that irrelevant results can be filtered efficiently. The quality of knowledge of the source of the search engine is ensured. The q-q filter scores the query and recall headers (corresponding to the knowledge segment headers) by relevance (0-10 points) through the prompt+LLM (namely the second prompt message and the large model, wherein the prompt corresponds to the second prompt message), and the candidates with the scores greater than 7 points are screened.

It should be noted that, in the embodiment of the present application, the first target neural network model and the second target neural network model may be a transducer model, or may be a cyclic neural network (Recurrent Neural Network, abbreviated as RNN) model, and are selected according to the actual needs of the large model.

The kernel of the transducer model, which is a neural network model based on an attention mechanism and is used for processing natural language processing tasks, such as machine translation, language modeling and text generation, is a self-attention mechanism, so that the model can simultaneously consider information of all positions in an input sequence, and does not gradually process the input sequence like a traditional cyclic neural network or a convolutional neural network. This enables the transducer model to better capture long range dependencies in the input sequence and to have better training and reasoning efficiency when processing large-scale data sets.

Wherein RNN is a neural network model with cyclic connections. The advantage in processing sequence data is that previous information can be utilized to assist in understanding subsequent data. The RNN classification model is a model for classifying an input sequence using RNNs, and is generally used for processing data having sequence properties such as text and time series. By learning patterns and rules in the sequence data, the RNN classification model can effectively classify the input data.

For the step S204, determining the similarity between the M second knowledge pieces included in the first knowledge piece and the input information includes: determining a source knowledge base of a fourth knowledge segment in the case that a plurality of second knowledge bases and/or third knowledge bases are included in the first knowledge base, wherein the fourth knowledge segment is any knowledge segment in the M second knowledge segments, and the plurality of second knowledge bases and/or third knowledge bases include the source knowledge base; determining semantic similarity of the fourth knowledge segment and the input information, and determining preset weights of the source knowledge base; and determining the product of the semantic similarity and the preset weight as the similarity of the fourth knowledge segment.

The product obtained by multiplying the preset weight of the source knowledge base of the fourth knowledge segment by the semantic similarity of the fourth knowledge segment can be determined as the similarity of the fourth knowledge segment. After the similarity is determined, the M second knowledge segments are ranked according to the similarity from high to low, and N second knowledge segments with highest similarity are selected from the M second knowledge segments.

For example, as shown in fig. 3, the input information is a home appliance query, and is classified into a home appliance domain knowledge base and an encyclopedia knowledge base, and then top5 and top10 are recalled from the home appliance domain knowledge base and the encyclopedia knowledge base respectively (top 5 or top10 can be obtained by sorting according to the semantic similarity sim of the knowledge segments and the query), and top5 is sorted according to the comprehensive score (equivalent to similarity) by q-q screening of the rest top5 (for example).

Optionally, determining the similarity between the M second knowledge pieces included in the first knowledge piece and the input information includes: the similarity score is calculated by the following formula:

score = a*sim*value；

That is, in the process of calculating the similarity, the semantic similarity of the fourth knowledge piece and the preset weight of the source knowledge base of the fourth knowledge piece may be considered, and the importance degree of the fourth knowledge piece may also be considered.

Further, the importance value of the fourth knowledge piece can be calculated by the following formula:

value=num（B）/len(B)+s_bm25；

Wherein num (B) is the number of keywords included in the fourth knowledge piece, len (B) is the length of the number of characters included in the fourth knowledge piece, and s_bm25 is the score obtained by calculating the fourth knowledge piece through a target algorithm. The user query is denoted as A, and the fourth knowledge segment is denoted as B.

Alternatively, the target algorithm may be tf-idf or bm25 algorithm, and the fourth knowledge piece is input into the target algorithm, so that the importance score of the fourth knowledge piece can be determined. Alternatively, the importance degree may be made equal to only the score of importance, or the importance degree may be made equal to only the ratio of the number of keywords included in the fourth knowledge piece to the length of the number of characters included in the fourth knowledge piece.

Further, cosine similarity between the fourth knowledge segment and the input information can be directly used as semantic similarity; however, considering the frequency of domain keywords (corresponding to keywords) involved in the knowledge segments comprehensively, the semantic similarity sim of the fourth knowledge segment can also be calculated by the following formula:

sim=similarity(A,B)*X+Y*min(1,n_B/n_A)；

The similarity (a, B) is used for representing cosine similarity between the input information and the fourth knowledge segment, n_b is the same number of keywords included in the input information and the fourth knowledge segment, n_a is the number of keywords of the input information, X represents a first weight of the cosine similarity, and Y represents a second weight of min (1, n_b/n_a). Wherein, X can take the value of 0.7, and Y can take the value of 0.3.

The cosine similarity formula is as follows:

；

wherein A represents the input information, can specifically represent a first vector corresponding to the input information, B represents the fourth knowledge segment, can specifically represent a third vector corresponding to the fourth knowledge segment, An i-th component vector representing the first vector,/>An i-th component vector representing the third vector.

For the step S206, determining, by a knowledge filter in the large model, K third knowledge segments having a response relationship with the input information from the N second knowledge segments, where the determining includes: splicing N second knowledge segments according to the sequence from the big similarity to the small similarity to obtain a first context; and screening K third knowledge segments with response relation with the input information from the first context through a knowledge screening device.

For the N second knowledge segments, determining effective information in the N second knowledge segments, namely a third knowledge segment with a response relation with the input information, through a knowledge filter. It will be appreciated that the screening process of the knowledge screener is a process of screening a third knowledge segment from the second knowledge segment. Before the screening operation of the knowledge screener is performed, the N second knowledge segments need to be spliced in order of similarity from large to small. It should be noted that, N pieces of the second knowledge are arranged in order of similarity from large to small before being input to the knowledge filter.

The K extracted third knowledge segments can be used as third prompt information of the large model to be input into the large model after being spliced according to the extraction sequence, and the third prompt information comprises the second context.

For example, under the condition of k=5, the knowledge filter is used to determine whether the recalled top5 second knowledge segment can respond to the input information, and noise and irrelevant data are filtered, so that the accuracy of LLM generation is improved. Screening can be performed by LLM or ragas, an open source tool to evaluate the effect of the generation. The specific screening steps are as follows: the second knowledge segments of top5 are spliced in sequence to obtain a complete context (corresponding to the first context); the local sentences related to the question (corresponding to the third knowledge segments) are then screened out based on the prompt+llm (prompt can be understood herein as the fourth hint information of the knowledge screener). Finally, the screened effective knowledge sentences (corresponding to the third knowledge segments) are spliced to form knowledge contexts (corresponding to the second contexts) generated by the knowledge.

For example: the tomato can be placed in the query=' refrigerator;

The first context= '' your will be very happy, the tomatoes can not be frozen in the refrigerator, 1, the tomatoes are frozen at low temperature, the meat is in a bubble shape, the meat appears soft or spallation phenomenon occurs, black spots are on the surface, the tomatoes are not cooked, the tomatoes have no delicate flavor, and the tomatoes are rancid and rotten seriously. 2. Tomatoes are not suitable for low temperature storage.

After the food is placed in the refrigerator for about 2-3 hours, the refrigerator is fully cooled, and the food can be placed at the moment.

When the temperature setting is changed, the temperature in the box can reach equilibrium after a period of time. And the length of the period of time depends on the size of the temperature setting change, the ambient temperature and the like.

Tomatoes are not recommended for a refrigerator. '''

The result_pre= {' in the second context is good, so that the tomato can not be frozen in a refrigerator, 1, the tomato is frozen at low temperature, the meat is in a bubble shape, appears soft and rotten, or the spallation phenomenon appears, the surface has black spots, the tomato is not cooked, the tomato has no delicate flavor, and the tomato is seriously rancid and rotten. 2. Tomatoes are not suitable for low temperature storage. 'can respond to the question', 'tomato does not suggest to put a refrigerator to the door', 'can respond to the question'. }

It will be appreciated that the feedback information is generated by a knowledge generator. Sequentially splicing knowledge piece sentences (corresponding to third knowledge pieces) which are finally recalled and screened as knowledge contexts (corresponding to second contexts), constructing a prompt (corresponding to third prompt information), and enabling LLM to generate answers (corresponding to feedback information) based on the knowledge contexts; if no relevant knowledge is retrieved from the knowledge base, the LLM is allowed to generate an answer directly from the user query (answer the question in combination with knowledge possessed by itself).

Alternatively, feedback information generated by the knowledge generator may be input into the knowledge evaluator for scoring. As shown in FIG. 4, answers generated by LLM are scored (0-10 points) for q-a (question-answer) matching once again by LLM capability, and high quality answers are screened out. Different audit logic is walked through for different quality answers.

Further, obtaining average grading values of the knowledge evaluator on a plurality of pieces of feedback information in a preset time period; adjusting parameters of the large model through the average grading value, wherein the parameters comprise at least one of the following: the knowledge base weight corresponding to the plurality of fourth knowledge bases in the large model respectively, and the number (first number, second number, etc.) of second knowledge segments which are set for each fourth knowledge base and allowed to be screened from each second knowledge base.

It will be appreciated that the accuracy of the high scoring answers is higher than the accuracy of the low scoring answers. The LLM model shown in fig. 3 and fig. 4 involves some parameters, the weights and top numbers can be adjusted through manual verification in the early stage, and the parameters can be self-learned and self-adjusted by the model according to the average score of the evaluator in the later stage. The goal of the adjustment is to ensure that the average score is highest, and the model can learn the parameters through means such as grid search (GRIDSEARCH) or reinforcement learning.

From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method of the various embodiments of the present application.

The embodiment also provides a device for generating feedback information based on a large model, which is used for implementing the foregoing embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

Fig. 5 is a block diagram of an alternative apparatus for generating feedback information based on a large model according to an embodiment of the present application, the apparatus including:

the classification module 52 is configured to classify input information input into the large model by using a knowledge classifier in the large model, obtain a classification result, and search a first knowledge base indicated by the classification result for a first knowledge segment corresponding to the input information;

A first determining module 54, configured to determine the similarity between M second knowledge pieces included in the first knowledge pieces and the input information, and determine second knowledge pieces corresponding to N similarities respectively, where a minimum similarity of the N similarities is greater than a maximum similarity of other similarities, where the other similarities are similarities of the M similarities except the N similarities;

A second determining module 56, configured to determine, by using a knowledge filter in the large model, K third knowledge segments having a response relationship with the input information from the N second knowledge segments;

A generating module 58 for inputting K third knowledge pieces into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge pieces and the input information; wherein M, N and K are positive integers, M is greater than N, and N is greater than or equal to K;

wherein, in the case that the first knowledge base includes a plurality of second knowledge bases, the value of M is determined by the following formula:

; wherein n represents the number of a plurality of second knowledge bases included in the first knowledge base, W is a first number of second knowledge segments which are preset for each second knowledge base and are allowed to be screened out of each second knowledge base, and the first numbers corresponding to the plurality of second knowledge bases are different;

wherein, in the case that the first knowledge base includes a plurality of second knowledge bases and third knowledge bases, the value of M is determined by the following formula:

; wherein S represents a second number of second knowledge pieces that are searched from the third knowledge base and allowed to be screened by the piece screener after being screened by the piece screener, and the large model includes the piece screener;

and the second knowledge segments searched from the second knowledge base are allowed to calculate the similarity, and S second knowledge segments allowed to calculate the similarity are required to be screened out by the segment screener.

Through the device, the input information of the target object input into the large model is classified through the knowledge classifier in the large model, a classification result is obtained, and a first knowledge segment corresponding to the input information is searched in a first knowledge base indicated by the classification result; determining the similarity between M second knowledge pieces included in the first knowledge pieces and the input information, and determining second knowledge pieces corresponding to N similarities respectively, wherein the minimum similarity of the N similarities is larger than the maximum similarity of other similarities, and the other similarities are similarities except the N similarities; determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model; inputting K third knowledge pieces into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge pieces and the input information; wherein M, N and K are positive integers, M is greater than N, and N is greater than or equal to K; wherein, in the case that the first knowledge base includes a plurality of second knowledge bases, the value of M is determined by the following formula: ; wherein n represents the number of a plurality of second knowledge bases included in the first knowledge base, W is a first number of second knowledge segments which are preset for each second knowledge base and are allowed to be screened out of each second knowledge base, and the first numbers corresponding to the plurality of second knowledge bases are different; wherein, in the case that the first knowledge base includes a plurality of second knowledge bases and third knowledge bases, the value of M is determined by the following formula: /(I) ; Wherein S represents a second number of second knowledge pieces that are searched from the third knowledge base and allowed to be screened by the piece screener after being screened by the piece screener, and the large model includes the piece screener; and the second knowledge segments searched from the second knowledge base are allowed to calculate the similarity, and S second knowledge segments allowed to calculate the similarity are required to be screened out by the segment screener. That is, by searching a plurality of second knowledge bases and/or third knowledge bases, searching out first knowledge segments, and screening out effective and high-quality K third knowledge segments by a segment screener, a knowledge screener and the like, the finally obtained feedback information can be generated by combining the K third knowledge segments and knowledge of a large model; therefore, by adopting the technical scheme, the problems that the large model cannot generate targeted feedback information for user input information belonging to different knowledge fields due to knowledge limitation, illusion problem, data security and the like in the related technology are solved; the technical effect of generating high-quality feedback information aiming at the input information of the user is achieved.

In an exemplary embodiment, the apparatus further comprises a first building block for: constructing first prompt information of the knowledge classifier based on a large model, wherein the first prompt information is used for indicating an input information label which is allowed to be classified by the knowledge classifier by the input information, and the input information label comprises at least one of the following: home class labels, general class labels, chat class labels, other class labels, wherein the large model is a deep learning model for performing natural language processing tasks.

In one exemplary embodiment, classification module 52 is further configured to: converting the input information into a first vector by a large model; inputting the first vector into a first target neural network model corresponding to the knowledge classifier, so as to determine a target label corresponding to the first vector in the input information label through the first target neural network model and the first prompt information; and determining the target label as the classification result.

In an exemplary embodiment, the apparatus further comprises a search module for: determining a first vector into which the input information is converted; under the condition that a plurality of second knowledge bases are included in the first knowledge base, L second knowledge segments are screened out from each second knowledge base according to second vectors included in the first vectors, wherein the second vectors are vectors obtained by converting metadata information included in the input information, and the metadata information is indication information used for screening the second knowledge segments in the input information; determining semantic similarity of the L second knowledge pieces in each second knowledge base with the first vector under the condition that L is higher than a first quantity W; determining W second knowledge segments corresponding to the semantic similarities respectively, so as to determine W second knowledge segments corresponding to each second knowledge base respectively, and determining a plurality of the W second knowledge segments as the first knowledge segments, wherein the minimum similarity in the W semantic similarities is larger than the maximum similarity in the rest similarity, and the rest similarity is the similarity except the W semantic similarities in the L semantic similarities; wherein L, W is a positive integer, L, W is less than M.

In an exemplary embodiment, the apparatus further comprises a search module for: determining a first vector into which the input information is converted; under the condition that the first knowledge base comprises a plurality of second knowledge bases, determining L second knowledge segments with highest semantic similarity with the first vector in each second knowledge base through an information retrieval technology; screening P second knowledge segments from the L second knowledge segments according to a second vector included in the first vector when L is higher than the first number W, wherein the second vector is a vector obtained by converting metadata information included in the input information, and the metadata information is indication information used for screening the second knowledge segments in the input information; under the condition that W is equal to P, determining W second knowledge segments corresponding to a plurality of second knowledge bases as the first knowledge segments; when W is larger than P, P second knowledge segments corresponding to the second knowledge bases and Q second knowledge segments corresponding to the second knowledge bases are determined to be the first knowledge segments, wherein the Q second knowledge segments are Q second knowledge segments with highest semantic similarity except the P second knowledge segments in the L second knowledge segments; wherein L, W, P, Q is a positive integer, L, W, P, Q is less than M.

In an exemplary embodiment, the apparatus further comprises a second building block for: constructing second prompt information of a segment screener based on a large model, wherein the segment screener is used for screening R second knowledge segments searched out from a third knowledge base under the condition that the third knowledge base is included in the first knowledge base, and the first knowledge segments comprise the R second knowledge segments; the second prompt information is used for indicating scoring standards of the segment screener on the R second knowledge segments.

In an exemplary embodiment, the apparatus further comprises a third determining module for: under the condition that the first knowledge base comprises the third knowledge base, inputting the R second knowledge segments into a second target neural network model corresponding to the segment screener, so as to determine similarity scores of knowledge segment titles respectively included in the R second knowledge segments and the input information through the second target neural network model and the second prompt information; determining S second knowledge segments and T second knowledge segments as the M second knowledge segments, wherein the S second knowledge segments are S second knowledge segments with highest similarity scores in the R second knowledge segments, the T second knowledge segments are knowledge segments except the R second knowledge segments in the first knowledge segments, R, S, T are positive integers, and R is larger than S.

In an exemplary embodiment, the first determining module 54 is further configured to: determining a source knowledge base of a fourth knowledge segment in the case that a plurality of second knowledge bases and/or third knowledge bases are included in the first knowledge base, wherein the fourth knowledge segment is any knowledge segment in the M second knowledge segments, and the plurality of second knowledge bases and/or third knowledge bases include the source knowledge base; determining semantic similarity of the fourth knowledge segment and the input information, and determining preset weights of the source knowledge base; and determining the product of the semantic similarity and the preset weight as the similarity of the fourth knowledge segment.

In an exemplary embodiment, the first determining module 54 is further configured to: the similarity score is calculated by the following formula:

score = a*sim*value；

In an exemplary embodiment, the first determining module 54 is further configured to: calculating the importance value of the fourth knowledge piece according to the following formula:

value=num（B）/len(B)+s_bm25；

In an exemplary embodiment, the first determining module 54 is further configured to: calculating the semantic similarity sim of the fourth knowledge piece according to the following formula:

sim=similarity(A,B)*X+Y*min(1,n_B/n_A);

In one exemplary embodiment, the second determination module 56 is further configured to: splicing N second knowledge segments according to the sequence from the big similarity to the small similarity to obtain a first context; and screening K third knowledge segments with response relation with the input information from the first context through a knowledge screening device.

In one exemplary embodiment, the generation module 58 is further configured to: determining the extraction sequence of the K third knowledge pieces from N second knowledge pieces; and splicing the K third knowledge segments according to the extraction sequence to obtain a second context, and inputting the second context into the large model.

In one exemplary embodiment, the generation module 58 is further configured to: directly generating feedback information of the input information through a knowledge generator of a large model in case that a target condition is satisfied, wherein the target condition comprises one of the following: the first knowledge segments are not searched from the first knowledge base; determining that no second knowledge segments with response relation with the input information exist in the N second knowledge segments through the knowledge screener; the knowledge generator is arranged behind a knowledge filter of the large model or behind an input information interface of the large model, and the input information interface is used for inputting the input information.

In an exemplary embodiment, the apparatus further comprises a scoring module for: determining the matching degree of the feedback information and the input information through a knowledge evaluator, and grading the feedback information according to the matching degree to obtain a grading value, wherein the large model comprises the knowledge evaluator, and the knowledge evaluator is arranged behind a knowledge generator of the large model and is connected with the knowledge generator; under the condition that the grading value is higher than a preset grading value, determining the feedback information as first feedback information, and sampling and auditing the first feedback information through a target object; and under the condition that the scoring value is lower than or equal to a preset scoring value, determining the feedback information as second feedback information, and modifying the second feedback information through the target object.

Embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.

Alternatively, in the present embodiment, the above-described storage medium may be configured to store a computer program for performing the steps of:

S1, classifying input information of a target object input into a large model through a knowledge classifier in the large model to obtain a classification result, and searching a first knowledge segment corresponding to the input information in a first knowledge base indicated by the classification result;

S2, determining the similarity between M second knowledge pieces included in the first knowledge pieces and the input information, and determining second knowledge pieces corresponding to N similarities respectively, wherein the minimum similarity of the N similarities is larger than the maximum similarity of other similarities, and the other similarities are similarities except the N similarities;

S3, determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model;

S4, inputting the K third knowledge segments into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge segments and the input information.

In one exemplary embodiment, the computer readable storage medium may include, but is not limited to: a usb disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing a computer program.

Specific examples in this embodiment may refer to the examples described in the foregoing embodiments and the exemplary implementation, and this embodiment is not described herein.

An embodiment of the application also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:

In an exemplary embodiment, the electronic apparatus may further include a transmission device connected to the processor, and an input/output device connected to the processor.

It will be appreciated by those skilled in the art that the modules or steps of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application, which are intended to be comprehended within the scope of the present application.

Claims

1. The method for generating the feedback information based on the large model is characterized by comprising the following steps of:

Classifying input information input into the large model by a knowledge classifier in the large model to obtain a classification result, and searching a first knowledge segment corresponding to the input information in a first knowledge base indicated by the classification result;

Determining the similarity between M second knowledge pieces included in the first knowledge pieces and the input information, and determining second knowledge pieces corresponding to N similarities respectively, wherein the minimum similarity of the N similarities is larger than the maximum similarity of other similarities, and the other similarities are similarities except the N similarities;

Determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model;

Inputting K third knowledge pieces into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge pieces and the input information; wherein,

M, N and K are positive integers, M is greater than N, and N is greater than or equal to K;

Wherein the second knowledge segments searched from the second knowledge base all allow the similarity to be calculated, the second knowledge segments searched from the third knowledge base need to be screened out S second knowledge segments allowing the similarity to be calculated by the segment screener, wherein,

Determining the similarity between M second knowledge pieces included in the first knowledge pieces and the input information comprises the following steps:

the similarity score is calculated by the following formula:

score = a*sim*value；

wherein a represents a preset weight of a source knowledge base, sim represents semantic similarity between a fourth knowledge segment and the input information, and value represents importance degree of the fourth knowledge segment;

Wherein the fourth knowledge segment is any knowledge segment of the M second knowledge segments, the source knowledge base is a source knowledge base of the fourth knowledge segment, the first knowledge base comprises a plurality of second knowledge bases and/or third knowledge bases, the plurality of second knowledge bases and/or third knowledge bases comprises the source knowledge base, wherein,

Comprising the following steps:

Calculating the semantic similarity sim of the fourth knowledge piece according to the following formula:

sim=similarity(A,B)*X+Y*min(1,n_B/n_A);

2. The method for generating feedback information based on a large model according to claim 1, wherein before classifying input information of a target object input into the large model by a knowledge classifier in the large model, the method further comprises:

Constructing first prompt information of the knowledge classifier based on a large model, wherein the first prompt information is used for indicating an input information label which is allowed to be classified by the knowledge classifier by the input information, and the input information label comprises at least one of the following: home class labels, general class labels, chat class labels, other class labels, wherein the large model is a deep learning model for performing natural language processing tasks.

3. The method for generating feedback information based on a large model according to claim 2, wherein classifying input information of a target object input into the large model by a knowledge classifier in the large model to obtain a classification result comprises:

Converting the input information into a first vector by a large model;

inputting the first vector into a first target neural network model corresponding to the knowledge classifier, so as to determine a target label corresponding to the first vector in the input information label through the first target neural network model and the first prompt information;

and determining the target label as the classification result.

4. The method for generating feedback information based on a large model according to claim 1, wherein searching a first knowledge segment corresponding to the input information in a first knowledge base indicated by the classification result comprises:

determining a first vector into which the input information is converted;

Under the condition that a plurality of second knowledge bases are included in the first knowledge base, L second knowledge segments are screened out from each second knowledge base according to second vectors included in the first vectors, wherein the second vectors are vectors obtained by converting metadata information included in the input information, and the metadata information is indication information used for screening the second knowledge segments in the input information;

Determining semantic similarity of the L second knowledge pieces in each second knowledge base with the first vector under the condition that L is higher than a first quantity W;

Determining W second knowledge segments corresponding to the semantic similarities respectively, so as to determine W second knowledge segments corresponding to each second knowledge base respectively, and determining a plurality of the W second knowledge segments as the first knowledge segments, wherein the minimum similarity in the W semantic similarities is larger than the maximum similarity in the rest similarity, and the rest similarity is the similarity except the W semantic similarities in the L semantic similarities;

wherein L, W is a positive integer, L, W is less than M.

5. The method for generating feedback information based on a large model according to claim 1, wherein searching a first knowledge segment corresponding to the input information in a first knowledge base indicated by the classification result comprises:

determining a first vector into which the input information is converted;

under the condition that the first knowledge base comprises a plurality of second knowledge bases, determining L second knowledge segments with highest semantic similarity with the first vector in each second knowledge base through an information retrieval technology;

Screening P second knowledge segments from the L second knowledge segments according to a second vector included in the first vector when L is higher than the first number W, wherein the second vector is a vector obtained by converting metadata information included in the input information, and the metadata information is indication information used for screening the second knowledge segments in the input information;

Under the condition that W is equal to P, determining W second knowledge segments corresponding to a plurality of second knowledge bases as the first knowledge segments;

When W is larger than P, P second knowledge segments corresponding to the second knowledge bases and Q second knowledge segments corresponding to the second knowledge bases are determined to be the first knowledge segments, wherein the Q second knowledge segments are Q second knowledge segments with highest semantic similarity except the P second knowledge segments in the L second knowledge segments;

Wherein L, W, P, Q is a positive integer, L, W, P, Q is less than M.

6. The method for generating feedback information based on a large model according to claim 1, wherein before classifying input information of a target object input into the large model by a knowledge classifier in the large model, the method further comprises:

Constructing second prompt information of a segment screener based on a large model, wherein the segment screener is used for screening R second knowledge segments searched out from a third knowledge base under the condition that the third knowledge base is included in the first knowledge base, and the first knowledge segments comprise the R second knowledge segments; the second prompt information is used for indicating scoring standards of the segment screener on the R second knowledge segments.

7. The method of generating large model-based feedback information according to claim 6, wherein before determining the similarity of M second knowledge pieces included in the first knowledge pieces to the input information, the method further comprises:

under the condition that the first knowledge base comprises the third knowledge base, inputting the R second knowledge segments into a second target neural network model corresponding to the segment screener, so as to determine similarity scores of knowledge segment titles respectively included in the R second knowledge segments and the input information through the second target neural network model and the second prompt information;

determining S second knowledge segments and T second knowledge segments as the M second knowledge segments, wherein the S second knowledge segments are S second knowledge segments with highest similarity scores in the R second knowledge segments, the T second knowledge segments are knowledge segments except the R second knowledge segments in the first knowledge segments, R, S, T are positive integers, and R is larger than S.

8. The method for generating feedback information based on a large model according to claim 1, wherein determining the similarity of M second knowledge pieces included in the first knowledge pieces to the input information comprises:

determining a source knowledge base of a fourth knowledge segment in the case that a plurality of second knowledge bases and/or third knowledge bases are included in the first knowledge base, wherein the fourth knowledge segment is any knowledge segment in the M second knowledge segments, and the plurality of second knowledge bases and/or third knowledge bases include the source knowledge base;

determining semantic similarity of the fourth knowledge segment and the input information, and determining preset weights of the source knowledge base;

And determining the product of the semantic similarity and the preset weight as the similarity of the fourth knowledge segment.

9. The method for generating large model-based feedback information according to claim 1, comprising:

Calculating the importance value of the fourth knowledge piece according to the following formula:

value=num（B）/len(B)+s_bm25；

10. The method for generating feedback information based on a large model according to claim 1, wherein determining, by a knowledge filter in the large model, K third knowledge segments having a response relationship with the input information from among the N second knowledge segments includes:

Splicing N second knowledge segments according to the sequence from the big similarity to the small similarity to obtain a first context;

and screening K third knowledge segments with response relation with the input information from the first context through a knowledge screening device.

11. The method for generating feedback information based on a large model according to claim 1, wherein inputting K third knowledge pieces into the large model includes:

Determining the extraction sequence of the K third knowledge pieces from N second knowledge pieces;

and splicing the K third knowledge segments according to the extraction sequence to obtain a second context, and inputting the second context into the large model.

12. The method for generating large model-based feedback information according to claim 1, comprising:

Directly generating feedback information of the input information through a knowledge generator of a large model in case that a target condition is satisfied, wherein the target condition comprises one of the following:

the first knowledge segments are not searched from the first knowledge base;

Determining that no second knowledge segments with response relation with the input information exist in the N second knowledge segments through the knowledge screener;

The knowledge generator is arranged behind a knowledge filter of the large model or behind an input information interface of the large model, and the input information interface is used for inputting the input information.

13. The method of generating feedback information based on a large model according to claim 1, wherein after inputting K third pieces of knowledge to the large model to instruct the large model to generate feedback information for the input information based on the K third pieces of knowledge and the input information, the method further comprises:

Determining the matching degree of the feedback information and the input information through a knowledge evaluator, and grading the feedback information according to the matching degree to obtain a grading value, wherein the large model comprises the knowledge evaluator, and the knowledge evaluator is arranged behind a knowledge generator of the large model and is connected with the knowledge generator;

Under the condition that the grading value is higher than a preset grading value, determining the feedback information as first feedback information, and sampling and auditing the first feedback information through a target object;

And under the condition that the scoring value is lower than or equal to a preset scoring value, determining the feedback information as second feedback information, and modifying the second feedback information through the target object.

14. A large model-based feedback information generation apparatus, comprising:

The classification module is used for classifying input information of a target object input into the large model through a knowledge classifier in the large model to obtain a classification result, and searching a first knowledge segment corresponding to the input information in a first knowledge base indicated by the classification result;

A first determining module, configured to determine similarities between M second knowledge pieces included in the first knowledge pieces and the input information, and determine second knowledge pieces corresponding to N similarities respectively, where a minimum similarity of the N similarities is greater than a maximum similarity of other similarities, where the other similarities are similarities of the M similarities except the N similarities;

the second determining module is used for determining K third knowledge segments with response relation with the input information in the N second knowledge segments through a knowledge screening device in the large model;

The generation module is used for inputting K third knowledge segments into the large model to instruct the large model to generate feedback information for the input information based on the K third knowledge segments and the input information; wherein M, N and K are positive integers, M is greater than N, and N is greater than or equal to K;

The first determining module is further configured to calculate the similarity score according to the following formula:

score = a*sim*value；

The first determining module is further configured to calculate a semantic similarity sim of the fourth knowledge piece according to the following formula:

sim=similarity(A,B)*X+Y*min(1,n_B/n_A);