CN116541536A - Knowledge-enhanced content generation system, data generation method, device, and medium - Google Patents

Knowledge-enhanced content generation system, data generation method, device, and medium Download PDF

Info

Publication number
CN116541536A
CN116541536A CN202310632730.2A CN202310632730A CN116541536A CN 116541536 A CN116541536 A CN 116541536A CN 202310632730 A CN202310632730 A CN 202310632730A CN 116541536 A CN116541536 A CN 116541536A
Authority
CN
China
Prior art keywords
intermediate output
knowledge
input data
sample
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310632730.2A
Other languages
Chinese (zh)
Other versions
CN116541536B (en
Inventor
王海峰
吴华
�田�浩
刘璟
陈艳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN202310632730.2A priority Critical patent/CN116541536B/en
Publication of CN116541536A publication Critical patent/CN116541536A/en
Application granted granted Critical
Publication of CN116541536B publication Critical patent/CN116541536B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
    • G06F16/367—Ontology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The disclosure provides a knowledge-enhanced content generation system, a data generation method, a data generation device and a medium, relates to the technical field of artificial intelligence, and particularly relates to the technical fields of natural language processing, deep learning and the like. The system comprises: the deep learning large model can generate reply data only based on input data of a user; a knowledge retrieval component for providing retrieval results based on the input query. The deep learning large model is configured to: in response to determining that the first input data of the user includes a plurality of knowledge points, outputting a plurality of intermediate output data including first intermediate output data and second intermediate output data, the first intermediate data and the second intermediate output data corresponding to respective knowledge points; obtaining a plurality of search results of the knowledge search component aiming at a plurality of intermediate output data, wherein the plurality of search results comprise a first search result and a second search result which respectively correspond to the first intermediate output data and the second intermediate output data; a reply is generated based on the first input data and the search result.

Description

Knowledge-enhanced content generation system, data generation method, device, and medium
Technical Field
The present disclosure relates to the field of artificial intelligence technology, and in particular, to the technical fields of natural language processing, deep learning, and the like, and in particular, to a knowledge-enhanced content generation system, a data generation method based on the knowledge-enhanced content generation system, an electronic device, a computer-readable storage medium, and a computer program product.
Background
Artificial intelligence is the discipline of studying the process of making a computer mimic certain mental processes and intelligent behaviors (e.g., learning, reasoning, thinking, planning, etc.) of a person, both hardware-level and software-level techniques. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, and the like; the artificial intelligence software technology mainly comprises natural language processing technology, computer vision technology, voice recognition technology, machine learning/deep learning technology, big data processing technology, knowledge graph technology and other big directions.
The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, the problems mentioned in this section should not be considered as having been recognized in any prior art unless otherwise indicated.
Disclosure of Invention
The present disclosure provides a knowledge-enhanced content generation system, a data generation method, an electronic device, a computer-readable storage medium, and a computer program product based on the knowledge-enhanced content generation system.
According to an aspect of the present disclosure, there is provided a knowledge-enhanced content generation system, comprising: a deep learning large model capable of generating reply data based only on input data of a user; and a knowledge retrieval component for providing corresponding retrieval results based on the input query, wherein the deep learning big model is configured to: in response to determining that the first input data of the user includes a plurality of knowledge points, outputting a plurality of intermediate output data including first intermediate output data and second intermediate output data, wherein the first intermediate data and the second intermediate output data correspond to respective ones of the plurality of knowledge points; obtaining a corresponding plurality of search results of the knowledge search component for the plurality of intermediate output data, wherein the plurality of search results includes a first search result corresponding to the first intermediate output data and a second search result corresponding to the second intermediate output data; and generating reply content based on the first input data and the plurality of search results.
According to another aspect of the present disclosure, there is provided a data generation method of a knowledge-based enhanced content generation system, the content generation system including: a trained large language model deep learning large model capable of generating reply data based only on input data of a user; and a knowledge retrieval component for providing corresponding retrieval results based on the input query, wherein the method comprises: inputting first input data of a user into a trained large language model deep learning large model to obtain first output data, wherein the first input data comprises a plurality of knowledge points according to the large language model deep learning large model, the first output data comprises a plurality of intermediate output data, and the plurality of intermediate output data at least comprises first intermediate output data and second intermediate output data, and the first intermediate data and the second intermediate output data respectively correspond to corresponding knowledge points in the plurality of knowledge points; inputting the plurality of intermediate output data into a knowledge retrieval component to obtain a corresponding plurality of retrieval results, wherein the plurality of retrieval results includes a first retrieval result corresponding to the first intermediate output data and a second retrieval result corresponding to the second intermediate output data; and inputting the first input data and the plurality of search results into the trained large language model deep learning large model to obtain reply content.
According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described above.
According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the above-described method.
According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the above-described method.
According to one or more embodiments of the present disclosure, the present disclosure makes a decision on whether input data of a user includes a plurality of knowledge points that need to be externally retrieved by using a deep learning big model, and outputs a plurality of intermediate output data corresponding to the plurality of knowledge points if it is determined that the input data includes the plurality of knowledge points, and further queries the intermediate output data as input of a knowledge retrieval component to obtain a plurality of retrieval results corresponding to the plurality of knowledge points, and finally processes the input data of the user and the plurality of retrieval results by using the deep learning big model to obtain reply contents of the input data of the user.
By the aid of the method, the deep learning large model and the content generation system which can perform tasks such as understanding and content generation are further improved in knowledge, and accordingly quality of finally generated reply content is improved. The content generation system can fully and fully understand the input data of the user by utilizing the deep learning large model to identify and split a plurality of knowledge points in the input data of the user, and can generate reply content with high fact accuracy and timeliness by generating a plurality of input queries corresponding to the plurality of knowledge points, utilizing a knowledge retrieval component to acquire retrieval results corresponding to the knowledge points and finally utilizing the acquired retrieval content to assist the deep learning large model to follow the input data of the user.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The accompanying drawings illustrate exemplary embodiments and, together with the description, serve to explain exemplary implementations of the embodiments. The illustrated embodiments are for exemplary purposes only and do not limit the scope of the claims. Throughout the drawings, identical reference numerals designate similar, but not necessarily identical, elements.
FIG. 1A shows a schematic diagram of an exemplary system in which various methods described herein may be implemented, according to an embodiment of the present disclosure;
FIG. 1B shows a schematic diagram of interactions with a knowledge-enhanced content generation system, in accordance with an embodiment of the present disclosure;
FIG. 1C illustrates a schematic diagram of a user interacting with a deep learning large model without knowledge enhancement, according to an embodiment of the present disclosure;
FIG. 1D shows a schematic diagram of a user interacting with a knowledge retrieval component in accordance with an embodiment of the disclosure;
FIG. 2 illustrates a block diagram of a knowledge-enhanced content generation system, in accordance with an embodiment of the present disclosure;
FIG. 3 shows a flowchart of training steps for a deep learning large model according to an embodiment of the present disclosure;
FIG. 4 illustrates a flow chart of joint optimization training in accordance with an embodiment of the present disclosure;
FIG. 5 illustrates a flow chart of a data generation method of a knowledge-based enhanced content generation system, in accordance with an embodiment of the disclosure; and
fig. 6 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the present disclosure, the use of the terms "first," "second," and the like to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of the elements, unless otherwise indicated, and such terms are merely used to distinguish one element from another element. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, they may also refer to different instances based on the description of the context.
The terminology used in the description of the various illustrated examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, the elements may be one or more if the number of the elements is not specifically limited. Furthermore, the term "and/or" as used in this disclosure encompasses any and all possible combinations of the listed items.
In the related art, content generation systems can only generate based on knowledge in the corpus they train, and cannot access real-time information. In addition, such a content generation system has problems of inaccurate knowledge memory and low generation controllability.
In order to solve the above problems, the present disclosure uses a deep learning big model to make a decision on whether the input data of the user includes a plurality of knowledge points that need to be searched externally, and outputs a plurality of intermediate output data corresponding to the plurality of knowledge points if it is determined that the input data includes the plurality of knowledge points, and further uses the intermediate output data as an input query of a knowledge search component to obtain a plurality of search results corresponding to the plurality of knowledge points, and finally uses the deep learning big model to process the input data of the user and the plurality of search results to obtain reply contents of the input data of the user.
By the aid of the method, the deep learning large model and the content generation system which can perform tasks such as understanding and content generation are further improved in knowledge, and accordingly quality of finally generated reply content is improved. The content generation system can fully and fully understand the input data of the user by utilizing the deep learning large model to identify and split a plurality of knowledge points in the input data of the user, and can generate reply content with high fact accuracy and timeliness by generating a plurality of input queries corresponding to the plurality of knowledge points, utilizing a knowledge retrieval component to acquire retrieval results corresponding to the knowledge points and finally utilizing the acquired retrieval content to assist the deep learning large model to follow the input data of the user.
Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
Fig. 1A shows a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented, in accordance with embodiments of the present disclosure. Referring to fig. 1A, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. Client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the data generation methods of the present disclosure. In one exemplary embodiment, a complete content generation system, or some component of the content generation system, such as a deep learning large model and a retrieval model in a knowledge retrieval component, may be deployed on a server.
In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and/or 106 under a software as a service (SaaS) model.
In the configuration shown in fig. 1A, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or a combination thereof that are executable by one or more processors. A user operating client devices 101, 102, 103, 104, 105, and/or 106 may in turn utilize one or more client applications to interact with server 120 to utilize the services provided by these components. It should be appreciated that a variety of different system configurations are possible, which may differ from system 100. Accordingly, FIG. 1A is one example of a system for implementing the various methods described herein and is not intended to be limiting.
The user may use client devices 101, 102, 103, 104, 105, and/or 106 to input to the content generation system. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface, e.g., may output a reply to the user generated by the content generation system for user input. Although fig. 1A depicts only six client devices, one skilled in the art will appreciate that the present disclosure may support any number of client devices.
Client devices 101, 102, 103, 104, 105, and/or 106 may include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, linux, or Linux-like operating systems (e.g., GOOGLE Chrome OS); or include various mobile operating systems such as MICROSOFT Windows Mobile OS, iOS, windows Phone, android. Portable handheld devices may include cellular telephones, smart phones, tablet computers, personal Digital Assistants (PDAs), and the like. Wearable devices may include head mounted displays (such as smart glasses) and other devices. The gaming system may include various handheld gaming devices, internet-enabled gaming devices, and the like. The client device is capable of executing a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications), short Message Service (SMS) applications, and may use a variety of communication protocols.
Network 110 may be any type of network known to those skilled in the art that may support data communications using any of a number of available protocols, including but not limited to TCP/IP, SNA, IPX, etc. For example only, the one or more networks 110 may be a Local Area Network (LAN), an ethernet-based network, a token ring, a Wide Area Network (WAN), the internet, a virtual network, a Virtual Private Network (VPN), an intranet, an extranet, a blockchain network, a Public Switched Telephone Network (PSTN), an infrared network, a wireless network (e.g., bluetooth, WIFI), and/or any combination of these and/or other networks.
The server 120 may include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-end servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination. The server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture that involves virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices of the server). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.
The computing units in server 120 may run one or more operating systems including any of the operating systems described above as well as any commercially available server operating systems. Server 120 may also run any of a variety of additional server applications and/or middle tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.
In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and/or event updates received from users of client devices 101, 102, 103, 104, 105, and/or 106. Server 120 may also include one or more applications to display data feeds and/or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and/or 106.
In some implementations, the server 120 may be a server of a distributed system or a server that incorporates a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in a cloud computing service system, so as to solve the defects of large management difficulty and weak service expansibility in the traditional physical host and virtual private server (VPS, virtual PrivateServer) service.
The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of databases 130 may be used to store information such as audio files and video files. Database 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. Database 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve the databases and data from the databases in response to the commands.
In some embodiments, one or more of databases 130 may also be used by applications to store application data. The databases used by the application may be different types of databases, such as key value stores, object stores, or conventional stores supported by the file system.
The system 100 of fig. 1A may be configured and operated in various ways to enable application of the various methods and apparatuses described in accordance with the present disclosure.
According to one aspect of the present disclosure, a knowledge-enhanced content generation system is provided. As shown in fig. 2, the content generation system 200 includes: deep learning large model 210, the trained large language model deep learning large model is capable of generating reply data based only on user input data; and a knowledge retrieval component 220 for providing corresponding retrieval results based on the input query. The deep-learning large model 210 is configured to: in response to determining that the first input data of the user includes a plurality of knowledge points, outputting a plurality of intermediate output data including first intermediate output data and second intermediate output data, wherein the first intermediate data and the second intermediate output data correspond to respective ones of the plurality of knowledge points; obtaining a corresponding plurality of search results of the knowledge search component for the plurality of intermediate output data, wherein the plurality of search results includes a first search result corresponding to the first intermediate output data and a second search result corresponding to the second intermediate output data; and generating reply content based on the first input data and the plurality of search results.
Therefore, by the aid of the method, the deep learning large model and the content generation system which can perform tasks such as understanding and content generation are further improved in knowledge, and accordingly quality of finally generated reply content is improved. The content generation system can fully and fully understand the input data of the user by utilizing the deep learning large model to identify and split a plurality of knowledge points in the input data of the user, and can generate reply content with high fact accuracy and timeliness by generating a plurality of input queries corresponding to the plurality of knowledge points, utilizing a knowledge retrieval component to acquire retrieval results corresponding to the knowledge points and finally utilizing the acquired retrieval content to assist the deep learning large model to follow the input data of the user.
FIG. 1B illustrates an exemplary embodiment of user interaction with a knowledge-enhanced content generation system 150. The first input data for the user may be "where tomorrow a and B are hotter? The first input data includes two knowledge points, namely, a city open air temperature and B city open air temperature. In response to the deep learning large model, determining that a plurality of knowledge points are included in the first input data of the user, the output includes a plurality of intermediate output data including the first intermediate output data "what the A city tomorrow temperature is" and the second intermediate output data "what the B city tomorrow temperature is". It can be seen that the first intermediate output data and the second intermediate output data correspond to the above two knowledge points, respectively. The first intermediate output data and the second intermediate output data are input as queries to the knowledge retrieval component 154, respectively, to obtain corresponding retrieval results. The search result corresponding to the first intermediate output result may be "a city open sun air temperature 10-15 degrees", and the search result corresponding to the second intermediate output result may be "B city open sun air temperature 18-25 degrees". Further, the first input data and the two search results may be input to the deep learning large model 152, so that the reply content "the air temperature of the open world in city a is 10 to 15 degrees and the air temperature of the open world in city B is 18 to 25 degrees" for the first input data generated by the deep learning large model 152 is obtained. From this, tomorrow's B market is hotter.
If the first input data in the above-described exemplary embodiment is not subjected to knowledge point recognition, splitting, and separate retrieval, but is directly processed as a whole by the deep learning large model 160 without knowledge enhancement, the deep learning large model generates reply content based on knowledge in the trained corpus. Because real-time information cannot be accessed, outdated reply content may be generated or the user may be prompted that the content reply system is unable to generate a corresponding reply, as shown in FIG. 1C. Moreover, the deep learning large model has the problems of inaccurate knowledge memory and low generation controllability, so that the generated reply content may have a fact error or low quality.
In addition, in consideration of the complexity of the first input data of the user, there may be no knowledge content in the knowledge base corresponding to the knowledge retrieval component that can be directly used as the retrieval result corresponding to the first input data, and thus the knowledge retrieval component cannot generate accurate reply content. In such a case, if the first input data is retrieved using the knowledge retrieval component 170, the resulting retrieval result may or may not be partially related to the first input data of the user, e.g., the knowledge retrieval component may return a climate comparison of city a and city B, as shown in fig. 1D. Even if such a search result is input into the deep learning large model, the deep learning large model cannot generate accurate reply content.
It can be seen that the recognition and splitting of multiple knowledge points in the input data of the user by the deep learning large model can further refine granularity of knowledge content required to be acquired by the knowledge retrieval component, so that the possibility of acquiring accurate and available knowledge content from the knowledge retrieval component is improved. And by processing the retrieval results corresponding to the thinned multiple knowledge points and the input data of the user by using the deep learning large model, the understanding capability and the reasoning capability of the deep learning large model can be fully utilized to generate accurate reply content.
In this disclosure, deep learning large models are also referred to as understanding generation unified interaction large models (simply understanding generation large models or unified large models). The deep-learning large model has end-to-end characteristics, and reply data can be directly generated based on input data of a user without resorting to functional components or other inputs outside the deep-learning large model. In other words, the deep learning large model itself has a generation function. The deep learning large model may be a large language model. Large language models are typically deep-learning large models with billions or even billions of parameters that are typically trained on large-scale text data or other modalities of data. The large language model may be used for various natural language processing tasks such as text generation, language translation, and question-answering systems, etc.
The deep learning large model may employ, for example, an N-layer fransformer network structure with an Encoder (Encoder) and a Decoder (Decoder), or a Unified pre-training language model (UniLM) network structure. It is understood that the deep learning large model may also be other neural network models based on a transducer network structure, which is not limited herein. The input and output of the deep learning large model are each composed of tokens (token). Each token may correspond to a single word, character, word, special symbol, or to some external component (e.g., a knowledge retrieval component), as will be described below. The deep learning large model can be trained by using a pre-training task and a generating task so as to have the generating function.
The knowledge retrieval component may be a model or system with knowledge retrieval functionality that is capable of providing corresponding retrieval results based on received input queries.
According to some embodiments, the knowledge retrieval component may be an external search engine. The external search engine may be a general search engine or a domain-specific knowledge engine. The intermediate output data generated by the deep-learning large model may be, for example, a search formula such that an external search engine may be utilized to search based on each search formula to obtain one or more search results corresponding to each search formula. These search results together constitute a plurality of search results obtained from the knowledge retrieval component.
According to some embodiments, the knowledge retrieval component may include a knowledge retrieval model and a knowledge base. The knowledge retrieval model may be an end-to-end large model based on a Transformer structure, which may include only a single model, or may further include a recall model and a sort model. The knowledge retrieval model may be trained with pre-training tasks and retrieval tasks to provide knowledge retrieval capabilities. The knowledge base can be a general knowledge base, a professional knowledge base or a privately owned database. The knowledge retrieval model and knowledge base may also have other forms, not limited herein.
In some embodiments, the knowledge points may be knowledge content that is referred to in the first input data of the user. For example, "who is star a and star B higher? "may include two knowledge points" height of Star A "and" height of Star B "; "is the specific weight of trade volume of country a versus country B in GDP of country a? "may include two knowledge points," trade amount of country a versus country B "and" GDP of country a ". Knowledge points split from the user's input data may relate to physical content such as news, counseling, data, general knowledge, knowledge of various areas, or to questions with specific answers. By inputting the query (query) corresponding to the knowledge point into the knowledge retrieval component, a corresponding retrieval result, namely a corresponding knowledge content, can be obtained, thereby realizing knowledge enhancement of the content generation system.
According to some embodiments, the plurality of intermediate output data may include a plurality of intermediate output tokens (tokens) generated one by one, and the plurality of intermediate output tokens may include a plurality of characters corresponding to each of the intermediate output data and a separator token for indicating that a single intermediate output data has been completely output. The plurality of intermediate output data may be defined as:
Q list ＝{q 1 ，q 2 ，q 3 ，[sep]，..，q n }
wherein q i Representing the ith token (token) result in the generated result. To output multiple query terms at a time, we use a special mark symbol [ sep ]]Representing a complete intermediate output data has been completely output. When q i ＝[sep]When it is, it represents a complete intermediate output data generation.
In some embodiments, in response to determining that the first input data of the user includes a plurality of knowledge points, outputting a plurality of intermediate output data including the first intermediate output data and the second intermediate output data includes: determining a probability distribution corresponding to an nth intermediate output token to be generated, wherein the probability distribution corresponding to the nth intermediate output token is equal to a product of a continuous multiplication of an ith intermediate output probability from i to N and a first intermediate output probability, the first intermediate output probability indicating a probability that the deep learning large model generates the first intermediate output token based on the first input data when determining that the first input data includes a plurality of knowledge points, the ith intermediate output probability indicating a probability that the deep learning large model generates the ith intermediate output token based on the first input data and all intermediate output tokens preceding the ith intermediate output token when determining that the first input data includes a plurality of knowledge points; and sampling based on the probability distribution corresponding to the nth intermediate output token to generate the nth intermediate output token.
In some embodiments, the probability distribution corresponding to the nth intermediate output token may be expressed as:
wherein S is search Indicating that the knowledge retrieval component is currently required to be invoked, promt refers to the user's input data.
Thus, by the method, the generated part in the plurality of intermediate output data can be completely considered when each token is generated, so that more accurate results can be generated.
In some embodiments, when sampling is performed based on the probability distribution corresponding to the nth intermediate output token, the token corresponding to the word having the highest probability may be selected, or a plurality of words having the highest probability may be selected, or other sampling methods may be used, which are not limited herein.
It should be noted that the plurality of intermediate output data are continuously output at once by the deep learning large model. That is, the deep-learning large model does not receive other input data between the deep-learning large model outputting a plurality of intermediate output data.
In some embodiments, the deep-learning large model may output an token indicating that a knowledge retrieval component needs to be invoked before outputting the plurality of intermediate output data.
In some embodiments, the first input data and the plurality of search results may be fused (e.g., spliced, etc.). The deep learning large model generates reply content for input data of the user based on the fusion result.
According to some embodiments, the reply content may include a plurality of reply content qualifiers generated one by one. The plurality of reply content reamers may include a plurality of character reamers corresponding to the reply content.
In some embodiments, generating the reply content based on the first input data and the plurality of search results may include: determining a probability distribution corresponding to an nth reply content token to be generated, wherein the probability distribution corresponding to the nth reply content token is equal to a continuous multiplication of an ith reply content probability from i equal to 1 to N, the ith reply content probability indicating a probability that the deep learning big model generates the ith reply content token based on the first input data, the plurality of search results and all reply content tokens preceding the ith reply content token; and sampling based on the probability distribution corresponding to the nth reply content token to generate the nth reply content token.
In some embodiments, the probability distribution corresponding to the nth reply content token may be expressed as:
wherein K represents a plurality of knowledge points, namely a plurality of acquired search results, and Prompt represents input data of a user.
Thus, by the method, the generated part in the reply content can be completely considered when each token corresponding to the reply content is generated, so that more accurate results can be generated.
According to some embodiments, the content generation system may further include another component other than the deep learning large model and knowledge retrieval component. Generating the reply content based on the first input data and the plurality of search results may include: in response to determining that generating reply content based on the first input data and the plurality of search results requires invoking another component, generating an token for invoking the other component and an intermediate query that is determined based on the first input data and that is identifiable by the other component; and generating reply content based at least on the first input data, the plurality of search results, and the intermediate result, wherein the intermediate result is determined by another component based on the intermediate query.
Therefore, through the mode, the deep learning large model which can execute tasks such as understanding, knowledge point splitting, search query generation, reply content generation and the like is further realized, and the capability enhancement is further realized, so that the quality of a finally generated reply is improved. In addition, the intermediate query which can be identified by another functional component is directly generated by utilizing the deep learning large model, so that the intermediate query and the intermediate result can be more consistent with the potential intention in the input data of the user, and the model can output a reply meeting the requirement of the user.
In some embodiments, another component may be, for example, an external memory bank, an application program interface, or the like. Each of these different functional components has a corresponding token. The deep-learning large model can make a decision as to whether to invoke these components (and/or which functional component to invoke), and the decision results can be embodied in the results output by the deep-learning large model as to whether to include the token corresponding to the invoking functional component (and/or specifically which functional component to include in the results).
In some embodiments, when predicting a deep learning large model based on a transform network structure, the model first receives initial input to generate a first output token_1. The model then receives token_1 to generate a second output token token_2. And repeating the cyclic call of the deep learning large model until the token_n of the model output indicates that the model output is finished. Each token output by the model may correspond to a particular functional component to embody a decision result of whether to invoke the functional component; or in the form of a specific tag (markup) to generate an intermediate query that can be identified by a specific functional component; but also specific individual words, characters or words to generate a reply to the user input; and may be a special symbol to indicate that the current content has been generated. Thus, automated decision making with the model is enabled to determine the tasks that need to be performed next (e.g., invoking a functional component or generating a reply).
According to some embodiments, the other component may be an external repository in which a first set of data sets associated with the user may be stored, and wherein each data set in the first set of data sets may include at least a history input data item and a history reply item generated by the deep learning big model for the history input data item.
Therefore, the external memory library is set to store the history dialogue of the user and the content generation system in a long period, so that the memory capacity of the content generation system is improved, and the deep learning large model can refer to the history dialogue to generate a reply with stronger pertinence and richer and more specific content by acquiring the history dialogue related to user input, so that the reply quality is improved, the intelligence of the dialogue is improved, and the user experience is enhanced.
According to some embodiments, the other component may be at least one application program interface capable of being invoked by a deep learning large model. The different APIs each have a corresponding tag (markup) form, i.e., a token for calling the API. When a large model outputs a token/tag corresponding to a particular API in deep learning large model prediction, the content generation system knows that it is necessary to trigger that API. The large model will then continue to output intermediate queries (i.e., inputs for the API, which may also be referred to as rewritten query) that can be recognized by the API. Further, the input of the deep-learning large model may be determined based on the intermediate result obtained by calling the API with the intermediate query, and the large model may be made to continue prediction.
In some embodiments, APIs used in the content generation system may include scientific calculators, form processing tools, smart home control, and the like. Therefore, the capability expansion of the content generation system is realized by calling the APIs capable of executing various tasks. By using external functional components such as a scientific calculator, the problem of weak logic computing capacity of the deep learning large model is solved, and the logic reasoning capacity of the whole content generation system is improved. Compared with the method of calling the API by using the mapping table of the keywords and the API calling instructions, the deep learning large model is used for directly generating the intermediate query which can be identified by the API, so that the intermediate query and the obtained intermediate result can be more fit with the potential intention of the user in the initial input, the quality of the finally generated reply is improved, and the intelligence of the system is enhanced. In addition, by combining the understanding generation large model with the API, the content generation system can be provided with automatic operation execution capability, and the capability expansion of the deep learning large model and the content generation system is realized.
According to some embodiments, the large model may directly generate reply content in the event that it is determined that no other components need to be invoked. Generating the reply content based on the first input data and the plurality of search results may include: responsive to determining that generating the reply content based on the first input data and the plurality of search results does not require invoking any of the components other than the deep learning big model and the knowledge search component, the reply content is directly generated.
According to some embodiments, as shown in fig. 3, the deep-learning large model may also be trained by: step S301, acquiring first sample input data and a plurality of real output data corresponding to the first sample input data, wherein the first sample input data comprises a plurality of first sample knowledge points, the plurality of real output data comprises first real middle output data and second real middle output data, the first real middle output data and the second real middle output data respectively correspond to corresponding first sample knowledge points in the plurality of first sample knowledge points, and the first real middle output data and the second real middle output data can be respectively used for inputting knowledge retrieval components to acquire knowledge content related to the corresponding first sample knowledge points; step S302, inputting first sample input data into an initial deep learning large model to be trained so as to obtain corresponding at least one prediction output data; and step S303, adjusting parameters of the initial deep learning large model based on the plurality of real output data and at least one predicted output data to obtain the deep learning large model.
Thus, by the method, the trained deep learning large model can be enabled to have the capability of identifying and splitting a plurality of knowledge points in input data.
In step S301, the first sample input data may be, for example, input data of a real user or data of a structure. The first sample input data includes a plurality of first sample knowledge points, and reply contents to the first sample input data can be obtained based on knowledge contents corresponding to the plurality of first sample knowledge points. The plurality of real output data corresponding to the first sample input data may be, for example, artificially noted and correspond to a plurality of first sample knowledge points in the first sample input data, respectively. These real output data can be used as query inputs into the knowledge retrieval component to obtain knowledge content related to the corresponding first sample knowledge points.
In step S302, the initial deep learning large model to be trained may not output predicted output data for inputting the knowledge retrieval component to acquire the knowledge content based on the first sample input data, may generate only one predicted output data, and may also generate two or more predicted output data. The purpose of steps S301-S303 is to enable the trained deep learning large model to output results similar to or even identical to the multiple real output data based on the first sample input data.
In step S303, a loss value may be determined based on the plurality of real output data and the at least one predicted output data according to a predetermined loss function, and further parameters of the initial deep learning large model may be reversely optimized by means of random gradient descent or the like based on the loss value. The loss function may be determined according to specific requirements and is not limited herein.
In some embodiments, the predetermined loss function may include calculating a matching value of the predicted output data and the actual output data. In one exemplary embodiment, the matching value may be calculated based on the number of output data satisfying a perfect match with any one of the real output data and the number of predicted output data generated by the deep-learning large model among the at least one predicted output data.
According to some embodiments, the knowledge retrieval model may be a result of joint optimization training with a deep learning large model. As shown in fig. 4, the joint optimization training includes: step S401, obtaining second sample input data and a sample input query, wherein the second sample input data at least comprises second sample knowledge points, and the sample input query corresponds to the second sample knowledge points; step S402, inputting a sample input query into an initial knowledge retrieval model to be jointly optimized so as to obtain a plurality of corresponding sample retrieval results; step S403, inputting a plurality of sample retrieval results and second sample input data into a deep learning large model respectively to obtain a plurality of sample reply contents corresponding to the plurality of sample retrieval results respectively; step S404, evaluating the reply contents of a plurality of samples based on a predetermined standard, and labeling the retrieval results of the plurality of samples based on the evaluation results; and step S405, based on the labeling results of the plurality of sample retrieval results, adjusting the parameters of the initial knowledge retrieval model to obtain the knowledge retrieval model.
Therefore, by adopting the mode to carry out joint optimization training on the knowledge retrieval model and the deep learning large model, the knowledge acquired by the trained knowledge retrieval model is beneficial to the deep learning large model to generate better reply content, and the quality of the finally obtained reply content is improved.
In step S401, the second sample input data may be similar to the first sample input data described above, and will not be described herein. The sample input query may be output by, for example, a deep learning large model based on the first sample input data, or may be obtained by other means (e.g., manual annotation), without limitation.
In some embodiments, in step S402, the initial knowledge retrieval model may be used to determine the relevance of the sample input query to the plurality of candidate contents in the knowledge base, respectively, and then a certain number of contents with the highest relevance among the candidate contents may be used as the plurality of sample search results. In some embodiments, an initial indicative search model may also be utilized to determine whether a plurality of candidate content in the knowledge base matches the sample input query, and the matched candidate content is used as a plurality of sample search results.
In step S403, each of the plurality of sample retrieval results may be fused with the second sample input data and then input into the deep learning large model, so as to obtain corresponding sample reply contents. In other words, a plurality of sample search results may be used as the retrieved knowledge content, respectively, and sample reply content may be generated based on each of the retrieved knowledge content, respectively, using the deep learning large model.
In step S404, the sample reply content generated based on each retrieved knowledge content may be evaluated using a predetermined criterion, and a plurality of sample retrieval results may be labeled based on the evaluation results. The labeling results of the multiple sample retrieval results can comprehensively embody the evaluation results of the reply contents of the multiple samples. For example, the better the evaluation result, the higher the score of the corresponding sample retrieval result.
According to some embodiments, step S404, evaluating the plurality of sample reply contents based on a predetermined criterion, and labeling the plurality of sample retrieval results based on the evaluation results may include: sorting the plurality of sample reply contents based on the evaluation result; and labeling the plurality of sample retrieval results based on the sequencing results of the reply contents of the plurality of samples.
Therefore, the sorting results of the quality of the replies to the respective corresponding sample contents can be reflected in the labeling of the plurality of sample retrieval results, and training of the initial knowledge retrieval model based on the labeling results enables the knowledge retrieval model after joint optimization to provide the knowledge contents which are favorable for the deep learning large model to generate better replies.
In an exemplary embodiment, for the second sample input data and the corresponding sample input query, two sample retrieval results may be obtained from the knowledge retrieval component, and sample reply contents may be generated based on the two sample retrieval results, respectively, so that the sample retrieval result corresponding to the sample reply content with the better evaluation result may be set as the 2 nd stage, and the sample retrieval result corresponding to the sample reply content with the worse evaluation result may be set as the 1 st stage. In addition, knowledge content that the knowledge retrieval component considers to be irrelevant to the sample input query may also be set to level 0. Through the labeling mode, the labeling result can better reflect the quality difference of sample reply contents generated based on different sample retrieval results, so that the knowledge retrieval model can be helped to train and optimize to provide knowledge contents which are favorable for deep learning of a large model to generate better reply contents.
According to some embodiments, the pre-determined criteria may include at least the accuracy and format of the generated content.
In some embodiments, the pre-determined criteria may include scoring the reply content generation results in terms of fluency, diversity, consistency, and taking the average of the generated scores as the acceptability score for the reply content. In one exemplary embodiment, the score may range from 1 to 5 points.
In step S405, a loss value may be determined according to a predetermined loss function based on labeling results of multiple sample retrieval results, and further parameters of the initial knowledge retrieval component may be reversely optimized by means of random gradient descent or the like based on the loss value. The loss function may be determined according to specific requirements and is not limited herein.
According to some embodiments, the first input data may include raw input data from a user and context information of the raw input data. The raw input data of the user may be, for example, data entered by the user through text, voice input, or other means. The context information may include a number of rounds of conversations that have occurred between the user and the content generation system prior to the acquired user's original input data.
In some embodiments, the context information includes multiple rounds of conversations that the user has occurred with the content generation system in a current session with the content generation system, but does not include conversations sent in a historical session of the user with the content generation system. In other words, when the user closes an application or service of the content generation system, the context information is cleared accordingly; and when the user opens the application or service of the content generation system again, the context information resumes recording.
Furthermore, the context information typically has a preset maximum encodable length, limited by the upper input length limit of the deep learning large model, with limited memory. Thus, after a user has made multiple rounds or a longer session with the content generation system, a portion of the content in the context information may be discarded.
According to another aspect of the present disclosure, a data generation method of a knowledge-based enhanced content generation system is disclosed. The content generation system includes: a deep learning large model capable of generating reply data based only on input data of a user; and a knowledge retrieval component for providing corresponding retrieval results based on the input query. As shown in fig. 5, the data generation method includes: step S501, inputting first input data of a user into a deep learning large model to obtain first output data, wherein the first input data comprises a plurality of knowledge points according to the deep learning large model, the first output data comprises a plurality of intermediate output data, and the plurality of intermediate output data at least comprises first intermediate output data and second intermediate output data, wherein the first intermediate data and the second intermediate output data respectively correspond to corresponding knowledge points in the plurality of knowledge points; step S502, inputting a plurality of intermediate output data into a knowledge retrieval component to obtain a corresponding plurality of retrieval results, wherein the plurality of retrieval results comprise a first retrieval result corresponding to the first intermediate output data and a second retrieval result corresponding to the second intermediate output data; and step S503, inputting the first input data and the plurality of search results into the deep learning large model to obtain reply content. It is to be understood that the operations of step S501-step S503 in the data generating method may refer to the description of the deep learning large model 210 hereinabove, and are not described herein.
Therefore, by the aid of the method, the deep learning large model and the content generation system which can perform tasks such as understanding and content generation are further improved in knowledge, and accordingly quality of finally generated reply content is improved. The content generation system can fully and fully understand the input data of the user by utilizing the deep learning large model to identify and split a plurality of knowledge points in the input data of the user, and can generate reply content with high fact accuracy and timeliness by generating a plurality of input queries corresponding to the plurality of knowledge points, utilizing a knowledge retrieval component to acquire retrieval results corresponding to the knowledge points and finally utilizing the acquired retrieval content to assist the deep learning large model to follow the input data of the user.
According to some embodiments, the knowledge retrieval component may be an external search engine.
According to some embodiments, the knowledge retrieval component may include a knowledge retrieval model and a knowledge base.
According to some embodiments, the plurality of intermediate output data comprises a plurality of intermediate output tokens generated one by one, the plurality of intermediate output tokens comprising a plurality of character tokens corresponding to each of the intermediate output data and a separator token for indicating that a single intermediate output data has been completely output, wherein the deep learning big model is configured to: determining a probability distribution corresponding to an nth intermediate output token to be generated, wherein the probability distribution corresponding to the nth intermediate output token is equal to a product of a continuous multiplication of an ith intermediate output probability from i to N and a first intermediate output probability, the first intermediate output probability indicating a probability that the deep learning large model generates the first intermediate output token based on the first input data when determining that the first input data includes a plurality of knowledge points, the ith intermediate output probability indicating a probability that the deep learning large model generates the ith intermediate output token based on the first input data and all intermediate output tokens preceding the ith intermediate output token when determining that the first input data includes a plurality of knowledge points; and sampling based on the probability distribution corresponding to the nth intermediate output token to generate the nth intermediate output token.
According to some embodiments, the reply content may include a plurality of reply content qualifiers generated one by one. The deep learning large model may be configured to: determining a probability distribution corresponding to an nth reply content token to be generated, wherein the probability distribution corresponding to the nth reply content token is equal to a continuous multiplication of an ith reply content probability from i equal to 1 to N, the ith reply content probability indicating a probability that the deep learning big model generates the ith reply content token based on the first input data, the plurality of search results and all reply content tokens preceding the ith reply content token; and sampling based on the probability distribution corresponding to the nth reply content token to generate the nth reply content token.
According to some embodiments, the content generation system may further include another component other than the deep learning large model and knowledge retrieval component. Step S503, inputting the first input data and the plurality of search results into the deep learning big model to obtain the reply content may include: inputting the first input data and the plurality of search results into a deep learning big model to obtain second output data, wherein the second component is required to be invoked by generating reply content based on the first input data and the plurality of search results in response to the deep learning big model, and the third intermediate output data comprises an token for invoking the second component and an intermediate query which is determined based on the first input data and can be identified by the second component; obtaining an intermediate result determined by another component based on the intermediate query; inputting at least the first input data, the plurality of search results and the intermediate result into a deep learning large model to obtain third output data; and generating reply content based on the third output data. It is to be understood that the operation of step S503 is similar to the operation of generating the reply content based on the first input data and the plurality of search results, and is not described herein.
According to some embodiments, the other component may be an external repository in which a first set of data sets associated with the user may be stored, and wherein each data set in the first set of data sets may include at least a history input data item and a history reply item generated by the deep learning big model for the history input data item.
According to some embodiments, the other component may be at least one application program interface capable of being invoked by a deep learning large model.
According to some embodiments, step S503, inputting the first input data and the plurality of search results into the deep learning big model to obtain the reply content may include: generating the reply content based on the first input data and the plurality of search results in response to the deep-learning big model determination does not require invoking any of the components that are distinct from the deep-learning big model and the knowledge search component.
According to some embodiments, the deep-learning large model may be trained by: acquiring first sample input data and a plurality of real output data corresponding to the first sample input data, wherein the first sample input data comprises a plurality of first sample knowledge points, the plurality of real output data comprises first intermediate output data and second intermediate output data, the first intermediate output data and the second intermediate output data respectively correspond to corresponding first sample knowledge points in the plurality of first sample knowledge points, and the first intermediate output data and the second intermediate output data can be respectively used for inputting a knowledge retrieval component to acquire knowledge content related to the corresponding first sample knowledge points; inputting the first sample input data into an initial deep learning large model to be trained to obtain corresponding at least one predicted output data; and adjusting parameters of the initial deep learning large model based on the plurality of real output data and the at least one predicted output data to obtain the deep learning large model.
According to some embodiments, the knowledge retrieval model is obtained by joint optimization training with a deep learning large model. The joint optimization training may include: acquiring second sample input data and a sample input query, wherein the second sample input data at least comprises second sample knowledge points, and the sample input query corresponds to the second sample knowledge points; inputting the sample input query into an initial knowledge retrieval model to be jointly optimized so as to obtain a plurality of corresponding sample retrieval results; inputting the multiple sample retrieval results and second sample input data into a deep learning large model respectively to obtain multiple sample reply contents corresponding to the multiple sample retrieval results respectively; evaluating the reply contents of the plurality of samples based on a predetermined standard, and labeling the retrieval results of the plurality of samples based on the evaluation results; and adjusting parameters of the initial knowledge retrieval model based on labeling results of the plurality of sample retrieval results to obtain the knowledge retrieval model.
According to some embodiments, evaluating the plurality of sample reply contents based on a predetermined criterion and labeling the plurality of sample retrieval results based on the evaluation results may include: sorting the plurality of sample reply contents based on the evaluation result; and labeling the plurality of sample retrieval results based on the sequencing results of the reply contents of the plurality of samples.
According to some embodiments, the pre-determined criteria may include at least the accuracy and format of the generated content.
According to some embodiments, the first input data may include raw input data from a user and context information of the raw input data.
In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the user accord with the regulations of related laws and regulations, and the public order colloquial is not violated.
According to embodiments of the present disclosure, there is also provided an electronic device, a readable storage medium and a computer program product.
Referring to fig. 6, a block diagram of an electronic device 600 that may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 6, the electronic device 600 includes a computing unit 601 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, ROM 602, and RAM603 are connected to each other by a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
A number of components in the electronic device 600 are connected to the I/O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, the input unit 606 may receive input numeric or character information and generate key signal inputs related to user settings and/or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and/or a remote control. The output unit 607 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. Storage unit 608 may include, but is not limited to, magnetic disks, optical disks. The communication unit 609 allows the electronic device 600 to exchange information/data with other devices through a computer network, such as the internet, and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth devices, 802.11 devices, wiFi devices, wiMax devices, cellular communication devices, and/or the like.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the respective methods and processes described above, such as a data generation method. For example, in some embodiments, the data generation method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 600 via the ROM 602 and/or the communication unit 609. When a computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the data generating method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the data generation method by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), the internet, and blockchain networks.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.
Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the foregoing methods, systems, and apparatus are merely exemplary embodiments or examples, and that the scope of the present invention is not limited by these embodiments or examples but only by the claims following the grant and their equivalents. Various elements of the embodiments or examples may be omitted or replaced with equivalent elements thereof. Furthermore, the steps may be performed in a different order than described in the present disclosure. Further, various elements of the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after the disclosure.

Claims (31)

1. A knowledge-enhanced content generation system, comprising:
A deep learning large model capable of generating reply data based only on input data of a user; and
a knowledge retrieval component for providing corresponding retrieval results based on the input query,
wherein the deep learning large model is configured to:
in response to determining that the first input data of the user includes a plurality of knowledge points, outputting a plurality of intermediate output data including first intermediate output data and second intermediate output data, wherein the first intermediate data and the second intermediate output data respectively correspond to respective knowledge points of the plurality of knowledge points;
obtaining a respective plurality of search results for the plurality of intermediate output data by the knowledge search component, wherein the plurality of search results includes a first search result corresponding to the first intermediate output data and a second search result corresponding to the second intermediate output data; and
and generating reply content based on the first input data and the plurality of search results.
2. The content generation system of claim 1, wherein the plurality of intermediate output data comprises a plurality of intermediate output commanders generated one by one, the plurality of intermediate output commanders comprising a plurality of character commars corresponding to each of the intermediate output data and a separator commar for indicating that a single intermediate output data has been completely output, wherein the outputting the plurality of intermediate output data comprising the first intermediate output data and the second intermediate output data in response to determining that the first input data of the user comprises a plurality of knowledge points comprises:
Determining a probability distribution corresponding to an nth intermediate output token to be generated, wherein the probability distribution corresponding to the nth intermediate output token is equal to a product of a continuous multiplication of an ith intermediate output probability from i equal to 2 to N and a first intermediate output probability, the first intermediate output probability indicating a probability that the deep learning large model generates a first intermediate output token based on the first input data when determining that the first input data includes a plurality of knowledge points, the ith intermediate output probability indicating a probability that the deep learning large model generates an ith intermediate output token based on the first input data and all intermediate output tokens preceding the ith intermediate output token when determining that the first input data includes a plurality of knowledge points; and
sampling is performed based on a probability distribution corresponding to the nth intermediate output token to generate the nth intermediate output token.
3. The content generation system according to claim 1 or 2, wherein the deep-learning large model is a large language model, the deep-learning large model being trained by:
acquiring first sample input data and a plurality of real output data corresponding to the first sample input data, the first sample input data including a plurality of first sample knowledge points, the plurality of real output data including first real intermediate output data and second real intermediate output data, the first real intermediate output data and the second real intermediate output data respectively corresponding to respective first sample knowledge points of the plurality of first sample knowledge points, and the first real intermediate output data and the second real intermediate output data respectively being usable for input to the knowledge retrieval component to obtain knowledge content related to the respective first sample knowledge points;
Inputting the first sample input data into an initial deep learning large model to be trained to obtain corresponding at least one predicted output data; and
based on the plurality of real output data and the at least one predicted output data, parameters of the initial deep-learning large model are adjusted to obtain the deep-learning large model.
4. A content generation system as claimed in any one of claims 1 to 3 wherein the knowledge retrieval component is an external search engine.
5. A content generation system as claimed in any one of claims 1 to 3 wherein the knowledge retrieval component comprises a knowledge retrieval model and a knowledge base.
6. The content generation system of claim 3, wherein the knowledge retrieval model is derived from a joint optimization training with the deep learning large model, the joint optimization training comprising:
acquiring second sample input data and a sample input query, wherein the second sample input data at least comprises a second sample knowledge point, and the sample input query corresponds to the second sample knowledge point;
inputting the sample into a query input initial knowledge retrieval model to be jointly optimized so as to obtain a plurality of corresponding sample retrieval results;
Inputting the plurality of sample retrieval results and the second sample input data into the deep learning large model respectively to obtain a plurality of sample reply contents corresponding to the plurality of sample retrieval results respectively;
evaluating the reply contents of the plurality of samples based on a predetermined standard, and labeling the retrieval results of the plurality of samples based on the evaluation results; and
and adjusting parameters of the initial knowledge retrieval model based on labeling results of the plurality of sample retrieval results to obtain the knowledge retrieval model.
7. The content generation system of claim 6, wherein evaluating the plurality of sample reply content based on predetermined criteria and labeling the plurality of sample retrieval results based on the evaluation results comprises:
sorting the plurality of sample reply contents based on the evaluation result;
labeling the plurality of sample retrieval results based on the sequencing results of the plurality of sample reply contents.
8. The content generation system of claim 6, wherein the pre-determined criteria includes at least an accuracy and a format of the generated content.
9. A content generation system as claimed in any one of claims 1 to 3, wherein the reply content includes a plurality of reply content qualifiers generated one by one, generating reply content based on the first input data and the plurality of search results includes:
Determining a probability distribution corresponding to an nth reply content token to be generated, wherein the probability distribution corresponding to the nth reply content token is equal to a continuous multiplication of an ith reply content probability from i equal to 1 to N, the ith reply content probability indicating a probability that the deep learning big model generates an ith reply content token based on the first input data, the plurality of search results, and all reply content tokens preceding the ith reply content token; and
sampling is performed based on a probability distribution corresponding to the nth reply content token to generate the nth reply content token.
10. A content generation system as claimed in any one of claims 1 to 3, wherein the first input data comprises raw input data from a user and context information of the raw input data.
11. The content generation system of any of claims 1 to 3, wherein the content generation system further comprises another component, different from the deep-learning big model and the knowledge retrieval component, to generate reply content based on the first input data and the plurality of retrieval results, comprising:
generating an token for invoking the other component and an intermediate query that is identifiable by the other component and is determined based on the first input data in response to determining that generating reply content based on the first input data and the plurality of search results requires invoking the other component; and
The reply content is generated based at least on the first input data, the plurality of search results, and an intermediate result, wherein the intermediate result is determined by the other component based on the intermediate query.
12. The content generation system of claim 11, wherein the other component is an external repository having stored therein a first set of data sets associated with the user, and wherein each data set in the first set of data sets includes at least a historical input data item and a historical reply item generated by the deep learning large model for the historical input data item.
13. The content generation system of claim 11, wherein the other component is at least one application program interface capable of being invoked by the deep-learning large model.
14. The content generation system of any of claims 11, wherein generating reply content based on the first input data and the plurality of search results comprises:
responsive to determining that generating reply content based on the first input data and the plurality of search results does not require invoking any component other than the deep learning macro model and the knowledge search component, the reply content is directly generated.
15. A data generation method of a knowledge-based enhanced content generation system, the content generation system comprising:
a deep learning large model capable of generating reply data based only on input data of a user; and
a knowledge retrieval component for providing corresponding retrieval results based on the input query,
wherein the method comprises the following steps:
inputting first input data of a user into the deep learning large model to obtain first output data, wherein the first input data comprises a plurality of knowledge points according to the deep learning large model, the first output data comprises a plurality of intermediate output data, the plurality of intermediate output data at least comprises first intermediate output data and second intermediate output data, and the first intermediate data and the second intermediate output data respectively correspond to corresponding knowledge points in the plurality of knowledge points;
inputting the plurality of intermediate output data into the knowledge retrieval component to obtain a corresponding plurality of retrieval results, wherein the plurality of retrieval results includes a first retrieval result corresponding to the first intermediate output data and a second retrieval result corresponding to the second intermediate output data; and
And inputting the first input data and the plurality of search results into the deep learning large model to obtain reply content.
16. The method of claim 15, wherein the plurality of intermediate output data comprises a plurality of intermediate output reams generated one by one, the plurality of intermediate output reams comprising a plurality of character reams corresponding to each of the intermediate output data and a separator reams for indicating that a single intermediate output data has been completely output, wherein the deep learning large model is configured to:
determining a probability distribution corresponding to an nth intermediate output token to be generated, wherein the probability distribution corresponding to the nth intermediate output token is equal to a product of a continuous multiplication of an ith intermediate output probability from i equal to 2 to N and a first intermediate output probability, the first intermediate output probability indicating a probability that the deep learning large model generates a first intermediate output token based on the first input data when determining that the first input data includes a plurality of knowledge points, the ith intermediate output probability indicating a probability that the deep learning large model generates an ith intermediate output token based on the first input data and all intermediate output tokens preceding the ith intermediate output token when determining that the first input data includes a plurality of knowledge points; and
Sampling is performed based on a probability distribution corresponding to the nth intermediate output token to generate the nth intermediate output token.
17. The method of claim 15 or 16, wherein the deep-learning large model is trained by:
acquiring first sample input data and a plurality of real output data corresponding to the first sample input data, the first sample input data including a plurality of first sample knowledge points, the plurality of real output data including first intermediate output data and second intermediate output data, the first intermediate output data and the second intermediate output data corresponding respectively to respective first sample knowledge points of the plurality of first sample knowledge points, and the first intermediate output data and the second intermediate output data being respectively usable for input to the knowledge retrieval component to acquire knowledge content related to the respective first sample knowledge points;
inputting the first sample input data into an initial deep learning large model to be trained to obtain corresponding at least one predicted output data; and
based on the plurality of real output data and the at least one predicted output data, parameters of the initial deep-learning large model are adjusted to obtain the deep-learning large model.
18. The method of any of claims 15 to 17, wherein the knowledge retrieval component is an external search engine.
19. The method of any of claims 15 to 17, wherein the knowledge retrieval component comprises a knowledge retrieval model and a knowledge base.
20. The method of claim 17, wherein the knowledge retrieval model is derived from joint optimization training with the deep learning large model, the joint optimization training comprising:
acquiring second sample input data and a sample input query, wherein the second sample input data at least comprises a second sample knowledge point, and the sample input query corresponds to the second sample knowledge point;
inputting the sample into a query input initial knowledge retrieval model to be jointly optimized so as to obtain a plurality of corresponding sample retrieval results;
inputting the plurality of sample retrieval results and the second sample input data into the deep learning large model respectively to obtain a plurality of sample reply contents corresponding to the plurality of sample retrieval results respectively;
evaluating the reply contents of the plurality of samples based on a predetermined standard, and labeling the retrieval results of the plurality of samples based on the evaluation results; and
And adjusting parameters of the initial knowledge retrieval model based on labeling results of the plurality of sample retrieval results to obtain the knowledge retrieval model.
21. The method of claim 20, wherein evaluating the plurality of sample reply content based on predetermined criteria and labeling the plurality of sample retrieval results based on the evaluation results comprises:
sorting the plurality of sample reply contents based on the evaluation result;
labeling the plurality of sample retrieval results based on the sequencing results of the plurality of sample reply contents.
22. The method of claim 20, wherein the pre-determined criteria includes at least accuracy and format of the generated content.
23. The method of any of claims 15 to 17, wherein the reply content includes a plurality of reply content qualifiers generated one by one, the deep-learning large model being configured to:
determining a probability distribution corresponding to an nth reply content token to be generated, wherein the probability distribution corresponding to the nth reply content token is equal to a continuous multiplication of an ith reply content probability from i equal to 1 to N, the ith reply content probability indicating a probability that the deep learning big model generates an ith reply content token based on the first input data, the plurality of search results, and all reply content tokens preceding the ith reply content token; and
Sampling is performed based on a probability distribution corresponding to the nth reply content token to generate the nth reply content token.
24. The method of any of claims 15 to 17, wherein the first input data comprises raw input data from a user and contextual information of the raw input data.
25. The method of any of claims 15 to 17, wherein the content generation system further comprises another component that is different from the deep-learning big model and the knowledge retrieval component, wherein inputting the first input data and the plurality of retrieval results into the deep-learning big model to obtain reply content comprises:
inputting the first input data and the plurality of search results into the deep learning big model to obtain second output data, wherein the further component needs to be invoked in response to the deep learning big model determining that generating reply content based on the first input data and the plurality of search results, the third intermediate output data comprising an order for invoking the further component and an intermediate query that is identifiable by the further component and is determined based on the first input data;
Obtaining an intermediate result determined by the other component based on the intermediate query;
inputting at least the first input data, the plurality of search results, and the intermediate result into the deep learning large model to obtain third output data; and
and generating the reply content based on the third output data.
26. The method of claim 25, wherein the other component is an external repository having stored therein a first set of data sets associated with the user, and wherein each data set in the first set of data sets includes at least a historical input data item and a historical reply item generated by the deep learning large model for the historical input data item.
27. The method of claim 25, wherein the other component is at least one application program interface capable of being invoked by the deep-learning large model.
28. The method of any of claims 25, wherein inputting the first input data and the plurality of search results into the deep-learning large model to obtain reply content comprises:
generating reply content based on the first input data and the plurality of search results in response to the deep-learning big model determination does not require invoking any component distinct from the deep-learning big model and the knowledge search component to directly generate the reply content.
29. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the method comprises the steps of
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 15-28.
30. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 15-28.
31. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 15-28.
CN202310632730.2A 2023-05-30 2023-05-30 Knowledge-enhanced content generation system, data generation method, device, and medium Active CN116541536B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310632730.2A CN116541536B (en) 2023-05-30 2023-05-30 Knowledge-enhanced content generation system, data generation method, device, and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310632730.2A CN116541536B (en) 2023-05-30 2023-05-30 Knowledge-enhanced content generation system, data generation method, device, and medium

Publications (2)

Publication Number Publication Date
CN116541536A true CN116541536A (en) 2023-08-04
CN116541536B CN116541536B (en) 2024-03-01

Family

ID=87450678

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310632730.2A Active CN116541536B (en) 2023-05-30 2023-05-30 Knowledge-enhanced content generation system, data generation method, device, and medium

Country Status (1)

Country Link
CN (1) CN116541536B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116842126A (en) * 2023-08-29 2023-10-03 青岛网信信息科技有限公司 Method, medium and system for realizing accurate output of knowledge base by using LLM

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109299133A (en) * 2017-07-24 2019-02-01 迅讯科技（北京）有限公司 Data query method, computer system and non-transitory computer-readable medium
CN114020896A (en) * 2021-11-26 2022-02-08 中国银行股份有限公司 Intelligent question and answer method, system, electronic equipment and storage medium
US20230083443A1 (en) * 2021-09-16 2023-03-16 Evgeny Saveliev Detecting anomalies in physical access event streams by computing probability density functions and cumulative probability density functions for current and future events using plurality of small scale machine learning models and historical context of events obtained from stored event stream history via transformations of the history into a time series of event counts or via augmenting the event stream records with delay/lag information
CN115952274A (en) * 2023-03-10 2023-04-11 北京百度网讯科技有限公司 Data generation method, training method and device based on deep learning model

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109299133A (en) * 2017-07-24 2019-02-01 迅讯科技（北京）有限公司 Data query method, computer system and non-transitory computer-readable medium
US20230083443A1 (en) * 2021-09-16 2023-03-16 Evgeny Saveliev Detecting anomalies in physical access event streams by computing probability density functions and cumulative probability density functions for current and future events using plurality of small scale machine learning models and historical context of events obtained from stored event stream history via transformations of the history into a time series of event counts or via augmenting the event stream records with delay/lag information
CN114020896A (en) * 2021-11-26 2022-02-08 中国银行股份有限公司 Intelligent question and answer method, system, electronic equipment and storage medium
CN115952274A (en) * 2023-03-10 2023-04-11 北京百度网讯科技有限公司 Data generation method, training method and device based on deep learning model

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116842126A (en) * 2023-08-29 2023-10-03 青岛网信信息科技有限公司 Method, medium and system for realizing accurate output of knowledge base by using LLM
CN116842126B (en) * 2023-08-29 2023-12-19 青岛网信信息科技有限公司 Method, medium and system for realizing accurate output of knowledge base by using LLM

Also Published As

Publication number Publication date
CN116541536B (en) 2024-03-01

Similar Documents

Publication Publication Date Title
CN115952274B (en) Data generation method, training method and device based on deep learning model
CN116501960B (en) Content retrieval method, device, equipment and medium
US20240104154A1 (en) Ranking of recall data
CN116541536B (en) Knowledge-enhanced content generation system, data generation method, device, and medium
CN116303962A (en) Dialogue generation method, training method, device and equipment for deep learning model
CN114443989B (en) Ranking method, training method and device of ranking model, electronic equipment and medium
CN115114424A (en) Response method and device for query request
CN114547244A (en) Method and apparatus for determining information
CN115879469B (en) Text data processing method, model training method, device and medium
CN115862031B (en) Text processing method, neural network training method, device and equipment
CN114219046B (en) Model training method, matching method, device, system, electronic equipment and medium
CN114547270B (en) Text processing method, training method, device and equipment for text processing model
CN115964462A (en) Dialogue content processing method, and training method and device of dialogue understanding model
CN116541537A (en) Knowledge graph-based enterprise trade information visual display method
CN115809364B (en) Object recommendation method and model training method
CN116521841B (en) Method, device, equipment and medium for generating reply information
CN116841506B (en) Program code generation method and device, and model training method and device
CN114861658B (en) Address information analysis method and device, equipment and medium
CN116842156B (en) Data generation method, device, equipment and medium
CN116450917B (en) Information searching method and device, electronic equipment and medium
CN115879468B (en) Text element extraction method, device and equipment based on natural language understanding
CN115033782B (en) Object recommendation method, training method, device and equipment of machine learning model
CN117992675A (en) Content recommendation method and device, electronic equipment and storage medium
CN116521841A (en) Method, device, equipment and medium for generating reply information
CN117291191A (en) Text processing method, device, equipment and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant