WO2021120543A1 - 基于自然语言和知识图谱的表示学习方法及装置 - Google Patents

基于自然语言和知识图谱的表示学习方法及装置 Download PDF

Info

Publication number
WO2021120543A1
WO2021120543A1 PCT/CN2020/095108 CN2020095108W WO2021120543A1 WO 2021120543 A1 WO2021120543 A1 WO 2021120543A1 CN 2020095108 W CN2020095108 W CN 2020095108W WO 2021120543 A1 WO2021120543 A1 WO 2021120543A1
Authority
WO
WIPO (PCT)
Prior art keywords
knowledge graph
learning
layer
representation
entity
Prior art date
Application number
PCT/CN2020/095108
Other languages
English (en)
French (fr)
Inventor
王海峰
姜文斌
吕雅娟
朱勇
吴华
Original Assignee
北京百度网讯科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京百度网讯科技有限公司 filed Critical 北京百度网讯科技有限公司
Priority to KR1020207036186A priority Critical patent/KR102524766B1/ko
Priority to EP20864301.5A priority patent/EP3866025A4/en
Priority to JP2020571787A priority patent/JP7250052B2/ja
Priority to US17/124,030 priority patent/US12019990B2/en
Publication of WO2021120543A1 publication Critical patent/WO2021120543A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
    • G06F16/367—Ontology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G06F16/9024—Graphs; Linked lists
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/02—Knowledge representation; Symbolic representation

Definitions

  • the embodiments of the application relate to the field of artificial intelligence (AI) technology, and in particular to a representation learning method and device based on natural language and knowledge graphs.
  • AI artificial intelligence
  • KG knowledge graph
  • knowledge graph and natural language are two independent fields. Both of these two fields have independently developed a representation learning technology system.
  • knowledge graph representation learning is usually in the form of vector space operations.
  • the co-occurrence law between the node (Point) and the edge (Edge) is modeled to represent the semantics of the learning knowledge graph;
  • the natural language representation learning is usually in the form of sequence generation to model the co-occurrence law between words or sentences , In order to learn the semantic representation of natural language.
  • the embodiments of the application provide a representation learning method and device based on natural language and knowledge graphs.
  • a representation learning method and device based on natural language and knowledge graphs.
  • an embodiment of the present application provides a text processing method based on natural language and knowledge graphs, including: receiving a text processing request input by a user, and the text processing request is used to request a semantic representation processing based on a predicted object in the text
  • the prediction object is input to a pre-trained joint learning model to obtain a semantic representation of the prediction object
  • the joint learning model is used for knowledge graph representation learning and natural language representation learning
  • the semantic representation is obtained by combining the knowledge graph representation learning and the natural language representation learning, and the text is processed according to the semantic representation.
  • the method before receiving the text processing request input by the user, the method further includes: training training samples to obtain the joint learning model, and the joint learning model includes a natural language learning layer, a joint learning correlation layer, and A knowledge graph learning layer, and the joint learning association layer is used to associate the knowledge graph learning layer with the natural language learning layer.
  • the training of training samples to obtain the joint learning model includes: determining neighbor samples of the target sample in the training sample at the natural language learning layer, and at the joint learning correlation layer According to the neighbor samples, determine the weight of the target training sample relative to each entity in the knowledge graph learning layer, determine the semantic representation of the knowledge graph of the target training sample according to the weight of each entity, and determine the semantic representation of the knowledge graph of the target training sample according to the knowledge The semantic representation of the graph spectrum and the neighbor samples determine the training result of the target sample.
  • the determining the weight of the target training sample relative to each entity in the knowledge graph learning layer at the joint learning association layer includes: for each entity in the knowledge graph learning layer , Determining M weights of the target training sample relative to the entity in the joint learning correlation layer, where M ⁇ 1 and an integer.
  • the determining the semantic representation of the knowledge graph of the target training sample according to the weight of each entity includes: for each entity in the knowledge graph learning layer, according to the corresponding entity The weight processes the semantic representation of the knowledge graph of the entity to obtain multiple processed semantic representations of the knowledge graph, and determines the semantic representation of the knowledge graph of the target training sample according to the multiple processed semantic representations of the knowledge graph.
  • the method further includes: optimizing the knowledge graph learning layer contains multiple information based on the training result.
  • the optimization of the semantic representation of the knowledge graph of each of the entities included in the knowledge graph learning layer according to the training result includes: judging whether the training result is correct, if the training If the result is correct, an excitation signal is generated, and the semantic representation of the knowledge graph of each entity in the multiple entities included in the knowledge graph learning layer is enhanced according to the excitation signal. If the training result is wrong, a penalty signal is generated, according to the The penalty signal adjusts the semantic representation of the knowledge graph of each of the multiple entities included in the knowledge graph learning layer.
  • an embodiment of the present application provides a text processing device based on natural language and knowledge graphs, including: a receiving module, configured to receive a text processing request input by a user, and the text processing request is used to request a prediction based on the text
  • the semantic representation of the object processes the text
  • the acquisition module is used to input the prediction object to a pre-trained joint learning model to obtain the semantic representation of the prediction object.
  • the joint learning model is used for knowledge graph representation learning and natural language representation learning, and the semantics
  • the representation is obtained by the joint learning model combining the knowledge graph representation learning and the natural language representation learning;
  • the processing module is used to process the text according to the semantic representation.
  • the above-mentioned device further includes:
  • the training module is used to train the training samples to obtain the joint learning model before the receiving module receives the text processing request input by the user.
  • the joint learning model includes a natural language learning layer, a joint learning correlation layer, and a knowledge graph Learning layer, the joint learning association layer is used to associate the knowledge graph learning layer with the natural language learning layer.
  • the training module is used to determine the neighbor samples of the target sample in the training sample at the natural language learning layer, and determine the target according to the neighbor samples at the joint learning correlation layer
  • the training sample is relative to the weight of each entity in the knowledge graph learning layer
  • the semantic representation of the knowledge graph of the target training sample is determined based on the weight of each entity
  • the semantic representation of the knowledge graph is determined based on the semantic representation of the knowledge graph and the neighbor samples.
  • the training module when the joint learning association layer determines the weight of the target training sample with respect to each entity in the knowledge graph learning layer, the training module is configured to determine the weight of each entity in the knowledge graph learning layer.
  • An entity determines M weights of the target training sample relative to the entity at the joint learning association layer, where M ⁇ 1 and an integer.
  • the training module determines the semantic representation of the knowledge graph of the target training sample according to the weights of the entities, for each entity in the knowledge graph learning layer, according to the The weight corresponding to the entity processes the semantic representation of the knowledge graph of the entity to obtain multiple processed semantic representations of the knowledge graph, and determines the knowledge graph semantics of the target training sample according to the multiple processed semantic representations of the knowledge graph Said.
  • the training module after determining the training result of the target sample according to the semantic representation of the knowledge graph and the neighbor samples, is also used to optimize the learning of the knowledge graph according to the training result
  • the semantic representation of the knowledge graph of each entity in the multiple entities contained in the layer is also used to optimize the learning of the knowledge graph according to the training result.
  • the training module is specifically used to determine whether the training result is correct; if the training result is correct, an excitation signal is generated, and the knowledge graph learning layer is enhanced according to the excitation signal.
  • an electronic device including:
  • At least one processor At least one processor
  • a memory communicatively connected with the at least one processor; wherein,
  • the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the first aspect or any possible implementation of the first aspect method.
  • the embodiments of the present application provide a computer program product containing instructions, which when run on an electronic device, cause the electronic device computer to execute the above-mentioned method in the first aspect or various possible implementation manners of the first aspect .
  • an embodiment of the present application provides a storage medium that stores instructions in the storage medium, and when it runs on an electronic device, the electronic device executes various possibilities as described in the first aspect or the first aspect.
  • the method in the implementation mode is not limited to.
  • an embodiment of the present application provides a method for optimizing processor video memory for deep learning training tasks, including: a first processor determines a path for transmitting a calculation result of a first calculation unit to a second calculation unit, and the first The computing unit and the second computing unit are included in the first processor, there is at least one intermediate computing unit between the first computing unit and the second computing unit, and the first processor passes through the The path sends the calculation result of the first calculation unit to the second calculation sheet.
  • An embodiment in the above application has the following advantages or beneficial effects: after receiving the text processing request input by the user, the electronic device inputs the predicted object in the text to the pre-trained joint learning model to learn the semantic representation of the predicted object, The semantic representation is obtained by the joint learning model combining knowledge graph representation learning and natural language representation learning. After that, the electronic device processes the text according to the semantic representation.
  • the joint learning model since the semantic representation obtained by the electronic device using the joint learning model is obtained by the joint learning model combining knowledge graph representation learning and natural language representation learning, it combines knowledge graph learning representation and natural language learning representation, compared to Using only knowledge graph representation learning or natural language representation learning to learn the semantic representation of the predicted object, the joint learning model considers more and more comprehensive factors, so the accuracy of semantic representation can be improved, and the accuracy of text processing can be improved.
  • Fig. 1 is a flowchart of a text processing method based on natural language and knowledge graph provided by an embodiment of the present application
  • FIG. 2 is a schematic structural diagram of a joint learning model applicable to a text processing method based on natural language and knowledge graph provided by an embodiment of the present application;
  • FIG. 3 is a schematic structural diagram of a text processing device based on natural language and knowledge graph provided by an embodiment of the application;
  • FIG. 4 is a schematic structural diagram of another text processing device based on natural language and knowledge graph provided by an embodiment of the application;
  • Fig. 5 is a block diagram of an electronic device used to implement a text processing method based on natural language and knowledge graphs according to an embodiment of the present application.
  • Knowledge graph and natural language processing are two separate fields.
  • the two fields develop independent representation learning systems, and the representation learning techniques in each field follow different modeling methods and optimization goals, and there is no cross-fusion.
  • Knowledge graph represents learning usually in the form of vector space operations to model the co-occurrence rules between nodes and edges, while natural language representation learning usually builds the co-occurrence rules between words or sentences in the form of sequence generation. mold.
  • this method is regarded as a preliminary version of introducing knowledge in natural language representation learning, but this method uses a multi-layer self-attention neural network as the learning model, and Use cloze, adjacent sentence pair judgment and other words and sentence co-occurrence tasks as learning goals.
  • the specific method of introducing knowledge is: according to the named entity dictionary, the corresponding word sequence in the natural language sentence is bound and treated, and as a unified processing object, it participates in the language representation learning process.
  • the so-called bound treatment means that if a word is in the named entity dictionary The corresponding entity does not separate the word as a whole.
  • adding a bracket to the word indicates that the word is an entity and cannot be split.
  • the effect of improving natural language representation learning with the help of named entity dictionary knowledge can be achieved.
  • multiple entities are recorded in the named entity dictionary, which can be regarded as a list of entities, which can be obtained according to the knowledge graph or obtained by collecting entities.
  • the above-mentioned method of improving natural language representation learning with the help of named entity dictionary knowledge also has two shortcomings: defect 1.
  • defect 1 In terms of the scope of knowledge use, only the named entity dictionary is used to judge whether a word or a word is not. There is a corresponding entity.
  • the knowledge graph contains not only a large number of nodes, but also edges, and the topological structure composed of nodes and edges, and the above method only uses the information of whether a word or character is an entity; defect two, in the use of knowledge
  • the above method uses natural language representation learning as the basic framework. On this basis, it is judged whether a word or character is an entity based on the entity list provided by the named entity dictionary.
  • the flow of information in this way is one-way, from external knowledge that is From named entity dictionary to natural language, the two-way correspondence rules between natural language and external knowledge cannot be effectively used.
  • the embodiments of the present application provide a text processing method and device based on natural language and knowledge graphs.
  • a better quality semantic representation can be learned.
  • the knowledge graph is composed of nodes and edges in the form of a graph, which is a structured representation form of knowledge.
  • the information represented by the nodes of the knowledge graph includes, but is not limited to, entities, concepts, interests, and events corresponding to specific knowledge graph types such as concept graphs, interest graphs, and time graphs; correspondingly, the information that edges can represent includes but is not limited to attributes , Subordination, timing and causality, etc.
  • the semantic representation of the nodes and edges of the knowledge graph can be used for artificial intelligence tasks such as knowledge base completion, knowledge base question and answer, intelligent recommendation, event analysis, language understanding, and machine translation.
  • natural language is composed of characters or words in a sequence. It is a tool and carrier for people to communicate and think. It can be used for cognitive intelligence tasks, such as reading comprehension, intelligent question and answer, and machine translation through text processing. And automatic writing, etc.
  • the electronic device performs joint modeling of the representation learning of the knowledge graph and the representation learning process of the natural language, and the association between the entities constituting the knowledge graph and the words constituting the natural language is introduced in the modeling process to obtain the joint Learning model, based on the joint learning model to learn better semantic representation of knowledge graph and natural language semantic representation.
  • the electronic device is, for example, a server or a terminal device.
  • FIG. 1 is a flowchart of a text processing method based on natural language and knowledge graph provided by an embodiment of the present application. This embodiment is described from the perspective of an electronic device, and this embodiment includes:
  • the joint learning model is pre-loaded on the electronic device, and when text processing is required, the user inputs a processing request to the electronic device through a click operation, touch operation, or voice input, and the electronic device receives and recognizes the text processing request.
  • the associative learning model is used to assist cognitive intelligence tasks
  • the text to be processed is processed by word marking, etc., and the text is split into individual words or words.
  • the word is the prediction object.
  • the electronic device when the associated learning model is used to assist the knowledge graph task, after receiving the text processing request, the electronic device recognizes the entities contained in the text to be processed, and these entities are the prediction objects.
  • the electronic device inputs the prediction object into the joint learning model, thereby obtaining a semantic representation of the prediction object.
  • the prediction object is a word or character
  • the associated learning model outputs a natural language semantic representation
  • the prediction object is an entity
  • the associated learning model outputs a semantic representation of the knowledge graph.
  • the semantic representation output by the associated learning model is obtained by the joint learning model combining knowledge graph representation learning and natural language representation learning in advance.
  • the electronic device trains the joint learning model, for a training sample that is a word or character, in addition to the neighbor training samples of the training sample, the knowledge map information of the training sample is also considered during the training process.
  • the training results of the training samples are also used to adjust the semantic representation of the knowledge graph of each entity in the knowledge graph, so that the semantic representation of each entity in the knowledge graph takes into account the natural language processing in addition to other entities in the knowledge graph. The training result of the training sample.
  • the electronic device processes the text according to these semantic representations.
  • the associated learning model is used to assist cognitive intelligence tasks
  • electronic devices perform reading comprehension, intelligent question answering, machine translation, or automatic writing based on semantic representations.
  • the electronic device performs knowledge base completion, knowledge base reasoning, and knowledge base question and answer according to the semantic representation.
  • the electronic device after receiving the text processing request input by the user, the electronic device inputs the prediction object in the text to the pre-trained joint learning model to learn the prediction object Semantic representation, which is obtained by a joint learning model combining knowledge graph representation learning and natural language representation learning. After that, the electronic device processes the text according to the semantic representation.
  • the joint learning model since the semantic representation obtained by the electronic device using the joint learning model is obtained by the joint learning model combining knowledge graph representation learning and natural language representation learning, it combines knowledge graph learning representation and natural language learning representation, compared to Using only knowledge graph representation learning or natural language representation learning to learn the semantic representation of the predicted object, the joint learning model considers more and more comprehensive factors, so the accuracy of semantic representation can be improved, and the accuracy of text processing can be improved.
  • the electronic device before the electronic device receives the text processing request input by the user, it also trains the training samples to obtain the above-mentioned joint learning model.
  • the above-mentioned joint learning model includes three core modules: the natural language learning layer and the joint learning association. Layer and a knowledge graph learning layer, and the joint learning association layer is used to associate the knowledge graph learning layer with the natural language learning layer.
  • FIG. 2 is a schematic structural diagram of a joint learning model applicable to a text processing method based on natural language and knowledge graph provided by an embodiment of the present application.
  • the joint learning model includes a natural language learning layer, a joint learning correlation layer and a knowledge graph learning layer, which correspond to the corresponding sub-neural network models.
  • the knowledge graph learning layer and the natural language learning layer cooperate through the joint learning correlation layer.
  • the coordination mechanism can use a variety of different neural network mechanisms.
  • the joint learning model also includes learning modules for other tasks, which are used to drive knowledge graph representation learning and natural language representation learning.
  • the solid line box shows the natural language learning layer.
  • Various mainstream neural network language models can be used, such as Recurrent Neural Network (RNN) model and Long Short-Term Memor.
  • LSTM model Transformer model, bidirectional Encoder Representations from Transformers (BERT) from the converter, and knowledge-enhanced semantic representation model (Enhanced Representation through kNowledge IntEgration, ERNIE), etc.
  • the essence of the natural language learning layer is to transfer information between sentences and words, and to model the association relationship between words, so as to achieve the effect of learning word representation.
  • the dashed box shows the knowledge graph learning layer, and graph neural networks can be used to better reflect the topological structure of the knowledge graph.
  • the knowledge graph learning layer may not adopt any model, but only regard the knowledge graph as a list composed of nodes and edges.
  • the dot-dash line shows the joint learning association layer, which is composed of one or more reading and writing mechanisms.
  • the role of the joint learning association layer is to establish associations between words and knowledge to realize natural language sentences and knowledge Information transfer between atlas knowledge bases.
  • the input of the reading and writing mechanism of the joint learning association layer is usually the words or words in the natural language learning layer, and the knowledge map in the knowledge graph learning layer, where the words or words are neighbor samples of the target training sample, for example, the training samples include n words or words, such as word 1 (word 1, w1) ⁇ wn in the figure, assuming that the target training sample is w3, then the neighbor samples of w3 when inputting, such as w1, w2, w4, w5, etc., and the knowledge graph,
  • the joint learning correlation layer obtains the semantic representation of the knowledge graph of the target training sample according to the input and outputs it.
  • the natural language learning layer obtains the training result of the target training sample according to the semantic representation of the knowledge graph of the target training sample and the
  • the purpose of training to obtain a joint learning model for knowledge graph representation learning and natural language representation learning is achieved.
  • the electronic device when the electronic device trains the training samples to obtain the joint learning model, it can use the knowledge graph learning layer to improve the natural language learning layer, and then the natural language learning layer can use the natural language learning layer to improve the knowledge graph learning layer.
  • the two aspects will be described in detail below.
  • the natural language learning layer determines the neighbor samples of the target sample in the training sample
  • the joint learning correlation layer determines the neighbor samples of the target sample in the joint learning correlation layer.
  • the neighbor samples determine the weight of the target training sample relative to each entity in the knowledge graph learning layer, and determine the semantic representation of the knowledge graph of the target training sample according to the weight of each entity, and according to the knowledge graph
  • the semantic representation and the neighbor samples determine the training result of the target sample.
  • the training samples include w1 ⁇ wn. Take w3 as the target training sample as an example.
  • the default training of training samples other than w3 is The results are known, and the known training results need to be used to predict the training results of w3.
  • the electronic device mainly uses the neighbor samples of w3, such as the training results of w1, w2, w4, and w5.
  • the training results of other training training samples other than w3 are known by default.
  • the electronic device predicts the training result of w3, it uses the neighbor samples w1 and w3 of w3.
  • the language representation of the knowledge graph of w3 also needs to be considered.
  • the knowledge spectrogram language representation of w3 can be obtained based on the neighbor samples of w3 and so on.
  • the electronic device integrates neighbor samples w1, w2, w4, w5, etc. of w3 to obtain integrated information, inputs the integrated information to the joint learning association layer, and determines that the integrated information is relative to the knowledge graph learning layer in the joint learning association layer
  • entity entity, e
  • the semantic representation of the knowledge graph of the target training sample is determined according to the five weights and the semantic representation of the knowledge graph of each entity; for example, for w3
  • the electronic device determines the weight of the neighbor sample relative to each entity (e) in the knowledge graph learning layer, such as ei, ej, ek, and el in the joint learning association layer, and obtains 20 weights, according to The 20 weights and the semantic representation of the knowledge graph of each entity determine
  • the knowledge graph is used to improve natural language representation learning. From the perspective of natural language representation learning, this method has a wider range of knowledge use and a more effective way of using knowledge.
  • representation learning joint modeling the method can be integrated Use the knowledge of nodes, edges, and the topological structure composed of nodes and edges to improve the learning effect of natural language.
  • the joint learning association layer can determine the semantic representation of the knowledge graph for the target training object from M different angles.
  • the electronic device determines the weight of the target training sample relative to each entity in the knowledge graph learning layer in the joint learning association layer, for each entity in the knowledge graph learning layer, the weight of the target training sample is determined in the joint learning association layer.
  • the learning association layer determines M weights of the target training sample relative to the entity, where M ⁇ 1 and an integer.
  • the joint learning association layer determines from M angles that the integrated information is relative to each of the knowledge graph learning layers.
  • Entities (e) such as the weights of ei, ej, ek, and el, obtain 5M weights. According to the 5M weights and the semantic representation of the knowledge graph of each entity, the semantic representation of the knowledge graph of the target training sample is determined.
  • the joint learning association layer is composed of one or more reading and writing mechanisms.
  • the most commonly used read-write mechanism is the attention mechanism.
  • the joint learning association layer can use one attention module or multiple attention modules.
  • Figure 2 shows the use of M attention modules.
  • the attention module can also be called a read write module.
  • the graph neural network has to be run M times to obtain M versions of knowledge graph data for the attention mechanism to access.
  • M copies of the same data structure namely the list of nodes and edges, need to be maintained. In this way, through M versions of the attention mechanism, these M copies are the same The data structure will learn the semantic representation of knowledge graphs from different perspectives.
  • the knowledge graph can be learned differently through multiple attention mechanisms to obtain the semantic representation of the knowledge graph from multiple angles of the target training sample, and the semantic representation of the knowledge graph from multiple angles and multiple neighbor samples are used to predict the target. Training samples to improve the accuracy of prediction.
  • the number of training samples in the embodiment of this application is extremely large, and the neighbor training samples of a target training sample can be 4 or more.
  • the number of entities in the knowledge graph learning presentation layer is also very special. Many, the above ei, ej, ek, and el are just examples. In actual implementation, the number of entities is tens of thousands or even more.
  • the electronic device After the electronic device obtains the semantic representation of the knowledge graph of the target training sample, it determines the training result of the target sample according to the semantic representation of the knowledge graph and the neighbor samples of the target training sample, that is, predicts the target training sample. For example, in Figure 2, the semantic representations of the knowledge graphs of w1, w2, w4, w5, and w3 are known, and which word or word w3 is predicted.
  • the embodiments of the present application are not limited thereto, and other feasible implementations are , Including but not limited to: a) Know the previous words, predict the current word, for example, know w1 and w2, predict w3; b) Know the previous and subsequent words, predict the current word, for example, Knowing w1, w2, w4, w5, predicting w3; c) predicting whether two sentences, that is, whether the two word sequences are adjacent sentences; d) knowing the previous multiple words, predicting the current multiple words, that is, the sentence, For example, if w1, w2, and w3 are known, w4 and w5 are predicted.
  • each entity processes the semantic representation of the knowledge graph of the entity according to the weight corresponding to the entity to obtain multiple processed semantic representations of the knowledge graph, and determines the semantic representation of the knowledge graph according to the multiple processed semantic representations of the knowledge graph The semantic representation of the knowledge graph of the target training sample.
  • the electronic device integrates the neighbor samples w1, w2, w4, w5, etc. of w3 to obtain integrated information, and inputs the integrated information to the joint learning association layer.
  • the joint learning association layer determines the weight of the integrated information relative to each entity (entity, e) in the knowledge graph learning layer, such as ei, ej, ek, and el, and multiplies the semantic representation of the knowledge graph of each entity with the corresponding weight , Get five processed semantic representations of the knowledge graph, and then perform a summation operation on the five processed semantic representations of the knowledge graph, thereby obtaining the semantic representation of the knowledge graph of the target training sample.
  • other operation methods may also be used, which are not limited in the embodiment of the present application.
  • the purpose of determining the semantic representation of the knowledge graph of the target training sample is achieved.
  • the electronic device determines the training result of the target sample according to the semantic representation of the knowledge graph and the neighbor samples, and then optimizes the multiple entities included in the knowledge graph learning layer according to the training result The semantic representation of each entity in the knowledge graph.
  • the semantic representation of each entity in the knowledge graph learning layer can be further optimized according to the training results of the target training samples.
  • this method can use natural language to express the learning training process, coordinate and optimize the knowledge map identification learning process, and assist and supplement the learning process of the knowledge map representation learning itself, thereby helping Learn a better knowledge graph representation.
  • the electronic device determines whether the training result of the target training object is correct. If the training result is correct, it generates an excitation signal, and enhances the knowledge graph learning layer according to the excitation signal.
  • the electronic device determines whether the training result of w3 is correct, that is, whether it correctly predicts which w3 is For the word or word, if the training result is correct, an excitation signal is generated and fed back to the knowledge graph learning layer, so that the knowledge graph learning layer enhances the semantic representation of the knowledge graph of entities ei, ej, ek, and el; if the training result is incorrect, it means The semantic representation of the knowledge graph of the entities ei, ej, ek, and el of the knowledge graph learning layer may be wrong.
  • the electronic device generates a penalty signal and feeds it back to the knowledge graph learning layer, so that the knowledge graph learning layer adjusts the knowledge graph learning model.
  • the semantic representation of each of the entities is, whether it correctly predicts which w3 is For the word or word, if the training result is correct, an excitation signal is generated and fed back to the knowledge graph learning layer, so that the knowledge graph learning layer enhances the semantic representation of the knowledge graph of entities ei, e
  • the purpose of improving the knowledge graph learning layer with the help of the natural language learning layer is achieved.
  • the entire joint learning model can be learned under the driving of the natural language learning layer.
  • the natural language learning layer adopts commonly used language model learning strategies, including but not limited to: a) Know the previous multiple words, predict the current word, for example, know w1 and w2, predict w3; b) have Know the multiple words before and after, predict the current word, for example, know w1, w2, w4, w5, predict w3; c) predict whether two sentences, that is, whether the two word sequences are adjacent sentences; d) know the previous The multiple words of, predict the current multiple words, that is, the sentence, for example, w1, w2, w3 are known, and w4 and w5 are predicted.
  • the natural language learning layer is driven by the above-mentioned language model learning strategy, and further drives the learning of the knowledge graph learning layer through the bridge function of the joint learning correlation layer, so that it can learn the natural semantic representation of natural language words and the knowledge graph simultaneously.
  • the semantic representation of the knowledge graph of nodes and edges can be regarded as the external knowledge base of the natural language learning layer, and the entire joint learning model can be regarded as a knowledge-enhanced language model.
  • the upload task can also be used to drive the training of the joint learning model.
  • the upper-level tasks refer to the cognitive intelligence tasks of the natural language learning layer, such as reading comprehension, problem systems, and machine translation. In specific implementation, it can be achieved by taking the representation of the words in the natural language learning layer as the input of the uppermost driving task. In this way, driven by the uppermost cognitive task, the parameters of the upper task itself, the parameters of the natural language learning layer, the parameters of the joint learning correlation layer, and the parameters of the knowledge graph learning layer can all be learned synchronously.
  • the knowledge graph not only includes the fact knowledge graph in the traditional sense, but also includes special knowledge graph types such as concept graphs, interest point graphs, and event graphs.
  • the nodes in the knowledge graph include, but are not limited to, entities, concepts, interests, and time
  • the edges include, but are not limited to, attributes, subordination, time series, and causal associations.
  • FIG. 3 is a schematic structural diagram of a text processing device based on natural language and knowledge graph provided by an embodiment of the application.
  • the device can be integrated in an electronic device or realized by an electronic device, and the electronic device can be a terminal device or a server.
  • the text processing apparatus 100 based on natural language and knowledge graph may include:
  • the receiving module 11 is configured to receive a text processing request input by a user, and the text processing request is used to request that the text be processed according to the semantic representation of the predicted object in the text;
  • the acquiring module 12 is configured to input the prediction object into a pre-trained joint learning model to obtain the semantic representation of the prediction object, and the joint learning model is used for knowledge graph representation learning and natural language representation learning.
  • the semantic representation is obtained by the joint learning model combining the knowledge graph representation learning and the natural language representation learning;
  • the processing module 13 is configured to process the text according to the semantic representation.
  • Fig. 4 is a schematic structural diagram of another text processing device based on natural language and knowledge graph provided by an embodiment of the application. According to Fig. 4, the above-mentioned text processing device 100 based on natural language and knowledge graph further includes:
  • the training module 14 is configured to train training samples to obtain the joint learning model before the receiving module 11 receives the text processing request input by the user.
  • the joint learning model includes a natural language learning layer, a joint learning correlation layer, and A knowledge graph learning layer, and the joint learning association layer is used to associate the knowledge graph learning layer with the natural language learning layer.
  • the training module 14 is used to determine the neighbor samples of the target sample in the training sample at the natural language learning layer, and determine the neighbor samples according to the neighbor samples at the joint learning association layer
  • the target training sample is relative to the weight of each entity in the knowledge graph learning layer
  • the semantic representation of the knowledge graph of the target training sample is determined according to the weight of each entity, and according to the semantic representation of the knowledge graph and the neighbor samples, Determine the training result of the target sample.
  • the training module 14 when the joint learning association layer determines the weight of the target training sample relative to each entity in the knowledge graph learning layer, the training module 14 is For each entity, M weights of the target training sample relative to the entity are determined at the joint learning association layer, where M ⁇ 1 and an integer.
  • the training module 14 determines the semantic representation of the knowledge graph of the target training sample according to the weight of each entity, for each entity in the knowledge graph learning layer, according to all The weight corresponding to the entity processes the semantic representation of the knowledge graph of the entity to obtain multiple processed semantic representations of the knowledge graph, and determines the knowledge graph of the target training sample according to the multiple processed semantic representations of the knowledge graph Semantic representation.
  • the training module 14 after determining the training result of the target sample according to the semantic representation of the knowledge graph and the neighbor samples, is also used to optimize the knowledge graph according to the training result The semantic representation of the knowledge graph of each of the multiple entities contained in the learning layer.
  • the training module 14 is specifically used to determine whether the training result is correct; if the training result is correct, an excitation signal is generated, and the knowledge graph learning layer is enhanced according to the excitation signal.
  • the device provided in the embodiment of the present application can be used in the method executed by the electronic device in the above embodiment, and its implementation principles and technical effects are similar, and will not be repeated here.
  • the present application also provides an electronic device and a readable storage medium.
  • Fig. 5 is a block diagram of an electronic device used to implement a text processing method based on natural language and knowledge graphs according to an embodiment of the present application.
  • Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.
  • Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices.
  • the components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the application described and/or required herein.
  • the electronic device includes: one or more processors 501, a memory 502, and interfaces for connecting various components, including a high-speed interface and a low-speed interface.
  • the various components are connected to each other using different buses, and can be installed on a common motherboard or installed in other ways as needed.
  • the processor may process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input/output device (such as a display device coupled to an interface).
  • an external input/output device such as a display device coupled to an interface.
  • multiple processors and/or multiple buses can be used with multiple memories and multiple memories.
  • multiple electronic devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).
  • a processor 501 is taken as an example.
  • the memory 502 is a non-transitory computer-readable storage medium provided by this application.
  • the memory stores instructions executable by at least one processor, so that the at least one processor executes the text processing method based on natural language and knowledge graph provided by this application.
  • the non-transitory computer-readable storage medium of the present application stores computer instructions, and the computer instructions are used to make the computer execute the text processing method based on the natural language and knowledge graph provided by the present application.
  • the memory 502 can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as the programs corresponding to the text processing method based on natural language and knowledge graph in the embodiment of this application.
  • Instructions/modules for example, the receiving module 11, the acquiring module 12, the processing module 13, and the training module 14 shown in FIG. 3 and FIG. 5).
  • the processor 501 executes various functional applications and data processing of the server by running the non-transient software programs, instructions, and modules stored in the memory 502, that is, realizing the text processing method based on natural language and knowledge graph in the above method embodiment .
  • the memory 502 may include a storage program area and a storage data area.
  • the storage program area may store an operating system and an application program required by at least one function; the storage data area may store data created according to the use of the electronic device of XXX.
  • the memory 502 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices.
  • the memory 502 may optionally include memories remotely provided with respect to the processor 501, and these remote memories may be connected to the electronic device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the electronic device of the text processing method based on natural language and knowledge graph may further include: an input device 503 and an output device 505.
  • the processor 501, the memory 502, the input device 503, and the output device 505 may be connected by a bus or in other ways. In FIG. 5, the connection by a bus is taken as an example.
  • the input device 503 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as touch screen, keypad, mouse, track pad, touch pad, indicator stick, one or more Input devices such as mouse buttons, trackballs, joysticks, etc.
  • the output device 505 may include a display device, an auxiliary lighting device (for example, LED), a tactile feedback device (for example, a vibration motor), and the like.
  • the display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.
  • Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, the programmable processor It can be a dedicated or general-purpose programmable processor that can receive data and instructions from the storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device. An output device.
  • machine-readable medium and “computer-readable medium” refer to any computer program product, device, and/or device used to provide machine instructions and/or data to a programmable processor ( For example, magnetic disks, optical disks, memory, programmable logic devices (PLD)), including machine-readable media that receive machine instructions as machine-readable signals.
  • machine-readable signal refers to any signal used to provide machine instructions and/or data to a programmable processor.
  • the systems and techniques described here can be implemented on a computer that has: a display device for displaying information to the user (for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) ); and a keyboard and a pointing device (for example, a mouse or a trackball) through which the user can provide input to the computer.
  • a display device for displaying information to the user
  • LCD liquid crystal display
  • keyboard and a pointing device for example, a mouse or a trackball
  • Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (for example, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, voice input, or tactile input) to receive input from the user.
  • the systems and technologies described herein can be implemented in a computing system that includes back-end components (for example, as a data server), or a computing system that includes middleware components (for example, an application server), or a computing system that includes front-end components (for example, A user computer with a graphical user interface or a web browser, through which the user can interact with the implementation of the system and technology described herein), or includes such back-end components, middleware components, Or any combination of front-end components in a computing system.
  • the components of the system can be connected to each other through any form or medium of digital data communication (for example, a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.
  • the computer system can include clients and servers.
  • the client and server are generally far away from each other and usually interact through a communication network.
  • the relationship between the client and the server is generated by computer programs that run on the corresponding computers and have a client-server relationship with each other.
  • the embodiment of the present application also provides a joint learning model training method, the joint learning model includes a knowledge graph learning layer and a natural language learning layer, and the method includes: training the natural language learning layer with the knowledge graph learning layer, The natural language learning layer after training is used to improve the knowledge graph learning layer.
  • the electronic device after receiving the text processing request input by the user, the electronic device inputs the predicted object in the text to the pre-trained joint learning model to learn the semantic representation of the predicted object, and the semantic representation is joint
  • the learning model combines knowledge graph representation learning and natural language representation learning. After that, the electronic device processes the text according to the semantic representation.
  • the semantic representation obtained by the electronic device using the joint learning model is obtained by the joint learning model combining knowledge graph representation learning and natural language representation learning, it combines knowledge graph learning representation and natural language learning representation, compared to Using only knowledge graph representation learning or natural language representation learning to learn the semantic representation of the predicted object, the joint learning model considers more and more comprehensive factors, so the accuracy of semantic representation can be improved, and the accuracy of text processing can be improved.
  • the joint learning model by jointly modeling the knowledge graph representation learning and natural language representation learning processes, it is possible to use the relationship between the elements composed of the knowledge graph and the elements composed of the natural language to learn better The semantic representation of the knowledge graph and the semantic representation of natural language.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • Animal Behavior & Ethology (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Machine Translation (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种基于自然语言处理和知识图谱的文本处理方法及装置，涉及人工智能技术的深度领域。具体实现方案为：电子设备使用联合学习模型得到的语义表示，而该联合学习模型结合知识图谱表示学习和自然语言表示学习得到的，其结合了知识图谱学习表示和自然语言学习表示，相较于仅利用知识图谱表示学习或自然语言表示学习学习预测对象的语义表示，联合学习模型考虑的因素更多更全面，因此可以提高语义表示的准确性，进而提高文本处理的准确性。

Description

基于自然语言和知识图谱的表示学习方法及装置
本申请要求于2019年12月17日提交中国专利局、申请号为201911297702X、申请名称为“基于自然语言和知识图谱的文本处理方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及人工智能(Artificial Intelligence，AI)技术领域，尤其涉及一种基于自然语言和知识图谱的表示学习方法及装置。
背景技术
目前，为了从海量数据中获取有价值的信息，知识图谱(knowledge graph，KG)应运而生。同时，自然语言作为人们用以交流和思维的工具和载体，自然语言的字符和词语的表示，是基于深度学习的语言处理类人工智能任务的基本处理对象。
通常情况下，知识图谱和自然语言是两个独立的领域，该两个领域均已独立发展起了表示学习技术体系，其中，知识图谱表示学习通常以向量空间运算的形式，对知识图谱包含的节点(Point)和边(Edge)之间的共现规律进行建模，以学习知识图谱语义表示；自然语言表示学习通常以序列生成的形式，对词语或语句之间的共现规律进行建模，以学习自然语言语义表示。
然而，基于上述表示学习方法学习到的语义表示准确度差，导致上述语义表示用于文本处理时，文本处理的准确度差。
发明内容
本申请实施例提供一种基于自然语言和知识图谱的表示学习方法及装置，通过将知识图谱的表示学习和自然语义的表示学习进行结合，以学习到质量更好的语义表示，实现提高文本处理准确性的目的。
第一方面，本申请实施例提供一种基于自然语言和知识图谱的文本处理方法，包括：接收用户输入的文本处理请求，所述文本处理请求用于请求根据文本中的预测对象的语义表示处理所述文本，将所述预测对象输入至预先训练好的联合学习模型，以获取所述预测对象的语义表示，所述联合学习模型用于知识图谱表示学习和自然语言表示学习，所述语 义表示是所述联合学习模型结合所述知识图谱表示学习和所述自然语言表示学习得到的，根据所述语义表示处理所述文本。采用该种方案，
一种可行的设计中，所述接收用户输入的文本处理请求之前，还包括：对训练样本进行训练以得到所述联合学习模型，所述联合学习模型包括自然语言学习层、联合学习关联层和知识图谱学习层，所述联合学习关联层用于关联所述知识图谱学习层和所述自然语言学习层。采用该种方案，
一种可行的设计中，所述对训练样本进行训练以得到所述联合学习模型，包括：在所述自然语言学习层确定所述训练样本中目标样本的邻居样本，在所述联合学习关联层根据所述邻居样本，确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重，根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示，根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果。采用该种方案，
一种可行的设计中，所述在所述联合学习关联层确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重，包括：对于所述知识图谱学习层中的每一个实体，在所述联合学习关联层确定所述目标训练样本相对于所述实体的M个权重，所述M≥1且为整数。采用该种方案，
一种可行的设计中，所述根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示，包括：对于所述知识图谱学习层中的每一个实体，根据所述实体对应的权重对所述实体的知识图谱语义表示进行处理，得到多个处理后的知识图谱语义表示，根据所述多个处理后的知识图谱语义表示，确定所述目标训练样本的知识图谱语义表示。采用该种方案，
一种可行的设计中，所述根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果之后，还包括：根据所述训练结果优化所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。采用该种方案，
一种可行的设计中，所述根据所述训练结果优化所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示，包括：判断所述训练结果是否正确，若所述训练结果正确，则生成激励信号，根据所述激励信号增强所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示，若所述训练结果错误，则生成惩罚信号，根据所述惩罚信号调整所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。采用该种方案，
第二方面，本申请实施例提供一种基于自然语言和知识图谱的文本处理装置，包括：接收模块，用于接收用户输入的文本处理请求，所述文本处理请求用于请求根据文本中的预测对象的语义表示处理所述文本；
获取模块，用于将所述预测对象输入至预先训练好的联合学习模型，以获取所述预测对象的语义表示，所述联合学习模型用于知识图谱表示学习和自然语言表示学习，所述语义表示是所述联合学习模型结合所述知识图谱表示学习和所述自然语言表示学习得到的；
处理模块，用于根据所述语义表示处理所述文本。
一种可行的设计中，上述的装置还包括：
训练模块，用于在所述接收模块接收用户输入的文本处理请求之前，对训练样本进行训练以得到所述联合学习模型，所述联合学习模型包括自然语言学习层、联合学习关联层和知识图谱学习层，所述联合学习关联层用于关联所述知识图谱学习层和所述自然语言学习层。
一种可行的设计中，所述训练模块，用于在所述自然语言学习层确定所述训练样本中目标样本的邻居样本，在所述联合学习关联层根据所述邻居样本，确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重，根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示，根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果。
一种可行的设计中，所述训练模块，在所述联合学习关联层确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重时，对于所述知识图谱学习层中的每一个实体，在所述联合学习关联层确定所述目标训练样本相对于所述实体的M个权重，所述M≥1且为整数。
一种可行的设计中，所述训练模块，在根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示时，对于所述知识图谱学习层中的每一个实体，根据所述实体对应的权重对所述实体的知识图谱语义表示进行处理，得到多个处理后的知识图谱语义表示，根据所述多个处理后的知识图谱语义表示，确定所述目标训练样本的知识图谱语义表示。
一种可行的设计中，所述训练模块，在根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果之后，还用于根据所述训练结果优化所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
一种可行的设计中，所述训练模块，具体用于判断所述训练结果是否正确；若所述训练结果正确，则生成激励信号，根据所述激励信号增强所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示；若所述训练结果错误，则生成惩罚信号，根据所述惩罚信号调整所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
第三方面、本申请实施例提供一种电子设备，包括：
至少一个处理器；以及
与所述至少一个处理器通信连接的存储器；其中，
所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行第一方面或第一方面任意可能实现的方法。
第四方面，本申请实施例提供一种包含指令的计算机程序产品，当其在电子设备上运行时，使得电子设备计算机执行上述第一方面或第一方面的各种可能的实现方式中的方法。
第五方面，本申请实施例提供一种存储介质，所述存储介质中存储有指令，当其在电子设备上运行时，使得电子设备执行如上述第一方面或第一方面的各种可能的实现方式中的方法。
第六方面，本申请实施例提供一种面向深度学习训练任务的处理器显存优化方法，包括：第一处理器确定第一计算单元的计算结果传输至第二计算单元的路径，所述第一计算单元和所述第二计算单元包含于所述第一处理器中，所述第一计算单元和所述第二计算单元之间存在至少一个中间计算单元，所述第一处理器通过所述路径向所述第二计算单发送所述第一计算单元的计算结果。
上述申请中的一个实施例具有如下优点或有益效果：电子设备接收到用户输入的文本处理请求后，将文本中的预测对象输入至预先训练好的联合学习模型以学习到预测对象的语义表示，该语义表示是联合学习模型结合知识图谱表示学习和自然语言表示学习得到的，之后，电子设备根据语义表示处理文本。该过程中，由于电子设备使用联合学习模型得到的语义表示，是该联合学习模型结合知识图谱表示学习和自然语言表示学习得到的，其结合了知识图谱学习表示和自然语言学习表示，相较于仅利用知识图谱表示学习或自然语言表示学习学习预测对象的语义表示，联合学习模型考虑的因素更多更全面，因此可以提高语义表示的准确性，进而提高文本处理的准确性。
上述可选方式所具有的其他效果将在下文中结合具体实施例加以说明。
附图说明
附图用于更好地理解本方案，不构成对本申请的限定。其中：
图1是本申请实施例提供的一种基于自然语言和知识图谱的文本处理方法的流程图；
图2是本申请实施例提供的基于自然语言和知识图谱的文本处理方法所适用的联合学习模型的结构示意图；
图3为本申请实施例提供的基于自然语言和知识图谱的文本处理装置的结构示意图；
图4为本申请实施例提供的另一种基于自然语言和知识图谱的文本处理装置的结构示意图；
图5是用来实现本申请实施例的基于自然语言和知识图谱的文本处理方法的电子设备的框图。
具体实施方式
以下结合附图对本申请的示范性实施例做出说明，其中包括本申请实施例的各种细节以助于理解，应当将它们认为仅仅是示范性的。因此，本领域普通技术人员应当认识到，可以对这里描述的实施例做出各种改变和修改，而不会背离本申请的范围和精神。同样，为了清楚和简明，以下的描述中省略了对公知功能和结构的描述。
一般来说，知识图谱和自然语言处理为两个独立的领域。该两个领域各自发展独立的表示学习体系，各领域的表示学习技术各自遵循不同的建模方法和优化目标，并没有交叉融合。知识图谱表示学习通常以向量空间运算的形式，对节点和边之间的共现规律进行建模，而自然语言表示学习通常以序列生成的形式，对词语或语句之间的共现规律进行建模。
虽然目前在自然语言领域出现了借助命名实体信息改进语言表示学习的方法，该方法视为在自然语言表示学习中引入知识的初步版本，但是该方法采用多层自注意神经网络为学习模型，并采用完型填空、相邻句对判断等词语、语句共现类任务为学习目标。其引入知识的具体方式是：依据命名实体词典对自然语言语句中对应的词语序列捆绑对待，作为统一的处理对象参与语言表示学习过程，所谓捆绑对待，是指若一个词语在命名实体词典中有对应的实体，则将该词语作为一个整体不分开，例如，为该词语加个括号，表示该词语是个实体，不能继续进行拆分。如此一来，可以达到借助命名实体词典知识改进自然语言表示学习的效果。其中，命名实体词典中记录了多个实体，可以视为实体列表，其可以是根据知识图谱得到的，也可以通过搜集实体得到。
上述借助命名实体词典知识改进自然语言表示学习，是非常初步的利用外部知识的方式。但是，该方法能够借助外部知识辅助自然语言表示学习，无法反过来借助自然语言辅助外部知识图谱表示学习。
站在自然语言表示学习的角度，上述借助命名实体词典知识改进自然语言表示学习的方法也存在两个缺陷：缺陷一、在知识使用范围方面，仅通过命名实体词典判断一个词或一个字是不是存在对应的实体。具体来说，知识图谱不仅包含海量的节点，还包含边，以及节点和边构成的拓扑结构，而上述的方法仅用到了一个词或字是否是实体这一个信息；缺陷二、在知识使用方式方面，上述的方法以自然语言表示学习为基本框架，在此基础上依据命名实体词典提供的实体列表判断一个词或字是否是实体，该种方式的信息流动是单向的，从外部知识即命名实体词典到自然语言，无法有效利用自然语言和外部知识之间的 双向对应规则。
有鉴于此，本申请实施例提供一种基于自然语言和知识图谱的文本处理方法及装置，通过将知识图谱的表示学习和自然语义的表示学习进行结合，以学习到质量更好的语义表示。
下面，对本申请实施例涉及的名词进行解释说明。
首先，知识图谱。
本申请实施例中，知识图谱由节点和边以图的形式构成，是知识的结构化表示形式。知识图谱的节点表示的信息包括但不限于实体、概念、兴趣和事件等分别对应于概念图谱、兴趣图谱和时间图谱等具体的知识图谱类型；相应的，边可以表示的信息包括但不限于属性、从属、时序和因果等。知识图谱的节点和边的语义表示，可以用于知识库补全、知识库问答、智能推荐、事件分析、语言理解、机器翻译等人工智能任务。
其次，自然语言学习。
本申请实施例中，自然语言由字符或词语以序列的方式构成，是人们用于交流和思维的工具和载体，可通过文本处理用于认知智能任务，如阅读理解、智能问答、机器翻译以及自动写作等。
本申请实施例中，电子设备通过将知识图谱的表示学习和自然语言的表示学习过程进行联合建模，建模过程中引入构成知识图谱的实体和构成自然语言的词语的关联关系，从而得到联合学习模型，基于该联合学习模型学习到更好的知识图谱语义表示和自然语言语义表示。其中，电子设备例如为服务器或终端设备等。
图1是本申请实施例提供的一种基于自然语言和知识图谱的文本处理方法的流程图，本实施例是从电子设备的角度进行说明，本实施例包括：
101、接收用户输入的文本处理请求，所述文本处理请求用于请求根据文本中的预测对象的语义表示处理所述文本。
示例性的，电子设备上预先加载联合学习模型，当需要进行文本处理时，用户通过点击操作、触摸操作或语音输入等方式向电子设备输入处理请求，电子设备接收并识别该文本处理请求。
例如，当关联学习模型用于辅助认知智能任务时，电子设备接收到文本处理请求后，对待处理文本进行划词等处理，将文本拆分成一个个的词语或字，该一个个的词语或字即为预测对象。
再如，当关联学习模型用于辅助知识图谱任务时，电子设备接收到文本处理请求后，识别出待处理文本包含的实体，该些实体即为预测对象。
102、将所述预测对象输入至预先训练好的联合学习模型，以得到所述预测对象的语义表示，所述联合学习模型用于知识图谱表示学习和自然语言表示学习，所述语义表示是所述联合学习模型结合所述知识图谱表示学习和所述自然语言表示学习得到的。
示例性的，对于每一个预测对象，电子设备将该预测对象输入至联合学习模型，从而得到该预测对象的语义表示。例如，当预测对象为词语或字时，关联学习模型输出的是自然语言语义表示；再如，当预测对象为实体时，关联学习模型输出的是知识图谱语义表示。
本实施例中，关联学习模型输出的语义表示，不论是知识图谱语义表示还是自然语言语义表示，均是联合学习模型预先结合知识图谱表示学习和自然语言表示学习得到的。也就是说，电子设备在训练联合学习模型时，对于一个为词语或字的训练样本，训练过程中除了考虑了该训练样本的邻居训练样本外，还考虑了该训练样本的知识图谱信息。而且，该训练样本的训练结果还被用于调整知识图谱中各实体的知识图谱语义表示，使得知识图谱中每个实体的语义表示除了考虑知识图谱中其他实体外，还考虑了自然语言处理中训练样本的训练结果。
103、根据所述语义表示处理所述文本。
示例性的，电子设备得到待处理文本中每个预测对象的语义表示后，根据该些语义表示处理文本。例如，当关联学习模型用于辅助认知智能任务时，电子设备根据语义表示进行阅读理解、智能问答、机器翻译或自动写作等。再如，当关联学习模型用于辅助知识图谱任务时，电子设备根据语义表示进行知识库补全、知识库推理、知识库问答等。
本申请实施例提供的基于自然语言和知识图谱的文本处理方法，电子设备接收到用户输入的文本处理请求后，将文本中的预测对象输入至预先训练好的联合学习模型以学习到预测对象的语义表示，该语义表示是联合学习模型结合知识图谱表示学习和自然语言表示学习得到的，之后，电子设备根据语义表示处理文本。该过程中，由于电子设备使用联合学习模型得到的语义表示，是该联合学习模型结合知识图谱表示学习和自然语言表示学习得到的，其结合了知识图谱学习表示和自然语言学习表示，相较于仅利用知识图谱表示学习或自然语言表示学习学习预测对象的语义表示，联合学习模型考虑的因素更多更全面，因此可以提高语义表示的准确性，进而提高文本处理的准确性。
上述实施例中，电子设备接收用户输入的文本处理请求之前，还对训练样本进行训练以得到上述的联合学习模型，上述的联合学习模型包括的三个核心模块为自然语言学习层、联合学习关联层和知识图谱学习层，所述联合学习关联层用于关联所述知识图谱学习层和所述自然语言学习层。示例性的，可参见图2，图2是本申请实施例提供的基于自然语言和知识图谱的文本处理方法所适用的联合学习模型的结构示意图。
请参照图2，联合学习模型包括自然语言学习层、联合学习关联层和知识图谱学习层，它们分别对应到相应的子神经网络模型，知识图谱学习层和自然语言学习层通过联合学习关联层配合，配合机制可以采用多种不同的神经网络机制。另外，联合学习模型还包括其他任务的学习模块，用于驱动知识图谱表示学习和自然语言表示学习。
请参照图2，实线框所示为自然语言学习层，可采用各种主流的神经网络语言模型，如循环神经网络(Recurrent Neural Network，RNN)模型、长短记忆网络(Long Short-Term Memor，LSTM)模型、转换器(transformer)模型、来自转换器的双向编码器特征(Bidirectional Encoder Representations from Transformers，BERT)以及知识增强的语义表示模型(Enhanced Representation through kNowledge IntEgration，ERNIE)等。自然语言学习层的本质是在语句和词语之间进行信息传递，对词语之间的关联关系进行建模，从而起到学习词语表示的效果。
请参照图2，虚线框所示为知识图谱学习层，可以采用图神经网络以更好的反应知识图谱的拓扑结构。另外，知识图谱学习层也可以不采用任何模型，而是仅仅把知识图谱当做一个由节点和边构成的列表。
请参照图2，点划线所示为联合学习关联层，其由一种或多种读写机制构成，联合学习关联层的作用是在词语和知识之间建立关联，实现自然语言语句和知识图谱知识库之间的信息传递。联合学习关联层的读写机制的输入通常为自然语言学习层中的词语或字，以及知识图谱学习层中的知识图谱，其中，词语或字是目标训练样本的邻居样本，例如，训练样本包括n个词语或字，如图中的词语1(word 1，w1)～wn，假设目标训练样本是w3，则输入时w3的邻居样本，如w1、w2、w4、w5等，以及知识图谱，联合学习关联层根据输入得到目标训练样本的知识图谱语义表示并输出。之后，自然语言学习层根据目标训练样本的知识图谱语义表示以及邻居训练样本，得到目标训练样本的训练结果。
本实施例中，实现训练得到用于知识图谱表示学习和自然语言表示学习的联合学习模型的目的。
上述实施例中，电子设备在对训练样本进行训练得到联合学习模型时，可以借助知识图谱学习层改进自然语言学习层，之后，可以利用自然语言学习层改进知识图谱学习层。下面，对该两个方面分别进行详细说明。
首先，借助知识图谱学习层改进自然语言学习层。
一种可行的设计中，电子设备对训练样本进行训练以得到所述联合学习模型时，在所述自然语言学习层确定所述训练样本中目标样本的邻居样本，在所述联合学习关联层根据所述邻居样本，确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重，根据 所述各实体的权重，确定所述目标训练样本的知识图谱语义表示，根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果。
示例性的，请参照图2，训练样本包括w1～wn，以w3为目标训练样本为例，传统的训练过程中，当w3为目标训练样本时，默认w3之外的其他训练训练样本的训练结果是已知的，需要利用该些已知的训练结果预测w3的训练结果，预测过程中，电子设备主要利用了w3的邻居样本，如w1、w2、w4、w5等的训练结果。本申请实施例中，当w3为目标训练样本时，默认w3之外的其他训练训练样本的训练结果是已知的，电子设备在预测w3的训练结果时，除了利用了w3的邻居样本w1、w2、w4、w5等的训练结果外，还需要考虑w3的知识图谱语言表示。
本申请实施例中，w3的知识谱图语言表示可以根据w3的邻居样本等获得。例如，电子设备将w3的邻居样本w1、w2、w4、w5等进行整合得到整合信息，将该整合信息输入至联合学习关联层，在联合学习关联层确定该整合信息相对于知识图谱学习层中每个实体(entity，e)，如ei、ej、ek和el的权重，根据该5个权重和各实体的知识图谱语义表示，确定目标训练样本的知识图谱语义表示；再如，对于w3的每一个邻居样本，电子设备在联合学习关联层分别确定该邻居样本相对于知识图谱学习层中每个实体(entity，e)，如ei、ej、ek和el的权重，得到20个权重，根据该20个权重和各实体的知识图谱语义表示，确定目标训练样本的知识图谱语义表示。
本实施例中，借助知识图谱改进自然语言表示学习，站在自然语言表示学习的角度，该方法的知识使用范围更广泛，且知识使用方式更有效，通过表示学习联合建模，该方法能够综合利用节点、边以及由节点和边构成的拓扑结构的知识，改进自然语言的表示学习效果。
上述实施例中，联合学习关联层可以从M个不同的角度为目标训练对象确定出知识图谱语义表示。此时，电子设备在所述联合学习关联层确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重时，对于所述知识图谱学习层中的每一个实体，在所述联合学习关联层确定所述目标训练样本相对于所述实体的M个权重，所述M≥1且为整数。
示例性的，以将目标训练样本的邻居样本整合为一条整合信息为例，该整合信息到达联合学习关联层后，联合学习关联层从M个角度确定该整合信息相对于知识图谱学习层中每个实体(entity，e)，如ei、ej、ek和el的权重，得到5M个权重，根据该5M个权重和各实体的知识图谱语义表示，确定目标训练样本的知识图谱语义表示。
为实现从M个不同的角度为目标训练对象确定出知识图谱语义表示，可以借助联合学习关联层的读写机制实现。示例性的，联合学习关联层由一种或多种读写机制构成，读写 机制最常用的就是注意力机制，联合学习关联层可以采用一个注意力模块，也可以采用多个注意力模块，图2所示为采用M个注意力模块。其中，注意力模块也可以称之为读写(read write)模块。当采用多个注意力模块时，利用知识图谱学习到的数据份数，如相对于每个实体的权重的个数等于注意力模块的个数相同。例如，对于知识图谱学习层采用图神经网络的情形，图神经网络要运行M次，从而获得M个版本的知识图谱数据供注意力机制访问。再例如，对于知识图谱学习层不采用任何神经网络模块的情形，只需要维护M份相同的数据结构即节点和边的列表即可，这样，通过M个版本的注意力机制，这M份相同的数据结构会学习到不同角度的知识图谱语义表示。
本实施例中，可以通过多个注意力机制对知识图谱进行不同学习，以得到目标训练样本的多个角度的知识图谱语义表示，利用多个角度的知识图谱语义表示和多个邻居样本预测目标训练样本，提高预测的准确性。
需要说明的是，本申请实施例中训练样本的数量是及其庞大的，一个目标训练样本的邻居训练样本可以为4个甚至更多，同理，知识图谱学习表示层中实体的数量也是特别多的，上述的ei、ej、ek和el仅是举例，实际实现中，实体的数量是数以万计甚至更多。
电子设备得到目标训练样本的知识图谱语义表示后，根据该知识图谱语义表示和目标训练样本的邻居样本，确定目标样本的训练结果，即对目标训练样本进行预测。例如，图2中，已知w1、w2、w4、w5以及w3的知识图谱语义表示，预测w3是哪个词语或哪个字。
需要说明的是，上述是以已知w1、w2、w4、w5，未知w3为例对本申请实施例进行详细说明的，然而，本申请实施例并不以此为限制，其他可行的实现方式中，包括但不限于：a)已知前面的多个词语，预测当前的词语，例如，已知w1和w2，预测w3；b)已知前面和后面的多个词语，预测当前词语，例如，已知w1、w2、w4、w5，预测w3；c)预测两个语句即两个词语序列是否是相邻语句；d)已知前面的多个词语，预测当前的多个词语，即语句，例如，已知w1、w2、w3，预测w4和w5。
上述实施例中，电子设备获得目标训练对象相对于知识图谱学习层中各实体的权重后，根据各实体的权重，确定目标训练样本的知识图谱语义表示时，对于所述知识图谱学习层中的每一个实体，根据所述实体对应的权重对所述实体的知识图谱语义表示进行处理，得到多个处理后的知识图谱语义表示，根据所述多个处理后的知识图谱语义表示，确定所述目标训练样本的知识图谱语义表示。
示例性的，以联合学习关联层采用一个注意力模块为例，电子设备将w3的邻居样本w1、w2、w4、w5等进行整合得到整合信息，将该整合信息输入至联合学习关联层，在联 合学习关联层确定该整合信息相对于知识图谱学习层中每个实体(entity，e)，如ei、ej、ek和el的权重，将每个实体的知识图谱语义表示与对应的权重相乘，得到5个处理后的知识图谱语义表示，然后对该5个处理后的知识图谱语义表示进行求和运算，从而得到目标训练样本的知识图谱语义表示。另外，除了乘法和加法运算外，还可以采用其他的运算方式，本申请实施例并不限制。
本实施例中，实现确定出目标训练样本的知识图谱语义表示的目的。
其次，借助自然语言学习层改进知识图谱学习层。
一种可行的设计中，电子设备根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果之后，还根据所述训练结果优化所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
示例性的，本申请实施例中，借助知识图谱学习表示层改进自然语言学习层后，还可以进一步的根据目标训练样本的训练结果，优化知识图谱学习层中各实体的语义表示。
本实施例中，实现借助自然语言学习层改进知识图谱学习层的目的。站在知识图谱标识学习的角度，该方法能够借助自然语言表示学习的训练过程，对知识图谱标识学习过程进行协调和优化，对知识图谱表示学习自身的学习过程进行辅助和补充，从而有助于学习到更好的知识图谱表示。
当借助自然语言学习层改进知识图谱学习层时，电子设备判断目标训练对象的训练结果是否正确，若训练结果正确，则生成激励信号，根据所述激励信号增强所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示；若所述训练结果错误，则生成惩罚信号，根据所述惩罚信号削弱所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
示例性的，再请参照图2，利用邻居样本w1、w2、w4、w5和w3的知识图谱语言表示预测出w3后，电子设备判断w3的训练结果是否正确，即是否正确预测出w3是哪个词语或哪个字，若训练结果正确，则生成激励信号并反馈给知识图谱学习层，使得知识图谱学习层增强实体ei、ej、ek和el的知识图谱语义表示；若训练结果不正确，则说明知识图谱学习层的实体ei、ej、ek和el的知识图谱语义表示可能错误，此时，电子设备生成惩罚信号并反馈给知识图谱学习层，使得知识图谱学习层调整知识图谱学习模型包含的多个实体中每个实体的语义表示。
本实施例中，实现借助自然语言学习层改进知识图谱学习层的目的。
下面，对如何驱动联合学习模型的训练进行详细说明。
一种实现方式中，可以在自然语言学习层的驱动下进行整个联合学习模型的学习。
示例性的，自然语言学习层采用常用的语言模型学习策略，包括但不限于：a)已知前面的多个词语，预测当前的词语，例如，已知w1和w2，预测w3；b)已知前面和后面的多个词语，预测当前词语，例如，已知w1、w2、w4、w5，预测w3；c)预测两个语句即两个词语序列是否是相邻语句；d)已知前面的多个词语，预测当前的多个词语，即语句，例如，已知w1、w2、w3，预测w4和w5。自然语言学习层以上述语言模型学习策略进行驱动，进而通过联合学习关联层的桥梁作用，进一步驱动知识图谱学习层的学习，从而能够同步学习到自然语言的词语的自然语义表示，以及知识图谱的节点和边的知识图谱语义表示。另外，当用户仅仅将联合学习模型作为语言模型使用的时候，知识图谱学习层可以视为自然语言学习层的外挂知识库，整个联合学习模型可以视为一个知识增强的语言模型。
另一种实现方式中，也可以采用上传任务驱动联合学习模型的训练。其中，上层任务指自然语言学习层的认知智能任务，如阅读理解、问题系统、机器翻译等。具体实现时，可以通过将自然语言学习层的词语的表示作为最上层驱动任务的输入来实现。这样一来，在最上层认知任务的驱动下，上层任务本身的参数、自然语言学习层的参数、联合学习关联层的参数以及知识图谱学习层的参数，都能够得到同步的学习。在此基础上，还可以进一步的引入知识图谱类的任务协助驱动整个联合学习模型的学习，如知识库补全、知识库问答以及其他知识推理任务，进行知识图谱表示部分的优化学习。这种情况下，可以采用多任务学习或者多目标学习的策略，来协同优化知识图谱的优化目标和认知智能任务的优化目标。
上述实施例中，通过将知识图谱表示学习和自然语言表示学习过程进行联合建模，能够利用知识图谱构成的元素和自然语言构成的元素之间的关联关系，学习到更好的知识图谱语义表示和自然语言语义表示。
需要说明的是，上述实施例中，知识图谱不仅包含传统意义上的事实知识图谱，还包括概念图谱、兴趣点图谱、事件图谱等特种知识图谱类型。相应的，知识图谱中的节点包括但不限于实体、概念、兴趣和时间等类型的信息，边包括但不限于属性、从属、时序和因果等类型的关联。
上述介绍了本申请实施例提到的基于自然语言和知识图谱的文本处理的具体实现，下述为本申请装置实施例，可以用于执行本申请方法实施例。对于本申请装置实施例中未披露的细节，请参照本申请方法实施例。
图3为本申请实施例提供的基于自然语言和知识图谱的文本处理装置的结构示意图。该装置可以集成在电子设备中或通过电子设备实现，电子设备可以终端设备或服务器等。如图3所示，在本实施例中，该基于自然语言和知识图谱的文本处理装置100可以包括：
接收模块11，用于接收用户输入的文本处理请求，所述文本处理请求用于请求根据文本中的预测对象的语义表示处理所述文本；
获取模块12，用于将所述预测对象输入至预先训练好的联合学习模型，以获取所述预测对象的语义表示，所述联合学习模型用于知识图谱表示学习和自然语言表示学习，所述语义表示是所述联合学习模型结合所述知识图谱表示学习和所述自然语言表示学习得到的；
处理模块13，用于根据所述语义表示处理所述文本。
图4为本申请实施例提供的另一种基于自然语言和知识图谱的文本处理装置的结构示意图，请按照图4，上述的基于自然语言和知识图谱的文本处理装置100还包括：
训练模块14，用于在所述接收模块11接收用户输入的文本处理请求之前，对训练样本进行训练以得到所述联合学习模型，所述联合学习模型包括自然语言学习层、联合学习关联层和知识图谱学习层，所述联合学习关联层用于关联所述知识图谱学习层和所述自然语言学习层。
一种可行的设计中，所述训练模块14，用于在所述自然语言学习层确定所述训练样本中目标样本的邻居样本，在所述联合学习关联层根据所述邻居样本，确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重，根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示，根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果。
一种可行的设计中，所述训练模块14，在所述联合学习关联层确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重时，对于所述知识图谱学习层中的每一个实体，在所述联合学习关联层确定所述目标训练样本相对于所述实体的M个权重，所述M≥1且为整数。
一种可行的设计中，所述训练模块14，在根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示时，对于所述知识图谱学习层中的每一个实体，根据所述实体对应的权重对所述实体的知识图谱语义表示进行处理，得到多个处理后的知识图谱语义表示，根据所述多个处理后的知识图谱语义表示，确定所述目标训练样本的知识图谱语义表示。
一种可行的设计中，所述训练模块14，在根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果之后，还用于根据所述训练结果优化所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
一种可行的设计中，所述训练模块14，具体用于判断所述训练结果是否正确；若所述训练结果正确，则生成激励信号，根据所述激励信号增强所述知识图谱学习层包含的多个 实体中每个实体的知识图谱语义表示；若所述训练结果错误，则生成惩罚信号，根据所述惩罚信号调整所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
本申请实施例提供的装置，可用于如上实施例中电子设备执行的方法，其实现原理和技术效果类似，在此不再赘述。
根据本申请的实施例，本申请还提供了一种电子设备和一种可读存储介质。
图5是用来实现本申请实施例的基于自然语言和知识图谱的文本处理方法的电子设备的框图。电子设备旨在表示各种形式的数字计算机，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本申请的实现。
如图5所示，该电子设备包括：一个或多个处理器501、存储器502，以及用于连接各部件的接口，包括高速接口和低速接口。各个部件利用不同的总线互相连接，并且可以被安装在公共主板上或者根据需要以其它方式安装。处理器可以对在电子设备内执行的指令进行处理，包括存储在存储器中或者存储器上以在外部输入/输出装置(诸如，耦合至接口的显示设备)上显示GUI的图形信息的指令。在其它实施方式中，若需要，可以将多个处理器和/或多条总线与多个存储器和多个存储器一起使用。同样，可以连接多个电子设备，各个设备提供部分必要的操作(例如，作为服务器阵列、一组刀片式服务器、或者多处理器系统)。图5中以一个处理器501为例。
存储器502即为本申请所提供的非瞬时计算机可读存储介质。其中，所述存储器存储有可由至少一个处理器执行的指令，以使所述至少一个处理器执行本申请所提供的基于自然语言和知识图谱的文本处理方法。本申请的非瞬时计算机可读存储介质存储计算机指令，该计算机指令用于使计算机执行本申请所提供的基于自然语言和知识图谱的文本处理方法。
存储器502作为一种非瞬时计算机可读存储介质，可用于存储非瞬时软件程序、非瞬时计算机可执行程序以及模块，如本申请实施例中的基于自然语言和知识图谱的文本处理方法对应的程序指令/模块(例如，附图3和图5所示的接收模块11、获取模块12、处理模块13和训练模块14)。处理器501通过运行存储在存储器502中的非瞬时软件程序、指令以及模块，从而执行服务器的各种功能应用以及数据处理，即实现上述方法实施例中的基于自然语言和知识图谱的文本处理方法。
存储器502可以包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、 至少一个功能所需要的应用程序；存储数据区可存储根据XXX的电子设备的使用所创建的数据等。此外，存储器502可以包括高速随机存取存储器，还可以包括非瞬时存储器，例如至少一个磁盘存储器件、闪存器件、或其他非瞬时固态存储器件。在一些实施例中，存储器502可选包括相对于处理器501远程设置的存储器，这些远程存储器可以通过网络连接至电子设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
基于自然语言和知识图谱的文本处理方法的电子设备还可以包括：输入装置503和输出装置505。处理器501、存储器502、输入装置503和输出装置505可以通过总线或者其他方式连接，图5中以通过总线连接为例。
输入装置503可接收输入的数字或字符信息，以及产生与电子设备的用户设置以及功能控制有关的键信号输入，例如触摸屏、小键盘、鼠标、轨迹板、触摸板、指示杆、一个或者多个鼠标按钮、轨迹球、操纵杆等输入装置。输出装置505可以包括显示设备、辅助照明装置(例如，LED)和触觉反馈装置(例如，振动电机)等。该显示设备可以包括但不限于，液晶显示器(LCD)、发光二极管(LED)显示器和等离子体显示器。在一些实施方式中，显示设备可以是触摸屏。
此处描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电路系统、专用ASIC(专用集成电路)、计算机硬件、固件、软件、和/或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和/或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。
这些计算程序(也称作程序、软件、软件应用、或者代码)包括可编程处理器的机器指令，并且可以利用高级过程和/或面向对象的编程语言、和/或汇编/机器语言来实施这些计算程序。如本文使用的，术语“机器可读介质”和“计算机可读介质”指的是用于将机器指令和/或数据提供给可编程处理器的任何计算机程序产品、设备、和/或装置(例如，磁盘、光盘、存储器、可编程逻辑装置(PLD))，包括，接收作为机器可读信号的机器指令的机器可读介质。术语“机器可读信号”指的是用于将机器指令和/或数据提供给可编程处理器的任何信号。
为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监 视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)和互联网。
计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。
本申请实施例还提供一种联合学习模型训练方法，所述联合学习模型包括知识图谱学习层和自然语言学习层，所述方法包括：借助所述知识图谱学习层训练所述自然语言学习层，利用训练后的自然语言学习层改进所述知识图谱学习层。
根据本申请实施例的技术方案，电子设备接收到用户输入的文本处理请求后，将文本中的预测对象输入至预先训练好的联合学习模型以学习到预测对象的语义表示，该语义表示是联合学习模型结合知识图谱表示学习和自然语言表示学习得到的，之后，电子设备根据语义表示处理文本。该过程中，由于电子设备使用联合学习模型得到的语义表示，是该联合学习模型结合知识图谱表示学习和自然语言表示学习得到的，其结合了知识图谱学习表示和自然语言学习表示，相较于仅利用知识图谱表示学习或自然语言表示学习学习预测对象的语义表示，联合学习模型考虑的因素更多更全面，因此可以提高语义表示的准确性，进而提高文本处理的准确性。另外，训练联合学习模型的过程中，通过将知识图谱表示学习和自然语言表示学习过程进行联合建模，能够利用知识图谱构成的元素和自然语言构成的元素之间的关联关系，学习到更好的知识图谱语义表示和自然语言语义表示。
应该理解，可以使用上面所示的各种形式的流程，重新排序、增加或删除步骤。例如，本发申请中记载的各步骤可以并行地执行也可以顺序地执行也可以不同的次序执行，只要能够实现本申请公开的技术方案所期望的结果，本文在此不进行限制。
上述具体实施方式，并不构成对本申请保护范围的限制。本领域技术人员应该明白的 是，根据设计要求和其他因素，可以进行各种修改、组合、子组合和替代。任何在本申请的精神和原则之内所作的修改、等同替换和改进等，均应包含在本申请保护范围之内。

Claims (17)

  1. 一种基于自然语言处理和知识图谱的文本处理方法，其特征在于，包括：
    接收用户输入的文本处理请求，所述文本处理请求用于请求根据文本中的预测对象的语义表示处理所述文本；
    将所述预测对象输入至预先训练好的联合学习模型，以获取所述预测对象的语义表示，所述联合学习模型用于知识图谱表示学习和自然语言表示学习，所述语义表示是所述联合学习模型结合所述知识图谱表示学习和所述自然语言表示学习得到的；
    根据所述语义表示处理所述文本。
  2. 根据权利要求1所述的方法，其特征在于，所述接收用户输入的文本处理请求之前，还包括：
    对训练样本进行训练以得到所述联合学习模型，所述联合学习模型包括自然语言学习层、联合学习关联层和知识图谱学习层，所述联合学习关联层用于关联所述知识图谱学习层和所述自然语言学习层。
  3. 根据权利要求2所述的方法，其特征在于，所述对训练样本进行训练以得到所述联合学习模型，包括：
    在所述自然语言学习层确定所述训练样本中目标样本的邻居样本；
    在所述联合学习关联层根据所述邻居样本，确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重；
    根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示；
    根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果。
  4. 根据权利要求3所述的方法，其特征在于，所述在所述联合学习关联层确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重，包括：
    对于所述知识图谱学习层中的每一个实体，在所述联合学习关联层确定所述目标训练样本相对于所述实体的M个权重，所述M≥1且为整数。
  5. 根据权利要求3或4所述的方法，其特征在于，所述根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示，包括：
    对于所述知识图谱学习层中的每一个实体，根据所述实体对应的权重对所述实体的知识图谱语义表示进行处理，得到多个处理后的知识图谱语义表示；
    根据所述多个处理后的知识图谱语义表示，确定所述目标训练样本的知识图谱语义表示。
  6. 根据权利要求3或4所述的方法，其特征在于，所述根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果之后，还包括：
    根据所述训练结果优化所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
  7. 根据权利要求6所述的方法，其特征在于，所述根据所述训练结果优化所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示，包括：
    判断所述训练结果是否正确；
    若所述训练结果正确，则生成激励信号，根据所述激励信号增强所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示；
    若所述训练结果错误，则生成惩罚信号，根据所述惩罚信号调整所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
  8. 一种基于自然语言处理和知识图谱的文本处理装置，其特征在于，包括：
    接收模块，用于接收用户输入的文本处理请求，所述文本处理请求用于请求根据文本中的预测对象的语义表示处理所述文本；
    获取模块，用于将所述预测对象输入至预先训练好的联合学习模型，以获取所述预测对象的语义表示，所述联合学习模型用于知识图谱表示学习和自然语言表示学习，所述语义表示是所述联合学习模型结合所述知识图谱表示学习和所述自然语言表示学习得到的；
    处理模块，用于根据所述语义表示处理所述文本。
  9. 根据权利要求8所述的装置，其特征在于，还包括：
    训练模块，用于在所述接收模块接收用户输入的文本处理请求之前，对训练样本进行训练以得到所述联合学习模型，所述联合学习模型包括自然语言学习层、联合学习关联层和知识图谱学习层，所述联合学习关联层用于关联所述知识图谱学习层和所述自然语言学习层。
  10. 根据权利要求9所述的装置，其特征在于，
    所述训练模块，用于在所述自然语言学习层确定所述训练样本中目标样本的邻居样本，在所述联合学习关联层根据所述邻居样本，确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重，根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示，根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果。
  11. 根据权利要求10所述的装置，其特征在于，
    所述训练模块，在所述联合学习关联层确定所述目标训练样本相对于所述知识图谱学习层中各实体的权重时，对于所述知识图谱学习层中的每一个实体，在所述联合学习关联 层确定所述目标训练样本相对于所述实体的M个权重，所述M≥1且为整数。
  12. 根据权利要求10或11所述的装置，其特征在于，
    所述训练模块，在根据所述各实体的权重，确定所述目标训练样本的知识图谱语义表示时，对于所述知识图谱学习层中的每一个实体，根据所述实体对应的权重对所述实体的知识图谱语义表示进行处理，得到多个处理后的知识图谱语义表示，根据所述多个处理后的知识图谱语义表示，确定所述目标训练样本的知识图谱语义表示。
  13. 根据权利要求10或11所述的装置，其特征在于，
    所述训练模块，在根据所述知识图谱语义表示和所述邻居样本，确定所述目标样本的训练结果之后，还用于根据所述训练结果优化所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
  14. 根据权利要求13所述的装置，其特征在于，
    所述训练模块，具体用于判断所述训练结果是否正确；若所述训练结果正确，则生成激励信号，根据所述激励信号增强所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示；若所述训练结果错误，则生成惩罚信号，根据所述惩罚信号调整所述知识图谱学习层包含的多个实体中每个实体的知识图谱语义表示。
  15. 一种电子设备，其特征在于，包括：
    至少一个处理器；以及
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行权利要求1-7中任一项所述的方法。
  16. 一种存储有计算机指令的非瞬时计算机可读存储介质，其特征在于，所述计算机指令用于使所述计算机执行权利要求1-7中任一项所述的方法。
  17. 一种联合学习模型训练方法，其特征在于，所述联合学习模型包括知识图谱学习层和自然语言学习层，所述方法包括：
    借助所述知识图谱学习层训练所述自然语言学习层；
    利用训练后的自然语言学习层改进所述知识图谱学习层。
PCT/CN2020/095108 2019-12-17 2020-06-09 基于自然语言和知识图谱的表示学习方法及装置 WO2021120543A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
KR1020207036186A KR102524766B1 (ko) 2019-12-17 2020-06-09 자연어 및 지식 그래프 기반 표현 학습 방법 및 장치
EP20864301.5A EP3866025A4 (en) 2019-12-17 2020-06-09 METHOD AND DEVICE BASED ON A KNOWLEDGE GRAPH AND A NATURAL LANGUAGE FOR LEARNING BY REPRESENTATION
JP2020571787A JP7250052B2 (ja) 2019-12-17 2020-06-09 自然言語及び知識グラフに基づく表現学習方法及び装置
US17/124,030 US12019990B2 (en) 2019-12-17 2020-12-16 Representation learning method and device based on natural language and knowledge graph

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911297702.X 2019-12-17
CN201911297702.XA CN110727806B (zh) 2019-12-17 2019-12-17 基于自然语言和知识图谱的文本处理方法及装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/124,030 Continuation US12019990B2 (en) 2019-12-17 2020-12-16 Representation learning method and device based on natural language and knowledge graph

Publications (1)

Publication Number Publication Date
WO2021120543A1 true WO2021120543A1 (zh) 2021-06-24

Family

ID=69226014

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/095108 WO2021120543A1 (zh) 2019-12-17 2020-06-09 基于自然语言和知识图谱的表示学习方法及装置

Country Status (4)

Country Link
EP (1) EP3866025A4 (zh)
JP (1) JP7250052B2 (zh)
CN (1) CN110727806B (zh)
WO (1) WO2021120543A1 (zh)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113505244A (zh) * 2021-09-10 2021-10-15 中国人民解放军总医院 基于深度学习的知识图谱构建方法、系统、设备及介质
CN113591442A (zh) * 2021-10-08 2021-11-02 北京明略软件系统有限公司 文本生成方法、装置、电子装置及可读存储介质
CN113642319A (zh) * 2021-07-29 2021-11-12 北京百度网讯科技有限公司 文本处理方法、装置、电子设备及存储介质
CN113656709A (zh) * 2021-08-24 2021-11-16 东北大学 一种融合知识图谱和时序特征的可解释兴趣点推荐方法
CN113961679A (zh) * 2021-09-18 2022-01-21 北京百度网讯科技有限公司 智能问答的处理方法、系统、电子设备及存储介质
CN115017337A (zh) * 2022-08-03 2022-09-06 中国电子科技集团公司第五十四研究所 一种面向深度学习模型的隐语义解释方法
CN115587175A (zh) * 2022-12-08 2023-01-10 阿里巴巴达摩院(杭州)科技有限公司 人机对话及预训练语言模型训练方法、系统及电子设备

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110727806B (zh) * 2019-12-17 2020-08-11 北京百度网讯科技有限公司 基于自然语言和知识图谱的文本处理方法及装置
CN113360751A (zh) * 2020-03-06 2021-09-07 百度在线网络技术（北京）有限公司 意图识别方法、装置、设备和介质
CN111462914B (zh) * 2020-03-13 2023-07-25 云知声智能科技股份有限公司 一种实体链接方法及装置
CN111488740B (zh) * 2020-03-27 2023-12-22 北京百度网讯科技有限公司 一种因果关系的判别方法、装置、电子设备及存储介质
CN111522944B (zh) * 2020-04-10 2023-11-14 北京百度网讯科技有限公司 用于输出信息的方法、装置、设备以及存储介质
CN111611410B (zh) * 2020-06-23 2024-01-12 中国人民解放军国防科技大学 基于多层网络空间知识表示的知识处理方法和装置
CN111950303B (zh) * 2020-10-19 2021-01-08 平安科技（深圳）有限公司 医疗文本翻译方法、装置及存储介质
CN112507040B (zh) * 2020-12-21 2023-08-08 北京百度网讯科技有限公司 多元关系生成模型的训练方法、装置、电子设备及介质
WO2024007119A1 (zh) * 2022-07-04 2024-01-11 华为技术有限公司 文本处理模型的训练方法、文本处理的方法及装置
CN116312971B (zh) * 2023-05-15 2023-08-29 之江实验室 认知训练素材生成方法、认知训练方法、装置和介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017149559A1 (en) * 2016-02-29 2017-09-08 Nec Corporation An information processing system, an information processing method and a computer readable storage medium
CN108256065A (zh) * 2018-01-16 2018-07-06 智言科技（深圳）有限公司 基于关系检测和强化学习的知识图谱推理方法
CN108681544A (zh) * 2018-03-07 2018-10-19 中山大学 一种基于图谱拓扑结构和实体文本描述的深度学习方法
CN109978060A (zh) * 2019-03-28 2019-07-05 科大讯飞华南人工智能研究院（广州）有限公司 一种自然语言要素抽取模型的训练方法及装置
CN110727806A (zh) * 2019-12-17 2020-01-24 北京百度网讯科技有限公司 基于自然语言和知识图谱的文本处理方法及装置

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106649878A (zh) * 2017-01-07 2017-05-10 陈翔宇 基于人工智能的物联网实体搜索方法及系统
CN108073673B (zh) * 2017-05-15 2019-02-26 北京华宇元典信息服务有限公司 一种基于机器学习的法律知识图谱构建方法、装置、系统和介质
EP3625699A1 (en) * 2017-10-25 2020-03-25 Google LLC Natural language processing with an n-gram machine
CN110083690B (zh) * 2019-04-10 2022-05-03 华侨大学 一种基于智能问答的对外汉语口语训练方法及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017149559A1 (en) * 2016-02-29 2017-09-08 Nec Corporation An information processing system, an information processing method and a computer readable storage medium
CN108256065A (zh) * 2018-01-16 2018-07-06 智言科技（深圳）有限公司 基于关系检测和强化学习的知识图谱推理方法
CN108681544A (zh) * 2018-03-07 2018-10-19 中山大学 一种基于图谱拓扑结构和实体文本描述的深度学习方法
CN109978060A (zh) * 2019-03-28 2019-07-05 科大讯飞华南人工智能研究院（广州）有限公司 一种自然语言要素抽取模型的训练方法及装置
CN110727806A (zh) * 2019-12-17 2020-01-24 北京百度网讯科技有限公司 基于自然语言和知识图谱的文本处理方法及装置

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"Physical Review Letters", vol. 09, 1 May 2016, AMERICAN PHYSICAL SOCIETY., US, ISSN: 0031-9007, article TIAN FEI: "Joint Learning Methods for Distributed Representations of Natural Language", pages: 1 - 129, XP055821925, DOI: 10.1103/PhysRevLett.121.132301 *
See also references of EP3866025A4 *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113642319A (zh) * 2021-07-29 2021-11-12 北京百度网讯科技有限公司 文本处理方法、装置、电子设备及存储介质
CN113642319B (zh) * 2021-07-29 2022-11-29 北京百度网讯科技有限公司 文本处理方法、装置、电子设备及存储介质
CN113656709A (zh) * 2021-08-24 2021-11-16 东北大学 一种融合知识图谱和时序特征的可解释兴趣点推荐方法
CN113656709B (zh) * 2021-08-24 2023-07-25 东北大学 一种融合知识图谱和时序特征的可解释兴趣点推荐方法
CN113505244A (zh) * 2021-09-10 2021-10-15 中国人民解放军总医院 基于深度学习的知识图谱构建方法、系统、设备及介质
CN113961679A (zh) * 2021-09-18 2022-01-21 北京百度网讯科技有限公司 智能问答的处理方法、系统、电子设备及存储介质
CN113591442A (zh) * 2021-10-08 2021-11-02 北京明略软件系统有限公司 文本生成方法、装置、电子装置及可读存储介质
CN113591442B (zh) * 2021-10-08 2022-02-18 北京明略软件系统有限公司 文本生成方法、装置、电子装置及可读存储介质
CN115017337A (zh) * 2022-08-03 2022-09-06 中国电子科技集团公司第五十四研究所 一种面向深度学习模型的隐语义解释方法
CN115587175A (zh) * 2022-12-08 2023-01-10 阿里巴巴达摩院(杭州)科技有限公司 人机对话及预训练语言模型训练方法、系统及电子设备

Also Published As

Publication number Publication date
CN110727806A (zh) 2020-01-24
JP2022517292A (ja) 2022-03-08
CN110727806B (zh) 2020-08-11
EP3866025A4 (en) 2022-02-09
EP3866025A1 (en) 2021-08-18
JP7250052B2 (ja) 2023-03-31

Similar Documents

Publication Publication Date Title
WO2021120543A1 (zh) 基于自然语言和知识图谱的表示学习方法及装置
US11842172B2 (en) Graphical user interface to an artificial intelligence engine utilized to generate one or more trained artificial intelligence models
US10679148B2 (en) Implicit bridging of machine learning tasks
US10007660B2 (en) Contextual language understanding for multi-turn language tasks
US10629193B2 (en) Advancing word-based speech recognition processing
US10832658B2 (en) Quantized dialog language model for dialog systems
US20210200947A1 (en) Event argument extraction method and apparatus and electronic device
WO2021169745A1 (zh) 基于语句前后关系预测的用户意图识别方法及装置
US11080073B2 (en) Computerized task guidance across devices and applications
US20230237277A1 (en) Aspect prompting framework for language modeling
US20210334659A1 (en) Method and apparatus for adversarial training of machine learning model, and medium
CN113641830A (zh) 模型预训练方法、装置、电子设备和存储介质
US20230367978A1 (en) Cross-lingual apparatus and method
US20210232775A1 (en) Language generation method and apparatus, electronic device and storage medium
US12019990B2 (en) Representation learning method and device based on natural language and knowledge graph
US20210192364A1 (en) Representation learning method and device based on natural language and knowledge graph
Kim et al. Sungjin Lee
Qian et al. An Interactive Multi-task Learning Model for Aspect-Based Sentiment Analysis
US20220327288A1 (en) Semantic Parsing of Utterance Using Contractive Paraphrasing
CN118227868A (zh) 文本处理方法、装置、电子设备和存储介质

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2020571787

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2020864301

Country of ref document: EP

Effective date: 20210325

NENP Non-entry into the national phase

Ref country code: DE