CN111813961A

CN111813961A - Data processing method and device based on artificial intelligence and electronic equipment

Info

Publication number: CN111813961A
Application number: CN202010860523.9A
Authority: CN
Inventors: 周辉阳
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-08-25
Filing date: 2020-08-25
Publication date: 2020-10-23
Anticipated expiration: 2040-08-25
Also published as: CN111813961B

Abstract

The application provides a data processing method, a device, electronic equipment and a computer readable storage medium based on artificial intelligence; the method comprises the following steps: acquiring a text to be subjected to knowledge extraction; acquiring a knowledge extraction template of a problem comprising a plurality of levels, traversing the plurality of levels according to the level order of the plurality of levels, and executing the following processing for each traversed level: performing answer prediction processing by combining the traversed hierarchy question and the text to obtain an answer corresponding to the traversed hierarchy question, and updating the next hierarchy question according to the answer corresponding to the traversed hierarchy question so as to perform answer prediction processing of the next hierarchy by combining the next hierarchy question and the text; and constructing a knowledge graph according to the questions and corresponding answers of each level in the knowledge extraction template. Through the method and the device, the efficiency of extracting knowledge and constructing the knowledge graph can be improved.

Description

Data processing method and device based on artificial intelligence and electronic equipment

Technical Field

The present application relates to artificial intelligence and natural language processing technologies, and in particular, to a data processing method and apparatus based on artificial intelligence, an electronic device, and a computer-readable storage medium.

Background

Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. Natural Language Processing (NLP) is an important direction in the field of artificial intelligence, and various theories and methods for realizing efficient communication between a person and a computer using natural Language are mainly studied.

The question-answering system is an important application in natural language processing, such as intelligent customer service, a chat robot, an intelligent sound box and the like, and is used for feeding back corresponding answers aiming at a question expressed by natural language. In the related art, knowledge is generally extracted from a specific text manually, and a knowledge graph is constructed to implement question answering based on the knowledge graph. However, the efficiency of extracting knowledge manually is low, and when the number of texts to be subjected to knowledge extraction is large, the knowledge graph cannot be constructed timely and effectively.

Disclosure of Invention

The embodiment of the application provides a data processing method and device based on artificial intelligence, electronic equipment and a computer readable storage medium, and the efficiency of knowledge extraction and knowledge graph construction can be improved.

The technical scheme of the embodiment of the application is realized as follows:

the embodiment of the application provides a data processing method based on artificial intelligence, which comprises the following steps:

acquiring a text to be subjected to knowledge extraction;

acquiring a knowledge extraction template of a problem comprising a plurality of levels, traversing the plurality of levels according to the level order of the plurality of levels, and executing the following processing for each traversed level:

performing answer prediction processing by combining the traversed hierarchy question and the text to obtain an answer corresponding to the traversed hierarchy question, and

updating the question of the next level according to the answer corresponding to the traversed question of the level, so as to be used for performing answer prediction processing of the next level in combination with the text;

and constructing a knowledge graph according to the questions and corresponding answers of each level in the knowledge extraction template.

The embodiment of the application provides a data processing device based on artificial intelligence, includes:

the text acquisition module is used for acquiring a text to be subjected to knowledge extraction;

a traversal module, configured to acquire a knowledge extraction template of a problem including a plurality of hierarchies, traverse the plurality of hierarchies according to a hierarchy order of the plurality of hierarchies, and perform the following processing for each traversed hierarchy:

and the construction module is used for constructing a knowledge graph according to the question and the corresponding answer of each level in the knowledge extraction template.

An embodiment of the present application provides an electronic device, including:

a memory for storing executable instructions;

and the processor is used for realizing the artificial intelligence-based data processing method provided by the embodiment of the application when the executable instructions stored in the memory are executed.

The embodiment of the application provides a computer-readable storage medium, which stores executable instructions for causing a processor to execute, so as to implement the artificial intelligence-based data processing method provided by the embodiment of the application.

The embodiment of the application has the following beneficial effects:

the questions of multiple levels in the knowledge extraction template are combined with the text respectively to perform answer prediction processing to obtain corresponding answers, and a knowledge graph is constructed based on the questions of each level and the corresponding answers, so that the efficiency and convenience of knowledge extraction and knowledge graph construction are improved, the time cost is reduced, and meanwhile, the accuracy of knowledge in the knowledge graph can be improved. Based on the constructed knowledge graph, accurate and reliable question answering service can be provided.

Drawings

FIG. 1 is an alternative architectural diagram of an artificial intelligence based data processing system provided by an embodiment of the present application;

fig. 2 is an alternative architecture diagram of a terminal device provided in the embodiment of the present application;

FIG. 3A is a schematic flow chart of an alternative artificial intelligence based data processing method according to an embodiment of the present application;

FIG. 3B is a schematic flow chart of an alternative artificial intelligence based data processing method according to an embodiment of the present application;

FIG. 3C is a schematic flow chart diagram illustrating an alternative artificial intelligence based data processing method according to an embodiment of the present application;

FIG. 3D is a schematic flow chart diagram illustrating an alternative artificial intelligence based data processing method according to an embodiment of the present application;

FIG. 3E is a schematic flow chart diagram illustrating an alternative artificial intelligence based data processing method according to an embodiment of the present application;

FIG. 4 is an alternative illustration of headings for multiple levels in text provided by an embodiment of the present application;

FIG. 5A is an alternative schematic diagram of a knowledge-graph listing interface provided by embodiments of the present application;

FIG. 5B is an alternative schematic diagram of a knowledge definition interface provided by embodiments of the present application;

FIG. 5C is an alternative schematic diagram of a graph construction interface provided by embodiments of the present application;

FIG. 5D is an alternative schematic diagram of a graph construction interface provided by embodiments of the present application;

FIG. 5E is an alternative schematic diagram of a graph construction interface provided by embodiments of the present application;

FIG. 5F is an alternative schematic diagram of a graph construction interface provided by embodiments of the present application;

fig. 6 is an alternative flow diagram of knowledge extraction provided by the embodiments of the present application.

Detailed Description

In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the attached drawings, the described embodiments should not be considered as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

In the following description, references to the terms "first \ second \ third" are only to distinguish similar objects and do not denote a particular order, but rather the terms "first \ second \ third" are used to interchange specific orders or sequences, where appropriate, so as to enable the embodiments of the application described herein to be practiced in other than the order shown or described herein. In the following description, the term "plurality" referred to means at least two.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the application.

Before further detailed description of the embodiments of the present application, terms and expressions referred to in the embodiments of the present application will be described, and the terms and expressions referred to in the embodiments of the present application will be used for the following explanation.

1) A knowledge extraction template: a template for knowledge extraction of text, the template including a plurality of levels of questions, the plurality of levels progressing sequentially.

2) Knowledge Graph (Knowledge Graph): a semantic network for revealing relationships between entities, a knowledge graph is composed of a plurality of pieces of knowledge, and each piece of knowledge can be represented in the form of a Subject-Predicate-Object (SPO) triple.

3) Machine Learning (ML): the study on how the computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is a fundamental approach for enabling computers to have intelligence, and can construct artificial intelligence models, such as neural network models and the like, by utilizing the machine learning principle.

4) Named Entity Recognition (NER): one key technology in the field of NLP is to identify named entities with specific meaning in text, such as names of people, places, organizations, proper nouns, etc.

5) Dependency syntax processing (DP): the method is also called dependency syntax analysis, which is also one of the key technologies in the NLP field, and refers to determining the dependency relationships among entities in a text, such as the major-predicate relationship, the dynamic guest relationship, the inter-guest relationship, and the like.

6) Question and answer pairs: one question corresponds to the form of one answer.

7) Machine Reading Comprehension (MRC): one of the core tasks in the NLP field is also an important task for evaluating the ability of the model to understand text, and its essence can be regarded as a sentence relation matching task, and its specific prediction result is related to the specific task.

8) Database (Database): data sets that are stored together in a manner that enables sharing with multiple users, has as little redundancy as possible, and is independent of the application, the users can perform additions, queries, updates, and deletions to the data in the database.

The embodiment of the application provides a data processing method and device based on artificial intelligence, an electronic device and a computer readable storage medium, which can improve the efficiency and convenience of knowledge extraction and knowledge graph construction and reduce time cost. An exemplary application of the electronic device provided in the embodiments of the present application is described below, and the electronic device provided in the embodiments of the present application may be implemented as various types of terminal devices such as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, and a portable game device), and may also be implemented as a server.

The electronic equipment can rapidly extract effective knowledge from the text by operating the data processing scheme provided by the embodiment of the application, so that question and answer service is provided, namely the question and answer processing performance of the electronic equipment is improved, and the method and the device are suitable for various question and answer scenes. For example, the electronic device may be an intelligent sound box, the text to be subjected to knowledge extraction may be corpora (e.g., function introduction text, etc.) related to the intelligent sound box, and the electronic device may provide an intelligent question and answer service by performing knowledge extraction on the text and constructing a knowledge graph, so that a user can accurately and quickly know each function of the intelligent sound box by calling the intelligent question and answer service.

Referring to fig. 1, fig. 1 is an architecture diagram of an alternative artificial intelligence based data processing system 100 provided in the embodiment of the present application, in which a terminal device 400 is connected to a server 200 through a network 300, and the server 200 is connected to a database 500, where the network 300 may be a wide area network or a local area network, or a combination of both.

In some embodiments, taking the electronic device as a terminal device as an example, the artificial intelligence based data processing method provided in the embodiments of the present application may be implemented by the terminal device. For example, after acquiring a text to be subjected to knowledge extraction, the terminal device 400 performs knowledge extraction on the text according to a knowledge extraction template and constructs a knowledge graph, and then provides question and answer service based on the knowledge graph. The knowledge extraction template may be pre-stored locally in the terminal device 400, or may be obtained by the terminal device 400 from the outside (e.g., the database 500) in real time.

In some embodiments, taking the electronic device as a server as an example, the data processing method based on artificial intelligence provided in the embodiments of the present application may also be implemented by the server. The server 200 acquires the text to be subjected to knowledge extraction and the knowledge extraction template from the database 500, performs knowledge extraction on the text according to the knowledge extraction template, constructs a knowledge graph, and provides question and answer service based on the knowledge graph. The terminal apparatus 400 can perform question answering by calling a question answering service. For example, the terminal apparatus 400 transmits an inquiry operation including a question to the server 200 by calling a question-and-answer service, the server 200 inquires a corresponding answer in the knowledge graph, and transmits the answer to the terminal apparatus 400 in response to the inquiry operation. In addition, the server 200 may also transmit a question-answering application (e.g., a chat robot application) constructed based on a knowledge graph to the terminal apparatus 400, or directly transmit the knowledge graph to the terminal apparatus 400, so that the terminal apparatus 400 locally provides a question-answering service. It should be noted that, in the embodiment of the present application, the storage locations of the text, the knowledge extraction template, and the knowledge graph are not limited, and may be, for example, locations of the database 500, the distributed file system of the server 200, or a blockchain.

The terminal device 400 is used to display various results and final results in the question-answering process in the graphical interface 410. In fig. 1, taking the server 200 providing the question-and-answer service as an example, a question sent by the terminal device 400 to the server 200 in the process of invoking the question-and-answer service, and an answer obtained by the server 200 by querying in the knowledge graph according to the question are shown.

In some embodiments, the server 200 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, a middleware service, a domain name service, a security service, a CDN, and a big data and artificial intelligence platform, where the cloud service may be a question and answer service that is called by the terminal device 400, so as to query in a knowledge graph according to a question sent by the terminal device 400, and send an answer to the terminal device 400. The terminal device 400 may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like. The terminal device and the server may be directly or indirectly connected through wired or wireless communication, and the embodiment of the present application is not limited.

Taking the electronic device provided in the embodiment of the present application as an example for illustration, it can be understood that, for the case where the electronic device is a server, parts (such as the user interface, the presentation module, and the input processing module) in the structure shown in fig. 2 may be default. Referring to fig. 2, fig. 2 is a schematic structural diagram of a terminal device 400 provided in an embodiment of the present application, where the terminal device 400 shown in fig. 2 includes: at least one processor 410, memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal device 400 are coupled together by a bus system 440. It is understood that the bus system 440 is used to enable communications among the components. The bus system 440 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 440 in fig. 2.

The Processor 410 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor, or the like.

The user interface 430 includes one or more output devices 431, including one or more speakers and/or one or more visual displays, that enable the presentation of media content. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, and the like. Memory 450 optionally includes one or more storage devices physically located remote from processor 410.

The memory 450 includes either volatile memory or nonvolatile memory, and may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (ROM), and the volatile Memory may be a Random Access Memory (RAM). The memory 450 described in embodiments herein is intended to comprise any suitable type of memory.

In some embodiments, memory 450 is capable of storing data, examples of which include programs, modules, and data structures, or a subset or superset thereof, to support various operations, as exemplified below.

An operating system 451, including system programs for handling various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and handling hardware-based tasks;

a network communication module 452 for communicating to other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: bluetooth, wireless compatibility authentication (WiFi), and Universal Serial Bus (USB), etc.;

a presentation module 453 for enabling presentation of information (e.g., user interfaces for operating peripherals and displaying content and information) via one or more output devices 431 (e.g., display screens, speakers, etc.) associated with user interface 430;

an input processing module 454 for detecting one or more user inputs or interactions from one of the one or more input devices 432 and translating the detected inputs or interactions.

In some embodiments, the artificial intelligence based data processing apparatus provided by the embodiments of the present application can be implemented in software, and fig. 2 shows an artificial intelligence based data processing apparatus 455 stored in a memory 450, which can be software in the form of programs and plug-ins, and the like, and includes the following software modules: a text acquisition module 4551, a traversal module 4552, and a construction module 4553, which are logical and thus arbitrarily combined or further split depending on the functions implemented. The functions of the respective modules will be explained below.

In other embodiments, the artificial intelligence based data processing apparatus provided in the embodiments of the present Application may be implemented in hardware, for example, the artificial intelligence based data processing apparatus provided in the embodiments of the present Application may be a processor in the form of a hardware decoding processor, which is programmed to execute the artificial intelligence based data processing method provided in the embodiments of the present Application, for example, the processor in the form of the hardware decoding processor may be implemented by one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs), or other electronic components.

The artificial intelligence based data processing method provided by the embodiment of the present application will be described in conjunction with exemplary applications and implementations of the electronic device provided by the embodiment of the present application.

Referring to fig. 3A, fig. 3A is an alternative flowchart of the artificial intelligence based data processing method according to the embodiment of the present application, which will be described with reference to the steps shown in fig. 3A.

In step 101, a text to be subjected to knowledge extraction is acquired.

In step 102, a knowledge extraction template including a plurality of levels of questions is obtained, and for a traversed level in the plurality of levels of the knowledge extraction template, answer prediction processing is performed by combining the traversed level of questions and the text, so as to obtain an answer corresponding to the traversed level of questions.

Here, the knowledge extraction template includes a plurality of hierarchical problems and a hierarchical order of the plurality of hierarchical problems, that is, the plurality of hierarchical problems are sequentially advanced. When a text to be subjected to knowledge extraction and a knowledge extraction template are acquired, firstly, traversing a plurality of levels in the knowledge extraction template, and performing machine reading understanding on the problems of the traversed levels, namely performing answer prediction processing by combining the problems and the text of the traversed levels, and determining the answer corresponding to the problems of the traversed levels in the text. The embodiment of the present application does not limit the answer prediction processing manner, and for example, the answer prediction processing may be performed by an artificial intelligence model.

It is worth noting that in the knowledge extraction template, each level includes at least one question. In the case where the same hierarchy includes a plurality of questions, the plurality of questions may respectively include a plurality of attributes that are semantically (actually) identical, for example, the questions of a certain hierarchy include "how much the birthday of a person is", "what the birth date of a person is", and "what the birth year, month and day of a person is", where the semantics of the attributes "birthday", "birth date", and "birth year, month and day" are identical, and there is only a difference in description. Then, a plurality of attributes with the same semantic meaning can be mapped to the same attribute, so that classification management is facilitated, for example, "birthday", "birth date" and "year, month and day of birth" are uniformly mapped to "birthday". If the answer corresponding to the question 'what the birthday of a person is' is obtained after the answer prediction processing, the answer is also determined as the answer corresponding to the question 'what the birthday of a person is', and then the question 'what the birthday of a person is' and the corresponding answer are unified to construct a knowledge graph. The method is suitable for the scene that the same vocabulary (attribute) has multiple synonyms, and certainly, multiple problems included in the same level in the knowledge extraction template can be different in semantics and are determined according to the actual application scene.

In some embodiments, after step 102, further comprising: obtaining a plurality of standby answer formats and a uniform answer format corresponding to the question of each level in a plurality of levels; when the answer corresponding to the question of any one of the multiple levels fails to be matched with the corresponding multiple standby answer formats, deleting the answer; and when the answer corresponding to the question of any one of the multiple levels is successfully matched with any one of the corresponding standby answer formats, updating the successfully matched answer to accord with the corresponding uniform answer format.

In the knowledge extraction template, a plurality of alternative answer formats and a uniform answer format corresponding to the question of each level can be set. For example, if a question at a certain level is "what the birthday of a person is", that is, the included attribute is "birthday", the corresponding alternative answer formats may be set to include "x.x.x", "x/x/x", and "x year x month x day", and the corresponding unified answer format is "x year x month x day", where the unified answer format may be the same as one of the alternative answer formats or may be different from all of the alternative answer formats.

And after the answer corresponding to the problem of the traversed hierarchy is obtained, matching the answer with a plurality of standby answer formats corresponding to the problem of the traversed hierarchy. When the answer fails to match with a plurality of standby answer formats, namely the answer is not consistent with all the standby answer formats, the answer is proved to be inaccurate, and the answer is deleted; and when the answer is successfully matched with any one of the standby answer formats, updating the answer so as to accord with the corresponding uniform answer format. For example again, if the obtained answer is "2020.8.8" and conforms to the format of "x.x.x", the answer is updated to obtain an updated answer "8/2020". By the mode, the obtained answer is subjected to data cleaning, namely the answer is cleaned into a uniform format, the effectiveness of the answer is improved, and the data quality is guaranteed.

In step 103, for the traversed level, the question of the next level is updated according to the answer corresponding to the question of the traversed level, so as to be used for performing answer prediction processing of the next level in combination with the text.

For the traversed level, in addition to determining the answer corresponding to the question of the traversed level, the question of the next level is updated according to the answer, for example, the answer is filled into the question of the next level for the answer prediction processing of the next level in combination with the text. When the traversed level is the last level in the plurality of levels, the answer corresponding to the question of the last level is determined only because the next level does not exist. In addition, if the answer corresponding to the question of the traversed hierarchy is not obtained after the answer prediction processing, the traversal is terminated (provided that the question of the traversed hierarchy includes only one question).

In some embodiments, the above-mentioned updating of the question of the next hierarchy according to the answer corresponding to the question of the traversed hierarchy may be implemented in such a way that: and filling the answer to a part to be filled in the question of the next level aiming at each answer corresponding to the question of the traversed level.

Here, the question of the next hierarchy is individually updated for each answer corresponding to the question of the traversed hierarchy. In the knowledge extraction template, for the levels except for the first level in the plurality of levels, a part to be filled in the questions included in the levels can be set, so that answers corresponding to the questions in the previous level can be filled conveniently for updating.

For example, the question of the first level in the knowledge extraction template is "who all the faces there are," the question of the next level, i.e., the second level, is "which companies xx work at," and "xx" in the question of the second level is the part to be filled. If the obtained answer corresponding to the question of the first level comprises Zhang III and lie IV, the Zhang III is filled into the part to be filled in the question of the second level to obtain the company in which Zhang III works, and the lie IV is filled into the part to be filled in the question of the second level to obtain the company in which lie IV works, namely after updating, the question of the second level comprises two questions. By the method, the problem is effectively updated, and the method is suitable for the problem of hierarchy progression.

In step 104, a knowledge graph is constructed according to the question and the corresponding answer of each level in the knowledge extraction template.

And if the answer corresponding to the question of each level in the knowledge extraction template is obtained after traversal, taking the question of each level and the corresponding answer as the knowledge extracted from the text to construct a knowledge graph. The question of each level (except the question of the first level) used for constructing the knowledge graph refers to the question updated in step 103. In embodiments of the present application, a knowledge-graph may be constructed from one text or multiple texts.

Based on the constructed knowledge graph, question answering service can be provided. For example, a query operation for a question at any one of a plurality of levels is received, and a corresponding answer is queried according to the knowledge graph in response to the query operation.

As shown in fig. 3A, the embodiment of the application performs knowledge extraction on the text through the knowledge extraction template, improves the convenience of knowledge extraction and knowledge graph construction, and is suitable for unstructured texts.

In some embodiments, referring to fig. 3B, fig. 3B is an optional flowchart of the artificial intelligence based data processing method provided in the embodiment of the present application, and step 102 shown in fig. 3A may be implemented by steps 201 to 203, which will be described in conjunction with the steps.

In step 201, a knowledge extraction template including a plurality of levels of questions is obtained, and the plurality of levels are traversed according to the level sequence of the plurality of levels in the knowledge extraction template.

In step 202, when the traversed hierarchy is any one of the plurality of hierarchies except the last hierarchy, probability mapping processing is performed by combining the questions and the texts of the traversed hierarchy to obtain a plurality of candidate answers in the texts and the probability of each candidate answer, and the candidate answer with the probability greater than the first probability threshold is determined as the answer corresponding to the question of the traversed hierarchy.

In the knowledge extraction template, the answer corresponding to the question of the last level is usually one, and the answer corresponding to the question of other levels may be multiple. Based on the above, when the traversed hierarchy is any one of the plurality of hierarchies except the last hierarchy, probability mapping processing is performed by combining the questions and the texts of the traversed hierarchy, namely, a plurality of candidate answers in the texts are predicted, and the probability that each candidate answer is a correct answer is determined. Then, the candidate answers with the probability larger than the first probability threshold are determined as answers corresponding to the questions of the traversed hierarchy. And when the probability of all the candidate answers is less than or equal to the first probability threshold, determining that the answer corresponding to the question of the traversed hierarchy does not exist in the text.

In step 203, when the traversed hierarchy is the last hierarchy, probability mapping processing is performed by combining the question and the text of the last hierarchy to obtain a plurality of candidate answers in the text and a probability of each candidate answer, and the candidate answer with the probability greater than a second probability threshold and the maximum probability is determined as the answer corresponding to the question of the last hierarchy.

Since the question of the last level generally corresponds to an answer, when the traversed level is the last level in the multiple levels, probability mapping processing is performed by combining the question of the last level and the text, so that multiple candidate answers in the text and the probability that each candidate answer is a correct answer are obtained. And then, determining the candidate answer with the probability greater than a second probability threshold and the maximum probability as the answer corresponding to the question of the last level. And when the probability of all the candidate answers is less than or equal to the second probability threshold, determining that the answer corresponding to the question of the last level does not exist in the text. The first probability threshold and the second probability threshold can be set according to actual application scenarios.

In some embodiments, the probability mapping process described above in connection with the problem and text of the traversed hierarchy may be implemented in such a way that: performing probability mapping processing on the traversed problem and the text of the hierarchy through a first artificial intelligence model; each sample question for training the first artificial intelligence model corresponds to a plurality of sample answers; the probability mapping process described above in connection with the last level of questions and text can be implemented in such a way that: performing probability mapping processing on the problem and the text of the last level through a second artificial intelligence model; each sample question for training the second artificial intelligence model corresponds to one sample answer.

Here, the probability mapping process may be implemented using an artificial intelligence model. Aiming at the difference between the last level and other levels in the multiple levels, when the traversed level is any one level except the last level in the multiple levels, performing probability mapping processing on the problems and texts of the traversed level through a first artificial intelligence model; and when the traversed hierarchy is the last hierarchy, performing probability mapping processing on the questions and texts of the last hierarchy through a second artificial intelligence model. The first artificial intelligence model is obtained by training first training data, wherein the first training data comprise a sample text, a sample question and a plurality of sample answers corresponding to the sample question, so that the trained first artificial intelligence model can adapt to the condition that one question corresponds to a plurality of answers; the second artificial intelligence model is obtained through training of second training data, and the second training data comprise a sample text, a sample question and a sample answer corresponding to the sample question. Through the mode, the two artificial intelligence models are trained pertinently, and the applicability to different levels is improved.

In some embodiments, before performing probability mapping processing on the traversed hierarchy of questions and texts through the first artificial intelligence model, the method further includes: performing probability mapping processing on the sample text and the corresponding sample questions through a first artificial intelligence model to obtain a plurality of candidate answers in the sample text and the probability of each candidate answer, and determining the candidate answer with the probability greater than a first probability threshold value as an answer to be compared; and performing back propagation in the first artificial intelligent model according to the difference between the answer to be compared and the sample answers corresponding to the sample question, and updating the weight parameter of the first artificial intelligent model in the process of back propagation.

When the first artificial intelligence model is trained according to the first training data, firstly, probability mapping processing is carried out on a sample text and a sample question in the first training data through the first artificial intelligence model, a plurality of candidate answers in the sample text and the probability of each candidate answer are obtained, and for convenience of distinguishing, the candidate answer of which the probability is greater than a first probability threshold value is named as an answer to be compared. Then, the difference between the answer to be compared and the multiple sample answers corresponding to the sample question is determined through the loss function of the first artificial intelligence model. And performing back propagation in the first artificial intelligent model according to the obtained difference, and updating the weight parameters of each network layer in the first artificial intelligent model along the gradient descending direction in the process of back propagation.

Similarly, when the second artificial intelligence model is trained according to the second training data, firstly, the sample text and the sample questions in the second training data are subjected to probability mapping processing through the second artificial intelligence model to obtain a plurality of candidate answers in the sample text and the probability of each candidate answer, and for convenience of distinguishing, the candidate answer with the probability greater than the second probability threshold and the maximum probability is named as the answer to be compared. Then, the difference between the answer to be compared and one sample answer corresponding to the sample question is determined through the loss function of the second artificial intelligence model. And performing back propagation in the second artificial intelligence model according to the obtained difference, and updating the weight parameters of each network layer in the second artificial intelligence model along the gradient descending direction in the process of back propagation. By the mode, the artificial intelligence model is effectively trained, so that when probability mapping processing is carried out on the trained artificial intelligence model, accurate probability can be obtained.

In some embodiments, the probability mapping process described above in connection with the problem and text of the traversed hierarchy may be implemented in such a way that: performing probability mapping processing on the traversed problem and the text of the hierarchy through a third artificial intelligence model; the probability mapping process described above in connection with the last level of questions and text can be implemented in such a way that: performing probability mapping processing on the problem and the text of the last level through a third artificial intelligence model; each sample question for training the third artificial intelligence model corresponds to one sample answer.

In the embodiment of the present application, probability mapping processing may be performed on the question and the text at each level through a single artificial intelligence model, and for convenience of distinguishing, the artificial intelligence model is named as a third artificial intelligence model. When the traversed hierarchy is any one of the plurality of hierarchies except the last hierarchy, performing probability mapping processing on the questions and texts of the traversed hierarchy through a third artificial intelligence model, and determining candidate answers with the probabilities larger than a first probability threshold value as answers to be compared; and when the traversed hierarchy is the last hierarchy, performing probability mapping processing on the question and the text of the last hierarchy through a third artificial intelligence model, and determining the candidate answer with the probability greater than a second probability threshold and the maximum probability as the answer corresponding to the question of the last hierarchy. When the probability mapping process is performed by the third artificial intelligence model, the first probability threshold and the second probability threshold may be the same.

In the case of performing probability mapping processing by the third artificial intelligence model, when the traversed hierarchy is any one of the plurality of hierarchies except the last hierarchy, probability mapping processing may be performed on the questions and texts of the traversed hierarchy by the third artificial intelligence model, and the candidate answer with the probability greater than the first probability threshold and the maximum probability is determined as the answer to be compared. I.e. for each of a plurality of levels, only one answer is determined.

It should be noted that the third training data for training the third artificial intelligence model includes a sample text, a sample question, and a sample answer corresponding to the sample question, and the training process of the third artificial intelligence model is similar to the training process of the second artificial intelligence model, and is not described herein again. By the mode, the workload of model training can be reduced, namely only one artificial intelligence model is trained.

As shown in fig. 3B, in the embodiment of the present application, the last hierarchy is distinguished from other hierarchies, so that the pertinence to the problem of different hierarchies is improved, and the accuracy of the obtained answer is improved.

In some embodiments, referring to fig. 3C, fig. 3C is an optional flowchart of the artificial intelligence based data processing method provided in the embodiment of the present application, and step 104 shown in fig. 3A may be updated to step 301, and in step 301, a triple is constructed according to the question and the corresponding answer of each level in the knowledge extraction template, so as to construct a knowledge graph according to a plurality of triples; wherein the triplets include entities in the question, attributes, and attribute values in the answer.

After the knowledge extraction of the text is completed according to the knowledge extraction template, an answer corresponding to the question of each hierarchy can be obtained, and certainly, in an actual application scenario, the answer corresponding to the question of a part of hierarchies may not exist in the text. The question of each level comprises an entity and an attribute, and the corresponding answer comprises an attribute value of the attribute, so that the SPO triple can be constructed based on the entity, the attribute and the attribute value, wherein the form of the attribute value is not limited to a numerical form, and can also be a text form. For example, if a question at a certain level is "what is a nationality of zhang san", and the corresponding answer is "china", the entity included in the question is "zhang san", the attribute is "nationality", and the attribute value included in the answer is "china", an SPO triple of zhang san-nationality-china "can be constructed. And further constructing a knowledge graph according to all the obtained triples.

As shown in FIG. 3C, after step 301, a first entity and a second entity may also be picked from a plurality of triples of the knowledge-graph in step 302.

In the text, there may be multiple descriptions of the same entity, such as a person's Chinese and English names included in the text. In the embodiment of the application, fusion processing can be performed on the entities with the substantially same property, so as to eliminate ambiguity and improve data quality. For example, a first entity and a second entity are randomly selected from all triples of the knowledge-graph until all possible combinations of the first entity and the second entity are obtained. In the combination of the first entity and the second entity, the order of the first entity and the second entity is not distinguished.

In step 303, the same attribute is determined from the plurality of attributes corresponding to the first entity and the plurality of attributes corresponding to the second entity.

For example, the knowledge-graph includes triplets "Zhang-birthday-1/1980", "Zhang-occupation-writer", "Zhang-hobby-tennis", "s, Zhang-birthday-1/1980", "s, Zhang-occupation-writer", and "s.zhang-nationality-china". The selected first entity is Zhang III, and the corresponding attributes comprise birthday, occupation and hobby; the second entity is selected as "s. Zhang", and the corresponding attributes include birthday, occupation, and nationality. It can be determined that the same attributes include birthday and occupation.

In step 304, a similarity between the attribute values of the same attribute corresponding to the first entity and the attribute values of the same attribute corresponding to the second entity is determined.

For example, for the attribute birthday, the attribute values of the first entity "Zhang" and the second entity "s. Zhang" are both "1 month and 1 day of 1980", i.e., the similarity between the attribute values is 100%; for attribute occupations, the attribute values of the first entity "Zhang" and the second entity "s. Zhang" are both "writers", i.e., the similarity between the attribute values is also 100%. It should be noted that the type of the similarity between the attribute values is not limited in the embodiments of the present application, and may be, for example, Edit Distance (ED), set similarity (e.g., jaccard coefficient), cosine similarity, or the like.

In step 305, when the similarity satisfies the similarity condition, the first entity and the second entity are fused.

When the similarity obtained in step 304 meets the set similarity condition, it is verified that the first entity and the second entity are the same entity, and the first entity and the second entity are subjected to fusion processing, so that the first entity and the second entity share the attribute and the attribute value. For example, in response to a query operation against entity "Zhang", queries are performed simultaneously in triples of the knowledge-graph with "Zhang" and "s. Zhang" as main bodies; for another example, new triples of "zhang san-nationality-china" and "s.zhang-hobby-tennis" are added to the knowledge graph to complete the triples.

Except for the mode of constructing the knowledge graph, the method can also be used for performing fusion processing on a plurality of triples, and after the fusion processing is finished, the knowledge graph is constructed according to all the obtained triples.

In some embodiments, the number of identical attributes is multiple; the fusion processing of the first entity and the second entity when the similarity satisfies the similarity condition can be realized in such a way that: any one of the following processes is performed: acquiring the weight corresponding to each same attribute, weighting the similarity corresponding to a plurality of same attributes according to the weight to obtain entity similarity, and fusing the first entity and the second entity when the entity similarity is greater than an entity similarity threshold; and acquiring an attribute similarity threshold corresponding to each same attribute, and fusing the first entity and the second entity when the similarity corresponding to the same attributes is greater than the corresponding attribute similarity threshold.

When the number of the same attributes corresponding to the first entity and the second entity is multiple, the embodiment of the present application provides the following two similarity conditions:

1) the weight corresponding to each identical attribute is obtained, and the weight may be preset. Then, according to the weight corresponding to each identical attribute, weighting processing is performed on the similarity (referring to the similarity between attribute values) corresponding to a plurality of identical attributes to obtain entity similarity, wherein the weighting processing may be weighted summation or weighted average. When the same attribute is only one, the similarity corresponding to the same attribute may be weighted according to the weight corresponding to the same attribute, that is, the product of the weight and the similarity is processed to obtain the entity similarity. And when the finally obtained entity similarity is larger than the set entity similarity threshold, carrying out fusion processing on the first entity and the second entity.

2) For each attribute, an attribute similarity threshold is preset. And when the similarity (referring to the similarity among the attribute values) corresponding to the same attributes is larger than the corresponding attribute similarity threshold, performing fusion processing on the first entity and the second entity. The same applies to the case where the same attribute is only one.

By the method, the flexibility of the fusion processing is improved, certainly, the similarity condition is not limited to the method, and other similarity conditions can be applied according to the actual application scene.

As shown in fig. 3C, by performing fusion processing on two entities with higher similarity, the comprehensiveness and effectiveness of the constructed knowledge graph are improved, i.e., more effective question and answer service can be provided.

In some embodiments, referring to fig. 3D, fig. 3D is an optional flowchart of the artificial intelligence based data processing method provided in the embodiment of the present application, and step 102 shown in fig. 3A may be implemented through steps 401 to 404.

In step 401, named entity recognition is performed on the text, and a plurality of named entities in the text and an entity type of each named entity are obtained.

In the embodiment of the application, a plurality of knowledge extraction templates can be preset, and each knowledge extraction template is set to correspond to one set entity type so as to be suitable for texts comprising entities of different entity types. For example, a problem is that a set entity type corresponding to a certain knowledge extraction template is set as a person name, and a plurality of hierarchies related to the person name are set in the knowledge extraction template; setting a place name as a setting entity type corresponding to another knowledge extraction template, and setting a plurality of hierarchies related to the place name in the knowledge extraction template.

After the text to be subjected to knowledge extraction is acquired, a knowledge extraction template can be manually selected for answer prediction processing. The embodiment of the application also provides another automatic selection mode, and firstly, named entity recognition is carried out on the text to obtain a plurality of named entities (namely entities) in the text and an entity type of each named entity. The named entity recognition method in the embodiments of the present application is not limited, and may be, for example, a rule and dictionary based method, a statistical based method (for example, a hidden markov model, a maximum entropy model, a support vector machine model, or a conditional random field model), or a mixture of the two methods.

In step 402, the entity type of the named entity in the text is matched with a plurality of set entity types.

Here, the entity type of each named entity in the text is matched with all the set entity types.

In some embodiments, before step 402, further comprising: carrying out dependency syntax processing on a plurality of named entities in the text to obtain a plurality of dependency relationships; and screening out the named entity with the grammar type as the subject in the text according to the main and subordinate relations in the dependence relations, and taking the named entity as the named entity matched with the plurality of set entity types.

Since the subject in the text is usually a main component, after obtaining the multiple named entities in the text, the dependency syntax processing is further performed on the multiple named entities to obtain the dependency relationship between the named entities, where the embodiment of the present application does not limit the dependency syntax processing manner. And then, according to the dominance-predicate relationship in the dependency relationships, the named entities with the grammar types as the subject are screened out from the named entities, so that the entity types of the screened named entities are conveniently matched with a plurality of set entity types. By the method, the named entities with higher importance degree can be screened out, so that the accuracy of subsequently selecting the knowledge extraction template is further improved.

In step 403, when the matching is successful, a knowledge extraction template corresponding to the set entity type that is successfully matched is obtained.

And when the entity type of any named entity in the text is the same as that of any set entity, determining that the matching is successful, further taking a knowledge extraction template corresponding to the set entity type which is successfully matched as a knowledge extraction template for answer prediction processing, and acquiring the knowledge extraction template from a database.

In step 404, for a traversed level in the plurality of levels of the knowledge extraction template, answer prediction processing is performed by combining the questions and texts of the traversed level, and an answer corresponding to the questions of the traversed level is obtained.

As shown in fig. 3D, by matching the entity type of the named entity in the text with the set entity type of the knowledge extraction template, the intelligent selection of the knowledge extraction template is realized, the manual workload is reduced, and the selected knowledge extraction template can be better applied to the text; meanwhile, the data volume of the data (knowledge extraction template) required to be acquired is reduced, and communication resources are saved.

In some embodiments, referring to fig. 3E, fig. 3E is an optional flowchart of the artificial intelligence based data processing method provided in the embodiment of the present application, and based on fig. 3A, after step 101, in step 501, when the text conforms to the set hierarchical format, titles of multiple levels in the text and a hierarchical order of the multiple levels may also be extracted according to the hierarchical format.

Here, after the text to be subjected to knowledge extraction is acquired, the text may be matched with a set hierarchical format. If the text does not conform to the hierarchical format, the text is proved to be an unstructured text, and then the step 102 to the step 104 are executed, namely, the text is subjected to knowledge extraction through a knowledge extraction template, and a knowledge graph is constructed; if the text conforms to the hierarchical format, the proof text is a structured text (for example, the text is an insurance contract book or a comment book), that is, the text has a title with multiple hierarchies, and the knowledge extraction is performed based on the hierarchical format without using the knowledge extraction template.

When the text is determined to be in accordance with the hierarchical format, the text is decomposed according to the hierarchical format to obtain titles of multiple hierarchies and hierarchical sequences of the multiple hierarchies in the text. For ease of understanding, the embodiment of the present application provides a schematic diagram of multiple levels as shown in fig. 4, and in fig. 4, the titles of the multiple levels in the text are decomposed in the form of a structure tree. Taking the text as an example of the function introduction text of the intelligent sound box, in fig. 4, the title (i) may be a "sound box function", the title (ii) may be a "song on demand", the title (iii) may be an "internet shopping", and the title (iv) may be an "intelligent home control".

In step 502, a plurality of levels are traversed according to the level sequence, a problem of the traversed level is constructed according to the title of the traversed level, and the title of the next level is determined as an answer corresponding to the problem of the traversed level.

As shown in fig. 4, a plurality of levels may be traversed in a level order of a first level to a third level. Taking the traversed hierarchy as the first hierarchy as an example, the problem of the first hierarchy is constructed according to the title of the first hierarchy, for example, the problem is 'what the function of a sound box is' or 'what the function of the sound box includes', and the titles of the second hierarchy, the third hierarchy and the fourth hierarchy in fig. 4 are combined to form the answer corresponding to the problem, namely 'song on demand, online shopping and intelligent home control'.

In step 503, a knowledge graph is constructed based on the questions and corresponding answers for each level of text.

After traversal, corresponding problems can be constructed for the levels except the last level in the plurality of levels. Then, a knowledge graph is constructed based on the constructed questions and corresponding answers (if any) to provide question-answering services based on the knowledge graph.

As shown in fig. 3E, in the embodiment of the present application, the structured text and the unstructured text are processed differentially, so that pertinence to different types of texts is improved, and for the structured text, accuracy of the obtained questions and answers can be improved by performing knowledge extraction based on a hierarchical format.

Next, an exemplary application of the embodiment of the present application in a practical application scenario will be described. For example, the embodiment of the application can be applied to an intelligent sound box, a background server of the intelligent sound box performs knowledge extraction on texts (for example, texts related to characters or scenic spots and the like) and constructs a knowledge graph, and the intelligent sound box can provide intelligent question and answer service based on the constructed knowledge graph.

An alternative schematic diagram of the background interface shown in fig. 5A is provided in the embodiment of the present application, and in the knowledge base of the knowledge platform shown in fig. 5A, a knowledge graph list including a plurality of constructed knowledge graphs (knowledge graphs 1 to 6) is shown, and a user can perform an editing or deleting operation on the constructed knowledge graphs. In addition, a search option 51 and a new option 52 for the knowledge graph are also shown, and a user can input a keyword in the search option 51 to search the corresponding knowledge graph, wherein the keyword of the knowledge graph can be at least one of an entity, an attribute and an attribute value included in the knowledge graph; a new knowledge graph can be constructed by triggering the new option 52.

When the new option 52 is triggered, a jump is made to the new background interface. As shown in fig. 5B, the background interface includes a knowledge definition module 53 and a map construction module 54, in the knowledge definition module 53, categories 531 are shown, and specifically include general categories, geographical categories and person categories under the general categories, and sight spot categories under the geographical categories, where each category at the lowest layer may be set to correspond to one depth multi-turn template (corresponding to the above knowledge extraction template), for example, a sight spot category is set to correspond to one depth multi-turn template, and a person category corresponds to another depth multi-turn template. And the user can select the corresponding category according to the characteristics of the text so as to extract knowledge according to the corresponding depth multi-turn template.

In fig. 5B, taking the depth multi-turn template corresponding to the sight spot category as an example, attribute identifiers, attribute names, data types, attribute aliases, attribute descriptions, and single/multiple-valued descriptions of the attributes included in the depth multi-turn template are shown, and the user may perform a deletion operation on some of the attributes, or may modify the depth multi-turn template by adding an option 532 to add a new attribute. In addition, a search option 533 identified by the attribute as a search condition is also shown in fig. 5B, and the user can search for the corresponding attribute by inputting a keyword in the search option 533. The search condition may be an attribute name, an attribute alias, or the like, and is not limited to the attribute identifier.

And (3) selecting categories by a user, namely binding the depth multi-round templates, and then constructing the map. As shown in fig. 5C, in the atlas construction module 54, an upload option 541 and a build option 542 are shown. When upload option 541 is triggered, a window to upload a file is presented in the interface. The user can select to upload the structured data (corresponding to the structured text) or the unstructured data (corresponding to the unstructured text) in the window, and the structured data and the unstructured data adopt different modes for knowledge extraction. It should be noted that the file format of the uploaded file is not limited in the embodiment of the present application, and may be, for example, doc format, docx format, xlsx format, and the like. Fig. 5C shows data files 1 to 5 uploaded by a user, and after the uploading is completed, the user may trigger a construction option 542 to perform knowledge extraction and map construction on the data files 1 to 5, where the data files 1 to 5 may be used to construct the same knowledge map, or may be used to construct different knowledge maps, respectively. Here, the user may also trigger the build option 543 to perform knowledge extraction and graph building separately for the data file 5.

After the completion of the map construction, the interface shown in fig. 5C may be updated to the interface shown in fig. 5D, the uploaded files, the number of entities included in the knowledge graph corresponding to each file, the construction state of the knowledge graph, the last operation time (i.e., the time when the knowledge graph was constructed last based on the file), and the executable operations including the reconstruction and deletion operations are shown. Also shown is a search option 545, where the user may enter keywords in the search option 545 to search for files that include the keywords. If the user is not satisfied with the constructed knowledgegraph, the knowledgegraph may be reconstructed, for example, by triggering a reconstruction option 544 to reconstruct the knowledgegraph based on the data file 5.

As shown in fig. 5E, when constructing the knowledge graph based on the data file 5, the operations mainly involved include knowledge extraction, attribute mapping, data cleansing, and knowledge fusion. Taking the data file 5 as unstructured data as an example, the knowledge extraction means extracting knowledge in the data file 5 through a deep multi-round template, wherein the knowledge is a question-answer pair; the attribute mapping refers to mapping a plurality of attributes with the same actual meaning to the same attribute, for example, uniformly mapping a birthday, a birth date and a birth year, month and day to a birthday; the data washing refers to washing answers into a uniform format, for example, washing 2020.08.08 and 08/08/2020 into 8.8.2020' in a uniform manner, and deleting some messy codes which do not conform to the format; knowledge fusion is to determine whether two entities are the same entity, and if the two entities are the same entity, the two entities are fused. At any time during the process of building the knowledge-graph, the user may trigger the termination option 546 to discontinue the build.

After completion of the construction of the knowledge-graph, the knowledge-graph may be presented, in FIG. 5F, with a corresponding knowledge-graph 56 being presented in the form of SPO triples for text 55 included in the data file 5. It should be noted that the presentation manner is not limited to the SPO triplet, and may be presented in the form of question-answer pairs, for example, where the question in question-answer pair is "what is [ P ] of [ S ], and the answer is" [ O ]; and can be presented in a visual and graphical way. The background server can send the constructed knowledge graph to the intelligent sound box, so that the intelligent sound box provides intelligent question and answer service based on the knowledge graph. For example, the smart speaker receives the question "what is the altitude of xx mountains", outputs the answer "1864 meters"; the question "what the address of xx mountain is" is received, and the answer "xx City in south of xx province" is output.

Next, the underlying implementation process of knowledge extraction is explained. The embodiment of the present application provides a flow diagram of knowledge extraction as shown in fig. 6, and for convenience of understanding, the flow diagram is described in a numbering form:

1) unstructured text.

Unstructured text refers to text that does not conform to a particular hierarchical format, for example, "mr. zhang, born 1973, nationality of china. In 1995, graduate university economic management specialty xx; from 1995 to 1998, owned by xx road transport limited, and assigned to do accounting; from 1998 to 2000, the post is in the xx accounting firm, the project manager. "

2) The model is understood by reading.

The reading comprehension model is used for predicting an answer corresponding to a question, the input of the model is unstructured text and a question, and the output is the answer corresponding to the question in the unstructured text. For example, in the above example, the model input is the entire unstructured text and the question "where mr. zhang's nationality is" is the answer to the model output "china". The reading understanding model is not limited in type in the embodiment of the present application, for example, the reading understanding model may be a transform-based Bidirectional Encoder Representation (BERT) model.

3) And (5) depth multi-wheel templates.

Here, different depth multi-pass templates are set for different scenes. For example, for attractions, the question will typically relate to a ticket, location, time, level, profile, and source. In many application scenarios, there often exists a dependency relationship between attributes, for example, a person works in a plurality of companies, the job position of each company is different, and the job time of each company is different, i.e. the job time depends on the company and the time. In the embodiment of the present application, a multi-turn dialog manner is adopted, each turn of dialog obtains a corresponding slot value (i.e., answer), the question is continuously asked after the corresponding slot (corresponding to the above portion to be filled) is filled until the answer corresponding to the final question is obtained, and if some questions do not obtain corresponding answers, the relationship is determined not to be true.

For example, in a deep multi-round template, the first level question is "who is there," which prompts reading of the understanding model to find subjects in unstructured text, eventually resulting in one or more names of people (answers), e.g., [ a1, a2, … … ]; in the second level, a question is asked for each answer corresponding to the question in the first level, the question in the second level is 'A works in which companies', and after the question is processed by reading the understanding model, one or more answers are obtained, for example [ B1, B2, … … ], wherein A is a slot for filling the answer corresponding to the question in the first level; in the third level, a question is asked for each answer corresponding to the question in the second level, the question in the third level is "when a is working in company B", and thus one or more answers are obtained, for example [ C1, C2, … … ], wherein B is a slot for filling the answer corresponding to the question in the second level; in the fourth level, a question is asked for each answer corresponding to the question in the third level, wherein the question in the fourth level is "what job the company B plays during C" and the answer [ D ] is obtained, wherein C is a slot for filling the answer corresponding to the question in the third level.

For example, if the answer extraction (knowledge extraction) is performed on the unstructured text according to the deep multi-round template, the answer "zhang san" can be obtained for the first level question "who is there" of the first level; the second level of the updated question is "Zhang III works in which company", and the answers "xx Highway transportation company Limited" and "xx accounting firm" can be obtained; the updated third level questions include "when Zusan is on work at xx Highway transportation Co., Ltd" and "when Zusan is on work at xx accountant's office", the answers obtained are "1995 to 1998" and "1998 to 2000", respectively; the updated fourth level questions included "zhang san was responsible for what duties at xx road transport limited during 1995 to 1998" and "zhang san was responsible for what duties at xx accounting affairs during 1998 to 2000", and the obtained answers were "sponsoring accounting" and "project manager", respectively.

In the deep multi-round template, different levels are interdependent, and if a question at a certain level has no corresponding answer, the relationship is not existed and needs to be abandoned; if there is a corresponding answer, it needs to be retained. It should be noted that, for the levels other than the last level, the obtained answers may include a plurality of answers, but the answer obtained by the last level is usually one, so in order to ensure the effect of the model, two reading understanding models can be obtained by Fine _ tune, one is a multi-answer model (corresponding to the above first artificial intelligence model) and the other is a single-answer model (corresponding to the above second artificial intelligence model), wherein the Fine _ tune process corresponds to the above model training process. Thus, for the levels except the last level, a multi-answer model is used for searching answers; for the last level, a single answer model is used to find the answer.

Of course, it is also possible to train only a single-answer model and determine an answer corresponding to the question of each level through the single-answer model. Yet another way is to output all answers with probabilities exceeding a set probability threshold, i.e. not just one answer, when using the single answer model.

4) The multi-answer reading understands the corpus.

The multi-answer reading understanding corpus corresponds to the first training data of the above, and in the corpus, one question corresponds to a plurality of answers. Therefore, the weight parameters of the multi-answer model can be adapted to the scene of a question corresponding to a plurality of answers.

5) The single answer reads the understanding corpus.

The single answer reading understands that a corpus in which a question corresponds to an answer corresponds to the above second training data. Thus, the weighting parameters of the single-answer model can be adapted to the scene of one answer corresponding to one question.

6) And (6) putting data on a line.

After multi-answer extraction is carried out through the multi-answer model and single-answer extraction is carried out through the single-answer model, the knowledge graph can be constructed according to the obtained question-answer pairs, so that intelligent question-answer service is provided based on the knowledge graph, and online experience is carried out. In order to ensure the accuracy of the intelligent question-answering service, the question-answering pair can be manually checked before the knowledge graph is constructed.

The embodiment of the application carries out knowledge extraction on the unstructured text through the depth multi-round template, improves the convenience of knowledge extraction and knowledge map construction, and greatly reduces labor cost.

Continuing with the exemplary structure of the artificial intelligence based data processing apparatus 455 provided by the embodiments of the present application implemented as software modules, in some embodiments, as shown in fig. 2, the software modules stored in the artificial intelligence based data processing apparatus 455 of the memory 450 may include: the text acquisition module 4551 is configured to acquire a text to be subjected to knowledge extraction; a traversing module 4552, configured to acquire a knowledge extraction template of a problem including a plurality of hierarchies, traverse the plurality of hierarchies according to a hierarchy order of the plurality of hierarchies, and perform the following processing for each traversed hierarchy: performing answer prediction processing by combining the traversed hierarchy question and the text to obtain an answer corresponding to the traversed hierarchy question, and updating the next hierarchy question according to the answer corresponding to the traversed hierarchy question so as to perform answer prediction processing of the next hierarchy by combining the next hierarchy question and the text; a building module 4553, configured to build a knowledge graph according to the question and the corresponding answer of each level in the knowledge extraction template.

In some embodiments, the traversal module 4552 is further configured to: when the traversed level is any one of the plurality of levels except the last level, performing probability mapping processing by combining the questions and the texts of the traversed level to obtain a plurality of candidate answers in the texts and the probability of each candidate answer, and determining the candidate answer with the probability greater than a first probability threshold value as the answer corresponding to the question of the traversed level; and when the traversed hierarchy is the last hierarchy, performing probability mapping processing by combining the question and the text of the last hierarchy to obtain a plurality of candidate answers in the text and the probability of each candidate answer, and determining the candidate answer with the probability greater than a second probability threshold and the maximum probability as the answer corresponding to the question of the last hierarchy.

In some embodiments, the traversal module 4552 is further configured to: performing probability mapping processing on the traversed problem and the text of the hierarchy through a first artificial intelligence model; each sample question for training the first artificial intelligence model corresponds to a plurality of sample answers; performing probability mapping processing on the problem and the text of the last level through a second artificial intelligence model; each sample question for training the second artificial intelligence model corresponds to one sample answer.

In some embodiments, the artificial intelligence based data processing device 455 further comprises: the sample processing module is used for performing probability mapping processing on the sample text and the corresponding sample questions through a first artificial intelligence model to obtain a plurality of candidate answers in the sample text and the probability of each candidate answer, and determining the candidate answers with the probability greater than a first probability threshold value as answers to be compared; and the parameter updating module is used for performing back propagation in the first artificial intelligent model according to the answers to be compared and the difference between the sample answers corresponding to the sample questions, and updating the weight parameters of the first artificial intelligent model in the back propagation process.

In some embodiments, the traversal module 4552 is further configured to: performing probability mapping processing on the traversed problem and the text of the hierarchy through a third artificial intelligence model; performing probability mapping processing on the problem and the text of the last level through a third artificial intelligence model; each sample question for training the third artificial intelligence model corresponds to one sample answer.

In some embodiments, the traversal module 4552 is further configured to: and filling the answer to a part to be filled in the question of the next level aiming at each answer corresponding to the question of the traversed level.

In some embodiments, the building module 4553 is further configured to: constructing a triple according to the question and the corresponding answer of each level in the knowledge extraction template so as to construct a knowledge graph according to a plurality of triples; wherein the triplets include entities in the question, attributes, and attribute values in the answer.

In some embodiments, the artificial intelligence based data processing device 455 further comprises: a selecting module for selecting a first entity and a second entity from a plurality of triples; the same attribute determining module is used for determining the same attribute in the plurality of attributes corresponding to the first entity and the plurality of attributes corresponding to the second entity; the similarity determining module is used for determining the similarity between the attribute values of the same attribute corresponding to the first entity and the attribute values of the same attribute corresponding to the second entity; and the fusion module is used for fusing the first entity and the second entity when the similarity meets the similarity condition.

In some embodiments, the number of identical attributes is multiple; the fusion module is further configured to: any one of the following processes is performed: acquiring the weight corresponding to each same attribute, weighting the similarity corresponding to a plurality of same attributes according to the weight to obtain entity similarity, and fusing the first entity and the second entity when the entity similarity is greater than an entity similarity threshold; and acquiring an attribute similarity threshold corresponding to each same attribute, and fusing the first entity and the second entity when the similarity corresponding to the same attributes is greater than the corresponding attribute similarity threshold.

In some embodiments, the number of the knowledge extraction templates is multiple, and each knowledge extraction template corresponds to a set entity type; the traversal module 4552 is further configured to: the recognition module is used for recognizing the named entities of the text to obtain a plurality of named entities in the text and the entity type of each named entity; the matching module is used for matching the entity type of the named entity in the text with a plurality of set entity types; and the template determining module is used for acquiring a knowledge extraction template corresponding to the successfully matched set entity type when the matching is successful, so as to carry out answer prediction processing.

In some embodiments, the artificial intelligence based data processing device 455 further comprises: the syntax analysis module is used for carrying out dependency syntax processing on a plurality of named entities in the text to obtain a plurality of dependency relationships; and the screening module is used for screening the named entity with the grammar type as the subject in the text according to the subject-predicate relationship in the dependency relationships to serve as the named entity for matching with the multiple set entity types.

In some embodiments, the artificial intelligence based data processing device 455 further comprises: the extraction module is used for extracting titles of a plurality of levels and a level sequence of the plurality of levels in the text according to the level format when the text conforms to the set level format; the structured traversal module is used for traversing a plurality of levels according to the level sequence and executing the following processing aiming at each traversed level: and constructing the problems of the traversed hierarchy according to the titles of the traversed hierarchy, and determining the title of the next hierarchy as the answer corresponding to the problems of the traversed hierarchy.

In some embodiments, the artificial intelligence based data processing device 455 further comprises: the answer format acquisition module is used for acquiring a plurality of standby answer formats and a uniform answer format corresponding to the question of each hierarchy in a plurality of hierarchies; the answer deleting module is used for deleting an answer when the answer corresponding to the question of any one of the multiple levels fails to be matched with the corresponding multiple standby answer formats; and the answer updating module is used for updating the successfully matched answers to accord with the corresponding uniform answer format when the answers corresponding to the questions of any one of the multiple hierarchies are successfully matched with any one of the corresponding standby answer formats.

Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, so that the computer device executes the artificial intelligence based data processing method according to the embodiment of the present application.

Embodiments of the present application provide a computer-readable storage medium having stored therein executable instructions that, when executed by a processor, cause the processor to perform a method provided by embodiments of the present application, for example, an artificial intelligence based data processing method as illustrated in fig. 3A, fig. 3B, fig. 3C, fig. 3D, or fig. 3E. Note that the computer includes various computing devices including a terminal device and a server.

In some embodiments, the computer-readable storage medium may be memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

In some embodiments, executable instructions may be written in any form of programming language (including compiled or interpreted languages), in the form of programs, software modules, scripts or code, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

By way of example, executable instructions may correspond, but do not necessarily have to correspond, to files in a file system, and may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a hypertext markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).

By way of example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.

In summary, the following technical effects can be achieved through the embodiments of the present application:

1) for unstructured texts, knowledge extraction is carried out through a knowledge extraction template; in the structured document, a plurality of hierarchical titles and hierarchical orders are extracted according to a set hierarchical format, and knowledge is extracted. Therefore, convenience of knowledge extraction and knowledge graph construction can be improved, and labor cost is reduced.

2) And multi-answer extraction is carried out on the levels except the last level, and single-answer extraction is carried out on the last level, so that the pertinence to the problems of different levels is improved, and the accuracy of the obtained answers is further improved.

3) The entity types of named entities in the text are matched with the set entity types of the knowledge extraction template, so that the intelligent selection of the knowledge extraction template is realized, and the selected knowledge extraction template can be well suitable for the text; meanwhile, the knowledge extraction template can be accurately acquired, and communication resource consumption during data transmission is reduced.

4) And the obtained answer is subjected to data cleaning, namely the cleaning is carried out in a uniform format, so that the validity of the answer is improved, and the data quality is ensured.

5) By fusing the two entities with higher similarity, the comprehensiveness and effectiveness of the constructed knowledge graph are improved, and more effective question and answer service can be provided.

The above description is only an example of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method based on artificial intelligence is characterized by comprising the following steps:

acquiring a text to be subjected to knowledge extraction;

2. The data processing method according to claim 1, wherein performing answer prediction processing in combination with the traversed hierarchy question and the text to obtain an answer corresponding to the traversed hierarchy question comprises:

when the traversed hierarchy is any one of the plurality of hierarchies except the last hierarchy, probability mapping processing is carried out by combining the questions of the traversed hierarchy and the text to obtain a plurality of candidate answers in the text and the probability of each candidate answer, and

determining candidate answers with the probability larger than a first probability threshold value as answers corresponding to the traversed hierarchy questions;

when the traversed hierarchy is the last hierarchy, probability mapping processing is carried out by combining the question of the last hierarchy and the text to obtain a plurality of candidate answers in the text and the probability of each candidate answer, and

and determining the candidate answer with the probability greater than a second probability threshold and the maximum probability as the answer corresponding to the question of the last level.

3. The data processing method of claim 2,

the probability mapping processing is carried out by combining the problems of the traversed hierarchy and the texts, and comprises the following steps:

performing probability mapping processing on the traversed problem of the hierarchy and the text through a first artificial intelligence model;

each sample question for training the first artificial intelligence model corresponds to a plurality of sample answers;

performing probability mapping processing by combining the question of the last hierarchy and the text, wherein the probability mapping processing comprises the following steps:

performing probability mapping processing on the problem of the last level and the text through a second artificial intelligence model;

wherein each sample question used for training the second artificial intelligence model corresponds to one sample answer.

4. The data processing method of claim 3, further comprising:

performing probability mapping processing on a sample text and corresponding sample questions through the first artificial intelligence model to obtain a plurality of candidate answers in the sample text and the probability of each candidate answer, and obtaining the probability of each candidate answer

Determining candidate answers with the probability larger than the first probability threshold value as answers to be compared;

according to the difference between the answers to be compared and a plurality of sample answers corresponding to the sample questions, performing back propagation in the first artificial intelligence model, and performing back propagation on the answers to be compared and the sample answers

And updating the weight parameters of the first artificial intelligence model in the process of back propagation.

5. The data processing method of claim 2,

performing probability mapping processing on the traversed problem of the hierarchy and the text through a third artificial intelligence model;

performing probability mapping processing on the problem of the last level and the text through the third artificial intelligence model;

wherein each sample question used for training the third artificial intelligence model corresponds to one sample answer.

6. The data processing method according to any one of claims 1 to 5,

the updating the question of the next hierarchy according to the answer corresponding to the traversed question of the hierarchy includes:

for each answer corresponding to the traversed level question, filling the answer to a part to be filled in the next level question;

constructing a knowledge graph according to the questions and corresponding answers of each level in the knowledge extraction template, wherein the construction comprises the following steps:

constructing a triple according to the question and the corresponding answer of each level in the knowledge extraction template so as to construct a knowledge graph according to a plurality of triples;

wherein the triplets include entities in the question, attributes, and attribute values in the answer.

7. The data processing method of claim 6, further comprising:

selecting a first entity and a second entity from a plurality of said triples;

determining the same attribute from the plurality of attributes corresponding to the first entity and the plurality of attributes corresponding to the second entity;

determining similarity between the attribute values of the same attribute corresponding to the first entity and the attribute values of the same attribute corresponding to the second entity;

and when the similarity meets a similarity condition, carrying out fusion processing on the first entity and the second entity.

8. The data processing method of claim 7,

the number of the same attributes is multiple;

when the similarity meets a similarity condition, performing fusion processing on the first entity and the second entity, including:

any one of the following processes is performed:

obtaining the weight corresponding to each same attribute, weighting the similarity corresponding to a plurality of same attributes according to the weight to obtain entity similarity, and obtaining the entity similarity

When the entity similarity is larger than an entity similarity threshold value, carrying out fusion processing on the first entity and the second entity;

obtaining attribute similarity threshold corresponding to each same attribute, and

and when the similarity corresponding to the same attributes is larger than the corresponding attribute similarity threshold, performing fusion processing on the first entity and the second entity.

9. The data processing method according to any one of claims 1 to 5,

the number of the knowledge extraction templates is multiple, and each knowledge extraction template corresponds to a set entity type;

the obtaining a knowledge extraction template for a problem comprising a plurality of tiers, comprising:

carrying out named entity identification on the text to obtain a plurality of named entities in the text and an entity type of each named entity;

matching the entity type of the named entity in the text with a plurality of set entity types;

and when the matching is successful, acquiring a knowledge extraction template corresponding to the set entity type successfully matched for carrying out answer prediction processing.

10. The data processing method of claim 9, wherein before matching the entity type of the named entity in the text with the plurality of set entity types, further comprising:

carrying out dependency syntax processing on the named entities in the text to obtain a plurality of dependency relationships;

and screening out the named entity with the grammar type as the subject in the text according to the main and subordinate relations in the plurality of dependency relations, and taking the named entity as the named entity for matching with the plurality of set entity types.

11. The data processing method according to any one of claims 1 to 5, further comprising:

when the text conforms to a set hierarchical format, extracting titles of a plurality of hierarchies in the text and a hierarchical sequence of the plurality of hierarchies according to the hierarchical format;

traversing the plurality of levels according to the level order, and for each level traversed performing the following:

constructing the problem of the traversed hierarchy according to the title of the traversed hierarchy, and

and determining the title of the next hierarchy as the answer corresponding to the question of the traversed hierarchy.

12. The data processing method according to any one of claims 1 to 5, further comprising:

obtaining a plurality of standby answer formats and a uniform answer format corresponding to the question of each hierarchy in the plurality of hierarchies;

deleting an answer corresponding to the question of any one of the multiple levels when the answer is matched with the corresponding multiple standby answer formats in a failure mode;

and when the answer corresponding to the question of any one of the multiple levels is successfully matched with any one of the corresponding standby answer formats, updating the successfully matched answer to accord with the corresponding uniform answer format.

13. An artificial intelligence-based data processing apparatus, comprising:

14. An electronic device, comprising:

a memory for storing executable instructions;

a processor for implementing the artificial intelligence based data processing method of any one of claims 1 to 12 when executing executable instructions stored in the memory.

15. A computer-readable storage medium storing executable instructions for implementing the artificial intelligence based data processing method of any one of claims 1 to 12 when executed by a processor.