WO2020133291A1 - 文本实体识别方法、装置、计算机设备及存储介质 - Google Patents

文本实体识别方法、装置、计算机设备及存储介质 Download PDF

Info

Publication number
WO2020133291A1
WO2020133291A1 PCT/CN2018/125107 CN2018125107W WO2020133291A1 WO 2020133291 A1 WO2020133291 A1 WO 2020133291A1 CN 2018125107 W CN2018125107 W CN 2018125107W WO 2020133291 A1 WO2020133291 A1 WO 2020133291A1
Authority
WO
WIPO (PCT)
Prior art keywords
entity
predicted
text
target
entities
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/125107
Other languages
English (en)
French (fr)
Inventor
熊友军
罗沛鹏
廖洪涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ubtech Robotics Corp
Original Assignee
Ubtech Robotics Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ubtech Robotics Corp filed Critical Ubtech Robotics Corp
Priority to PCT/CN2018/125107 priority Critical patent/WO2020133291A1/zh
Publication of WO2020133291A1 publication Critical patent/WO2020133291A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G06F40/295—Named entity recognition

Definitions

  • the present invention relates to the field of computer processing, and in particular to a text entity recognition method, device, computer equipment, and storage medium.
  • deep learning technology is widely used in text entity recognition tasks, especially robot conversation corpus, which needs to obtain entities in order to perform related conversation orchestration.
  • an embodiment of the present invention provides a text entity recognition method.
  • the method includes:
  • the word segmentation mechanism is used to perform word segmentation processing on the text, and the second predicted entity set is extracted according to the word segmentation processing result;
  • the identified target entity set is determined according to the first predicted entity set and the second predicted entity set, and the target entity set is a set of target entities.
  • an embodiment of the present invention provides a text entity recognition device, and the device includes:
  • Text acquisition module used to acquire the text of the entity to be identified
  • An input-output module configured to use the text as an input of an entity recognition model to obtain a first set of predicted entities output by the entity recognition model;
  • the word segmentation extraction module is used to perform word segmentation processing on the text using the word segmentation mechanism, and extract the second predicted entity set according to the word segmentation processing result;
  • the determining module is configured to determine the identified target entity set according to the first predicted entity set and the second predicted entity set, and the target entity set is a set of target entities.
  • an embodiment of the present invention provides a computer device, including a memory and a processor.
  • the memory stores a computer program.
  • the processor is caused to perform the following steps:
  • the word segmentation mechanism is used to perform word segmentation processing on the text, and the second predicted entity set is extracted according to the word segmentation processing result;
  • the identified target entity set is determined according to the first predicted entity set and the second predicted entity set, and the target entity set is a set of target entities.
  • an embodiment of the present invention provides a computer-readable storage medium that stores a computer program.
  • the processor is caused to perform the following steps:
  • the word segmentation mechanism is used to perform word segmentation processing on the text, and the second predicted entity set is extracted according to the word segmentation processing result;
  • the identified target entity set is determined according to the first predicted entity set and the second predicted entity set, and the target entity set is a set of target entities.
  • the above text entity recognition method obtains the text of the entity to be recognized, uses the text as the input of the entity recognition model, obtains the first predicted entity set output by the entity recognition model, and at the same time uses the word segmentation mechanism to perform word segmentation processing on the text, and extracts according to the word segmentation processing result
  • the second predicted entity set is generated, and then the identified target entity set is determined according to the first predicted entity set and the second predicted entity set.
  • the above text entity recognition method extracts the second predicted entity set through the word segmentation mechanism, and then uses the second predicted entity set to supplement the first predicted entity set, that is, uses the entity recognition model and the word segmentation mechanism Together to determine the target entity set, more, more credible, and more accurate entities can be obtained, thereby improving the accuracy of entity recognition.
  • FIG. 1 is an application environment diagram of a text entity recognition method in an embodiment
  • FIG. 2 is a flowchart of a text entity recognition method in an embodiment
  • FIG. 3 is a flowchart of a method for extracting a second set of prediction entities in an embodiment
  • FIG. 5 is a schematic flowchart of a text entity recognition method in an embodiment
  • FIG. 6 is a structural block diagram of a text entity recognition device in an embodiment
  • FIG. 7 is a structural block diagram of a text entity recognition device in another embodiment
  • FIG. 8 is a structural block diagram of a text entity recognition device in yet another embodiment
  • FIG. 9 is an internal structure diagram of a computer device in an embodiment.
  • FIG. 1 is an application environment diagram of a text entity recognition method in an embodiment.
  • the text entity recognition is applied to a text entity recognition system.
  • the text entity recognition system includes a terminal 110 and a server 120.
  • the terminal 110 and the server 120 are connected through a network.
  • the terminal 110 may specifically be a desktop terminal or a mobile terminal, and the mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a notebook computer, and the like.
  • the server 120 may be implemented by an independent server or a server cluster composed of multiple servers.
  • the terminal 110 is used to upload the text of the entity to be recognized to the server 120, and the server 120 is used to receive the text of the entity to be recognized, use the text as an input of the entity recognition model, and obtain a first prediction output by the entity recognition model Entity set; using word segmentation mechanism to perform word segmentation processing on the text, and extract a second predicted entity set according to the word segmentation processing result; determine the identified target entity set according to the first predicted entity set and the second predicted entity set, Then, the obtained target entity set is sent to the terminal 110.
  • the above text entity recognition method may be directly applied to the terminal 110, and the terminal 110 is used to obtain text of an entity to be recognized, use the text as an input of an entity recognition model, and obtain the output of the entity recognition model.
  • a set of predicted entities using a word segmentation mechanism to perform word segmentation processing on the text, extracting a second predicted entity set according to the word segmentation processing result, and determining the identified target entity according to the first predicted entity set and the second predicted entity set set.
  • the text entity recognition method can be applied to a terminal or a server.
  • the method is applied to a terminal as an example.
  • the text entity recognition method specifically includes The following steps:
  • Step 202 Obtain the text of the entity to be identified.
  • the entity refers to keywords in the text.
  • Entity recognition refers to the identification of various entities in the text, such as the names of people, places, institutions, or other uniquely identified keywords.
  • Text entity recognition is a very important basic technology in natural language processing.
  • the user is queried for the music he wants to listen to by identifying the entity in the text. For example, enter “I want to listen to Andy Lau's song”, and by identifying the entity “Andy Lau", you can find the corresponding song list of Andy Lau in the corresponding music library.
  • the text of the entity to be recognized may be obtained directly by the text input by the user, or by obtaining the user's voice, and then obtaining the corresponding text by voice-to-text, of course, other methods may also be used.
  • step 204 the text is used as the input of the entity recognition model to obtain the first set of predicted entities output by the entity recognition model.
  • the entity recognition model refers to a model obtained by training for identifying entities in the text.
  • the entity recognition model is trained based on deep learning methods, such as deep convolutional neural networks.
  • a first predicted entity set is obtained.
  • the first predicted entity set refers to a set of predicted entities.
  • Step 206 Use the word segmentation mechanism to perform word segmentation processing on the text, and extract the second predicted entity set according to the word segmentation processing result.
  • the word segmentation mechanism refers to a mechanism for word segmentation processing of the text, for example, a word segmentation device.
  • the word segmentation mechanism uses a jieba word segmenter, but other word segmentators can also be used.
  • Word segmentation refers to words that divide the text into individual words. For example, suppose the text is "Is the temperature high today", and the result of the word segmentation processing is "Is the temperature high today?" After the word segmentation processing is used to obtain the word segmentation processing result, the entities in the word segmentation processing result are extracted.
  • entities are extracted based on part-of-speech, assuming nouns as entities, then extract the nouns in a sentence as entities, for example, "temperature” in “is today's temperature high” as a noun, then "temperature” As the extracted noun.
  • the entity type is customized, and then the desired entity is extracted according to the entity type. For example, assuming that "Today" is an entity in the entity library, you can set today's entity type to time and store it To the entity library, when extracting the entity, the corresponding entity is extracted according to the custom entity type in the entity library.
  • the entity extracted according to the word segmentation processing result is added to the second predicted entity set.
  • the second predicted entity set refers to a set of extracted entities. Because the entity recognition model may not accurately recognize the entity in all cases, the keywords of the text are extracted by using the word segmentation mechanism as a supplementary mechanism to make up for the deficiency of the entity recognition model.
  • Step 208 Determine the identified target entity set according to the first predicted entity set and the second predicted entity set, where the target entity set is a set of target entities.
  • the identified target entity set can be determined.
  • the target entity set refers to the set of identified target entities.
  • the intersection of the first predicted entity set and the second predicted entity set is taken as the target entity set, that is, the entities that coincide with the first predicted entity set and the second predicted actual set are taken as the target entity.
  • the union of the first predicted entity set and the second predicted entity set is used as the target entity set.
  • the above text entity recognition method obtains the text of the entity to be recognized, uses the text as the input of the entity recognition model, obtains the first predicted entity set output by the entity recognition model, and at the same time uses the word segmentation mechanism to perform word segmentation processing on the text, and extracts based on the word segmentation processing results
  • the second predicted entity set is generated, and then the identified target entity set is determined according to the first predicted entity set and the second predicted entity set.
  • the above text entity recognition method extracts the second predicted entity set through the word segmentation mechanism, and then uses the second predicted entity set to supplement the first predicted entity set, that is, uses an entity recognition model and a word segmentation mechanism Together to determine the target entity set, more, more credible, and more accurate entities can be obtained, which helps to improve the accuracy of text entity recognition.
  • the word segmentation mechanism is used to perform word segmentation processing on the text, and the second predicted entity set is extracted according to the word segmentation processing result, including:
  • Step 206A Perform word segmentation processing on the text through the word segmentation mechanism to obtain multiple words and corresponding word types;
  • the dictionary corresponding to the word segmentation mechanism is established based on the entity library, and the entities in the entity library are added to the word segmentation mechanism as a dictionary.
  • the entity library includes entities and corresponding entity types.
  • the entity type can be customized according to business requirements, which is also convenient for subsequent entity extraction according to the entity type.
  • Word segmentation processing is performed to obtain multiple words, and each word corresponds to a corresponding word type.
  • part-of-speech is used as the word type.
  • Step 206B Extract words that match the entity type from multiple words according to the word type to obtain a second set of predicted entities.
  • the word is an entity in the entity library according to the word type of each word, and if so, the word is used as the second prediction Entities in the entity set.
  • the above extracts entities according to the word segmentation mechanism as a supplementary mechanism of the entity recognition model, which is helpful to improve the accuracy of entity recognition.
  • the identified target entity set is determined according to the first predicted entity set and the second predicted entity set.
  • the target entity set is a set of target entities, including: acquiring the first predicted entity set and the second predicted entity set Intersection, the entity in the intersection as the target entity.
  • the entity obtained by the entity recognition model and the entity extracted by the word segmentation mechanism are taken as an intersection.
  • the entity in the intersection is the target entity, and the target entity is the confident entity.
  • the entities in the intersection are more representative, and one or more target entities can be determined. For example, the intersection of "Shenzhen Municipal Party Committee Office” and “Shenzhen Municipal Party Committee Office” is "Shenzhen", and it can be confirmed that "Shenzhen” is a certain entity.
  • determining the identified target entity according to the first predicted entity set and the second predicted entity set further includes: matching the first entity in the first predicted entity set with the second entity in the second predicted entity set ; When the first entity contains the second entity, the first entity containing the second entity as the target entity; When the second entity contains the first entity, the second entity containing the first entity as Target entity.
  • the entities in the first prediction entity set are called “first entities”, and the entities in the second prediction entity set are called “second entities”.
  • the first entity is matched with the second entity using the longest matching principle, and then the longer entity is retained.
  • the second entity is included in the first entity
  • the first entity including the second entity is used as the target entity.
  • the target entity For example, if the first entity is "Shenzhen University City" and the second entity is "Shenzhen", since "Shenzhen” is included in the first entity, "Shenzhen University City” is reserved as the target entity.
  • the second entity including the first entity is used as the target entity.
  • the above text entity recognition method further includes:
  • Step 210 Match the target entity in the target entity set with the entity in the preset entity library.
  • the target entity is in the preset entity library, go to step 212.
  • the target entity is not in the preset entity library, go to step 214 .
  • the entity library refers to a database that stores entities, which contains many entities.
  • the entity library is generally customized to establish according to business needs. After predicting the target entity, find in the preset entity library whether the predicted target entity is in the entity library. If the entity exists in the entity library, it means that the target entity is a confirmation entity. If the entity is not found in the entity library, then the entity has a certain uncertainty, and it needs to be clearly informed that the entity is the predicted entity when it is returned, and is not in the entity library.
  • step 212 it is determined that the target entity is a first type entity.
  • Step 214 Determine that the target entity is a second type entity.
  • the entity that exists in the entity library is called “the first type entity”, and the entity that does not exist in the entity library is called the “second type entity”.
  • the first type entity if it is the first type of entity, it returns "confirmation”, if it is the second type of entity, it returns "prediction”.
  • the above text entity recognition method further includes: when the target entity set contains English entities, acquiring multiple deformed English entities corresponding to the English entities; and performing the multiple deformed English entities with the entities in the preset entity library Match; when at least one deformed English entity among the plurality of deformed English entities is in the preset entity library, the corresponding English entity is determined to be the first type entity, otherwise, the corresponding English entity is determined to be the second type entity.
  • the English entity refers to an entity that exists in English.
  • the target entity is an English entity
  • the English entity is rewritten into multiple deformed English entities. For example, capitalize each first letter in an English entity, capitalize the first character, lowercase all letters, and capitalize a single English word.
  • match the multiple deformed English entities match the multiple deformed English entities with the entities in the preset entity library, and then when at least one of the deformed English entities is in the preset entity library, it is considered
  • the English entity is in the preset entity library, so that the English entity is determined as the first type entity, otherwise, it is determined as the second type entity.
  • the identified English entity is computer. Since the entity store may be Computer or COMPUTER, if the query is directly performed, it may not be queried, so by modifying the case of computer and then querying, you can Queryed in the entity library.
  • the above text entity recognition method further includes: when the first predicted entity set is empty, acquiring the entities and corresponding entity types in the second predicted entity set; acquiring the intention of the text, and determining the target entity according to the intention of the text Type: According to the target entity type, the target entity that matches the target entity type is selected from the second predicted entity set.
  • Text intent refers to the intent of text recognition, for example, to search for music or movies.
  • the entity of Andy Lau has two entity types, one is a singer and the other is an actor. If the text is intended to be a music scene, then Andy Lau’s corresponding target entity type should be a singer. If the target entity in the second prediction entity set is Andy Lau and the corresponding entity type is singer, Andy Lau is the target entity that matches the target entity type. If the corresponding entity type is an actor, then Andy Lau does not meet the target entity type, so it is a non-target entity.
  • FIG. 5 is a schematic flowchart of a text entity recognition method in an embodiment.
  • obtain the text of the entity to be recognized for example, receive the text input by the user
  • use the text as the input of the entity recognition model to obtain the first set of predicted entities
  • at the same time use the word segmentation mechanism to extract the second set of predicted entities
  • use the longest matching principle to check the first predicted entity Concentration and the second predicted entity concentrated the existence of the entity containing the relationship, leaving the longest. Then, check whether there are English entities in the remaining entities (ie, target entities).
  • a text entity recognition device which includes:
  • the text obtaining module 602 is used to obtain the text of the entity to be identified
  • the input and output module 604 is configured to use the text as an input of an entity recognition model to obtain a first set of predicted entities output by the entity recognition model;
  • the word segmentation extraction module 606 is used to perform word segmentation processing on the text by using a word segmentation mechanism, and extract a second predicted entity set according to the word segmentation processing result;
  • the determining module 608 is configured to determine the identified target entity set according to the first predicted entity set and the second predicted entity set, where the target entity set is a set of target entities.
  • the word segmentation extraction module is further used to perform word segmentation processing on the text through a word segmentation mechanism to obtain multiple words and corresponding word types; according to the word types, extracts matches from the multiple words The words of the entity type obtain the second predicted entity set.
  • the determination module is further configured to obtain an intersection of the first set of predicted entities and the second set of predicted entities, and use an entity in the intersection as the target entity.
  • the determination module is further used to match the first entity in the first prediction entity set with the second entity in the second prediction entity set; when the first entity includes the first In the case of two entities, the first entity containing the second entity is used as the target entity; when the second entity contains the first entity, the second entity containing the first entity is used as the target entity.
  • the device further includes:
  • the matching module 610 is configured to match the target entity in the target entity set with an entity in a preset entity library, and when the target entity is in the preset entity library, determine that the target entity is the first For a class entity, when the target entity is not in the preset entity library, it is determined that the target entity is a second class entity.
  • the matching module is further used to obtain a plurality of deformed English entities corresponding to the English entity when the target entity set contains English entities, and compare the plurality of deformed English entities with the pre- Assume that the entities in the entity library are matched.
  • the matching module is further used to obtain a plurality of deformed English entities corresponding to the English entity when the target entity set contains English entities, and compare the plurality of deformed English entities with the pre- Assume that the entities in the entity library are matched.
  • the device further includes:
  • the type obtaining module 612 is configured to obtain the entity and the corresponding entity type in the second predicted entity set when the first predicted entity set is empty;
  • the entity type determination module 614 is used to obtain the intention of the text and determine the target entity type according to the intention of the text;
  • the screening module 616 is configured to screen out target entities matching the target entity type from the second predicted entity set according to the target entity type.
  • FIG. 9 shows an internal structure diagram of a computer device in an embodiment.
  • the computer device may be a terminal or a server.
  • the computer device includes a processor, a memory, and a network interface connected through a system bus.
  • the memory includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium of the computer device stores an operating system and may also store a computer program.
  • the processor may cause the processor to implement a text entity recognition method.
  • a computer program may also be stored in the internal memory.
  • the processor may be caused to execute the text entity recognition method.
  • the network interface is used to communicate with the outside world.
  • FIG. 9 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied.
  • the specific computer equipment may It includes more or fewer components than shown in the figure, or some components are combined, or have a different component arrangement.
  • the text entity recognition method provided in this application may be implemented in the form of a computer program, and the computer program may run on a computer device as shown in FIG. 9.
  • Various program templates constituting the text entity recognition device can be stored in the memory of the computer device.
  • a computer device includes a memory and a processor.
  • the memory stores a computer program.
  • the processor is caused to perform the following steps: obtain text of an entity to be recognized;
  • the text is used as the input of the entity recognition model to obtain the first predicted entity set output by the entity recognition model;
  • the word segmentation mechanism is used to perform word segmentation processing on the text, and the second predicted entity set is extracted according to the word segmentation processing result; according to the first The predicted entity set and the second predicted entity set determine the identified target entity set, and the target entity set is a set of target entities.
  • the word segmentation mechanism is used to perform word segmentation processing on the text, and the second predicted entity set is extracted according to the word segmentation processing result, which includes: performing word segmentation processing on the text through the word segmentation mechanism to obtain multiple words and corresponding The word type of the word; extracting words matching the entity type from the plurality of words according to the word type to obtain a second set of predicted entities.
  • the determining the identified target entity set according to the first predicted entity set and the second predicted entity set includes: acquiring the first predicted entity set and the second predicted entity set , The entity in the intersection is used as the target entity.
  • the determining the identified target entity according to the first predicted entity set and the second predicted entity set further includes: combining the first entity in the first predicted entity set with the first Match the second entity in the second prediction entity set; when the first entity contains the second entity, the first entity containing the second entity is used as the target entity; when the second entity contains the first entity As an entity, the second entity containing the first entity is used as the target entity.
  • the computer program when executed by the processor, it is also used to execute The following steps: match the target entity in the target entity set with an entity in a preset entity library, and when the target entity is in the preset entity library, determine that the target entity is a first-type entity; When the target entity is not in the preset entity library, it is determined that the target entity is a second type entity.
  • the computer program when executed by the processor, the computer program is further used to perform the following steps: when the target entity set includes English entities, acquiring multiple deformed English entities corresponding to the English entities; Matching the plurality of deformed English entities with entities in the preset entity library; when at least one deformed English entity in the plurality of deformed English entities is in the preset entity library, the corresponding The English entity is a first type entity, otherwise, it is determined that the corresponding English entity is a second type entity.
  • the computer program when executed by the processor, it is also used to perform the following steps: when the first set of predicted entities is empty, obtain the entities in the second set of predicted entities and the corresponding Entity type; acquiring the intention of the text, and determining a target entity type according to the intention of the text; filtering out target entities that match the target entity type from the second predicted entity set according to the target entity type.
  • a computer-readable storage medium storing a computer program, which when executed by a processor, causes the processor to perform the following steps: acquiring text of an entity to be recognized; using the text as an input of an entity recognition model, Obtain the first predicted entity set output by the entity recognition model; use a word segmentation mechanism to perform word segmentation processing on the text, and extract a second predicted entity set according to the word segmentation processing result; based on the first predicted entity set and the second The predicted entity set determines the identified target entity set, and the target entity set is a set of target entities.
  • the word segmentation mechanism is used to perform word segmentation processing on the text, and the second predicted entity set is extracted according to the word segmentation processing result, which includes: performing word segmentation processing on the text through the word segmentation mechanism to obtain multiple words and corresponding The word type of the word; extracting words matching the entity type from the plurality of words according to the word type to obtain a second set of predicted entities.
  • the determining the identified target entity set according to the first predicted entity set and the second predicted entity set includes: acquiring the first predicted entity set and the second predicted entity set , The entity in the intersection is used as the target entity.
  • the determining the identified target entity according to the first predicted entity set and the second predicted entity set further includes: combining the first entity in the first predicted entity set with the first Match the second entity in the second prediction entity set; when the first entity contains the second entity, the first entity containing the second entity is used as the target entity; when the second entity contains the first entity As an entity, the second entity containing the first entity is used as the target entity.
  • the computer program when executed by the processor, it is also used to execute The following steps: match the target entity in the target entity set with an entity in a preset entity library, and when the target entity is in the preset entity library, determine that the target entity is a first-type entity; When the target entity is not in the preset entity library, it is determined that the target entity is a second type entity.
  • the computer program when executed by the processor, the computer program is further used to perform the following steps: when the target entity set includes English entities, acquiring multiple deformed English entities corresponding to the English entities; Matching the plurality of deformed English entities with entities in the preset entity library; when at least one deformed English entity in the plurality of deformed English entities is in the preset entity library, the corresponding The English entity is a first type entity, otherwise, it is determined that the corresponding English entity is a second type entity.
  • the computer program when executed by the processor, it is also used to perform the following steps: when the first set of predicted entities is empty, obtain the entities in the second set of predicted entities and the corresponding Entity type; acquiring the intention of the text, and determining a target entity type according to the intention of the text; filtering out target entities that match the target entity type from the second predicted entity set according to the target entity type.
  • Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory can include random access memory (RAM) or external cache memory.
  • RAM random access memory
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDRSDRAM double data rate SDRAM
  • ESDRAM enhanced SDRAM
  • SLDRAM synchronous chain Road (Synchlink) DRAM
  • RDRAM direct RAM
  • DRAM direct memory bus dynamic RAM
  • RDRAM memory bus dynamic RAM

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Machine Translation (AREA)

Abstract

一种文本实体识别方法，该方法包括：获取待识别实体的文本（S202）；将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集（S204）；采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集（S206）；根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合（S208）。通过采用分词机制作为补充识别，提高了实体识别的准确率。此外，还提出了一种文本实体识别装置、计算机设备及存储介质。

Description

文本实体识别方法、装置、计算机设备及存储介质 技术领域
本发明涉及计算机处理领域，尤其是涉及一种文本实体识别方法、装置、计算机设备及存储介质。
 背景技术
随着深度学习技术的成熟以及计算机性能的提升，深度学习技术广泛运用于文本实体识别任务中，尤其是机器人对话语料，需要获取实体才能进行相关的会话编排。
 技术问题
然而，深度学习实体识别模型预测的准确率具有一定的瓶颈（只能达到80%到90%之间），所以传统的语料中的实体识别的准确率偏低。
 技术解决方案
基于此，有必要针对上述问题，提供了一种实体识别准确率高的文本实体识别方法、装置、计算机设备及存储介质。
第一方面，本发明实施例提供一种文本实体识别方法，所述方法包括：
获取待识别实体的文本；
将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；
采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；
根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合。
第二方面，本发明实施例提供一种文本实体识别装置，所述装置包括：
文本获取模块，用于获取待识别实体的文本；
输入输出模块，用于将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；
分词提取模块，用于采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；
确定模块，用于根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合。
第三方面，本发明实施例提供一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述计算机程序被所述处理器执行时，使得所述处理器执行如下步骤：
获取待识别实体的文本；
将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；
采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；
根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合。
第四方面，本发明实施例提供一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时，使得所述处理器执行如下步骤：
获取待识别实体的文本；
将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；
采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；
根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合。
 有益效果
上述文本实体识别方法，通过获取待识别实体的文本，将文本作为实体识别模型的输入，获取实体识别模型输出的第一预测实体集，同时采用分词机制对文本进行分词处理，根据分词处理结果提取出第二预测实体集，然后根据第一预测实体集和第二预测实体集确定识别得到的目标实体集。上述文本实体识别方法在得到第一预测实体集后，通过分词机制提取出第二预测实体集，然后采用第二预测实体集对第一预测实体集进行补充识别，即采用实体识别模型和分词机制共同来确定目标实体集，能够得到更多、更可信、更准确的实体，从而提高了实体识别的准确率。
 附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图示出的结构获得其他的附图。
图1为一个实施例中文本实体识别方法的应用环境图；
图2为一个实施例中文本实体识别方法的流程图；
图3为一个实施例中提取第二预测实体集的方法流程图；
图4为另一个实施例中文本实体识别方法的流程图；
图5为一个实施例中文本实体识别方法的流程示意图；
图6为一个实施例中文本实体识别装置的结构框图；
图7为另一个实施例中文本实体识别装置的结构框图；
图8为又一个实施例中文本实体识别装置的结构框图；
图9为一个实施例中计算机设备的内部结构图。
 本发明的实施方式
图1为一个实施例中文本实体识别方法的应用环境图。参照图1，该文本实体识别应用于文本实体识别系统。该文本实体识别系统包括终端110和服务器120。终端110和服务器120通过网络连接，终端110具体可以是台式终端或移动终端，移动终端具体可以是手机、平板电脑、笔记本电脑等中的至少一种。服务器120可以用独立的服务器或者是多个服务器组成的服务器集群来实现。终端110用于获取到的待识别实体的文本上传到服务器120，服务器120用于接收待识别实体的文本，将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，然后将得到的目标实体集发送给终端110。
在另一个实施例中，上述文本实体识别方法可以直接应用于终端110，终端110用于获取待识别实体的文本，将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集，采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集，根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集。
如图2所示，提出了一种文本实体识别方法，该文本实体识别方法可以应用于终端，也可以应用于服务器，本实施例中以应用于终端为例说明，该文本实体识别方法具体包括以下步骤：
步骤202，获取待识别实体的文本。
其中，实体是指文本中的关键字。实体识别是指识别文本中的各种实体，如人名、地名、机构名或其他特有标识的关键字。文本实体识别是自然语言处理中非常重要的一项基础性技术。在一个音乐应用的场景中，通过识别文本中的实体来为用户查询想要听的音乐。比如，输入“我想听刘德华的歌”，通过识别实体“刘德华”就可以在相应的音乐曲库中查找到刘德华相应的歌曲列表。
在一个实施例中，待识别实体的文本的获取方式可以是直接获取用户输入的文本，也可以是通过获取用户语音，然后通过语音转文字的方式得到相应的文本，当然也可以是其他方式。
步骤204，将文本作为实体识别模型的输入，获取实体识别模型输出的第一预测实体集。
其中，实体识别模型是指训练得到的用于对文本中的实体进行识别的模型。在一个实施例中，实体识别模型是基于深度学习方法训练得到的，比如，深度卷积神经网络。通过将文本作为实体识别模型的输入，然后得到输出的第一预测实体集，第一预测实体集是指预测得到的实体的集合。
步骤206，采用分词机制对文本进行分词处理，根据分词处理结果提取出第二预测实体集。
其中，分词机制是指用于对文本进行分词处理的机制，比如，分词器。在一个实施例中，分词机制采用jieba（结巴）分词器，当然也可以采用其他分词器。分词是指将文本切分为一个个的词语。比如，假设文本为“今天温度高吗”，相应地分词处理结果为“今天 温度 高 吗”。在使用分词处理得到分词处理结果后，提取分词处理结果中的实体。在一个实施例中，根据词性来提取出实体，假设将名词作为实体，那么提取一句话中的名词作为实体，比如，“今天 温度 高 吗”中的“温度”为名词，那么将“温度”作为提取到的名词。在另一个实施例中，自定义设置实体类型，然后根据实体类型来提取出想要的实体，比如，假设“今天”是实体库中的实体，可以将今天的实体类型设置为time，并存储到实体库中，在提取实体时，根据实体库中自定义的实体类型来提取得到相应的实体。将根据分词处理结果提取出的实体加入第二预测实体集。第二预测实体集是指提取到的实体的集合。由于实体识别模型不一定在所有情况下都能准确地识别到实体，通过采用分词机制提取到文本的关键词，作为补充机制在弥补实体识别模型的不足。
步骤208，根据第一预测实体集和第二预测实体集确定识别得到的目标实体集，目标实体集为目标实体的集合。
其中，在确定了第一预测实体集和第二预测实体集后，就可以确定识别得到的目标实体集。目标实体集是指识别得到的目标实体的集合。在一个实施例中，将第一预测实体集和第二预测实体集的交集作为目标实体集，即将第一预测实体集和第二预测实际集中的重合的实体作为目标实体。在另一个实施例中，将第一预测实体集和第二预测实体集的并集作为目标实体集。通过将采用分词机制提取出的第二预测实体集作为后处理机制来对实体识别模型得到的第一预测实体集进行补充和确认，有利于得到更多、更可信、更准确的实体，从而提高了实体识别的准确度。
上述文本实体识别方法，通过获取待识别实体的文本，将文本作为实体识别模型的输入，获取实体识别模型输出的第一预测实体集，同时采用分词机制对文本进行分词处理，根据分词处理结果提取出第二预测实体集，然后根据第一预测实体集和第二预测实体集确定识别得到的目标实体集。上述文本实体识别方法在得到第一预测实体集后，通过分词机制提取出第二预测实体集，然后采用第二预测实体集对第一预测实体集进行补充识别，即采用实体识别模型和分词机制共同来确定目标实体集，能够得到更多、更可信、更准确的实体，从而有利于提高文本实体识别的准确率。
如图3所示，在一个实施例中，采用分词机制对文本进行分词处理，根据分词处理结果提取出第二预测实体集，包括：
步骤206A，通过分词机制对文本进行分词处理，得到多个词语和相应的词语类型；
其中，分词机制对应的词典是基于实体库建立的，将实体库中的实体作为词典加入到分词机制中。实体库中包括实体和相应的实体类型。实体类型可以根据业务需求自定义设置，这样也便于后续根据实体类型进行实体的提取。
对文本进行分词处理得到多个词语，每个词语都对应有相应的词语类型。在一个实施例中，将词性作为词语类型。对于实体库中的实体可以自定义相应的实体类型，并将实体类型作为该词的词语类型。比如，自定义“今天”这个词语的实体类型为“time”，将“time”作为今天的词语类型，对“今天温度高吗”进行分词后，得到“今天（time）温度（n）高（adj）吗（y）”，其中，括号里面的time、n、adj、y分别表示相应的词语类型。time为自定义的词语类型，n为名词，adj为形容词，y为语气助词。由于time为实体类型，所以后续根据词语类型就可以确定哪个词语是实体。
步骤206B，根据词语类型从多个词语中提取出符合实体类型的词语，得到第二预测实体集。
其中，通过预先对每个实体的类型进行自定义，获取到多个词语后，根据每个词语的词语类型来确定该词语是否为实体库中的实体，若是，则将该词语作为第二预测实体集中的实体。上述根据分词机制提取出实体，作为实体识别模型的补充机制，有利于提高实体识别的准确度。
在一个实施例中，根据第一预测实体集和第二预测实体集确定识别得到的目标实体集，目标实体集是目标实体的集合，包括：获取第一预测实体集和第二预测实体集的交集，将交集中的实体作为目标实体。
其中，为了提高实体识别的准确率，通过对实体识别模型识别得到的实体与分词机制提取到的实体取交集，交集中的实体为目标实体，该目标实体为确信实体。交集部分的实体比较具有代表性，可以确定一个到多个目标实体。如：“深圳 市委办公室”与“深圳 市委 办公室”的交集为“深圳”，可以确认“深圳”为确信的实体。
在一个实施例中，根据第一预测实体集和第二预测实体集确定识别得到的目标实体，还包括：将第一预测实体集中的第一实体与第二预测实体集中的第二实体进行匹配；当第一实体中包含有第二实体时，将包含有第二实体的第一实体作为目标实体；当第二实体中包含有第一实体时，将包含有第一实体的第二实体作为目标实体。
其中，为了区分，将第一预测实体集中的实体称为“第一实体”，将第二预测实体集中的实体称为“第二实体”。为了更多更好地保留实体信息，在得到第一预测实体集和第二预测实体集后，采用最长匹配原则将第一实体与第二实体进行匹配，然后保留较长的实体。具体地，当第一实体中包含有第二实体时，将包含有第二实体的第一实体作为目标实体。比如，如果第一实体为“深圳大学城”，第二实体为“深圳”，由于第一实体中包含有“深圳”，所以保留“深圳大学城”为目标实体。同样地，当第二实体中包含有第一实体时，将包含有第一实体的第二实体作为目标实体。
如图4所示，在一个实施例中，上述文本实体识别方法还包括：
步骤210，将目标实体集中的目标实体与预设实体库中的实体进行匹配，当目标实体在预设实体库中时，进入步骤212，当目标实体不在预设实体库中时，进入步骤214。
其中，实体库是指存储实体的数据库，里面包含有很多个实体。实体库一般是根据业务需求自定义建立的。在预测得到目标实体后，在预设实体库中查找预测得到的目标实体是否在实体库中，如果实体库中有该实体，则说明该目标实体为确认实体。如果实体库中没有查找到该实体，那么该实体具有一定的不确定性，返回时需要明确告知该实体为预测到的实体，并不在实体库中。
步骤212，判定目标实体为第一类实体。
步骤214，判定目标实体为第二类实体。
其中，为了将确认的实体和不确定的实体进行区分，将在实体库中存在的实体称为“第一类实体”，将不在实体库中存在的实体称为“第二类实体”。在一个实施例中，如果是第一类实体，返回“确认”，如果是第二类实体，返回“预测”。
在一个实施例中，上述文本实体识别方法还包括：当目标实体集中包含有英文实体时，获取英文实体对应的多个变形英文实体；将多个变形英文实体与预设实体库中的实体进行匹配；当多个变形英文实体中的至少一个变形英文实体在预设实体库中时，则判定对应的英文实体为第一类实体，否则，判定对应的英文实体为第二类实体。
其中，英文实体是指以英文形式存在的实体。当目标实体为英文实体时，由于英文分为大小写，为了能够匹配的完整，将英文实体改写为多个变形英文实体。比如，将英文实体中的每个首字母大写，将第一首字符大写，将所有字母全小写，单个英文词全大写等。在得到多个变形英文实体后，将多个变形英文实体与预设实体库中的实体进行匹配，然后当多个变形英文实体中的至少一个变形英文实体在预设实体库中时，则认为该英文实体是在预设实体库中，从而判定该英文实体为第一类实体，否则，判定为第二类实体。举个例子，假设识别得到的英文实体为computer，由于实体库中存储的可能为Computer或COMPUTER，如果直接进行查询很可能查询不到，所以通过将computer进行大小写变形，然后再查询，就可以在实体库中查询到。
在一个实施例中，上述文本实体识别方法还包括：当第一预测实体集为空时，获取第二预测实体集中的实体和相应的实体类型；获取文本的意图，根据文本的意图确定目标实体类型；根据目标实体类型从第二预测实体集中筛选出符合目标实体类型的目标实体。
其中，当通过实体识别模型没有预测到实体时，即第一预测实体集为空时，则需要依赖分词机制提取到的第二预测实体集。但是第二预测实体集中的实体很可能包含有非目标实体，所以为了准确地得到目标实体，还需要进行进一步筛选。文本意图是指文本识别的意图，比如，是为了搜索音乐，还是为了搜索电影等。比如，刘德华这个实体具有两个实体类型，一个是歌手，一个是演员。如果文本意图为音乐场景，那么刘德华相应的目标实体类型应该歌手。如果第二预测实体集中的目标实体为刘德华，相应的实体类型为歌手，那么刘德华就是符合目标实体类型的目标实体。如果相应的实体类型为演员，那么刘德华就不符合目标实体类型，所以为非目标实体。
如图5所示为一个实施例中，文本实体识别方法的流程示意图。首先，获取待识别实体的文本（比如，接收用户输入的文本），然后将该文本作为实体识别模型的输入，得到输出的第一预测实体集，同时采用分词机制提取第二预测实体集，接下来，判断第一预测实体集是否为空，若否，则获取第一预测实体集与第二预测实体集的交集，保留交集中的实体，接下来，采用最长匹配原则检查第一预测实体集中和第二预测实体集中存在包含关系的实体，留下最长的。然后，检查保留下来实体（即目标实体）中是否有英文实体，如果有，则使用英文改写机制得到多个变形英文实体，然后判断提取到的实体是否在实体库中，若没有英文实体，则直接进入判断提取到的实体是否在实体库中，若在实体库中，则返回确认，若不在实体库中，则返回预测。若第一预测实体集为空，则直接将第二预测实体集中实体作为目标实体，进入检查留下来的实体是否有英文实体的步骤。
如图6所示，在一个实施例中，提出了一种文本实体识别装置，该装置包括：
文本获取模块602，用于获取待识别实体的文本；
输入输出模块604，用于将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；
分词提取模块606，用于 采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；
确定模块608，用于根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合。
在一个实施例中，所述分词提取模块还用于通过分词机制对所述文本进行分词处理，得到多个词语和相应的词语类型；根据所述词语类型从所述多个词语中提取出符合实体类型的词语，得到第二预测实体集。
在一个实施例中，所述确定模块还用于获取所述第一预测实体集和所述第二预测实体集的交集，将所述交集中的实体作为目标实体。
在一个实施例中，所述确定模块还用于将所述第一预测实体集中的第一实体与所述第二预测实体集中的第二实体进行匹配；当所述第一实体中包含有第二实体时，将包含有第二实体的第一实体作为目标实体；当所述第二实体中包含有第一实体时，将包含有第一实体的第二实体作为目标实体。
如图7所示，在一个实施例中，所述装置还包括：
匹配模块610，用于将所述目标实体集中的目标实体与预设实体库中的实体进行匹配，当所述目标实体在所述预设实体库中时，则判定所述目标实体为第一类实体，当所述目标实体不在所述预设实体库中时，则判定所述目标实体为第二类实体。
在一个实施例中，所述匹配模块还用于当所述目标实体集中包含有英文实体时，获取所述英文实体对应的多个变形英文实体，将所述多个变形英文实体与所述预设实体库中的实体进行匹配，当所述多个变形英文实体中的至少一个变形英文实体在所述预设实体库中时，则判定对应的所述英文实体为第一类实体，否则，判定对应的所述英文实体为第二类实体。
如图8所示，在一个实施例中，所述装置还包括：
类型获取模块612，用于当所述第一预测实体集为空时，获取所述第二预测实体集中的实体和相应的实体类型；
实体类型确定模块614，用于获取所述文本的意图，根据所述文本的意图确定目标实体类型；
筛选模块616，用于根据所述目标实体类型从所述第二预测实体集中筛选出符合所述目标实体类型的目标实体。
图9示出了一个实施例中计算机设备的内部结构图。该计算机设备可以是终端，也可以是服务器。如图9所示，该计算机设备包括通过系统总线连接的处理器、存储器和网络接口。其中，存储器包括非易失性存储介质和内存储器。该计算机设备的非易失性存储介质存储有操作系统，还可存储有计算机程序，该计算机程序被处理器执行时，可使得处理器实现文本实体识别方法。该内存储器中也可储存有计算机程序，该计算机程序被处理器执行时，可使得处理器执行文本实体识别方法。网络接口用于与外界进行通信。本领域技术人员可以理解，图9中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，本申请提供的文本实体识别方法可以实现为一种计算机程序的形式，计算机程序可在如图9所示的计算机设备上运行。计算机设备的存储器中可存储组成该文本实体识别装置的各个程序模板。比如，文本获取模块602、输入输出模块604、分词提取模块606和确定模块608。
一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述计算机程序被所述处理器执行时，使得所述处理器执行如下步骤：获取待识别实体的文本；将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合。
在一个实施例中，所述采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集，包括：通过分词机制对所述文本进行分词处理，得到多个词语和相应的词语类型；根据所述词语类型从所述多个词语中提取出符合实体类型的词语，得到第二预测实体集。
在一个实施例中，所述根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，包括：获取所述第一预测实体集和所述第二预测实体集的交集，将所述交集中的实体作为目标实体。
在一个实施例中，所述根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体，还包括：将所述第一预测实体集中的第一实体与所述第二预测实体集中的第二实体进行匹配；当所述第一实体中包含有第二实体时，将包含有第二实体的第一实体作为目标实体；当所述第二实体中包含有第一实体时，将包含有第一实体的第二实体作为目标实体。
在一个实施例中，在所述根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集之后，所述计算机程序被所述处理器执行时，还用于执行以下步骤：将所述目标实体集中的目标实体与预设实体库中的实体进行匹配，当所述目标实体在所述预设实体库中时，则判定所述目标实体为第一类实体；当所述目标实体不在所述预设实体库中时，则判定所述目标实体为第二类实体。
在一个实施例中，所述计算机程序被所述处理器执行时，还用于执行以下步骤：当所述目标实体集中包含有英文实体时，获取所述英文实体对应的多个变形英文实体；将所述多个变形英文实体与所述预设实体库中的实体进行匹配；当所述多个变形英文实体中的至少一个变形英文实体在所述预设实体库中时，则判定对应的所述英文实体为第一类实体，否则，判定对应的所述英文实体为第二类实体。
在一个实施例中，所述计算机程序被所述处理器执行时，还用于执行以下步骤：当所述第一预测实体集为空时，获取所述第二预测实体集中的实体和相应的实体类型；获取所述文本的意图，根据所述文本的意图确定目标实体类型；根据所述目标实体类型从所述第二预测实体集中筛选出符合所述目标实体类型的目标实体。
一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时，使得所述处理器执行如下步骤：获取待识别实体的文本；将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合。
在一个实施例中，所述采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集，包括：通过分词机制对所述文本进行分词处理，得到多个词语和相应的词语类型；根据所述词语类型从所述多个词语中提取出符合实体类型的词语，得到第二预测实体集。
在一个实施例中，所述根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，包括：获取所述第一预测实体集和所述第二预测实体集的交集，将所述交集中的实体作为目标实体。
在一个实施例中，所述根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体，还包括：将所述第一预测实体集中的第一实体与所述第二预测实体集中的第二实体进行匹配；当所述第一实体中包含有第二实体时，将包含有第二实体的第一实体作为目标实体；当所述第二实体中包含有第一实体时，将包含有第一实体的第二实体作为目标实体。
在一个实施例中，在所述根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集之后，所述计算机程序被所述处理器执行时，还用于执行以下步骤：将所述目标实体集中的目标实体与预设实体库中的实体进行匹配，当所述目标实体在所述预设实体库中时，则判定所述目标实体为第一类实体；当所述目标实体不在所述预设实体库中时，则判定所述目标实体为第二类实体。
在一个实施例中，所述计算机程序被所述处理器执行时，还用于执行以下步骤：当所述目标实体集中包含有英文实体时，获取所述英文实体对应的多个变形英文实体；将所述多个变形英文实体与所述预设实体库中的实体进行匹配；当所述多个变形英文实体中的至少一个变形英文实体在所述预设实体库中时，则判定对应的所述英文实体为第一类实体，否则，判定对应的所述英文实体为第二类实体。
在一个实施例中，所述计算机程序被所述处理器执行时，还用于执行以下步骤：当所述第一预测实体集为空时，获取所述第二预测实体集中的实体和相应的实体类型；获取所述文本的意图，根据所述文本的意图确定目标实体类型；根据所述目标实体类型从所述第二预测实体集中筛选出符合所述目标实体类型的目标实体。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一非易失性计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink) DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对本申请专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (10)

  1. 一种文本实体识别方法，其特征在于，所述方法包括：
    获取待识别实体的文本；
    将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；
    采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；
    根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合。
  2. 根据权利要求1所述的方法，其特征在于，所述采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集，包括：
    通过分词机制对所述文本进行分词处理，得到多个词语和相应的词语类型；
    根据所述词语类型从所述多个词语中提取出符合实体类型的词语，得到第二预测实体集。
  3. 根据权利要求1所述的方法，其特征在于，所述根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，包括：
    获取所述第一预测实体集和所述第二预测实体集的交集，将所述交集中的实体作为目标实体。
  4. 根据权利要求3所述的方法，其特征在于，所述根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体，还包括：
    将所述第一预测实体集中的第一实体与所述第二预测实体集中的第二实体进行匹配；
    当所述第一实体中包含有第二实体时，将包含有第二实体的第一实体作为目标实体；
    当所述第二实体中包含有第一实体时，将包含有第一实体的第二实体作为目标实体。
  5. 根据权利要求1所述的方法，其特征在于，在所述根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集之后，还包括：
    将所述目标实体集中的目标实体与预设实体库中的实体进行匹配，当所述目标实体在所述预设实体库中时，则判定所述目标实体为第一类实体；
    当所述目标实体不在所述预设实体库中时，则判定所述目标实体为第二类实体。
  6. 根据权利要求5所述的方法，其特征在于，所述方法还包括：
    当所述目标实体集中包含有英文实体时，获取所述英文实体对应的多个变形英文实体；
    将所述多个变形英文实体与所述预设实体库中的实体进行匹配；
    当所述多个变形英文实体中的至少一个变形英文实体在所述预设实体库中时，则判定对应的所述英文实体为第一类实体，否则，判定对应的所述英文实体为第二类实体。
  7. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    当所述第一预测实体集为空时，获取所述第二预测实体集中的实体和相应的实体类型；
    获取所述文本的意图，根据所述文本的意图确定目标实体类型；
    根据所述目标实体类型从所述第二预测实体集中筛选出符合所述目标实体类型的目标实体。
  8. 一种文本实体识别装置，其特征在于，所述装置包括：
    文本获取模块，用于获取待识别实体的文本；
    输入输出模块，用于将所述文本作为实体识别模型的输入，获取所述实体识别模型输出的第一预测实体集；
    分词提取模块，用于采用分词机制对所述文本进行分词处理，根据分词处理结果提取出第二预测实体集；
    确定模块，用于根据所述第一预测实体集和所述第二预测实体集确定识别得到的目标实体集，所述目标实体集为目标实体的集合。
  9. 一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述计算机程序被所述处理器执行时，使得所述处理器执行如权利要求1至7中任一项所述方法的步骤。
  10. 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时，使得所述处理器执行如权利要求1至7中任一项所述方法的步骤。
PCT/CN2018/125107 2018-12-28 2018-12-28 文本实体识别方法、装置、计算机设备及存储介质 Ceased WO2020133291A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/125107 WO2020133291A1 (zh) 2018-12-28 2018-12-28 文本实体识别方法、装置、计算机设备及存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/125107 WO2020133291A1 (zh) 2018-12-28 2018-12-28 文本实体识别方法、装置、计算机设备及存储介质

Publications (1)

Publication Number Publication Date
WO2020133291A1 true WO2020133291A1 (zh) 2020-07-02

Family

ID=71125616

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/125107 Ceased WO2020133291A1 (zh) 2018-12-28 2018-12-28 文本实体识别方法、装置、计算机设备及存储介质

Country Status (1)

Country Link
WO (1) WO2020133291A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113204967A (zh) * 2021-05-25 2021-08-03 山东师范大学 简历命名实体识别方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060136208A1 (en) * 2004-12-17 2006-06-22 Electronics And Telecommunications Research Institute Hybrid apparatus for recognizing answer type
CN107943786A (zh) * 2017-11-16 2018-04-20 广州市万隆证券咨询顾问有限公司 一种中文命名实体识别方法及系统
CN108108344A (zh) * 2016-11-24 2018-06-01 北京智能管家科技有限公司 一种联合识别和连接命名实体的方法及装置
CN108460014A (zh) * 2018-02-07 2018-08-28 百度在线网络技术（北京）有限公司 企业实体的识别方法、装置、计算机设备及存储介质
CN108763201A (zh) * 2018-05-17 2018-11-06 南京大学 一种基于半监督学习的开放域中文文本命名实体识别方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060136208A1 (en) * 2004-12-17 2006-06-22 Electronics And Telecommunications Research Institute Hybrid apparatus for recognizing answer type
CN108108344A (zh) * 2016-11-24 2018-06-01 北京智能管家科技有限公司 一种联合识别和连接命名实体的方法及装置
CN107943786A (zh) * 2017-11-16 2018-04-20 广州市万隆证券咨询顾问有限公司 一种中文命名实体识别方法及系统
CN108460014A (zh) * 2018-02-07 2018-08-28 百度在线网络技术（北京）有限公司 企业实体的识别方法、装置、计算机设备及存储介质
CN108763201A (zh) * 2018-05-17 2018-11-06 南京大学 一种基于半监督学习的开放域中文文本命名实体识别方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113204967A (zh) * 2021-05-25 2021-08-03 山东师范大学 简历命名实体识别方法及系统
CN113204967B (zh) * 2021-05-25 2022-06-21 山东师范大学 简历命名实体识别方法及系统

Similar Documents

Publication Publication Date Title
CN108595695B (zh) 数据处理方法、装置、计算机设备和存储介质
CN109063221B (zh) 基于混合策略的查询意图识别方法和装置
CN111382570B (zh) 文本实体识别方法、装置、计算机设备及存储介质
US11017178B2 (en) Methods, devices, and systems for constructing intelligent knowledge base
US9223779B2 (en) Text segmentation with multiple granularity levels
US20220261545A1 (en) Systems and methods for producing a semantic representation of a document
CN108664595B (zh) 领域知识库构建方法、装置、计算机设备和存储介质
CN113297366B (zh) 多轮对话的情绪识别模型训练方法、装置、设备及介质
US10558701B2 (en) Method and system to recommend images in a social application
WO2022142613A1 (zh) 训练语料扩充方法及装置、意图识别模型训练方法及装置
US11361759B2 (en) Methods and systems for automatic generation and convergence of keywords and/or keyphrases from a media
WO2020133039A1 (zh) 对话语料中实体的识别方法、装置和计算机设备
WO2020057022A1 (zh) 关联推荐方法、装置、计算机设备和存储介质
CN109543007A (zh) 提问数据生成方法、装置、计算机设备和存储介质
CN111159546A (zh) 事件推送方法、装置、计算机可读存储介质和计算机设备
CN109858010A (zh) 领域新词识别方法、装置、计算机设备和存储介质
CN113761923B (zh) 命名实体识别方法、装置、电子设备及存储介质
WO2021238337A1 (zh) 用于实体标注的方法和装置
CN117112595B (zh) 一种信息查询方法、装置、电子设备及存储介质
WO2021063089A1 (zh) 规则匹配方法、规则匹配装置、存储介质及电子设备
CN107169021A (zh) 用于预测应用功能标签的方法和设备
US10496751B2 (en) Avoiding sentiment model overfitting in a machine language model
CN111126084B (zh) 数据处理方法、装置、电子设备和存储介质
CN107329964B (zh) 一种文本处理方法及装置
US11036926B2 (en) Generating annotated natural language phrases

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18944859

Country of ref document: EP

Kind code of ref document: A1