WO2020252950A1 - 基于预训练模型和微调技术的医疗文本命名实体识别方法 - Google Patents

基于预训练模型和微调技术的医疗文本命名实体识别方法 Download PDF

Info

Publication number
WO2020252950A1
WO2020252950A1 PCT/CN2019/105506 CN2019105506W WO2020252950A1 WO 2020252950 A1 WO2020252950 A1 WO 2020252950A1 CN 2019105506 W CN2019105506 W CN 2019105506W WO 2020252950 A1 WO2020252950 A1 WO 2020252950A1
Authority
WO
WIPO (PCT)
Prior art keywords
medical
named entity
text
training model
entity recognition
Prior art date
Application number
PCT/CN2019/105506
Other languages
English (en)
French (fr)
Inventor
陈涛
杨开漠
Original Assignee
五邑大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 五邑大学 filed Critical 五邑大学
Publication of WO2020252950A1 publication Critical patent/WO2020252950A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G06F16/355—Class or cluster creation or modification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G06F40/295—Named entity recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data

Definitions

  • the invention relates to the technical field of data mining, in particular to a medical text named entity recognition method based on a pre-training model and fine-tuning technology.
  • Clinical medicine is the science of studying the cause, diagnosis, treatment and prognosis of diseases, improving the level of clinical treatment, and promoting human health.
  • Medical text data such as electronic medical records in clinical medicine are of great value to medical research. With the popularization of the Internet, more and more online clinical medical communities and clinical medical consultation websites have appeared. These online clinical medical websites have also produced rich medical text data. There are a large number of real personal cases in these medical texts, and there are rich clinical medical values hidden in them. However, most of these clinical medical texts are in an unstructured state. In order to fully tap the value, and lay the foundation work for the next application of clinical medical question answering robot and clinical medical decision support system. How to use more effective and reasonable methods to accurately dig out the potential value in medical text data is already the trend of future development of information technology and the background of medical text data mining technology.
  • Medical institutions build and organize text data analysis teams for text analysis of medical text data to mine the useful information contained in the medical text data generated in clinical medicine, especially the named entity information contained in the medical text, in order to improve medical treatment. Obtain greater commercial benefits from competition among institutions. At the same time, the country also regards medical text data mining as an important component of the national strategic layout to improve domestic medical standards.
  • Named entity recognition also known as entity recognition and entity extraction, is a sub-task of information extraction tasks in the field of natural language processing. It aims to automatically identify and extract proper nouns from text corpus by using machine learning related methods And be classified. Common named entities are: names of persons, names of places, names of organizations, specific entities, etc.
  • Named entity recognition is a very important basic task in natural language processing, and it is a basic technology for many high-level applications of natural language processing, such as search engines, question answering systems, recommendation systems, translation systems, etc. With the continuous development of information technology and the continuous deepening of the application scope of information systems, automatic named entity recognition of text data is an important means and measure to improve the production efficiency and competitive advantage of enterprises.
  • the analysis methods of text named entity recognition can be divided into rule-based named entity recognition method, dictionary-based named entity recognition method, named entity recognition method based on traditional machine learning, and named entity recognition method based on deep learning method.
  • the rule-based named entity recognition method uses pattern matching technology to identify named entities by matching text according to a predefined rule template.
  • the rule template is generally designed based on linguistic knowledge such as morphology, syntax, and grammar. This method has high accuracy because the rules used are generally written by experts. However, due to the non-reusability of expert knowledge, this method has disadvantages such as very high labor costs.
  • the dictionary-based named entity recognition method is a named entity recognition method that builds a dictionary and matches the content in the text according to the dictionary.
  • Traditional named entity categories such as person names, place names, organization names, etc., have a huge number of named entities, and it is difficult to include them all in the dictionary.
  • the names of some entities change frequently and there is no strict rule to follow, so they are purely based on dictionary matching. Named entity recognition usually cannot meet the needs of practical applications.
  • the named entity recognition method based on traditional machine learning uses the knowledge of statistics and probability to design various complex statistical models for specific named entity recognition problems, and uses a large amount of labeled data for supervised learning to adjust the parameters of the model, and then use The trained model is used for named entity recognition. Compared with the first two methods, this type of method does not require strong linguistic knowledge, has high flexibility and strong adaptability, but the effect of named entity recognition depends on the quantity and quality of the labeled data.
  • the deep learning-based named entity recognition method uses deep neural network training models to identify specific named entities in different fields.
  • representative methods include text relation extraction methods based on Recurrent Neural Network (RNN) and its variants, and methods based on Convolutional Neural Networks (CNN) and its variants. These methods not only can more accurately identify and extract named entities in the text corpus of related fields, but also reduce labor costs.
  • RNN Recurrent Neural Network
  • CNN Convolutional Neural Networks
  • Medical text named entity recognition is the use of computer technology to automatically identify useful named entities in the medical field from medical texts, such as anatomical parts, independent symptoms, symptom descriptions, operations, drugs and other named entities, forming structured data and storing them in a database. The process for users to query or further analyze information. Specifically, it is the named entity recognition technology in the field of natural language processing that automatically recognizes medical named entities from medical texts.
  • the present invention provides a medical text named entity recognition method based on a pre-training model and fine-tuning technology.
  • the method trains a named entity recognition model based on a deep learning method by using medical text that has been marked with medical named entities. , And then use the trained model to recognize the named entity of the unlabeled medical text, and extract the named entity contained in the document.
  • the technical scheme of the present invention is: a medical text named entity recognition method based on a pre-training model and fine-tuning technology, including the following steps:
  • step S3 Use the stacked expanded convolutional neural network IDCNN to fine-tune the pre-trained model trained in step S2), and train a deep neural network model that can recognize the relevant medical named entities contained in the medical text;
  • step S1 the preprocessing of the medical text using the related technology of text data mining mainly includes the following steps:
  • BIO sequence labeling method to label the anatomical parts, independent symptoms, symptom descriptions, surgery, and drug-related clinical medical named entities in the medical text;
  • B represents the beginning word of the medical named entity;
  • I represents the medical named entity The middle or ending word of;
  • O means non-medical named entity;
  • step S2 the large-scale unstructured unlabeled medical text is pre-trained to obtain rich semantic representation information contained in the text.
  • the neural network word vector training model trains word vectors from large-scale unlabeled medical texts, and uses the word vectors as the input of the next layer of the network.
  • the BERT pre-training model can obtain richer text semantic representation information, which can be more effective Migration to specific tasks.
  • step S3) the stacked dilated convolutional neural network is combined with the BERT pre-training model, and the pre-training model generated in step S2) is fine-tuned to obtain a deep neural network for medical text named entity recognition Network model.
  • step S4) by inputting the medical text that has not been labeled with the named entity into the deep neural network model trained by the pre-training model and fine-tuning technology in step S3), the corresponding input text is obtained
  • the medical named entity BIO mark presents the above-mentioned mark to the user using a visual method, so that the user of the system can conveniently obtain the automatically recognized medical named entity information in the input medical text.
  • the present invention uses a pre-training model instead of a word vector as the input of the medical text named entity recognition model.
  • the pre-training model can more accurately capture the semantic information in the text, and can more effectively migrate to specific In the task, improve the accuracy of named entity recognition by the model;
  • the present invention uses stacked dilated convolutional neural network for medical text named entity recognition. Compared with the prior art, the stacked dilated convolutional neural network used in the present invention can not only capture the semantic information in the text well, but also the method can Perform parallel calculations to increase the speed of model training;
  • the present invention uses the stacked expanded convolutional neural network in combination with the pre-training model, and uses the stacked expanded convolutional neural network to fine-tune the pre-trained model obtained by the pre-training to improve the performance of the model.
  • Figure 1 is a schematic flow diagram of the present invention
  • Figure 2 is a schematic diagram of the BIO labeling results of the embodiment of the present invention.
  • the present invention provides a medical text named entity recognition method based on a pre-training model and fine-tuning technology.
  • the present invention first uses large-scale unstructured electronic medical records and other medical texts to pre-train the BERT pre-training model. To train a pre-training model that contains the semantic representation information in the text. Then, the generated pre-training model is fine-tuned using the stacked dilated convolutional neural network to obtain a deep neural network model that can automatically recognize named entities in the medical field.
  • BIO sequence labeling method uses the three letters B, I, and O to mark text information.
  • B represents the beginning word of a medical named entity
  • I represents the middle word or ending word of a medical named entity
  • O represents a non-medical named entity.
  • step S3 Use the stacked dilated convolutional neural network to fine-tune the pre-training model generated in step S2).
  • train a deep neural network model that can automatically identify named entities in the medical field;
  • the present invention uses a pre-training model instead of a word vector as the input of the medical text named entity recognition model.
  • the pre-training model is a neural network model designed to learn the semantic information in the corpus in an unsupervised manner from a large-scale unlabeled corpus. It is a complex learning model composed of a multi-layer neural network, and the word vector is a word vector.
  • the learning model generates a set of one-dimensional vectors from large-scale unlabeled corpus. Compared with word vectors, the pre-training model can more accurately capture the semantic information in the text, and improve the accuracy of the model's named entity recognition.
  • the present invention adopts a method of combining an artificial neural network based on deep learning and a natural language processing named entity recognition method, which can improve system accuracy, response speed and fault tolerance.
  • Pre-processing the medical field entity recognition corpus and fine-tuning the pre-training model by using stacked expansion convolutional neural networks to improve the accuracy and efficiency of named entity recognition can meet the needs of different applications.
  • the present invention analyzes the sentence "the left kidney area is obviously painful when knocking on the left kidney area", as shown in Figure 2, specifically:

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Data Mining & Analysis (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Epidemiology (AREA)
  • Medical Informatics (AREA)
  • Primary Health Care (AREA)
  • Public Health (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Medical Treatment And Welfare Office Work (AREA)

Abstract

一种基于预训练模型和微调技术的医疗文本命名实体识别方法，首先利用大规模非结构化的电子病历等医疗文本对BERT预训练模型进行预训练，以训练出包含文本中语义表示信息的预训练模型。利用堆叠扩张卷积神经网络对所产生的预训练模型进行微调，以获得能够进行医疗领域命名实体自动识别的深度神经网络模型。该方法提供的预训练模型能够更为准确的捕获文本中的语义信息，能够更有效的迁移到特定的任务中，提高模型进行命名实体识别的准确性；该方法将堆叠扩张卷积神经网络与预训练模型结合以对模型进行微调，最终进行医疗文本命名实体的识别，不仅能够很好的捕获文本中的语义信息，而且能够进行并行计算，以提高模型训练速度。

Description

基于预训练模型和微调技术的医疗文本命名实体识别方法 技术领域
本发明涉及数据挖掘技术领域，尤其是一种基于预训练模型和微调技术的医疗文本命名实体识别方法。
背景技术
临床医学是研究疾病的病因、诊断、治疗和预后，提高临床治疗水平，促进人体健康的科学。临床医学中的电子病历等医疗文本数据对于医学领域研究具有巨大的价值。随着互联网的普及，出现了越来越多的在线临床医疗社区以及临床医疗咨询网站。这些在线临床医疗网站也产生了丰富的医疗文本数据。这些医疗文本中有大量真实的个人案例，潜藏着丰富的临床医疗价值。但是这些临床医疗文本大多处于一种非结构化的状态。为充分挖掘其中的价值，并为接下来临床医疗问答机器人、临床医疗决策支持系统等应用打好基础工作。如何采用更加有效合理的方法，准确地挖掘出医疗文本数据中所存在的潜在价值，已经是未来信息科技发展的趋势，也是医疗文本数据挖掘技术产生的背景。
医疗机构通过构建和组织对医疗文本数据进行文本分析的文本数据分析团队，来挖掘临床医学中产生的医疗文本数据中所包含的有用信息，尤其是医疗文本中包含的命名实体信息，为提高医疗机构之间竞争中获取更大的商业利益。同时，国家也将医疗文本数据挖掘作为国家战略布局的重要组成成分，以提高国内医疗水平。
命名实体识别，又名实体识别、实体抽取，是自然语言处理领域信息抽取任务中的一项子任务，旨在通过利用机器学习相关方法从文本语料中自动识别并抽取出文本中的专有名词并加以分类。常见的命名实体有：人名、地名、机构名、特定实体等。命名实体识别是自然语言处理中一个相当重要的基础任务，是许多自然语言处理的高层应用，如搜索引擎、问答系统、推荐系统、翻译系统等的一个基础技术。随着信息技术的不断发展与信息系统应用范围的不断深化，对文本数据进行自动化命名实体识别是提高企业生产效率和竞争优势的一种重要手段及措施。
文本命名实体识别的分析方法可分为基于规则的命名实体识别方法、基于词典的命名实体识别方法、基于传统机器学习的命名实体识别方法和基于深度学习方法的命名实体识别方法。
基于规则的命名实体识别方法是利用模式匹配技术根据预先定义好的规则模板对文本进行匹配识别出命名实体。规则模板一般是根据词法、句法、语法等语言学知识来进行设计的。 该方法因为所采用的规则一般是由专家编写的，所以具有较高的准确性。但是因为专家知识具有不可复用性，导致该方法存在人工成本非常高等缺点。
基于词典的命名实体识别方法是通过构建词典并根据词典对文本中的内容进行匹配的命名实体识别方法。传统的命名实体类别，如人名、地名、机构名等的命名实体数量巨大，很难全部收录在词典中，而且部分实体名称变化频繁，并没有严格的规律可循，因此单纯的基于词典匹配的命名实体识别通常无法满足实际应用的需求。
基于传统机器学习的命名实体识别方法利用统计学和概率学的知识，针对特定的命名实体识别问题设计各种复杂的统计模型，利用大量的标记数据进行有监督的学习调整模型的参数，进而使用训练好的模型来进行命名实体识别。相比前两种方法，此类方法不需要很强的语言学知识，灵活性高，适应性强，但命名实体识别的效果依赖标注数据的数量和质量。
基于深度学习的命名实体识别方法利用深度神经网络训练模型来识别不同领域中的特定命名实体。其中，具有代表性意义的方法有基于循环神经网络(Recurrent Neural Network，RNN)及其变体的文本关系抽取方法和基于卷积神经网络(Convolutional Neural Networks，CNN)及其变体的方法。这些方法不仅能够更加准确地识别并抽取出相关领域文本语料中的命名实体，而且减少了人工成本消耗。
医疗文本命名实体识别是利用计算机技术从医疗文本中自动识别出医疗领域有用的命名实体，如解剖部位、独立症状、症状描述、手术、药物等命名实体，形成结构化的数据储存在数据库中，以供用户对信息的查询或进一步分析利用的过程。具体的说，是自然语言处理领域的命名实体识别技术从医疗文本中自动识别出医疗命名实体。
随着各类临床医疗症状的发现解决以及相关药物的研究发明，如何采用更加高效合理的技术从数目繁多的临床医疗文本中识别并抽取不同与医学相关的命名实体，并为不同用户群体及医学专家提供在线查询平台，已经是目前大数据行业所面临的直接问题与挑战。它是医疗知识挖掘、临床医疗问答机器人、临床医疗决策支持系统等应用领域的重要基础工作。综上所述，需要研究与开发一种科学准确的医疗文本命名实体识别的方法，具有很强的现实意义。
发明内容
针对现有技术的不足，本发明提供基于预训练模型和微调技术的医疗文本命名实体识别方法，该方法通过利用已标注好医疗命名实体的医疗文本对基于深度学习方法的命名实体识别模型进行训练，然后利用训练好的模型对未标注的医疗文本进行命名实体识别，并抽取出文档中所包含的命名实体。
本发明的技术方案为：基于预训练模型和微调技术的医疗文本命名实体识别方法，包括以下步骤：
S1)、利用文本数据挖掘的相关技术对医疗文本进行预处理；
S2)、利用BERT预训练模型对大规模非结构化的无标注的医疗文本进行预训练，训练出包含医疗领域语义表示信息的预训练模型；
S3)、利用堆叠扩张卷积神经网络IDCNN对步骤S2)所训练出的预训练模型进行微调，训练出能够识别医疗文本中所包含的相关医疗命名实体的深度神经网络模型；
S4)、将训练好的深度神经网络模型用于医疗文本的命名实体识别任务中，以识别更多有价值的命名实体。
进一步的，上述方法中，步骤S1)中，利用文本数据挖掘的相关技术对医疗文本进行预处理，主要包含以下步骤：
S101)、对医疗文本进行分词处理；
S102)、删除以分词句子中的停用词；
S103)、利用BIO序列标记法标注医疗文本中所存在的解剖部位、独立症状、症状描述、手术、药物相关的临床医疗命名实体；其中，B表示医疗命名实体的开始词；I表示医疗命名实体的中间词或结尾词；O表示非医疗命名实体；
进一步的，上述方法中，步骤S2)中，对大规模非结构化的无标注医疗文本进行预训练，以获得文本中所包含的丰富语义表示信息。相对于神经网络的词向量训练模型从大规模无标注医疗文本中训练词向量，将词向量作为下一层网络的输入，利用BERT预训练模型能够获得更加丰富的文本语义表示信息，能够更有效的迁移到特定的任务中。
进一步的，上述方法中，步骤S3)中，将堆叠扩张卷积神经网络与BERT预训练模型进行结合，对步骤S2)所产生的预训练模型进行微调，得到进行医疗文本命名实体识别的深度神经网络模型。
进一步的，上述方法中，步骤S4)中，通过将未进行命名实体标注的医疗文本输入到步骤S3)中利用预训练模型和微调技术训练出的深度神经网络模型中，得到该输入文本对应的医疗命名实体BIO标记，将上述标记利用可视化方法呈现给用户，使该系统的使用者能够方便的获取输入的医疗文本中被自动识别出的医疗命名实体信息。
本发明的有益效果为：
1、本发明采用预训练模型而不是词向量作为医疗文本命名实体识别模型的输入，相对于词向量，预训练模型能够更为准确的捕获文本中的语义信息，能够更有效的迁移到特定的任务中，提高模型进行命名实体识别的准确性；
2、本发明利用堆叠扩张卷积神经网络进行医疗文本命名实体识别，相对于现有技术，本发明采用的堆叠扩张卷积神经网络不仅能够很好的捕获文本中的语义信息，而且该方法能够进行并行计算，以提高模型训练速度；
3、本发明将堆叠扩张卷积神经网络与预训练模型结合使用，并使用堆叠扩张卷积神经网络来微调预训练得到的预训练模型，以提高模型性能。
附图说明
图1为本发明的流程示意图；
图2为本发明实施例的BIO标记结果示意图。
具体实施方式
下面结合附图对本发明的具体实施方式作进一步说明：
如图1所示，本发明提供一种基于预训练模型和微调技术的医疗文本命名实体识别方法，本发明首先利用大规模非结构化的电子病历等医疗文本对BERT预训练模型进行预训练，以训练出包含文本中语义表示信息的预训练模型。然后，利用堆叠扩张卷积神经网络对所产生的预训练模型进行微调，以获得能够进行医疗领域命名实体自动识别的深度神经网络模型。
具体为：S1)、利用文本数据挖掘相关技术对电子病历等医疗文本进行预处理操作；该过程具体包括以下步骤：
S101)、对电子病历等医疗文本进行分词处理；
S102)、删除已分词句子中停用词；
S103)、利用BIO序列标记法对医疗文本中存在的解剖部位、独立症状、症状描述、手术、药物等命名实体进行标注。BIO序列标记法是一种使用B、I、O三个字母来标记文本信息的方法。其中，B表示医疗命名实体的开始词；I表示医疗命名实体的中间词或结尾词；O表示非医疗命名实体。
S2)、利用BERT预训练模型大规模非结构化的无标注医疗文本进行预训练，生成包含文本中语义表示信息的预训练模型。
S3)、利用堆叠扩张卷积神经网络对步骤S2)所产生的预训练模型进行微调。在微调过程中，训练出能够自动识别医疗领域命名实体的深度神经网络模型；
S4)、将训练好的深度神经网络模型用于医疗文本命名实体识别任务中。
本发明采用预训练模型而不是词向量作为医疗文本命名实体识别模型的输入。预训练模型是专门为从大规模无标注语料中以无监督的方式学习语料中的语义信息的神经网络模型，它是由多层神经网络组成的复杂的学习模型，而词向量是使用词向量学习模型从大规模无标 注语料中生成的一组一维向量。相对于词向量，预训练模型能够更为准确的捕获文本中的语义信息，提高模型进行命名实体识别的准确性。
同时，本发明采用基于深度学习的人工神经网络与自然语言处理命名实体识别方法相结合的方法，能够提高系统准确性、反应速度和容错性。针对医疗领域实体识别语料进行预处理和利用堆叠扩张卷积神经网络微调预训练模型，提高命名实体识别的准确性和效率，能够满足不同应用的需求。
为了更好的理解本发明，本发明对句子“左肾区叩击疼明显”进行分析，如图2所示，具体为：
1)、对待处理句子进行预处理的分词结果如下：“左肾区叩击疼明显”。
2)、将上述结果输入给基于预处理模型和微调技术的医疗文本命名实体识别模型中，进行命名实体识别，得到的命名实体识别结果为身体部位实体“左肾区”和症状实体“叩击疼”。
上述实施例和说明书中描述的只是说明本发明的原理和最佳实施例，在不脱离本发明精神和范围的前提下，本发明还会有各种变化和改进，这些变化和改进都落入要求保护的本发明范围内。

Claims (5)

  1. 基于预训练模型和微调技术的医疗文本命名实体识别方法，其特征在于，包括以下步骤：
    S1)、利用文本数据挖掘的相关技术对医疗文本进行预处理；
    S2)、利用BERT预训练模型对大规模非结构化的无标注的医疗文本进行预训练，训练出包含医疗领域语义表示信息的预训练模型；
    S3)、利用堆叠扩张卷积神经网络IDCNN对步骤S2)所训练出的预训练模型进行微调，训练出能够识别医疗文本中所包含的相关医疗命名实体的深度神经网络模型；
    S4)、将训练好的深度神经网络模型用于医疗文本的命名实体识别任务中，以识别更多有价值的命名实体。
  2. 根据权利要求1所述的基于预训练模型和微调技术的医疗文本命名实体识别方法，其特征在于：步骤S1)中，利用文本数据挖掘的相关技术对医疗文本进行预处理，主要包含以下步骤：
    S101)、对医疗文本进行分词处理；
    S102)、删除已分词句子中的停用词；
    S103)、利用BIO序列标记法标注医疗文本中所存在的解剖部位、独立症状、症状描述、手术、药物相关的临床医疗命名实体；其中，B表示医疗命名实体的开始词；I表示医疗命名实体的中间词或结尾词；O表示非医疗命名实体；
  3. 根据权利要求1所述的基于预训练模型和微调技术的医疗文本命名实体识别方法，其特征在于：步骤S2)中，对大规模非结构化的无标注医疗文本进行预训练，以获得文本中所包含的丰富语义表示信息。
  4. 根据权利要求1所述的基于预训练模型和微调技术的医疗文本命名实体识别方法，其特征在于：步骤S3)中，将堆叠扩张卷积神经网络与BERT预训练模型进行结合，对步骤S2)所产生的预训练模型进行微调，得到进行医疗文本命名实体识别的深度神经网络模型。
  5. 根据权利要求1所述的基于预训练模型和微调技术的医疗文本命名实体识别方法，其特征在于：步骤S4)中，通过将未进行命名实体标注的医疗文本输入到步骤S3)中利用预训练模型和微调技术训练出的深度神经网络模型中，得到该输入文本对应的医疗命名实体BIO标记，将上述标记利用可视化方法呈现给用户，使该系统的使用者能够方便的获取输入的医疗文本中被自动识别出的医疗命名实体信息。
PCT/CN2019/105506 2019-06-17 2019-09-12 基于预训练模型和微调技术的医疗文本命名实体识别方法 WO2020252950A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910520186.6A CN110348008A (zh) 2019-06-17 2019-06-17 基于预训练模型和微调技术的医疗文本命名实体识别方法
CN201910520186.6 2019-06-17

Publications (1)

Publication Number Publication Date
WO2020252950A1 true WO2020252950A1 (zh) 2020-12-24

Family

ID=68182151

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/105506 WO2020252950A1 (zh) 2019-06-17 2019-09-12 基于预训练模型和微调技术的医疗文本命名实体识别方法

Country Status (2)

Country Link
CN (1) CN110348008A (zh)
WO (1) WO2020252950A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112800767A (zh) * 2021-01-31 2021-05-14 云知声智能科技股份有限公司 一种病历文本中患者基本信息的检查方法及系统
CN112989831A (zh) * 2021-03-29 2021-06-18 华南理工大学 一种应用在网络安全领域的实体抽取方法
CN113158677A (zh) * 2021-05-13 2021-07-23 竹间智能科技（上海）有限公司 一种命名实体识别方法和系统
CN114970522A (zh) * 2022-05-20 2022-08-30 北京百度网讯科技有限公司 语言模型的预训练方法、装置、设备、存储介质
CN116070602A (zh) * 2023-01-05 2023-05-05 中国科学院计算机网络信息中心 一种pdf文档智能标注与抽取方法

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111160023B (zh) * 2019-12-23 2023-06-20 华南理工大学 一种基于多路召回的医疗文本命名实体识别方法
CN111127191B (zh) * 2019-12-25 2023-07-14 北京百度网讯科技有限公司 风险评估方法及装置
CN111222339B (zh) * 2020-01-13 2023-05-23 华南理工大学 一种基于对抗多任务学习的医疗咨询命名实体识别方法
CN111222340B (zh) * 2020-01-15 2021-12-07 东华大学 基于多标准主动学习的乳腺电子病历实体识别系统
CN111274820B (zh) * 2020-02-20 2023-04-07 齐鲁工业大学 一种基于神经网络的智能医疗命名实体识别方法和装置
CN111159416B (zh) * 2020-04-02 2020-07-17 腾讯科技（深圳）有限公司 语言任务模型训练方法、装置、电子设备及存储介质
CN111584021B (zh) * 2020-05-08 2023-08-18 青岛国新健康产业科技有限公司 病案信息校验方法、装置、电子设备及存储介质
CN111444721B (zh) * 2020-05-27 2022-09-23 南京大学 一种基于预训练语言模型的中文文本关键信息抽取方法
CN112016295B (zh) * 2020-09-04 2024-02-23 平安科技（深圳）有限公司 症状数据处理方法、装置、计算机设备及存储介质
CN112016319B (zh) * 2020-09-08 2023-12-15 平安科技（深圳）有限公司 预训练模型获取、疾病实体标注方法、装置及存储介质
CN111931490B (zh) * 2020-09-27 2021-01-08 平安科技（深圳）有限公司 文本纠错方法、装置及存储介质
CN113808752A (zh) * 2020-12-04 2021-12-17 四川医枢科技股份有限公司 一种医学文档识别方法、装置及设备
CN112733543A (zh) * 2021-01-26 2021-04-30 上海交通大学 基于文本编辑生成模型的机构命名实体归一化方法和系统
CN113283244B (zh) * 2021-07-20 2021-10-01 湖南达德曼宁信息技术有限公司 一种基于预训练模型的招投标数据命名实体识别方法
CN117709441B (zh) * 2024-02-06 2024-05-03 云南联合视觉科技有限公司 通过逐步迁移领域训练专业医疗大模型的方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140359409A1 (en) * 2007-08-02 2014-12-04 Google Inc. Learning Synonymous Object Names from Anchor Texts
CN108920460A (zh) * 2018-06-26 2018-11-30 武大吉奥信息技术有限公司 一种多类型实体识别的多任务深度学习模型的训练方法及装置
CN109062983A (zh) * 2018-07-02 2018-12-21 北京妙医佳信息技术有限公司 用于医学健康知识图谱的命名实体识别方法及系统
CN109670179A (zh) * 2018-12-20 2019-04-23 中山大学 基于迭代膨胀卷积神经网络的病历文本命名实体识别方法
CN109710925A (zh) * 2018-12-12 2019-05-03 新华三大数据技术有限公司 命名实体识别方法及装置
CN109710770A (zh) * 2019-01-31 2019-05-03 北京牡丹电子集团有限责任公司数字电视技术中心 一种基于迁移学习的文本分类方法及装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104298651B (zh) * 2014-09-09 2017-02-22 大连理工大学 一种基于深度学习的生物医学命名实体识别和蛋白质交互关系抽取在线方法
US11593558B2 (en) * 2017-08-31 2023-02-28 Ebay Inc. Deep hybrid neural network for named entity recognition

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140359409A1 (en) * 2007-08-02 2014-12-04 Google Inc. Learning Synonymous Object Names from Anchor Texts
CN108920460A (zh) * 2018-06-26 2018-11-30 武大吉奥信息技术有限公司 一种多类型实体识别的多任务深度学习模型的训练方法及装置
CN109062983A (zh) * 2018-07-02 2018-12-21 北京妙医佳信息技术有限公司 用于医学健康知识图谱的命名实体识别方法及系统
CN109710925A (zh) * 2018-12-12 2019-05-03 新华三大数据技术有限公司 命名实体识别方法及装置
CN109670179A (zh) * 2018-12-20 2019-04-23 中山大学 基于迭代膨胀卷积神经网络的病历文本命名实体识别方法
CN109710770A (zh) * 2019-01-31 2019-05-03 北京牡丹电子集团有限责任公司数字电视技术中心 一种基于迁移学习的文本分类方法及装置

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112800767A (zh) * 2021-01-31 2021-05-14 云知声智能科技股份有限公司 一种病历文本中患者基本信息的检查方法及系统
CN112800767B (zh) * 2021-01-31 2023-11-21 云知声智能科技股份有限公司 一种病历文本中患者基本信息的检查方法及系统
CN112989831A (zh) * 2021-03-29 2021-06-18 华南理工大学 一种应用在网络安全领域的实体抽取方法
CN112989831B (zh) * 2021-03-29 2023-04-28 华南理工大学 一种应用在网络安全领域的实体抽取方法
CN113158677A (zh) * 2021-05-13 2021-07-23 竹间智能科技（上海）有限公司 一种命名实体识别方法和系统
CN113158677B (zh) * 2021-05-13 2023-04-07 竹间智能科技（上海）有限公司 一种命名实体识别方法和系统
CN114970522A (zh) * 2022-05-20 2022-08-30 北京百度网讯科技有限公司 语言模型的预训练方法、装置、设备、存储介质
CN114970522B (zh) * 2022-05-20 2023-11-24 北京百度网讯科技有限公司 语言模型的预训练方法、装置、设备、存储介质
CN116070602A (zh) * 2023-01-05 2023-05-05 中国科学院计算机网络信息中心 一种pdf文档智能标注与抽取方法
CN116070602B (zh) * 2023-01-05 2023-10-17 中国科学院计算机网络信息中心 一种pdf文档智能标注与抽取方法

Also Published As

Publication number Publication date
CN110348008A (zh) 2019-10-18

Similar Documents

Publication Publication Date Title
WO2020252950A1 (zh) 基于预训练模型和微调技术的医疗文本命名实体识别方法
WO2020211275A1 (zh) 基于预训练模型与微调技术的医疗文本关系抽取方法
Chen et al. Research on text sentiment analysis based on CNNs and SVM
Qiang et al. Lsbert: Lexical simplification based on bert
CN112925918B (zh) 一种基于疾病领域知识图谱的问答匹配系统
Song et al. Classification of traditional chinese medicine cases based on character-level bert and deep learning
Gan et al. Semglove: Semantic co-occurrences for glove from bert
CN116775812A (zh) 一种基于自然语音处理的中医药专利分析与挖掘工具
Subramanyam et al. Deep contextualized medical concept normalization in social media text
Zhang et al. AIA-Net: adaptive interactive attention network for text–audio emotion recognition
Jin Application optimization of nlp system under deep learning technology in text semantics and text classification
Abubakar et al. Explainable emotion recognition from tweets using deep learning and word embedding models
Li et al. Interactive lexical and semantic graphs for semisupervised relation extraction
Jawad et al. Combination Of Convolution Neural Networks And Deep Neural Networks For Fake News Detection
CN115828854B (zh) 一种基于上下文消歧的高效表格实体链接方法
Wang et al. A BERT-based named entity recognition in Chinese electronic medical record
CN116227594A (zh) 面向多源数据的医疗行业高可信度知识图谱的构建方法
Saikh et al. COVIDRead: A large-scale question answering dataset on COVID-19
Marerngsit et al. A two-stage text-to-emotion depressive disorder screening assistance based on contents from online community
Liao et al. A text classification framework for depression tendency detection over online social platform
Wang et al. Comparison between calculation methods for semantic text similarity based on siamese networks
Xing et al. Guest Editorial Neurosymbolic AI for Sentiment Analysis
Chen et al. A multi-label classification algorithm for non-standard text
Nguyen et al. A two-channel model for representation learning in vietnamese sentiment classification problem
Liu et al. Comparative research on structure function recognition based on deep learning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19933604

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19933604

Country of ref document: EP

Kind code of ref document: A1