CN103049501A - Chinese domain term recognition method based on mutual information and conditional random field model - Google Patents

Abstract

The invention discloses a Chinese domain term recognition method based on mutual information and a conditional random field model. The Chinese domain term recognition method includes the following steps: (1) gathering domain text corpus and marking all the punctuations, spaces, numbers, ASSCII (American Standard Code for Information Interchange) characters and characters except Chinese characters in the corpus; (2) setting character strings and computing the mutual information values of the character strings, (3) computing the left comentropy and the right comentropy of every character string, (4) defining character string evaluation function, setting evaluation function threshold, computing the evaluation function values of every character string, determining that every character string is a word, comparing in sequence the evaluation function value of the former character with the evaluation function value of the latter character in the character string and segmenting character meaning character strings one by one, (5) utilizing conditional random fields to train a conditional random field model and recognizing domain terms with the conditional random field model. When the Chinese domain term recognition method is used to recognize terms, the data sparsity of legitimate terms is overcome, the amount of calculation of conditional random fields is reduced, and the accuracy of the Chinese domain term recognition is improved.

Description

Chinese domain term recognition method based on mutual information and conditional random field model

technical field

The invention relates to a method for recognizing Chinese domain terms based on mutual information and a conditional random field model, which belongs to the field of information technology.

Background technique

The definition of the national standard GB/T15237.1-2000 "Terminology Working Vocabulary" is that a term refers to a term referring to a general concept in a specific professional field, and is a word or phrase used in a subject area to express a concept or relationship in the subject area . Terminology can be divided into general terms used in daily life and domain terms used in specific fields. General terms are mostly formed according to people's living and working habits, and they are not required to be strictly accurate in the expression of concepts, and their meanings are often vague; field terms are systematic and general descriptions of a professional concept, and are not allowed Ambiguity, the concept expressed by each technical term must be accurate and cannot be different due to different users.

Domain term recognition refers to the extraction of professional domain terms from a corpus in a specific scientific or technical field. As an important content of information extraction, automatic recognition of domain terms is widely used in the field of natural language processing. It is important for improving the processing accuracy of domain text indexing and retrieval, text mining, ontology construction, text classification and clustering, and latent semantic analysis. meaning. The existing domain term recognition methods in Chinese text information mainly include:

(1) A method for identifying Chinese domain terms based on statistical methods. The main idea is to extract domain terms by using the higher degree of correlation between the components of the domain term and the domain characteristic information of the term. The general process of the method based on statistics is as follows: First, use the methods in statistics or information theory to establish various statistical information, and determine the more accurate seed words according to the statistical results; then continue to expand on this basis to obtain the final domain terms . Word frequency, mean and variance are relatively commonly used statistical methods, and more scholars use hypothesis testing methods, mainly including T test, chi-square test, log likelihood ratio, point mutual information, etc. Using statistical methods to identify domain terms does not require syntactic and semantic information, is not limited to a specific field, does not depend on any resources, and has strong versatility.

Among them, the mutual information algorithm based on statistics is the most widely used. For example, there is an article report titled "Chinese Term Extraction System Based on Mutual Information" (the author of this article is: Zhang Feng, Xu Yun, Hou Yan, Fan Xiaozhong, published in "Computer Application Research", Volume 22, Issue 5, Issue 2005 72-73, p. 77), which discloses an automatic Chinese term extraction system. The system first calculates the internal combination strength of word strings based on mutual information to obtain a term candidate set; then removes basic words from the term candidate set, and Use common words with prefix and suffix information to further filter; finally, perform lexical analysis on the term candidates, use the part-of-speech composition rules of the terms to judge, and obtain the final term extraction results. The experimental results show that the accuracy rate of term extraction using mutual information algorithm is 72.19%, the recall rate is 77.98%, and the F measurement value is 74.97%. For example, there is a literature report, "Term Extraction Combining C Value and Mutual Information" (authors are: Liang Yinghong, Zhang Wenjing, Zhang Youcheng, published in "Computer Application and Software", Volume 27, Issue 4, pp. 108-110, published in 2010), This paper discloses a term extraction method that combines C-value and mutual information. This method proposes that the comprehensive C-value parameter has advantages in long term extraction. Experimental results show that the accuracy of this method for long term extraction is 75.7% %, the recall rate is 68.4%, and the F-measure value is 71.9%, which are higher than other methods under the same corpus. However, the performance of this algorithm directly depends on the size of the corpus and the word frequency of candidate domain terms. It is difficult to solve the problem of data sparseness that some low-frequency candidate terms may also be legal terms. The rate, recall rate and F measurement value are all difficult to reach more than 80%, and it is difficult to obtain the ideal recognition effect;

(2) The main steps of the machine learning-based Chinese domain term recognition method are: Construct the training corpus manually or semi-automatically, learn and generate a model from the training corpus according to a certain machine learning algorithm, and then use the model to extract domain terms from the test corpus Experiments are carried out to verify the effectiveness of this algorithm. The machine learning theories that have been used for term recognition in the Chinese field mainly include decision trees, support vector machines, hidden Markov models, maximum entropy models, maximum entropy Markov models, and conditional random field algorithms. The term recognition method based on machine learning does not require expert domain knowledge and language knowledge, and it is highly feasible to implement. It can obtain better recognition or extraction results when considering multiple term features.

At present, the conditional random field model is the most widely used in the recognition method of Chinese domain terms based on machine learning. For example, there is a literature report, "A method for automatic extraction of terminology in traditional Chinese medicine" (authors are: Zhang Wudai, Bai Yu, Wang Peiyan, Zhang Guiping, published in "Journal of Shenyang Aerospace University", Volume 28, Issue 1, Issue 72, published in 2011 -75 pages), this paper discloses a term extraction method based on conditional random field for the field of traditional Chinese medicine. , use the CRF toolkit to train a domain term model, and then use the model for term extraction. Selecting "Famous Doctor Class Cases" as the text in the field of traditional Chinese medicine for term extraction experiments, the accuracy rate reached 83.11%, the recall rate reached 81.04%, and the F measurement value reached 82.06%. And the article "Research on Automatic Extraction of Military Intelligence Terminology Using CRF Technology" (Authors: Jia Meiying, Yang Bingru, Zheng Dequan, Yang Jing, published in "Computer Engineering and Application", Volume 45, Issue 32, Pages 126-129, published in 2009), This paper discloses a conditional random field-based term extraction method for the military intelligence field. This method regards domain term recognition as a sequence labeling problem, quantifies the feature distribution of domain terms as the training feature, and uses the CRF toolkit to train A domain term feature template is generated, and then the domain term extraction is performed using the template. Experiments show that the method has good recognition results for terms in the field of military intelligence, with an accuracy rate of 73.24%, a recall rate of 69.57%, and an F-measurement value of 71.36%.

When the conditional random field algorithm is used for domain term recognition, the training corpus is basically manually and semi-automatically annotated. The degree of human participation is high and the workload is heavy, resulting in a small amount of general recognition, which restricts the recognition accuracy and application of the algorithm. At the same time, it is necessary to use general-purpose word segmentation tools to segment the corpus, and then conduct conditional random field training and testing on the segmented corpus to finally realize term recognition. Therefore, the premise of using conditional random field algorithm to identify domain terms is assuming that the existing general word segmentation tools can accurately segment the vocabulary in the field, and it is believed that the domain term is larger than the word granularity segmented by the segmentation tool. However, due to the gap between professional field terminology and common vocabulary, it is difficult to achieve accurate word segmentation of professional field corpus with general word segmentation tools. Therefore, the current mutual information and conditional random field methods have a low degree of automatic recognition in the process of field term recognition, and the recognition accuracy is not high.

Contents of the invention

In view of the problems in the prior art described above, the purpose of the present invention is to provide a method for identifying Chinese domain terms based on mutual information and conditional random field models, which can not only overcome the data sparseness of legal terms when identifying terms, but also The computational load of the conditional random field algorithm is reduced, and the recognition accuracy of Chinese domain terms can be improved.

In order to achieve the above object, the present invention adopts following technical scheme:

The Chinese domain term recognition method based on mutual information and conditional random field model of the present invention, concrete steps are as follows:

(1) Collect domain text corpus and mark all punctuation marks, spaces, numbers, ASCII characters and characters other than Chinese characters in the corpus;

，计算字串

的互信息值； (2), set the string

, to calculate the string

mutual information value;

左右信息熵； (3), calculate the string

left and right information entropy;

评价函数，设置评价函数阈值，计算各字串的评价函数值，确定字串

为词，依次比较该字串

中前一字

的评价函数值与后一字

评价函数值相比较，得到各字串

中对应的比值，其比值再与评价函数阈值比较，逐一对字义字串

分词；  (4), define the string

evaluation function, set the evaluation function Threshold, calculate the evaluation function value of each string, determine the string

is a word, compare the strings in turn

Chinese character

The value of the evaluation function and the next word

Compare the evaluation function values to get each string

The corresponding ratio in , and its ratio is then compared with the evaluation function Threshold comparison, one-by-one for literal strings

Participle;

(5) Using the training characteristics of the random field of word, part of speech, and word frequency, use the conditional random field method to train a field term conditional random field model, and use this model to identify field terms.

，计算字串

的互信息值，其计算公式如下： (2) setting string described in step (2) above

, to calculate the string

The mutual information value of , its calculation formula is as follows:

为一个领域术语，那么字串

由

、

、

……

个字组成，字串

的互信息值计算公式如下： Assuming that a domain term is composed of n words, if the string

is a domain term, then the string

Depend on

,

,

...

composed of words, string

The formula for calculating the mutual information value is as follows:

            （1）

(1)

表示一个由n个字组成的字串； in,

Represents a string consisting of n characters;

的第i个字（i=1，2，3，…，n）；  Indicates the composition of the string

The ith word of (i=1, 2, 3, ..., n);

表示语料库中字

出现的频次；

Represents words in the corpus

frequency of occurrence;

表示语料库中字

出现的频次；

Represents words in the corpus

frequency of occurrence;

表示语料库中字出现的频次；

Represents words in the corpus frequency of occurrence;

Represents words in the corpus frequency of occurrence;

表示字

、

、

、…、

同时出现的频次；

Indicates the word

,

,

,...,

frequency of simultaneous occurrences;

中所有字与字之间的互信息。 Represents a string

Mutual information between all words in .

The calculation formula for calculating the left and right information entropy described in the above step (3) is as follows:

     （2） The left information entropy calculation formula is:

(2)

      （3） The right information entropy calculation formula is:

(3)

表示为给定的一个由n个字组成的字串； in,

Represented as a given string consisting of n characters;

和分别表示出现在

左侧和右则时的条件概率；

and Respectively Appear in

Conditional probabilities for left and right time;

和

表示

左边和右边所有出现的词集合；

and

express

The set of all occurrences of words on the left and right;

表示组成字串

的第i个字，其中，i=1，2，3，…，n 。 

Indicates the composition of the string

The i-th word of , where i=1, 2, 3,..., n.

为词的可信度进行评价，判断该字串是否为词，其中，字串W评价函数计算公式如下：  The definition of the word string W evaluation function described in the above step (4), and the use of the evaluation function to segment the corpus refers to the use of the mutual information and left and right information entropy values calculated in steps (2) and (3), and the corpus string in

Evaluate the credibility of the word, and judge whether the word string is a word, wherein, the word string W evaluation function calculation formula is as follows:

                 （4）

(4)

表示为给定的一个由n个字组成的字串； in,

Represented as a given string consisting of n characters;

表示字串

中字符之间的互信息值；

Represents a string

Mutual information value between characters in ;

表示字串

的左信息熵值；

Represents a string

The left information entropy value of ;

表示字串的右信息熵值；

Represents a string The right information entropy value of ;

为平衡因子，用以调节信息熵与互信息值在字串

评价函数中的权值。

is a balance factor, used to adjust the value of information entropy and mutual information in the string

Weights in the evaluation function.

In the above step (5), the training characteristics of the random field of word, part of speech, and word occurrence frequency are used to train a field term conditional random field model by using the conditional random field method, and the field term recognition is carried out by using this model. The operation steps are as follows:

(51), mark in the corpus with the word itself, part of speech, and frequency of occurrence of the word;

(52), use the CRF++ 0.53 toolkit to train the marked feature sequence to obtain the conditional random field parameter, which is the conditional random field model for term recognition in this field;

(53). Using the conditional random field model of field term recognition to test the field term recognition of the marked feature sequence.

Compared with the prior art, the Chinese domain term recognition method based on mutual information and conditional random field model of the present invention has the following effects:

(1) This method organically combines two types of term recognition methods based on statistics and machine learning, and effectively solves the problem of data sparseness when simply using statistical methods for term recognition;

(2) This method uses the mutual information algorithm to segment and label the corpus, realizing the automatic labeling of the corpus;

(3) This method only uses the three most common word features as the training of the conditional random field method, which makes the method have strong field versatility, effectively reduces the calculation amount of the conditional random field, and reduces the condition Random field training time.

Description of drawings

Fig. 1 is the flow chart of the Chinese domain term recognition method based on mutual information and conditional random field model of the present invention;

Fig. 2 is the flowchart of step (4) among Fig. 1;

Fig. 3 is a flowchart of step (5) in Fig. 1 .

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

In this embodiment, the field term recognition of plant-bamboo is taken as an example to illustrate the present invention, but it is not used to limit the scope of the present invention.

With reference to Fig. 1, the Chinese field term recognition method based on mutual information and conditional random field model of the present invention, comprises the steps:

(1) Collect domain text corpus, and mark all punctuation marks, spaces, numbers, ASCII characters and characters other than Chinese characters in the corpus.

For example, this example selects the e-book manuscript of the ninth volume of "Flora of China" as the domain text corpus.

First, the corpus is randomly divided into two parts according to the ratio of 4:1: training corpus and test corpus;

Then, retrieve all punctuation marks, spaces, numbers, ASCII characters and characters other than Chinese characters in the corpus, and mark them with "//" symbols before and after the above characters;

Finally, referring to the Chinese Parts of Speech Table, for all pronouns, interjections, auxiliary words and function words, as well as the initial characters of "he, you, de, get, will, put, from, got, is, then, in, every, this, the , Give, So, Make, For, Not, Write, Got, Very, Should, With, Get, The front and back of the words are marked with "//" symbols respectively.

，计算字串的互信息值，其计算公式如下： (2), set the string

, to calculate the string The mutual information value of , its calculation formula is as follows:

为一个领域术语，那么字串

由

、

、

……

个字组成，字串

的互信息值计算公式如下： Assuming that a domain term is composed of n words, if the string

is a domain term, then the string

Depend on

,

,

...

composed of words, string

The formula for calculating the mutual information value is as follows:

          （1）

(1)

表示一个由n个字组成的字串； in,

Represents a string consisting of n characters;

表示组成字串

的第i个字，其中，i=1，2，3，…，n； 

Indicates the composition of the string

The i-th word of , where i=1, 2, 3,..., n;

表示语料库中字出现的频次；

Represents words in the corpus frequency of occurrence;

表示语料库中字出现的频次；

Represents words in the corpus frequency of occurrence;

表示语料库中字

出现的频次；

Represents words in the corpus

frequency of occurrence;

表示语料库中字出现的频次；

Represents words in the corpus frequency of occurrence;

、、

、…、同时出现的频次； Indicates the word

, ,

,..., frequency of simultaneous occurrences;

中所有字与字之间的互信息。 Represents a string

Mutual information between all words in .

Because the present invention thinks that the length of the term in the Chinese field is not more than 4 characters, and thinks that punctuation marks, spaces, numbers, ASCII characters, and characters other than Chinese characters cannot appear in the middle of the Chinese field term, and it is also impossible to appear interjections, function words, and indicators Pronouns and other words, so the present invention calculates the mutual information values of 2-word, 3-word, and 4-word for all words in the corpus text, and stops calculation when the marker "//" is encountered, and the calculation of the mutual information value For formulas, refer to formulas (1), (2) and (3) in step (2) in the above summary of the invention.

，

，

，

，； For example: the corpus "edge is fringed hair //, //", where 2-words include: "edge", "marginal cover", "be flowed", "tassel", "su-like" and "like hair"; 3-word includes: "edge is flowed", "edge is flowed", "be tasseled", "fringe-like" and "su-like hair"; 4-word includes: "edge is flowed", "edge is tasseled", "Fringed" and "fringed hair", the partial mutual information calculation results are:

,

,

,

, ;

左右信息熵，其计算公式如下：   (3), calculate the string

Left and right information entropy, its calculation formula is as follows:

The left information entropy calculation formula is:

               （2）

(2)

The right information entropy calculation formula is:

                （3）

(3)

表示为给定的一个由n个字组成的字串； in,

Represented as a given string consisting of n characters;

和

分别表示

出现在

左侧和右则时的条件概率；

and

Respectively

Appear in

Conditional probabilities for left and right time;

表示

左边和右边所有出现的词集合； and

express

The set of all occurrences of words on the left and right;

的第i个字，其中，i=1，2，3，…，n。  Indicates the composition of the string

The i-th word of , where i=1, 2, 3,..., n.

To judge whether a string is a word or not, not only the degree of combination between characters within the string, that is, the size of the mutual information between words, but also the degree of freedom of boundaries between strings, that is, the The more types of adjacent words appearing on the string boundary, the greater the left and right information entropy of the string is considered, that is, the greater the degree of freedom of the word string boundary, and the calculation formula of the left and right information entropy refers to the formula of step (3) in the above-mentioned content of the invention ( 2), (3).

，

,

，

，

，

；右信息熵计算结果为：

，，

，， 

，

； For example: in the corpus "the edge is fringed hair //, //", the calculation result of part of the left information entropy is:

,

,

,

,

,

; The calculation result of right information entropy is:

, ,

, ,

,

;

评价函数，设置评价函数

阈值，计算各字串的评价函数值，确定字串为词，依次比较该字串

中前一字

的评价函数值与后一字

评价函数值相比较，得到各字串

中对应的比值，其比值再与评价函数

阈值比较，逐一对字义字串

分词，其操作步骤如下： (4), define the string

evaluation function, set the evaluation function

Threshold, calculate the evaluation function value of each string, determine the string is a word, compare the strings in turn

Chinese character

The value of the evaluation function and the next word

Compare the evaluation function values to get each string

The corresponding ratio in , and its ratio is then compared with the evaluation function

Threshold comparison, one-by-one for literal strings

Word segmentation, the operation steps are as follows:

评价函数，其计算表达式为： (41), define the string

Evaluation function, its calculation expression is:

                  （4）

(4)

表示为给定的一个由n个字组成的字串； in,

Represented as a given string consisting of n characters;

表示字串

中字符之间的互信息值；

Represents a string

Mutual information value between characters in ;

表示字串

的左信息熵值；

Represents a string

The left information entropy value of ;

表示字串的右信息熵值；

Represents a string The right information entropy value of ;

It is a balance factor, which is used to adjust the weight of information entropy and mutual information value in the evaluation function.

为词。 (42), respectively calculate the value of the evaluation function, determine the string

for words.

取0.5，并认为当评价函数大于阈值0.8时，该字串

为词， Calculate the evaluation function values of all strings according to the evaluation function formula of step (4) in the above-mentioned summary of the invention, wherein

Take 0.5, and think that when the evaluation function When greater than the threshold 0.8, the string

for words,

，，

，

， 

，

； For example: in the corpus "the edge is fringed hair //, //", the calculation result of some evaluation functions is:

, ,

,

,

,

;

中前一字

的评价函数值与后一字

评价函数值相比，得到各字串

中对应的比值“？”，其比值再与评价函数

阈值比较，逐一对字义字串

分词。 (43), compare the above strings in turn

Chinese character

The value of the evaluation function and the next word

Evaluation function values are compared to get each string

The corresponding ratio "?" in, and its ratio is then compared with the evaluation function

Threshold comparison, one-by-one for literal strings

Participle.

、

、

和

； For example, firstly, starting from the first word of the corpus, select substrings with lengths of 4, 3, 2, and 1 respectively, and write them as

,

,

and

;

和

的评价函数进行比较，如果

，认为字串为新词，d在字串

前后分别以符号“*”进行标注；反之，认为字串

不是新词，则其丢弃尾部的最后一个字，对

和

的评价函数进行比较，如果

，认为字串

为新词，在字串

前后分别以符号“*”进行标注；反之，认为字串

不是新词，其丢弃尾部的最后一个字对

的评价函数进行判断，如果

，认为字串

为新词，在字串

前后分别以符号“*”进行标注；反之，认为字串

为新词，在字串

前后分别以符号“*”进行标注；只要有新词被标注，就从新词后的第一个字开始，再分别选取长度为4、3、2、1的子字串，记作

、、

和，重新进行评价函数的比较，当遇到“//”符号跳过。如此反复, 直至所以语料处理完为止，例如：语料“边缘被流苏状毛//，//”,首先，从第一个字开始截取长度分别为4、3、2、1的子字串，即：“边缘被流”、“边缘被”、“边缘”和“边”；然后，首先判断

是否大于等于0.8,根据步骤（41）评价函数的计算结果，可知

小于0.8，即字串“边缘被流”不是新词；然后，判断

是否大于等于0.8,根据步骤（41）评价函数的计算结果，可知

小于0.8，故字串“边缘被”也不是新词；接着判断

是否大于等于0.8,根据步骤（41）评价函数的计算结果，可知

大于0.8，故字串“边缘”是新词；当有判断出新词后，从新词后的第一个字开始再选取4、3、2、1个字串，作为新一轮的作

、

、

和，即“被流苏状”、“波流苏”、“被流”和“被”，再重复以上步骤进行比较，当遇到“//”符号跳过，直到结束，所以语料“边缘被流苏状毛//，//”，最后分词结果为“*边缘*被*流苏状*毛//,//” ； Then, for the string

and

The evaluation function is compared, if

, consider the string as a new word, d in the string

Mark with the symbol "*" before and after; otherwise, consider the string

is not a new word, it discards the last word at the end, for

and

The evaluation function is compared, if

, consider the string

for new words, in the string

Mark with the symbol "*" before and after; otherwise, consider the string

is not a new word, which discards the last word pair at the end

Evaluation function to judge, if

, consider the string

for new words, in the string

Mark with the symbol "*" before and after; otherwise, consider the string

for new words, in the string

The symbol "*" is used to mark the front and back respectively; as long as a new word is marked, start from the first word after the new word, and then select substrings with lengths of 4, 3, 2, and 1 respectively, and write them as

, ,

and , to re-comparison the evaluation function, and skip when the "//" symbol is encountered. Repeat this until all the corpus is processed, for example: the corpus "the edge is fringed hair //, //", first, start from the first word to intercept substrings with lengths of 4, 3, 2, 1 respectively, That is: "edge is flowed", "edge is", "edge" and "edge"; then, first judge

Whether it is greater than or equal to 0.8, according to the calculation result of the evaluation function in step (41), it can be known that

is less than 0.8, that is, the word string "edge is flowed" is not a new word; then, judge

Whether it is greater than or equal to 0.8, according to the calculation result of the evaluation function in step (41), it can be known that

is less than 0.8, so the word string "marginal quilt" is not a new word; then judge

Whether it is greater than or equal to 0.8, according to the calculation result of the evaluation function in step (41), it can be known that

is greater than 0.8, so the word string "edge" is a new word; when a new word is judged, select 4, 3, 2, and 1 word strings from the first word after the new word as a new round of writing

,

,

and , that is, "be fringed", "wave tassel", "be flowed" and "be", repeat the above steps for comparison, when encountering the "//" symbol skip until the end, so the corpus "edge is fringed" Mao//, //", the final word segmentation result is "*edge*be*fringed*hair//,//";

(5) Using the training characteristics of the random field of word, part of speech, and word frequency, use the conditional random field to train a field term conditional random field model, and use this model to identify field terms. The operation steps are as follows:

(51), mark in the corpus with the word itself, part of speech, and frequency of occurrence of the word, as follows:

分词标注特征序列，该词的标注的特征序列分别为：当前词本身；当前词的词性；当前词的出现频率，采用K-Means聚类方法，将上述当前词的出现频率分为10个等级，每个等级为一类，10个类分别表示为A、B、C、D、E、F、G、H、I、J、K，将已标注的特征序列分为：训练已标注的特征序列、测试已标注的特征序列两部份； literal string

The feature sequence of word segmentation, the feature sequence of the tag of the word is: the current word itself; the part of speech of the current word; the frequency of occurrence of the current word, using the K-Means clustering method to divide the frequency of the above current word into 10 levels , each level is a class, and the 10 classes are respectively represented as A, B, C, D, E, F, G, H, I, J, K, and the marked feature sequence is divided into: training marked features Sequence, test the two parts of the marked feature sequence;

(52), use the CRF++ 0.53 toolkit to train the marked feature sequence, and obtain the conditional random field parameters, which are the conditional random field models for domain term recognition;

(53), use the conditional random field model of field term recognition to test the field term recognition of the marked feature sequence, which is as follows:

Input the marked feature sequence of the test into the conditional random field model for field term recognition after training in step (5.2), use the conditional random field model to calculate the feature value, identify the field term, and output the recognized field term , For example: the corpus "edge is fringed hair //, //", and finally recognizes "edge" and "fringe" as domain terms.

The above are the best implementation modes of the present invention. According to the disclosed content of the present invention, those skilled in the art can obviously think of some similarities and alternatives, which should all belong to the technical innovation scope of the present invention.

Claims (5)

1. Chinese field term recognition methods based on mutual information and conditional random field models, concrete steps are as follows:

(1), the assembling sphere corpus of text, character beyond punctuation marks all in the language material, space, numeral, ascii character and the Chinese character is carried out mark;

(2), word string is set

, calculate word string

The mutual information value;

(3), calculate word string

Left and right sides information entropy;

(4), definition word string Evaluation function arranges evaluation function

Threshold value is calculated the evaluation function value of each word string, determines word string

Be word, successively this word string relatively

Middle prev word

Evaluation function value and a rear word

The evaluation function value is compared, and obtains each word string

The ratio of middle correspondence, its ratio again with evaluation function

Threshold ratio, one by one to meaning of word word string

Participle;

(5), with the training characteristics of the random field of the frequency of occurrences of word, part of speech, word, utilize the condition random field method to train a field term conditional random field models, with this model to carrying out field term identification.

2. the Chinese field term recognition methods based on mutual information and conditional random field models according to claim 1 is characterized in that, described in the above-mentioned steps (2) word string is set

, calculate word string

The mutual information value, its computing formula is as follows:

Suppose that a field term is comprised of n word, if word string

It is field term, so a word string

By

,

,

Individual word forms, word string

Mutual information value computing formula as follows:

（1）

Wherein,

Represent a word string that is formed by n word;

Expression forms word string

I word (i=1,2,3 ..., n);

Word in the expression corpus

The frequency that occurs;

Word in the expression corpus

The frequency that occurs;

Word in the expression corpus The frequency that occurs;

Word in the expression corpus

The frequency that occurs;

The expression word

, ,

...,

The frequency that occurs simultaneously;

The expression word string

In mutual information between all words and the word.

3. the Chinese field term recognition methods based on mutual information and conditional random field models according to claim 1 is characterized in that, the calculating left and right sides information entropy described in the above-mentioned steps (3), and its computing formula is as follows:

Left information entropy computing formula is:

(2)

Right information entropy computing formula is:

(3)

Wherein,

Be expressed as a given word string that is formed by n word;

With

Respectively expression Appear at

Left side and right conditional probability then the time;

With

Expression

The set of words that the left side and the right occur;

Expression forms word string

I word, wherein, i=1,2,3 ..., n.

4. the Chinese field term recognition methods based on mutual information and conditional random field models according to claim 1, it is characterized in that, definition word string W evaluation function described in the above-mentioned steps (4), and utilize evaluation function that language material is carried out participle, refer to the mutual information and the left and right sides information entropy that utilize step (2) and step (3) to calculate, to the word string in the language material

Estimate for the confidence level of word, judge whether this word string is word, and wherein, word string W evaluation function computing formula is as follows:

（4）

Wherein, Be expressed as a given word string that is formed by n word;

The expression word string

Mutual information value between the middle character;

The expression word string

Left information entropy;

The expression word string

Right information entropy;

Be balance factor, in order to regulate information entropy and mutual information value in word string

Weights in the evaluation function.

5. the Chinese field term recognition methods based on mutual information and conditional random field models according to claim 1, it is characterized in that, the training characteristics with the random field of the frequency of occurrences of word, part of speech, word described in the above-mentioned steps (5), utilize the condition random field method to train a field term conditional random field models, utilize this model to carrying out field term identification, its operation steps is as follows:

(51), the frequency of occurrences with word itself, part of speech, word marks in language material;

(52), utilize CRF++ 0.53 kit to the training of the characteristic sequence that marked, obtain the condition random field parameter, the conditional random field models that this condition random field parameter is identified for this field term;

(53), with field term identification the field term identification of conditional random field models characteristic sequence that test has been marked.

Images