JP5239161B2 - Language analysis system, language analysis method, and computer program - Google Patents

Description

  The present invention relates to a language analysis system, a language analysis method, and a computer program. More specifically, the present invention relates to a language analysis system, a language analysis method, and a computer program that execute term extraction by text analysis.

  For example, a process for extracting a term applied in data processing from a natural language document such as a search key applied in a database search or an index as an index of a term dictionary is a technique required in various data processing fields. is there.

  A collection of various text data is called a corpus. Research on extracting terms from documents included in a corpus, that is, extracting terms as meaningful language units, has been performed. For example, if it is a common document such as [car runs on road], it is possible to extract morphemes such as [car], [road], and [run] by applying a general morphological analysis system. . The morpheme analysis system is known as a system that applies a predetermined dictionary for morpheme analysis and performs a part-of-speech recognition process by segmenting into morphemes that are semantic minimum units based on dictionary registered words. .

  However, it is difficult to divide technical terms in highly specialized fields such as the medical field into appropriate morphemes. That is, there are few dictionaries that comprehensively register terms used in highly specialized fields such as the medical field, and technical terms that are not extracted are generated even if language analysis that depends only on the dictionary is performed.

  Examples of conventional techniques disclosing the term extraction processing from text include the following conventional techniques. Non-Patent Document 1 (Shimohata, S. Sugio, T. Nagata, J. Retrieving collections by co-ocurrenses and word order constraints. Proc. Of ACL scale). Discloses a technique for extracting collocations. This non-patent document 1 is configured to calculate entropy (information amount) in a corpus of collocations when extracting collocations (collocations), and extract collocations where the entropy of words on both sides exceeds a set threshold.

  Japanese Patent Laid-Open No. 9-138801 discloses a technique for extracting an arbitrary continuous character string from a text described in a natural language in consideration of surrounding characters. A configuration that extracts characters that appear in the vicinity of a continuous character string in the text, and those that appear at the same time as the continuous character string exceed a preset threshold, are extracted as words or idioms together as a continuous character string It is.

  Non-patent document 2 (Nakakawa, Mori, Yumoto. Extraction of technical terms based on appearance frequency and connection frequency. Natural language processing Vol.10 No.1, January 2003) automatically extracts technical terms from a specialized corpus. The method to do is disclosed. It is a configuration that performs morphological analysis on the corpus and extracts nouns, calculates a score using the frequency of appearance of nouns or compound nouns in the corpus and the frequency of nouns connected to the constituent nouns, This is a configuration for ranking compound nouns.

  Further, Patent Document 2 (Japanese Patent Laid-Open No. 2006-139686) discloses a technique for performing unknown word extraction with high accuracy. When text is given, processing to extract word set by obtaining word sequence of text using statistic of character string in text set prepared in advance and processing to obtain morpheme set by morphological analysis These processes are used to perform the two processes, and the character strings obtained by these processes, except for those included in the morpheme of the specific part-of-speech obtained, those whose statistics are above a certain level are unknown words It is the structure extracted as.

Further, in Patent Document 3 (Japanese Patent Laid-Open No. 2006-3295), n-gram probabilities of words are calculated from a first text set that has been divided into words and a second text set that has not been divided into words. A technique for improving accuracy is disclosed. The probability of adjacent characters or character types becoming word boundaries is determined from the first text set, and a division probability is assigned between each character in the second corpus to improve the accuracy of natural language processing for the second corpus It is the structure to make.
Japanese Patent Laid-Open No. 9-138801 JP 2006-139686 A JP 200631295 A Shimohata, S .; Sugio, T .; Nagata, J .; Retrieving associations by co-ocurrences and word order constraints. Proc. of ACL / EACL-97. Nakagawa, Mori, Yumoto. Terminology extraction based on appearance frequency and connection frequency. Natural language processing Vol. 10 No. 1,2003 January

  The present invention relates to a language analysis system, a language analysis method, and a language analysis method capable of extracting even a term not registered in a dictionary such as a morphological analysis dictionary in a configuration for extracting a term by text analysis, and The object is to provide a computer program.

The first aspect of the present invention is:
A character string extraction unit that extracts a set of character strings having a predetermined number of characters or less from text data;
Using the analysis processing for text in the text database, the character string extracted by the character string extraction unit is used as the analysis target character string, and an evaluation value representing the branch state of the appearance character string at the boundary of the analysis target character string is calculated. A branch state evaluation value calculation unit for
A score setting unit that sets a score corresponding to the analysis target character string based on the evaluation value calculated by the branch state evaluation value calculation unit;
A word determination unit that determines whether or not the analysis target character string is a word based on the score set by the score setting unit;
A language analysis system characterized by having

  Furthermore, in one embodiment of the language analysis system of the present invention, the evaluation value calculated by the branch number calculation unit is an outward appearance character of the analysis target character string at one or more characters at each end of the analysis target character string. An evaluation value indicating a branching state of a column, and an evaluation value indicating a branching state of an appearance character string in an analysis target character string direction at each of one or more characters at both ends contacting the analysis target character string, The score setting unit is configured to calculate a score based on the evaluation value calculated by the branch number calculation unit.

  Furthermore, in one embodiment of the language analysis system of the present invention, the score setting unit is an evaluation value indicating a branching state of the outward appearance character string of the analysis target character string at one or more characters at both ends of the analysis target character string And a score is calculated based on the minimum value of the evaluation values indicating the branching state of the appearance character string in the direction of the analysis target character string in one or more characters at both ends in contact with the analysis target character string. Features.

  Furthermore, in one embodiment of the language analysis system according to the present invention, the language analysis system further divides a word registered in the word database into units of m characters (where m ≧ 1 is a predetermined number). And the branch state evaluation value calculation unit calculates a plurality of evaluation values representing the branch state of the character string appearing at the boundary of the character string to be analyzed corresponding to the character string in m characters divided by the word dividing unit. The average value of a plurality of evaluation values excluding the evaluation value indicating the branching state of the externally appearing character string at the end of the character string in m characters from the plurality of evaluation values calculated by the branch state evaluation value calculation unit And the word determination unit performs a comparison between the score set by the score setting unit and the threshold set by the threshold setting unit, and the analysis target according to a comparison result String is word Characterized in that it is configured for judging that there.

  Furthermore, in an embodiment of the language analysis system of the present invention, the language analysis system further executes a filtering process for deleting a character string that does not hold as a word from the set of character strings extracted by the character string extraction unit. The branch state evaluation value calculation unit includes a character string after filtering in the character string filter unit as the analysis target character string, and the branch state of the appearance character string at the boundary of the analysis target character string. It is the structure which performs the process which calculates the evaluation value to represent.

  Furthermore, in one embodiment of the language analysis system of the present invention, the language analysis system further includes a head end character for extracting a head character and a tail character string of the extracted character string from the extracted character string extracted by the character string extraction unit. The branch state evaluation value calculation unit represents a branch state of the appearance character string at the boundary of the analysis target character string based on the head / end character string extracted by the head / end character string extraction unit; It is the structure which performs calculation of an evaluation value, It is characterized by the above-mentioned.

  Furthermore, in one embodiment of the language analysis system of the present invention, the language analysis system further includes a partial character string extraction unit that extracts a partial character string of the extracted character string from the extracted character string extracted by the character string extraction unit. The branch state evaluation value calculation unit calculates an evaluation value representing a branch state of the appearance character string inside the analysis target character string based on the partial character string extracted by the partial character string extraction unit. The word determination unit executes the internal set based on the score set by the score setting unit and the evaluation value calculated by the branch state evaluation value calculation unit regarding the partial character string extracted by the partial character string extraction unit It is the structure which performs the process which determines whether the said analysis object character string is a word based on a score.

  Furthermore, in one embodiment of the language analysis system of the present invention, the language analysis system is configured to execute word extraction processing in units of text stored in a text database.

  Furthermore, in one embodiment of the language analysis system of the present invention, the evaluation value calculated by the branch state evaluation value calculation unit is a perplexity that represents the number of branches of the appearance character string at the boundary of the analysis target character string. It is characterized by.

Furthermore, the second aspect of the present invention provides
It is a dictionary that registers word data extracted based on text analysis by a language analysis system,
In the language analysis system,
A character string extraction unit that extracts a set of character strings having a predetermined number of characters or less from text data;
Using the analysis processing for text in the text database, the character string extracted by the character string extraction unit is used as the analysis target character string, and an evaluation value representing the branch state of the appearance character string at the boundary of the analysis target character string is calculated. A branch state evaluation value calculation unit for
A score setting unit that sets a score corresponding to the analysis target character string based on the evaluation value calculated by the branch state evaluation value calculation unit;
A word determination unit that determines whether or not the analysis target character string is a word based on the score set by the score setting unit;
Have
It exists in the dictionary which has the word data determined as a word in the said word determination part as registration data.

Furthermore, the third aspect of the present invention provides
A language analysis method for executing language analysis processing in a language analysis system,
A character string extraction step in which the character string extraction unit extracts a set of character strings having a predetermined number of characters or less from the text data;
The branch number calculation unit uses the character string extracted in the character string extraction step by the analysis processing for the text in the text database as the analysis target character string, and the branch state of the appearance character string at the boundary of the analysis target character string A branch state evaluation value calculating step for calculating an evaluation value representing
A score setting step in which the score setting unit sets a score corresponding to the character string to be analyzed based on the evaluation value calculated in the branch state evaluation value calculation step;
A word determination step in which a word determination unit determines whether or not the analysis target character string is a word based on the score set in the score setting step;
A language analysis method characterized by comprising:

Furthermore, the fourth aspect of the present invention provides
A computer program that executes language analysis processing in a language analysis system,
A character string extraction step for causing the character string extraction unit to extract a set of character strings having a predetermined number of characters or less from the text data;
In the branch number calculation unit, the character string extracted in the character string extraction step by the analysis processing for text in the text database is set as the analysis target character string, and the branch state of the appearance character string at the boundary of the analysis target character string A branch state evaluation value calculating step for calculating an evaluation value representing
A score setting step for causing the score setting unit to set a score corresponding to the analysis target character string based on the evaluation value calculated in the branch state evaluation value calculating step;
A word determination step for causing a word determination unit to determine whether or not the analysis target character string is a word based on the score set in the score setting step;
In a computer program characterized by causing

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

According to the first aspect of the present invention, it is possible to extract a character string more legitimate as a word even if the appearance frequency is low as compared with a method of extracting a word that is not registered using a word registered in a dictionary. Can do.
According to the invention described in claim 2, since the boundary of the extracted character string can be determined more accurately, a character string more legitimate as a word can be extracted.
According to the third aspect of the present invention, even if the boundary where the diversity of the branch state of the appearing character string is small is adjacent to the end of the character string or this end, the character more legitimate as the word A column can be extracted.
According to the fourth aspect of the present invention, the threshold for the character string divided by the predetermined number of characters is set more appropriately.
According to the fifth aspect of the present invention, unnecessary branch state evaluation processing is reduced.
According to the sixth aspect of the present invention, the extraction of the character string can be evaluated based on the branch state at the boundary of the character string, which is highly appropriate for determining the word break position.
According to the seventh aspect of the present invention, the extraction of the character string can be evaluated based on whether or not the partial character string of the extracted character string exists inside.
According to the eighth aspect of the present invention, it is possible to use texts that are easily available and abundant in amount of information as compared with a dictionary in which a predetermined number of words are registered. It is possible to extract more legitimate character strings as fewer words.
According to the ninth aspect of the present invention, it is possible to provide a system that can easily set and adjust conditions related to character string extraction.
According to the invention described in claim 10, a character string that has not been registered can be used as a registered word in the dictionary.
According to the eleventh aspect of the present invention, a more legitimate character string can be extracted as a word even if the frequency of occurrence is low compared to a method of extracting a word that is not registered using a word registered in the dictionary. Can do.
According to the twelfth aspect of the present invention, compared to a method of extracting a word that is not registered using a word registered in the dictionary, a character string that is more legitimate as a word is extracted even if the appearance frequency is low. Can provide a computer program that can

  Hereinafter, a language analysis system, a language analysis method, and a computer program according to embodiments of the present invention will be described in detail with reference to the drawings.

[Example 1]
With reference to FIG. 1, the configuration and processing of a language analysis system according to an embodiment of the present invention will be described. As shown in FIG. 1, a language analysis system 100 according to an embodiment of the present invention includes a text input unit 101, a character string extraction unit 102, a character string filter unit 103, a head / end character string extraction unit 104, and a peripheral character string extraction. Unit 105, branch number calculation unit 106, word division unit 107, score setting unit 108, threshold setting unit 109, word determination unit 110, text database 121, and word database 122.

  The language analysis system according to the present embodiment is a system that performs word extraction processing from document text to be analyzed, and extracts words that are not registered in the dictionary or words that do not appear frequently in the text. It has a configuration that enables. In other words, when an analysis target text set for word extraction is given, it is an apparatus that can extract a character string as an unknown word regardless of the result of morphological analysis or the appearance frequency in the text set. .

  As typical word extraction processing, for example, there is processing for selecting a word registered in a dictionary from text by referring to a word registration dictionary (standard dictionary) by morphological analysis. In addition, for example, when an effective special area dictionary such as the medical field cannot be used, a process of extracting an unknown word by regarding a word frequently appearing in the text set of the special area as a word similar to the dictionary registration data may be performed. . However, with such a configuration, it is difficult to extract words that are not registered in the dictionary or character strings that appear less frequently in the text set as words.

  The language analysis system 100 according to the present embodiment shown in FIG. 1 does not depend on dictionary registration data, and extracts even a character string having a low appearance frequency in a specific text set as a word. Realize. Details of the processing of the language analysis system 100 shown in FIG. 1 will be described below.

  First, the configuration of the text database 121 and the word database 122 applied in the language analysis system 100 shown in FIG. 1 will be described with reference to FIG. The text database 121 is a database storing various texts as shown in FIG. 2A, and an ID as an identifier is set for each text. In the present embodiment, an example using a text database in which texts in the medical field are collected will be described.

  The word database 122 registers data associating [surface layer] indicating a morpheme as a word obtained as a result of morpheme analysis with a dictionary or text [type] as data recorded for the word [surface layer]. In each registered entry, [ID] as an identifier is set.

  In the language analysis system according to the present embodiment shown in FIG. 1, a threshold setting process is executed as a process before the word extraction process is performed. Hereinafter, “threshold setting processing” and “word extraction processing” executed in the language analysis system according to the present embodiment will be described in order.

[Threshold setting process]
In the word extraction processing from the text to which the language analysis system 100 according to the present embodiment shown in FIG. 1 is applied, first, threshold setting processing based on the words registered in the word database 122 is executed. This processing is executed as processing of the word division unit 107, the branch number calculation unit 106, and the threshold setting unit 109 of the language analysis system 100 shown in FIG. As an evaluation value of the branch state in this embodiment, perplexity that is an index of the number of branches is used. The calculation method of string perplexity is, for example, Kitakenji. Stochastic language model. Details are described in “The University of Tokyo Press, 1999”. First, the threshold setting process will be described with reference to the flowchart shown in FIG.

First, in step S101, the word dividing unit 107 divides a word registered in the word database 122 by m characters on the surface layer, and extracts a set of “different character strings”. m is set to a small value in the first to third places. It is registration data in the word database shown in FIG.
[Lymph node]
[Hand fractures]
A word division process for these [surface] data will be described.

When m = 1 and different character string sets are extracted,
[Lymph node] = "li", "n", "pa", "node"
[Hand joint fracture] = "hand", "between", "contact", "bone", "fold"
A set of these “different character strings” is extracted.

  In the next step S102, the branch number calculation unit (perplexity calculation unit) 106 calculates the perplexity of the “different character string” extracted in step S101.

The perplexity of the character string is calculated based on the entropy value of the character string that appears to the left and right with respect to a certain character string [W]. First, in the entropy calculation formula below, the entropy value of a character string appearing on the left and right with respect to the character string [W] is obtained.

In the above formula,
W _L : Character string set appearing on the left side W _R : Character string set appearing on the right side n: Number of character strings in the set
The following entropy value is calculated by the above formula.
H (W _L ): Entropy value of the character string appearing on the left with respect to the character string [W],
H (W _R ): the value of the entropy of the character string that appears to the right of the character string [W],

Next, the perplexity of the character string [W] is calculated by the following calculation formula.

In the above formula,
PP (W _L ): Perplexity of the left character string with respect to the character string [W],
PP (W _R ): right string perplexity with respect to the string [W],
It is.

  The perplexity is a value indicating the diversity of character strings appearing on the left and right with respect to a certain character string [W]. Perplexity is often used as an evaluation index in a language model. In this embodiment, perplexity defined by Equation 2 is used as an index indicating the number of branches. When the perplexity value is high, it indicates that there are various character strings appearing on the left and right with respect to a certain character string [W]. When the perplexity value is low, the character string [W] On the other hand, it means that the character strings appearing on the left and right are not diverse, that is, limited.

Next, in step S103, the threshold setting unit 109 executes threshold setting processing. In the perplexity calculated in step S102, an average is calculated by excluding character strings that appear only on the left side at the beginning and the right side at the end of the word, and set as a threshold value [t]. For example, if m = 1 and the word is “lymph node”,
To the right of "Li", "N", "Pa",
"N", "Pa", the left side of "Section",
A perplexity average value is calculated using only these perplexity values, and the calculated perplexity average value is defined as a threshold value [t]. The perplexity on the left side of "Li" and the right side of "Clause" is not used for calculating the average.
The threshold value calculation method described above is an example, and other threshold value calculation methods may be applied. For example, a midpoint may be used instead of the average, and the threshold values may be calculated separately on the left and right sides.

  The threshold value [t] is calculated by the above-described process, and the word extraction process is performed using the threshold value [t].

[Word extraction processing]
Next, details of the word extraction processing executed in the language analysis system 100 shown in FIG. 1 will be described with reference to the flowchart shown in FIG.

First, in step S201, text as a word extraction process target is input to the text input unit 101 of the system. For example, in this embodiment, a processing example for a text included in one text “latest anatomy glossary” in the medical field will be described.
For example, the text “Latest Anatomy Glossary” includes the following text in which each entry is separated by a line feed.
(Text example)
“Parasympathetic nerve roots of the wing palate ganglion Right inner lung branch of the right lung (B7)
Posterior intertemporal vein scleral sinus ... "

Next, in step S202, the character string extraction unit 102 divides the input text with delimiters such as line breaks and punctuation marks, and extracts all partial character strings that can be taken from each divided text. At this time, duplicates are excluded. Further, the maximum length of the partial character string may be set in order to suppress the extraction amount. For example,
"Scleral venous sinus"
When a partial character string is extracted with a maximum length of 3, the following extracted character string is obtained.
{Strong, membranous, static, pulse, sinus, sclera, membranous, vein, sinus, scleral, membranous vein, sinus}

In step S203, the character string filtering unit 103 executes character string filtering processing. Among the character strings extracted in step S202, those including a specific character pattern, those starting with a specific character, those ending with a specific character, those included in the word database 122, and the like are deleted. For example,
(A) including, as a collective of Hiragana, "to," to, to, by, for, and from in to
(B) Things that begin with one of the characters “Ai ぅ e ぉ yayyayuyakyakyyayen + − /% ~ :;”,
(C) Those ending with any of the characters "., ~"
Since these cannot basically be Japanese words, they are deleted. The character string filtering unit 103 stores character string information to be deleted in advance as registration information, and executes character string filtering by applying the registration information.

Further, in the present embodiment, since it is executed for the purpose of extracting words that are not registered in the dictionary, the character string filtering unit 103 includes the character strings extracted in step S202.
(D) a word already registered in the dictionary;
The process of deleting is also executed.
In this processing example, an entry of “latest anatomical terminology” or an IPA dictionary (http://chasen.naist.jp/hiki/ChaSen/), which is known as a standard dictionary for morphological analysis, is pre-worded. It is registered in the database 122 and the matching character string is deleted.

Furthermore, when the purpose is only to extract words that are not extracted by the conventional method,
(E) a character string determined to be a word that can be extracted by a conventional method;
It is good also as a structure which deletes about.
For example, a character string having a high appearance frequency in the text stored in the text database 121 is a word that can be extracted even by a process to which a conventional method is applied, and even if a character string corresponding to these words is deleted. Good.
In addition, it is good also as a structure which does not delete about the character string determined to be the word which can be extracted in a conventional method. In this case, these character strings can also be extracted as words by applying the processing according to the present embodiment.

In step S203, the character string filtering unit 103 executes such a filtering process. In this embodiment, from the character string extracted in step S202,
(1) Character strings that cannot basically be Japanese words (above (a) to (c)),
(2) A word already registered in the dictionary (above (d)),
(3) a character string having a high appearance frequency in the text stored in the text database 121 ((e) above),
Delete these strings. As a result of this filtering process, for example, the character string corresponding to the above (1) to (3) is deleted from the character string extracted in step S202, and the other character string is selected as the analysis target character string. become. Here, “a partial character string having an appearance frequency of 5 or less extracted from the latest anatomical glossary” is set as an analysis target character string set. For example, the following character strings are extracted.
(Analysis target character string set)
BIO
Lower leg bone

Next, in step S204, the first character string is acquired as the analysis target character string from the analysis target character string set selected in step S203. In step S205, the head / end character string extraction unit 104 performs the start of the target character string. And the last m characters are extracted. m is a preset value of 1 or more.
For example, if m = 1 and the selected character string to be analyzed is “lower leg bone”, the first “lower” and the last “bone” are extracted.

The next step S206 is a process of the peripheral character string acquisition unit 105, and a set of m character strings that appear around the target character string by the analysis process for the text data stored in the text database 121. To extract.
For example, from a medical text set stored in the text database 121,
Extract the surrounding character string of the character string "lower leg bone". With the setting of m = 1, one character on the left side and one character on the right side of the character string “crus bone” are extracted. As a result,
Left side string: "・"","
These two types of surrounding strings are obtained,
As the surrounding character string on the right side, “to” “fold”
These two types of surrounding character strings are obtained.

Specifically, the above processing result is stored in a medical text set stored in the text database 121.
(A) "... extensive on thighs and lower leg bones ..."
(B) "... the elephant, the shadow of the broken leg ..."
This is the result when these documents are detected. That is,
From the above (a), one character “·” on the left side of “crus bone”, one character “ni” on the right side,
From (b) above, one letter “,” on the left side of “crus bone”, one letter “fold” on the right side,
These are extracted as surrounding character strings.

The next step S207 is a process of the branch number calculation unit (perplexity calculation unit) 106, and constitutes a document including the analysis target character string “lower leg bone” extracted in the peripheral character string extraction process of step S206. The first and last character strings of the character string to be analyzed “lower leg bone”, and the left and right surrounding character strings of the character string to be analyzed “lower leg bone”. The parsing of these character string sets in the text data stored in the text database 121 Calculate plexity. In other words, in the present processing example, there are the following two documents including the analysis target character string “crus bone” extracted in the surrounding character string extraction process in step S206.
(A) "... extensive on thighs and lower leg bones ..."
(B) "... the elephant, the shadow of the broken leg ..."

From the two documents, the branch number calculation unit (perplexity calculation unit) 106
Select the first and last character strings of the analysis target character string “lower leg bone” and the left and right surrounding character strings of the analysis target character string “lower leg bone”, and calculate the perplexity of each. A specific processing example will be described with reference to FIGS.

As shown in FIG.
(A) "... extensive on thighs and lower leg bones ..."
From this document,
The first character string [lower] and the last character string [bone] of the analysis target character string “crus bone” are selected,
The left perplexity of the first character string [lower] of “lower leg bone”,
The right-side perplexity of the last character string [bone] of “lower leg bone”,
Calculate these, and
The right perplexity of the surrounding character string [•] on the left side of “lower leg bone”,
The left perplexity of the character string [to] to the right of “lower leg bone”,
These are calculated by processing for text stored in the text database 121.

The perplexity is calculated based on the entropy value of the character string that appears to the left and right with respect to a certain character string [W], as described in the description section of [Threshold setting processing], and the character string [ W] is a value indicating the diversity of character strings appearing on the left and right,
PP (W _L ): Perplexity of the left character string with respect to the character string [W],
PP (W _R ): right string perplexity with respect to the string [W],
It is.

Furthermore, as shown in FIG. 6, another extracted document,
(B) "... the elephant, the shadow of the broken leg ..."
Similarly, for this document, a perplexity calculation target character string is selected. Since the first character string [lower] and the last character string [bone] of the analysis target character string “crus bone” have already been selected,
The right perplexity of the left peripheral character string [,] of “lower leg bone”,
The left perplexity of the surrounding character string [fold] on the right side of “lower leg bone”,
These are calculated by processing for text stored in the text database 121.

The branch number calculation unit (perplexity calculation unit) 106 obtains the following value.
(1) Left perplexity of the first character string [lower] of the character string to be analyzed “lower leg bone” (2) Right perplexity of the last character string [bone] of the character string to be analyzed “lower leg bone” (3) Analysis The average value of the right perplexity of the surrounding character string [•] [,] on the left of the target character string “lower leg bone”,
(4) The average value of the left-side perplexity of the right peripheral character string [ni] [fold] of the character string to be analyzed “lower leg bone”,
Each of these values is calculated.

The next step S208 sets the score of the target character string from the calculated perplexity. For example,
(1) Perplexity a on the left side of the first m characters of the character string to be analyzed,
(2) Perplexity b on the right side of the last m characters of the character string to be analyzed,
(3) The average value c of the right perplexity of the set of surrounding character strings of the left m characters of the character string to be analyzed,
(4) Perplexity average value d on the left side of a set of surrounding character strings of m characters on the right side of the analysis target character string.
The minimum value of these four values is set as the score of the target character string. By setting in this way, even a character string having a boundary with a small number of branches can be extracted.
Note that this score setting method is merely an example, and other methods may be used. For example, an average of the above four values a to d may be used as a score. In this case, since a character string having a similar boundary is extracted as a word, an accurate word is easily extracted.

  The above four perplexities a to d represent the perplexity on the boundary surface of the character string in the text database 121. If a certain amount of text exists in the text database 121, the value of m With regard to the appearance frequency of the character string of m characters having a small size, in most cases, it is possible to perform calculation to such an extent that sufficient reliability can be obtained. For this reason, regarding the four perplexities a to d, reliable values are obtained in most cases even when the appearance frequency of the target character string is small.

  The word determination process in the next step S209 is a process of the word determination unit 110. The word determination unit 110 receives the score of the analysis target character string from the score setting unit 109 and receives the threshold value [t] from the threshold setting unit 109. The threshold value [t] received from the threshold value setting unit 109 is the threshold value [t] set in the threshold value setting process described above with reference to the flowchart of FIG.

If the score of the character string to be analyzed received from the score setting unit 109 is greater than or equal to the threshold t received from the threshold setting unit 109, the word determination unit 110 recognizes it as a word. That is, it recognizes as an unknown word without dictionary registration. For example,
Threshold t = 10.0
In this case, the score set based on the perplexity calculation performed on the text database 121 of the analysis target character string “lower leg bone” described above is as follows:
Score = 34.95
Then,
Score: 34.95 ≧ 10.0 (threshold)
Since the above formula is established, “lower leg bone” is recognized as a word.

  In step S210, it is determined whether or not there is an unprocessed analysis target character string. If there is an unprocessed analysis target character string, the next analysis target character string is selected, and the processes in steps S205 to S209 are repeated. And determine whether or not to recognize as a word of the character string to be analyzed. This process is executed for all the character strings to be analyzed, and the process ends when there are no more unprocessed character strings.

  The words extracted in this way can be used, for example, by registering them in a dictionary database applied to morphological analysis. In the processing example described above, a word as a technical term in the medical field is extracted. Therefore, for example, it can be set as a registered word in a word dictionary in the medical field or a specialized dictionary for morphological analysis. By performing morphological analysis using a dictionary in which words extracted in this way are registered, it is possible to extract words that cannot be extracted by a conventional dictionary. Moreover, the word extracted by the above-mentioned process can be used as a word used for data search, for example. By such processing, even a user who does not have knowledge in the medical field can perform effective data retrieval using words that are medical terms obtained by the above-described processing. The dictionary obtained in this way may be a database that stores only the character strings extracted by this method in a storage device and is managed separately from other dictionaries, or may be stored in another dictionary database as described above. May be added. It is also possible to store information indicating that the character string has been extracted by this method, and further store the character string score in association with the character string. When a column is used for language processing, use of a character string, for example, acceptance / rejection of the character string, change of a display mode, and the like can be controlled based on such information.

[Evaluation of word extraction processing according to this embodiment]
Since the evaluation process was performed on the result of the word extraction process according to the above-described embodiment of the present invention, the result will be described below.
(A) The result of performing the processing according to the above-described embodiment;
(B) Non-Patent Document 1 described above (Shimohata, S. Sugio, T. Nagata, J. Retrieving collaborations by co-ocurrenses and word order constants. Proc. Processing, that is, in extracting collocation (collocation), the entropy (information amount) in the corpus of collocations is calculated, and as a result of executing the process of extracting collocations where the entropy of the words on both sides exceeds the set threshold,
These were compared.

The processing target was
(1) “Latest Anatomical Glossary”,
(2) “MEDIS Standard Disease Name Master 2.4.2” (http://www.medis.or.jp/)
It is text data included in these, and regarding the character string whose appearance frequency in about 65,000 actual medical texts in these texts is 5 or less,
(A) A result obtained by performing processing according to the above-described embodiment, extracting the top 200 character strings in the order of scores, and excluding those included in the IPA dictionary,
(B) Result B obtained by extracting the top 200 character strings in the order of the score of the character strings extracted by the process applying the method described in the above prior art (Non-patent Document 1) and excluding those included in the IPA dictionary,
Regarding these results A and B, when a doctor who is an expert confirmed what was established as a word, the evaluation results shown in FIG. 7 were obtained.

  FIG. 7 (1) is a comparison data of the word extraction processing results of the present embodiment based on the text data included in the “latest anatomical terminology” and the conventional method, and FIG. 7 (2) is a “MEDIAS standard”. It is the comparison data of the present Example based on the text data contained in "Sick name master 2.4.2", and the word extraction processing result of each conventional method.

The “correct answer rate” is the proportion of the target character string that is established as a word. RR _SUM is the sum of Reciprocal Ranks (the reciprocal of the correct ranking), and is an evaluation index considering ranking. As is clear from FIG. 7, in the process for the “latest anatomy glossary”, the process according to the present embodiment is compared with the conventional method.
16.8% in correct answer rate,
0.29 for RR _SUM
These performance improvements are seen,
In addition, in the process for the “standard disease name master”, the process according to this example is compared with the conventional method.
0.6% correct answer rate,
0.52 for RR _SUM
These performance improvements were obtained, confirming the effectiveness of the present invention.

[Example 2]
Next, processing of the language analysis system according to the second embodiment of the present invention will be described. FIG. 8 shows a processing flow executed by the language analysis system according to the second embodiment. In the second embodiment, the system configuration described with reference to FIG. 1 is applied as the system configuration. The present embodiment is a processing example in which processing in units of text included in the text database 121 is executed. For example, as described with reference to FIG. 2A, the text database 121 stores various texts in association with identifiers (ID). In this embodiment, the word extraction process is executed for each text unit in which these identifiers are set. By obtaining a word (for example, an unknown word) for each text unit, it is possible to obtain word information corresponding to each text. Furthermore, the ranking process of the score of the extracted word from each text becomes possible.

  The processing sequence of the present embodiment will be described with reference to the flowchart shown in FIG. In the process flow shown in FIG. 8, the processes in steps S302 to S311 are the same as the processes in steps S201 to S210 in the process flow of the first embodiment described above with reference to FIG.

In this embodiment, first, in step S301, the first one text to be analyzed is selected from the text database 211. For example, in the example shown in FIG. 2A, for example, as the analysis target text, the first registered text, that is,
ID1: Superior in bilateral lung fields
Select the text above.

In step S302, the selected text is input to the text input unit 101 of the system.
Next, in step S303, the character string extraction unit 102 divides the input text by delimiters such as line breaks and punctuation marks, and extracts all partial character strings that can be taken from each divided text.
Next, in step S304, the character string filtering unit 103 executes character string filtering processing. As described above in the first embodiment, this process is performed as follows.
(1) Character strings that cannot basically be Japanese words,
(2) words already registered in the dictionary,
(3) a character string having a high appearance frequency in the text stored in the text database 121;
This process is executed to delete these character strings.

  Next, in step S305, the first character string is acquired as the analysis target character string from the analysis target character string set selected in step S304. In step S306, the head / end character string extraction unit 104 determines the head of the target character string. And the last m characters are extracted. m is a preset value of 1 or more.

The next step S307 is a process of the peripheral character string acquisition unit 105, and a set of m character string surrounding character strings appearing around the target character string by the analysis process for the text data stored in the text database 121. To extract.
The next step S308 is processing of the branch number calculation unit (perplexity calculation unit) 106, and constitutes a document including the analysis target character string extracted in the surrounding character string extraction processing of step S307. First and last character strings, and the left and right surrounding character strings of the analysis target character string,
The perplexity in the text data stored in the text database 121 of these character string sets is calculated. In particular,
(1) The left perplexity of the first character string of the analysis target character string (2) The right perplexity of the last character string of the analysis target character string (3) The right perplexity of the peripheral character string to the left of the analysis target character string The average value of
(4) The average value of the left perplexity of the surrounding character string to the right of the analysis target character string,
Each of these values is calculated.

The next step S309 is processing of the score setting unit 108. The score setting unit 108 sets the score of the target character string from the calculated perplexity. For example,
(1) Perplexity a on the left side of the first m characters of the character string to be analyzed,
(2) Perplexity b on the right side of the last m characters of the character string to be analyzed,
(3) The average value c of the right perplexity of the set of surrounding character strings of the left m characters of the character string to be analyzed,
(4) Perplexity average value d on the left side of a set of surrounding character strings of m characters on the right side of the analysis target character string.
The minimum value of these four values a to d is set as the score of the target character string. The processing example described above with reference to FIGS. 5 and 6 is a case where m = 1.

  The word determination process in the next step S310 is a process of the word determination unit 110. The word determination unit 110 receives the score of the analysis target character string from the score setting unit 109 and receives the threshold value [t] from the threshold setting unit 109. The threshold value [t] received from the threshold value setting unit 109 is the threshold value [t] set in the threshold value setting process described above with reference to the flowchart of FIG.

  If the score of the character string to be analyzed received from the score setting unit 109 is greater than or equal to the threshold t received from the threshold setting unit 109, the word determination unit 110 recognizes it as a word. That is, it recognizes as an unknown word without dictionary registration.

  Next, in step S311, the presence / absence of an unprocessed analysis target character string is determined. If there is an unprocessed analysis target character string, the next analysis target character string is selected, and the processing in steps S306 to S310 is performed. Is repeatedly executed, and it is determined whether or not it is recognized as a word of the analysis target character string. This process is executed for all the character strings to be analyzed, and if there is no unprocessed character string, the process proceeds to step S312.

  In step S312, it is determined whether or not unprocessed analysis target text exists in the text database 121. If the unprocessed analysis target text exists in the text database 121, the next analysis target text is selected, and step S302 is performed. ˜S311 is repeatedly executed. This process is executed for all the texts to be analyzed, and when there is no unprocessed text, the process ends.

  In this processing example, word extraction processing is executed for each text unit in which identifiers stored in the text database 121 are set. Thus, by acquiring a word (for example, an unknown word) for each text unit, it is possible to obtain word information corresponding to each text. Furthermore, the ranking process of the score of the extracted word from each text becomes possible.

  Also in this processing example, the extracted words can be set, for example, as registered words in a specialized word dictionary or a specialized dictionary for morphological analysis. By performing morphological analysis using a dictionary in which extracted words are registered in this way, it becomes possible to extract words that could not be extracted by conventional dictionaries, and use extracted words as words used for data retrieval can do. By such processing, even a user who does not have knowledge in a specialized field can perform effective data retrieval using words that are technical terms obtained by the above-described processing.

[Example 3]
Next, Embodiment 3 of the language analysis system of the present invention will be described with reference to FIG. In the first and second embodiments described above, for example, the head / end character string extraction unit 104 in the system configuration of FIG. 1 includes the following data that forms a document including the analysis target character string:
A string of m characters at the beginning and end of the string to be analyzed, and
A string of m characters around the left and right of the string to be analyzed,
The perplexity in the text data stored in the text database 121 is calculated for these m character string sets.

That is, in the first and second embodiments described above,
(1) Left-side perplexity of the first m characters of the character string to be analyzed (2) Right-side perplexity of the last m characters of the character string to be analyzed (3) Left of the character string to be analyzed The average value of the right perplexity of the surrounding m character string,
(4) The average value of the left perplexity of the character string of the surrounding m characters to the right of the analysis target character string,
These values were calculated.

  In the third embodiment, the head / end character string extraction unit 104 shown in FIG. 1 is replaced with a partial character string extraction unit 301 as shown in FIG. The partial character string extraction unit 301 extracts not only the first and last character strings of the analysis target character string but also all the m character partial character strings constituting the analysis target character string, and extracts the extracted m character partial character strings. The perplexity corresponding to is calculated.

In the first and second embodiments, when m = 1, for example, the head / end character string extraction unit 104 extracts the first and last m = 1 characters of the target character string, and the analysis target character string is “lower leg”. If it was “bone”, the first “bottom” and the last “bone” were extracted.
On the other hand, in this embodiment, when m = 1, the partial character string extraction unit 301 extracts all partial character strings made up of m = 1 character of the target character string. If the character string to be analyzed is “lower leg bone”, the first “lower”, the last “bone”, and the center “thigh” are extracted.

The branch number calculation unit (perplexity calculation unit) 106 calculates perplexity corresponding to the character string extracted by the partial character string extraction unit 301. For example, when m = 1 and the character string to be analyzed is “crus bone”, the perplexity at the boundary of the character string to be analyzed described in the first embodiment with reference to FIGS. 5 and 6 is calculated. As shown in FIG.
(A) Perplexity on the right side of “below”,
(B) The right and left perplexity of the “thigh”,
(C) Perplexity on the left side of “bone”,
These perplexities are also calculated.

Furthermore, in the internal score setting unit 302,
(1) The average perplexity on the right side of “bottom” and “thigh”,
(2) The average perplexity on the left of “thigh” and “bone”
Each of these average values is calculated, and the maximum value of these average values is set as the internal score.

In the score setting unit 108, as in the first embodiment, the perplexity at the boundary of the analysis target character string described with reference to FIG. 5 and FIG.
(1) Perplexity a on the left side of the first m characters of the character string to be analyzed,
(2) Perplexity b on the right side of the last m characters of the character string to be analyzed,
(3) The average value c of the right perplexity of the set of surrounding character strings of the left m characters of the character string to be analyzed,
(4) Perplexity average value d on the left side of a set of surrounding character strings of m characters on the right side of the analysis target character string.
The minimum value of these four values a to d is set as the boundary score [S] of the target character string. The processing example described above with reference to FIGS. 5 and 6 is a case where m = 1.

  The word determination unit 110 compares the boundary score [S] set by the score setting unit 108 with the internal score [SIN] set by the internal score setting unit 302, and the boundary score [S] is the internal score [SIN]. If it is larger, it is recognized as a word. That is, it recognizes as an unknown word without dictionary registration.

  According to this configuration, the partial character string of the analysis target character string is extracted, the internal score [SIN] is calculated, and compared with the boundary surface score [S], the unknown that more strongly reflects the characteristics of the target character string It is possible to extract words.

  Also in this processing example, the extracted words can be set, for example, as registered words in a specialized word dictionary or a specialized dictionary for morphological analysis. By performing morphological analysis using a dictionary in which extracted words are registered in this way, it becomes possible to extract words that could not be extracted by conventional dictionaries, and use extracted words as words used for data retrieval can do. By such processing, even a user who does not have knowledge in a specialized field can perform effective data retrieval using words that are technical terms obtained by the above-described processing.

  In the above-described embodiment, as the processing of the partial character string extraction unit 301, m = 1 character partial character strings constituting the analysis target character string are extracted as m = 1, and the extracted m = 1 character character is extracted. Although the perplexity corresponding to the partial character string is calculated, m can be set to various values of 1 or more. For example, the value of m can be changed according to the type of character (kanji, hiragana, katakana, alphanumeric, etc.). Thus, by changing the value of m depending on the character type, it is possible to cope with the difference in information amount depending on the character type. For example, by setting m = 1 for kanji and m = 2 for hiragana / katakana / alphanumeric characters, the amount of information of kanji can be handled as twice that of hiragana / katakana / alphanumeric characters. Note that the number of characters m to be extracted at the beginning, end, left side, and right side of the analysis target character string may be different. However, when the scores are set by combining these evaluation values, since the evaluation values under different conditions are combined unless the same number of characters is used, it is desirable that they be the same.

  Further, in the above-described embodiment, the example in which the perplexity (average branch number) represented by Equation 2 is used as the evaluation value indicating the branch state of the appearance character string at the boundary of the analysis target character string has been described. As the evaluation value of diversity, for example, entropy (Equation 1), which is used as an evaluation index for language analysis, can also be used. By calculating using a predetermined conversion formula, it can be used as an evaluation value. Alternatively, the evaluation value of diversity based on the number of appearances of the character string before and after the target character string can be obtained by applying a predetermined arithmetic expression, and the calculation result can be used as the evaluation value. However, since perplexity is commonly used in the field of language processing as a value indicating diversity, the perplexity is set more appropriately in condition setting and adjustment when extracting words than when entropy is used as an index. Easy to operate as a system.

  Finally, a hardware configuration example of the information processing apparatus that configures the language analysis system that executes the above-described processing will be described with reference to FIG. A CPU (Central Processing Unit) 501 performs processing corresponding to an OS (Operating System), perplexity calculation processing, score calculation processing, word extraction processing described in the above-described embodiment, and generation of a dictionary in which extracted words are registered. Processing, morphological analysis processing using the generated dictionary, etc. are executed. These processes are executed according to a computer program stored in a data storage unit such as a ROM or a hard disk of each information processing apparatus.

  A ROM (Read Only Memory) 502 stores programs used by the CPU 501, calculation parameters, and the like. A RAM (Random Access Memory) 503 stores programs used in the execution of the CPU 501, parameters that change as appropriate during the execution, and the like. These are connected to each other by a host bus 504 including a CPU bus.

  The host bus 504 is connected to an external bus 506 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 505.

  A keyboard 508 and a pointing device 509 are input devices operated by the user. The display 510 includes a liquid crystal display device, a CRT (Cathode Ray Tube), or the like, and displays various types of information as text and images.

  An HDD (Hard Disk Drive) 511 includes a hard disk, drives the hard disk, and records or reproduces a program executed by the CPU 501 and information. The hard disk includes, for example, words extracted by the word extraction described in the above embodiments, a dictionary in which words are registered, data processing programs applied in the above embodiments, parameters such as threshold values and scores, Stores the program.

  The drive 512 reads data or a program recorded on a removable recording medium 521 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and the data or program is read out from the interface 507 and the external bus 506. , And supplied to the RAM 503 connected via the bridge 505 and the host bus 504.

  The connection port 514 is a port for connecting the external connection device 522 and has a connection unit such as USB or IEEE1394. The connection port 514 is connected to the CPU 501 and the like via the interface 507, the external bus 506, the bridge 505, the host bus 504, and the like. The communication unit 515 is connected to a network and executes communication with various databases and other information processing apparatuses.

  Note that the hardware configuration example of the information processing apparatus as the language analysis system shown in FIG. 11 is an example of an apparatus configured by applying a PC, and the language analysis system of the present invention is not limited to the configuration shown in FIG. Any configuration capable of executing the processing described in the above-described embodiments may be used.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

It is a figure which shows the structural example of the language analysis system which concerns on one Example of this invention. It is a figure explaining the example of a structure of the text database applied in the language analysis system which concerns on one Example of this invention, and a word database. It is a figure which shows the flowchart explaining the sequence of the threshold value setting process performed in the language analysis system which concerns on one Example of this invention. It is a figure which shows the flowchart explaining the sequence of the word extraction process performed in the language analysis system which concerns on one Example of this invention. It is a figure explaining the example of a perplexity calculation process performed in the language analysis system which concerns on one Example of this invention. It is a figure explaining the example of a perplexity calculation process performed in the language analysis system which concerns on one Example of this invention. It is a figure explaining the evaluation data of the word extraction result performed in the language analysis system which concerns on one Example of this invention. It is a figure which shows the flowchart explaining the sequence of the threshold value setting process performed in the language analysis system which concerns on one Example of this invention. It is a figure which shows the structural example of the language analysis system which concerns on one Example of this invention. It is a figure explaining the calculation process example of the perplexity and internal score performed in the language analysis system which concerns on one Example of this invention. It is a figure explaining the hardware structural example of the language analysis system which concerns on one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Language analysis system 101 Text input part 102 Character string extraction part 103 Character string filter part 104 Leading / ending character string extraction part 105 Peripheral character string extraction part 106 Branch number calculation part (perplexity calculation part)
DESCRIPTION OF SYMBOLS 107 Word division part 108 Score setting part 109 Threshold setting part 110 Word determination part 121 Text database 122 Word database 301 Partial character string extraction part 302 Internal score setting part 501 CPU (Central Processing Unit)
502 ROM (Read-Only-Memory)
503 RAM (Random Access Memory)
504 Host bus 505 Bridge 506 External bus 507 Interface 508 Keyboard 509 Pointing device 510 Display 511 HDD (Hard Disk Drive)
512 drive 514 connection port 515 communication unit 521 removable recording medium 522 external connection device

Claims (11)

A character string extraction unit that extracts a set of character strings having a predetermined number of characters or less from text data;
From the text data in a text database, a character string where the character string extraction unit and extracted as an analysis target character string, the parsed character string word based on perplexity occurred string at the boundary of the analyzed text A score setting unit that calculates and sets a score indicating the probability of being ,
A word determination unit that determines whether or not the analysis target character string is a word based on the score set by the score setting unit;
A language analysis system comprising: The score setting unit
The perplexity of the appearance character string in the outward direction of the analysis target character string in each of one or more characters at both ends of the analysis target character string;
The perplexity of the appearance character string in the direction of the analysis target character string in each of one or more characters at both ends in contact with the analysis target character string;
Is calculated,
The language analysis system according to claim 1, wherein a score is calculated based on the two types of calculated perplexities . The score setting unit
The language analysis system according to claim 2, wherein a score is calculated based on a minimum value of the two types of perplexities . The language analysis system further includes:
A word dividing unit that divides words registered in the word database into units of m characters (however, a predetermined number of m ≧ 1);
The perplexity of the occurred string at the boundary of the parsed character string corresponding to the character string of the divided m characters, of the previous SL word dividing section and calculated, respectively and the calculated from perplexity, the m character units A threshold value setting unit that calculates an average value for a plurality of perplexities excluding the perplexity of the appearance character string in the external direction at the end of the character string, and calculates the average value as a threshold value,
The word determination unit
Comparing the score set by the score setting unit with the threshold set by the threshold setting unit, and determining that the analysis target character string is a word according to the comparison result The language analysis system according to claim 1, wherein: The language analysis system further includes:
A character string filter unit that executes a filtering process to delete a character string that does not hold as a word from the set of character strings extracted by the character string extraction unit;
The score setting unit
The language analysis system according to claim 1, wherein the score is calculated and set using the character string after filtering in the character string filter unit as an analysis target character string. The language analysis system further includes:
From the extracted character string extracted by the character string extracting unit, a leading and trailing character string extracting unit that extracts the leading and trailing character strings of the extracted character string,
The score setting unit
2. The configuration according to claim 1, wherein the perplexity of an appearance character string at a boundary of the analysis target character string is calculated based on the head / end character string extracted by the head / end character string extraction unit. The language analysis system described. The language analysis system further includes:
A partial character string extraction unit that extracts a partial character string of the extracted character string from the extracted character string extracted by the character string extraction unit ;
Based on the partial character string extracted by the partial character string extraction unit , the calculation of the perplexity of the appearance character string within the analysis target character string is performed , and an internal score is set based on the perplexity A score setting unit, the word determination unit,
It is configured to execute a process of determining whether or not the analysis target character string is a word based on the score set by the score setting unit and the internal score set by the internal score setting unit. The language analysis system according to claim 1. The language analysis system includes:
The language analysis system according to claim 1, wherein the word extraction process is executed in units of text stored in a text database. The language analysis system according to any one of claims 1 to 8, wherein the analysis target character string is registered in a dictionary database for morphological analysis based on a determination result of the word determination unit. A language analysis method for executing language analysis processing in a language analysis system,
A character string extraction step in which the character string extraction unit extracts a set of character strings having a predetermined number of characters or less from the text data;
Score setting unit, the analysis processing for the text data in the text database, a character string extracted in the character string extraction step as an analysis target character string, the occurred string at the boundary of the parsed string Papurekishi A score setting step for calculating and setting a score indicating the probability that the analysis target character string is a word based on the tee ;
A word determination step in which a word determination unit determines whether or not the analysis target character string is a word based on the score set in the score setting step;
A language analysis method characterized by comprising: A computer program that executes language analysis processing in a language analysis system,
A character string extraction step for causing the character string extraction unit to extract a set of character strings having a predetermined number of characters or less from the text data;
The score setting unit, the analysis processing for the text data in the text database, a character string extracted in the character string extraction step as an analysis target character string, the occurred string at the boundary of the parsed string Papurekishi A score setting step for calculating and setting a score indicating the probability that the analysis target character string is a word based on the tee ;
A word determination step for causing a word determination unit to determine whether or not the analysis target character string is a word based on the score set in the score setting step;
A computer program for executing

Images