JP2002269083A - Morpheme analysis system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform morpheme analysis without removing a tag, i.e., document description information in a document and to easily know the position where the tag, i.e., document description information has been located from the morpheme analysis result. SOLUTION: A sentence division part 11 divides an inputted document into sentences and phrases and when a dictionary referring range determination part 12 extracts a word for referring to a morpheme analysis dictionary 30 by a dictionary referring part 13 and document description information is included in the inputted document, a language unit level decision part 20 decides the language unit level of the document description information according to a language unit level registered in a language unit table 23 and changes the operation for morpheme analysis regarding the inputted document according to the language unit level of the decided document description information. Then an optimum path determination part 14 determines the best morpheme analysis result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a morphological analysis system applied to language processing, syntax analysis, and the like.
The present invention relates to a morphological analysis system applied to a language analysis system that handles tagged documents such as TML, XML, and SGML.

[0002]

2. Description of the Related Art Japanese Unexamined Patent Publication No. Hei 9-16594 discloses a document which removes tags in a document from a tagged document such as HTML, XML, and SGML, performs morphological analysis, and outputs the result and the tags together. There is a morphological analysis technique disclosed in "Morphological analyzer". In this morphological analysis technology, a sentence break is recognized by detecting a tag included in a document, then the tag is removed from the document, a morphological analysis is performed normally, and finally, the morphological analysis result and the tag Is output by merging the information about the documents, and morphological analysis can be performed even on a document with a tag.

[0003]

However, in the morphological analysis technology disclosed in the prior art, since the tag is once removed, it is difficult to obtain the positional relationship between the morphological analysis result and the tag later. There is.

[0004] For example, consider the following input document. "On Saturday <hinshi name =" proper noun "> Hanami <
/ Hinshi> line <br>let's go <p>

Here, <hinshi>, </ hinshi>, <b
r> and <p> indicate tags, respectively, and the tags <hinshi> and </ hinshi> are tags specifying the part of speech of a word interposed between the two tags. Indicates that "hanami" is a proper noun (in this example, a proper noun meaning the name of a bar). The <br> tag indicates that a physical line feed is specified, and a line feed is made at the insertion position of <br>. The tag <p> means that the paragraph is divided, and the paragraph is vertically separated at the insertion position of <p>.

Therefore, the input document described above is to go to “bar“ Hanami ”(here,“ hanami ”is a proper noun indicating the store name of the bar) on Saturday. This is an example of a text consisting of two paragraphs, a text inviting "" and a text confirming "I see.

From such an input document, Japanese Patent Laid-Open Publication No.
As in the case of the morphological analysis in Japanese Patent No. 594, if the tag is first removed, the text of "Let's go to Hanami on Saturday, I wonder if it's good" is obtained. Originally, "Saturday (noun) / ni (adjunct particle) / Hanami (proper noun) / ni (adjunct particle) / go (verb conjunctive form) / macho (auxiliary verb) / u (final particle) / ne (adjective) / , (Reading point) / good (adjective) / or (final particle) / na (final particle), but there is no tag, so "Saturday (noun) / ni (adjunct particle) / flower (Noun) / look (verb conjunctive form) / ni (adjunct verb) / go (verb conjunctive form) / macho (auxiliary verb) / u (final verb) / ne (final particle) /
Or (final particle) / na (final particle) ".

That is, although “hanami” should be analyzed as a proper noun, it is divided into verb conjunctive forms of “hana”, common nouns and “mi”. Despite the fact that it should be analyzed as a verb at the beginning of the text, it is interpreted as a final verb added to the end of the first text, and the purpose of "Let's go see the flowers on Saturday. The result is not analyzed correctly. After this, it becomes difficult to associate the tag with the result of the morphological analysis.

The present invention has been made in view of such a problem, and performs morphological analysis without removing tags in a document, ie, document description information, and finds tags, ie, document description information, from the analysis result. The purpose is to make the position easily understandable. It also aims to improve the analysis rate of morphological analysis using tags.

[0010]

According to a first aspect of the present invention, there is provided a morphological analysis dictionary having at least a word in the morphological analysis dictionary that matches a word included in an input document. In the morphological analysis system that performs morphological analysis of the input document and outputs an optimal morphological analysis result, when the input document includes document description information, the document description information A morphological analysis system that identifies a linguistic unit level and changes a morphological analysis operation on the input document in accordance with the identified linguistic unit level.

According to a second aspect of the present invention, in the morphological analysis system according to the first aspect, when the linguistic unit level of the identified document description information is a low level of characters or less, When the dictionary is indexed, the morphological analysis system extracts the words to be indexed by skipping the document description information.

According to a third aspect of the present invention, in the morphological analysis system according to the first aspect, when the linguistic unit level of the identified document description information is a level higher than a morpheme, the morphological analysis dictionary Is indexed, the morphological analysis system does not index the morphological analysis dictionary based on a character string sandwiching the document description information.

According to a fourth aspect of the present invention, in the morphological analysis system according to the first aspect, when the linguistic unit level of the identified document description information is a level equal to or higher than a sentence, The morphological analysis system performs a morphological analysis by dividing each document divided by.

[0014] The invention according to claim 5 provides the invention according to claims 1 to 4.
The morphological analysis system according to any one of the above, wherein the document description information is a tag of SGML and / or XML and / or HTML.

[0015] The invention described in claim 6 is the invention according to claims 1 to 4.
In the morphological analysis system according to any one of the above, the document description information is a morphological analysis system that is a layout control character including a line feed.

The invention according to claim 7 is the invention according to claims 1 to 4
Is characterized in that the document description information is a morphological analysis system in which delimiters such as punctuation marks and punctuation marks are used.

[0018] The invention according to claim 8 has at least a morphological analysis dictionary, and a dictionary is searched for words in the morphological analysis dictionary that match words contained in the input document. A morphological analysis of the document and outputting an optimal morphological analysis result, when the input document includes document description information, identifies a language unit level of the document description information. And a morphological analysis method that changes a morphological analysis operation for the input document according to the identified linguistic unit level.

According to a ninth aspect of the present invention, there is provided a morphological analysis dictionary having at least a morphological analysis dictionary. A morphological analysis of the document and outputting an optimal morphological analysis result, when the input document includes document description information, identifies a language unit level of the document description information. And a program recording medium readable as a computer program so that the computer can execute a morphological analysis method for changing an operation of morphological analysis on the input document according to the identified linguistic unit level. It is characterized by doing.

[0019]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows a functional block diagram of a morphological analysis system according to the present invention as an embodiment.
In FIG. 1, the morphological analysis system includes a morphological analysis unit 10, a linguistic unit level determination unit 20, and a morphological analysis dictionary 30. The morphological analysis unit 10 includes the sentence division unit 1
1, a dictionary lookup range determination unit 12, a dictionary lookup unit 13,
And an optimal path determination unit 14. The sentence division unit 11
It has a function of dividing an input document into sentences and phrases based on document description information (tags, layout control characters and delimiters) included in the document.

The dictionary lookup range determining unit 12 includes a sentence dividing unit 11
Has a function of extracting a word as a unit for indexing the morphological analysis dictionary 30 from among the phrases (texts for each section included in the sentence) divided by. In extracting words,
When the document description information is extracted, it is taken over by the linguistic unit level determination unit 20 and is not used as a word to index the morphological analysis dictionary 30. The dictionary lookup unit 13 is a dictionary lookup range determination unit 12
Morphological analysis dictionary 30 based on the words extracted by
And has a function of obtaining the notation, part of speech, cost, and the like for each word. The optimum path determination unit 14 is configured to search the dictionary
3 has a function of outputting an optimal morphological analysis result based on a dictionary index result for each word constituting a document.

On the other hand, the linguistic unit level determining unit 20 includes a linguistic unit level comparing unit 21 and a document description information table 22.
And a language unit level table 23. The language unit level determining unit 20 has a function of determining the language level of the document description information included in the document, and the sentence dividing unit 1
1 or the document description information specified by the dictionary lookup range determination unit 12 is determined to be registered in the document description information table 22. If registered, the document description information is registered in the language unit level table 23. Based on the linguistic unit table information, the linguistic unit level comparing unit 21 has, for example, a function of determining whether or not the linguistic unit level indicates a unit of a sentence or more.

The document input to the morphological analysis system is input to the morphological analysis unit 10 and, from the beginning of the document,
Although the morphological analysis is sequentially performed, if the word divided and extracted in the sentence dividing unit 11 is the document description information corresponding to the above-described tag, the language unit level of the document description information is changed to the language unit. The level is determined by the level determination unit 20. Thus, the processing method of the morphological analysis is changed according to the identification result.

Next, with respect to the embodiments according to the first to ninth aspects of the invention, [Problems to be Solved by the Invention]
In the section below, taking the following input document as an example,
This will be described below.

[0024] That is, "bar" Hanami "(name of bar)
Let's go on Saturday. "And the text"
Is it okay. "Indicates a text consisting of two paragraphs of text identified as" Saturday <hinshi name
= "Proper noun"> Hanami </ hinshi> line <br>Let's go <p>
Morphological analysis is performed by the following steps. Note that control information for controlling the structure of a document such as a tag is
In the present invention, the term is referred to as “document description information”, and the term of the document description information is used in the following description.

First, in the sentence division unit 11, the input document is examined sequentially from the beginning, and when the document description information is searched, the language unit level determination unit 20 indicates that the language unit level is higher than the sentence. If it is determined that the document description information has been searched, the document is divided into sentences at the position of the document description information (step S1). That is, here, <p> indicating the above paragraph is searched, divided into two texts before and after, and as shown in Table 1, two sentences “1” and “2” are added to each divided text. A number is assigned.
The "position" column shown in Table 1 indicates the position of the first character of the text indicated by each sentence number by the number of characters from the head of the input document. Although not shown in the input document, document description information indicating the end of the input document is also added to the end of the sentence.

[0026]

[Table 1]

Subsequently, after performing a morphological analysis of the sentence for each text corresponding to each sentence number (step S2), the "processing accompanying the analysis" relating to the document description information included in each text. (Step S
3) Finally, a morphological analysis result is output (step S4).

Next, the morphological analysis of the sentence shown in step S2 will be described. Further, the sentence division unit 11 examines the sentence which is the text of the sentence number “1” sequentially from the beginning, and when the document description information is searched, the language unit level determination unit 20 determines that the language unit level is the word As described above, when it is determined that the document description information less than the sentence has been searched, the sentence is further divided into the segments of each section at the position of the document description information by the document description information (step S2.
1). For example, for the sentence with the sentence number “1”, the document description information <hinshi name = “proper noun”> and </ hinshi>, which are less than the sentence and delimit more than words, are searched. As shown in Table 2, “1” is assigned to each of the divided sections.
Three section numbers “2” and “3” are assigned. The "position" column shown in Table 2 indicates the position of the first character of the text indicated by each section number by the number of characters from the beginning of the input document.

[0029]

[Table 2]

Next, in step S2.1, the following is executed for each of the segments in each of the divided sections. The current position (p) is set to the first character position of the phrase indicating the section (step S2.2). The dictionary lookup range determination unit 12 extracts a word starting from the set character position (step S2.3), delivers the word to the dictionary lookup unit 13, causes the morphological analysis dictionary 30 to index the word, and
Acquire information on part of speech and cost (step S2.
4). Here, when document description information is detected as the word, the language unit level determination unit 20 determines the language unit level of the document description information, and when the language unit level is the word level, Then, a "process accompanying the analysis" to be described later is performed, and a dictionary lookup for the document description information is not performed.

On the other hand, if the document description information is not at the word level but at the character level or lower, the word is skipped, the word is extracted and passed to the dictionary lookup unit 13, and the morphological analysis dictionary 30 is read. By indexing, information on the notation, part of speech, and cost of the word is acquired. The current position (p) is updated to the next character position (step S2.5). If the phrase indicating the section has not been finished yet (step S2.).
6) Returning to step S2.3, the extraction of the next word is repeated.

For example, if the extraction of words (step S2.3) and dictionary lookup (step S2.4) are repeatedly performed for the phrase having the section number "2" in Table 2, as shown in Table 3, , "Flower" and "mi </ hinshi>", and each word is lexicographically searched to obtain information on notation, part of speech, and cost. Here, the cost indicates the morphological analysis cost, and is registered in the dictionary for each word. Note that the document description information indicating the end of the section is decomposed by attaching to the last word, and the cost of the last word of the section is calculated by adding the cost of the document description information to the cost obtained by dictionary lookup. Is calculated. In Table 3, the “position” column indicates the head position of the word as the number of characters from the head of the document, and the “length” column indicates the number of characters of the word.

[0033]

[Table 3]

When the extraction of words (step S2.3) and the dictionary lookup (step S2.4) are repeatedly performed for the phrase having the section number "3" in Table 2, as shown in Table 4, , "Ni", "row <br>ki","ki",
Decomposed into five words, "Mashou" and "U <p>"
Each word is looked up in a dictionary, and information on the notation, part of speech, and cost is obtained.

[0035]

[Table 4]

However, in practice, each word in the phrase can be obtained from the position and length in the input text, and therefore it is not necessary to store and store the words in the format shown in Tables 3 and 4.

Next, in the above-described word extraction in step S2.3, when the document description information at the word level is detected as the language unit level of the document description information, the "analysis" accompanying the document description information to be performed is performed. (Step S3) will be described.

For example, the sentence with the sentence number "1" shown in the above example is divided into three clauses, that is, sections in step S2.1. As shown in the figure, “<hinshi name =“ proper noun ”>” is accompanied as document description information indicating the end of a phrase, and “proper noun” is loaded on the stack. Regarding the clause of “2”, created by repeating steps S2.3 and S2.4,
The dictionary lookup result using the two words “flower” and “mi hinshi” is changed to the part of speech indicated by the document description information loaded on the stack with the section number “1” (step S3). Therefore, in the dictionary lookup result of the section number “2”, as shown in Table 5, the two words are corrected to the proper noun “hanami </ hinshi>” synthesized into one word. As shown in Table 5, the part-of-speech is set to “proper noun” as specified in the document description information, and the notation becomes “hanami” as it is. Further, as for the cost, "10" is always set for simplicity.

[0039]

[Table 5]

Finally, in step S4, an optimal combination of morphemes is obtained based on the cost of each word of the result obtained by dictionary lookup, and a morphological analysis result is output. It should be noted that a method for obtaining an optimum morpheme can be performed by a general morphological analysis method, and thus will not be described here. As a result of the morphological analysis described above, a morphological analysis result as shown in Table 6 is output.

[0041]

[Table 6]

As described above, morphological analysis can be performed even if document description information (tag) is present in a document, and where the document description information (tag) is located as a result of morphological analysis. Easy to judge. Furthermore, <hins
By providing a tag such as hi>, the analysis rate is improved. In the present embodiment, XML, SGML, and HTML tags are shown as examples of the document description information. However, a specific 1-byte layout control character (for example, a line feed code) or a specific delimiter ( For example, the same processing can be performed in the case of a period, a reading mark, etc.).

Table 7 shows an example of the document description information table 22 in FIG. 1. When examining the language unit level of the document description information at an arbitrary position in the document, the document description information table 22 shown in Table 7 is used. By indexing table 22,
Language unit level can be determined. It is assumed that the document description information registered in the “document description information” column of the document description information table 22 is registered using a normalized regular expression. Here, as the document description information, S
GML and / or XML and / or HTML tags may be registered, or layout control characters such as line feeds, or delimiters such as punctuation marks and reading marks may be registered.

[0044]

[Table 7]

Table 8 shows an example of the language unit level table 23 in FIG. 1. In step S1, the language unit level of the document description information is
The language unit level table 23 is indexed when it is determined whether or not the document description information indicates “sentence” or more. Here, as shown in Table 8, when the linguistic unit level of the document description information is a low level of characters or less, when indexing the morphological analysis dictionary, the document description information is skipped and the words to be indexed are searched. When the linguistic unit level is equal to or higher than the morpheme, when the morphological analysis dictionary is indexed, the morphological analysis dictionary is not indexed based on a character string sandwiching the document description information,
Further, when the language unit level is equal to or higher than a sentence, the morphological analysis is performed by dividing each document divided by the document description information.

[0046]

[Table 8]

[0047]

As described above, according to the morphological analysis system according to the present invention, the following effects can be obtained. The morphological analysis of the document can be performed while the document description information (that is, tags and control characters) in the document is stored as it is. The positional relationship between the morphological analysis result and the document description information (that is, tags and control characters) can be easily known. The analysis rate of morphological analysis can be improved by using document description information (that is, tags and control characters).

[Brief description of the drawings]

FIG. 1 is a diagram showing an embodiment of a functional block diagram of a morphological analysis system according to the present invention.

[Explanation of symbols]

Reference Signs List 10: morphological analysis unit, 11: sentence division unit, 12: dictionary lookup range determination unit, 13: dictionary lookup unit, 14: optimal path determination unit, 20: language unit level determination unit, 21: language unit level comparison unit, 22 ... a document description information table, 23 ... a language unit level table, 30 ... a morphological analysis dictionary.

Claims (9)

[Claims] 1. A morphological analysis of an input document is performed by dictionary-searching words in the morphological analysis dictionary having at least a morphological analysis dictionary and matching words included in the input document. In a morphological analysis system that outputs an optimal morphological analysis result, when the input document includes document description information, a language unit level of the document description information is identified, and the identified language is identified. A morphological analysis system that changes an operation of morphological analysis on the input document according to a unit level. 2. The morphological analysis system according to claim 1, wherein when the linguistic unit level of the identified document description information is a low level equal to or lower than a character, the morphological analysis dictionary is indexed. A morphological analysis system for skipping document description information and extracting words to be indexed. 3. The morphological analysis system according to claim 1, wherein, when the linguistic unit level of the identified document description information is a level equal to or higher than a morpheme, the document is indexed in the morphological analysis dictionary. A morphological analysis system wherein the morphological analysis dictionary is not indexed based on a character string sandwiching description information. 4. The morphological analysis system according to claim 1, wherein, when the linguistic unit level of the identified document description information is a level equal to or higher than a sentence, each of the documents separated by the document description information. A morphological analysis system that performs division and performs morphological analysis. 5. The morphological analysis system according to claim 1, wherein the document description information is SGM.
A morphological analysis system characterized by being an L and / or XML and / or HTML tag. 6. The morphological analysis system according to claim 1, wherein the document description information is a layout control character including a line feed. 7. The morphological analysis system according to claim 1, wherein the document description information is a delimiter such as a period or a punctuation mark. 8. A morphological analysis of the input document by at least having a morphological analysis dictionary and dictionary-searching words in the morphological analysis dictionary that match words included in the input document. In the morphological analysis method for outputting an optimal morphological analysis result, when the input document includes document description information, a language unit level of the document description information is identified, and the identified language is identified. A morphological analysis method, wherein an operation of morphological analysis relating to the input document is changed according to a unit level. 9. A morphological analysis of the input document by having at least a morphological analysis dictionary and dictionary-searching words in the morphological analysis dictionary that match words included in the input document. In the morphological analysis method for outputting an optimal morphological analysis result, when the input document includes document description information, a language unit level of the document description information is identified, and the identified language is identified. A program recording medium, which is readable as a computer program so that the computer can execute a morphological analysis method for changing an operation of morphological analysis on the input document according to a unit level.