JP5373998B1 - Dictionary generating apparatus, method, and program - Google Patents

Abstract

A dictionary generation device executes a word generation in which a word division model is incorporated in a set of collected text and a model generation unit that generates a word division model using a corpus and a word group prepared in advance. An analysis unit for adding boundary information to each text; a selection unit for selecting a word to be registered in the dictionary from the text to which boundary information has been added by the analysis unit; and a registration unit for registering the word selected by the selection unit in the dictionary With. Each text included in the corpus is given boundary information indicating a word boundary.

Description

  One embodiment of the present invention relates to an apparatus, a method, a program, and a computer-readable recording medium for generating a word dictionary.

  Conventionally, a technique (word division) for obtaining a plurality of words by dividing a sentence using a word dictionary is known. In relation to this, Japanese Patent Application Laid-Open No. 2004-228561 searches for a word to be matched with a partial character string of an input text from a word dictionary and generates it as a word candidate. Is selected as an unknown word candidate, and the unknown word model is used to estimate the word appearance probability of each unknown part of speech using the unknown word model, and the word sequence that maximizes the joint probability is determined using dynamic programming. The required technology is described.

JP 2001-051996 A

  In order to divide the text correctly, it is desirable to prepare a large number of words in the dictionary in order to enhance lexical knowledge. However, it is not easy to build a large dictionary manually. Therefore, it is required to easily construct a large-scale word dictionary.

A dictionary generation apparatus according to an aspect of the present invention is a model generation unit that generates a word division model using a corpus and a word group prepared in advance, and each text included in the corpus indicates a word boundary. Boundary information is provided, and the boundary information includes first information indicating that no boundary exists at the inter-character position, second information indicating that a boundary exists at the inter-character position, and a boundary at the inter-character position. The model generation unit including the third information indicating the existence of the probability, and the word division in which the word division model is embedded is performed on the collected text set, and each text is bounded. An analysis unit for providing information, a selection unit for selecting a word to be registered in the dictionary from the text to which boundary information is provided by the analysis unit, and a registration unit for registering the word selected by the selection unit in the dictionary.

A dictionary generation method according to an aspect of the present invention is a dictionary generation method executed by a dictionary generation device, and includes a model generation step of generating a word division model using a corpus and a word group prepared in advance. Each text included in the corpus is given boundary information indicating a word boundary, and the boundary information includes first information indicating that there is no boundary at the inter-character position and a boundary at the inter-character position. The word generation model includes the model generation step including the second information indicating the boundary and the third information indicating that the boundary is probabilistically present at the position between the characters. An analysis step for executing built-in word segmentation to give boundary information to each text, and a selection step for selecting a word to be registered in the dictionary from the text to which the boundary information was given in the analysis step. Including a flop, and a registration step of registering a word selected in the selection step in the dictionary.

A dictionary generation program according to an aspect of the present invention is a model generation unit that generates a word division model using a corpus and a word group prepared in advance, and each text included in the corpus indicates a word boundary. Boundary information is provided, and the boundary information includes first information indicating that no boundary exists at the inter-character position, second information indicating that a boundary exists at the inter-character position, and a boundary at the inter-character position. The model generation unit including the third information indicating the existence of the probability, and the word division in which the word division model is embedded is performed on the collected text set, and each text is bounded. An analysis unit for providing information, a selection unit for selecting a word to be registered in the dictionary from the text to which boundary information has been added by the analysis unit, and a registration unit for registering the word selected by the selection unit in the dictionary are implemented on a computer. Make.

A computer-readable recording medium according to an aspect of the present invention is a model generation unit that generates a word division model using a corpus and a word group prepared in advance, and each text included in the corpus includes a word Boundary information indicating a boundary is given , and the boundary information includes first information indicating that no boundary exists at the inter-character position, second information indicating that a boundary exists at the inter-character position, and the inter-character position. Each of the model generation unit including the third information indicating that the boundary is probabilistically present, and executing word division incorporating a word division model on the collected text set, An analysis unit for adding boundary information to the text, a selection unit for selecting a word to be registered in the dictionary from the text to which boundary information has been added by the analysis unit, and a registration unit for registering the word selected by the selection unit in the dictionary Con Storing a dictionary generation program to be executed by Yuta.

According to such a form, a word division model is generated using a corpus to which boundary information is given and a word group, and the word division incorporating the model is applied to the text set. Then, a word is selected from the text set to which boundary information is given by this application and registered in the dictionary. In this way, by adding boundary information to a text set by analysis using a corpus with boundary information, a word dictionary extracted from the text set is registered to easily build a large-scale word dictionary. be able to. In addition, the text can be more appropriately divided into a plurality of words by introducing the third information indicating an intermediate concept rather than simply selecting whether or not a boundary exists.

  In the dictionary generation device according to another aspect, the selection unit may select a word to be registered in the dictionary based on the appearance frequency of each word calculated from the boundary information given by the analysis unit. The accuracy of the dictionary can be increased by considering the appearance frequency calculated in this way.

  In the dictionary generation device according to another aspect, the selection unit may select a word whose appearance frequency is equal to or higher than a predetermined threshold. By registering only words that appear more than a certain number of times in the dictionary, the accuracy of the dictionary can be improved.

  In the dictionary generation device according to another aspect, the selection unit extracts words having an appearance frequency equal to or higher than a threshold as registration candidates, selects a predetermined number of words from the registration candidates in order from the word having the highest appearance frequency, and registers them. The unit may add the word selected by the selection unit to the dictionary in which the word group is recorded. By registering only words with a relatively high appearance frequency in the dictionary, the accuracy of the dictionary can be improved. Further, by adding words to a word group dictionary prepared in advance, the configuration of the dictionary can be simplified.

  In the dictionary generation device according to another aspect, the selection unit extracts words having an appearance frequency equal to or higher than a threshold as registration candidates, selects a predetermined number of words from the registration candidates in order from the word having the highest appearance frequency, and registers them. The unit may register the word selected by the selection unit in a dictionary different from the dictionary in which the word group is recorded. By registering only words with a relatively high appearance frequency in the dictionary, the accuracy of the dictionary can be improved. Further, by adding words to a dictionary different from the dictionary of existing word groups (existing dictionary), it is possible to generate a dictionary having characteristics different from those of the existing dictionary.

  In the dictionary generation device according to another embodiment, the registration unit may register the word selected by the selection unit in a dictionary different from the dictionary in which the word group is recorded. By adding words to a dictionary different from a dictionary of existing word groups (existing dictionary), a dictionary having characteristics different from those of the existing dictionary can be generated.

  In the dictionary generation device according to another aspect, the selection unit extracts words whose appearance frequency is equal to or higher than a threshold as registration candidates, groups the registration candidate words according to the appearance frequency, and the registration unit The plurality of groups generated by the selection unit may be individually registered in a plurality of dictionaries different from the dictionary in which the word group is recorded. By grouping words according to the appearance frequency and registering the generated groups in different dictionaries, a plurality of dictionaries having different characteristics due to the appearance frequency can be generated.

  In the dictionary generation device according to another aspect, each of the collected texts is associated with information indicating the field of the text, and the registration unit includes the word selected by the selection unit. It may be individually registered in a dictionary prepared for each field based on the field of the text. By generating a dictionary for each field, a plurality of dictionaries having different characteristics can be generated.

In the dictionary generation device according to another aspect, the appearance frequency of each word may be calculated based on the first, second, and third information. The text can be more appropriately divided into a plurality of words by introducing the third information indicating an intermediate concept instead of simply selecting whether or not a boundary exists.

  In the dictionary generation device according to another aspect, the analysis unit includes a first binary classifier and a second binary classifier, and the first binary classifier has a first position for each character position. Whether to allocate information other than the first information, or whether the second binary classifier assigns information other than the first information by the first binary classifier For the inter-position, it may be determined whether the second information or the third information is assigned. By using a plurality of binary classifiers to determine boundary information in stages, it is possible to add boundary information to texts at high speed and efficiently.

  In the dictionary generation device according to another embodiment, the collected text set is divided into a plurality of groups, and the analysis unit, the selection unit, and the registration unit perform processing based on one of the plurality of groups. The model generation unit generates a word division model using the corpus, the word group, and the word registered by the registration unit, and then the analysis unit, the selection unit, and the registration unit are another one of the plurality of groups. One process may be executed.

  According to one aspect of the present invention, a large-scale word dictionary can be easily constructed.

It is a figure which shows the hardware constitutions of the dictionary production | generation apparatus which concerns on embodiment. It is a block diagram which shows the function structure of the dictionary production | generation apparatus shown in FIG. It is a figure for demonstrating the setting of boundary information (word boundary tag). It is a flowchart which shows operation | movement of the dictionary production | generation apparatus shown in FIG. It is a figure which shows the structure of the dictionary production | generation program which concerns on embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description is omitted.

  First, the functional configuration of the dictionary generation device 10 according to the embodiment will be described with reference to FIGS. The dictionary generation apparatus 10 extracts a word from the text set by analyzing a set of collected large amounts of text (hereinafter also referred to as “large-scale text”), and adds the extracted word to the dictionary It is.

  As shown in FIG. 1, the dictionary generation apparatus 10 includes a CPU 101 that executes an operating system, application programs, and the like, a main storage unit 102 that includes a ROM and a RAM, an auxiliary storage unit 103 that includes a hard disk, and the like. The communication control unit 104 includes a network card, an input device 105 such as a keyboard and a mouse, and an output device 106 such as a display.

  Each functional component of the dictionary generation device 10 to be described later reads predetermined software on the CPU 101 and the main storage unit 102, and controls the communication control unit 104, the input device 105, the output device 106, and the like under the control of the CPU 101. The operation is realized by reading and writing data in the main storage unit 102 and the auxiliary storage unit 103. Data and databases necessary for processing are stored in the main storage unit 102 and the auxiliary storage unit 103. In FIG. 1, the dictionary generation device 10 is illustrated as being configured by a single computer, but the functions of the dictionary generation device 10 may be distributed to a plurality of computers.

  As illustrated in FIG. 2, the dictionary generation device 10 includes a model generation unit 11, an analysis unit 12, a selection unit 13, and a registration unit 14 as functional components. When executing the word extraction process, the dictionary generation device 10 refers to the learning corpus 20, the existing dictionary 31, and the large-scale text 40 prepared in advance, and stores the extracted words in the word dictionary 30. Note that the word dictionary 30 includes at least the existing dictionary 31 and may further include one or more additional dictionaries 32. Before describing the dictionary generation device 10 in detail, these data will be described.

  The learning corpus 20 is a set of texts to which boundary information (annotations) indicating word boundaries (division positions when a sentence is divided into words) is attached (associated), and is prepared in advance as a database. Text is a sentence or character string consisting of a plurality of words. In the present embodiment, a predetermined number of texts randomly extracted from the titles and descriptions of the products stored in the website of the virtual shopping street are used as the material of the learning corpus 20.

  Boundary information is given to each extracted text by the evaluator manually. The setting of boundary information is performed based on two techniques of word division by point estimation and a three-stage word division corpus.

[Word segmentation by point estimation]
A word boundary tag b = b ₁ b ₂ ... B _n is assigned to text (character string) x = x ₁ x ₂ ... X _n (x ₁ , x ₂ ,..., X _n are characters). Here, b _i is a tag indicating whether or not a word boundary exists between characters x _i and x _{i + 1} (inter-character position), b _i = 1 is division, and b _i = 0 is non-division Respectively. Here, the value indicated by this tag b _i can be said to be the intensity of the split.

  Figure 3 shows the Japanese sentence “bo-rupen wo katta” (“(I) bought a ballpoint pen.” In English) with “n (n)” and “wo (wo)”. An example of determining a tag between “and“. The value of the word boundary tag is determined by referring to the feature obtained from the characters existing around it. For example, the value of the word boundary tag is set using three types of features, that is, a character feature, a character type feature, and a dictionary feature.

Character feature is in contact with the boundary b _i, or the boundary b _i all characters length n of the enclosing (n-gram), is a feature represented by a combination of relative positions with respect to the position b _i. For example, when n = 3 in FIG. 3, “−1 / n (n)”, “1 /” is set for the boundary b _i between “n (n)” and “wo (wo)”. (Wo) ""-2 / pen (pen) ""-1 / n (n wo) "" 1 / buy (wo ka) ""-3 / rupen ""-2 / pen (( 9 features of “pen wo”, “-1 / buy (n wo ka)”, “1 / buy (wo kat)”.

  Character type features are the same as the character features described above, except that character types are used instead of characters. As the character types, eight types of hiragana, katakana, kanji, upper case alphabet, lower case alphabet, arabic numerals, kanji numerals, and middle black (•) were considered. In addition, the character type to be used and the number of the types are not limited at all.

The dictionary feature is a feature representing whether or not a word having a length j (1 ≦ j ≦ k) located around the boundary exists in the dictionary. The dictionary feature is a flag indicating whether the boundary b _i is located at the end point of the word (L), whether it is located at the start point (R) or contained in the word (M), It is shown in combination with the length j of the word. If the words “pen” and “wo” are registered in the dictionary, dictionary features L2 and R1 are created for the boundary bi in FIG. As will be described later, when a plurality of dictionaries are used, a dictionary identifier is assigned to the dictionary feature. For example, if “pen” is registered in the dictionary A having the identifier DIC1, and “wo” is registered in the dictionary B having the identifier DIC2, the dictionary feature is DIC1-L2. It is expressed as DIC2-R1.

  In the present embodiment, the maximum length n of n-gram in the character feature and the character type feature is 3 and the maximum word length k in the dictionary feature is 8, but these values may be arbitrarily determined.

[Three-stage word division corpus]
In Japanese, there is a problem that it is difficult to uniquely determine the word boundary, and the mode of appropriate word division differs depending on the scene. As an example, assume a case where a keyword search is performed on a text set including the word “ballpoint pen” (“ballpoint pen” in English). If the “ballpoint pen (bo-rupen)” is not divided, text will not be extracted even if the search is performed using the keyword “pen” (“pen” in English) (decrease in recall). On the other hand, when the “ball-point pen (bo-rupen)” is divided into “ball (bo-ru)” (“ball” in English) and “pen (pen)”, the sports equipment “ball (bo-rupen)” (ru) "as a keyword, a text including" ballpoint pen (bo-rupen) "is extracted (decrease in accuracy).

  Therefore, a three-stage word division corpus that introduces the concept of “half division” as well as the binary of “division” and “non-division” as described above. The three-stage word division corpus is a technique that develops probabilistic word division that indicates a division mode with a probabilistic value. The three-stage word division corpus is used because the number of word division strengths that humans can actually recognize is only a few levels at most, and it is not necessary to indicate the mode of division with continuous probability values. For words that contain half-splitting, both the whole word and the components of the word are extracted, so it is possible to record words that are difficult for humans to determine whether they are split or non-split for the time being as half-splitting At the same time, it becomes easy to add boundary information. “Half-splitting” is an aspect indicating that a boundary is probabilistically present (within a probability range greater than 0 and less than 1) at the position between characters.

The three-stage word division corpus is a three-stage discrete stochastic word obtained by adding “half-division” (b _i = 0.5) to “division” (b _i = 1) and “non-division” (b _i = 0). This is a corpus generated by division. For example, a compound noun such as “ball / pen (bo-ru / pen)”, a compound verb such as “ori / tatam” (“fold” in English), “o / sume (o / ") (" Recommendation "in English), a division in a word that is lexicalized including an affix (indicated by" / "in these examples) is defined as a half division Is natural. In addition, “rechargeable battery” (in English, “rechargeable battery”) means “rechargeable” (in English, “recharge”) and “denchi” (in English, “battery”). Although it can be said that it is a compound word of the type “AB + BC → ABC”, such a word is divided in half as “charge / electricity / pond (juu / den / chi)”.

“I bought a ballpoint pen.” (Bo-rupen
The text “wo katta) is divided as shown in FIG. 3, for example, using the word division based on the above point estimation and the three-stage word division corpus. In the example of FIG. 3, the word boundary tag of “divided” (b _i = 1) is given at the beginning of the text or between “n (n)” and “m (wo)”. The word boundary tag of “half-divided” (b _i = 0.5) is assigned between “le” and “pe”. In FIG. 3, the word boundary tag of “non-divided” (b _i = 0) is omitted, but places where no boundary is expressed between characters (for example, “pe (pe)” and “n (n)”) The tag is assigned to (between).

  Each text is given a word boundary tag as boundary information and stored in the database as a learning corpus 20. The method for adding the boundary information to the text is arbitrary. As an example, boundary information may be embedded in each text so that “divided” is indicated by a space, “half-divided” is indicated by a hyphen, and the display of “non-divided” is omitted. In this case, the text with the boundary information can be recorded as a character string.

  The existing dictionary 31 is a set of a predetermined number of words, and is prepared in advance as a database. The existing dictionary 31 may be a generally used electronic dictionary, for example, a UniDic morphological analysis dictionary.

  The large-scale text 40 is a set of collected text and is prepared in advance as a database. The large-scale text 40 may include an arbitrary sentence or character string according to the word to be extracted and the field of the word. For example, a large amount of product titles and explanations may be collected from a virtual shopping street website, and the large-scale text 40 may be constructed from these raw data. The number of texts prepared as the large-scale text 40 is overwhelmingly larger than the number of texts included in the learning corpus 20.

  Based on the above, functional components of the dictionary generation device 10 will be described.

  The model generation unit 11 is means for generating a word division model using the learning corpus 20 and the word dictionary 30. The model generation unit 11 includes a support vector machine (SVM), and generates a word division model by inputting a learning corpus 20 and a word dictionary 30 to the machine and executing learning processing. To do. This word segmentation model shows the rules on how to segment text, and is output as a parameter group used for word segmentation. The algorithm used for machine learning is not limited to SVM, and may be a decision tree or logistic regression.

  In order to analyze the large-scale text 40, the model generation unit 11 causes the SVM to perform learning based on the learning corpus 20 and the existing dictionary 31, thereby generating an initial word division model (baseline model). Then, the model generation unit 11 outputs this word division model to the analysis unit 12.

  Thereafter, when a word is added to the word dictionary 30 by the processing of the analysis unit 12, the selection unit 13, and the registration unit 14 described later, the model generation unit 11 performs learning (re-execution) based on the learning corpus 20 and the entire word dictionary 30. A corrected word division model is generated by causing the SVM to execute (learning) processing. Here, the whole word dictionary 30 means all the words stored in the existing dictionary 31 from the beginning and the words obtained from the large-scale text 40.

  The analysis unit 12 is a unit that performs analysis (word division) in which the word division model is incorporated on the large-scale text 40 and gives (associates) boundary information to each text. As a result, a large amount of text as shown in FIG. 3 is obtained. The analysis unit 12 performs such word division on each text constituting the large-scale text 40, so that the “division” (second information), “half-division” (third information), and Boundary information indicating “non-divided” (first information) is assigned to each text, and all processed texts are output to the selection unit 13.

  The analysis unit 12 includes two binary classifiers, and uses these classifiers in order to give three types of boundary information to each text. The first classifier is means for determining whether the inter-character position is “non-divided” or otherwise, and the second classifier is whether the boundary determined not to be “non-divided” is “divided” or “ It is a means for determining whether it is “half-split”. In reality, since the majority of the inter-character positions are “non-divided”, it is first determined whether or not the inter-character positions are “non-divided”, and then determined to be other than “non-divided”. By determining the division mode for, boundary information can be efficiently given to a large amount of text. Moreover, the structure of the analysis part 12 can be simplified by combining a binary classifier.

  The selection unit 13 is means for selecting a word to be registered in the word dictionary 30 from the text to which boundary information is given by the analysis unit 12.

ここで、Ｏ_１は単語ｗの表記の出現を示しており、下記の通りに定義される。

First, the selection part 13 calculates | requires total frequency _fr (w) of each word w contained in the input text group by following formula (1). This calculation means that the appearance frequency can be obtained from the boundary information b _i given to each inter-character position.

Here, O ₁ indicates the appearance of the notation of the word w and is defined as follows.

  The frequency of appearance of the word “ball-point” (bo-rupen) in one sentence “bo-rupen wo katta” shown in FIG. 3 is 1.0 * 1.0 * 1.0 * 0. .5 * 1.0 * 1.0 = 0.5, and the appearance frequency of the word “pen” in the sentence is 0.5 * 1.0 * 1.0 = 0.5. These mean that the words “bo-rupen” and “pen” appear to appear 0.5 times each in the sentence. The selection part 13 calculates | requires the appearance frequency of each word contained in each text, and obtains the total appearance frequency of each word by totaling the appearance frequency for every word.

  Subsequently, the selection unit 13 selects only words whose total appearance frequency is equal to or higher than the first threshold THa from the word group in the large-scale text 40 as a registration candidate V (word truncation based on frequency). Then, the selection unit 13 selects a word to be finally registered in the word dictionary 30 from the registration candidates V, and determines a dictionary (database) for storing the word as necessary. The method of determining the word to be finally registered and the dictionary of the storage destination is not limited to one, and various methods can be used as described below.

  The selection unit 13 may determine to add only words whose total appearance frequency is equal to or higher than a predetermined threshold among the registration candidates V to the existing dictionary 31. In this case, the selection unit 13 may select only words whose total appearance frequency is the second threshold THb (where THb> THa), or may select only words whose total appearance frequency is the top n. Hereinafter, such processing is also referred to as “APPEND”.

  Alternatively, the selection unit 13 may determine to register only words whose total appearance frequency is greater than or equal to a predetermined threshold among the registration candidates V in the additional dictionary 32. Also in this case, the selection unit 13 may select only words whose total appearance frequency is the second threshold THb (where THb> THa), or may select only words whose total appearance frequency is the top n. Hereinafter, such processing is also referred to as “TOP”.

  Alternatively, the selection unit 13 may determine that all the registration candidates V are registered in the additional dictionary 32. Hereinafter, such processing is also referred to as “ALL”.

Alternatively, the selection unit 13 may decide to divide the registration candidates V into a plurality of subsets according to the total appearance frequency and register each subset in the individual additional dictionary 32. Of the registration candidates V, a subset having a total appearance frequency up to the top _n is represented as V _n . In this case, for example, the selection unit 13 sets the subset V ₁₀₀₀ composed of the words up to the top 1000, the subset V ₂₀₀₀ composed of the words up to the top _2000, and the subset V ₃₀₀₀ composed of the words up to the top _3000. And generate Then, the selection unit 13 determines to register the subsets V ₁₀₀₀ , V ₂₀₀₀ , and V ₃₀₀₀ in the first additional dictionary 32, the second additional dictionary 32, and the third additional dictionary 32. Note that the number of subsets to be generated and the size of each subset may be arbitrarily determined. Hereinafter, such processing is referred to as “MULTI”.

  When the word to be registered is finally selected and the storage destination dictionary is determined, the selection unit 13 outputs the selection result to the registration unit 14.

  The registration unit 14 is means for registering the word selected by the selection unit 13 in the word dictionary 30. Which dictionary to register the word in the word dictionary 30 depends on the processing in the selection unit 13, so the registration unit 14 may register the word only in the existing dictionary 31, or the word only in one additional dictionary 32. May register. In the case of the above “MULTI” process, the registration unit 14 divides the selected word into a plurality of additional dictionaries 32 and registers them.

  As described above, the word added to the word dictionary 30 is used for correcting the word division model, but the word dictionary 30 may be used for purposes other than word division. For example, the word dictionary 30 may be used for morphological analysis, display of input candidate words in an input box having an automatic input function, a knowledge database for extracting proper nouns, and the like.

  Next, the operation of the dictionary generation device 10 will be described with reference to FIG. 4 and the dictionary generation method according to the present embodiment will be described.

  First, the model generation unit 11 generates an initial word division model (baseline model) by causing the SVM to perform learning based on the learning corpus 20 and the existing dictionary 31 (step S11, model generation step). Subsequently, the analysis unit 12 performs an analysis (word division) in which the baseline model is incorporated on the large-scale text 40, and indicates a boundary indicating “division”, “half-division”, or “non-division”. Information is assigned (associated) to each text (step S12, analysis step).

  Subsequently, the selection unit 13 selects a word to be registered in the dictionary (selection step). Specifically, the selection unit 13 calculates the total appearance frequency of each word based on the text with boundary information (step S13), and selects a word whose frequency is a predetermined threshold or more as a registration candidate (step S14). ). Then, the selection unit 13 selects a word to be finally registered in the dictionary from registration candidates and determines a dictionary in which the word is registered (step S15). The selection unit 13 can select a word and designate a dictionary by using the above-described techniques such as APPEND, TOP, ALL, and MULTI.

  Subsequently, the registration unit 14 registers the selected word in the designated dictionary based on the processing in the selection unit 13 (step S16, registration step).

  With the above processing, the addition of words to the word dictionary 30 is completed. In the present embodiment, the word division model is corrected using the expanded word dictionary 30. That is, the model generation unit 11 generates a corrected word division model by relearning based on the learning corpus 20 and the entire word dictionary 30 (step S17).

  Next, a dictionary generation program P1 for causing a computer to function as the dictionary generation device 10 will be described with reference to FIG.

  The dictionary generation program P1 includes a main module P10, a model generation module P11, an analysis module P12, a selection module P13, and a registration module P14.

  The main module P10 is a part that comprehensively controls the dictionary generation function. The functions realized by executing the model generation module P11, the analysis module P12, the selection module P13, and the registration module P14 are the functions of the model generation unit 11, the analysis unit 12, the selection unit 13, and the registration unit 14, respectively. It is the same.

  The dictionary generation program P1 is provided after being fixedly recorded on a tangible recording medium such as a CD-ROM, DVD-ROM, or semiconductor memory. Further, the dictionary generation program P1 may be provided via a communication network as a data signal superimposed on a carrier wave.

  As described above, according to the present embodiment, a word division model is generated using the learning corpus 20 to which boundary information is given and the existing dictionary 31, and the word division incorporating the model is large-scale. Applies to text 40. Then, by this application, a word is selected from the text set to which boundary information is given and registered in the word dictionary 30. As described above, by adding boundary information to a text set by analysis using the learning corpus 20 and registering words extracted from the text set, a large-scale word dictionary 30 can be easily constructed. Can do.

  For example, “sumahoke-su” (in English, “smartphone case”) is divided into “sumaho” and “case” (ke-su). sumaho) can be registered in the dictionary. Note that “sumaho” is an abbreviation for “suma-tofon” in Japanese. Also, the phrase “uttororin” (an unknown word corresponding to “uttori” in Japanese (“fascinated” in English)) can be registered in the dictionary. Then, by performing text analysis using the constructed dictionary, word segmentation of sentences containing registered words (for example, sentences containing “sumaho” or “uttororin”) can be performed more accurately. Executed.

Next, an example of evaluation of word division performance by the dictionary generation device 10 in the present embodiment will be shown. Accuracy (Prec), recall (Rec), and F value were used as indexes for evaluating word division performance. If the total number of words included in the correct corpus is N _REF , the total number of words included in the analysis result is N _{SY S} , and the total number of words included in both the analysis result and the correct corpus is N _COR , the above three indicators are as follows: Is defined as follows.
Prec = N _COR / N _SYS
Rec = N _COR = N _REF
F = 2Prec · Rec / (Prec + Rec)

  UniDic's headword list (304,267 different words) was used as an existing dictionary, and LIBLINEAR was used as a support vector machine with default parameters. All the half-width characters in the learning corpus and large text were unified, but no further normalization was performed.

  First, the effectiveness when the learning corpus and the large-scale text are in the same field (learning in the same field) will be described. Here, the field is a concept for grouping sentences and words based on style, contents (genre), and the like. In learning in the same field, from the title and description of 590 products randomly extracted from the website of the virtual shopping mall A without genre bias, and the description of 50 products randomly extracted from the website of the virtual shopping mall B A learning corpus with three-level word division was created. The number of words in this learning corpus was about 110,000 and the number of characters was about 340,000. The performance was evaluated using this learning corpus.

  As large-scale texts, the titles and explanations of all product data in the virtual shopping street A were used. The number of products was about 27 million and the number of characters was about 16 billion.

  When this large-scale text is analyzed by the baseline model and two-stage word division is executed, different 576,954 words are extracted, and when three-stage word division is executed after the analysis, different 603, 187 words were extracted. Here, the frequency threshold used for word cut-off is 20. When the above “MULTI” was adopted, the top 100,000 words, the top 200,000 words, the top 300,000 words, the top 400,000 words and the whole of the total appearance frequency were added as separate dictionaries. When the above “TOP” was adopted, only the top 100,000 words were used.

Table 1 shows the learning result by the baseline model, the result of relearning using the word dictionary obtained by the two-stage word division, and the result of relearning using the word dictionary obtained by the three-stage word division. All values in Table 1 are percentages (%).

  When re-learning using two-stage word segmentation, the F-value is improved no matter which method (APPEND / TOP / ALL / MULTI) is used, which means that the proposed large text It shows that the learning used is effective. The increment of the F value was larger in the order of APPEND <TOP <ALL <MULTI. From this result, when adding a word, it is more effective to add it to another dictionary than to add it to an existing dictionary, and furthermore, it appears more than registering the word to be added to one additional dictionary. It was found that it was more effective to add to different dictionaries according to frequency.

  From Table 1, it can be considered that the classifier automatically learns different contributions and weights depending on the appearance frequency of words. Furthermore, when re-learning using three-stage word division, performance improved in all cases over the baseline model and two-stage word division. Specifically, by taking into account half-division, improvements such as accurately acquiring words with affixes were obtained.

  Next, the effectiveness when the learning corpus and the large-scale text are different fields will be described. The learning corpus used was the same as that used for learning in the same field. On the other hand, the large-scale text used a user review in the travel reservation site C, an accommodation facility name, an accommodation plan name, and a response from the accommodation facility. The number of texts was 348,564, and the number of characters was about 126 million. Of this large-scale text, 150 and 50 reviews were randomly extracted and manually divided into words, which were used as a test corpus and an active learning corpus (additions to the learning corpus), respectively.

  First, a large-scale text in the travel field was analyzed using the baseline model learned from the above-mentioned learning corpus in the product field. This analysis performance is the “baseline” in Table 2 below.

  Next, we added a field-adaptive corpus to the product-area learning corpus to learn the word segmentation model, and then used it to analyze large text. This analysis performance is “field adaptation” in Table 2 below. After analyzing large-scale text, 41,671 words were extracted when using two-stage word division, and 44,247 words were extracted when using three-stage word division. In all cases, only words with a total appearance frequency of 5 or more were used.

Table 2 shows the results of adding these obtained words to the dictionary and re-learning the model using the learning corpus and the field adaptation corpus. All values in Table 2 are percentages (%).

  As can be seen from the table, when the learning corpus and the large-scale text are different, the performance was improved in the case of three-stage word division.

  The present invention has been described in detail based on the embodiments. However, the present invention is not limited to the above embodiment. The present invention can be variously modified without departing from the gist thereof.

  In the above embodiment, the selection unit 13 selects a word based on the appearance frequency. However, the selection unit 13 may register all the words in the existing dictionary 31 or the additional dictionary 32 without referring to the appearance frequency. . Also, word truncation is not an essential process.

  In the above embodiment, the processing by the selection unit 13 and the registration unit 14 is performed after the analysis unit 12 analyzes the entire large-scale text 40. However, the analysis unit 12 analyzes a large amount of collected text in multiple times. May be. In this case, a series of processes including a model generation step, an analysis step, a selection step, and a registration step are repeated a plurality of times. For example, when the large-scale text 40 is divided into groups 1 to 3, the group 1 is analyzed and the word is registered in the first loop process, and the group 2 is analyzed in the second loop process to further add the word. Registered, group 3 is analyzed in the process of the third loop, and further words are registered. In the processing after the second loop, the model generation unit 11 refers to the entire word dictionary 30 and generates a corrected word division model.

In the above embodiment, since the three-stage division method is used, there are three types of boundary information. However, the mode of the boundary information is not limited to this example. For example, two-stage word division may be performed using only two types of boundary information “division” and “non-division”. Furthermore, word division may be performed in four or more stages using “division”, “non-division”, and a plurality of types of probabilistic division. For example, four-stage word division may be performed using probabilistic division (third information) of b _i = 0.33 and b _i = 0.67. In any case, the strength of the division corresponding to the third information is larger than the strength when the boundary information is “non-divided” (for example, b _i = 0), and the strength when the boundary information is “divided” (for example, smaller than b _i = 1).

  According to this embodiment, a large-scale word dictionary can be easily constructed.

  DESCRIPTION OF SYMBOLS 10 ... Dictionary production | generation apparatus, 11 ... Model production | generation part, 12 ... Analysis part, 13 ... Selection part, 14 ... Registration part, 20 ... Learning corpus, 30 ... Word dictionary, 31 ... Existing dictionary (word group), 32 ... Additional dictionary , 40 ... large-scale text (collected text collection), P1 ... dictionary generation program, P10 ... main module, P11 ... model generation module, P12 ... analysis module, P13 ... selection module, P14 ... registration module.

Claims (14)

A model generation unit that generates a word division model using a corpus and a word group prepared in advance, and each text included in the corpus is provided with boundary information indicating a word boundary, and the boundary information is The first information indicating that the boundary does not exist at the inter-character position, the second information indicating that the boundary exists at the inter-character position, and the probability that the boundary exists at the inter-character position. The model generation unit including third information to be shown ;
An analysis unit that executes word division in which the word division model is incorporated with respect to a set of collected texts and gives the boundary information to each text;
A selection unit for selecting a word to be registered in the dictionary from the text to which the boundary information is given by the analysis unit;
A dictionary generation apparatus comprising: a registration unit that registers a word selected by the selection unit in the dictionary. The selection unit selects a word to be registered in the dictionary based on the appearance frequency of each word calculated from the boundary information given by the analysis unit.
The dictionary generation device according to claim 1. The selection unit selects a word having the appearance frequency equal to or higher than a predetermined threshold;
The dictionary generation device according to claim 2. The selection unit extracts words having the appearance frequency equal to or higher than the threshold as registration candidates, and selects a predetermined number of words from the registration candidates in order from the word having the highest appearance frequency,
The registration unit adds the word selected by the selection unit to a dictionary in which the word group is recorded;
The dictionary generation device according to claim 3. The selection unit extracts words having the appearance frequency equal to or higher than the threshold as registration candidates, and selects a predetermined number of words from the registration candidates in order from the word having the highest appearance frequency,
The registration unit registers the word selected by the selection unit in a dictionary different from the dictionary in which the word group is recorded;
The dictionary generation device according to claim 3. The registration unit registers the word selected by the selection unit in a dictionary different from the dictionary in which the word group is recorded;
The dictionary generation device according to claim 3. The selection unit extracts words having the appearance frequency equal to or higher than the threshold as registration candidates, and groups the registration candidate words according to the height of the appearance frequency.
The registration unit individually registers a plurality of groups generated by the selection unit in a plurality of dictionaries different from a dictionary in which the word group is recorded.
The dictionary generation device according to claim 3. Each of the collected texts is associated with information indicating the field of the text,
The registration unit individually registers the words selected by the selection unit in a dictionary prepared for each field based on the field of the text in which the word was included.
The dictionary generation device according to claim 3. The appearance frequency of each word is calculated based on the first, second, and third information.
The dictionary production | generation apparatus as described in any one of Claims 2-8. The analysis unit includes a first binary classifier and a second binary classifier,
The first binary classifier determines whether to assign the first information or information other than the first information for each inter-character position;
Whether the second binary classifier assigns the second information for the inter-character position determined by the first binary classifier to assign information other than the first information, or the third binary classifier Determine whether to assign information,
The dictionary generation device according to claim 9. The collected set of text is divided into a plurality of groups;
After the analysis unit, the selection unit, and the registration unit execute processing based on one of the plurality of groups, the model generation unit is registered by the corpus, the word group, and the registration unit. Generating the word division model using a word, and subsequently, the analysis unit, the selection unit, and the registration unit perform processing based on another one of the plurality of groups,
The dictionary production | generation apparatus as described in any one of Claims 1-10. A dictionary generation method executed by a dictionary generation device,
In the model generation step of generating a word division model using a corpus and a word group prepared in advance, each text included in the corpus is given boundary information indicating a word boundary, and the boundary information is The first information indicating that the boundary does not exist at the inter-character position, the second information indicating that the boundary exists at the inter-character position, and the probability that the boundary exists at the inter-character position. The model generation step including third information to be shown ;
Analyzing the set of collected texts by performing word division incorporating the word division model and adding the boundary information to each text;
A selection step of selecting a word to be registered in the dictionary from the text given the boundary information in the analysis step;
A registration step of registering the word selected in the selection step in the dictionary. A model generation unit that generates a word division model using a corpus and a word group prepared in advance, and each text included in the corpus is provided with boundary information indicating a word boundary, and the boundary information is The first information indicating that the boundary does not exist at the inter-character position, the second information indicating that the boundary exists at the inter-character position, and the probability that the boundary exists at the inter-character position. The model generation unit including third information to be shown ;
An analysis unit that executes word division in which the word division model is incorporated with respect to a set of collected texts and gives the boundary information to each text;
A selection unit for selecting a word to be registered in the dictionary from the text to which the boundary information is given by the analysis unit;
A dictionary generation program that causes a computer to execute a registration unit that registers a word selected by the selection unit in the dictionary. A model generation unit that generates a word division model using a corpus and a word group prepared in advance, and each text included in the corpus is provided with boundary information indicating a word boundary, and the boundary information is The first information indicating that the boundary does not exist at the inter-character position, the second information indicating that the boundary exists at the inter-character position, and the probability that the boundary exists at the inter-character position. The model generation unit including third information to be shown ;
An analysis unit that executes word division in which the word division model is incorporated with respect to a set of collected texts and gives the boundary information to each text;
A selection unit for selecting a word to be registered in the dictionary from the text to which the boundary information is given by the analysis unit;
A computer-readable recording medium that stores a dictionary generation program that causes a computer to execute a registration unit that registers a word selected by the selection unit in the dictionary.

Images