JP2006293767A - Sentence categorizing device, sentence categorizing method, and categorization dictionary creating device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To resolve a problem that it is difficult to categorize sentence data into a plurality of categories. <P>SOLUTION: A categorization dictionary storing part 24 stores dictionary data wherein a component of a sentence is associated with an appearance frequency of the component in the sentence categorized into each category in regard to the plurality of predetermined categories. A sentence receiving part 36 receives un-categorized sentence data being a new categorization target from an external device. A sentence analyzing part 18 analyzes the un-categorized sentence data in accordance with predetermined rules to extract sentence components. A category attribute probability calculating part 26 refers to the dictionary data to calculate an attribute probability in regard to each category indicating a probability of the un-categorized sentence data including the extracted components attributing to each category. A determining part 28 refers to the attribute probability to determine a category of the un-categorized sentence data. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a sentence classification technique, and more particularly to a sentence classification technique for classifying uncategorized sentence data into a plurality of predetermined categories.

  As the Internet has become widespread in society, the amount of text distributed on the network has increased dramatically. For this reason, it is becoming difficult to manually perform operations such as classifying and displaying web pages collected through a network and classifying a large amount of e-mails into appropriate folders. Therefore, a text classification technique for automatically classifying text data into a predetermined category has been devised.

For example, there is a sentence classification technique using a feature vector method (see, for example, Patent Document 1). In this technique, sentences are classified by the following steps. First, W _i = {w _ij } representing the importance w _ij of each word j with respect to category i as a vector is generated from a collection of sentence examples belonging to category i. Next, using a word that appears in an uncategorized sentence, a feature vector W of the sentence is generated. Then, a vector W _n closest to the feature vector W is obtained, and the sentence is classified into a corresponding category n.
Japanese Unexamined Patent Publication No. 2000-222431

  However, in sentence classification using the feature vector method, when a sentence to be classified contains a certain word, it is very likely that it is classified into a specific category or not classified into a specific category. Under such conditions, such conditions cannot be reflected well in the sentence classification, and misclassification increases.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a technique for automatically classifying sentence data according to a predetermined classification system.

  One embodiment of the present invention is a sentence classification device. The apparatus includes a classification dictionary holding unit that holds dictionary data in which a sentence component is associated with an appearance frequency at which the component appears in a sentence to be classified into each category for a plurality of predetermined categories. A sentence receiving unit that receives unclassified sentence data to be newly classified from an external device, a sentence decomposing unit that analyzes the unclassified sentence data according to a predetermined rule and extracts constituent elements of the sentence, and the dictionary With reference to the data, with reference to the attribution probability, a category attribution probability calculating unit that calculates an attribution probability representing the probability that the uncategorized sentence data including the extracted constituent element belongs to each category, A determination unit that determines a category into which the uncategorized sentence data is classified.

  According to this aspect, it is determined according to the belonging probability calculated for each category to which category the uncategorized sentence data belongs. Therefore, one sentence data can be classified into one category, and can be classified into two or more categories. In addition to classifying categories based on one viewpoint, categories based on a plurality of viewpoints can be mixed, and the belonging probability can be calculated collectively.

  “Probability of belonging” is a probability that uncategorized text data including several components should be classified into a certain category, and can be calculated for each category. This attribution probability is calculated for each component extracted from uncategorized text data by calculating the appearance probability by taking out the appearance frequency in the category for which the attribution probability is calculated from the classification dictionary holding unit. Can be calculated by obtaining the probability of belonging to the category by combining the appearance probabilities.

  Another aspect of the present invention is a classification dictionary creation device. This apparatus stores a sentence example storage that stores, for a plurality of predetermined categories, a category sentence example data group that includes sentence data to be classified into each category, and a non-category sentence example data group that includes sentence data that is not classified into a category. A sentence decomposition unit that analyzes sentence data according to a predetermined rule and extracts constituent elements of the sentence, and sentence data included in the category sentence example data group and the non-category sentence example data group, A classification dictionary creating unit that counts the appearance frequency of the extracted component element for each category, a classification dictionary holding unit that stores dictionary data in which the component element and the appearance frequency in each category are associated, and an external device A dictionary providing unit for providing the dictionary data.

  According to this aspect, the classification dictionary creation device is held in the classification dictionary holding unit when dictionary data is requested from an external device in order to calculate the attribution probability of uncategorized sentence data to each category. Dictionary data can be transmitted to an external device.

  It should be noted that any combination of the above-described components and a representation of the present invention by a method, apparatus, system, recording medium, and computer program are also effective as an aspect of the present invention.

  According to the present invention, uncategorized text data can be automatically classified along a predetermined category.

  In one embodiment of the present invention, dictionary data for classification is created using a large number of example sentence data that is determined in advance as to whether or not it belongs to a specific category. It is a sentence classification device that determines a category in which sentence data should be classified by calculating a probability relating to whether or not each category belongs to the category. Hereinafter, the sentence classification device according to the present embodiment will be described with reference to the drawings.

  FIG. 1 shows an example of a usage pattern of a sentence classification device 10 according to the present embodiment. The sentence classification device 10 is connected to the client terminal 82 and the server 84 via the network 80. The sentence classification device 10 classifies the sentence data transmitted from the client terminal 82 or the server 84 into any of a plurality of preset categories. The sentence classification device 10 classifies sentence data collected from a large number of client terminals 82 and servers 84 connected to a network by a web crawler (also called a search robot) (not shown) into any of a plurality of categories. . The sentence classification apparatus 10 according to the present embodiment is characterized in that it can also determine that the sentence data to be classified belongs to two or more categories.

  FIG. 2 is a functional block diagram of the sentence classification device 10. The sentence classification device 10 includes a sentence example storage unit 12, a sentence decomposition unit 18, a sorting unit 20, a classification dictionary creation unit 22, a classification dictionary holding unit 24, a category attribution probability calculation unit 26, a determination unit 28, an element narrowing reference provision unit 30, The determination result storage unit 32 and the text reception unit 36 are provided. These configurations can be realized in terms of hardware by a CPU, memory, or other LSI of any computer, and in terms of software, they are realized by programs loaded into the memory. It depicts the functional blocks that are realized. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The functional blocks included in the sentence classification device 10 are used in a learning stage for creating dictionary data and in a classification stage for classifying unclassified sentence data using the dictionary data after the dictionary data is created. Can be divided into functional blocks. First, functional blocks used in the learning stage will be described.

  The sentence example storage unit 12 stores, for each of a plurality of predetermined categories, a category sentence example 14 including sentence data to be classified into the category and a non-category sentence example 16 including sentence data not classified into the category. Here, “category” refers to grouping for classifying text data according to a specific standard. This category can be variously set according to the purpose of use of the sentence classification result. For example, when the classification result by the sentence classification device 10 is used on a news article distribution site, the category types such as “politics”, “economy”, “society”, and “sports” are conceivable. When the classification result by the sentence classification device 10 is provided on a directory-type search site, categories such as “shopping”, “travel”, “movie”, and “music” can be considered. In the case of classifying a questionnaire or the like transmitted from a client terminal by the sentence classification device 10, categories such as “female” and “young people” can be considered.

  “Text data” refers to data including text information such as text data, HTML, XML, and XHTML files, but the data format is not limited.

  Assuming that M categories (category 1,..., Category M) are defined, a category sentence example 14 and a non-category sentence example 16 are prepared in the sentence example storage unit 12 for each category. The category sentence example is a file in which one or a plurality of sentence data to be classified into a certain category is accumulated. Non-category sentence example data is a file in which one or a plurality of sentence data that are not classified into a certain category are accumulated. The determination of which category sentence example or non-category sentence example the sentence data is included is made manually. A pair of category sentences and non-category sentence examples are prepared for each category.

  Text data included in a non-category sentence example for a certain category may be different from or identical to sentence data included in a non-category sentence example for another category. That is, the sentence data in the “non-category 1 sentence example” and the sentence data in the “non-category 2 sentence example” may overlap. The same applies to category sentence examples. For example, sentence data classified into both “category 1 sentence example” and “category 2 sentence example” may exist. However, it is desirable that there is no sentence data classified into both category sentence examples and non-category sentence examples within the same category.

  The sentence decomposition unit 18 analyzes sentence data included in the category sentence example 14 and the non-category sentence example 16 described above according to a predetermined rule, and extracts constituent elements of the sentence. The sentence decomposition unit 18 can use any known sentence decomposition algorithm. The sentence decomposition unit 18 extracts words, combinations of words and parts of speech, phrases, simple sentences, and the like from the sentence data as constituent elements according to the sentence decomposition algorithm. An example of the sentence decomposition algorithm used in the sentence decomposition unit 18 will be described later.

  The sorting unit 20 rearranges the components extracted from the sentence data of the category sentence example 14 and the non-category sentence example 16 by the sentence decomposition unit 18 according to a predetermined rule. As an example, the readings of the constituent elements may be rearranged in the order of 50 sounds. Or you may rearrange in order of the ASCII code of the first character of a component. When the sentence decomposition unit 18 extracts combinations of words and parts of speech as constituent elements, they may be arranged in the part of speech order of the constituent elements. By rearranging the constituent elements by the sorting unit 20, it becomes easy to search dictionary data using the constituent elements described later as keys, so that the time required for creating the classification dictionary can be shortened.

The classification dictionary creation unit 22 calculates the frequency at which each component extracted by the sentence decomposition unit 18 appears in the sentence data in the category sentence example 14 and the non-category sentence example 16 for each category, and for each component and each category Create dictionary data in association with the appearance frequency of. “Frequency” here may be the number of appearances or the ratio of appearance to the total number of words. Alternatively, it may be an appearance ratio with respect to the number of sentence data included in the category sentence example or the non-category sentence example. Hereinafter, these are collectively referred to as “appearance frequency”. In any case, any numerical value can be used as long as it is a numerical value representing the degree of appearance of a certain component in a category sentence example and a non-category sentence example for one category.
The dictionary data created by the classification dictionary creation unit 22 is stored in the classification dictionary holding unit 24. A more detailed configuration and function of the classification dictionary creation unit 22 will be described later with reference to FIG.

  The element narrowing criteria providing unit 30 stores selection criteria for excluding some components from the components extracted by the text decomposition unit 18. This selection criterion is, for example, a condition that specifies a specific part of speech (for example, a noun + verb, only a noun, only a particle, etc.), specifies the upper limit of the number of characters, specifies a component consisting of only hiragana or only kanji I mean. A combination of a plurality of conditions may be used as a selection criterion.

The classification dictionary creating unit 22 uses the selection criteria provided from the element narrowing criteria providing unit 30 to narrow down the number of components for which dictionary data is to be created, thereby storing the dictionary data stored in the classification dictionary holding unit 24. It is possible to create dictionary data effective for classification while suppressing the amount of data.
The sentence classification apparatus 10 of the present embodiment does not limit the language of sentence data to be classified, but the element narrowing reference providing unit 30 is particularly useful when processing Japanese sentence data.

  Subsequently, functional blocks used in the classification stage will be described.

  The text receiving unit 36 receives text data 34 to be classified (hereinafter referred to as “unclassified text data 34”) from an external device (not shown). The data format of the uncategorized text data 34 is not limited as described above. The external device is, for example, a client terminal or server connected to the network, but is not limited to these.

  The sentence decomposing unit 18 receives the uncategorized sentence data 34 from the sentence receiving unit 36 and extracts the constituent elements of the sentence in the same manner as described above. The extracted components are rearranged by the sorting unit 20 according to a predetermined rule. This rule is preferably the same rule as when the components extracted from the category sentence example and the non-category sentence example are rearranged. This rearrangement facilitates the search of the dictionary data in the classification dictionary holding unit 24 using the constituent elements as keys, thereby speeding up the processing in the category attribution probability calculating unit 26 described later.

  The category attribution probability calculation unit 26 refers to the dictionary data stored in the classification dictionary holding unit 24, and calculates the probability that uncategorized sentence data including some components should be classified into each category. calculate. Hereinafter, this probability is referred to as “attribute probability”.

Similar to the classification dictionary creation unit 22, the category attribution probability calculation unit 26 may limit the number of constituent elements serving as a basis for calculating the attribution probability according to the selection criterion provided from the element narrowing criterion provision unit 30.
A more detailed configuration and function of the category attribution probability calculation unit 26 will be described later with reference to FIG.

  The determination unit 28 acquires the belonging probability calculated for each category by the category belonging probability calculating unit 26 and determines whether to classify the unclassified sentence data into any category based on the value of the belonging probability. More specifically, the determination unit 28 classifies the unclassified sentence data into the category having the maximum attribution probability. Alternatively, uncategorized sentence data may be classified into all categories for which an attribution probability equal to or higher than a preset threshold is obtained. By doing so, one uncategorized sentence data can be classified into two or more categories by a series of operations. When there is no category having the attribution probability equal to or higher than the threshold, the determination unit 28 may determine that the uncategorized sentence data is a sentence not classified into any category, or the category having the maximum attribution probability. May be classified. The determination result of the uncategorized sentence data by the determination unit 28 is stored in the determination result storage unit 32 or output to an external device (not shown).

  Next, an outline of the sentence decomposition algorithm used in the sentence decomposition unit 18 will be described.

(1) Morphological Analysis FIG. 3 shows an example in which sentence data is decomposed into constituent elements by morphological analysis. The text data used is the text "The Japan Meteorological Agency announced that the first spring in the Kanto region blew on the 23rd." As shown in FIG. 3, this sentence is “Meteorological Agency / Ha / 2/3 / Day /, / Kanto / Region / De / Spring Ichiban / Ga / Blow / Ta / To / Announcement / Shi / Ta /. Thus, it is broken down into 18 elements. In the morphological analysis, the utilization form 50, the original form 52, and the part of speech 54 can be determined from the target sentence. Of these utilization forms, original forms, and parts of speech, (original form + part of speech) may be a constituent element, or only the original form may be an element. The utilization form may be used as an element instead of the original form.

  Since components extracted using morphological analysis have high sentence resolution, classification into categories based on dictionary data using these components is expected to be highly accurate. Since morphological analysis is a well-known technique, further explanation is omitted.

(2) Syntax analysis Next, syntax analysis will be described. Parsing breaks a sentence into phrases. When the same sentence data as in the example of FIG. 3 is decomposed by parsing, it is decomposed into six components as follows: “The Japan Meteorological Agency / 23 days / In the Kanto area / Spring first / Announced / I announced.” Is done.

  When parsing is used, the number of components is significantly reduced compared to morphological analysis, which is suitable for high-speed classification, but the accuracy of classification is reduced. Since parsing is a well-known technique, further explanation is omitted.

(3) Minimum Composition Sentence Next, an example of extracting a minimum construction sentence from a sentence using morphological analysis and syntax analysis will be described. Here, the “minimum component sentence” is a sentence that has a minimum meaning. For details, see “Invention of similarity between documents used for imitation report determination, Takahisa Ota, Shigeru Masuyama, 10th Annual Meeting of the Language Processing Society” Pp.729-732, 2004 ”, published in the next conference.

  Extracting the minimum sentence from the same text data as in the example of FIG. 3, “Ministry of Meteorological Agency announced.” “Announced on 23rd.” “Announced that the first spring in the Kanto region was blown.” A composition sentence is obtained. By creating dictionary data using these minimum constituent sentences as constituent elements, it is possible to capture the meaning of words in the context, so it is possible to classify sentences containing words that can be captured in multiple meanings into appropriate categories. Classification is possible, but the calculation cost is high.

  Note that, by using the part-of-speech information obtained as a result of the morphological analysis, it is possible to extract the minimum constituent sentence consisting only of the noun, the adjective, and the verb form. Using the same example as above, three minimum composition sentences are obtained: “Meteorological Agency to announce”, “23 days to announce”, “Kanto region, spring first, blow, to announce”.

  In this manner, by extracting the constituent elements using different sentence decomposition algorithms in the sentence decomposition unit 18, dictionary data having different tendencies can be created in the classification dictionary creation unit 22. Therefore, it is possible to improve classification accuracy and processing speed by selecting an appropriate sentence decomposition algorithm according to the category type.

  FIG. 4 is a detailed functional block diagram of the classification dictionary creation unit 22. The classification dictionary creation unit 22 includes a component reception unit 102, a narrowing information reception unit 104, a category information provision unit 106, a component selection unit 108, a dictionary data search unit 110, and a dictionary data update unit 112.

  The component reception unit 102 receives the components rearranged according to a predetermined rule from the sorting unit 20 and passes them to the component selection unit 108. The category information providing unit 106 informs the element narrowing-down criterion providing unit 30 of information about the categories to which the category examples and the non-category example sentences from which the component elements received by the component element receiving unit 102 are extracted belong. The refinement information receiving unit 104 receives the selection criteria from the element refinement criteria providing unit 30 and passes them to the component selection unit 108. The component selection unit 108 compares the selection criterion with the component, selects a component that satisfies the selection criterion, and passes it to the dictionary data search unit 110. The dictionary data search unit 110 searches the dictionary data held in the classification dictionary holding unit 24 for dictionary data for the same component as the component that satisfies the selection criteria, and the corresponding dictionary data is found. If there is, it is passed to the dictionary data update unit 112. The dictionary data updating unit 112 counts the number of each constituent element that satisfies the selection criteria, adds the number to the dictionary data, and stores it in the classification dictionary holding unit 24. When the component is new, new dictionary data is created and stored in the classification dictionary holding unit 24.

FIG. 5 is a data structure diagram of dictionary data stored in the classification dictionary holding unit 24. In the dictionary data 40, the constituent element 42 is associated with the appearance frequency 44 at which the constituent element appears in sentences included in the category sentence examples of category 1 to M and the non-category sentence example. If the component elements are represented as W _n (n = 1 to N) and the frequency of appearance in sentences included in the category sentence example or the non-category sentence example where W _n is the category m (m = 1 to M), respectively, X _nm and Y _nm. The dictionary data 46 for a certain component W _n can be expressed as (W _n , X _n1 , Y _n1 , X _n2 , Y _n2 ,..., X _nm , Y _nm ).

In this embodiment, for each component W _n , two values of (appearance frequency in sentences included in category sentence examples of category m) and (appearance frequency in sentences included in non-category sentence examples of category m) are set. Hold in pairs. This is for facilitating rewriting of the frequency information when new sentence data is added to the category sentence example or the non-category sentence example and the dictionary data in the classification dictionary holding unit 24 is to be expanded.
In another embodiment, the appearance frequency of the component W _n may be held as a single value. When the appearance frequency of the component W _n with respect to the category m is expressed as F _nm , F _nm = X _nm / Y _nm may be set, or F _nm = X _nm / (X _nm + Y _nm ) may be set. In this case, the dictionary data 46 for a certain component W _n can be expressed as (W _n , F _n1 , F _n2 ,..., F _nM ).

  FIG. 6 is a data structure diagram of the selection criteria stored in the element narrowing criteria providing unit 30. Selection criteria are prepared corresponding to the category type. In FIG. 6, “theme classification”, “style”, and “age” are included as category types. The element narrowing criteria providing unit 30 receives the category sentence information from which the constituent elements are extracted from the category information providing unit 106, and returns the selection criteria in the right column 58 in the case of a specific category shown in the left column 56 in the figure. . If the category received from the category information providing unit 106 does not exist in the left column 56, the standard “noun” selection criterion is returned.

  For example, if the category type is related to a theme or topic classification, for example, a category such as “travel”, “music”, or “movie”, the criterion “noun” is provided. This is because, for such themes and topic classifications, the presence of a specific noun often determines the category classification. When the category type is related to a style, for example, a category such as “formal”, “careful”, or “random”, a criterion of “adjective or particle” is provided. This is because the style can often be determined by a particle such as “Tenanoha” or emotional expression. Furthermore, for the age and gender of the person who created the text, for example “female” and “young people”, a criterion “noun of hiragana” is provided. In this way, the element refinement criterion providing unit 30 selects different criteria for each category depending on which category sentence example or non-category sentence example includes the constituent elements for which dictionary data is to be created. Can be provided. The classification dictionary creation unit 22 can create dictionary data that narrows down the components prepared as a dictionary with reference to the selection criteria.

  The element narrowing criteria providing unit 30 may provide the number of characters as a selection criterion instead of providing the part of speech of the component as a selection criterion. As a result, the classification dictionary creation unit 22 can create a dictionary for components having a certain number of characters or less. Alternatively, the element refinement criterion providing unit 30 may provide a specific component (for example, a noun “automobile”) as a selection criterion. The classification dictionary creation unit 22 may exclude components that match the classification dictionary from the dictionary data creation target. For example, it is preferable to exclude common nouns (for example, “I” and “thing”) used in a very large number of sentences because they have little influence on the category classification.

  FIG. 7 is a detailed functional block diagram of the category attribution probability calculator 26. The category attribution probability calculation unit 26 includes a component reception unit 122, a narrowing information reception unit 124, a component selection unit 126, a dictionary data search unit 128, an appearance probability calculation unit 130, and an attribution probability calculation unit 132.

  The component receiving unit 122 receives the components rearranged from the sorting unit 20 according to a predetermined rule. The refinement information receiving unit 124 receives the selection criterion from the element refinement criterion providing unit 30 and passes it to the component selection unit 126. The component selection unit 126 compares the selection criterion with the component, selects a component that satisfies the selection criterion, and passes it to the dictionary data search unit 128. The dictionary data search unit 128 searches the dictionary data held in the classification dictionary holding unit 24 for dictionary data for the same component as the component that satisfies the selection criteria, and the corresponding dictionary data is found. If there is, it is passed to the appearance probability calculation unit 130.

The appearance probability calculation unit 130 calculates the appearance probability a _nm of each component W _n extracted from the unclassified data for each category m. Here, the appearance probability a _nm is calculated by the following equation using the appearance frequencies X _nm and Y _nm in the sentence included in the above-described category sentence example or non-category sentence example.

FIG. 8 shows the appearance probabilities a _{11 to} a _N1 of the components W _{1 to} W _N with respect to the category 1 calculated by Equation ₁ .
Note that the calculation of the appearance probability is not limited to Equation 1. For example, the above-described F _nm may be used as it is.

The attribution probability calculation unit 132 adds up the calculated appearance probabilities for all the components, and calculates the attribution probability for each category for the unclassified sentence data. Preferably, belonging probability calculation unit 132 uses the Bayesian filter method to calculate a membership probability E _n to category n by the following equation.

  Note that the attribution probability may be calculated using a method other than the Bayesian filter method. For example, the attribution probability may be calculated by simply multiplying the appearance probabilities of all the constituent elements, or the average value of the appearance probabilities may be used as the attribution probability.

FIG. 9 is a flowchart showing a process of creating a classification dictionary.
First, the sentence decomposition unit 18 acquires a pair of category sentence examples or non-category sentence examples from the sentence example storage unit 12 (S10). Next, the sentence decomposition unit 18 decomposes the sentence data in the category sentence example and the non-category sentence example into constituent elements based on a predetermined sentence decomposition algorithm, and the sorting part 20 separates the decomposed constituent elements according to a predetermined rule. Rearrange (S12). The execution of the rearrangement is not essential for the present embodiment, and the data search time from the classification dictionary holding unit becomes long, so that the calculation speed can be reduced, but the accuracy of category classification is not affected.

  Next, the classification dictionary creation unit 22 compares the extracted component with the selection criterion received from the element refinement criterion providing unit 30 to determine whether the component is a component for which dictionary data is to be created. (S14). If it is not a dictionary data creation target (N in S14), the process proceeds to S24. If it is a dictionary data creation target (Y in S14), the classification dictionary creation unit 22 searches the classification dictionary holding unit 24 for dictionary data for the component (S16). If the corresponding dictionary data exists (Y in S18), the number of components existing in the current sentence example data is added to the frequency of the category in the dictionary data (S20). If there is no corresponding dictionary data (N in S18), new dictionary data is created (S22). Then, it is determined whether or not all the components decomposed by the text decomposition unit 18 have been processed (S24). If the processing has not ended (N in S24), the processing from S14 is performed on another component. repeat. When the creation of dictionary data for all the constituent elements is finished (Y in S24), this flow is finished.

FIG. 10 is a flowchart showing a process of classifying uncategorized text data into categories.
The text receiving unit 36 receives unclassified text data (S30). The sentence decomposition unit 18 decomposes the sentences in the uncategorized sentence data into constituent elements, preferably based on the same sentence decomposition algorithm as S12 in FIG. 9, and the sorting unit 20 converts the decomposed constituent elements according to a predetermined rule. Rearrange (S32). Next, the category attribution probability calculating unit 6 compares the extracted one component with the selection criterion received from the element narrowing criterion providing unit 30 to calculate the attribution probability of uncategorized sentence data. It is determined whether or not the appearance probability of the component is to be calculated (S34). If it is an appearance probability calculation target (Y in S34), the category attribution probability calculation unit 26 searches the classification dictionary holding unit 24 for dictionary data for the component, and if there is corresponding dictionary data (in S36). Y) Based on the dictionary data, the appearance probability of the component is calculated for each category (S38). If the component is not the target of appearance probability calculation in S34 (N in S34), or if the corresponding dictionary data does not exist in S36 (N in S36), S38 is skipped.
Subsequently, it is determined whether or not all the components extracted by the text decomposition unit 18 have been processed (S40). If the processing has not ended (N in S40), the processing from S34 is performed on another component. repeat.

  When the processing for all the constituent elements is completed (Y in S40), the category attribution probability calculation unit 26 calculates the attribution probability of uncategorized sentence data for each category according to the above-described procedure (S42), and the determination unit 28 Determines the category to which the uncategorized text data belongs based on the attribution probability (S44).

(Example)
Hereinafter, based on a specific Example, operation | movement of the text classification apparatus 10 which concerns on this Embodiment is demonstrated. In this embodiment, it is assumed that two categories of “gambling” and “education” are prepared as categories in order to simplify the explanation. The dictionary data that has already been created is used.

  FIG. 11 shows dictionary data used in this embodiment, and corresponds to FIG. 5 in the entire description above. This dictionary data is included in the category sentence example for the category “gambling”, the constituent elements extracted from the 30 sentences included in the category sentence example and the 15 sentences included in the non-category sentence example, and the category “education”. And the components extracted from the 18 sentences included in the non-category sentence examples. The number of sentences used is shown in the column 154 as “number of sentences”.

  As shown in the figure, this dictionary includes words such as “Pachinko”, “Manga ticket”, “Youth”, “Healthy”, and “Nurture” as constituent elements. Each component has appearance frequency information for each category. Taking the component “pachinko” as an example, the appearance frequency in the sentence of the category sentence example is 10 times and the appearance frequency in the sentence of the non-category sentence example is 2 times for the category “gambling”. For the category “education”, the frequency of appearance in the sentence of the category sentence example is once, and the frequency of appearance in the sentence of the non-category sentence example is 20 times. The same applies to other components.

  The “total” column 152 is a number obtained by adding the number of sentences in the category sentence example and the number of sentences in the non-category sentence example for all the constituent elements.

  On the assumption that such dictionary data is held in the classification dictionary holding unit 24, the sentence “Let's nurture the pachinko industry soundly” as unclassified sentence data is classified into any of the two categories. Explain how. A component is extracted from the sentence by the sentence decomposition unit 18. In this embodiment, the sentence decomposing unit 18 decomposes the sentence by morphological analysis, and as a result, ten constituent elements such as “Pachinko / industry / to / health / to / nurture / do / make / to /.” Is extracted. Subsequently, the component element selection unit 126 in the category attribution probability calculation unit 26 receives the selection criterion “noun” from the element narrowing criterion provision unit 30 and selects only the noun from the extracted component elements. Therefore, four components of “pachinko”, “industry”, “sound” and “nurturing” are selected.

  The dictionary data retrieval unit 128 retrieves corresponding dictionary data from the classification dictionary holding unit 24. In this case, three dictionary data of “pachinko”, “sound” and “nurturing” are obtained. The appearance probability calculation unit 130 calculates the appearance probability for each category using the above equation 1 from the appearance frequencies of the three components in the dictionary data. The result is shown in FIG. Taking the component “Pachinko” as an example, the appearance probability for the category “gambling” is 0.714, and the appearance probability for the category “education” is 0.043. The same applies to other components.

The attribution probability calculation unit 132 calculates the attribution probability of the uncategorized sentence data for each category using the appearance probability according to the above formula 2.
This will be described using specific numerical values. 13, corresponding to Figure 8 of the entire description, is a table summarizing the probability a _n for the category "gambling", "Education" (1-a _n). For the category “gambling”, the appearance probability a ₁₁ of the element “pachinko” is 0.714, the appearance probability a ₂₁ of the element “sound” is 0.200, and the appearance probability a ₃₁ of the element “nurturing” is 0.273. Therefore, when calculated according to Equation 2, the attribution probability E ₁ of the uncategorized sentence data to the category “gambling” is calculated as follows.
(Equation 3)
E ₁ = (0.714 × 0.200 × 0.273) /
{(0.714 × 0.200 × 0.273) + (1−0.714) × (1−0.200) × (1−0.273)} ≈0.190

Category for the "education", the occurrence probability of _{a 12} element "pachinko" is 0.043, the occurrence probability of _{a 22} element "healthy" is 0.783, the occurrence probability of _{a 32} element "training" is 0.844 Therefore, when calculated according to Equation ₂ , the belonging probability E ₂ of the uncategorized sentence data to the category “education” is calculated as follows.
(Equation 4)
E ₂ = (0.043 × 0.783 × 0.844) /
{(0.043 × 0.783 × 0.844) + (1−0.043) × (1−0.783) × (1−0.844)} ≈0.467

  As a result, the determination unit 28 determines that the sentence “Let's nurture the pachinko industry soundly” is classified into the category “education” having the higher attribution probability. Thus, a series of processes for classifying unclassified sentence data is completed.

  As described above, according to the sentence classification device of the present embodiment, uncategorized sentence data can be automatically classified according to a predetermined category.

  Incidentally, a spam filter using a Bayesian filter method has been conventionally known. This spam filter calculates the probability of spam when each word is included from a collection of sentence examples that belong to spam and a collection of sentence examples that does not belong to spam. Detect spam emails by determining spam probability according to: However, this method can only determine whether a certain sentence belongs to a single category, that is, in this case, a category of “spam mail”.

  On the other hand, in the sentence classification device according to the present embodiment, to which category the unclassified sentence data belongs is determined by the belonging probability of the unclassified sentence data calculated for each category. Therefore, one sentence data can be classified into one category, and can be classified into two or more categories. In addition to classifying categories based on one viewpoint, categories based on a plurality of viewpoints can be mixed, and the belonging probability of unclassified sentence data can be calculated collectively. More specifically, classification of content types (for example, politics / economy / society) and classification such as sentence types (news articles / blogs / essays) can be executed simultaneously with a single calculation. . Therefore, it is possible to classify sentences from a multiaxial viewpoint such as “political news articles” and “essays on social issues”.

  The sentence classification device of the present embodiment is assumed to be applied as described below.

Application form 1.
When a directory-type search site is created, the web page HTML file collected by the web crawler is given to the sentence classification device, so that the web page can be classified into categories based on various topics. By using this classification result, the construction of a directory-type search site can be facilitated. In this application mode, the classification dictionary creation unit may weight the appearance frequency according to whether the text data is in the header, tag, or body of the HTML file or XML file.

Application form 2.
In the electronic bulletin board system, post data transmitted from a contributor to a server connected via a network is given to a text decomposing apparatus, whereby the post data can be classified into categories based on contents. As a result, the posted data can be automatically classified and displayed without human intervention. In addition, the contributor can also transmit the posting data to the electronic bulletin board system without selecting the posting destination.

Application form 3.
By preparing spam mail data as category sentence examples and other mail data as non-category sentence examples, the sentence classification device can also be used as a spam mail detection filter.

  The present invention has been described based on some embodiments. It is understood by those skilled in the art that these embodiments are exemplifications, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. By the way.

  It should also be understood by those skilled in the art that the functions to be fulfilled by the constituent elements described in the claims are realized by the individual functional blocks shown in the present embodiment or their linkage.

  FIG. 14 shows a configuration of a sentence classification system according to another embodiment. In this embodiment, a sentence classification system 100 is constructed from a dictionary unit 60 that creates a dictionary from category sentence examples and non-category sentence examples, and a classification unit 70 that classifies unclassified sentence data into categories.

  In the dictionary unit 60, the sentence example storage unit 12, the sentence decomposition unit 18, the sorting unit 20, the classification dictionary creation unit 22, the classification dictionary holding unit 24, and the element refinement criterion providing unit 30 have the same functions as those described with reference to FIG. Have When the dictionary providing unit 62 is requested to provide dictionary data from an external device such as the classification unit 70, the dictionary providing unit 62 searches the dictionary dictionary corresponding to the component from the classification dictionary holding unit 24 and transmits the dictionary data to the external device.

In addition, the sentence reception unit 36, the category attribution probability calculation unit 26, the determination unit 28, and the determination result storage unit 32 in the classification unit 70 also have the same functions as those described with reference to FIG. The sentence decomposing unit 76 and the sorting unit 78 have the same functions as the sentence decomposing unit 18 and the sorting unit 20, respectively. The category attribution probability calculation unit 26 requests the dictionary unit 60 to provide dictionary data for the components extracted from the uncategorized sentence data 34, receives the dictionary data transmitted from the dictionary providing unit 62, and described above. A series of processing is executed. The category attribution probability calculation unit 26 may narrow down elements by receiving provision of narrowing information from the element narrowing unit 74.
In this way, by configuring the dictionary unit and the classification unit separately, each unit can be remotely arranged via the network.

  In another embodiment, data obtained when classifying unclassified sentence data may be reflected in the dictionary data in the classification dictionary holding unit 24. Specifically, when the dictionary data corresponding to the component extracted from the uncategorized sentence data exists in the classified dictionary holding unit 24, the frequency information of the dictionary data is updated. In this way, the dictionary data can be enriched each time classification of uncategorized text data is repeated.

  When recording the appearance frequency for each category as dictionary data, the classification dictionary creation unit 22 may weight the constituent elements by part of speech. For example, if the constituent element is a noun, the frequency may be doubled, and if the constituent element is a particle, the frequency may be recorded by 0.1 times. Further, the category attribution probability calculation unit 26 may calculate the attribution probability of the unclassified sentence data by weighting the appearance probability according to the part of speech of the component. This makes it possible to categorize sentence data that reflects the importance of the influence of the part of speech of the constituent element.

It is a figure which shows an example of the usage condition of the text classification device which concerns on this Embodiment. It is a functional block diagram of a text classification device. It is a figure which shows the example which decomposed | disassembled text data into the component by morphological analysis. It is a detailed functional block diagram of a classification dictionary creation unit. It is a data structure figure of the dictionary data stored in the classification dictionary holding part. It is a structural diagram of the narrowing-down criterion data stored in the element narrowing-down criterion providing unit. It is a detailed functional block diagram of a category attribution probability calculation unit. It is a figure which shows the appearance probability of the component about one category. It is a flowchart which shows the process in which a classification dictionary is produced. It is a flowchart which shows the process in which uncategorized text data is classified into a category. It is a figure which shows the dictionary data used in one Example. It is a figure which shows the result of having calculated the appearance probability for every component using the dictionary data of FIG. It is a figure which shows the result of having calculated the belonging probability about each category using the appearance probability of FIG. It is a block diagram of the text classification system which concerns on another embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 sentence classification device, 12 sentence example storage part, 14 category sentence example data group, 16 non-category sentence example data group, 18 sentence decomposition part, 20 sort part, 22 classification dictionary creation part, 24 classification dictionary holding part, 26 category attribution probability calculation part , 28 determination unit, 30 element narrowing reference providing unit, 32 determination result storage unit, 34 uncategorized text data, 36 text receiving unit, 62 dictionary providing unit, 102 component receiving unit, 104 narrowing information receiving unit, 106 category information providing , 108 component selection unit, 110 dictionary data search unit, 112 dictionary data update unit, 122 component element reception unit, 124 refinement information reception unit, 126 component element selection unit, 128 dictionary data search unit, 130 appearance probability calculation unit, 132 Attribution probability calculation unit.

Claims (11)

A classification dictionary holding unit that holds dictionary data in which the constituent elements of the sentence and the appearance frequency of the constituent elements appearing in the sentences to be classified into the categories for a plurality of predetermined categories are associated;
A text receiving unit that receives unclassified text data to be newly classified from an external device;
A sentence decomposing unit that analyzes the unclassified sentence data according to a predetermined rule and extracts constituent elements of the sentence;
With reference to the dictionary data, a category attribution probability calculator for calculating an attribution probability representing the probability that the uncategorized sentence data including the extracted component belongs to each category,
A determination unit that determines a category in which the unclassified sentence data is classified with reference to the probability of belonging;
A sentence classification device comprising:   The category attribution probability calculation unit calculates the appearance probability by taking out the appearance frequency in the category for which the attribution probability is calculated for each component extracted from the unclassified sentence data from the classification dictionary holding unit, The sentence classification device according to claim 1, wherein the probability of belonging to the category is obtained by synthesizing the appearance probabilities of the constituent elements.   3. The sentence classification according to claim 1, wherein the classification dictionary holding unit holds dictionary data in which words and parts of speech obtained as a result of morphological analysis on the sentence data and appearance frequencies in each category are associated with each other. apparatus.   4. The sentence according to claim 1, wherein the classification dictionary holding unit holds the dictionary data in a state in which the elements are rearranged according to a predetermined rule that facilitates searching. Classification device. An element refining criterion providing unit storing a criterion for selecting a component;
The category attribution probability calculating unit receives the criterion from the element narrowing criterion providing unit, and among the components extracted from the uncategorized sentence data, the component satisfying the criterion is the calculation target of the appearance probability. The classification dictionary creation device according to claim 3.   6. The sentence classification apparatus according to claim 5, wherein the element narrowing-down criterion providing unit has a plurality of the criteria and gives different criteria to the category attribution probability calculating unit according to the type of category.   The sentence classification apparatus according to claim 5 or 6, wherein the category attribution probability calculation unit calculates the appearance probability after weighting the appearance frequency according to a part of speech of a corresponding component. About a plurality of predetermined categories, a sentence example storage unit that stores a category sentence example data group including sentence data to be classified into each category, and a non-category sentence example data group including sentence data not classified into a category,
A sentence decomposition unit that analyzes sentence data according to a predetermined rule and extracts constituent elements of the sentence;
A classification dictionary creation unit that counts, for each category, the frequency of appearance of components extracted by the sentence decomposition unit in the sentence data included in the category sentence example data group and the non-category example data group;
A classification dictionary holding unit that holds dictionary data in which a component and an appearance frequency in each category are associated;
A dictionary providing unit for providing the dictionary data to an external device;
A classification dictionary creation device comprising: An element refining criterion providing unit storing a criterion for selecting a component;
The classification dictionary creating unit receives the criteria from the element narrowing criteria providing unit, and among the components extracted from the sentence data included in the category sentence example data group and the non-category sentence example data group, the constituent elements satisfying the criterion The classification dictionary creating apparatus according to claim 8, wherein the dictionary data is to be created.   10. The classification dictionary creating apparatus according to claim 9, wherein the element narrowing-down criterion providing unit has a plurality of the criteria and gives different criteria to the category attribution probability calculating unit according to the type of category. For a plurality of predetermined categories, a category sentence data group including sentence data to be classified into each category and a non-category sentence example data group including sentence data not classified into a category are stored in the storage means.
Analyzing the sentence data according to the prescribed rules and extracting the constituent elements of the sentence,
Calculating the frequency of appearance of each component extracted in the sentence data included in the category sentence example data group and the non-category example data group for each category;
Storing dictionary data in which the frequency and the component are associated;
Newly received uncategorized text data to be classified from an external device,
For each component extracted from the uncategorized sentence data, refer to the dictionary data and calculate for each category an attribution probability representing the probability that the uncategorized sentence data including the extracted component belongs to each category. ,
A sentence classification method, wherein the category into which the unclassified sentence data is classified is determined with reference to the probability of belonging.

Images