JP2010055253A - Unnecessary word deciding apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To decide an unnecessary word without using a threshold for deciding whether or not a word is an unnecessary word. <P>SOLUTION: By a parameter learning section 14, appearance probability of each topic of each word in a word group included in learning document data, which maximizes likelihood for the learning document data is learned and searched. By a word classification section 16, each word in the word group is classified by a topic having highest appearance probability. By an unnecessary word decision section 20, a word group of a topic, by which words with appearance probabilities for every topic respectively falling within a predetermined range and distributed uniformly are classified, is decided as an unnecessary word. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an unnecessary word determination device and a program.

  In document retrieval and document clustering, word indexing is indispensable. If a word that cannot be a keyword of a document is registered in the index when creating this index, the efficiency in document search and the accuracy in clustering are adversely affected. Therefore, it is necessary to appropriately extract and remove words called unnecessary words that cannot be used as keywords for documents, and various techniques have been proposed.

For example, a technique is known that uses document data classified for each field and removes words that appear in all documents in the same field (Patent Document 1). In a similar document search device, a technique is known that counts the frequency of appearance of words and registers words that appear above (or below) a certain ratio as unnecessary words (Patent Document 2). In addition, as a technique for effectively removing unnecessary words in automatic document classification, words are extracted from a learning document set, and unnecessary words in that category are relatively removed based on the appearance frequency and appearance ratio in each category. A technique is known (Patent Document 3).
JP 7-78182 A Japanese Patent Laid-Open No. 11-259515 JP 2004-139222 A

  An object of the present invention is to provide an unnecessary word determination device and a program capable of determining an unnecessary word without using a threshold value for determining whether or not the word is an unnecessary word.

  In order to achieve the above object, the unnecessary word determination device according to claim 1 maximizes the likelihood for the document information for the word group included in the document information prepared in advance. Search means for searching for an appearance probability that each word of the word appears in each of a predetermined number of themes, and each word of the word group based on the appearance probability searched by the search means And classifying means for classifying the subject number into the themes with the highest number of themes, and each of the appearance probabilities of appearing in the themes with the number of themes is within a predetermined range. Unnecessary words determining means for determining each of the words classified as the subject in which the word is classified as an unnecessary word.

  The unnecessary word determination device according to the second aspect of the present invention provides a subject in which each word of the word group is set in advance for maximizing the likelihood of the document information for a word group included in document information prepared in advance. Search means for searching for the appearance probability of appearing in each of a number of themes, and, based on the appearance probability searched by the search means, for each word of the word group, the appearance probability of the word is highest Classifying means for classifying into any of the themes of the number of themes, and among the themes of the number of themes, the themes in which each of the occurrence probabilities appearing in the themes of the number of themes falls within a predetermined range are classified Unnecessary word candidate determination means for determining each of the classified words as an unnecessary word candidate, and a display device for the user to select the unnecessary word candidates determined by the unnecessary word candidate determination means for selecting an unnecessary word Displayed on It is configured to include a display control means for controlling.

  According to a third aspect of the present invention, in the unnecessary word determining apparatus according to the first or second aspect of the present invention, the search means determines that each of the appearance probabilities appearing in the themes corresponding to the number of themes is within a predetermined range. The number of themes is sequentially increased from the preset number of themes until the subject in which the word is classified appears, and the appearance probability that maximizes the likelihood for the document information is repeatedly searched.

  According to a fourth aspect of the present invention, there is provided a program that causes a computer to maximize the likelihood for the document information for a word group included in document information prepared in advance. Search means for searching for an appearance probability appearing in each of the themes, and the subject having the highest occurrence probability of the word for each word of the word group based on the appearance probability searched by the search means Classifying means for classifying into any of a number of themes, and among the themes of the number of themes, classifying the words into which the occurrence probabilities appearing in the themes of the number of themes are within a predetermined range. This is a program for causing each of the generated words to function as unnecessary word determining means for determining as an unnecessary word.

  According to a fifth aspect of the present invention, there is provided a program that causes a computer to maximize the likelihood for the document information for a word group included in document information prepared in advance. Search means for searching for an appearance probability appearing in each of the themes, and the subject having the highest occurrence probability of the word for each word of the word group based on the appearance probability searched by the search means Classifying means for classifying into one of a number of themes, among the themes of the number of themes, words whose occurrence probabilities appearing in the themes of the number of themes are within a predetermined range are classified into the classified themes Unnecessary word candidate determining means for determining each of the words as unnecessary word candidates, and displaying the unnecessary word candidates determined by the unnecessary word candidate determining means for the user to select the unnecessary words Is a program for functioning as display control means for displaying on the location.

  As described above, according to the unnecessary word determination device according to claim 1, there is an effect that an unnecessary word can be determined without using a threshold value for determining whether or not the word is an unnecessary word. It is done.

  According to the unnecessary word determining device of the second aspect, there is an effect that an unnecessary word can be determined without using a threshold value for determining whether or not the word is an unnecessary word.

  According to the unnecessary word determining device of the third aspect, it is possible to determine that many unnecessary words can be determined as compared with the case where the present configuration is not provided.

  According to the program of the fourth aspect, there is an effect that an unnecessary word can be determined without using a threshold for determining whether or not the word is an unnecessary word.

  According to the program of the fifth aspect, it is possible to determine an unnecessary word without using a threshold value for determining whether or not the word is an unnecessary word.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The unnecessary word determination device 10 according to the first embodiment of the present invention is configured by a conventionally known personal computer, and includes an unnecessary word determination processing routine, which will be described later, with a keyboard, mouse, display, CPU, ROM, RAM, and the like. It is comprised with memory | storage devices, such as HDD which memorize | stored various data, such as the program for implementing this, and document data.

  When the unnecessary word determination device 10 is represented by functional blocks according to an unnecessary word determination processing routine described below, as shown in FIG. 1, a document database 12 storing learning document data prepared in advance, and a learning document data A parameter α for adjusting the appearance frequency of each of a plurality of subjects (hereinafter also referred to as a topic), and a parameter β representing an appearance probability of each word in the word group included in the learning document data. Based on the learned parameter learning unit 14, the learned parameter β, the word classifying unit 16 that classifies each word into a topic having the highest appearance probability, and the topic in which words uniformly distributed in each topic are classified A uniform distribution determining unit 18 that determines whether or not a word is present, and each word classified as a topic in which uniformly distributed words are classified. And a unnecessary word determination portion 20 that determines a word.

  Hereinafter, the principle of determining unnecessary words in the present embodiment will be described.

  In recent years, a lot of statistical document processing methods have been studied and put into practical use, and a probabilistic document model has been proposed and studied. As a model for expressing a plurality of topics appearing in one document, Probabilistic Latent Semantic Analysis has been proposed (Non-Patent Document 1: T. Hofmann, “Probabilistic Latent Semantic Analysis”, Proceedings of Conference on Uncertainty in Artificial Intelligence, pp. 289-296. (1999)).

  In addition, as a model obtained by generalizing the above model from the framework of Bayesian statistics, a technique called “Lent Dirichlet Allocation” (hereinafter referred to as LDA) has been proposed (Non-Patent Document 2: DM Blei, AY Ng, and MI Jordan, “Latent Dirichlet Allocation”, Journal of Machine Learning Research, vol. 3, pp.993-1022. (2003).).

  The probabilistic document generation model of LDA is represented by an acyclic directed graph as shown in FIG. Here, M is the number of documents, N is the number of words per document, Z is a topic variable for each word and a latent variable, θ is the appearance probability of the topic, and represents a parameter of the Dirichlet distribution. Further, α is a parameter for adjusting the appearance frequency of the topic, and represents a hyper parameter of the Dirichlet distribution, and β represents the appearance probability of the word on the condition of the topic. The Dirichlet distribution is a probability distribution function often used to represent the prior probability distribution or posterior probability distribution of discrete data in Bayesian statistics (Non-Patent Document 3: A. Gelman et al., “Bayesian Data Analysis”, Chapman & Hall / CRC, second edition (2004).).

  In the present embodiment, unnecessary words are determined using LDA, which is a representative technique among techniques called latent topic models. Here, LDA will be briefly described.

  The LDA proposed by Blei et al. In Non-Patent Document 2 is a framework for handling discrete data generated based on a central concept called a topic. Specifically, the LDA is a model in which a document generation process is modeled and described by Bayesian statistics. For document data generated from N words, a topic is assumed for each word, the predetermined number of topics is K, model parameters estimated from the data are α, β, By assuming a so-called “bag-of-words” model where the appearance probability is θ, the set of K topics is z, and the set of N words is w, these connection probabilities are expressed by the following equation (1): It expresses by.

  The marginalization probability distribution of the document data is expressed by the following equation (2).

  The generation probability of the document data D in the entire corpus is expressed by the following equation (3).

  Also, the random variable θ according to the K-dimensional Dirichlet distribution is expressed as the following equation (4).

 From the above formula (4), the above formula (2) is expressed by the following formula (5). Further, p (w | α, β) calculated by the equation (5) represents the likelihood for the target document data.

In the above equations (1) to (5), i is a variable for identifying a topic, n is a variable for identifying a word in the document data, and j is a word in the dictionary. It is a variable to do. Also, w _n ^j is 1 when the nth word matches the jth word in the dictionary, and 0 otherwise. V is the number of words indexed in the dictionary. In addition, the likelihood for the target document data calculated by the above equation (5) is how likely the document data generated stochastically based on the parameters α and β is for the target document data. It is a degree which represents.

  In the above equation (5), since θ and β are coupled, it cannot be easily integrated. Therefore, some approximate calculation methods are used, for example, Markov chain Monte Carlo method (Non-patent document 4: Yukito Iba, "Frontier of statistical science 12-Computational statistics II-Markov chain Monte Carlo method and its surroundings"), Iwanami Approximate calculation is performed using the variational Bayes method described in the bookstore, 2005) and the nonpatent literature 2 mentioned above.

  In this embodiment, a variational Bayes method, which is one of efficient calculation methods, is used. The variational Bayes method is a method of variationally approximating the logarithmic likelihood of the entire document data set using α and β as parameters. This will be described later as part of the parameter learning method.

  In the parameter learning unit 14 according to the present embodiment, the number of topics is set and the LDA model is learned without a teacher by using the experience Bayes method described in Non-Patent Document 2. For example, the combination of the parameters α and β that maximizes the likelihood for the learning document data is searched while changing the parameters α and β of the LDA model for the set number of topics to a plurality of types of values.

  Here, α is a parameter for adjusting the appearance frequency of each topic. Β is a parameter representing the appearance probability for each topic of K words registered in the dictionary that can be generated from the learning document data, and the size of the dictionary is V and a K × V matrix It is represented by

  In this embodiment, as described in Non-Patent Document 2, parameter learning combining the variational Bayes method and the experience Bayes method is performed. An algorithm based on this combination is also called a variational EM algorithm, and has the following procedure.

  First, variational approximation of marginal likelihood is performed and variation parameters are introduced. Next, the variation parameter is optimized for each document. Then, the parameters α and β are optimized so as to maximize the lower bound of the marginal likelihood subjected to variational approximation.

  The lower bound in the above algorithm refers to the right side of the following expression (6), which is the following Jensen inequality regarding arbitrary functions p and q.

  In the above equation (6), D, Z, and θ represent variables. Then, the left side of Equation (6) is set as the peripheral log likelihood, and the lower bound on the right side is maximized. A technique for optimizing Bayesian statistical parameters by maximizing this likelihood is called an experience Bayesian method. It is a method called variational approximation that assumes that the function q introduced here can be expressed in a variable separation form as shown in the following equation (7).

  Here, γ and θ are newly introduced auxiliary parameters called variational parameters described above. Note that more detailed description of these methods is described in Non-Patent Document 2 above, and is therefore omitted here.

  If β that maximizes the likelihood for the learning document data is obtained by the parameter learning unit 14, a word having a high appearance probability can be determined for each topic. Therefore, the word classification unit 16 registers the word in the dictionary. Each word is classified into a topic having the highest appearance probability.

  In the above LDA model, when the number of topics to be set is increased, a topic in which a set of words that do not react strongly to a specific topic is classified may be formed. A feature of a word set that is strongly associated with such a topic is that the appearance probabilities on other (partial) topics are almost equal, and such a set of words becomes an unnecessary word.

Therefore, in this embodiment, the uniform distribution determination unit 18, among all the topics for each of all the words w _n classified, appearance probability to all topics _{T i p (w n | T} i ) Determines whether there is a topic that falls within the range represented by the following equation (8).
(1-δ) / K ≦ p (w _n | T _i ) ≦ (1 + δ) / K (i = 1,..., K) (8)
Here, K represents the number of topics, and δ is a constant for defining the range of appearance probabilities. δ may be set by the user for each learning document data. Further, the value of δ may be determined from an index (for example, 1%, 3%, 5%) indicating significance that is usually used in statistical tests.

  If there is no topic with a word set that has a uniform distribution for all topics, a topic with a word set that has a uniform distribution for all topics will appear. Until the number of topics is increased, the parameter learning unit 14 repeatedly learns.

  When there is a topic in which a set of words having a uniform distribution with respect to all the topics as described above exists, the words having a uniform distribution classified into the topic by the unnecessary word determination unit 20 The set is determined as an unnecessary word.

  Next, the operation of the unnecessary word determination device 10 according to the first embodiment will be described.

  First, in the unnecessary word determination device 10, the learning document data is read from the document database 12, the morphological analysis is performed on the learning document data, the noun word is extracted, and the word dictionary is used as a word group included in the learning document data. Register in a database (not shown).

  And the unnecessary word determination apparatus 10 performs the unnecessary word determination processing routine shown in FIG. In step 100, a word group included in the learning document data is read from the word dictionary database. In step 102, 2 is set as the initial value for the topic number K, and a predetermined upper limit value is set for the topic number upper limit value Kmax.

  Then, in step 104, the parameters α and β that obtain the maximum likelihood for the learning document data are searched by learning the word group obtained in step 100 in the topic number K. In the next step 106, each word obtained in step 100 is classified into K topics based on the appearance probability represented by the parameter β obtained in step 106.

  In step 108, all words classified in step 106 are uniformly distributed to each topic according to the above equation (8) based on the appearance probability represented by the parameter β obtained in step 106. It is determined whether or not a topic to be present exists. If there is no topic in which all the classified words are uniformly distributed, it is determined in step 110 whether or not the topic number K is less than the upper limit value Kmax. If the value Kmax has not been reached, the number of topics to be set is incremented in step 112, and the process returns to step 104 above.

  On the other hand, if the topic number K has reached the upper limit value Kmax in step 110, it is determined that a topic in which all classified words are uniformly distributed has not been obtained, and unnecessary words are not registered. Then, the unnecessary word determination processing routine is terminated.

  If it is determined in step 108 that a topic in which all classified words are uniformly distributed has been obtained, a set of words classified in the topic is converted into an unnecessary word list in step 114. Register and complete the unnecessary word determination processing routine.

  Next, the learning result of the appearance probability by the method using the LDA model of the present embodiment will be described. The clustered patent document set was used as a learning corpus, morphological analysis was performed, noun words were extracted as word groups, and 2 or 3 were set as the number of topics to perform LDA learning.

  When the number of topics is set to 2, LDA learning is performed, and the appearance probabilities of the obtained words for each topic are sorted by using the appearance probability of one of the topics as a condition, as shown in FIG. It can be seen that the word sets classified into (2) have different ratios of appearance probabilities between topics in the order of powers of 10. Further, as shown in FIG. 5, it can be seen that the word sets classified into the topic 2 have different ratios of appearance probabilities between topics in the order of the power of 10. That is, in this case, it is understood that topics 1 and 2 are meaningful topic groups that make a difference in the contents of different documents.

  Further, when the number of topics is set to 3 and LDA learning is performed again, as shown in FIG. 6, the word set classified as topic 2 has a relatively small ratio of appearance probabilities between topics, that is, It can be seen that the word set is not biased toward one specific topic. The word set of topic 2 into which this word set is classified is registered as an unnecessary word list. If you look at the words listed at the top, you will find relatively abstract words such as “formation”, “devise”, “work”, “placement”, and such words are registered as unnecessary words. I understand that

  As described above, according to the unnecessary word determination device according to the first embodiment, the words of the topic into which the words that are uniformly distributed within the predetermined range where the appearance probabilities for all topics are within the predetermined range are classified. The set is determined as an unnecessary word. This makes it possible to determine an unnecessary word without using a threshold value for determining whether or not the word is an unnecessary word that is difficult to set appropriately.

  Also, the appearance probability of each word for each topic that maximizes the likelihood of the learning document data is searched by learning, and unnecessary words are determined using the searched appearance probability. Thus, unnecessary words can be determined in consideration of the document characteristics of the learning document data.

  Next, a second embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and the description regarding a structure is abbreviate | omitted.

  The second embodiment is mainly different from the first embodiment in that an unnecessary word candidate list is presented to the user and the user selects an unnecessary word.

  As illustrated in FIG. 7, the unnecessary word determination device 210 according to the second exemplary embodiment includes a document database 12, a parameter learning unit 14, a word classification unit 16, and a uniform distribution determination unit 18. An unnecessary word candidate display control unit 220 that displays each word classified as a topic in which distributed words are classified on a display (not shown) as an unnecessary word candidate list, and an unnecessary word by a user operating a mouse or a keyboard. An unnecessary word selection determining unit 222 that determines a word selected from the candidate list as an unnecessary word.

  The unnecessary word candidate display control unit 220 displays each word of a topic in which a word set having a uniform distribution with respect to all topics is displayed on the display as an unnecessary word candidate list. An appearance probability or a ratio of appearance probabilities between topics is displayed together with an unnecessary word candidate list. Thus, the user selects a word to be registered as an unnecessary word with reference to the appearance probability or the ratio of the appearance probability displayed on the display.

  The unnecessary word selection determination unit 222 determines a set of words selected from the unnecessary word candidate list as unnecessary words by the user as unnecessary words.

  Next, an unnecessary word determination processing routine according to the second embodiment will be described. In addition, about the process similar to 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  First, in step 100, a word group included in the learning document data is read from the word dictionary database, and in step 102, an initial value 2 is set as the topic number K and a topic number upper limit value Kmax is set in advance. Set the upper limit.

  In step 104, the parameters α and β that maximize the likelihood of the learning document data are searched for by learning in the number of topics K. In the next step 106, based on the parameter β obtained in step 106, each word obtained in step 100 is classified into K topics. Then, in step 108, it is determined whether or not there is a topic in which all classified words are uniformly distributed for each topic, and there is a topic in which all classified words are uniformly distributed. If not, it is determined in step 110 whether the topic number K is less than the upper limit value Kmax. If the topic number K has not reached the upper limit value Kmax, the number of topics is determined in step 112. Is incremented and the process returns to step 104. On the other hand, if the number of topics K has reached the upper limit Kmax in step 110, the unnecessary word determination processing routine is terminated.

  If it is determined in step 108 that a topic in which all classified words are uniformly distributed has been obtained, a set of words classified in the topic is converted into an unnecessary word candidate list in step 250. On the display.

  In the next step 252, it is determined whether or not an unnecessary word has been selected by the user from the unnecessary word candidate list, and the user operates the mouse or keyboard to select a word to be an unnecessary word from the unnecessary word candidate list. Then, the process proceeds to step 254, the set of words selected in step 252 is registered in the unnecessary word list, and the unnecessary word determination processing routine is terminated.

  As described above, according to the unnecessary word determination device according to the second embodiment, the words of the topic in which the words that are uniformly distributed within the predetermined range where the appearance probabilities for all the topics are within the predetermined range are classified. The set is determined as an unnecessary word candidate. Thereby, an unnecessary word candidate can be determined without using a threshold value for determining whether or not the word is an unnecessary word that is difficult to set appropriately.

  Next, a third embodiment will be described. In addition, since the unnecessary word determination apparatus which concerns on 3rd Embodiment becomes a structure similar to 1st Embodiment, the same code | symbol is attached | subjected and the description regarding a structure is abbreviate | omitted.

  In the third embodiment, the word set of a topic in which a predetermined number of words having a uniform distribution with respect to all topics are determined as unnecessary words is determined as the first embodiment. Mainly different from the form.

  In the unnecessary word determination device according to the third exemplary embodiment, the uniform distribution determination unit 18 uses the above expression (8) to determine the appearance probability of all the topics in the classified word set. It is determined whether or not there is a topic having a predetermined number or more of words within the range to be represented.

  If there are no topics with more than a predetermined number of words having a uniform distribution for all topics as described above, the number of topics is increased and the parameter learning unit 14 is made to learn again.

  When there are topics classified as more than a predetermined number of words having a uniform distribution with respect to all the topics as described above, the unnecessary word determination unit 20 generates a word set having a uniform distribution. Determine as an unnecessary word.

  In addition, about the other structure and effect | action of an unnecessary word determination apparatus which concern on 3rd Embodiment, since it is the same as that of 1st Embodiment, description is abbreviate | omitted.

  The third embodiment described above may be applied to the second embodiment described above. In this case, a word set of topics classified by a predetermined number of words having a uniform distribution over all topics is determined as an unnecessary word candidate, and the user selects an unnecessary word from the unnecessary word candidates. You can make it.

  In the first to third embodiments described above, the case where the determined unnecessary word is registered has been described as an example. However, the present invention is not limited to this. For example, the unnecessary word is used. May be removed from the corpus.

  Further, similar document search, document clustering, keyword search, and the like may be performed using an unnecessary word list including determined unnecessary words. In this case, using the appearance probability for each topic of each word obtained by LDA learning, the appearance probability for each topic is assigned to each word included in the input new document data, and the assigned topic An average topic of document data and an average topic for each paragraph are obtained by weighting using the appearance probability for each. Then, similar document search, document clustering, keyword search, and the like may be performed using the obtained average topic.

  Further, the program according to the present invention can be provided by being stored in a storage medium such as a CDROM.

It is a block diagram which shows the structure of the unnecessary word determination apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the stochastic document production | generation model of LDA. It is a flowchart which shows the content of the unnecessary word determination processing routine in the unnecessary word determination apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows the appearance probability for every topic of each word obtained by LDA learning when the number of topics is 2. It is a figure which shows the appearance probability for every topic of each word obtained by LDA learning when the number of topics is 2. It is a figure which shows the appearance probability for every topic of each word obtained by LDA learning when the number of topics is three. It is a block diagram which shows the structure of the unnecessary word determination apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the content of the unnecessary word determination processing routine in the unnecessary word determination apparatus which concerns on the 2nd Embodiment of this invention.

Explanation of symbols

10, 210 Unnecessary word determination device 12 Document database 14 Parameter learning unit 16 Word classification unit 18 Uniform distribution determination unit 20 Unnecessary word determination unit 220 Unnecessary word candidate display control unit 222 Unnecessary word selection determination unit

Claims (5)

A search for searching for the probability of occurrence of each word of the word group appearing in each of a predetermined number of subjects for the word group included in the document information prepared in advance, which maximizes the likelihood for the document information. Means,
Classifying means for classifying each word of the word group into any of the themes having the highest appearance probability of the word based on the appearance probability searched by the search means;
Of the themes of the number of themes, it is unnecessary to determine, as unnecessary words, each of the words classified into the themes into which the words whose occurrence probabilities appear in the predetermined number of themes are within a predetermined range Word determination means;
Unnecessary word determination device including A search for searching for the probability of occurrence of each word of the word group appearing in each of a predetermined number of themes, which maximizes the likelihood for the document information for the word group included in the document information prepared in advance Means,
Classifying means for classifying each word of the word group into any of the themes having the highest occurrence probability of the word based on the appearance probability searched by the search means;
Among the themes of the number of themes, each of the words classified into the themes into which the words whose appearance probabilities appearing in the themes of the number of themes are within a predetermined range is determined as unnecessary word candidates. Unnecessary word candidate determination means;
Display control means for causing the user to display the unnecessary word candidates determined by the unnecessary word candidate determining means on the display device for the user to select unnecessary words;
Unnecessary word determination device including   The search means sequentially increases the number of themes from the preset number of themes until a theme in which words whose appearance probabilities appearing in the themes for the number of themes fall within a predetermined range appears. The unnecessary word determination device according to claim 1, wherein the appearance probability that maximizes the likelihood for the document information is repeatedly searched. Computer
A search for searching for the probability of occurrence of each word of the word group appearing in each of a predetermined number of subjects for the word group included in the document information prepared in advance, which maximizes the likelihood for the document information. means,
Classification means for classifying each word of the word group into any of the themes having the highest appearance probability of the word based on the appearance probability searched by the search means, and the number of themes Unnecessary words determining means for determining, as unnecessary words, each of the words classified into the themes in which words whose appearance probabilities appearing in the number of themes within the number of themes are within a predetermined range are classified as unnecessary words Program to function as. Computer
A search for searching for the probability of occurrence of each word of the word group appearing in each of a predetermined number of subjects for the word group included in the document information prepared in advance, which maximizes the likelihood for the document information. means,
Classification means for classifying each word of the word group into any of the themes having the highest appearance probability of the word based on the appearance probability searched by the search means;
Among the themes of the number of themes, each of the words classified into the themes into which the words whose appearance probabilities appearing in the themes of the number of themes are within a predetermined range is determined as unnecessary word candidates. An unnecessary word candidate determining unit, and a program for causing an unnecessary word candidate determined by the unnecessary word candidate determining unit to function as a display control unit that causes a user to display the unnecessary word candidate on a display device in order to select an unnecessary word.

Images