KR20140049659A - An automatic device for training and classifying documents based on n-gram statistics and an automatic method for training and classifying documents based on n-gram statistics therefor - Google Patents

Abstract

The present invention relates to an apparatus for automatically learning documents and a method for automatically learning documents using the same, and an apparatus for automatically classifying documents and a method for automatically classifying documents using the same, which are capable of automatically learning and classifying mass documents on the web through a process of automatically learning and classifying documents based on n-gram. The apparatus for automatically classifying documents according to the present invention includes: a learning document pool including a plurality of learning document groups which are classified according to categories; a preprocessing unit configured to preprocess each of the learning document groups of the learning document pool; and an n-gram data set pool configured to store a set of n-gram data of the learning document pool, which is formed by being learned through the preprocessing of the preprocessing unit. Additionally, the apparatus for automatically classifying documents includes: an automatic document learning unit configured to allow the preprocessing unit to preprocess a corresponding new document to form a bigram set, when the new document occurs, which is not identified through the learning document pool; and an automatic document classifying unit configured to compare the bigram set of the new document, formed through the preprocessing unit, with a bigram set of the n-gram data set pool and to allocate and store the bigram set of the new document to one of n-gram data sets of the n-gram data set pool. [Reference numerals] (220) Automatic document classifying unit; (230) Learned n-gram data set(bigram example); (AA) Non-identified document; (BB) Appearance of a new document; (CC) Preprocessing

Description

Technical Field [0001] The present invention relates to an automatic learning apparatus for a document, an automatic document learning method using the same, a document automatic classification apparatus, and an automatic document classification method using the automatic classification apparatus. on N-gram statistics therefor}

The present invention relates to a document automatic learning apparatus, an automatic document learning method and a document automatic classification apparatus using the same, and a document automatic classification method using the automatic document learning apparatus and method. And a method for automatically classifying a document using the automatic classifying device and a method for automatically classifying a document using the automatic classifying device.

Document classification is a field that has been started to be developed by the library management system in the past and is being continuously researched and developed so as to efficiently manage and index large documents, Respectively.

Especially, due to the rapid development of the Internet and smartphone, the amount of information on the web has increased in a large amount, and therefore, it is very important for the users to efficiently manage and search the information on the web.

For example, an information retrieval system that processes countless web document information on a web site in recent years, extracts only information corresponding to a user's request, and provides the extracted information to the user is widely used. In addition, An Internet information retrieval system capable of searching the Internet is also used. In order to improve the functionality of such an information retrieval system, development of the document classification technology as described above is an essential requirement.

A brief description of the widely used document classification techniques is as follows: a method of mapping the distance between the documents by mapping them to the vector space based on the frequency of the words appearing in the document, and then classifying the distances between the documents and applying the Naive Bayes theory And probabilistic models.

However, when the conventional document classification technique is applied to a natural language, the technique of limiting the number of words to the number of word frequencies has a limitation on the reliability of the natural language. This is a factor of deterioration of search performance in the search process using the natural language of the Internet information search system described above.

Although a method utilizing a knowledge database such as WordNet has been proposed for the above reasons, there is a problem that the accuracy of document classification depends on the reliability of the knowledge database.

The present applicant has proposed a method in which an N-gram is a method that combines the number of occurrences of air information in the document with the number of appearance frequencies, and is a method that is actively used in the field of speech recognition and natural language processing. , And the present invention can be effectively utilized for selecting representative words.

Korean Patent No. 10-0842080, " Document classification method by group " Korean Patent No. 10-1092352, " Automatic Classification Method and Apparatus for Sentence Corpus "

The present invention relates to a device for automatically learning a document and automatically classifying a large amount of documents on the web through an automatic document learning and classification process based on the document, an automatic document learning method using the same, The purpose of this paper is to provide a method for automatically classifying documents.

In addition, the present invention provides a semantic analysis of a web document and a search result closely related to a user's query word through a method of automatically classifying a new document when an unidentified new document appears after learning a web document in a predefined field And a method for automatically classifying a document using the apparatus and a method for automatically classifying a document using the same.

In addition, the present invention constructs an engram that can represent a document based on an enumeration which is a set of words appearing at the same time and the frequency of appearance thereof, so that when an unidentified new document is found, A document automatic learning apparatus and a document automatic classification apparatus using the automatic document learning apparatus and a document automatic classification method using the same.

According to another aspect of the present invention, there is provided an automatic learning apparatus for a document according to the present invention, comprising: a learning document pool including a plurality of learning document groups classified into categories; and a preprocessing process for each learning document group of the learning document pool And an engram data set pool in which an engram data set of the learning document pool formed by learning through the preprocessing process of the preprocessing unit is stored.

The preprocessing unit may further include a special character removal unit for removing special characters for each learning document in the learning document group, a stopword removing unit for removing stopwords for each learning document in the learning document group, And a part-of-speech extraction unit for extracting a noun and a verb from a learning document that has been rejected by the part-of-speech recognition unit.

In addition, the engram data set is generated by storing a Bigram set for each category.

Also, the bi-gram set is generated by extracting and constructing a bigram in units of words.

According to another aspect of the present invention, there is provided a method for automatically learning a document, the method comprising: a plurality of learning documents classified into categories to form a plurality of learning document groups; a step of preprocessing the plurality of learning document groups individually; And separately generating an engram data set from the plurality of learning document groups.

The step of individually preprocessing the plurality of learning document groups may include the steps of removing special characters for each learning document in the learning document group, removing a stopword for each learning document in the learning document group, Extracting nouns and verbs from the learning documents that have been subjected to the morphological analysis and the process of extracting some parts of speech, and then extracting the nouns and verbs from the learning documents, Extracting a number of simultaneous occurrence frequencies of the verbs; and constructing an engram data set based on the number of simultaneous occurrence frequencies.

In addition, the step of constructing an engram data set based on the number of simultaneous appearance frequencies is a step of constructing a bi-gram data set through the preprocessing step for each learning document group.

The automatic classifying apparatus for a document according to the present invention may further comprise: a learning document pool including a plurality of learning document groups classified into categories; a preprocessing unit for preprocessing each learning document group of the learning document pool; And an engram data set pool in which an engram data set of the learning document pool formed by learning is learned, and when a new document unidentifiable through the learning document pool is present, the preprocessor preprocesses the new document to generate a bi- gram set A bi-gram set of the new document formed through the preprocessing unit and a bi-gram set of the engram data set pool, and comparing the bi-gram set of the new document with one of the engram data set pools Automatically classify documents that are assigned to and stored in an enum data set It is configured to include a.

In addition, the automatic document classifying unit may classify

"

Bigram Weight: The similarity between the biggram set of the new document and the bigogram set for each engram data set of the engram data set pool, and if the biggram data extracted from the new document exists in the biggram for each engram data set, Multiply the number of occurrences of the word bigram.

CW: A value obtained by multiplying the calculated Bigram Weight by the number of times the same number of bigrams appear in each of the new document and the Engram data set.

The similarity between the bi-gram set of the engram dataset pool and the bi-gram set of the new document is measured.

Also, the CW is allocated to a maximum value among values obtained by multiplying the pre-computed Bigram Weight by the number of bigrams appearing simultaneously in the new document and the set of engram data.

Also, the automatic document classification unit may classify the new document into a set of the engram data sets through the learning of the new document and the corresponding learning, based on the air information of the words, the number of occurrence frequencies of the words, Is determined.

According to another aspect of the present invention, there is provided a method of automatically classifying a document, comprising: forming a plurality of learning document groups by categorizing a plurality of learning documents into a plurality of categories; extracting special characters, stopwords, A preprocessing step of extracting only a verb from the plurality of learning document groups, a preprocessing step of extracting only a verb from the plurality of learning document groups, and a preprocessing step of separately generating an engram data set from the plurality of learning document groups through the preprocessing, A step of preprocessing a new document by the same process as the pre-processing to form a corresponding biagram of the new document; comparing the biagram of the new document with the biagram of the engram data set, And assigning the same to one.

According to the present invention, large documents on the web can be automatically learned and classified through the automatic document learning and classification process based on an engram.

In addition, it is possible to provide semantic analysis of web documents and search results closely related to the user's query, through a method of automatically classifying new documents when an unidentified new document is found after learning a predefined area of web documents .

In addition, an engram that can represent a document based on the set of words that appear at the same time and the frequency of appearance of the document is constructed so that a new document that is not identified at a later time is found. Based on the previously constructed information, Can be automatically classified.

1 is a block diagram illustrating an apparatus for automatic learning of a document according to an embodiment of the present invention.
2 is a block diagram conceptually showing an apparatus for automatically classifying a document according to an embodiment of the present invention.
FIG. 3 is a flowchart illustrating a method of automatic document learning according to an embodiment of the present invention.
4 is a flow chart illustrating a document automatic classification method according to an embodiment of the present invention.

Hereinafter, a document automatic learning apparatus according to an embodiment of the present invention, an automatic document learning method using the automatic document learning apparatus, an automatic document classification apparatus, and a document automatic classification method using the same will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an apparatus for automatic learning of documents according to an embodiment of the present invention.

As shown in the figure, the automatic document learning apparatus 100 according to an exemplary embodiment of the present invention includes a learning document pool 110, a preprocessing unit 120, and an engram data set pool 130.

The learning document pool 110 is formed by including a plurality of learning document groups 111 classified into categories.

The preprocessing unit 120 preprocesses each learning document group in the learning document pool 110. The preprocessing unit 120 includes a special character removal unit 121, a stopword removal unit 122, A part-of-speech rejection 123, and a part-of-speech extraction unit 124.

The special character removal unit 121 removes special characters for each learning document in the learning document group 111 and the stopword removal unit 122 removes a stopword for each learning document in the learning document group 111, . Then, the part-of-speech rejection unit 123 analyzes the morpheme for each learning document in the learning document group 111, and extracts parts of speech other than nouns and verbs. The part-of-speech extraction unit 124 extracts a learning document To extract nouns and verbs.

The engram data set pool 130 stores an engram data set 131 of the learning document pool 110 formed by learning through the preprocessing process of the preprocessing unit 120. Then, the engram data set 131 generates and stores a Bigram set for each category, where the bi-gram set is generated by extracting and building Bigram in units of words.

The automatic classification apparatus for documents according to an embodiment of the present invention will now be described with reference to FIG.

As shown in the figure, the automatic document classification apparatus 200 according to an embodiment of the present invention includes a document automatic learning unit 210 and a document automatic classification unit 220.

The automatic document learning unit 210 corresponds to the automatic document learning apparatus 100 described above with reference to FIG. 1, that is, the automatic document learning unit 210 includes a learning unit 220 that includes a plurality of learning document groups A preprocessor for preprocessing a preprocess for each learning document group of a document pool and a learning document pool; and an engram data set pool storing an engram data set of the learning document pool formed by learning through the preprocessing process of the preprocessing unit . The automatic document learning unit 210 pre-processes the new document when a new document that is not identified through the learning document pool appears on the automatic document learning unit 210, thereby forming a bi-gram set.

The configuration including the preprocessing unit of the automatic document learning unit 210 and the configuration of the preprocessing unit are the same as those of the preprocessing unit of the automatic document learning apparatus according to FIG. 1, and therefore, do.

 The automatic document classification unit 220 compares the bi-gram set of the new document formed through the pre-processing unit of the automatic document learning unit 210 with the bi-gram set of the engram data set pool and stores the bi- gram set of the new document in the engram data set pool 230, only the document data set pool of the automatic document learning unit 210 is shown separately for the sake of understanding).

Here, the automatic document sorting unit 220 measures the similarity between the biagram set of the engram data set pool 230 and the bi-gram set of the new document through the following equation.

[Formula 1]

Bigram Weight: The similarity between the bigram set of a new document and the bigram set for each engram data set in the engram data set pool. When the big-gram data extracted from the new document exists in the big-gram for each engram data set, the occurrence of the corresponding big-gram word Multiplied by the number of frequencies.

CW: The calculated Bigram Weight multiplied by the number of times the newgrams appear in each new document and engram data set.

In other words, the automatic document classification unit 220 learns new documents based on the air information of the words, the number of occurrence frequencies of the words, and the number of occurrence occurrences of the engrams, and either of the engram data set pools 230 To determine classification into an engram dataset. Table 1 below shows this example.

Bigram Data Bigram weight CW
New Document chosun university 5,
computer science 4,
master students 5,
...
Category  One
chosun university 15,
computer science 40,
gwangju korea 37
...
ND * C1 = 5 * 15 + 4 * 40
= 235
1 + 235 * 2 = 471
Category  2 republic korea 64,
chosun university 5,
president korea 41,
...
ND * C2 = 5 * 5 = 25
1 + 25 * 1 = 26
Category  3 chosun university 71,
artificial intelligence 41,
international conference 13,
...
ND * C3 = 5 * 71 = 355
1 + 355 * 1 = 356

The CW may be assigned to a maximum value of a value obtained by multiplying a pre-calculated Bigram Weight by the number of times of a bigram simultaneously appearing in each of the new document and the Engram data set.

Next, a method of automatic document learning according to an embodiment of the present invention will be described with reference to FIG. Here, the automatic learning apparatus of the document is based on the embodiment of FIG. 1, and therefore, in the following description, the configuration of the automatic learning apparatus of the document follows the embodiment of FIG. 1 and uses the same reference numerals.

First, in step S110, a plurality of learning documents are classified into categories and a plurality of learning document groups 111 are formed.

Subsequently, in step S120, a plurality of learning document groups 111 are preprocessed individually.

Here, the preprocessing process is a process in which special characters are removed for each learning document in the learning document group 111 in step S121, a stopword is removed for each learning document in the learning document group 111 in step S122, , Analyzing morphemes for each learning document in the learning document group 111 in step S123 and extracting parts of speech other than nouns and verbs, morphological analysis in step S124, Extracting nouns and verbs from the learning document, extracting the number of simultaneous appearance frequencies of the nouns and verbs, and building an engram data set based on the number of simultaneous appearance frequencies in step S125.

After the series of preprocessing steps, a step of separately generating the engram data sets 131 from the plurality of learning document groups 111 is performed in step S130.

In addition, step S130 corresponds to the step of constructing the bi-gram data set through the preprocessing process for each learning document group 111. [

The following will describe a document automatic classification method according to an embodiment of the present invention with reference to FIG. Here, the automatic classification apparatus of the document is based on the embodiment of FIG.

First, in step S210, a plurality of learning documents are classified into categories and a plurality of learning document groups are formed.

Next, in step S220, preprocessing for extracting only nouns and verbs is performed by removing special characters, stopwords, and morpheme analysis for each learning document group.

Then, in step S230, a set of the engram data is separately generated from the plurality of learning document groups through the preprocessing in step S220.

Then, in step S240, a new document that is not identified through the set of engram data appears.

Then, in step S250, the new document of step S240 is pre-processed in the same process as the preprocessing of step S220 to form a corresponding bi-gram of the new document.

Then, in step S260, the biagram of the new document is compared with the biagram of the engram data set, and the new document is assigned to one of the learning document groups and classified.

As can be seen from the embodiments of FIGS. 1 to 4, the automatic learning apparatus for a document according to the present invention, the automatic document learning method using the same, and the automatic document classification apparatus and the automatic document classification method using the same, It enables automatic learning and classification of large documents on the web through gram-based document automatic learning and classification process.

In addition, it is possible to provide semantic analysis of web documents and search results closely related to the user 's query term through a method of automatically classifying web documents after learning stages of predefined fields.

It is also possible to construct an engram that can represent a document based on the set of words that appear at the same time and the frequency of occurrence of the new document, so that when a new document that is not identified is found later, To be automatically classified.

In addition, if a new document appears after constructing a set of statistical distribution values for each word in a learning document corresponding to a predefined classification item, the distribution of the engrams existing in the new document and the distribution of the previously learned engrams New documents are automatically classified through similarity measure. At this time, the distribution data of the generated documents in the new document accumulates in the distribution of the learning learning engrams, and it gradually expands to improve the reliability with the data that can adapt to the change of information.

Although the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. I would say that there is a technical mind.

100: Automatic learning device of document 110: Learning document pool
111: learning document group 120: preprocessing unit
121: Special character removal 122:
123: Rejection of bodily injury 124:
130: engram data set pool 131: engram data set
200: automatic classification apparatus of document 210: document automatic learning unit
220: Document automatic classification section

Claims (13)

A learning document pool including a plurality of learning document groups classified by category;
A preprocessor for preprocessing each learning document group of the learning document pool;
And an engram data set pool in which an engram data set of the learning document pool formed by being learned through a preprocessing process of the preprocessor is stored. The method of claim 1, wherein the pretreatment unit
A special character removing unit for removing a special character for each learning document of the learning document group;
A stopword removing unit for removing a stopword for each learning document in the learning document group;
Parsing the morpheme for each learning document in the learning document group and rejecting parts of speech other than nouns and verbs;
And a part-of-speech extracting unit for extracting nouns and verbs from the learning document having passed through the part-of-speech tagger. The method of claim 1,
The engram data set generates and stores a bigram set for each category. The method of claim 3, wherein
The bigram set is generated by extracting and constructing a word unit biggram (Bigram). Forming a plurality of learning document groups by classifying the plurality of learning documents by category;
Separately preprocessing the plurality of groups of learning documents;
And automatically generating an engram data set from the plurality of learning document groups through the preprocessing. 6. The method of claim 5,
The preprocessing of the plurality of learning document groups individually
Removing a special character for each learning document of the learning document group;
Removing stopwords for each learning document of the learning document group;
Analyzing parts of each learning document of the learning document group to screen out parts of speech other than nouns and verbs;
Extracting nouns and verbs from the learning document that has undergone the morphological analysis and selecting parts of speech to extract the number of simultaneous occurrences of the nouns and verbs;
And building an engram data set based on the number of co-occurrence frequencies. The method according to claim 6,
Building an engram data set based on the number of co-occurrence frequencies
Automatic learning method of a document, characterized in that the step of building a by-gram data set through the pre-processing for each learning document group. A preprocessing unit for preprocessing each learning document group of the learning document pool, wherein the preprocessing unit includes a plurality of learning document groups classified into categories, Wherein the preprocessing unit pre-processes the new document to form a bi-gram set when a new document that is not identified through the learning document pool is present;
By comparing the by-gram set of the new document formed by the pre-processing unit and the by-gram set of the engram data set pool, allocates and stores the by-gram set of the new document to any one of the engram data set pool. An automatic classification device for documents including a document automatic classification unit. The method of claim 8,
The automatic document classifying unit may classify
"

Bigram Weight: The similarity between the biggram set of the new document and the bigogram set for each engram data set of the engram data set pool, and if the biggram data extracted from the new document exists in the biggram for each engram data set, Multiply the number of occurrences of the word bigram.
CW: A value obtained by multiplying the calculated Bigram Weight by the number of times the same number of bigrams appear in each of the new document and the Engram data set.
And measures the similarity between the bi-gram set of the engram dataset pool and the bi-gram set of the new document. The method of claim 9,
And the CW is assigned to a maximum value of a value obtained by multiplying the pre-calculated Bigram Weight by the number of times the new document and the Engram data set are simultaneously displayed. 9. The image processing apparatus according to claim 8, wherein the pre-
A special character removing unit for removing a special character for each learning document of the learning document group;
A stopword removing unit for removing a stopword for each learning document in the learning document group;
Parsing the morpheme for each learning document in the learning document group and rejecting parts of speech other than nouns and verbs;
And a part-of-speech extracting unit for extracting a noun and a verb from the learning document that has been rejected. The method of claim 8,
The document automatic classification unit classifies an engram data set into one of the engram data set pools through learning and corresponding learning of the new document based on the word air information, the number of occurrence frequencies, and the number of engram occurrence frequencies. Automatic classification apparatus of the document, characterized in that for determining. Forming a plurality of learning document groups by classifying the plurality of learning documents by category;
A preprocessing step of extracting only nouns and verbs by removing special characters and stopwords and morphological analysis for each learning document group;
Separately generating an engram data set from the plurality of learning document groups through the preprocessing;
Appearing a new document not identified through the engram data set;
Preprocessing the new document in the same process as the preprocess to form a corresponding vigram of the new document;
And comparing the bygram of the new document with the bygram of the engram data set, and assigning the new document to one of the learning document groups to classify the new document.

Images