KR102492881B1 - Classification Dictionary Building Method Per Subject of Text Data and Storage Medium Recording Program for Executing the Same - Google Patents

Abstract

A subject-specific classification dictionary construction method for text data according to the present invention is a subject-specific classification dictionary construction method for text data performed through a management server in which a subject-specific classification dictionary construction program for text data stored in a storage medium is installed. In the step (a) of extracting text data from an online service, the step of (b) dividing and extracting sentences included in the text data extracted by step (a) into predetermined word units, and the step of (b) Step (c) of pre-processing the words divided by the step, step (d) of deriving subject names of the text data by analyzing the words pre-processed by step (c), and steps (a) to (d) It includes a step (e) of storing the process of the step as an initial classification system.

Description

Classification Dictionary Building Method Per Subject of Text Data and Storage Medium Recording Program for Executing the Same}

The present invention relates to a method for constructing a subject-specific classification dictionary for text data and a computer-readable storage medium recording a program for executing the same, and more particularly, to a dictionary by recommending a classification pattern highly related to the subject of text data. It relates to a subject-specific classification dictionary construction method for text data provided to facilitate construction and a computer-readable storage medium recording a program for executing it.

Recently, as the amount of data existing online has become vast, the task of constructing a classification dictionary by text mining big data consisting of text has become important.

Conventionally, in order to build a classification dictionary, a method of manually extracting words corresponding to data classification by sampling large-volume text data and reading it from beginning to end was used.

However, such a method of manually constructing a classification dictionary by a person has a problem in that a large amount of labor costs are incurred to reflect all of the vast amount of data, and a completely different classification dictionary is built according to the operator's subjectivity.

In addition, patterns that do not fit the subject are likely to be included in the classification dictionary, resulting in an incorrect classification system and a problem in which classification accuracy of data is greatly reduced.

Furthermore, there is a problem in that it is difficult to reflect the latest trends in the classification dictionary because there is a clear limit for a person to check the amount of latest data, which is recently increasing exponentially.

Therefore, a method for solving these problems is required.

Korean Patent Publication No. 10-2014-0010930

The present invention has been made to solve the above-mentioned problems of the prior art, and provides a method for constructing a classification dictionary by subject for text data that can facilitate dictionary construction by recommending a classification pattern highly related to the subject of text data. has a purpose to provide

The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, the subject-specific classification dictionary construction method for text data of the present invention is a subject for text data that is performed through a management server in which a subject-specific classification dictionary construction program for text data stored in a storage medium is installed. In the method for constructing a separate classification dictionary, (a) extracting text data from an online service, and (b) dividing and extracting sentences included in the text data extracted by the step (a) into preset word units. Step (c) of pre-processing the words divided by step (b), step (d) of deriving the subject title of the text data by analyzing the words pre-processed by step (c), and step of (a) and (e) storing the process of steps to (d) as an initial classification system.

In this case, the step (b) corresponds to the step (b-1) of dividing the sentence included in the text data by word and the group of nouns, verbs, and adjectives in the words divided by the step (b-1). It may include a step (b-2) of separating and extracting only the word.

In the step (c), the step (c-1) of configuring a combination group by combining the n words extracted in the step (b), extracting the subject index of each word included in the combination group ( Step c-2) and step (c-3) of assigning a weight to each word included in the combination group may be included.

Here, in step (c-3), when the combination group includes a noun and an adjective, additional weight may be given to the noun of the corresponding combination group.

Alternatively, in step (c-3), when the adjectives included in the combination group are pre-set and classified emphasized adjectives, additional weight may be given to the adjectives.

In the step (d), the step (d-1) of calculating the number of times the random word preprocessed by the step (c) is exposed in the text data, the words constituting the text data (d-2) step of calculating the ratio representing the same topic as the random word in step -1), calculating the topic index by multiplying the result values of the step (d-1) and step (d-2) Step d-3) and step (d-4) of deriving the subject name of the text data in consideration of the subject index of step (d-3).

Meanwhile, the present invention may further include a step (f) of updating to an extended classification system by reflecting the new classification system derived by steps (a) to (d) newly performed after the initial classification system is stored.

In the step (f), the similarity between the subject name of the text data included in the initial classification system and the subject name of the text data included in the new classification system is measured (f-1) and the step (f-1) (f-2) updating the extended classification system by summing or separating the subject name of the text data included in the new classification system with the subject name of the text data included in the initial classification system according to the similarity measurement result of can do.

At this time, the step (f-1),

The similarity between the subject name of the text data included in the initial classification system and the subject name of the text data included in the new classification system can be determined through the formula of .

Also, in the step (f-2), as a result of the similarity determination in the step (f-1), the subject names of the text data having an absolute value of 0.5 or more are integrated and stored, and the subject names of the text data having an absolute value of less than 0.5 are stored together. As it is recognized as a new classification and stored separately, the extended classification system can be updated.

In addition, in the step (f-2), if there are multiple subject names of the text data having an absolute value of 0.5 or more as a result of the similarity determination in the step (f-1), only the subject name of the text data having the highest absolute value is integrated and stored. can

* Meanwhile, the subject-specific classification dictionary construction method for text data of the present invention may be provided in the form of a computer-readable storage medium in which a program for executing it is recorded.

To solve the above problems, a method for constructing a subject-specific classification dictionary for text data and a computer-readable storage medium recording a program for executing the same according to the present invention to solve the above problems are conventional large-capacity text data checked one by one by a person to create a classification system. It has the advantage of reducing costs by automating the construction process mechanically, promptly reflecting the latest data, and constructing a dictionary with consistent classification standards.

In particular, in the prior art, there was a problem of not being able to build dictionaries in various fields because a person had to see tens of thousands of data in one field over a long period of time. It is possible to build a classification dictionary with accuracy, and has the advantage of making great progress in increasing data classification productivity.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a diagram showing the entire process of a method for constructing a classification dictionary by subject for text data according to an embodiment of the present invention;
2 is a view showing a detailed process of step (b) in a method for constructing a classification dictionary for each subject for text data according to an embodiment of the present invention;
3 is a view showing a detailed process of step (c) in a method for constructing a classification dictionary for each subject for text data according to an embodiment of the present invention;
4 is a diagram showing a detailed process of step (d) in a method for constructing a dictionary for classification by subject for text data according to an embodiment of the present invention; and
5 is a diagram illustrating a detailed process of step (f) in the method for constructing a dictionary for subject classification of text data according to an embodiment of the present invention.

Hereinafter, a preferred embodiment of the present invention in which the object of the present invention can be realized in detail will be described with reference to the accompanying drawings. In describing the present embodiment, the same name and the same reference numeral are used for the same configuration, and additional description thereof will be omitted.

The subject-specific classification dictionary construction method for text data according to the present invention is performed through a management server equipped with a storage medium in which a subject-specific classification dictionary construction program for text data is installed, which is installed in the management server and is installed in the management server's processor. can be driven by

In addition, the subject-specific classification dictionary construction program for the text data driven thereby may be output through an image output device such as a display module, and may provide visual information to the user through a visualized graphic user interface.

In particular, the storage medium in which the subject-specific classification dictionary construction program for text data is stored can be installed in the management server using a removable disk or a communication network, and the subject-specific classification dictionary construction program for text data is provided by the management server as a variety of functional means. can be made to operate. That is, in the present invention, information processing by software is concretely realized through hardware.

Hereinafter, the algorithm of the subject-specific classification dictionary construction method for text data, which is executed through the management server, of the present invention will be described.

1 is a diagram showing the entire process of a method for constructing a classification dictionary by subject for text data according to an embodiment of the present invention.

As shown in FIG. 1, a method for constructing a subject-specific classification dictionary for text data according to an embodiment of the present invention includes steps (a) of extracting text data from an online service, and the text extracted by step (a). Step (b) of dividing and extracting sentences included in the data into predetermined word units, step (c) of preprocessing the words divided by step (b), and analyzing the words preprocessed by step (c) and (d) step of deriving the subject name of the text data by using this method, and (e) step of storing the process of steps (a) to (d) as an initial classification system.

Step (a) is a process of extracting predetermined text data from various online services.

Here, the online service may be various online service media using a communication network such as the web and SNS, and is not limited to any one online service. It is also possible to include future online services that will appear in the future.

Step (b) is a process of dividing and extracting sentences included in the text data extracted by step (a) into preset word units, and may include a plurality of processes in detail.

2 is a diagram illustrating a detailed process of step (b) in the method for constructing a dictionary for classification by subject for text data according to an embodiment of the present invention.

As shown in FIG. 2, step (b) includes step (b-1) of dividing sentences included in text data by word in detail, and noun groups and verbs from the words divided by step (b-1). and (b-2) step of separating and extracting only the words corresponding to the group and the adjective group.

In step (b-1), the words included in the sentences included in the text data are separately processed. For example, if there is a sentence such as 'Good cosmetics make the skin moist' in the text data, it is classified according to the words 'good', 'cosmetics', 'skin', 'moisturize', and 'make'. can be divided

And in step (b-2), in order to separate and extract only words corresponding to the noun group, verb group, and adjective group from the words divided by step (b-1), , the remaining elements except for the noun group including pronouns, the verb group including verbs representing 'action', and the adjective group including adjectives representing 'emphasis', that is, particles, final endings, numerals, adjectives, adverbs, etc. elements of will be removed.

The reason for doing this is to clarify the objects described by the sentences included in the text data and to increase the accuracy of classification, that is, the topic, through association between words in each sentence.

Next, step (c) is a process of pre-processing the words divided by step (b), and may include a plurality of processes in detail.

3 is a diagram illustrating a detailed process of step (c) in the method for constructing a dictionary for classification by subject for text data according to an embodiment of the present invention.

As shown in FIG. 3, step (c) includes, in detail, step (c-1) of configuring a combination group by combining n words extracted in step (b), and each word included in the combination group. A step (c-2) of extracting subject indices and a step (c-3) of assigning a weight to each word included in the combination group are included.

In step (c-1), a combination group is formed by combining n words corresponding to the noun group, verb group, and adjective group, which are the words extracted in the above-mentioned step (b). Here, n may be a natural number of 1 or more, and in the case of this embodiment, it is exemplified that a combination group is formed by combining one word to three words.

In this way, the reason why the combination group does not exceed three in this embodiment is that the subject of the data may be too subdivided when the amount of data increases.

In addition, the combination group that can be combined according to this embodiment can be configured as shown in Table 1 below.

The configuration criteria of Table 1 are combinations to express the association of words well, and the meaning of each word according to the combination of words can express the subject of the text data well. At this time, the combination group is made up of sentence units, and the criteria for sentences were based on final endings, periods, and line breaks.

On the other hand, if words made of text are used as they are in the machine learning algorithm, the speed is very slow and the consumption of computer hardware resources is very extreme. After creating a temporary word dictionary, large-volume data divided by word can be replaced by integers with large-data word vectors.

An example of this may be configured as shown in Table 2 below.

In step (c-2), the subject index of each word included in the combination group is extracted, and in step (c-3), a weight is assigned to the subject index of each word included in the combination group. .

Here, the weighting of the subject index may be performed according to various criteria.

In this embodiment, when a combination group includes a noun and an adjective, additional weight is given to the noun of the combination group. A method of giving additional weight to was applied.

The reason for doing this is that combination groups that include adjectives with nouns put emphasis on the nouns, and adjectives that show strong subject meanings for nouns show higher subject meanings than other adjectives. because bet

Next, in step (d), a process of deriving the subject title of the text data by analyzing the words preprocessed in step (c) is performed. Similarly, step (d) may include a plurality of processes in detail.

4 is a diagram showing a detailed process of step (d) in the method for constructing a dictionary for subject classification of text data according to an embodiment of the present invention.

As shown in FIG. 4, step (d) includes, in detail, step (d-1) of calculating the number of times the random word preprocessed by step (c) is exposed in text data, and step (d-1) of configuring text data. The topic index is obtained by multiplying the result values of steps (d-1) and (d-2) with step (d-2), which calculates the ratio of words representing the same topic as any word in step (d-1). Step (d-3) of calculating, and step (d-4) of deriving the subject name of the text data in consideration of the subject index of step (d-3).

The reason for calculating the number of exposures of a random word preprocessed in step (d-1) in the text data is that the more often a word is mentioned in each text data, the more likely it is to be an important pattern in the text data.

In addition, as in step (d-2), the reason why the ratio of words constituting text data representing the same subject as the selected random word is calculated is that the word corresponding to the subject that appears in a consistently high ratio in text data This is because it is likely to be an important pattern in the data.

Accordingly, in step (d-3), a topic index is calculated by multiplying the result values of steps (d-1) and (d-2), and patterns are recommended in the order of highest. The subject name of the text data is derived by considering the subject index of step (d-3).

And in each process of step (d), the weight for each word assigned in step (c-3) may be applied.

Through the above process, the subject name of the text data is derived, and in step (e), the process of steps (a) to (d) is stored as an initial classification system.

However, since the initial classification system cannot classify new words or emerging topics, it is necessary to always collect new data, create a classification system, and expand it by adding it to the initial classification system.

Therefore, in the subject-specific classification dictionary construction method for text data according to an embodiment of the present invention, after the initial classification system is stored by step (e), the newly performed steps (a) to (d) derive A step (f) of updating to an extended classification system by reflecting the new classification system may be further included.

5 is a diagram illustrating a detailed process of step (f) in the method for constructing a dictionary for subject classification of text data according to an embodiment of the present invention.

As shown in FIG. 5, step (f) includes, in detail, the step (f-1) of measuring the similarity between the subject name of the text data included in the initial classification system and the subject name of the text data included in the new classification system; ( According to the similarity measurement result in step f-1), the subject name of the text data included in the new classification system is added or separated from the subject name of the text data included in the initial classification system to update the extended classification system (f-2) can include

In the case of step (f-1), a process of measuring similarity between subject names of text data included in the initial classification system and subject names of text data included in the new classification system is performed to expand the initial classification system.

At this time, of course, the similarity measuring method between the subject name of the text data included in the initial classification system and the subject name of the text data included in the new classification system can be applied in various ways.

In the case of this embodiment, as a method for determining the degree of similarity, Equation 1 below was applied. In Equation 1, the closer the absolute value of the result is to 1, the higher the degree of similarity.

And in step (f-2), as a result of similarity determination in step (f-1), the subject names of text data having an absolute value of 0.5 or more are integrated and stored, and the subject names of text data having an absolute value of less than 0.5 are stored. As it is recognized as a new classification and stored separately, the extended classification system can be updated.

In addition, as a result of the similarity determination in step (f-1), when there are multiple subject names of text data having an absolute value of 0.5 or more, only the subject names of the text data having the highest absolute value can be collectively stored.

For example, when a table as shown in Table 3 above is derived as a result of determining similarities between initial classification 1 and initial classification 2 and new classification 1 and new classification 2, new classification 1 is integrated into initial classification 1, and new classification 2 is the initial classification. 2 can be integrated.

At this time, the new category 2 also satisfies the integration condition with the initial category 1 because the absolute value is 0.5 or more. However, since the absolute value of the new category 2 is higher than that of the new category 1, the new category 1 is integrated with the initial category 1.

As described above, the present invention reduces costs by mechanically automating the process of establishing a classification system by checking conventional large-volume text data one by one by a person, promptly reflecting the latest data, and constructing a dictionary with consistent classification criteria. You will be able to do it.

As described above, the preferred embodiments according to the present invention have been reviewed, and the fact that the present invention can be embodied in other specific forms without departing from the spirit or scope in addition to the above-described embodiments is a matter of ordinary knowledge in the art. It is self-evident to them. Therefore, the embodiments described above are to be regarded as illustrative rather than restrictive, and thus the present invention is not limited to the above description, but may vary within the scope of the appended claims and their equivalents.

Claims (9)

A subject-specific classification dictionary construction method for text data performed through a management server in which a subject-specific classification dictionary construction program for text data stored in a storage medium is installed,
(a) extracting text data from an online service;
(b) dividing and extracting sentences included in the text data extracted in step (a) into preset word units;
(c) step of pre-processing the words divided by step (b);
(d) deriving subject names of the text data by analyzing the words preprocessed by step (c);
(e) storing the process of steps (a) to (d) as an initial classification system; and
Including a step (f) of updating to an extended classification system by reflecting the new classification system derived by steps (a) to (d) newly performed after the initial classification system is stored,
In step (c),
(c-1) forming a combination group by combining n words (where n is a natural number equal to or greater than 1) extracted in step (b); and
And (c-2) step of pre-processing the extracted word based on the combination group,
The combination group is formed in sentence units, and the criterion of the sentence is based on a final ending, a period, or a line break,
In the step (c-1), one to three words are combined to form the combination group,
In step (d),
(d-1) calculating the number of exposures of the words preprocessed by step (c) in the text data;
(d-2) calculating a ratio in which the words constituting the text data represent the same subject as the preprocessed words;
(d-3) calculating a subject index by multiplying the result values of steps (d-1) and (d-2); and
(d-4) deriving a subject name of the text data in consideration of the subject index of the step (d-3);
In step (f),
(f-1) measuring similarity between subject names of text data included in the initial classification system and subject names of text data included in the new classification system; and
According to the similarity measurement result of step (f-1), updating the extended classification system by adding or separating the subject name of the text data included in the new classification system with the subject name of the text data included in the initial classification system ( Step f-2); Including,
A method for constructing a subject-specific classification dictionary for text data.
According to claim 1,
In step (b),
(b-1) dividing a sentence included in the text data by word; and
(b-2) step of separating and extracting only words corresponding to the noun group, verb group, and adjective group from the words divided by the step (b-1);
including,
A method for constructing a subject-specific classification dictionary for text data. According to claim 1,
In the step (c-2),
(c-2-1) extracting the subject index of each word included in the combination group; and
(c-2-2) assigning a weight to each word included in the combination group;
including,
A method for constructing a subject-specific classification dictionary for text data. According to claim 3,
In the step (c-2-2),
When the combination group includes a noun and an adjective, to give additional weight to the noun of the combination group,
A method for constructing a subject-specific classification dictionary for text data. According to claim 3,
In the step (c-2-2),
If the adjective included in the combination group is a pre-set and classified emphatic adjective, additional weight is given to the adjective,
A method for constructing a subject-specific classification dictionary for text data. According to claim 1,
In the step (f-1),

Judging the similarity between the subject name of the text data included in the initial classification system and the subject name of the text data included in the new classification system through the formula of
A method for constructing a subject-specific classification dictionary for text data. According to claim 6,
In the step (f-2),
As a result of the similarity determination in step (f-1), subject names of text data having an absolute value of 0.5 or more are integrated and stored, and subject names of text data having an absolute value of less than 0.5 are recognized as a new category and stored separately. to update the extended taxonomy,
A method for constructing a subject-specific classification dictionary for text data. According to claim 7,
In the step (f-2),
As a result of the similarity determination in step (f-1), if there are multiple subject names of text data having an absolute value of 0.5 or more, only the subject name of the text data having the highest absolute value is integrated and stored.
A method for constructing a subject-specific classification dictionary for text data. A computer-readable storage medium storing a program for executing the subject-specific classification dictionary construction method for text data of any one of claims 1 to 8 in a computer.

Images