KR101975419B1 - Device and method for terminology clustering informal text data for big data analysis - Google Patents

Abstract

The present invention relates to a terminology clustering device of informal text data for big data analysis. The terminology clustering device of informal text data for big data analysis includes: a database including a data set; a data preprocessing unit selecting data from the data set included in the database and preprocessing the selected data; a recommendation term determining unit separating morphemes of original terms included in the preprocessed data and calculating a recommendation score of the original terms; and a data clustering unit separating phonemes of the original terms, calculating similarity between the original terms from which the phonemes are separated, and clustering the original terms having a similarity calculation value equal to or greater than a predetermined threshold value, wherein the recommendation term determining unit is able to determine a recommended term among the plurality of original terms based on the recommendation score and a clustering result. The present invention is able to easily perform big data analysis.

Description

TECHNICAL FIELD [0001] The present invention relates to an apparatus and method for clustering termed unstructured text data for large data analysis,

The present invention relates to an apparatus and method for terminating clusters of unstructured text data for big data analysis.

For big data analysis, 70% to 80% of the total effort is used for data preprocessing. Big data analysis is explosively growing, but the development of data preprocessing technology is slow as much as the development of big data analysis technology. Therefore, there is a need to develop automated data preprocessing technology.

Despite the emphasis on irregular data preprocessing, the need for analysis of unstructured data is increasing due to the open environment of public information, and most of the preprocessing is manual.

Fuzzy matching algorithm is the most used algorithm for calculating the similarity between data in text data. This algorithm is an algorithm that calculates the similarity between data using the calculated values based on the edit distance (Levenshtein Distance).

The Fuzzy Matching algorithm is simply an algorithm that calculates the similarity between two data. This algorithm is applied to group data with a certain degree of similarity in data. In addition, since the Fuzzy Matching algorithm is developed based on English, it has a problem of calculating similarity based on non-phonological syllables when applied to Korean.

In addition, morpheme analysis is the most important technique in natural language processing, separating words or sentences into morphemes, which are the smallest unit of words, and judging the parts of the separated morpheme. Through the morphological analysis of the dataset, you can identify key morphemes in the dataset.

The background technology of the present application is disclosed in Korean Patent Laid-Open Publication No. 10-2016-0075974.

It is an object of the present invention to overcome the above problems of the prior art and to overcome the difficulty of the preprocessing of big data unstructured text by clustering similar terms in the data to make large data analysis easier, And to provide a terminology clustering apparatus and method.

SUMMARY OF THE INVENTION The present invention has been made to solve the problems of the prior art described above and it is an object of the present invention to provide a method and apparatus for analyzing large data that can reduce the time required for an arbitrary data preprocessing process, And an apparatus and method for clustering terms of unstructured text data.

It is an object of the present invention to provide a method for automatically recommending a representative term to a user by applying morphological analysis to a user, And to provide a term clustering apparatus and method for unstructured text data.

In order to solve the problem of the conventional art described above, in order to solve the problem of calculating the edit distance of Korean text data in syllable units in the similarity calculation process, the syllables are grouped using a method of separating and calculating phonemes, And an object of the present invention is to provide a term clustering apparatus and method of unstructured text data that can correct human errors such as user's mistakes and help standardize irregular data.

It is to be understood, however, that the technical scope of the embodiments of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to an aspect of the present invention, there is provided an apparatus for grouping atypical text data for analysis of big data according to an embodiment of the present invention includes a database including a data set, a data set included in the database, A data preprocessing unit for selecting data and performing preprocessing, a recommendation term determination unit for separating the morpheme of the original term included in the preprocessed data, calculating the recommendation score of the original term, and the phonemes of the original term, And a data clustering unit for performing clustering of original terms having the similarity calculation value equal to or greater than a preset threshold value, wherein the recommendation term determining unit determines a plurality of clusters based on the recommendation score and the clustering result The recommended terms can be determined from the original terms.

The pre-processing unit according to an embodiment of the present invention determines a first data set to perform clustering of terms among the data sets included in the database, and performs clustering of terms among a plurality of column items of the first data set 1 column to perform data preprocessing of the selected column item.

The preprocessing unit according to an embodiment of the present invention may perform preprocessing to remove data that does not include redundant terms and terms included in the selected column.

The recommendation term determination unit according to an embodiment of the present invention calculates a recommendation score using a value obtained by quantizing the extraction frequency of the separated morpheme and a weight extracted based on the separated morpheme, And may be a value obtained by proportioning the specific gravity.

The recommendation score according to an exemplary embodiment of the present invention may be determined based on the extraction frequency of the morpheme separated on the basis of the value obtained by quantifying the extraction frequency of the separated morpheme and the length of each of the plurality of morphemes classified in the first source term, May be calculated using the sum of the results of dividing by the full length of the term.

According to one embodiment of the present invention, when the data is in Hangul, the data grouping unit can separate the phonemes according to the Korean alphabet with the initial, neutral, and trailing characters, and calculate the similarity using an artificial intelligence based algorithm.

The term clustering apparatus according to an embodiment of the present invention may further include a data output unit for providing a clustering result of the data clustering unit, wherein the clustering result may include a recommended term, an original term, and a similarity value.

The term clustering apparatus according to an embodiment of the present invention may further include a user input receiving unit for receiving term clustering performance information from a user terminal.

The recommendation term determiner according to an exemplary embodiment of the present invention may perform morpheme division based on the preprocessed data based on part-of-speech decision information included in the term clustering performance information when morpheme is separated.

According to an embodiment of the present invention, a method for grouping atypical text data for analysis of big data includes the steps of: selecting data from a data set included in a database and performing preprocessing; Calculating a recommendation score of the original terms, separating the phonemes of the original terms, performing a similarity calculation between the original terms separated by the phonemes, and clustering the original terms whose similarity calculation values are equal to or greater than a predetermined threshold value Determining a recommendation term among a plurality of original terms based on the recommendation score, the recommendation score, and the clustering result.

The above-described task solution is merely exemplary and should not be construed as limiting the present disclosure. In addition to the exemplary embodiments described above, there may be additional embodiments in the drawings and the detailed description of the invention.

According to the above-mentioned problem solving means of the present invention, an item is selected from a data set of a database, a syllable is converted into a phonemic unit by selecting a recommendation term through morphological analysis, word clustering is performed, Clustering can be done.

According to the task resolution means of the present invention, it is possible to cluster the terms with high precision by providing the user with a recommendation term in which priority is set by performing clustering after performing morpheme analysis.

According to the above-mentioned problem solving means of the present invention, since the terms are grouped by calculating the similarity in phonemic units, typographical errors such as typos can be substituted for the recommended terms in the data set.

According to the task resolution means of the present invention, it is possible to more precisely classify the atypical big data by grouping the notation errors of data or terms expressed differently in a recommendation term.

However, the effects obtainable here are not limited to the effects as described above, and other effects may exist.

1 is a schematic block diagram of a term clustering apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram schematically showing a part of a data item for performing clustering of a term clustering apparatus according to an embodiment of the present invention.
3 is a diagram for explaining morphemic separation of a term clustering apparatus according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating an exemplary ranking result of the morphemes in an inverse order in a term clustering apparatus according to an exemplary embodiment of the present invention.
FIG. 5 is a diagram exemplifying a result of calculating a recommendation score of a term in a term clustering apparatus according to an embodiment of the present invention. FIG.
FIG. 6 is a diagram schematically showing the ranking of the recommended terms according to the calculation of the recommended score of the terms in the term clustering apparatus according to the embodiment of the present invention.
FIG. 7 is an exemplary illustration of phonologicalization of terms of recommendation in a term clustering apparatus according to an embodiment of the present invention; FIG.
FIG. 8 is a result of exemplary clustering of terms in a terminology clustering apparatus according to an embodiment of the present invention.
FIG. 9 is a flowchart illustrating a method for grouping unstructured text data for analysis of big data according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when an element is referred to as being "connected" to another element, it is intended to be understood that it is not only "directly connected" but also "electrically connected" or "indirectly connected" "Is included.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as " including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

1 is a schematic block diagram of a term clustering apparatus according to an embodiment of the present invention.

1, the term clustering apparatus 100 includes a data base 110, a data preprocessing unit 120, a recommended term determiner 130, a data cluster unit 140, a data output unit 150, And a receiving unit 160.

According to an embodiment of the present invention, the data terminology clustering apparatus 100 may select a single column item from a data set, and select a preferred term by weight calculation according to morphological analysis within a selected column. The data terminology clustering apparatus 100 can group the original terms based on the recommended term and similarity calculation. The similarity calculation may include a preprocessing process for separating syllables into phonemes. In addition, if the recommended term does not represent the clustering original terms, the user can input any recommended term to cluster the original terms.

In addition, the data terminology clustering apparatus 100 may cluster the original terms in the column using an automated clustering algorithm in the column of the selected dataset. In addition, the data terminology clustering apparatus 100 can provide a term clustering method of unstructured text data for analysis of big data considering the convenience of the user by providing a recommendation term using a weight calculation using morphological analysis.

The database 110 may include a dataset used for clustering terms. The database 110 may include unstructured data. Unstructured data, unstructured data, and unstructured data may refer to information that does not have a predefined data model or is not organized in a predefined manner. Unstructured data can mean unstructured data, such as pictures, images and documents, which are different in shape and structure from the numerical data having a certain standard or form.

FIG. 2 is a diagram schematically showing a part of a data item for performing clustering of a term clustering apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the data set included in the database 110 may include two column items or more column items. The column items included in the dataset can be divided into a representative key and a general column. For example, the representative key of the column item of the data set of FIG. 2 may be a 'patient ID' and the generic column may be a 'disease name'. In this case, the general column 'disease name' can be made of unstructured text data. Atypical text data may include terms.

The data preprocessing unit 120 may select data from the data set included in the database 110 and perform preprocessing. The data preprocessing unit 120 may select a column to be subjected to data clustering among a plurality of columns of the data set stored in the database 110.

In other words, the data preprocessing unit 120 determines a first data set to perform clustering of terms among the data sets included in the database 110, and performs a term clustering among a plurality of column items of the first data set 1 column to perform data preprocessing of the selected column item. Referring to FIG. 2, the data preprocessing unit 120 determines a first data set to be used for grouping terms among a plurality of data sets included in the database 110, It is possible to perform data preprocessing of the selected 'disease name' column item by selecting the 'disease name' column to be performed.

The data preprocessing unit 120 may perform preprocessing to remove data that does not include redundant terms and terms included in the selected column. In other words, the data preprocessing unit 120 may perform a preprocessing process of eliminating redundant processing of data of a determined column and null value (data not including terms).

According to an embodiment of the present invention, the term pre-processing unit 120 does not require clustering in terms of completely matching terms and null data (data that does not include a term) Null values can be removed. In addition, the user can replace the null value (data not including the term) with another term as needed.

Illustratively, the user input receiving unit 160 may receive term clustering information from the user terminal. The term clustering information may include alternative terms for replacing data that does not include the term with another term. In other words, the data preprocessing unit 120 may input the alternative term in the data that does not include the alternative term provided from the user input receiving unit 160. [

FIG. 3 is a diagram for explaining morpheme separation of a term clustering apparatus according to an embodiment of the present invention, and FIG. 4 illustrates an example of a term clustering apparatus according to an exemplary embodiment of the present invention, FIG. 5 is a diagram exemplifying a result of calculating a recommendation score of terms in a term clustering apparatus according to an embodiment of the present invention.

Referring to FIG. 3, the recommendation term determiner 130 may separate morphemes of original terms included in the preprocessed data. The recommended term determination unit 130 may separate the morphemes of original terms included in the preprocessed data. Separation of a morpheme can be to separate the sentence or text into the smallest units and automatically identify the part of the morpheme. In addition, morpheme separation may be a separation of the morpheme into the minimum unit of meaning by using the original term. Morphological analysis is the most basic skill of natural language processing.

Illustratively, referring to FIG. 3, the recommendation terminology determining unit 130 may divide 'colon polyp removal' into 'colon', 'polyp', and 'disinfection'. In addition, the recommendation terminology determining unit 130 may separate the 'unspecified pneumonia' into 'detailed', 'unknown', and 'pneumonia'. The recommended term determination unit 130 may perform morpheme segmentation based on a specific part-of-speech.

According to one embodiment of the present invention, the separated morpheme can be used for prioritizing the recommended terms using the weights. The recommended term determinator 130 may rank the frequencies of the separated morphemes. This allows the user to identify the most used morpheme in the column (data) selected by the user. As an example, the recommendation term determiner 130 may sort the ranking of the separated morpheme (Rank) and sort the results as shown in FIG.

The recommended term determination unit 130 can calculate the recommended score of the original term using the separated morpheme. The recommended term determination unit 130 may calculate the recommendation score using the values obtained by digitizing the extraction frequency of the separated morpheme and the weight extracted based on the morpheme. Illustratively, the value obtained by quantifying the extraction frequency of the separated morpheme may be a value of Rank (frequency) shown in FIG. In this case, the weight may be a value obtained by proportioning the weight of the morpheme in the original term. For example, as shown in FIG. 4, the recommendation term determiner 130 ranks the inverse order of the frequencies so that the higher the frequency, the higher the ranking (rank 1), the proportion that the morpheme occupies in all the terms is proportional to the weight Can be used. In other words, the recommended term determiner 130 may quantify the order of the separated morphemes in reverse order, and determine the score of the morpheme so that the higher the morpheme having the higher frequency, the higher the value. The weight of the morpheme can be calculated with the proportion of the length of each morpheme in terms containing the morpheme. The recommended term determination unit 130 may calculate the recommendation score of the term by multiplying the weight and the score of the morpheme by the sum of the terms.

The recommendation score is calculated by dividing the extraction frequency of the separated morpheme based on the numerical value of the extraction frequency of the separated morpheme and the length of each of the plurality of morphemes classified in the first original term by the total length of the first original term Of the total sum of the two. For example, when the first morpheme includes the first morpheme to the third morpheme, the length of the first morpheme is divided by the total length of the first original term, And the sum of the results obtained by dividing the length of the third morpheme by the total length of the first original term to calculate the recommended score of the term.

By way of example, the recommendation score can be expressed as [Equation 1].

[Formula 1]

Where n is the number of morphemes separated from the term and rank is the rank of the frequency (in reverse order).

Illustratively, referring to FIGS. 4 and 5, the numerical values of the frequencies of 'detailed', 'unknown', and 'pneumonia' are 402, 399, and 330, respectively, and the total length of 'unspecified pneumonia' And each of the separated morphemes is 2 letters. The weights are weighted by dividing the total length of 7 by the length of the letters in the morpheme. As a result, the recommendation score of 'Unspecified pneumonia' is 323 points, which is the sum of the numbers of 'detailed', 'unknown', 'pneumonia'

According to one embodiment of the present application, the recommended term determiner 130 calculates the frequency of all parts of speech in morphological analysis, but the present invention is not limited thereto. For example, the recommendation terminology determining unit 130 may perform morpheme analysis by determining the part-of-speech based on the part-of-speech decision information included in the term clustering performance information provided through the user input receiving unit 160. In other words, the recommended term determiner 130 may perform morpheme segmentation based on the preprocessed data based on the part-of-speech decision information included in the term clustering performance information when the morpheme is separated.

FIG. 6 is a diagram schematically showing the ranking of the recommended terms according to the calculation of the recommended score of the terms in the term clustering apparatus according to the embodiment of the present invention.

Referring to FIG. 6, the recommended term determiner 130 may sort the terms calculated by applying the calculation method of the recommendation score described above in favor of terms having a high recommendation score.

According to one embodiment of the present invention, the recommendation term determiner 130 may select a recommendation term by calculating recommendation scores using frequency and weight of separated morphemes. In addition, the recommendation term determiner 130 may determine the priority of the recommendation term using the weight and the frequency of ranking. In addition, the recommendation term determiner 130 may determine the preference order of the recommendation terms using the weights using the ranking morpheme and the length of the morpheme.

FIG. 7 is an exemplary illustration of phonologicalization of terms of recommendation in a term clustering apparatus according to an embodiment of the present invention; FIG.

Referring to FIG. 7, the data collecting unit 140 separates phonemes of original terms and performs similarity calculation between original terms separated by phonemes. When the original term is Korean, the data cluster 140 separates the original word into phonemes according to the Korean alphabet, and can calculate the similarity using an artificial intelligence based algorithm. The data grouping unit 140 may perform similarity calculation between respective original terms using a fuzzy data matching algorithm, but is not limited thereto. The Fuzzy Data Matching algorithm is an algorithm that performs matching between data using the calculated values based on the edit distance (Levenshtein Distance).

The data clustering unit 140 can separate the terms made up of the syllables into the phonemes according to the Korean alphabet as the first, neutral, and last words. As an example, Fuzzy Data Matching algorithm, which is one of artificial intelligence based algorithms, uses a method of calculating the similarity only between shapes of words. Unlike English, which is the language on which the algorithm is based, Korean alphabets corresponding to alphabetic alphabets are combined to produce letters. Thus, the data grouping unit 140 solves syllables such as alphabets, The degree of similarity can be calculated. For example, if the similarity is calculated without phonemic separation, 'ri' 'and' ball 'are completely different letters, but since' ㄱ ㅏ 'and' ㅗ ㅗ ㅇ ' (140) can be separated into phonemes according to the Korean alphabet when the original term is Korean.

For example, referring to FIG. 7, in the case of 'unspecified pneumonia', the data collecting unit 140 can calculate the similarity by dividing the data into '' '' '' '' '' '' '' '' '' '' '. However, this process can be omitted in English.

According to one embodiment of the present invention, the data aggregation unit 140 may group the similarities between irregular data selected by the user. In addition, the data grouping unit 140 may calculate the similarities between the terms separated by the phonemes and group them into the recommended term groups according to the priority order of the recommended terms when the similarity value is exceeded. In other words, the data grouping unit 140 may group the original terms whose similarity calculation value is equal to or greater than a preset threshold value. The threshold value can be modified and changed to a threshold value of the similarity value according to the convenience of the user. In other words, the threshold value may be changed based on the threshold correction information included in the term clustering performance information received by the user input receiving unit 160.

Illustratively, referring to FIG. 7, the data grouping unit 140 may perform similarity calculation of each of a plurality of terms. For example, the data clustering unit 140 may perform a similarity calculation between the first original term (unspecified pneumonia) and the second original term (the spine of the thighbone) 1 Similarity calculations between the original term (unspecified pneumonia) and the third original term (acute hepatitis) can be performed. The data grouping unit 140 may perform similarity calculation for each of the first original term and the nth original term included in the column. The similarity calculation performed in the data cluster unit 140 may be used to determine a recommendation term of the recommendation term determiner 130.

According to one embodiment of the present invention, the data collecting unit 140 can separate and group the recommendation term priority data determined by the recommendation term deciding unit 130 into phonemes. Clustering can group clusters of original terms when they exceed a certain degree of similarity based on similarity based on editing distance.

FIG. 8 is a result of exemplary clustering of terms in a terminology clustering apparatus according to an embodiment of the present invention.

Referring to FIG. 8, the recommended term determination unit 130 may determine a recommended term among a plurality of original terms based on a recommendation score and a clustering result.

The recommended term determiner 130 generates a first clustering result 11 including a first original term 1, a second original term 2, a third original term 3, a fourth original term 4, A recommendation term among a plurality of original terms can be determined. The recommended term determiner 130 determines whether or not the first term 1, the second term 2, the third term 3 and the fourth term 4 included in the first clustering result 11, The recommended term can be determined based on the recommendation score of the 5th original term (5), the 6th original term (6), and the 7th original term (7). The recommended term determination unit 130 may select the original term having the highest recommended score among the original terms included in the first clustering result 11 as a recommended term. The first clustering result 11 may be a result of clustering original terms that are equal to or greater than a predetermined threshold value in the data clustering unit 140.

Illustratively, the recommended score for the first original term (unspecified pneumonia) is 377, the recommended score for the second original term (unspecified pneumonia) is 323, and the recommended score for the third original term (unspecified pneumonia) 323, and the recommendation score of the fourth original term (unspecified bacterial pneumonia) may be 310. The recommended term determination unit 130 may determine the first original term (unspecified pneumonia) having the highest recommendation score among the recommended scores of the first original term to the fourth original term as the recommended term. The similarity value may be a similarity value between a first original term (unspecified pneumonia) and a second original term (unspecified pneumonia) determined in a recommended term. In other words, the recommendation terminology determining unit 130 selects, as a recommendation term, the original term having the highest recommendation score among a plurality of original terms included in the first clustering result (11), and the similarity is a similarity degree between the recommendation term and the original term Lt; / RTI >

For example, referring to FIG. 8, the clustered data based on the recommended term 'unspecified pneumonia' includes' unspecified pneumonia ',' unspecified pneumonia ',' unspecified pneumonia ',' unspecified bacterial pneumonia ',' Unspecified pneumonia ',' unspecified pneumonia ', and' unspecified pneumonia '. It can be seen that the original terms clustered in the recommended terms are similar terms to 'unspecified pneumonia', except for spacing and survey which are different according to the user who entered the original term.

In addition, terms (data) clustered based on the recommended term 'neck bone spine sprains' include 'neck spine back bone spine', 'neck spine spine back bone spine', 'neck spine spine', 'waist spine spine' Sprains of the back bone, sprains of the back bone, sprains of the back bone, sprains of the back bone, and sprains of the back bone. Thus, by using special symbols, spaces, surveys, and conjunctions included in the terms, it can be seen that the terms that have been expressed differently are grouped into the recommended term 'neck bone spine sprains'.

Illustratively, referring to FIG. 8, the data result unit 150 may provide a cluster result of the data cluster unit 140. Clustering results may include suggested terms, original terms, and similarity values. Here, the original term may be a term included in the column. In other words, the original term may be data of an initial value included in the database 110. By way of example, a disease name may correspond to the original term. The recommended term may be a term determined among a plurality of original terms based on the recommendation score and the clustering result. The similarity may be a similarity value between the recommended term and the original term. The degree of similarity may be a similarity value between respective original terms calculated by phonemes in the data grouping unit 140. The user can provide the data result unit 150 with the percentage of similarity between the original terms and the recommended terms and the original terms clustered according to the recommended terms.

For example, the similarity value between the 'unspecified pneumonia', which is the recommended term of the first clustering result (11), and the 'unspecified pneumonia', the first original term, may be 100. Further, the similarity value between the 'unspecified pneumonia', which is a recommended term of the first clustering result (11), and the 'unspecified pneumonia', which is the second original term, may be 94. Further, the similarity value between the 'unspecified pneumonia', which is a recommended term of the first clustering result (11), and the 'unspecified pneumonia', which is the third original term, may be 100. When the existing clustering method is applied, the recommended terms 'unspecified pneumonia' and the third original term 'unspecified pneumonia' are included in the same term including the spacing, but the problem of clustering can not be obtained. By performing the similarity calculation after the segmentation by the phoneme of the data cluster unit 140, it is possible to solve the problem that it is not clustered due to the spacing.

According to one embodiment of the present invention, the data result unit 150 can identify, store, and modify the results of the data grouping unit 140. [ The clustering result that can be checked in the data result unit 150 is a result of clustering data included in the column items of the selected data set. The clustering result shows the similarity between the original data included in the column items of the dataset, the recommended terms, the recommended terms, and the original data. The user can check the clustering result in the data result section and modify the recommended term.

According to one embodiment of the present invention, the data result unit 150 can modify the recommendation term based on the data term clustering result. The data output unit 150 may modify the recommended term by a request for correction of a recommended term provided by the user terminal. In other words, the recommended terms can be modified by the user in terms of user convenience or standardization.

According to one embodiment of the present invention, the data result unit 150 may store a term (data) clustering result. It is possible to store data in which the term (data) clustering is completed in a form desired by the user such as the database 110. At this time, the user can re-confirm the cluster result by storing the new term (the column containing the recommended term) instead of modifying the recommended term instead of the original term.

According to one embodiment of the present invention, the user input receiving unit 160 may receive term clustering performance information from a user terminal (not shown). The term clustering performance information may include user input information for determining a first data set to be clustered among the datasets included in the database. And may include user input information for determining the part-of-speech in the morphological separation of terms. In addition, the term clustering performance information may include user input information for setting a threshold for classifying the similarity between respective original terms separated by a phoneme.

According to one embodiment of the present invention, the user input receiving unit 160 may provide a term clustering menu to a user terminal (not shown). For example, a user terminal (not shown) may download and install an application program provided by the term clustering apparatus 100, and a term clustering menu may be provided through the installed application.

The user input receiving unit 160 may include any type of server, terminal, or device that transmits and receives data, contents, and various communication signals through a network, and has a function of data storage and processing, with a user terminal (not shown) .

A smart pad, a tablet PC, a wearable device, and the like, and a personal communication system (PCS), for example, ), GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication) -2000, CDMA , W-CDMA (W-Code Division Multiple Access), and Wibro (Wireless Broadband Internet) terminals, desktop computers, and smart TVs.

An example of a network for information sharing between the user input receiving unit 160 and a user terminal (not shown) includes a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a 5G network, a World Interoperability for Microwave Access ) Network, a wired / wireless Internet, a LAN (Local Area Network), a wireless LAN (Local Area Network), a WAN A near field communication network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, and the like.

Hereinafter, the operation flow of the present invention will be briefly described based on the details described above.

FIG. 9 is a flowchart illustrating a method for grouping unstructured text data for analysis of big data according to an embodiment of the present invention.

The term clustering method of unstructured text data for the big data analysis shown in FIG. 9 can be performed by the term clustering apparatus 10 described above. Therefore, even if omitted below, the description of the term clustering apparatus 10 can be equally applied to the description of the clustering method of the unstructured text data for the big data analysis.

In step S901, the term clustering apparatus 10 may select data from a data set included in the database and perform preprocessing.

In step S902, the term clustering apparatus 10 can separate the morpheme of the original term included in the preprocessed data and calculate the recommendation score of the original term.

In step S903, the term clustering apparatus 10 separates the phonemes of the original terms, performs similarity calculation between the original terms separated by the phonemes, and clusters the original terms whose similarity calculation value is equal to or greater than a predetermined threshold value .

In step S904, the term clustering apparatus 10 may include a step of determining a recommended term among a plurality of original terms based on the recommendation score and the clustering result.

In the above description, steps S901 to S904 may be further divided into additional steps, or combined in fewer steps, according to embodiments of the present disclosure. Also, some of the steps may be omitted as necessary, and the order between the steps may be changed.

The term clustering method of unstructured text data for analysis of big data according to an embodiment of the present invention may be implemented in a form of a program command which can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The term clustering method of unstructured text data for the above-described big data analysis can also be implemented in the form of a computer program or an application executed by a computer stored in a recording medium.

It will be understood by those of ordinary skill in the art that the foregoing description of the embodiments is for illustrative purposes and that those skilled in the art can easily modify the invention without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

100: Term clustering device
110: Database
120: Data preprocessing section
130: Reference Glossary
140: Data cluster part
150:
160: User input receiver

Claims (10)

A term clustering apparatus for unstructured text data for analysis of big data,
A database containing a dataset;
A data preprocessing unit for selecting data from a data set included in the database and performing preprocessing;
A recommendation term determination unit for separating morphemes of original terms included in the preprocessed data and calculating recommendation scores of original terms; And
A data cluster unit for separating the phonemes of original terms, performing a similarity calculation between original terms separated by phonemes, and clustering original terms having a similarity calculation value equal to or greater than a preset threshold value,
, ≪ / RTI &
The recommendation term determination unit may determine recommendation terms among a plurality of original terms based on the recommendation score and the result of the grouping, and use a value obtained by digitizing the extraction frequency of the separated morpheme and a weight extracted based on the separated morpheme Wherein the recommendation score is calculated by weighting the weight of the morpheme in the original term. The method according to claim 1,
The pre-
Determining a first data set to perform clustering of terms among the data sets included in the database, selecting a first column for clustering terms among the plurality of column items of the first data set, The term clustering device. delete delete The method according to claim 1,
The recommendation score,
The sum of the extraction frequency of the separated morpheme based on the value obtained by digitizing the extraction frequency of the separated morpheme and the result of dividing the length of each of the plurality of morphemes classified in the first original term by the total length of the first original term Is computed using the term clustering. The method according to claim 1,
The data-
Wherein when the original term is Hangul, it is divided into phonemes according to Korean alphabet with a prefix, a neutral, and a longitudinal, and the similarity is calculated using an artificial intelligence based algorithm. The method according to claim 1,
And a data result unit for providing a cluster result of the data cluster unit,
Wherein the clustering result includes a recommendation term, an original term, and a similarity value. The method according to claim 1,
And a user input receiving unit for receiving term clustering performance information from the user terminal. 9. The method of claim 8,
The term "
Wherein morphologic separation is performed based on the preprocessed data based on part-of-speech decision information included in the term clustering performance information when morpheme is separated. A method for grouping unstructured text data for big data analysis,
Selecting data from a data set included in the database and performing preprocessing;
Separating morphemes of original terms included in the preprocessed data and calculating recommendation scores of original terms; And
Separating the phonemes of the original terms, performing a similarity calculation between the original terms separated by the phonemes, and clustering the original terms having the similarity calculation value equal to or greater than a predetermined threshold value;
Determining a recommendation term among a plurality of original terms based on the recommendation score and the clustering result,
The step of calculating the recommendation score of the original term may include:
The recommendation score is calculated using the numerical value of the extraction frequency of the separated morpheme and the weight extracted based on the separated morpheme,
Wherein the weights are values obtained by scaling the weight of the morphemes in the original term.

Images