KR20190052437A - Method for analyzing unstructured data and apparatus using the same - Google Patents

Abstract

According to one embodiment of the present invention, provided are a method and an apparatus for analyzing data. The method for analyzing data comprises the steps of: collecting atypical data related to a plurality of predefined entities; analyzing the collected atypical data to extract at least one word; comparing the at least one extracted word to calculate scores indicating a comparison result by the at least one word; and determining ranking for the plurality of entities based on the calculated scores.

Description

[0001] METHOD FOR ANALYZING UNSTRUCTED DATA AND APPARATUS USING THE SAME [0002]

The present invention relates to a method and apparatus for analyzing atypical data.

As the IT related technologies such as computers, smart phones, and telecommunication are developed rapidly, a large amount of information is generated in various fields, and the amount of information is expected to continuously increase. Such a large amount of information is a resource that can be used in various fields such as forecasting the near future or creating new business and creating new value, so it is positioned as an independent industrial field of big data industry rather than simple data . In recent years, online social services based on high-speed networks and large-scale multimedia services have rapidly developed, and interest in data analysis using unstructured data generated in online social services and large-capacity multimedia services is increasing. Unstructured data is unformatted data such as text documents, images, moving images, audio data, and the like. Such large-volume unstructured data is continuously generated not only in the online but also in various industrial fields such as finance, communication, and electric power.

Generally, in data analysis using unstructured data, at least one document related to an entity, which is a meaningful word determined in advance, is used to determine which keyword is used frequently through prior learning, and attributes are given to the entity according to the determination result. For example, an entity may include a specific person, a brand, a commodity, etc., and the document may include a webpage, an SNS (social network service), an e-mail and an application related to the entity .

In this case, although it is possible to provide desired information for each entity, there is a problem that the user can not provide such information even if he wants to know which of the various entities is popular with the public.

SUMMARY OF THE INVENTION The present invention provides a data analysis method and apparatus for solving the above problems.

In particular, it is an object of the present invention to provide a data analysis method and apparatus capable of analyzing atypical data related to a plurality of entities and determining ranking of each entity.

The problems of the present invention are not limited to the above-mentioned problems, and other problems not mentioned can be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, a data analysis method according to an embodiment of the present invention is provided. A data analysis method includes collecting unstructured data associated with a plurality of predefined entities, extracting one or more words by analyzing the collected unstructured data, comparing the extracted one or more words, Calculating a score indicating a result of the comparison, and determining a ranking for the plurality of entities based on the calculated score.

According to an aspect of the present invention, there is provided an apparatus for analyzing data according to an embodiment of the present invention. The data analysis apparatus includes a data collection unit for collecting unstructured data related to a plurality of predefined entities, a word extraction unit for analyzing the collected irregular data and extracting one or more words, A score calculation unit for calculating scores indicating comparison results of one or more words, and a ranking determination unit for determining a ranking for a plurality of entities based on the calculated scores.

The details of other embodiments are included in the detailed description and drawings.

The present invention provides a method and system for collecting unstructured data for a plurality of entities and analyzing the collected unstructured data to determine the rankings of the entities so that the popularity of each entity in the unstructured data can be conveniently and easily confirmed .

The effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the specification.

1 is a schematic diagram for explaining a data analysis system according to an embodiment of the present invention.
2 is a schematic diagram for explaining a data analysis apparatus according to an embodiment of the present invention.
3 is a schematic flowchart for explaining a data analysis method according to an embodiment of the present invention.
4 is a schematic flowchart for explaining a method of calculating a score according to an embodiment of the present invention.
5 is a schematic flow diagram illustrating a method for determining ranking for a plurality of entities in accordance with an embodiment of the present invention.
6A, 6B, 6C, 6D, 6E and 6F are diagrams for explaining a method of analyzing data according to an embodiment of the present invention.
7 is a schematic diagram for explaining a data analysis apparatus according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. It should be understood, however, that the invention is not limited to the disclosed embodiments, but is capable of many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

Although the first, second, etc. are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are used only to distinguish one component from another. Therefore, it is needless to say that the first component mentioned below may be the second component within the technical spirit of the present invention.

Like reference numerals refer to like elements throughout the specification.

It is to be understood that each of the features of the various embodiments of the present invention may be combined or combined with each other partially or entirely and technically various interlocking and driving is possible as will be appreciated by those skilled in the art, It may be possible to cooperate with each other in association.

In this specification, a data analysis apparatus according to an embodiment of the present invention is not limited, and may include any apparatus capable of analyzing data. For example, the data analysis device may include a general purpose computer, a laptop, a network-connected storage, a data server, a mobile device (e.g., a smart phone, a tablet device), and the like. In addition, software for data analysis is installed in the data analysis apparatus, and data analysis can be performed through the loaded software. Hereinafter, for convenience of explanation, the data analysis apparatus will be described as a data server that analyzes data and provides results, but the present invention is not limited thereto and the manner in which the present invention is implemented may vary.

Various embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

1 is a schematic diagram for explaining a data analysis system according to an embodiment of the present invention.

Referring to FIG. 1, a data analysis apparatus 100 includes a data analysis apparatus 100 for collecting and analyzing atypical data related to a plurality of entities through a communication network 120 and providing analysis results, and analysis results At least one receiving client device 110 is shown. The entity may be a word having a predefined meaning, for example, a specific person, a brand, a commodity, or the like. The entity is not limited to only the words mentioned by way of example, but may be any meaningful word.

The data analysis apparatus 100 collects atypical data from at least one document associated with each entity, and analyzes the collected atypical data to extract one or more words (or natural language). The document may be a file, a combination of files, or one or more files having a link embedded in another file. The one or more files may be of any type, such as text, HTML, XML, audio, image, video, For example, the document may include web pages, SNS, email and applications.

The data analysis apparatus 100 compares the extracted at least one word to calculate comparison data, and determines a ranking between a plurality of entities based on the calculated comparison data. For example, the comparison data may include data indicating the frequency of each word and the similarity between the words. The data analysis apparatus 100 may provide analysis results including ranking information for a plurality of entities to at least one client 110.

2 is a schematic diagram for explaining a data analysis apparatus according to an embodiment of the present invention. 2, the data analysis apparatus 100 includes a data analysis unit 100, a data collection unit 102, a word extraction unit 104, a score calculation unit 106, and a ranking determination unit 108 .

The data collection unit 102 collects unstructured data related to a plurality of entities. Each entity may be pre-matched to at least one document, and atypical data may be collected from the at least one document. For example, the first brand may match the first document and the second document, and the data collection unit 102 may collect the atypical data for the first brand from the first document and the second document. The atypical data may include, for example, articles and comments disclosed on a blog if the first document is a blog related to the first brand, and inquiries disclosed on the website if the second document is a website related to the first brand Articles, reviews, and the like.

According to various embodiments of the present invention, when collecting irregular data, the data collecting unit 102 acquires at least one of the type of irregular data, the location, information on the creator, the collection date, You can collect more metadata that you include.

The word extracting unit 104 extracts one or more words by analyzing the collected irregular data. For example, the word extracting unit 104 may extract one or more words from posts, comments, and the like of a blog related to the first brand.

The score calculation unit 106 compares the extracted one or more words and calculates a score indicating a comparison result for each word. Specifically, the score calculation unit 106 classifies the extracted one or more words into topics, calculates a first score indicating a frequency for each of the one or more classified words, and a second score indicating a correlation between the classified one or more words do. The score calculation unit 106 calculates a first score for each word by using a topic modeling technique such as a Latent Dirichlet Allocation (LDA) technique, and calculates a score using a word index technique such as a TF-IDF (Term Frequency-Inverse Document Frequency) The second score for each word can be calculated.

For example, when word 1, word 2, and word 3 are classified in correspondence with the first topic, the ranking determining unit 108 determines the first score of the word 1 indicating the frequency of the word 1, the word indicating the frequency of the word 2 The second score of 2 and the third score of word 3 representing the frequency of word 3 can be calculated. The ranking determining unit 108 compares the word 1 and the word 2 to calculate a score 1-1 indicating a relationship between the word 1 and the word 2 and compares the word 1 and the word 3 to show a relation between the word 1 and the word 3 And calculates the score 1-2. The ranking determining unit 108 compares the word 2 with the word 1 to calculate a score 2-1 indicating a relationship between the word 2 and the word 1 and compares the word 2 with the word 3 to indicate a relationship between the word 2 and the word 3 And calculates score 2-2. The ranking determining unit 108 compares the word 3 with the word 1 to calculate a score 3-1 indicating the correspondence between the word 3 and the word 1 and compares the word 3 with the word 2 to determine a relationship between the word 3 and the word 2 3 " The second score of the word 1 is the score obtained by adding the score 1-1 and the score 1-2, the second score of the word 2 is the score obtained by adding the score 2-1 and the score 2-2, The score can be a score that is the sum of the score 3-1 and the score 3-2.

Next, the ranking determining unit 108 calculates a third score obtained by adding the first score and the second score to one or more words. According to various embodiments of the present invention, the ranking determiner 108 may calculate the third score using an equation such as " Ts = a x Fs + b x Ss 1.97 ". Here, Ts is a third score, a and b are weights, Fs is a first score, and Ss may be a second score.

The ranking determining unit 108 calculates a final score obtained by summing the third scores calculated for one or more words corresponding to each of a plurality of entities, and determines a ranking for a plurality of entities based on the calculated final score . For example, if one or more words corresponding to entity 1 are word 1, word 2, and one or more words corresponding to entity 2 are word 3, word 4, ranking determination section 108 determines the third score of word 1 And the third score of word 2 may be summed to yield a final score for entity 1 and a third score of word 3 and a third score of word 4 may be summed to yield a final score for entity 2. If the final score of entity 1 is greater than the final score of entity 2, then entity 1 can be determined to rank higher than entity 2.

3 is a schematic flowchart for explaining a data analysis method according to an embodiment of the present invention. Referring to FIG. 3, the data analysis apparatus 100 collects unstructured data related to a plurality of predefined entities (S300). For example, atypical data may be collected from at least one document matched to each of a plurality of entities. Such unstructured data may be posts or comments collected from web pages corresponding to at least one document, email content collected from emails, posts collected from SNS, or posts or comments collected from applications. According to various embodiments, meta data of irregular data may also be collected when collecting unstructured data. For example, the meta data of unstructured data may be a location such as a post, comment or email content, information about the author, date of collection, date the post, comment, or email content was created,

Next, the data analysis apparatus 100 analyzes the collected irregular data and extracts one or more words (S310). For example, the data analysis apparatus 100 may extract one or more words by analyzing a post or comment of a web page, an e-mail content, a post of an SNS, or a post or comment of an application.

The data analysis apparatus 100 compares the extracted one or more words and calculates a score indicating a result of comparison for one or more words (S320). Hereinafter, a method of calculating a score will be described in more detail with reference to FIG. 4 is a schematic flowchart for explaining a method of calculating a score according to an embodiment of the present invention.

First, the data analysis apparatus 100 classifies the extracted one or more words into topics (S400). For example, word 2, word 6 and word 10 are classified corresponding to the first topic, word 3, word 4 and word 9 are classified corresponding to the second topic, word 1 corresponding to the third topic, word 8 and word 5 can be classified.

The data analysis apparatus 100 calculates a first score indicating a frequency of each word classified by topic (S410), compares the words classified by the topic, and calculates a second score indicating a relation between words in each word (S420). For example, the data analysis apparatus 410 compares the word 2 and the word 6 classified corresponding to the first topic, and calculates a score indicating the association between the word 2 and the word 6 and a score indicating the association between the word 2 and the word 10 , And calculate the second score of the word 2 by summing the calculated scores.

Referring again to FIG. 3, the data analysis apparatus 100 may determine a ranking for a plurality of entities based on the calculated scores (S330). Hereinafter, with reference to FIG. 5, a method of determining ranking for a plurality of entities will be described in more detail. 5 is a schematic flow diagram illustrating a method for determining ranking for a plurality of entities in accordance with an embodiment of the present invention.

First, the data analysis apparatus 100 calculates a third score obtained by summing the first score and the second score calculated for at least one word (S500). For example, a third score obtained by summing the first score and the second score of the word 2 can be calculated.

The data analysis apparatus 100 calculates a final score by summing a third score for at least one word corresponding to each of the plurality of entities (S510). For example, if entity 1 corresponds to document 1 and document 2, the third scores of words 2, 6, and 10 extracted from document 1 and the third scores of words 3, 4, and 9 extracted from document 2 are summed And can be calculated as the final score of entity 1.

The data analysis apparatus 100 may determine a ranking for a plurality of entities based on the calculated final scores (520). For example, the data analysis apparatus 100 may compare the final scores of Entity 1, Entity 2 and Entity 3 to determine the highest ranking entities with the highest score.

6A, 6B, 6C, 6D, 6E and 6F are diagrams for explaining a method of analyzing data according to an embodiment of the present invention. 6A, a data analysis apparatus 100 includes a plurality of documents (e.g., Document 1, Document 2, Document 3, Document 4, and Document 2) associated with a plurality of entities (e.g., Entity 1, Entity 2, Entity 3) Document 5). For example, atypical data was "Word 1 had word 2, word 3 had word 4", "word 4, word 8, word 9 had word 10", "word 5 had word 6 and word 8 was word 9 . "," Word 1 and word 3 are word 5, word 6 ".

Referring to FIG. 6B, the data analysis apparatus 100 extracts one or more words from each of a plurality of documents. For example, " word 1, word 2, word 3, word 4 " is extracted from document 1, " word 1, word 3, word 5 and word 6 " is extracted from document 2, Word 2, word 3, word 4, and word 6 " are extracted from document 4, and words " 1, 9, 8, 3, Word 2 " can be extracted.

Referring to FIG. 6C, the data analysis apparatus 100 classifies the extracted one or more words into topics, and calculates a first score indicating frequency of each of the words classified by the topics. For example, " word 2, word 6, word 10 " is categorized as the first topic, " word 3, word 4, May be classified as the third topic. The first score of word 2 classified as the first topic is "0.51", the first score of word 6 is "0.26", the first score of word 10 is "0.14", and the first score of word 3 1 score of "0.46", the first score of word 4 is "0.23", the first score of word 9 is "0.23", the first score of word 1 classified as the third topic is "0.39" The first score may be " 0.27 " and the first score of word 5 may be " 0.14 ".

Referring to FIG. 6D, the data analysis apparatus 100 calculates a comparison score by comparing each word classified according to a topic, and calculates a second score obtained by summing the calculated comparison score for each word. For example, if the comparison score obtained by comparing word 2 and word 6 in correspondence with the first topic is " 0.59 ", the comparison score obtained by comparing word 2 and word 10 is " 0.80 ", and word 6 and word 2 are compared The comparative score obtained by comparing the word 2 and the word 6 is " 0 ", the comparison score obtained by comparing the word 10 and the word 2 is " 0.80 "Quot; 0 ". In this case, the second score of word 2 may be "1.39", the second score of word 6 may be "0.59", and the second score of word 10 may be "0.80". Further, in a case where the comparison score obtained by comparing the word 3 and the word 4 in correspondence to the second topic is " 0.54 ", the comparison score obtained by comparing the word 3 and the word 9 is " 0.55 ", and the comparison score Is 0.55, the comparison score obtained by comparing the word 4 and the word 9 is " 1.20 ", the comparison score obtained by comparing the word 9 and the word 3 is " 0.55 ", and the comparison score obtained by comparing the word 9 and the word 4 is " 1.20 ". In this case, the second score of the word 3 may be "1.09", the second score of the word 4 may be "1.74", and the second score of the word 9 may be "1.75". Next, a comparison score obtained by comparing word 1 and word 8 corresponding to the third topic is " 0.64 ", a comparison score obtained by comparing word 1 and word 5 is " 0.64 &Quot; is 0.64, the comparison score obtained by comparing word 8 and word 5 is " 0.16 ", the comparison score obtained by comparing word 5 and word 1 is " 0.64 ", and the comparison score obtained by comparing word 5 and word 8 May be " 0.16 ". In this case, the second score of word 1 may be "1.28", the second score of word 8 may be "0.80", and the second score of word 5 may be "0.80".

Referring to FIG. 6E, the data analysis apparatus 100 calculates a third score obtained by summing the first score and the second score. For example, the third score of word 2 is "1.09", the third score of word 6 is "0.85", the third score of word 10 is "0.94", the third score of word 3 is "1.55" 1.97, " the third score of word 9 is " 1.98 ", the third score of word 1 is " 1.67 ", the third score of word 8 is & The third score of word 5 may be " 0.94 ".

Referring to FIG. 6F, the data analysis apparatus 100 calculates the sum of the third scorers of one or more words corresponding to each of a plurality of entities as a final score, and calculates a rank for each entity based on the calculated final score . For example, " 9.88 " which is the sum of the third scores of word 1, word 2, word 3, word 4, word 5 and word 6 corresponding to entity 1 is calculated as the final score of entity 1, 10.26 ", which is the sum of the third scores of word 2, word 3, word 4, word 6, word 8 and word 10 is calculated as the final score of entity 2, and word 1, word 2, word 3.95, " 7.95 " which is the sum of the third scores of word 3, word 6, word 8 and word 9 can be calculated as the final score of entity 3. In this case, the first ranking among the plurality of entities is the entity 2, the second ranking is the entity 1, and the third ranking is the entity 3.

7 is a schematic diagram for explaining a data analysis apparatus according to an embodiment of the present invention. Referring to FIG. 2, the data analysis apparatus 700 includes a communication unit 710, a storage unit 720, and a processor 730.

The communication unit 710 connects the data analysis apparatus 700 so that it can communicate with the external apparatus. For example, the communication unit 710 may connect to at least one external device through a wireless network and receive atypical data from at least one external device.

The storage unit 720 may be configured to store the irregular data received through the communication unit 110 and to store various data used for analyzing the irregular data. For example, the storage unit 720 may store meta data of irregular data or an algorithm for analyzing unstructured data.

Processor 730 is operatively coupled to communication unit 710 and storage unit 720 and performs various instructions for analyzing unstructured data. Hereinafter, an operation of analyzing the unstructured data will be briefly described. The processor 730 collects unstructured data from at least one document associated with each of the plurality of entities received via the communication unit 110, and analyzes the collected unstructured data to extract one or more words. Next, the processor 730 compares the extracted one or more words to calculate a score indicating the result of the comparison, and determines a ranking for each of the plurality of entities based on the calculated score. The determined ranking may be provided to at least one client through the communication unit 110. [

Thus, the present invention can easily check the popularity of each entity in the unstructured data by collecting the unstructured data for a plurality of entities and analyzing the collected unstructured data to determine the ranking of each entity.

The apparatus and method according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination.

Program instructions to be recorded on a computer-readable medium may be those specially designed and constructed for the present invention or may be available to those skilled in the computer software arts. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. The above-mentioned medium may also be a transmission medium such as a light or metal wire, wave guide, etc., including a carrier wave for transmitting a signal designating a program command, a data structure and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the embodiments of the present invention have been described in detail with reference to the accompanying drawings, it is to be understood that the present invention is not limited to those embodiments and various changes and modifications may be made without departing from the scope of the present invention. . Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, it should be understood that the above-described embodiments are illustrative in all aspects and not restrictive. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

100: Data analysis device
102: Data collecting unit
104: word extracting unit
106: score calculation unit
108: Ranking decision unit
110: at least one client device
120: communication network

Claims (15)

Collecting atypical data related to a plurality of predefined entities;
Analyzing the collected atypical data to extract one or more words;
Comparing the extracted one or more words to calculate a score indicating a comparison result for each of the one or more words; And
And determining a ranking for the plurality of entities based on the calculated score. The method of claim 1, wherein the atypical data collection step comprises:
And collecting the atypical data from at least one document that is pre-matched with the plurality of entities. The method of claim 1, further collecting metadata of the atypical data when collecting the atypical data. 2. The method according to claim 1,
Classifying the one or more words by topic;
Calculating a first score indicative of a frequency for the one or more words classified; And
And calculating a second score indicative of the association between the one or more words sorted. 5. The method according to claim 4,
Calculating a third score obtained by adding the first score and the second score to the one or more words;
Computing a final score summing the third score for the one or more words corresponding to each of the plurality of entities; And
And determining a ranking for the plurality of entities based on the calculated final score. 5. The method of claim 4, wherein the first score calculation step uses a topic modeling technique. 5. The method of claim 4, wherein the second score calculation step uses a word index technique. A data collection unit for collecting irregular data related to a plurality of predefined entities;
A word extracting unit for analyzing the collected irregular data and extracting one or more words;
A score calculation unit for comparing the extracted one or more words to calculate a score indicating a comparison result for each of the one or more words; And
And a ranking determiner for determining a ranking for the plurality of entities based on the calculated score. 9. The data processing apparatus according to claim 8,
And collecting the atypical data from at least one document that is pre-matched with the plurality of entities. 9. The data processing apparatus according to claim 8,
And further collects the meta data of the atypical data when collecting the atypical data. 9. The apparatus according to claim 8,
Classify the one or more words by topics, calculate a first score that represents the frequency for the one or more classified words, and calculate a second score that represents the association between the one or more classified words. 12. The apparatus according to claim 11,
Calculating a third score by summing the first score and the second score for the one or more words, and calculating a final score summing the third score for the one or more words corresponding to each of the plurality of entities And determines a ranking for the plurality of entities based on the calculated final score. 12. The apparatus according to claim 11,
Wherein the first score is calculated using a topic modeling technique. 12. The apparatus according to claim 11,
And calculates the second score using a word index technique. Collecting atypical data related to a plurality of predefined entities,
Analyzing the collected atypical data to extract one or more words,
Comparing the extracted one or more words to calculate a score indicating a relation between the one or more words,
And determining a ranking for the plurality of entities based on the calculated score. ≪ Desc / Clms Page number 21 >

Images