KR20190047938A - Method and apparatus for collecting and analyzing text data with user interface for data analysis - Google Patents

Abstract

The present invention relates to a method for collecting and analyzing text data, which comprises the following steps: receiving input related to a keyword and period information; calculating the number of articles posted on a web including the keyword on the basis of the keyword and the period information; calculating the number of web pages which can be crawled based on the number of the articles; providing the calculated number of the web pages; receiving an input related to a range of the web pages to be crawled; crawling the web pages including the articles including the keyword within the received range of the web pages; collecting text data of the articles included in the crawled web pages; pre-processing the collected text data on the basis of a preset dictionary-defined word; and performing text data analysis on the basis of a preprocessed result.

Description

[0001] METHOD AND APPARATUS FOR COLLECTING AND ANALYZING TEXT DATA WITH USER INTERFACE FOR DATA ANALYSIS [

The present invention relates to a text data collection and analysis method and apparatus including a user interface for data analysis.

Recently, demand for intelligent services such as civil affairs analysis, news curation, preference survey, and recommendation is increasing rapidly. Most of these services collect and analyze textual data contained in web publishing articles or posts to provide users with the necessary information.

For example, people's concerns about the human health hazards of electromagnetic waves emitted from mobile phones, tablet PCs, Wi-Fi, and living wireless devices and wireless communication technologies are rapidly increasing. Cases and lawsuits related to electromagnetic waves are increasing worldwide, and it is urgent to analyze the causes of complaints and to provide effective countermeasures. However, the existing countermeasures for civil complaints related to electromagnetic waves are difficult to respond quickly to electromagnetic wave related civil complaints, which are currently a social issue, because only the responses to local complaints are taken into consideration or only the general harmfulness to electromagnetic waves is considered. It may not be a suitable countermeasure. In order to solve these problems, countermeasures that reflect the content analysis of civil affairs related to the current issue should be essential. However, existing civil affairs analysis may take a long time because it depends on the experts, and it may not be possible to derive accurate analysis results according to the proficiency of the experts. As a result, it may take a long time, a lot of money and manpower to solve the anxiety of raised people, and it may fail to respond to complaints due to inaccurate analysis results.

In order to provide a seamless intelligent service, it is essential to analyze various kinds of articles displayed on the web in real time, to provide web crawling technology for collecting text data, and to provide a big data analysis technique for analyzing collected data.

The background technology of the present invention is disclosed in Korean Patent Laid-Open Publication No. 2010-0094263 (published on Aug. 26, 2010).

The present invention has been made to solve the above-mentioned problems of the conventional art, and it is an object of the present invention to provide a method and apparatus for receiving a keyword from a user, collecting articles on internet related to input keywords using web crawling, And to provide a method of collecting and analyzing text data including a database capable of storing and outputting analysis results.

According to another aspect of the present invention, there is provided a text data collecting and analyzing method for visualizing a keyword text data collection and analysis result in various forms and providing the same to a user based on a keyword input from a user, .

It is another object of the present invention to provide an integrated framework that considers both text data collection technology and data analysis technology.

It should be understood, however, that the technical scope of the embodiments of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to one embodiment of the present invention, a method for collecting and analyzing text data includes receiving input regarding keyword and period information. Calculating the number of articles displayed on the web including the keyword based on the keyword and the period information, calculating the number of crawlable web pages based on the number of articles, Receiving an input relating to a range of a web page to be crawled, crawling a web page including an article containing the keyword within the range of the input web page, Collecting text data of an article included in the web page, pre-processing based on a predefined word set for the collected text data, and performing text data analysis based on the pre-processed result can do.

According to one embodiment of the present invention, the period information may be a publication period of an article.

According to an embodiment of the present invention, the step of collecting text data of an article included in the crawled web page can be performed by repeatedly crawling the web page within the range of the inputted web page.

According to one embodiment of the present application, the preprocessing step includes the steps of: receiving an input relating to the predefined predefined word; comparing an article containing the predefined predefined word with an article that does not include the predefined predefined word And removing the predetermined removed text element from the text data of the article including the predetermined predefined word.

According to one embodiment of the present application, a method for collecting and analyzing text data further comprises receiving an input regarding a word frequency ranking and forming a base data set to be used for text data analysis based on the preprocessed text data Wherein the base data set includes a cognate set included in the preprocessed text data, a frequent word set within the word frequency rank among the words included in the cognate set, and a matrix data set . ≪ / RTI >

According to one embodiment of the present invention, the step of performing the text data analysis may include analyzing the frequency of the words used in the article and analyzing association rules between words in the article.

According to one embodiment of the present disclosure, the step of performing the text data analysis comprises the steps of: determining a first word set that includes word combinations that can be made up of frequent words in the matrix data set, Comprising the steps of: receiving an input relating to a second set of words comprising a word; and, if the word combination contained in the first set of words is included in an article, And determining a rule.

According to one embodiment of the present application, the step of performing the text data analysis may include receiving an input regarding minimum support and minimum reliability for the association rule, and receiving support for at least the minimum support and minimum reliability among the determined association rules. And outputting an analysis result of the association rule having reliability.

According to an embodiment of the present invention, the step of performing the text data analysis may include receiving an input regarding a minimum frequency, performing frequency analysis by determining words of the minimum frequency among the words included in the cognitive sets, . ≪ / RTI >

According to one embodiment of the present application, a text data collection and analysis apparatus includes a user interface unit for receiving an input relating to a keyword and period information and receiving an input relating to a range of a web page to be crawled, Calculating the number of articles displayed on the web including the keyword based on the information, calculating the number of crawlable web pages based on the number of articles, providing the calculated number of web pages to the user interface unit A web crawler unit for crawling a web page including an article including the keyword within a range of a web page inputted by the user interface unit and a text data of an article included in the crawled web page is collected , Preprocessing the collected text data on the basis of a predefined word set in advance, And a data analysis unit for analyzing the text data based on the preprocessed result.

According to an embodiment of the present invention, the period information received by the user interface unit may be a publication period of an article.

According to an embodiment of the present invention, the web crawler unit can repeatedly perform the crawling of the web page within the range of the web page received by the user interface unit.

According to one embodiment of the present invention, the user interface unit receives an input relating to the predetermined predefined word, and the data analyzing unit includes an article including the predefined predefined word, And a preprocessing unit for classifying articles and eliminating predetermined absent text elements from the text data of the articles including the predetermined dictionary words to perform preprocessing on the collected text data.

According to an embodiment of the present invention, the user interface unit receives an input regarding a word frequency ranking, and the data analysis unit includes a data forming unit that forms a base data set to be used for text data analysis based on the preprocessed text data Wherein the base data set includes a cognate set included in the preprocessed text data, a frequent word set within the word frequency rank among the words included in the cognate set, and a matrix data set . ≪ / RTI >

According to an embodiment of the present invention, the data analysis unit may include an analysis unit for analyzing frequency of words used in an article based on the base data set and analyzing association rules between words in an article.

According to an embodiment of the present invention, the user interface unit receives an input relating to a second word set including at least one word of the frequent word set, and the analyzing unit generates the frequent words in the matrix data set Determining a first word set including a word combination that can be included in the second word set if the word combination included in the first word set is included in the article, .

According to an embodiment of the present invention, the user interface unit receives an input relating to a minimum support degree and a minimum reliability degree of the association rule, and the analyzing unit compares the support degree and reliability of the minimum support degree and the minimum reliability among the determined association rules You can output the analysis results of association rules.

According to an embodiment of the present invention, the user interface unit receives an input relating to a minimum frequency, and the analyzing unit may perform frequency analysis by determining words of the minimum frequency among the words included in the cognitive sets.

According to one embodiment of the present application, a text data collection and analysis system comprises a text data collection and analysis device and an input for the text data collection and analysis device about keywords and duration information, And a user terminal for receiving and outputting the result of the text data analysis.

The above-described task solution is merely exemplary and should not be construed as limiting the present disclosure. In addition to the exemplary embodiments described above, there may be additional embodiments in the drawings and the detailed description of the invention.

According to an embodiment of the present invention, a keyword is input from a user, an Internet-related article related to an input keyword is collected using a web crawl, a text data analysis related to an input keyword is performed, And providing a text data collection and analysis method that includes a database that can be stored in a database.

According to the present invention, there is provided a method of collecting and analyzing keyword text data and analyzing results of the keyword text data in various forms and providing the same to a user based on the keyword received from the user .

In addition, according to the task resolution means of the present invention, it is possible to collect the Korean text data of the web site, and to obtain the integrated framework through the analysis time reduction through the data analysis automation and the text data collection and analysis.

1 is a diagram showing a schematic configuration of a text data collection and analysis system according to an embodiment of the present invention.
2 is a block diagram showing a schematic configuration of a text data collection and analysis apparatus according to an embodiment of the present invention.
3 is a flowchart schematically illustrating a text data collection and analysis method according to an embodiment of the present invention.
FIG. 4 is a flowchart schematically illustrating the step of acquiring information of an article according to an embodiment of the present invention.
5 is a flow chart schematically illustrating collecting text data of an article according to one embodiment of the present application.
6 is an exemplary diagram illustrating a collected text data set according to one embodiment of the present application.
FIG. 7 is a flow chart outlining the collected text data preprocessing step according to one embodiment of the present application.
FIG. 8 is a diagram illustrating a text data classification process according to an embodiment of the present invention.
FIG. 9 is a diagram illustrating a process of removing a predetermined removed text element according to an embodiment of the present invention.
FIG. 10 is a flow chart schematically illustrating the steps of forming a base data set according to one embodiment of the present application.
11 is a schematic representation of a matrix data set according to one embodiment of the present invention.
12 is a flowchart schematically illustrating a step of performing text data analysis according to an embodiment of the present invention.
FIG. 13 is a diagram illustrating an example of outputting a word having a minimum frequency determined according to an embodiment of the present invention in word cloud form by determining output position, output size, and output color according to the frequency.
14 is a diagram illustrating an example of a group matrix output for outputting a relation between a first word set and a second word set in a matrix form according to an embodiment of the present invention.
FIG. 15 is a diagram illustrating an example of a graph output for outputting a network graph form showing a relation between a word belonging to a first word set and a word belonging to a second word set according to an embodiment of the present invention.
16 is a diagram showing an example of a distribution diagram output that graphically outputs the distribution of determined association rules according to an embodiment of the present invention.
17 is a diagram illustrating an example of a graph including extraction frequency and extraction probability according to an embodiment of the present invention.
FIG. 18 is a diagram schematically illustrating a request purpose according to a requestFlag value according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when a part is referred to as being "connected" to another part, it is not limited to a case where it is "directly connected" but also includes the case where it is "electrically connected" do.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as " including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when a part is referred to as being "connected" to another part, it is not limited to a case where it is "directly connected" but also includes the case where it is "electrically connected" do.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as " including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

The present invention relates to a text data collection and analysis method, which collects text data included in a web publication article or a post through web crawling from a user's input keyword, stores collected data for analysis in a database, The collected text is classified according to the predefined word, and the association rule analysis between the keyword and the extracted word is performed based on the classified word, and the analysis result can be provided to the user using the visualization function.

FIG. 1 is a diagram illustrating a schematic configuration of a text data collection and analysis system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a schematic configuration of a text data collection and analysis apparatus according to an embodiment of the present invention. Referring to FIGS. 1 and 2, a text data collection and analysis system may include a text data collection and analysis apparatus 100, a user terminal 200, and a web 300. The text data collection and analysis system collects Internet publication articles from the web 300 based on the keyword and period information and provides the related keyword text data collection and analysis results to the user terminal 200 based on the collected articles . The user terminal 200 may provide input to the text data collection and analysis apparatus 100 regarding keywords and period information and may receive and output the results of the text data analysis from the text data collection and analysis apparatus 100.

The user terminal 200 may be a personal communication system (PCS), a global system for mobile communication (GSM), a personal digital cellular (PDC), a Personal Digital Cellular (PHS) Handyphone System, PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication) -2000 terminal, CDMA (Code Division Multiple Access) -2000 terminal, W-CDMA (W-Code Division Multiple Access) terminal, Wibro ) Terminal and all kinds of wireless communication devices capable of processing and outputting data.

The text data collection and analysis apparatus 100, the user terminal 200, and the web 300 may be connected by a network. A network refers to a connection structure between a wireless network and a wireless network capable of exchanging information between nodes such as a terminal and a server. An example of such a network is a 3rd Generation Partnership Project (3GPP) network, an LTE (Long Term Evolution) 5G network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a LAN (Local Area Network), a wireless LAN (Local Area Network), a WAN Bluetooth) network, a satellite broadcast network, an analog broadcast network, a DMB (Digital Multimedia Broadcasting) network, and the like.

Referring to FIG. 1, information input from the user terminal 200 is transmitted to the text data collection and analysis apparatus 100, and text data related to input keywords is collected and analyzed by the text data collection and analysis apparatus 100 . The text data collection and analysis apparatus 100 may provide text data collection and analysis results visualized via the network to the user terminal 200. The text data collection and analysis apparatus 100 may collect Internet publication articles related to the keyword input from the web 300 based on the input keyword information of the user.

The text data collection and analysis apparatus 100 acquires article information including a keyword from the web based on the keyword and the term information, crawls the web page including the article based on the information of the article , Analyze the text data from the articles included in the crawled web page, and store the analysis results.

Referring to FIG. 2, the text data collection and analysis apparatus 100 may include a user interface unit 110, a web crawler unit 120, a data analysis unit 130, and a database 140. The web crawler unit 120 may include a parsing unit 121, an extracting unit 122, a language support unit 123, and a file generating unit 124. The data analysis unit 130 may include a preprocessing unit 131, a data forming unit 132, and an analysis unit 133. In addition, the database 140 may include a crawling database 141, an analysis database 142, and a predefined word database 143.

According to one embodiment of the present invention, the user interface unit 110 helps to set detailed values for web crawling and text data analysis by providing a GUI to the user, and provides the derived association rule analysis result and word cloud ) And output it to the user. Although not shown, the user interface unit 110 may include three functional blocks: 1) a User Input Panel, 2) a Result View Panel, and 3) a Pre-define Knowledge Panel. The User Input Panel can be used to set various options for data analysis, including settings for data collection, such as keyword, article delivery period, page and crawl range setting, output conditions, and minimum frequency setting for Word Cloud. The Result View Panel is used to provide visualized results after analyzing the data, and the Pre-define Knowledge Panel can be used to add / delete words stored in the Pre-define Knowledge DB.

The web crawler unit 120 searches for an article that can be collected on the condition of a specific keyword and an article publication period input from a user to extract a data set necessary for data analysis, and collects text data of all articles retrieved. The web crawler unit 120 generates URLs based on the keywords and option values input by the user, and searches the articles displayed on the web using the generated URLs. And then retrieves an HTML file containing the list of articles retrieved and the article body from the Web 300. In addition, the web crawler unit 120 performs encoding for Korean language support, extracts text data corresponding to the title and content of the article of the HTML file, and generates and stores the extracted text data in a txt format.

The data analysis unit 130 performs data preprocessing based on the data set collected as a result of the web crawler unit 120 and forms matrix data for association rule analysis. Then, the association rule analysis is performed, and the analysis result can be visualized in various forms. The data analysis unit 130 classifies the articles to be analyzed by comparing the data of the collected articles with preset words. In addition, a cognate (noun) can be extracted from the body of the classified article, and a data set for association rule analysis can be formed based on words having a high frequency among the extracted words. Thereafter, the data analysis unit 130 performs an association rule analysis using the Apriori algorithm, and visualizes the analysis results in the form of Word Cloud, Grouped Matrix, Graph, Scatter Plot, and Extraction Probability.

The database 140 may store the data set extracted by the web crawler unit 120 and may store the analysis result obtained through the data analysis unit 130 and the visualized image. In particular, it stores predefined words defined by the user for an article classification operation performed in the data preprocessing process, and stores a data set for pre-processing results and association analysis. The database 140 includes 1) a crawling database, 2) an analysis database, and 3) a predefined word database. The crawl database is used for storing and managing the data collected through the web crawler unit 120. The analysis database is used for storing and managing data analysis results, and the predefined word database is composed of user defined predefined words Stored and managed.

The description of each part of the text data collection and analysis apparatus 100 will be described in detail below.

3 is a flowchart schematically illustrating a text data collection and analysis method according to an embodiment of the present invention. The description of each step shown in Fig. 3 will be described in detail with reference to Figs. 4 to 12. Fig.

Referring to FIG. 3, in step S310, the user interface unit 110 transmits keyword and period information to the web crawler unit 120, and the web crawler unit 120 can receive keyword and period information. For example, the period information may be the publication period of an article.

In step S320, the web crawler unit 120 can acquire information of an article including a keyword from the web based on the keyword and period information. Further, in step S330, the web crawler unit 120 may crawl the web page including the article based on the information of the article. In step S320 and step 330, the web crawler unit 120 generates a URL including keyword and period information, transmits the URL to the web, and receives the HTML file including the search result from the web.

In step S340, the web crawler unit 120 may transmit the text data of the articles included in the web page crawled by the data analysis unit 130. [ As another example, in step S340, the data analysis unit 130 may read the text data of the articles collected by the web crawler unit 120 from the database.

In step S350, the data analysis unit 130 may preprocess the crawled web page based on a predefined word. More specifically, the data analysis unit 130 may preprocess text data of an article included in a web page crawled based on a predefined word. The data analysis unit 130 may classify the text data of the article including the predetermined predefined word and the text data of the article not including the predetermined predefined word. In addition, the data analysis unit 130 may remove a predetermined removed text element from the text data of the article including the preset predefined word.

In step S360, the data analysis unit 130 may form a base data set to be used for data analysis from the articles included in the crawled web page. More specifically, the data analysis unit 130 may form a base data set from the text data of the articles of the preprocessed web page. The data analysis unit 130 may form an underlying data set for the text data of the article from which the preset text removal element has been removed. According to one embodiment of the present invention, the base data set includes a set of chords included in an article from which a preset text removal element is removed, a frequent word set within a predetermined rank, and matrix data And a set.

In step S370, the data analysis unit 130 may analyze the text data based on the base data set. According to one embodiment of the present invention, the data analysis unit 130 may analyze the text data including the frequency analysis of the words used in the article and the association rule analysis between words in the article. Also, the data analysis unit 130 may output the results of the frequency analysis and the results of the association rule analysis in different graphics.

In step S380, the database 140 may store the result of the text data analysis. According to one embodiment of the present invention, the database 140 may store the result of the text data analysis in association with the result of the frequency analysis and the identifier of the result of the association rule analysis. For example, the identifier of the result of an association rule analysis may include Word Cloud, Grouped Matrix, Graph, Scatter Plot, Extraction Probability, and the like.

FIG. 4 is a flowchart schematically illustrating the step of acquiring information of an article according to an embodiment of the present invention.

Referring to FIG. 4, in step S410, the user interface unit 110 may receive input regarding keyword and period information. Period information may be the publication period of an article. For example, a user can be set to perform a search without restrictions on the article publication period, and if a user wants to search for articles published within a certain period of time, the start date (year, month, day) , Day) can be input respectively to set the publication period. For example, the user interface unit 110 may enable the user to set the start date and the end date of the publication period using the calendar UI.

In step S420, the user interface unit 110 may transmit a request flag (RequestFlag) including request information to the web crawler unit 120 together with keyword and period information. Illustratively, referring to FIG. 18, request 1 may be a request for the number of articles retrieved. The user interface unit 110 can transmit a request flag including request 1 (request 1). That is, the user interface unit 110 may transmit the keyword, the period information, and the request 1 to the web crawler unit 120.

The web crawler unit 120 may calculate the number of retrieved articles for the purpose of request 1 received from the user interface unit 110. [

In step S430, the parsing unit 121 of the web crawler unit 120 may generate a URL including keywords, period information, and information of a web page. For example, when the period information is not specified, the parsing unit 121 can generate a URL for obtaining an article search result without limiting the publication period of the article. In another example, if the start date and the end date of the period information are set, the parsing unit 121 may generate a URL for obtaining an article search result based on a publication period set to a specific value.

In step S440, the parsing unit 121 may transmit the generated URL to the Web 300, and may request the Web 300 for the HTML file including the search result on the Web 300. [

In step S450, the web 300 generates an HTML file including keyword and period information. In step S460, the web 300 may transmit an HTML file including the search result to the web crawler unit 120. [

In step S470, the parsing unit 121 may reconstruct the HTML file received from the Web 300 in a tree form. The extracting unit 122 may extract data having a predetermined attribute value (e.g., span class = " result_num ") associated with extracting the number of articles from the reconstructed HTML file.

In step S480, the extracting unit 122 may remove the blank and special characters from the extracted data and count the number of articles. Illustratively, the web crawler unit 120 can transmit the number of retrieved articles to the user interface unit 100 in the form of "total: number of articles". The web crawler unit 120 can transmit the search result to the user interface unit 120 in the form of " total: NA "

In step S490, the web crawler unit 120 may calculate the number of web pages that can be crawled based on the number of articles. For example, one web page may include at least one article. The web crawler unit 120 can transmit the number of the maximum crawlable web pages to the user interface unit 120 by expressing the number of the maximum crawlable web pages as numerals if the result of the article including the keyword is present based on the keyword and the period information. If the search result does not exist, the web crawler unit 120 may transmit the search result to the user interface unit 120 in the form of a text.

5 is a flow chart schematically illustrating collecting text data of an article according to one embodiment of the present application.

The web crawler unit 120 can crawl a web page including an article including a keyword within a range of a web page input by a user or the like in the user interface unit 110. [

In step S510, the user interface unit 110 can receive a crawling range input by a user or the like. The crawling range means a range of a web page of an article to be crawled, and should be set to a range smaller than the number of crawlable web pages calculated in step S490. The user interface unit 110 can provide the maximum number of web pages that can be crawled. A user can set a crawl range by entering a start web page and an end page.

In step S520, the user interface unit 110 may transmit a start web page that performs a web crawl with the keyword. In addition, the user interface unit 110 may transmit a request flag including a request 2 (request 2) when transmitting a web page range to the web crawler unit 120. Referring to FIG. 18, request 2 may be a request flag for an article crawl request. The web crawler unit 120 can perform article crawling according to the request 2 of the user interface unit 110. [

In step S530, the web crawler unit 120 may calculate the number of articles including keywords based on the keyword and the term information. The parsing unit 121 may generate a URL for a specific page crawl in order to calculate the number of articles. The URL generation process is similar to the process of generating the URL including the keyword and period information in step S430. The parsing unit 121 generates a URL including a keyword, period information, and information of a web page, and transmits the URL to the web 300. The parsing unit 121 receives an HTML file including a list of articles included in the web page from the web 300 , The HTML file can be reconstructed in a tree form.

In step S540, the web crawler unit 120 may crawl articles containing keywords within the range of the input web page. In step S540, the parsing unit 121 extracts the URL of the article list from the HTML file, transmits the URL of the article list to the web 300, receives the HTML file containing the article text data from the web, The HTML file containing the article text data can be reconstructed in tree form.

Further, in step S540, the extracting unit 122 of the web crawler unit 120 may extract the text data from the HTML file of the article list reconstructed in a tree form. The extracting unit 122 may extract text data corresponding to the title and contents of the article from the extracted text data. The language support unit 123 can perform encoding on an HTML file including article text data reconstructed in a tree form. For example, the encoding may be Universal Transformation Format-8 (UTP-8) encoding to prevent loss of Hangul text data.

In step S550, the web crawler unit 120 may repeat the crawling process as many times as the number of article lists. The web crawler unit 120 may terminate the iterative process and transmit the crawled article text data to the crawl database if the number of articles currently being crawled is greater than the total number of articles.

In step S560, the web crawler unit 120 may repeat the article text data to the crawl database 300 when the article text data request process of the specific page is repeated by the number of URLs of the article list. The user interface unit 110 can compare the start page and the end page of the range of the web page designated by the user when the article text data is stored in the crawl database 141. [

If the number of end pages is greater than the number of web pages currently being crawled in step S570, the web crawler unit 120 increments the start page of the web page by one, The crawling process of the text data of the article can be repeatedly performed. That is, the web crawler unit 120 can repeatedly perform the crawling of the web page within the range of the inputted web page.

In addition, the file generating unit 124 can generate the extracted text data as a file in the txt format. The crawling database 141 can store the generated file in the txt format.

6 is an exemplary diagram illustrating a collected text data set according to one embodiment of the present application.

Referring to FIG. 6, text data of an article included in a crawled web page may be collected as a collection result, and collected text data may be stored as a file in a txt format. In the crawling process, the language support unit 123 performs encoding on the HTML file including the article text data, thereby preventing the loss of the Hangul data and obtaining the title of the Hangul type article.

FIG. 7 is a flow chart outlining the collected text data preprocessing step according to one embodiment of the present application.

Referring to FIG. 7, the preprocessing unit 131 may perform preprocessing based on a predefined word set for the collected text data set. Data analyzer 130 may perform data preprocessing to increase the accuracy of data analysis when information of requests 3 and 4 is received. The preprocessing process may be a data classification process and a data filtering process. The data filtering process may be a process of removing a predetermined removed text element from the text data of the article including the preset predefined word.

As a preliminary task for data preprocessing, a predefined word can be defined in the predefined word database 143. Predefined words can be defined by the user.

In step S710, the user interface unit 110 may receive a predefined word input added by the user. The user can add or delete predefined words as needed. The user interface unit 110 can receive inputs for addition and deletion and transmit them to the predefined word database 143 based on the updated predefined words.

Further, in step S710, the user interface unit 100 may transmit a request flag including request 3 (request 3) when transmitting a predefined word. Request 3 may be a requestFlag value intended for data classification requests.

In step S720, the crawling database 141 may transmit the collected text data to the data analysis unit 130 at the request of the data analysis unit 130. [ The preprocessing unit 131 of the data analysis unit 130 may receive the text data collected in the crawl database 141 to classify the text data.

In step S730, the preprocessing unit 131 may compare the content of the article body of the collected text data with the dictionary definition word. If the pre-defined word is included, the preprocessing unit 131 can generate a temporary file and classify the data set first.

 In step S740, the preprocessing unit 131 may request the crawl database 141 to delete the text data so as to prevent speedup of the data classification operation and duplicate classification of the text data set.

That is, the preprocessing unit 131 maintains the text data of the article that does not include the predefined predefined word in the crawl database 141, deletes the text data of the article including the predefined predefined word from the crawl database, Can be stored in the database 142.

In step S750, the crawling database 141 may delete the text data of the article including the predetermined predefined word. Further, in step S760, the analysis database 142 stores the text data of the article including the predetermined predefined word.

The preprocessing unit 131 may repeat the article classification by the number of predefined words preset for the collected text data.

FIG. 8 is a diagram illustrating a text data classification process according to an embodiment of the present invention.

8 (a) may be a predefined word preset by the user. The user can specify a word to be classified, that is, a word to be included in the article, as a predefined word, and judge whether or not the predefined word is included in the collected text data. For example, the preprocessing unit 131 may receive an input from the user interface unit 110, which defines a predefined word set by a user in advance as propagation, negative, human body, side effect, and exposure. The preprocessing unit 131 deletes the data of the article including the word "propagation", "negation", "body", "side effects", and "exposure" from the crawling database 141 among the text data sets collected in the crawling database And store the text data of the article including the predefined word in the analysis database 142. [ 8B shows a process of deleting a word that does not include a predefined word preset from the text data set. FIG. 8C shows an example of a set of preprocessed text data performed by the preprocessing unit 131 Lt; / RTI >

In addition, the preprocessing unit 131 may receive a request flag including the request 4 (request 4) from the user interface unit 110. Request 4 may be a requestFlag value intended for text data filtering requests. In step S770, the preprocessing unit 131 may receive the text data of the article containing the predetermined predefined word from the analysis database 142 to perform the text data filtering.

In step S780, the preprocessing unit 131 receiving the request 4 may perform a process of removing a predetermined removed text element from the text data including the predefined word stored in the analysis database 142. [ For example, the pre-established uninstall text element may include at least one of the name of the reporter who wrote the article, the email address of the reporter, and the advertisement and authoring mark. It is possible to remove unnecessary elements in text data analysis by removing preset text elements removed.

Specifically, the preprocessing unit 131 searches all the positions of the punctuation in the text data of the article including the preset predefined word when performing the process of removing the pre-set removed text elements, The last sentence of the article can be determined on the basis of the sentence, and the next text area of the determined last sentence can be removed as a preset removed text element.

FIG. 9 is a diagram illustrating a process of removing a predetermined removed text element according to an embodiment of the present invention.

Referring to FIG. 9, in the text data of an article including a predefined dictionary word, elements such as journal name, journal e-mail, advertisement, copyright notation, and the like may be stored after the last sentence. The preprocessing unit 131 may search all the positions of ". &Quot; of the classified data, and may remove the text element based on a position located second from the end of the retrieved position. For example, in general, the article's name, e-mail address, copyright notation, etc. are listed after the last sentence of the article. Because the e-mail address contains a period, Can be determined as the point at which the last sentence of the article ends, and the text area located after the last sentence can be removed. The preprocessing unit 131 may delete the contents after the reference point is set, and store the text data in the analysis database 142 after the deletion is completed.

FIG. 10 is a flow chart schematically illustrating the steps of forming a base data set according to one embodiment of the present application.

Referring to FIG. 10, the data forming step is a step for forming a base data set to be used for data analysis, and a chin extraction, a frequent word extraction, and a matrix data set forming process can be sequentially performed.

First, the data forming unit 132 of the data analysis unit 130 may receive a request flag including request 5 (request 5) to the user interface unit 110. Request 5 may be a requestFlag value for the purpose of requesting a snail extract. The censor extraction process can be a process of extracting pronouns such as pronouns, nouns, and rhetoric from collected articles.

In step S810, the data forming unit 132 may receive the text data of the preprocessed article from the analysis database 142. [

In step S820, based on the text data of the preprocessed article, the data forming unit 132 may extract only the cheong except for unnecessary investigation, adjective, adverb, and the like in data analysis. According to one embodiment of the present invention, the data forming unit 132 may classify words of two or more words among word sets composed of words extracted through the chin extraction process, and store them in the analysis database 142. [

After the snooze extracting process, the data forming unit 132 may receive a request flag including request 6 (request 6) to the user interface unit 110. Request 6 may be a requestFlag value intended for frequent word extraction requests. Frequent word extraction can be performed by sorting the extracted vowels based on the frequency of extracted vowels, and extracting vowels of a certain rank or higher.

In step S830, the user interface unit 110 may receive an input relating to the user's word frequency rank. The word frequency rank refers to the frequency rank order of words for extracting frequent words and can be used to extract words within a certain frequency rank in the frequent word extraction process. The data forming unit 132 can request the cynergy set data to the analysis database 142.

In step S840, the analysis database 142 may transmit the extracted census data set.

In step S850, the data forming unit 132 may extract a frequent word based on the cognitive aggregate data and the word frequency rank. The data forming unit 132 may extract only words in the cognitive aggregate data set whose frequencies are within the word frequency rank value input by the user and store the extracted words in the analysis database 142. [

After the frequent word extraction process, the data forming unit 132 may receive a request flag including request 7 (request 7) to the user interface unit 110. Request 7 may be a requestFlag value intended to request a matrix data set formation request. The process of forming the matrix data set may be a process of forming a data set in which the frequent word inclusion of each article is indicated for the association rule analysis and word extraction probability.

In step S860, the data forming unit 132 may receive the text data set and the frequent word set of the preprocessed article from the analysis database 142. [

In step S870, the data forming unit 132 determines whether or not the frequent word of each article is included for the association rule analysis based on the text data set of the preprocessed article, the frequent word set, and the keyword value included in the request flag of request 7 A matrix data set can be formed. In step S880, the data forming unit 132 forms a base data set including a cognate set, a frequent word set, and a matrix data set.

11 is a schematic representation of a matrix data set according to one embodiment of the present invention.

Referring to FIG. 11, a column name of a matrix data set represents a word having a certain frequency or more, and a row name is an identifier of an article, and may indicate a number from 1 to the number of articles. The data forming unit 132 compares each article with a word having a specific frequency or more. If the article includes a word having a certain frequency or more, the data forming unit 132 may set the data to 1. Otherwise, the data forming unit 132 may set the data to zero. In order to analyze the relationship between the keyword inputted by the user and the extracted words, the data forming unit 132 inserts the keyword inputted by the user into the name of the first column and inserts the value of the row corresponding to the first column into 1 . The keyword input by the user may be keyword information received in step S310.

12 is a flowchart schematically illustrating a step of performing text data analysis according to an embodiment of the present invention.

Referring to FIG. 12, the data analysis step may be a step for analyzing the extracted data set and providing the analysis result to the user in various visualization forms. In data analysis, frequency analysis of the words used in the article and association rule analysis between words in the article can be performed. The word frequency analysis can analyze the frequency of use of specific words and analyze the extraction probability of specific words. Association rule analysis analyzes the correlation of words used in an article and can be performed based on a matrix data set.

The analysis unit 133 may analyze the frequency of the words used in the article and analyze the association rules between words in the article. The analysis unit 130 compares all the word sets that can be made with frequent words with the word sets included in each article and finds a set of words used simultaneously in a plurality of articles and defines them as IF-THEN type rules Method can be used to perform association rule analysis.

Illustratively, the association rule can be defined and expressed as {electromagnetic wave, repeater} -> {vice versa}. In other words, if the word 'electromagnetic wave, repeater' is used in the article, it can be interpreted that the word 'opposition' is also used together. Here, {electromagnetic wave, repeater} corresponds to a first word set (Left-Hand Side (LHS), and {reverse} corresponds to a second word set (Right-Hand Side (RHS)). A first word set (LHS) is a word set that can be made up of frequent words in a matrix data set, and a second word set (RHS) is a word set that consists of one word of frequent words in a matrix data set.

If the matrix data set includes three frequent words of electromagnetic waves, repeaters and transmission towers, the first word set (LHS) is {electromagnetic wave}, {repeater}, {transmission tower}, {electromagnetic wave, repeater}, {electromagnetic wave, , {Repeater, transmission tower}, {electromagnetic wave, repeater, transmission tower}, and the second word set RHS becomes {electromagnetic wave}, {repeater}, {transmission tower}. In the association rule, the first word set (LHS) and the second word set (RHS) do not use overlapping words. The analysis unit 133 may generate a number of association rules related to the first word set LHS and the second word set RHS as an analysis result. If the user specifies the second word set (RHS) as a specific word, the user can easily confirm the association rule analysis result for the desired word.

Association rule analysis is a task that compares all sets that can be made into words in the matrix data set by each article, so that the number of operations can increase exponentially as the number of words and articles increases. The analysis unit 133 may use an Apriori algorithm for eliminating operations on an infrequent word set based on two principles in order to reduce the number of operations in the association rule analysis.

The Apriori algorithm is 1) if a set of items is frequent, all subsets of this set are also frequent itemsets. 2) If a yield set is infrequent, then all the sets containing this set of items are non-frequent itemsets. And the like.

Since the item set is a set that can be made up of words in the data set (set), applying the Apriori algorithm can reduce the number of operations by reducing the word set relatively.

The analysis unit 133 may use three factors, such as support, confidence, and lift, in order to establish a relationship between keywords and extracted words through association rule analysis. Support, reliability, and degree of improvement are calculated as follows, where X is the first word set (LHS) and Y is the second word set (RHS).

① Support (support, s (X → Y))

= Number of articles containing both X and Y at the same time / Total number of articles

② Confidence (c → X)

= Number of articles containing X and Y simultaneously / Number of articles containing X

③ Improvement degree (Lift, Lift (X → Y))

= Percentage of articles containing Y / percentage of articles containing Y among all articles

Illustratively, the degree of support means the proportion of articles containing both X and Y of the extracted articles, and the reliability means the proportion of articles containing Y among the articles containing X. Also, the degree of improvement refers to the ratio of articles containing Y among the articles containing X to the ratio of articles including Y among the extracted articles when the keyword is input.

More specifically, support measures how many articles X and Y were used simultaneously. If the support value is very small, it means that the number of articles that follow the association rule is very small. That is, by measuring support, it is possible to measure whether a particular word set is used only in a specific article or in many articles.

Reliability means the probability that the rule is correct, and the percentage of articles in which X is used in the article. If the reliability of a certain set of words is very low, it means that the rule is not a correct answer, and it is a rule that does not occur in a normal case. Even though association rules and reliability are both highly related rules, they can not be defined as meaningful rules. This is because there are rules that have high relevance and reliability that seem to be related by chance. Thus, an enhancement is used to measure the usefulness of association rules.

 The degree of enhancement provides information on 'how much the probability of including Y when including X is improved compared to the probability of including Y if X is not considered'. If the degree of enhancement is 1, there is a high possibility that the two word sets X and Y are not correlated with each other because the probability of finding Y in the articles containing X and the probability of finding Y in all articles is the same. On the other hand, if the degree of improvement is more than 1, the two word sets are more likely to have a close correlation (positive correlation), and if they are less than 1, there is a high possibility of having a mutually opposite correlation (negative correlation). For example, bread and butter are likely to have a close and positive correlation, and leprosy and laxatives are likely to have a negative correlation with each other.

In step S901, the user interface unit 110 may transmit the request flag and the minimum frequency. The data analysis unit 130 may receive a request flag including request 8 from the user interface unit 110. [ Request 8 may be a requestFlag value intended for a Word Cloud visualization request.

In step S902, the analysis unit 133 may receive the census data from the analysis database 142. [

In step S903, the analyzing unit 133 can determine a word having a minimum frequency of words included in the vocabulary set and perform frequency analysis. In addition, the analyzing unit 133 may determine the output position, the output size, and the output color according to the frequency of the word having the determined minimum frequency and output it in word cloud form.

FIG. 13 is a diagram illustrating an example of outputting a word having a minimum frequency determined according to an embodiment of the present invention in word cloud form by determining output position, output size, and output color according to the frequency.

Illustratively, referring to FIG. 13, visualization in word cloud form can be visualized to enhance core words, and words used in the article can be expressed in different sizes and colors according to frequency.

In addition, the analysis unit 133 may store the visualization result output in the generated word cloud form in the analysis database 142.

Next, the data analysis unit 130 may receive a request flag including request 9 (request 9) from the user interface unit 110. Request 9 may be a requestFlag value intended for a Grouped Matrix visualization request.

In step S904, the data analysis unit 130 may receive minimum support degree and minimum reliability input information. In addition, the data analysis unit 130 may receive an input relating to the second word set.

In step S905, the analysis database 142 may transmit the matrix data set at the request of the data analysis unit 133. [

In step S906, the analysis unit 133 may perform association rule analysis based on the matrix data set. The analysis unit 133 determines a first word set including a word combination that can be made up of frequent words in the matrix data set, and when the word combination included in the first word set is included in the article, it is included in the second word set Can determine the association rules as to whether or not the word is included in the article.

The analyzing unit 133 may calculate the degree of support, reliability, and degree of improvement of the association rule, and output the analysis result of the association rule having the support degree and the reliability higher than the minimum support degree and the minimum reliability level among the determined association rules.

14, in step S907, the analyzer 133 may output the association between the first word set and the second word set in the form of a matrix. 14 is a diagram showing an example of a group matrix output. In FIG. 14, it can be confirmed that the association between the respective word sets is expressed in a matrix form on the basis of the first word set (LHS) and the second word set (RHS) of the association rule. The circle size of the group matrix output graph indicates the degree of support of each rule, and the shade of color may mean an improvement. The number before the first word set (LHS) name means the number of association rules with that condition, and the number marked with "+" in the first word set (LHS) means the number of words omitted can do. Group Matrix Output It can be interpreted that the larger the circle size of the graph, the larger the color is, the more the rule is generated.

The analysis unit 133 may generate different numbers of rules for each search keyword. The reason why different numbers of rules are generated is that the number of articles included in the matrix data set and the number of words included in the articles are different for each keyword. The analysis unit 133 can generate different numbers of rules for each search keyword since the number of rules having a value equal to or greater than the option value (support degree = 0.1, reliability = 0.1) is different for each search keyword. In other words, when the association rule analysis is performed, the analysis unit 133 can determine the support degree and the reliability value of the rules generated according to the word sets included in each article. The smaller the number of articles in the matrix data set and the number of words contained in each article, the greater the number of rules that are below the option value and can be excluded from the results.

In addition, the data analysis unit 130 may receive a request flag including the request 10 (request 10) from the user interface unit 110. Request 10 may be a requestFlag value intended for a Graph visualization request.

In step S908, the data analysis unit 130 may request the matrix data set from the analysis database 142. [ The analysis unit 133 may perform association rule analysis based on the matrix data set. The analysis unit 133 can perform graph visualization based on rules having a specific support ranking or more.

In step S909, the association rule analysis analysis unit 133 may receive the minimum support value and the minimum reliability value from the user interface unit 110. [ In addition, the analyzer 133 may set a second word set (RHS) value to check only the association rule analysis result for the word desired by the user.

Referring to FIG. 15, in step S910, the analyzer 133 may output a graph outputting the association between a word belonging to the first word set and a word belonging to the second word set in a network form. In the network graph type word shown in FIG. 15, a circle-directed arrow means a first word set (LHS), and an arrow from a circle to a word means (RHS). For example, the analysis unit 133 can analyze the association of the 'Transmission Tower (LHS) -> O-> Resident (RHS)'. That is, if the word "transmission tower" is used in the article, the analysis unit 133 can analyze the rule that the word "residents" is also used in the article. When analyzed as 'O-> electromagnetic waves (RHS)', it may mean the rule that electromagnetic waves are used in the article, regardless of whether or not the word presented in the first word set (LHS) is used.

In addition, the size of the circle may mean the degree of support of each rule, and the increase of the color of the circle may mean the degree of improvement. The position of the word indicates the association with other words, and in FIG. 15, it can be confirmed that the relay is at the center of the association. In other words, it can be interpreted that the larger the size of the circle, the higher the frequency is. For example, if there are two words apart from other words, it may be that the two words have only two associations.

The analysis unit 133 may store the graph output output in the generated network form in the analysis database 142. [

In addition, the data analysis unit 130 may receive a request flag including the request 11 (request 11) from the user interface unit 110. Request 11 may be a requestFlag value for the purpose of requesting a Scatter Plot visualization (distribution diagram output) request.

In step S911, the data analysis unit 130 may request the matrix data set from the analysis database 142. [ The analysis unit 133 may perform association rule analysis based on the matrix data set. The analysis unit 133 can visualize the distribution of the determined association rule as a distribution diagram output and provide it.

In step S912, the association rule analysis analysis unit 133 may receive the minimum support value and the minimum reliability value from the user interface unit 110. [ In addition, the analyzer 133 may set a second word set (RHS) value to check only the association rule analysis result for the word desired by the user.

In step S913, the analysis unit 133 may provide a graph of the distribution of the generated association rules. Illustratively, referring to FIG. 16, the X-axis of the minute application output graph means the support degree of the association rule, and the Y-axis can mean the reliability of the association rule. The enhancement of the color of each point may be an improvement. Distribution graph The output graph can represent the distribution of support, reliability, and degree of improvement of the generated rules. Distribution points Each point in the output graph represents a rule, and the density of points can be proportional to the number of rules. Regardless of the search keyword, the distribution graph of the output graph shows that the higher the reliability, the greater the number of generated rules, and the higher the degree of rule enhancement. In the association rule analysis, a rule with a high degree of reliability among the rules with low occurrence frequency may mean that the association between the two word sets is high.

Finally, the data analysis unit 130 may receive a request flag including the request 12 (request 12) from the user interface unit 110. Request 12 may be a requestFlag value for purposes of Extraction Probability visualization requests.

In step S914, the analysis unit 133 may request the matrix data set to the analysis database 142. [ The analysis unit 133 may perform the frequency analysis based on the matrix data set. The analysis unit 133 can provide a probability analysis and a frequency of extraction of a specific word from the article in a visualized form. In step S915, the analysis unit 133 reads the matrix data set, calculates the probability that the words included in the cognoset are extracted from the respective articles of the matrix data set, A frequency analysis can be performed by outputting a graph including extraction frequency and extraction probability.

Referring to FIG. 17, in step S915, the analyzer 133 may represent a graph including an extraction frequency and an extraction probability as a bar-shaped graph. The graph including the extraction frequency and the extraction probability shown in FIG. 17 can indicate the probability of extracting a specific word from an article in the form of a bar graph when an arbitrary keyword is input. The X axis of the graph including the extraction frequency and extraction probability means a word set having a high frequency, and the Y axis means a probability of extraction of word sets. In the graph shown in FIG. 17, the top 30 word sets having a high frequency among all the extracted word sets represent examples of the X axis. The number at the top of each bar graph shows the specific probability of extraction by word set.

The probability that a specific word is extracted from the article when an arbitrary search keyword is input in step S915 can be calculated based on the matrix data set. The analyzing unit 133 can express the extraction probability by calculating (the number of articles including a specific word) / (the number of articles crawled).

The analysis unit 133 may store the generated extraction frequency and extraction probability graph in the analysis database 142. [

In the above description, each step may be further divided into further steps, or combined into fewer steps, according to embodiments of the present disclosure. Also, some of the steps may be omitted as necessary, and the order between the steps may be changed.

The method for collecting and analyzing text data according to one embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Furthermore, the text data collection and analysis method described above can also be implemented in the form of a computer program or an application executed by a computer stored in a recording medium.

It will be understood by those of ordinary skill in the art that the foregoing description of the embodiments is for illustrative purposes and that those skilled in the art can easily modify the invention without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

100: Text data collection and analysis device
110:
120: Web crawler part
121: parsing unit 122:
123: language support unit 124: file generation unit
130: Data analysis section
131: preprocessing unit 132: data forming unit
133:
140: Database
141: Crawl database 142: Analysis database
143: Predefined word database
200: user terminal
300: Web

Claims (20)

A method for collecting and analyzing text data,
Receiving input regarding keyword and period information;
Calculating the number of articles displayed on the web including the keyword based on the keyword and the period information;
Calculating a number of crawlable web pages based on the number of articles;
Providing the calculated number of web pages;
Receiving an input relating to a range of a web page to be crawled;
Crawling a web page including an article including the keyword within a range of the input web page;
Collecting text data of an article included in the crawled web page;
Preprocessing the collected text data based on a predefined word set in advance; And
Performing a text data analysis based on the preprocessed result,
Gt; a < / RTI > text data collection and analysis method. The method according to claim 1,
And the period information is a publication period of an article. The method according to claim 1,
Wherein the crawling of the web page is repeatedly performed within a range of the input web page. The method according to claim 1,
The pre-
Receiving an input relating to the predetermined predefined word;
Classifying articles that do not include the predefined word and an article that includes the predefined predefined word; And
Removing a predetermined removed text element from the text data of the article containing the predetermined predefined word,
The method comprising the steps of: The method according to claim 1,
Receiving an input regarding a word frequency ranking; And
Forming a base data set to be used for text data analysis based on the preprocessed text data,
Further comprising:
The base data set includes a cognate set included in the preprocessed text data, a frequent word set within the word frequency rank among the words included in the cognate set, and a matrix data set that is information on whether the frequent word is included in each article To a text data collection and analysis method. 6. The method of claim 5,
Wherein performing the text data analysis comprises analyzing frequency of words used in the article and analyzing association rules between words in the article. The method according to claim 6,
Wherein performing the text data analysis comprises:
Determining a first word set comprising a word combination that can be made up of frequent words in the matrix data set;
Receiving an input relating to a second set of words including at least one word of the frequent word set; And
Determining an association rule as to whether a word included in the second word set is included in an article if the word combination included in the first word set is included in the article,
The method comprising the steps of: 8. The method of claim 7,
Wherein performing the text data analysis comprises:
Receiving an input regarding minimum support and minimum reliability for the association rule; And
Outputting an analysis result on an association rule having a support degree and a reliability higher than the minimum support degree and the minimum reliability out of the determined association rules;
The method comprising the steps of: The method according to claim 6,
Wherein performing the text data analysis comprises:
Receiving an input regarding a minimum frequency; And
Determining a word having the minimum frequency among the words included in the vocabulary set and performing frequency analysis,
The method comprising the steps of: A text data collection and analysis apparatus comprising:
A user interface unit for receiving an input regarding keyword and period information and receiving an input relating to a range of a web page to be crawled;
Calculating the number of articles displayed on the web including the keyword based on the keyword and the period information, calculating the number of web pages that can be crawled based on the number of articles, A web crawler unit providing a user interface unit with a web page including an article including the keyword within a range of the web page input by the user interface unit; And
A data analysis unit for collecting text data of an article included in the crawled web page, preprocessing the collected text data based on a predefined word, and performing text data analysis based on the preprocessed result, ,
And a text data acquisition and analysis device. 11. The method of claim 10,
And the period information is a publication period of an article. 11. The method of claim 10,
Wherein the web crawler unit repeatedly performs crawling of the web page within a range of the web page received by the user interface unit. 11. The method of claim 10,
The user interface unit receives an input relating to the predetermined predefined word,
The data analysis unit may include:
Classifying an article including the predefined dictionary word and an article not including the predetermined dictionary definition word and removing a predetermined removed text element from the text data of the article including the predetermined dictionary definition word, And a preprocessing unit for performing preprocessing on one piece of text data. 11. The method of claim 10,
The user interface receives an input relating to a word frequency ranking,
The data analysis unit may include:
And a data forming unit for forming a base data set to be used for text data analysis based on the preprocessed text data,
The base data set includes a cognate set included in the preprocessed text data, a frequent word set within the word frequency rank among the words included in the cognate set, and a matrix data set that is information on whether the frequent word is included in each article The text data collection and analysis device. 15. The method of claim 14,
The data analysis unit may include:
And analyzing a frequency of the words used in the article based on the base data set and analyzing association rules between words in the article. 16. The method of claim 15,
Wherein the user interface unit receives an input relating to a second word set including at least one word of the frequent word set,
Wherein the analyzing unit determines a first word set including word combinations that can be made up of frequent words in the matrix data set, and when the word combination included in the first word set is included in the article, And determining association rules as to whether embedded words are also included in the article. 17. The method of claim 16,
Wherein the user interface unit receives an input regarding a minimum support degree and minimum reliability regarding the association rule,
Wherein the analyzing unit outputs an analysis result related to an association rule having a support degree and a reliability higher than or equal to the minimum support degree and the minimum reliability degree among the determined association rules. 16. The method of claim 15,
The user interface unit receives an input relating to a minimum frequency,
Wherein the analyzing unit determines the words having the minimum frequency among the words included in the vocabulary set and performs the frequency analysis. 18. A text data collection and analysis apparatus according to any one of claims 10 to 18; And
A user terminal providing the text data collection and analysis apparatus with an input relating to keyword and period information, receiving and outputting results of text data analysis from the text data collection and analysis apparatus,
And a text data collection and analysis system. A computer-readable recording medium on which a program for executing the method of any one of claims 1 to 9 is recorded.

Images