KR20220095654A - Social data collection and analysis system - Google Patents

Abstract

The present invention relates to a social data collection analysis system and a method thereof. The method comprises the following steps: (a) a collection part collects data corresponding to a set keyword from a connected social site; (b) a filter part filters the collected data in accordance with a score given in accordance with preset criteria; and (c) an analysis part generates analysis information by extracting the frequency of a keyword included in the filtered data. In accordance with the present invention, by collecting, from a social site, big data for all kinds of tendency/preference/approval rating/market/response research, learning the big data, and analyzing the similarity of the learned big data in accordance with a set keyword to thus generate various indexes and statistical information, the social data collection analysis system is capable of reducing an analysis period, and analysis and storage costs for the big data, and improving a response rate in accordance with the utilization of the analyzed big data.

Description

Social data collection and analysis system { Social data collection and analysis system}

The present invention relates to a social data collection and analysis system and method, and more particularly, it performs learning on big data collected from social sites, and analyzes similarity according to keywords that set the learned big data to obtain various indicators and It relates to techniques for generating statistical information.

Big data refers to a large amount of structured or unstructured data sets that go beyond the ability to collect, store, manage, and analyze data with existing database management tools, and technology to extract values from these data and analyze the results.

The development of big data technology, which is characterized by the generation, collection, analysis, and expression of various types of large-capacity data, more accurately predicts the diversified modern society and makes it work efficiently, and provides customized information for each individualized modern society member; management and analysis.

As described above, big data presents the possibility of providing useful information to society and mankind in all areas such as politics, society, economy, culture, and science and technology, and its importance is being highlighted.

Data mining, machine learning, natural language processing, pattern recognition, etc. used in existing statistics and computer science can be mobilized for big data analysis. In particular, text mining, opinion mining, social network analysis, and cluster analysis among analysis techniques are receiving attention due to the recent increase in unstructured data such as social media.

However, the conventional big data collection has a uniform disadvantage because the search for information to be collected is judged only according to the rules input by the user through the rule-based engine, and the information collection method is limited to crawling or scraping.

In addition, there is a problem of having to designate the variables and parameters every time the information to be collected is searched for, and the information collection target is limited to a web browser.

It is an object of the present invention to perform learning by collecting big data for research on various tendencies/preferences/approvals/market/reactions from social sites, and by analyzing similarity according to keywords set with the learned big data, various indicators and statistics It relates to a technology that reduces the analysis period, analysis cost, and storage cost of big data by generating information, and improves the response rate according to the use of analyzed big data.

One embodiment of the present invention for achieving this technical problem is a social data collection and analysis system, the collection unit for collecting data corresponding to a keyword set from an accessed social site; a filter unit for filtering the collected data according to a score given according to a preset criterion; and an analysis unit for generating analysis information by extracting the frequency of keywords included in the filtered data, but the collection unit sets and manages the basic data for data collection as a json file, and any one of text, comments, voice, or images It is characterized in that it collects data including

Preferably, the basic data is characterized in that it includes any one of an access address of a social site, an account, a search word for data collection, and a storage path of the collected data.

The collection unit temporarily stores data collected from social sites in a database, but has a crawling engine or scraping engine for data collection for a plurality of social sites, and API or Restful of social sites that are not supported In this case, it is characterized by collecting data through a scraping engine.

The collection unit is characterized in that it collects data in any one of API, Restful, crawling, and scraping methods through setting changes.

The filter unit assigns a score to each of the keywords or prohibited words according to any one of the criteria of Hangeul morpheme analysis, similarity between sentences and keywords, or the number of occurrences of words for the collected data, and results by Bayesian Machine Learning It is characterized in that the data collected in the database is sorted or filtered based on the

The analysis unit classifies the analysis information according to preset values stored in the database to generate a data flow chart in the form of a time-series chart, diagram, or moving picture.

A social data collection and analysis method according to an embodiment of the present invention based on the above-described system includes: (a) collecting data corresponding to a keyword set from a social site accessed by a collection unit; (b) filtering the collected data by the filter unit according to a score given according to a preset criterion; and (c) generating analysis information by the analysis unit extracting the frequency of keywords included in the filtered data.

Preferably, step (a) comprises: (a-1) in which the collection unit sets and manages the basic data for data collection from the social site as a json file; (a-2) step of the collection unit setting any one data collection method among API, Restful, crawling, or scraping; (a-3) step of the collecting unit accessing a social site that matches the basic data; (a-4) collecting, by a collecting unit, data corresponding to the set keyword from the accessed social site in a set data collecting method; and (a-5) temporarily storing the data collected by the collection unit from the social site in a database.

The step (b) includes: (b-1) giving the collected data by the filter unit a score for each keyword or prohibited word according to a preset criterion; (b-2) performing Bayesian machine learning on the keyword or forbidden word to which the filter unit is scored; and (b-3) of the filter unit sorting or filtering the data collected in the database based on the Bayesian machine learning result.

Step (c) includes: (c-1) generating analysis information by extracting the frequency of keywords included in the filtered data by the analysis unit; (c-2) indexing the report setting values previously stored in the database by the analysis unit ; (c-3) generating a banner of a page including the analysis information generated by the analysis unit as an image when the index result of step (c-2) and the report setting value is '0'; (c-4) generating the analysis information generated by the analysis unit as a word count report when the index result of step (c2) and the report setting value is '1'; and (c-5) generating a data flow chart in the form of a time-series chart, diagram, or moving picture by classifying the analysis information generated by the analysis unit according to preset values stored in the database.

According to the present invention as described above, learning is performed by collecting big data for research on various tendencies/preferences/approvals/market/reactions from social sites, and similarity is analyzed according to keywords set with the learned big data to analyze various indicators and statistical information, it has the effect of reducing the analysis period, analysis cost, and storage cost of big data, and improving the response rate according to the use of the analyzed big data.

1 is a block diagram illustrating a social data collection and analysis system according to an embodiment of the present invention.
2 is a diagram illustrating a detailed configuration of a social data collection and analysis system according to an embodiment of the present invention.
3 is a diagram illustrating a code generation unit and a short-range communication unit of a social data collection and analysis system according to an embodiment of the present invention.
4 is a diagram illustrating a framework configuration of a social data collection and analysis system according to an embodiment of the present invention.
5 is a diagram illustrating an operational flow of a social data collection and analysis system according to an embodiment of the present invention.
6 is a diagram illustrating a network configuration of a social data collection and analysis system according to an embodiment of the present invention.
7 is a flowchart illustrating a social data collection and analysis method according to an embodiment of the present invention.
8 is a flowchart illustrating a detailed process of step S100 of the social data collection and analysis method according to an embodiment of the present invention.
9 is a flowchart illustrating a detailed process of step S200 of the social data collection and analysis method according to an embodiment of the present invention.
10 is a flowchart illustrating a detailed process of step S300 of the social data collection and analysis method according to an embodiment of the present invention.

The specific features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. Prior to this, the terms or words used in the present specification and claims are based on the technical idea of the present invention based on the principle that the inventor can appropriately define the concept of the term in order to describe his invention in the best way. It should be interpreted with the corresponding meaning and concept. In addition, it should be noted that, if it is determined that the detailed description of the well-known functions related to the present invention and its configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof is omitted.

1 is a block diagram showing a social data collection and analysis system (S) according to an embodiment of the present invention, Figure 2 shows a detailed configuration of the social data collection and analysis system (S) according to an embodiment of the present invention it is one drawing

1 and 2 , the social data collection and analysis system (S) according to an embodiment of the present invention is configured to include a collection unit 100 , a filter unit 200 , and an analysis unit 300 . .

First, the collection unit 100 sets and manages basic data for data collection from a social site as a json file, and collects data corresponding to a keyword set from an accessed social site.

In this case, the data collected by the collecting unit 100 may include any one of text, voice, and image, and comments may be collected based on tags and keywords.

In addition, the basic data includes any one of an access address of a social site, an account, a search word for data collection, and a storage path of the collected data, and the value included in the basic data may be added or changed by an administrator.

The collection unit 100 temporarily stores data collected from social sites in a database, has a crawling engine or scraping engine for data collection for each social site, and API or Restful is not supported. For social sites, data is collected through scraping engines.

The collection unit 100 may include a separate module for collecting data corresponding to keywords from a legacy database in addition to the social site.

The collection unit 100 may change the data collection method to any one of API, Restful, crawling, or scraping through setting change.

The filter unit 200 gives a score for each keyword or prohibited word according to a preset criterion, and sorts or filters data collected in the database according to the score.

In this case, the preset criterion includes any one of Hangul morpheme analysis, similarity between sentences and keywords, or the number of occurrences of words, and a value included in the set criterion may be added or changed by an administrator.

In addition, the filter unit 200 is configured to sort or filter the data collected in the database based on the results of Bayesian machine learning (Bayesian MachineLearning) according to the score given to each of the keywords or prohibited words.

Accordingly, in the filter unit 200 according to an embodiment of the present invention, active filtering is possible as an administrator determines parameters and variables by Bayesian machine learning without setting variables and parameters every time.

In other words, it is possible to search and filter by redefining patterns and rules through rule-based learning, rather than judging only according to the rules entered by the administrator.

The analysis unit 300 extracts the frequency of keywords included in the filtered data to generate a banner of a page including the generated analysis information as an image, or generates analysis information including a word count report, and sets pre-stored in the database It selects, stores, and manages filtered data through machine learning that matches the value.

The analysis unit 300 classifies the generated analysis information according to preset values stored in the database to generate a data flow chart in the form of a time-series chart, diagram, or video.

In addition, the social data collection and analysis system (S) according to an embodiment of the present invention may be configured to further include a code generation unit 400 and a short-range communication unit 500 as shown in FIG. 3 .

The code generation unit 400 is configured to generate the url of the report page for checking the analysis information as a barcode so that the analysis information can be checked through the mobile device. In this case, the barcode may be generated as either a one-dimensional barcode or a two-dimensional barcode (QR code).

The short-distance communication unit 500 is configured to move the screen to the report page when tagging by NFC or a beacon so that analysis information can be checked through a mobile device.

That is, according to an embodiment of the present invention, by configuring the code generating unit 400 and the short-distance communication unit 500, it is not limited to simply accessing data through a web browser, but also analysis information through a mobile device through a barcode or NFC. make it possible to check

4 is a diagram illustrating a framework configuration of a social data collection and analysis system (S) according to an embodiment of the present invention.

As shown in Fig. 4, based on Scikit-learn, Tensorflow is configured as a backend engine, and Keras is configured as a front engine.

For data summary, newspaper module is configured, and pandas and seaborn are used for morphological analysis of texts included in collected data and numerical analysis of konply library.

An analyzer is implemented for the mining function to classify and analyze data, and a collector is configured to collect data from social networks and legacy databases. At this time, MYSQL's memory storage engine-based Integrity DB was configured to temporarily store processing information for analysis and classification and to store the ledger information of the filter processor.

In addition, matplotlib, ffmpeg and ggplot libraries were used to show various chart plotting and video-based trend information of the report to be checked by the user.

The configuration of the web server is composed of the express server-based language of node.js, and Electron technology is used so that users can use mobile devices, web and application programs through a single UI.

5 is a diagram showing an operation flow according to the framework configuration of the social data collection and analysis system (S) according to an embodiment of the present invention. Referring to FIG. 5, the operation flow according to the framework configuration is as follows.

1) Collector reads social cloud or legacy database by configuration.

2) Execution Environment Collect information from legacy databases or social cloud sites.

3) Read information from memory-based database for integration and classify/analyze it.

4) The predictor learns the analyzed data and creates a report by visualizing it based on the prediction.

5) Shows web/mobile/application programs as one integrated viewer.

6 is a diagram illustrating a network configuration of a social data collection and analysis system (S) according to an embodiment of the present invention.

As shown in Fig. 6, Collector provides Integration DB information to process and temporarily store information collected from Social Network and Leagacy DBMS, and provides Predictor to learn/predict based on the analyzed information, User provides Electron Viewer so that one view can be checked equally on various devices.

Hereinafter, a social data collection and analysis method according to an embodiment of the present invention will be described with reference to FIG. 7 .

First, the collection unit 100 collects data corresponding to the keyword set from the accessed social site (S100).

Next, the filter unit 200 filters the collected data according to a score given according to a preset criterion (S200).

Then, the analysis unit 300 extracts the frequency of keywords included in the filtered data to generate analysis information (S300).

Hereinafter, the step S100 of the social data collection and analysis method according to an embodiment of the present invention will be described with reference to FIG. 8 .

First, the collection unit 100 sets and manages the basic data for data collection from the social site as a json file (S102).

Next, the collection unit 100 sets any one data collection method among API, Restful, crawling, and scraping (S104).

Subsequently, the collection unit 100 accesses a social site that matches the basic data (S106).

Next, the collecting unit 100 collects data corresponding to the set keyword from the accessed social site in a set data collecting method ( S108 ).

Then, the collection unit 100 temporarily stores the data collected from the social site in the database (S110).

Hereinafter, the detailed process of step S200 of the social data collection and analysis method according to an embodiment of the present invention will be described with reference to FIG. 9 .

After step S100, the filter unit 200 assigns a score to each of the keywords or prohibited words according to a preset criterion for the collected data (S202).

Next, the filter unit 200 performs Bayesian machine learning on the scored keywords or prohibited words ( S204 ).

Then, the filter unit 200 sorts or filters the data collected in the database based on the Bayesian machine learning result (S206).

Hereinafter, the detailed process of step S300 of the social data collection and analysis method according to an embodiment of the present invention will be described with reference to FIG. 10 .

After step S200, the analysis unit 300 extracts the frequency of keywords included in the filtered data to generate analysis information (S302).

Next, the analysis unit 300 indexes the report setting values previously stored in the database (S304).

When the index result report setting value in step S304 is '0', the banner of the page including the analysis information generated by the analysis unit 300 is generated as an image (S306).

On the other hand, when the index result report set value in step S304 is '1', the analysis information generated by the analysis unit 300 is generated as a word count report (S308)

Then, the analysis information generated by the analysis unit 300 is classified according to the preset values stored in the database to generate a data flow chart in the form of a time-series chart, diagram, or moving picture (S310).

Although the above has been described and illustrated in relation to a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as such, and deviates from the scope of the technical idea. It will be apparent to those skilled in the art that many changes and modifications may be made to the present invention without reference to the invention. Accordingly, all such suitable alterations and modifications and equivalents are to be regarded as being within the scope of the present invention.

S: Social data collection and analysis system
100: collection unit
200: filter unit
300: analysis unit
400: code generator
500: short-distance communication department

Claims (10)

a collection unit that collects data corresponding to keywords set from the accessed social sites; a filter unit for filtering the collected data according to a score given according to a preset criterion; and an analysis unit generating analysis information by extracting the frequency of keywords included in the filtered data, wherein the collection unit sets and manages the basic data for data collection as a json file, text, comment, voice or image A social data collection and analysis system, characterized in that it collects data including any one of them.
According to claim 1,
The basic data, Social data collection and analysis system, characterized in that it includes any one of an access address of a social site, an account, a search word for data collection, and a storage path of the collected data.
According to claim 1,
The collecting unit temporarily stores data collected from social sites in a database,
Social data comprising a crawling engine or scraping engine for data collection for each social site, and collecting data through a scraping engine for social sites that do not support API or Restful collection analysis system.
According to claim 1,
The collection unit, Social data collection and analysis system, characterized in that by collecting data in any one of API, Restful, crawling, and scraping through setting change.
According to claim 1,
The filter unit assigns a score to each of the keywords or prohibited words according to any one of the criteria of Hangul morphological analysis of the collected data, the similarity between sentences and keywords, or the number of occurrences of the word, and in Bayesian Machine Learning Social data collection and analysis system, characterized in that for sorting or filtering the data collected in the database based on the results of
According to claim 1,
The analysis unit, Social data collection and analysis system, characterized in that by classifying the analysis information according to preset values stored in the database to generate a data flow chart in the form of a time-series chart, diagram, or video.
(a) collecting, by a collecting unit, data corresponding to the keyword set from the accessed social site;
(b) filtering the collected data by the filter unit according to a score given according to a preset criterion; and
(c) generating analysis information by the analysis unit extracting the frequency of keywords included in the filtered data.
8. The method of claim 7,
The step (a) is,
(a-1) the collecting unit setting and managing the basic data for data collection from the social site as a json file;
(a-2) the collection unit setting any one data collection method among API, Restful, crawling, or scraping;
(a-3) step of the collecting unit accessing a social site that matches the basic data;
(a-4) collecting, by a collection unit, data corresponding to the set keyword in a set data collection method; and
(a-5) temporarily storing the data collected by the collecting unit in the database;
Social data collection and analysis method comprising the.
8. The method of claim 7,
The step (b) is,
(b-1) the filter unit giving the collected data a score for each keyword or prohibited word according to a preset criterion;
(b-2) performing, by the filter unit, machine learning on the scored keywords or prohibited words; and
(b-3) a social data collection and analysis method comprising the step of sorting or filtering the data collected in the database by the filter unit based on the machine learning results.
8. The method of claim 7,
Step (c) is,
(c-1) generating analysis information by the analysis unit extracting the frequency of keywords included in the filtered data;
(c-2) indexing the report setting values previously stored in the database by the analysis unit;
(c-3) generating a banner of a page including the analysis information generated by the analysis unit as an image when the index result of the step (c-2) and the report setting value is '0';
(c-4) generating the analysis information generated by the analysis unit as a word count report when the index result of step (c-2) and the report setting value is '1'; and
(c-5) classifying the analysis information generated by the analysis unit according to preset values stored in the database to generate a data flow chart in the form of a time-series chart, diagram, or video.

Images