KR101900285B1 - Dynamic tag cloud system and method in real-time SNS data - Google Patents

Abstract

The present invention provides a dynamic tag cloud method in a dynamic tag cloud system including a client and a server, the method comprising the steps of: transmitting one or more keywords from a user on a client side to a server side, The server stores the received session ID and keyword in a database. The server periodically searches all the keywords in the database. The server performs keyword filtering to collect data including keywords, The server extracts the hashtags from the collected data and counts the number of extracted hashtags, and the server classifies the users using the session ID and visualizes and provides the tag cloud to each user . According to the present invention, it is possible to check the change of the unstructured data, which is difficult to express with existing visualization technology, in real time.

Description

[0001] The present invention relates to a dynamic tag cloud system and method for real-time SNS data,

The present invention relates to SNS data, and more particularly, to dynamic tag cloud technology for real-time SNS data.

Since the late 2000s, the use of social networking services (SNS) has increased rapidly due to popularization of various smart devices. SNS is aimed at sharing opinions, sharing information, and socializing among users. Typically, SNS includes Facebook, Twitter, and Instagram.

Most SNS users mix their text with text, pictures, links, and other types of data and post their opinions. These posts are generated quickly around the globe, and the data is larger than tens of exabytes per year. For example, Facebook has about 2.5 billion shared content every day, and Twitter has about 5,700 shared content every second. The generated SNS data can be divided into big data because it has various kinds of data, fast generation speed and volume. In particular, since the vast majority of people use SNS as a tool for expressing their opinions, SNS data makes it easy to see people's reactions and opinions on issues at a certain point in time, and on specific topics. Therefore, the analysis and visualization results using this are effective to easily identify issue change, product trend, and customer needs.

Real-time data processing technology is a technology to collect and process stream data that occurs in real-time. Apache Storm, Apache Spark, and EsperTech Esper are examples. Among them, Storm is a real-time distributed processing technology that processes data quickly in a large number of distributed servers, and has the advantage that a cluster can be scaled-out as needed. Storm has a master / slave structure similar to Apache Hadoop.

1 is a diagram showing an operation structure of Apache Storm.

Referring to FIG. 1, Nimbus corresponds to a master, and a supervisor corresponds to a slave, which are managed through a distributed coordinator Zookeeper. The nimbus distributes / allocates tasks to the supervisor acting as a slave, periodically grasps the status of the supervisor and responds to the failure. The supervisor directly performs the tasks assigned by nimusbus and reports the status to nimusbus.

2 is a diagram showing a topology of Apache Storm.

As shown in FIG. 2, the Apache Storm expresses a series of operations from input to output of data in a topology when processing a work. The topology consists of a spout and a bolt.

First, the spout receives data from the source of the streaming data, converts it into a tuple, which is the data type used by Storm, and sends it to the vault.

Next, the bolt processes the data received from the spout and transfers it to the next bolt, or sends the result to the output or storage device.

In general, Storm handles tasks in tuples, which makes it difficult to process operations such as aggregation and grouping. To solve this problem, Storm provides a high-level abstraction technology called Trident. Trident compiles a set of tuples and processes them on a window-by-window basis, and can perform operations such as JOIN, AGGREGATION, GROUPING, FUNCTION, and FILTER. So, to calculate the number of words on a per-window basis, we use a trident rather than a tuple-based operation.

Twitter's unofficial Java API, Twitter4J, is a popular method for retrieving data from Twitter, and streaming APIs allow you to retrieve articles posted by Twitter users in real time. Data imported via Twitter4J includes various information such as user information such as ID and area, post information such as article number, article contents, comment information, retweet information, and the like.

Currently, research on visualization technology using Twitter data is actively conducted, and most of the results of Twitter data are visualized by various methods such as chart, map, and tag cloud.

3 is a diagram showing various visualization techniques using twitter data.

In (a) of FIG. 3, Followthhashtag visualizes the statistical analysis result of the Twitter data related to the user's keyword by using a chart, a map, etc., but provides it as a batch unit for a certain period rather than real time.

In (b), TweetStats visualizes the statistical results of a particular user's Twitter data as a graph. However, like Followthehashtag, it analyzes all the data of the user in batches and requires long waiting time after the search request.

The TwitterMap in (c) visualizes on the map based on the local information of the tweets that contain the keywords. You can search the current region, period, etc., but there is no function other than visualization, and it has a disadvantage that it is very slow.

4 is a diagram illustrating a conventional tweeter tag cloud technology.

Referring to FIG. 4, there are (a) trends24.in, (b) Cloud.li, (c) TwitterSheep, and (d) sentiment viz.

First, treands24.in provides the tag cloud for each country in batches on a daily basis. Cloud.li visualizes in real time using each word in a tweet that contains or excludes keywords, but it is not currently running. TwitterSheep visualizes the tag cloud based on the bio of the follower who subscribes to other users' posts, and likewise is not currently running. Next, sentiment viz is visualized based on tweets about 5 minutes including keywords. Tag cloud is divided into upset, happy, releaxed and unhappy according to the tendency of words. Visualize. In addition, Potwiture provides the appropriate photos for the tweets you create, and Tori's Eye has the ability to search for tweets that contain keywords.

Of the various visualization techniques, tag cloud technology is very useful for visually representing the frequency of certain words from a set of documents. In the initial tag cloud study, the words in the document set are listed in order of occurrence, word order, and frequency, and word sizes are shown differently according to the frequency. However, in this method, all the words occupy the same space irrespective of the frequency, and as the number of words increases, the empty space becomes wider and the intuitiveness of words having a higher frequency becomes lower. In order to improve this, a study has been conducted to express the tag cloud by dividing the whole space according to the frequency of the word, but the disadvantage is that the space between the words is large because all the words have the same direction.

Korean Patent No. 10-1060487

SUMMARY OF THE INVENTION It is an object of the present invention to provide a dynamic tag cloud system and method capable of visually confirming a change of unstructured data in real time in order to solve the above problems.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

According to another aspect of the present invention, there is provided a method for dynamic tag cloud in a dynamic tag cloud system including a client and a server, wherein when one or more keywords are input from a user on the client side, Storing the received session ID and keyword in a database, the server periodically searching all the keywords in the database, and the server retrieving keywords including keywords The server extracts a hash tag from the collected data, and counts the number of extracted hash tags. The server identifies the user using the session ID, and visualizes the tag cloud to each user .

The dynamic tag cloud system performs real-time data processing on the tweeter based on the Apache storm represented by a topology composed of a spout and a bolt, performs visualization on twitter data, and the Apache storm processes data using a trident In the keyword search step, the spout may periodically search all the keywords in the database in a polling manner.

Extracting the hashtags from the Apache storm topology and counting the number of extracted hashtags, wherein the spout comprises: filtering tweets containing keywords stored in a database from tweeter; Extracting a hash tag, grouping a window ID and a hash tag into a key, performing an aggregation for each key, and aggregating all the hash tags in the window ID.

The dynamic tag cloud system of the present invention stores a client transmitting a session ID and a keyword of a corresponding user to the server and the received session ID and a keyword in a database when one or more keywords are inputted from a user, Keywords are searched, keywords are filtered to collect the data including the keywords, the hashtags are extracted from the collected data, the number of extracted hashtags is counted, the users are classified using the session ID, And a server that visualizes and provides the tag cloud to the user.

The dynamic tag cloud system performs real-time data processing on the tweeter based on the Apache storm represented by a topology composed of a spout and a bolt, performs visualization on twitter data, and the Apache storm processes data using a trident And the spout can periodically search all the keywords in the database in a polling manner.

Extracting the hashtags from the Apache storm topology and counting the number of extracted hashtags, wherein the spout filters tweets including keywords stored in the database from tweeter, extracts the hashtags from the tweets filtered from the spout, Grouping the window ID and the hash tag into a key, performing an aggregation for each key, and aggregating all the hash tags within the window ID.

In one embodiment of the present invention, the dynamic tag cloud system further includes a hash tag periodically aggregated in the Apache storm and a web socket for asynchronously providing the number of hash tags to a user, and when the session ends, When the session is deleted and the data is aggregated in the Apache storm, the session ID is searched in the database based on the keyword included in the data, and the web socket is classified into the session ID To provide a visualization of the tag cloud to the user.

According to the present invention, it is possible to check the change of the unstructured data, which is difficult to express with existing visualization technology, in real time. Especially, we build the analysis environment by making full use of the functional characteristics of Apache Storm for real-time collection / processing of Twitter data.

In addition, the present invention has the effect of newly providing a real-time, dynamic tag cloud visualization method by improving the limitations of the existing tag cloud. That is, the present invention can be applied to various dynamic analysis environments such as grasping social issues changing in real time, analyzing responses and opinions on specific topics, collecting information on unexpected unexpected situations, and the like.

1 is a diagram showing an operation structure of Apache Storm.
2 is a diagram showing a topology of Apache Storm.
3 is a diagram showing various visualization techniques using twitter data.
4 is a diagram illustrating a conventional tweeter tag cloud technology.
5 is a diagram showing a representative example of a tag cloud.
6 is an example screen image of a first web page that receives a keyword according to an embodiment of the present invention.
7 is an exemplary diagram of a result page for tag cloud visualization in accordance with an embodiment of the present invention.
Figure 8 illustrates various tag clouds in accordance with an embodiment of the present invention.
9 is an overall structure diagram of a dynamic tag cloud system according to an embodiment of the present invention.
10 is a schematic diagram of a trident topology for hash tag aggregation according to an embodiment of the present invention.
11 is an exemplary diagram of data that the topology delivers to the web server.
12 is a trident spurt algorithm for tweet filtering according to an embodiment of the present invention.
13 is an ER diagram of a database used in a dynamic tag cloud system according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted in an ideal or overly formal sense unless expressly defined in the present application Do not.

In the following description of the present invention with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and redundant explanations thereof will be omitted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

The present invention focuses on the visualization of SNS data. Visualization techniques of Big Data include not only simple charts and graphs that numerically represent data but also diagrams and trees that show the relationship of objects, and heat maps that show distribution. Among them, Tag Cloud is a technique to visualize words in a cloud form. By expressing color or size differently according to the frequency of words, it is possible to grasp more specific words more intuitively.

5 is a diagram showing a representative example of a tag cloud.

A common tag cloud technology is to visualize the number of words in a fixed set of documents. However, this method has a limitation in instantly expressing the change of the SNS data generated in real time.

In order to intuitively express the change of the SNS data, the present invention proposes a dynamic visualization technique utilizing a real-time data processing environment. That is, visualization is performed on twitter data among several SNSs, and an implementation and experiment environment based on Apache Storm is constructed to collect and analyze the data.

Apache Storm is an open source platform that distributes data collected in real time on a cluster of servers. In particular, the Apache Storm is used for processing big data streams on Twitter, Yahoo, and Baidu because it has a faster data collection speed than MapReduce in Hadoop.

In addition to the Apache Storm, the present invention designs and implements a web interface for inputting a visualization topic and confirming analysis results for ease of use of the entire system. This allows users to see dynamic tag clouds that change in real time based on visualization themes.

Recently, a variety of JavaScript libraries have emerged to support data visualization based on the Web. In particular, the tag cloud provided by D3.js expresses the tag cloud by rotating the word according to the space between words. With this method, you can minimize the space between words without overlapping words. The tag cloud library of D3.js is used in the present invention.

The system proposed in the present invention constitutes two web pages for a user interface. The first page receives keywords from the user, and the second page periodically receives words and numbers from the Apache storm topology and visualizes them in the tag cloud.

6 is an example screen image of a first web page that receives a keyword according to an embodiment of the present invention.

Referring to FIG. 6, when a keyword is divided by a tab or a comma (,) key, the keyword is added to the list, which allows multiple words separated by a blank to be separated into one keyword.

When the user enters all the keywords, the user can go to the next page by pressing the search button. Clicking the Search button sends a list of keywords to the server and visualizes the window periodically processed by the Apache Storm Topology on the second page. For example, in the present invention, the window time of the spout can be set to process the window in units of 5 seconds.

7 is an exemplary diagram of a result page for tag cloud visualization in accordance with an embodiment of the present invention. 7 is a result page for visualization. The keywords used for illustration are "BigData, Big Data, Hadoop, Hadoop, Real-time, Visualization, Visualization, Tag Cloud, Tag Cloud". As mentioned above, in the present invention, the D3.js library is used for tag cloud visualization.

7 (a) is the initial state of the second page, and the tag cloud is not expressed because there is no processing result yet.

Next, FIG. 7 (b) shows the tag cloud processed by processing the first window data after 5 seconds in FIG. 7 (a). A total of three words were entered in the first window data, and the word "BigData" was weighted higher than the other words and expressed in larger letters.

7 (c) shows that the second window data is processed after five seconds in FIG. 7 (b), and the tag cloud is changed. In the second process, five words are further input.

Figures 7 (d) and 7 (e) continue to process the data and visualize more words, each with the additional word " Analytics "

Finally, Figure 7 (f) is a tag cloud visualized for a 10 minute time period. Of these, the word "BigData" has the highest weight and is therefore represented by the largest character, which can be changed in the future as the window continues to be processed.

Figure 8 illustrates various tag clouds in accordance with an embodiment of the present invention. FIG. 8 shows a result of 20 minutes visualization using different keywords, wherein (a) is the tag cloud of the automobile manufacturer, and (b) is the tag cloud of the mobile device manufacturer.

The keywords used for the tag cloud of FIG. 8 (a) are the car manufacturers "Toyota, Volkswagen, Mercedes-Benz, BMW, Hyundai, Ford, Kia, Volvo, Nissan, Ferrari, Lamborghini, Chevrolet" "ParisMotorShow" showed the most frequency.

Next, as the keyword in FIG. 8 (b), the mobile device manufacturers "Samsung, Apple, LG, Huawei, Lenovo, Xiaomi, Motorola and HTC" were used and the word "Apple"

As described above, the system proposed in the present invention collects hash tags from tweeters and visualizes them as tag clouds. In addition, the present invention can dynamically change the tag cloud every time a tag is collected, and simultaneously processes the keywords input by a plurality of users to provide different visualization results to each user. Therefore, the real-time dynamic tag cloud system proposed by the present invention is useful for real-time analysis of trends according to a subject.

9 is an overall structure diagram of a dynamic tag cloud system according to an embodiment of the present invention.

Referring to FIG. 9, the dynamic tag cloud system according to an embodiment of the present invention is a client on the basis of a Web socket, and a server in the following. The reason for adopting such a client-server structure is that various keywords are received from different users in the Apache Storm cluster, and the results are provided to each user.

In the dynamic tag cloud system of the present invention, the user basically inputs a desired keyword and receives the result as a tag cloud that changes in real time

The operation procedure illustrated in FIG. 9 will be described in detail with respect to the user.

(1) Input keywords: The user enters keywords into the dynamic tag cloud system. The entered keyword is used to filter tweets that include this keyword among the twitter streaming data.

(2) Insert session id and keywords: When the user inputs one or more keywords and clicks the search button, the user's session ID and keywords are transmitted to the server. The session ID is used to identify users in later visualizations. The Web Socket Server stores the session ID and keywords that it receives in the database.

(3) Select keywords: Spout searches all the keywords periodically in the database by polling method. This allows the user's input and topology to operate asynchronously.

(4) Filter Tweet keywords: Spout filters keywords through Twitter4J. In other words, only tweets that contain keywords from Twitter are collected in real time.

(5) Extract hashtags: Extract hashtags from collected tweets.

(6) Aggregate: Aggregate the number of hash tags.

(7) Send hashtags: Pass the aggregated hashtags to the web server.

(8) Visualize tag cloud: Use the session ID to distinguish users and visualize the tag cloud for each user.

The core of the system structure of FIG. 9 is the portion that aggregates the tag and the portion that visualizes it. Therefore, we will describe the Apache storm topology that collects tweets and aggregates hash tags, and describes the web server and database for user management and tag cloud visualization.

In one embodiment of the present invention, a non-Apache storm is used. This is because the hash tags collected by the system proposed by the present invention are aggregated in window form at predetermined time intervals.

10 is a schematic diagram of a trident topology for hash tag aggregation according to an embodiment of the present invention.

Referring to FIG. 10, the topology filters tweets using keywords of users stored in the database, and compiles hash tags of tweets for each window.

The detailed description of each step in FIG. 10 is as follows.

(1) Filter tweet: Spout filters tweets that contain keywords stored in the database from twitter. At this time, the same window ID is assigned to the data generated within the period so as to aggregate data in units of windows of a certain period.

(2) Extract hashtag: Extract a hash tag from a tweet filtered from a spout. At this time, one tweet may include one or more hash tags, and the window ID, the hashtag, and the list of the keywords included in the tweet are transmitted to the next tweet.

(3) Count hashtag: Group the window ID and the hash tag into a key, and perform an aggregation for each key. As a result, hash tags and their number are calculated in the same window ID.

(4) Bind hashtag and count: It compiles all the hash tags in the window ID.

(5) Submit batch: Transfers the previously aggregated window to the web server via Web socket. At this time, JSON can be used as a type of data to be transmitted, and FIG. 11 is an example of expressing this data structure.

11 is an exemplary diagram of data that the topology delivers to the web server.

In Fig. 11, data has a list structure in which data is a key, and a tag, a count of words, and a keyword are stored therein. In the example of FIG. 11, the hash tags of apple, banana, and car are collected five times, three times, and two times as keywords fruit, yellow, and vehicle, respectively.

During the topology configuration phase, in the Filter tweet phase, you need to know which keywords filtered the filtered tweets. This is to support a large number of users, to distinguish input keywords on the basis of the user, and to correctly transmit the aggregated result to each user. However, since the existing Twitter API does not provide a keyword re-extraction function, it is necessary to re-search for a tweet. To this end, the present invention adds a related function to the spout.

12 is a trident spurt algorithm for tweet filtering according to an embodiment of the present invention. The algorithm of FIG. 12 can be implemented in the emitBatch () method in the actual spout.

Referring to FIG. 12, the start time is initialized in line 2 for a window of a certain period, and the repeat is terminated when the difference between the current time and the start time exceeds the window time in line 3. Next, filter the tweet based on the keyword in line 4, and re-search the tweet in lines 5 through 9 to re-extract the keywords contained in the tweet.

Finally, line 10 tuples the list of keywords contained in the window ID, tweets, and tweets and forwards them to the vault. This algorithm is continuously repeated when Apache Storm is executed and generates window data in real time, and processes it at a later stage to perform visualization.

The dynamic tag cloud system proposed by the present invention uses hash tags periodically counted in Apache Storm and a web socket to provide the number of hash tags asynchronously to a user. When users enter different keywords, the results of the hash tag aggregation provided are also different. Therefore, a method for distinguishing a user from a web socket is required, and the present invention uses a session ID of the user for this purpose.

That is, when a user searches for a keyword, the session ID of the user is stored in the database together with the keyword, and when the session ends, the session is deleted from the database. As a result, when the window data is aggregated in Apache Storm, it retrieves the session ID from the database based on the keywords contained in the data, and provides the result to the user corresponding to the session ID.

13 is an ER diagram of a database used in a dynamic tag cloud system according to an embodiment of the present invention.

Referring to FIG. 13, the table keyword_tb manages the keyword entered by the user and the number of users who input the keyword, and the table session_tb manages the session ID of each user. Here, one user can input multiple keywords, and one keyword can be input by multiple users.

The table session_keyword_tb supports the N: M relationship between these keywords and sessions.

However, in the dynamic tag cloud system proposed in the present invention, the frequency of window data is continuously calculated through the Apache storm, and the word of the existing tag cloud And correct the tag cloud to reflect the latest results in frequency. In this way, the tag cloud is dynamically changed at certain times.

As described above, the dynamic tag cloud system proposed in the present invention aggregates the SNS data using the Apache storm and visualizes the result in the tag cloud. Therefore, by using the dynamic tag cloud system of the present invention, it is possible to intuitively determine various issues and changes occurring in the SNS data generated rapidly.

While the present invention has been described with reference to several preferred embodiments, these embodiments are illustrative and not restrictive. It will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit of the invention and the scope of the appended claims.

Claims (7)

A dynamic tag cloud method in a dynamic tag cloud system including a client and a server,
When one or more keywords are inputted from a user on the client side, transmitting a session ID and a keyword of the user to the server side;
Storing the received session ID and keyword in a database;
The server periodically searching all the keywords in the database;
The server performing keyword filtering to collect data including keywords;
The server extracts a hash tag from the collected data, and counts the number of extracted hash tags. And
Wherein the server identifies users using a session ID and visualizes and provides a tag cloud to each user.
The method according to claim 1,
The dynamic tag cloud system performs visualization on twitter data,
Realtime data processing of the tweeter based on Apache Storm, which is represented by a topology composed of spout and bolt,
The Apache Storm processes the data using Trident,
Wherein, in the keyword search step, the spout searches all the keywords periodically in the database in a polling manner.
The method of claim 2,
Extracting the hashtag from the Apache storm topology and counting the number of extracted hashtags,
The spout includes the steps of filtering tweets containing keywords stored in a database from a tweeter, extracting a hash tag from a tweet filtered from the spout, grouping the window ID and the hash tag into a key, Performing an aggregation, and aggregating all the hash tags within the window ID.
A client that transmits a session ID and a keyword of the user to the server when one or more keywords are input from the user; And
Storing the received session ID and keyword in a database, periodically searching all the keywords in the database, performing keyword filtering to collect data including keywords, extracting a hash tag from the collected data, A server that aggregates the number of hash tags, identifies users using session IDs, and provides each user with a visualized tag cloud.
The method of claim 4,
The dynamic tag cloud system performs visualization on twitter data,
Realtime data processing of the tweeter based on Apache Storm, which is represented by a topology composed of spout and bolt,
The Apache Storm processes the data using Trident,
Wherein the spout searches all the keywords periodically in the database in a polling manner.
The method of claim 5,
Extracting the hashtag from the Apache storm topology, counting the number of extracted hashtags,
Spout filters tweets that contain keywords stored in the database from Twitter, extracts the hashtags from the tweets filtered from the spout, groups the window IDs and hashtags into keys, performs an aggregation on each key And aggregates all the hash tags within the window ID.
delete

Images