TW201407390A - Data clustering apparatus and method - Google Patents

Abstract

A data clustering apparatus includes a news database, configured to store data; a calculation module, setting a Global Silhouette Pattern for obtaining a rough clusting number according to the distance relationship between the data; a clustering module, dividing the data into a plurality of clusters by a clustering algorithm according to the rough clusting number and calculating the Intra-Cluster distance of each cluster; and a comparing module, comparing the Intra-Cluster distance with a threshold, wherein when the Intra-Cluster distance is less than the threshold, the cluster corresponding to the Intra-Cluster distance is stored in an event database.

Description

Data grouping device and method

The invention mainly relates to a data grouping technology, in particular to a technique for data grouping by using an Auto-detect Text Recursively Clusting (ADTR) method.

In recent years, due to the rapid development of wireless communication technology, a wide variety of portable and handheld devices, such as mobile phones, smart phones, personal digital assistants (PDAs), tablets (Tablet PCs) ) Electronic products are constantly being introduced in the market, and the functions of these electronic products are increasingly diversified. In addition, due to the convenience of these devices, these devices have also become one of the necessities of people's lives.

In addition to the aforementioned hard-wired wireless communication devices, many software and functions that can be implemented and used with the aforementioned hardware are continuously developed to make the user more convenient, more instant, and more convenient to manage money anytime, anywhere. , work, play or grab information, and more. With the popularity of mobile networks, the popularity of handheld mobile devices has become an important trend for modern people to read news through mobile networks, such as by car and MRT. There are a large number of news source websites, and there are various sorting methods for the newsletter of the news provided by various media at present, which are very complicated. Although it is easy to get a large number of news events, it cannot be tracked or classified according to news events. In addition, the current application software for reading news is mainly for the simple news aggregation (RSS) source of news, and the presentation of news by the news general program. Cause the reader to read the scripture In fact, it is not easy to find the news events that you care about, and it is not easy to find important news events.

In addition, the writing of Chinese news is an unstructured format. Therefore, when artificial intelligence is automatically classified or grouped, it is difficult to easily judge similar articles as a group; on the other hand, when different groups are often different, the news is easy to be compared. Non-representative words are grouped together in the same group, making it less difficult to find the same news event. In addition, when data is grouped, the number of clusters is often very difficult to determine. Generally, it is defined by prior definition or prior observation, and any method needs to be manually assisted.

In view of the above prior art problems, the present invention provides a data grouping technique, in particular, a technique for data grouping by using an Auto-detect Text Recursively Clusting (ADTR) method.

According to an embodiment of the present invention, a data grouping method is provided, comprising the steps of: obtaining a plurality of materials from a news database; and establishing an overall silhouette pattern according to a distance relationship between the data to obtain a preliminary group reference number. And using a cluster algorithm to divide the plurality of data into complex clusters according to the preliminary grouping reference number; calculating an average distance within a group of each of the clusters; and comparing whether the average distance within the group is less than a threshold, wherein If the distance is less than the threshold value, the cluster corresponding to the average distance within the group is stored in an event database.

According to an embodiment of the present invention, a data grouping device includes: a news database for storing complex data; a computing module, a root According to the distance relationship between the above data, an overall silhouette pattern table is established, and a preliminary group reference number is obtained according to the overall silhouette pattern table; a group module is determined by using a group algorithm according to the preliminary grouping reference number. The plurality of data is divided into a plurality of clusters, and then an average distance within a group of each of the clusters is calculated, and a comparison module is configured to compare whether the average distance within the group is less than a threshold value, wherein if the average distance within the group is less than the threshold value , the above cluster corresponding to the average distance within the group is stored in an event database.

1 is a block diagram showing a data grouping apparatus 100 according to an embodiment of the present invention. As shown in the figure, a data grouping apparatus 100 according to an embodiment of the present invention includes a news database 110, a preprocessing module 120, a computing module 130, a grouping module 140, and a comparison module. 150. An event database 160.

According to an embodiment of the present invention, the news database 110 is configured to store and provide plural data, and the data stored in the news database 110 can be updated in an instant, wherein the data described herein can include various types of news events, such as International news, political news, social news, sports news, performing arts news, etc., can also contain various special reports or written materials.

According to an embodiment of the present invention, the pre-processing module 110 is configured to perform a pre-processing operation on the complex data stored in the news database 110, that is, to perform a vectorization process on the complex features of the complex data, so that the data can be converted into one. The spatial model facilitates the processing of data grouping afterwards. The characteristics described herein refer to the content contained in the data after the word breaking or the sentence is broken. The different keywords extracted, for example, the phrase "global warming caused the melting of the Arctic icebergs, thus raising the sea level", the words "global warming", "arctic", "iceberg", "return Keywords such as “flat rise” are extracted, and after the keywords are extracted, these keywords are vectorized and converted into vector points with different weight values. Therefore, after such vectorization processing, the original The data can be converted from text to a spatial model represented by a vector.

According to an embodiment of the invention, the computing module 130 is configured to receive the pre-processed data via the pre-processing module 110, and to establish a global silhouette pattern according to the distance relationship between the spatial models. Then, according to the established overall silhouette style sheet, obtain a preliminary group reference number. More specifically, in this embodiment, the step of the calculation module 130 for establishing a total silhouette pattern table to obtain the preliminary group reference number includes: first, according to the silhouette formula (as shown below), according to the distance between the data in the cluster. The relationship calculates a complex silhouette factor, wherein the silhouette factor described herein is an indicator for assessing the group validity and state that can be used to present a good degree of cluster status. Next, the clustering results for different cluster numbers are used to generate a Global Silhouette value (GS _u ) having a number of different clusters corresponding to a number of clusters, wherein the cluster number ranges from 2 to the above data. Between the total. Finally, the calculation module 130 establishes an overall silhouette style sheet according to the complex overall silhouette value, and records the overall silhouette value (GS _u ) corresponding to the number of each cluster, and sets the number of clusters corresponding to the maximum value of the silhouette value to The initial number of group references, the detailed calculation process will be explained below.

Silhouette formula: Calculate the Silhouette coefficient of an i _th data:

1. Calculate the average distance (a _i ) of the i _th data points for all other data points in the same cluster.

2. For the i _th data points and other clusters, calculate the average distance of this data for all data in each of the other clusters, and take the minimum value (b _i ).

3. Calculate the Silhouette coefficient (S _i ) of i _th , the formula is defined as follows:

The max operand is used to take the maximum value between ai and bi as the denominator and the above equation obeys -1≦Si≦1.

In order to determine the overall silhouette coefficient value (GS _u ), the calculation module 130 first calculates a cluster silhouette value corresponding to each of the clusters for each cluster corresponding to a certain number of clusters. The cluster silhouette value (S _j ) is calculated as follows: Where m is the number of data contained in a single cluster.

For example, when the data is divided into groups c, that is, when the number of clusters is c, if the overall silhouette coefficient value (GS _u ) is to be obtained, it can be obtained by calculating the average of the cluster silhouette values of all the clusters. The overall silhouette coefficient value (GS _u ) is defined as follows:

2 is a schematic diagram showing the overall silhouette value and the number of clusters according to an embodiment of the invention. As shown in FIG. 2, if there is m data in the news database 110, that is, the range of the number of clusters to be calculated is from 2 clusters to divided into m clusters, and the calculation module 130 calculates the number of clusters according to the calculation. Range, calculation divides the data into the overall silhouette values corresponding to the 2~m clusters, and records the calculated overall silhouette values in the overall silhouette pattern table. If the data is divided into N groups, the maximum value of the silhouette value can be obtained. The calculation module 130 sets the N group as the preliminary group reference number.

According to an embodiment of the present invention, the grouping module 140 divides the complex data into a plurality of clusters according to the preliminary grouping reference number, and then calculates an intra-Cluster Distance corresponding to each cluster, wherein The step of the group module 140 calculating the average distance within the group of each cluster includes: first, calculating a center point of the data included in each cluster in the vector space; and then calculating the data included in each cluster to the center point. An average distance, the calculated average distance is the average distance within each group of each cluster. In this embodiment, the intra-group average distance is determined using a Cosine Distance formula, and the intra-group average distance can be used to assess the cohesion of a cluster.

In addition, it is particularly noted that the grouping algorithm used in the above embodiments is a hierarchical grouping algorithm, but is not limited to this algorithm in the present invention, and is anyone skilled in the art, Can be used in After reading this manual, use other suitable grouping algorithms to replace the hierarchical grouping algorithm used in the specification, for example: K-means algorithm in K-partitional clustering, K object (K-medoids) algorithm and so on.

According to an embodiment of the present invention, the comparison module 150 is configured to compare whether the average distance within the cluster is less than a threshold. If the average distance within the group is less than the threshold, the average distance within the group is less than the threshold. The cluster is stored in an event database 160. If the average distance within the group is not less than the threshold, a recursive grouping operation is performed, that is, the data included in the cluster whose average distance within the group is not less than the threshold is retransmitted back. The computing module 130 continues to calculate the overall silhouette pattern table to obtain the preliminary group reference number, and then re-executes the processes performed by the other modules of the data grouping device 100 until all the data is stored in the event database 160. Indicates that all data has been grouped. In particular, regarding the setting of the threshold value, any person skilled in the art can use the appropriate value to set the threshold value (for example, 0.2 to 0.3) after referring to this specification. According to an embodiment of the present invention, the user can obtain the data result grouped by the data grouping device 100 by the event database 160 by using a display unit (not shown) and a search unit (not shown), and display the result. On the display unit.

Figure 3 is a flow chart 300 showing a method of data grouping according to an embodiment of the invention. First, in step S310, a plurality of materials are obtained from a news database; in step S320, a pre-processing operation is performed to perform a vectorization process on the complex features of the data to convert the data into a spatial model; in step S330, According to the distance relationship between the above materials, Establishing an overall silhouette pattern table to obtain a preliminary grouping reference number; in step S340, using the grouping algorithm to divide the plurality of data into a plurality of clusters according to the preliminary grouping reference number; and in step S350, obtaining one of each of the clusters If the average distance within the group is less than the threshold value, the step S370 is performed to store the cluster corresponding to the average distance within the group into an event database. If the average distance in the group is not less than the threshold value, proceed to step S380, and recalculate the silhouette coefficient corresponding to the cluster corresponding to the average distance within the group to obtain the preliminary grouping reference number, that is, return to step S330 again. Continue the steps of data grouping. In addition, it is particularly noted that the grouping algorithm used in the above embodiments is a hierarchical grouping algorithm, but is not limited to this algorithm in the present invention, and is anyone skilled in the art, After reading this specification, other suitable clustering algorithms can be used instead of the hierarchical grouping algorithm used in the specification, for example, the K-means algorithm in partition clustering, K object (K-medoids) algorithm and so on.

4 is a flow chart 400 showing the creation of an overall silhouette style sheet in accordance with an embodiment of the present invention. First, in step S410, according to the distance relationship of the data in the space vector, the one-side shadow formula is used to calculate a complex overall silhouette value corresponding to a number of different cluster numbers, wherein the number of clusters ranges from 2 to the above data. Between the total number; in step S420, the overall silhouette value is recorded in the overall silhouette pattern table; and in step S430, the number of clusters corresponding to the maximum value of the overall silhouette value is set as the preliminary group reference number.

Figure 5 is a flow chart 500 showing the acquisition of the corresponding intra-group average distance for each cluster, in accordance with an embodiment of the present invention. First, in step S510, a center point of the data included in each of the clusters is obtained; in step S520, an average distance of the data included in each of the clusters to the center point is obtained as the group Average distance.

In the face of the needs of users and the existing problems of RSS information sources, in order to give users a better reading experience, we use the data grouping method based on the field of artificial intelligence text mining (Text Mining). Automatic file detection and retransfer grouping (ADTR) to improve the automatic detection of clustering parameters in traditional clustering algorithms, which can cluster messy news to get RSS feeds of different news. However, it is a cluster of similar news events, thus achieving the result of improving the accuracy of news event clustering. In addition, the data grouping method proposed by the present invention can help identify important names and potentially important lexicons in the news, along with the news situation. Different, it can also be applied to different situational changes, and has a good performance on the over-the-counter anti-interference ability of the thesaurus. In addition, compared with the traditional single-pass clustering method, the single-pass grouping method deals with one article at a time, and then compares the current cluster similarity to the clustering basis. However, the data grouping method proposed by the present invention uses the file automatic detection rewinding grouping technique (ADTR) to establish an overall silhouette style sheet for all existing data and find the initial cluster number, and then perform clustering. Recursive clustering algorithm.

The specific features, structures, or properties mentioned in the "invention" or "embodiment" referred to in the specification may be included in at least one embodiment of the present specification. Therefore, the statements that appear in different places "in one implementation In the example, it may not all refer to the same embodiment. In addition, this particular feature, structure, or property may be combined with one or more embodiments in any suitable manner. In addition, it should be noted that the following illustrations are only for the purpose of explanation and are not drawn to the actual scale.

While the present invention has been described with respect to the embodiments of the present invention, the disclosed embodiments are intended to protect the scope of the invention and the scope of the invention. Thus, the embodiments disclosed herein will readily appreciate the advantages described above for anyone skilled in the art. After reading the contents of the specification, any person skilled in the art can make appropriate changes and substitutions in a broad manner without departing from the spirit and scope of the invention.

100‧‧‧ data grouping device

110‧‧‧News database

120‧‧‧Pre-processing module

130‧‧‧Computation Module

140‧‧‧Group Module

150‧‧‧Comparative Module

160‧‧‧ Event Database

300, 400, 500‧‧‧ flow chart

S310, S320, S330, S340, S350, S360, S370, S380, S410, S420, S430, S510, S520‧‧ steps

1 is a block diagram showing a data grouping apparatus 100 according to an embodiment of the present invention.

2 is a schematic diagram showing the correspondence between the overall silhouette value and the number of clusters according to an embodiment of the invention.

Figure 3 is a flow chart 300 showing a method of data grouping according to an embodiment of the invention.

4 is a flow chart 400 showing the creation of an overall silhouette style sheet in accordance with an embodiment of the present invention.

Figure 5 is a flow chart 500 showing the calculation of the corresponding intra-group average distance for each cluster, in accordance with an embodiment of the present invention.

100‧‧‧ data grouping device

110‧‧‧News database

120‧‧‧Pre-processing module

130‧‧‧Computation Module

140‧‧‧Group Module

150‧‧‧Comparative Module

160‧‧‧ Event Database

Claims (12)

A data grouping device includes: a news database for storing a plurality of data; a computing module, based on a distance relationship between the data, for establishing an overall silhouette style sheet, and then obtaining the overall silhouette style sheet according to the overall pattern a preliminary grouping reference number; a grouping module, according to the preliminary grouping reference number, using a grouping algorithm to divide the complex data into a plurality of clusters, and then calculating an intra-Cluster distance of each of the clusters, and a The comparison module is configured to compare whether the average distance in the group is less than a threshold value, wherein if the average distance within the group is less than the threshold value, the cluster corresponding to the average distance within the group is stored in an event database. The data grouping device of claim 1, further comprising a pre-processing module for performing a pre-processing operation on the plurality of data to perform a vectorization process on the plurality of features of the data to convert the data. Into a space model. The data grouping device of claim 1, wherein if the average distance within the group is not less than the threshold value, the cluster corresponding to the average distance within the group is returned to the computing module to establish the overall The pattern style sheet is obtained, and the number of preliminary grouping references mentioned above is obtained. The data grouping device of claim 1, wherein the step of establishing the overall silhouette pattern table by the calculating module comprises: calculating a complex silhouette coefficient by using a one-side shadow formula according to the distance relationship between the data to generate a corresponding a complex overall silhouette value (GS _u ) of a number of different clusters in a range of clusters, wherein the number of clusters ranges from 2 to the total number of the above data; recording the above-mentioned silhouette values to the overall grouping pattern style sheet; and The number of clusters of the maximum value of the overall silhouette value is set to the above-mentioned preliminary grouping reference number. The data grouping device of claim 1, wherein the step of calculating, by the grouping module, the average distance within the group of each of the clusters comprises: obtaining a center point of the data included in each of the clusters; And obtaining an average distance from the above-mentioned center point of the above-mentioned data included in each of the clusters, wherein the average distance is the average distance within the group. The data grouping device according to claim 1, wherein the grouping algorithm is a hierarchical grouping algorithm. A data grouping method includes the following steps: obtaining a plurality of materials from a news database; and establishing an overall silhouette style sheet according to a distance relationship between the data to obtain a preliminary grouping reference number; using one according to the preliminary grouping reference number The clustering algorithm divides the plurality of data into a plurality of clusters; obtains an average distance within a group of each of the clusters; and compares whether the average distance within the group is less than a threshold, wherein if the average distance within the group is less than the threshold, the corresponding The above clusters of average distances within the above groups are stored in an event database. For example, the data grouping method described in claim 7 of the patent application, wherein before the establishment of the above-mentioned overall silhouette style sheet, A pre-processing operation is performed to perform a vectorization process on the complex features of the above data to convert the data into a spatial model. The data grouping method according to claim 7, wherein if the average distance within the group is not less than the threshold value, the cluster corresponding to the average distance within the group is re-established to form the overall silhouette pattern to obtain the preliminary grouping. Reference number. The data grouping method according to Item 7 of the patent application scope, wherein the step of establishing the overall silhouette pattern table comprises: calculating a complex silhouette coefficient by using a one-side shadow formula according to the distance relationship between the data to generate a corresponding number of clusters a complex overall silhouette value of the number of different clusters, wherein the number of clusters ranges from 2 to the total number of the above data; recording the silhouette value to the overall silhouette pattern table; and the cluster corresponding to the maximum value of the overall silhouette value The number is set to the above preliminary group reference number. The data grouping method of claim 7, wherein the step of obtaining the average distance within the group of each of the clusters comprises: obtaining a center point of the above-mentioned data included in each of the clusters; and obtaining each The average distance of the above-mentioned data included in the cluster to the center point is taken as the average distance within the group. For example, the data grouping method described in claim 7 is wherein the grouping algorithm is a hierarchical grouping algorithm.