TW201822030A - Website classification method and computer program product thereof capable of classifying the target website into various categories according to the relevancy - Google Patents

Abstract

The present invention provides a website classification method and a computer program product thereof. The website classification method comprises the steps: capturing one or a plurality of website description information under the condition that the website description content can be obtained from a target website; analyzing the relevancy between each website description information and one or more category keywords of categories to be classified so as to classify the target website into a category according to the relevancy. Automatic website classification operation can be realized by the aforementioned operations, to thereby replace the manual classification employed by prior art.

Description

   URL classification method and computer program product   

The invention relates to a method for classifying a website address and a computer program product thereof, and in particular to a method for classifying a website address and a computer program product thereof that can be automatically classified without human labor.

In many network management applications, such as spam or pornographic web filtering, a large number of networks need to be manually classified by network service providers. As the number of websites increases, the conventional classification method is implemented. Face great challenges.

In addition, the conventional classification scheme usually needs to provide complete web pages and categories during the training phase, and configure one-to-one manual tagging results. When the categories are changed, all of them need to be re-marked, so that the cost of performing URL classification operations has always High.

In summary, how to provide a solution that can solve the foregoing problems is a technical problem that needs to be solved in the art.

In order to solve the problems disclosed previously, the object of the present invention is to provide a technical solution for URL classification.

In order to achieve the above object, the present invention proposes a method for classifying a website. It includes the condition that the destination URL can obtain the URL description content, then extract one or more URL description information, and analyze the correlation between each URL description information and one or more category keywords of the category to be classified, based on Relevance categorizes destination URLs into categories.

To achieve the above object, the present invention provides a computer program product for categorizing web addresses. When a computer device loads and executes the computer program product, the steps described in the foregoing method can be completed.

In summary, the URL classification method and computer program product of the present invention analyze the correlation between the URL description information and the category keywords, and automatically classify the target URL into a specific category based on the correlation, thereby reducing the one-to-one URL. The cost of manual tagging with categories.

S1000 ~ S1200‧‧‧step

FIG. 1 is a flowchart of establishing a classification mode according to an implementation example of the present case.

FIG. 2 is a flowchart of establishing a website classification model according to an embodiment of the present invention.

Figure 3 is a flowchart of the steps of fetching and preprocessing the URL description.

FIG. 4 is a flowchart of steps for establishing and implementing a website classification model in this case.

FIG. 5 is a flowchart of steps for judging a new URL category and generating a classification result of the new URL for this case.

The following describes specific embodiments to illustrate the implementation of the present invention, but it is not intended to limit the scope of the present invention.

The first embodiment of the present invention provides a method for classifying a website. This method includes extracting one or more URL description information under the condition that the destination URL can obtain the URL description content, and analyzing the correlation between each URL description information and one or more category keywords of the category to be classified. Classify destination URLs into categories based on relevance.

In another embodiment, under the condition that the destination URL fails to obtain the description content of the URL, the destination URL extracts one or more URL keywords and compares the URL keywords with one or more training set URL keywords. Match to classify the destination URL into the category corresponding to the relevant training set URL keywords.

In another embodiment, the aforementioned method analyzes the relevance of the URL keywords and the training set URL keywords by the number of matching strings. In another embodiment, the aforementioned method analyzes the relevance based on the URL level of the matching string. In another embodiment, the foregoing method performs weight calculation based on the number of matching strings and the level of the web site where it is located to analyze its relevance.

In another embodiment, the aforementioned method analyzes the probability distribution between the description information of each URL and the category keywords to calculate the degree of relevance. In another embodiment, the aforementioned method analyzes the probability distribution based on a potential Dirichlet allocation model. In another embodiment, the potential Dirichlet allocation model of the aforementioned method uses Gibbs sampling to estimate the degree of association. In another embodiment, the foregoing method further translates keywords of the destination URL for classification.

The second embodiment of the present invention further provides a computer program product for categorizing web addresses. When a computer device loads and executes the computer program product, the steps described in the foregoing method can be completed.

In the following, the present invention is exemplified by the URL classification method of the first embodiment. However, the computer program product of the URL classification of the second embodiment can also achieve the same or similar technical effects.

Please refer to FIG. 1, which is a flowchart of establishing a classification mode according to an implementation example of this case. The steps are explained as follows:

S1000: Establish a classification model and output its model training results.

S1200: Determine the new URL category and generate a classification result of the new URL.

Please refer to Figure 2 for the flowchart of establishing the URL classification model of this case, including the following steps:

S1110: Create a training set of URLs.

S1120: URL description retrieval and pre-processing.

S1130: Generate a URL description set.

S1140: Establish category knowledge.

S1150: Establish and execute the URL classification model.

The aforementioned S1100 to S1130 are the data preparation stage. First, S1110 establishes a training set URL, and collects a group of URLs that can retrieve URL descriptions. Next, S1120 executes pre-processing to extract the description content of the URL. Figure 3 shows a detailed flow chart of S1120. First, S1121 extracts the URL description for each URL in the URL training set. The URL description can be obtained by different methods, such as using web crawler technology to retrieve the URL content. S1122 uses natural language technology to segment words and restore parts of speech. S1123 judges whether the hyphenation of the restored part of speech is a stop word (the stop word is usually considered to be unimportant or discriminating words). Finally, S1124 further removes the words of a specific part of speech, and only retains the specific part of speech of interest. In this case, the words left after S1122 to 1124 are treated as keywords. S1130 collects all the URL description keywords and then generates a single file.

The S1140 establishment category knowledge can provide interest categories and several keywords of each category, known category URL classification knowledge, and known similar URL database. This step can assist in the establishment and implementation of the S1150 website classification model, which can be regarded as a semi-supervised method.

After obtaining the S1130 URL description set and S1140 category knowledge, they are considered together as model input data and S1150 is executed to establish the URL classification model. The URL classification modeling mechanism here is based on the Latent Dirichlet allocation model (LDa). ). LDA is a probabilistic generation model that can be used to identify hidden topic messages in large-scale document sets. The model assumes that a document is a set of words, and there is no sequential relationship between words. It can contain multiple topics, and each word in the file is generated by one of them. The original LDA is an unsupervised learning algorithm, that is, there is no need to manually tag information, all that is needed is the file set and the number of specified topics. This case uses this same concept to find several categories hidden in large-scale URL descriptions, that is, files are URLs, file sets are URL description sets, and topics are categories. The model finally produces two matrices, namely the website-category relationship matrix θ and the category-keyword relationship matrix φ, and these two matrices can be obtained by the probability inference method.

The method used for LDA probability inference in this case is Gibbs sampling. During the implementation process, it repeatedly estimates the probability that each keyword belongs to each category, and finally selects a category based on the probability distribution of this category and gives this keyword (that is, forms a keyword Corresponding to the category), this whole process is executed by S1150. Given K = {k ₁ , k ₂ , ..., k _n } is all category types, W = {w ₁ , w ₂ , ..., w _v } is a set of all keywords, and D = {d ₁ , d ₂ , ..., d _m } is the set of all URL descriptions. For the following simplified symbols, k is used to represent a certain category k _j , 1 j n; w represents a certain keyword w _i , 1 i v; d represents a URL description d _f , 1 f m. As shown in equation (1), the probability p (w → k) of the keyword w belonging to the category k is estimated by two expressions, and p (w | k) is the probability of the keyword w in the category k; ; P (k | d _w ) is the ratio of the number of keywords assigned to category k to all keywords in the URL description d _w to which w belongs.

In S1150, when all keywords are associated with categories through formula (1), it is called a round. When the number of training rounds reaches a set threshold, the training phase is terminated. According to the correspondence between each keyword and category, The website-category relationship matrix θ and the category-keyword relationship matrix φ can be calculated, where θ describes the category probability distribution of each website, and φ describes the category probability distribution of each keyword.

In S1140 category knowledge, interest categories and their keywords (known category keywords) should be collected first. An interest category can be composed of multiple keywords. This step can also use other known knowledge to further improve the performance of the classification model, including URL classification knowledge and similar URL databases. Known URL classification Knowledge description URL corresponds to its category. For example, www.youtube.com.tw is considered to belong to the category of audiovisual entertainment; Known similar URL library record URLs correspond to other similar URLs. Similar sites are www.vimeo.com and www.dailymotion.com.

In the establishment and implementation of the S1150 URL classification model, this case uses the probability method of replacing the equation (1) with the correspondence between the interest category and its keywords in the S1140 category knowledge to directly generate the correspondence; using the knowledge of known URL categories and known similar URL libraries To adjust equation (1). The details of S1150 are shown in Figure 4. S1150A is for each keyword w in the training set, and S1150B is the keyword set where w is already included in category k. Then execute S1151 to directly assign w to category k; if not, but If you have both known URL category and known similar URL database knowledge (S1150C), execute S1152, use formula (2) to determine the category of w, and repeat the same steps for the next keyword in the training set. Classify knowledge for known URLs, Known for similar URL library knowledge (S1150D), where Represents a known probability. If there is no category knowledge, execute S1153 to determine the category of w using formula (1) and repeat the same steps for the next keyword in the training set. If you only have knowledge of known URL classifications, execute S1154 to determine w using formula (3). And repeat the same steps for the next keyword in the training set; if there is only a known similar URL database, execute S1155 to determine the category of w using formula (4) and repeat the same steps for the next keyword in the training set.

In summary, the input URL description set of the URL classification model, combined with known category keywords, known URL category knowledge, and known similar URL libraries to help improve the model, its final output is a classification model, namely θ and φ. In S1140 category knowledge, the known interest category and its keywords directly determine the correspondence between keywords and categories in Gibbs sampling, so that the keywords will not be assigned to other unrelated categories, and the formation of this correspondence relationship will also be affected. The category mapping of other keywords in the same URL description then affects the category distribution of other URL descriptions with the same keyword. Even when a keyword does not belong to any known category keyword, knowledge of known URL classifications and similar URL libraries can help this keyword be assigned to a more appropriate category. The design improvement of this case can improve the performance of the classification model. At the same time, θ records the probability that each URL belongs to each category, breaking the limit of one category per URL in the past.

Please refer to FIG. 5, which is a flowchart showing that S1200 determines a new URL category. For the classification problem of the new website, based on the model training results obtained in S1100, this case provides two classification mechanisms, namely an online classification mechanism and an offline classification mechanism. If the new URL has a retrievable URL description (S1200A, S1200B), the online classification mechanism is used; otherwise, the offline classification mechanism is used.

When a new URL is obtained, S1210 extracts keywords after pre-processing the URL description like S1120, leaving only the more meaningful words called the keyword set d. Next, in S1220, a translation technique is used to translate all keywords in d into the corresponding language, and the keywords described in the original URL are combined to form an augmented URL description d ' . Finally, S1230 will calculate the probability that each keyword in d ' belongs to each category, as shown in equation (5), | d ' | is the number of keywords in d ' , and E (w _g ) is the The weight of g keywords w _g should be between [0,1], and p (w _g | k) is obtained from θ. The weight calculation should be determined according to the use situation, and the weight deformation that can achieve the same effect should be regarded as the equivalent implementation of this method. In other words, formula (5) reflects that if most of the keywords in d ' appear in the social category, it means that d ' also has a higher probability in the social category. It is worth noting that because each URL or keyword in the model result obtained by S1100 can belong to multiple categories, the new URL can also be assigned to multiple categories through this design. If the user wants to classify only one URL To a category, you can directly select the category with the highest probability.

When real-time URL category identification or dynamic URL description retrieval is not allowed, an offline classification mechanism is used. The design spirit behind it is that for two similar URLs, their categories should be similar. Therefore, the offline classification mechanism will compare the training URL with the new URL in all training sets, and assign the category distribution of its most similar URL to its new URL. In this case, each of the URLs separated by "." Or "/" is regarded as a string, and the similarity between the two URLs is defined as the number of longest string matches, and the longest matching sequential string is It is called the longest common subsequence, and the number of strings in the sequence is defined as the length. For example, as shown in Table 1, suppose the new website mod.cht.com.tw/channelrw/01.php has similarities with s1, s2, s3, and s4 of 6, 4, 3, and 6, respectively. In addition, this case defines a unit whose URL can be divided by "/" as layers. For example, mod.cht.com.tw/channelrw/01.php has three layers, which are mod.cht.com.tw, channelrw and 01. .php. Strings also have different importance in different layers of the URL. Generally speaking, strings that belong to the upper layers are more representative and unique. Therefore, this case considers a weighted computer system that adjusts the similarity according to the level at which the matching strings are located. The weight of the keyword w is {1} / {the level to which w belongs}. As shown in Table 2, the similarity between the new website and s1, s2, s3, and s4 is changed to 4.83, 4, 3, and 4.67. The weight deformation that can achieve the same effect should be regarded as the equivalent implementation of this method.

In summary, S1240 will calculate the similarity for a new URL, identifying the longest subsequence length of each URL in the training set and the level of matching strings. Finally, S1250 finds the URL with the highest similarity and assigns its URL in the category distribution to the new URL. In the example in Table 2, s1 is the most similar to the new URL, so the category probability distribution of the new URL is the same as s1. When there is more than one URL with the same maximum similarity, the category distribution of these URLs is added up and assigned to the new URL.

In S1110, a bunch of training set URLs are first established. As indicated in the first column of Table III, only three representatives are listed here. After extracting the URL description and pre-processing in S1120, the URL description set of S1130 can be obtained, which is shown in the second column of Table 3.

Tables 4, 5 and 6 are examples of S1140 category knowledge. Table 4 is an example of keywords of known categories, which are divided into three categories: social, audio and video, and e-commerce. The keywords should be determined by application and target. Table 5 is an example of known URL classification knowledge. It is assumed that it is obtained through the category to which the multi-person tagged URL belongs, and the calculation method of the category probability is {number of people who think of category k} / {number of people participating in feedback}. Table 6 is an example of a known similar URL library, assuming Google search results.

Next, the S1150 model is established. The model establishment process is to continuously adjust the correspondence between each keyword and category according to different given conditions. The process is shown in Figure 4. First, determine whether the keyword is already included in a certain category of keywords. If it is, then the category is directly corresponding to the keyword; if not, the formula should be selected according to the known knowledge conditions ( 1), (2), (3), or (4) to determine the corresponding relationship. If the keyword is not in the keyword and the URL description in which it is located has no known classification knowledge and similar URL database, then use the formula (1) to determine its category; when the keyword is not a keyword and the URL where it is located has both known classification knowledge and a known similar URL database, then formula (2) is used; when the keyword is not critical When the word is located and the URL where the word is located only has knowledge of known URL classifications, then formula (3) is used; when the keyword is not in a keyword and the URL where it is located has only known similar URL libraries, the formula is used (4). After executing S1150, the category-key matrix φ is one of its results.

In Table 7, the obtained φ is used. In this case, a representative keyword in each category can be found to represent this category. The representativeness of a category keyword is determined by the probability of this keyword in the category. In other words, the greater the probability Can represent this category, this table only lists the top five keyword representations. Table 8 is the second result of the classification model, that is, the website-category relationship matrix θ, which lists the category probability distribution of each website. It is particularly noted that the probability distribution of this category is different from the known classification knowledge of URL categories. This category probability is calculated from the keywords and category correspondence in the URL description. Because the model assumes a relationship, each URL has a certain probability of belonging to each category, but if the probability is too low, it is temporarily represented by 0 here.

Finally, the model result obtained in S1200 identifies the category of the new website. Suppose you want to identify which category the new URL shopping.friday.tw belongs to. First, S1210 first retrieves the URL description, as shown in the first column of Table 9 (this is just an example, the URL description can be different according to individual scenarios). The keywords are formed after removing the special part of speech, as shown in the second column of Table 9. S1230 uses automatic translation technology to translate keywords. The language of translation should be determined according to the situation. Here, English is used as an example. Finally, the original keywords and the translated keywords are used to form an augmented URL description, as shown in the third column of Table 9. Suppose the weight of the keyword w is the number of occurrences of {w in d _w × } The formula is calculated, as shown in the third column of each category in Table 7, and for each keyword in the description of the expanded URL, go to Table 7 to find the probability of each category, and finally use the weighted average according to formula (5) to obtain Probability of this URL in category. For example, shopping.friday.tw is calculated using formula (5) based on the description of the augmented URL. The classification probability distribution may be {Social = 0.05, Video = 0.05, E-commerce = 0.9}, because its keywords are in the e-commerce category. There is a higher probability. In addition, assuming that the new website www.momoshop.com.tw/category/food does not have (or cannot obtain) a URL description, S1240 calculates the similarity between Table 8 and all the training set URLs, and the similarities are 0 and 0, respectively. With 1.

Therefore, S1250 assigned the category probability of www.momoshop.com.tw to this new URL. In the end, the probability of this new URL category is {social = 0.01, video = 0.01, e-commerce = 0.98}. If each URL is divided into only one category, the classification result can be set to the category with the highest probability of ownership, namely, the e-commerce category.

The above detailed description is a specific description of a feasible embodiment of the present invention, but this embodiment is not intended to limit the patent scope of the present invention. Any equivalent implementation or change that does not depart from the technical spirit of the present invention should be included in Within the scope of the patent in this case.

Claims (10)

    A method for classifying a URL, comprising: if the destination URL can obtain the URL description content, extracting one or more URL description information, and analyzing each URL description information and one or more category keywords of the category to be classified Of relevance to categorize the destination URL into that category based on that relevance.         According to the URL classification method described in claim 1, and in the condition that the destination URL fails to obtain the description content of the URL, one or more URL keywords are extracted from the destination URL, and the URL keywords are combined with one or Multiple training set URL keywords are compared to classify the target URL into the category corresponding to the training set URL keywords that are relevant.         The URL classification method described in claim 2 is based on the number of matching strings between the URL keywords and the training set URL keywords to analyze their relevance.         The URL classification method described in claim 3 is based on analyzing the relevance of the URL level where the matching string is located.         The URL classification method described in claim 4 is based on the number of matching strings and the URL level of the website to calculate the weight to analyze its relevance.         The URL classification method described in claim 1, analyzes the probability distribution between each URL description and the keywords in the category to calculate the relevance.         The URL classification method described in claim 6, analyzes the probability distribution based on a potential Dirichlet allocation model.         The URL classification method according to claim 7, wherein the latent Dirichlet allocation model uses Gibbs sampling to estimate the degree of relevance.         According to the URL classification method described in claim 1, the keywords of the destination URL are translated to perform the classification operation.         A website classification computer program product. When a computer device loads and executes the computer program product, the method described in any one of claims 1 to 9 can be completed.