TWI757957B - Automatic classification method and system of webpages - Google Patents

Abstract

An automatic classification method and system of webpages is provided. The automatic classification method includes the following steps. A plurality of keywords contained in a webpage of a website is extracted using an application programming interface (API), and an identifier (ID) is given to each keywords contained in the webpage. A TF-IDF value of each keywords contained in the webpage is calculated based on the number of all webpages in the website. A matrix is generated according to the identifier of each keywords contained in the webpage and the TF-IDF value of each keywords contained in the webpage. The matrix is inputted into a webpage classification model to generate a predicted category name. The webpage is saved with the predicted category name.

Description

Web page automatic classification method and system

The present invention relates to an automatic classification method and system, and in particular, to an automatic classification method and system of web pages.

The Internet has become an indispensable part of life. People often browse web pages through computers. When they browse to favorite web pages or important web pages, they can save web pages through the browser's function, such as saving web pages in "My Favorites". , so that the next time you open your browser, you can quickly browse the saved pages from the pages saved in Favorites.

However, when storing web pages, users often spend a lot of time thinking about the category names of the web pages. If the category names are inaccurate, it will be difficult for users to quickly find the previously stored web pages for browsing when the browser is opened next time, resulting in inconvenience. .

Therefore, how to provide accurate classification names for web pages has become the direction of the industry's efforts.

The present invention relates to an automatic classification method and system for web pages.

According to an embodiment of the present invention, an automatic classification method for web pages is provided. The automatic classification method of web pages includes the following steps. An application programming interface (API) is used to retrieve a plurality of keywords contained in a webpage in a website, and an identifier (ID) is given to each keyword contained in the webpage. Taking all webpages in the website as the parent, calculate the TF-IDF value of each keyword contained in the webpage. A matrix is generated according to the identifier of each keyword contained in the web page and the TF-IDF value of each keyword contained in the web page. The matrix is input to a web page classification model to generate a predicted category name. Save the page with the predicted category name.

According to another embodiment of the present invention, an automatic classification system for web pages is provided. The automatic classification system for webpages includes a processor and a webpage classification model. The processor is used for using an application programming interface (API) to retrieve a plurality of keywords contained in a webpage in a website, and assigns an identifier (ID) to each keyword contained in the webpage. The processor is used for calculating the TF-IDF value of each keyword included in the webpage by taking all the webpages in the website as the parent. The processor is used for generating a matrix according to the identifier of each keyword contained in the web page and the TF-IDF value of each keyword contained in the web page. The processor is used to input the matrix to the webpage classification model to generate a predicted classification name. The processor is used for storing the web page with the predicted category name.

In order to have a better understanding of the above-mentioned and other aspects of the present invention, the following specific examples are given and described in detail in conjunction with the accompanying drawings as follows:

100: Automatic Classification System

110: Processor

120-1, 120-2, 120-10: web page

120: Website

130: Web Page Classification Model

140,160: Website

140-1, 140-2, 140-8, 160-1, 160-2, 160-3: web page

180: Website

180-1, 180-2: web pages

180-11, 180-21, 180-22, 180-23: Articles

API: Application Programming Interface

CN _120-1, CN _140-1 , CN _140-8 , CN _160-1 , CN _160-3 , CN _180-11 : Classification name

KW ₁₂₀₁ ,KW ₁₂₀₂ ,KW _1205, KW ₁₄₀₁ ,KW ₁₄₀₂ ,KW _1406,

KW ₁₈₀₁ , KW ₁₈₀₂ , KW ₁₈₀₆ : Keywords

PCN, PCN _180-11 : Predicted class names

MX,MX _140-1 ,MX _140-8 ,MX _160-1 ,MX _160-3 ,MX _180-11 : Matrix

S110,S120,S130,S140,S150,S210,S220,S230,S240,S310,S320,S330,S340,S350,S360,S370,S410,S420,S430,S440,S450,S460,S510,S520,S530, S540, S550, S560, S570: Steps

FIG. 1 shows a block diagram of an automatic classification system for webpages and a website according to an embodiment of the present invention.

FIG. 2 shows a flow chart of a method for automatically classifying webpages according to an embodiment of the present invention.

FIG. 3 shows a schematic diagram of a web page according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a matrix according to an embodiment of the present invention.

FIG. 5 shows a block diagram of an automatic classification system for webpages and a website according to another embodiment of the present invention.

FIG. 6 is a flowchart illustrating a training method of the webpage classification model 130 in the automatic webpage classification method according to another embodiment of the present invention.

FIG. 7 is a schematic diagram of a web page according to another embodiment of the present invention.

FIG. 8 is a schematic diagram of a matrix according to another embodiment of the present invention.

FIG. 9 is a flow chart of a method for automatically classifying webpages according to another embodiment of the present invention.

FIG. 10 shows a block diagram of an automatic classification system for webpages and a website according to another embodiment of the present invention.

FIG. 11 is a flowchart illustrating a method for automatically classifying webpages according to another embodiment of the present invention.

FIG. 12 shows a schematic diagram of an article according to an embodiment of the present invention.

FIG. 13 is a flowchart illustrating a method for automatically classifying webpages according to another embodiment of the present invention.

Please refer to FIG. 1 , which shows a block diagram of an automatic web page classification system 100 and a website 120 according to an embodiment of the present invention. The automatic classification system 100 for webpages includes a processor 110 and a webpage classification model 130 . The automatic classification system 100 of web pages is, for example, a smart phone, a tablet computer, a notebook computer or a desktop computer. Website 120 includes a plurality of web pages, such as web pages 120-1, 120-2, . . . , 120-10. The automatic classification system 100 for webpages can browse webpages 120-1, 120-2, . . . , 120-10 in the website 120, and can also retrieve webpages 120-1, 120- 2, ..., information in 120-10.

The operation of the above components is described in detail with the flow chart below. Please refer to FIG. 2 , which shows a flowchart of a method for automatically classifying webpages according to an embodiment of the present invention.

Step S110 , using an application programming interface to extract a plurality of keywords contained in a web page of a website, and assign an identifier (ID) to each keyword contained in the web page. Please refer to FIG. 3, which illustrates a schematic diagram of a web page 120-1 according to an embodiment of the present invention. The web page 120-1 contains the category name CN _120-1 , and the keywords KW ₁₂₀₁ , KW ₁₂₀₂ , . . . , KW ₁₂₀₅ . For example, the category name is "sports news" or "political news".. etc. Keywords such as "Chinese Team", "Kickoff", "Home Run", "President" or "Mayor"...etc. The processor 110 retrieves a plurality of keywords KW ₁₂₀₁ , _{KW 1202} , _. , KW ₁₂₀₂ , ..., KW ₁₂₀₅ - an identifier. Each keyword KW ₁₂₀₁ , KW ₁₂₀₂ , . . . , KW ₁₂₀₅ is given a different identifier. In one embodiment, the API has a dictionary, and the API gives each keyword KW ₁₂₀₁ , KW ₁₂₀₂ , . . . , KW ₁₂₀₅ a different identifier according to the dictionary.

Step S120: Calculate the TF-IDF value of each keyword included in the webpage based on the number of all webpages in the website. The calculation of the TF-IDF value requires the definition of a matrix. In this embodiment, the parent is all the web pages 120-1, 120-2, . . . , 120-10 in the website 120. The processor 110 calculates each keyword KW ₁₂₀₁ , KW ₁₂₀₂ , . , TF-IDF value of KW ₁₂₀₅ .

In step S130, a matrix is generated according to the identifier of each keyword contained in the webpage and the TF-IDF value of each keyword contained in the webpage. Please refer to FIG. 4 , which is a schematic diagram of a matrix MX according to an embodiment of the present invention. The processor 110 determines the identifier of each keyword KW ₁₂₀₁ , KW ₁₂₀₂ , . . . , KW ₁₂₀₅ included in the web page 120-1 and each keyword KW ₁₂₀₁ , KW ₁₂₀₂ , . The TF-IDF values of the KW ₁₂₀₅ yield the matrix MX. In other words, a web page 120-1 corresponds to a matrix MX.

Step S140, input the matrix into the webpage classification model to generate a predicted classification name. The processor 110 inputs the matrix MX to the webpage classification model 130 to generate a predicted classification name PCN.

In step S150, the webpage is stored with the predicted category name. The processor 110 stores the web page 120-1 with the predicted category name PCN. In one embodiment, executing Before step S110, the processor 110 determines whether the webpage has been previously stored, and when the webpage has not been previously stored, steps S110 to S150 are executed. For example, the processor 110 creates a custom field in the browser's cookie to record whether the web page 120-1 has been previously stored.

In this way, the automatic web classification method proposed in this case can generate a matrix for the identifier and TF-IDF value corresponding to each keyword contained in a web page, and input it into the trained web page classification model to accurately Generates the category name of the web page.

Please refer to Figures 5 and 6. FIG. 5 illustrates a block diagram of an automatic classification system 100 for web pages and websites 140 and 160 according to another embodiment of the present invention. FIG. 6 is a flowchart illustrating a training method of the webpage classification model 130 in the automatic webpage classification method according to another embodiment of the present invention. The website 140 includes web pages 140-1, 140-2, . . . , 140-8. Website 160 includes web pages 160-1, 160-2, 160-3. For the convenience of description, the following two websites 140 and 160 are used as training data to train the webpage classification model 130 as an example.

In step S210, the application programming interface is used to extract a plurality of keywords and a category name contained in the webpage of the website, and an identifier is given to each keyword contained in the webpage. Please refer to FIG. 7, which shows a schematic diagram of a web page 140-1 according to another embodiment of the present invention. The web page 140-1 contains the category name CN _140-1 , and the keywords KW ₁₄₀₁ , KW ₁₄₀₂ , . . . , KW ₁₄₀₆ . The processor 110 uses the application programming interface to retrieve a plurality of keywords _{KW 1401} , _{KW 1402} , . Each keyword of KW ₁₄₀₁ , KW ₁₄₀₂ , . . . , KW ₁₄₀₆ has an identifier.

Step S220: Calculate the TF-IDF value of each keyword included in the webpage based on the number of all webpages in the plurality of websites. The calculation of the TF-IDF value requires the definition of a matrix. In this embodiment, the parent is all web pages 140-1, 140-2, . The processor 110 calculates based on the number (11) of all web pages 140-1, 140-2, . The TF-IDF value of each keyword KW ₁₄₀₁ , KW ₁₄₀₂ , . . . , KW ₁₄₀₆ contained in the web page 140-1.

Step S230, generating a matrix according to the identifier of each keyword contained in the webpage and the TF-IDF value of each keyword contained in the webpage. Please refer to FIG. 8, which shows a schematic diagram of a matrix MX _140-1 according to another embodiment of the present invention. The processor 110 determines the identifier of each keyword KW ₁₄₀₁ , KW ₁₄₀₂ , . . . , KW ₁₄₀₆ included in the web page 140-1 and each keyword KW ₁₄₀₁ , KW ₁₄₀₂ , . The TF-IDF values of KW ₁₄₀₆ yield a matrix MX _140-1 .

Step S240, training a webpage classification model according to the matrix and the classification name. The processor 110 trains the webpage classification model 130 according to the matrix MX _140-1 and the classification name CN _140-1 . By analogy, steps S210 to S240 are repeated until each matrix MX corresponding to each web page 140-1, . . . 140-8, 140-1, . _140-1 , ..., MX _140-8 , MX _160-1 , ..., MX _160-3 and classification names CN _140-1 , ..., CN _140-8 , CN _160-1 , ... ., CN _160-3 , to train the webpage classification model 130.

In this way, the automatic classification method of webpages proposed in this case can train a webpage classification model to accurately generate the classification names of webpages.

Please refer to Figures 1, 3, 4 and 9. FIG. 9 is a flow chart of a method for automatically classifying webpages according to another embodiment of the present invention. The following takes the webpage 120-1 of the website 120 as the browsed webpage, and the webpage 120-1 is not stored as an example.

In step S310, it is determined whether a browsed webpage has been stored. If yes, end the process; if not, execute step S320. The processor 110 determines that the webpage 120-1 is a browsed webpage and the webpage 120-1 is not stored, and then executes step S320.

Step S320 , when the browsed webpages are not stored, use the application program interface to retrieve a plurality of keywords contained in the browsed webpages, and assign an identifier to each keyword of the browsed webpages. The processor 110 retrieves a plurality of keywords KW ₁₂₀₁ , _{KW 1202} , . Keywords KW ₁₂₀₁ , KW ₁₂₀₂ , ..., KW ₁₂₀₅ - an identifier.

Step S330: Calculate the TF-IDF value of each keyword of the browsed webpage based on the number of all webpages in the website to which the browsed webpage belongs. The calculation of the TF-IDF value requires the definition of a matrix. In this embodiment, the parent is all the web pages 120-1, 120-2, . . . , 120-10 in the website 120 to which the web page 120-1 that has been viewed belongs. The processor 110 calculates each keyword KW ₁₂₀₁ , KW ₁₂₀₂ included in the browsed webpage 120-1 based on the number (10) of all webpages 120-1, 120-2, . . . , 120-10 in the website 120 , ..., TF-IDF value of KW ₁₂₀₅ .

Step S340, generating a matrix according to the identifier of each keyword of the browsed webpage and the TF-IDF value of each keyword of the browsed webpage. The processor 110 is based on the identifiers of each keyword KW ₁₂₀₁ , _{KW 1202} , _. The TF-IDF values of , KW ₁₂₀₂ , ..., KW ₁₂₀₅ generate a matrix MX.

Step S350, input the matrix into the webpage classification model to generate predicted classification names. The processor 110 inputs the matrix MX to the webpage classification model 130 to generate a predicted classification name PCN.

In step S360, the browsed web pages are stored in a database with the predicted category names. The processor 110 stores the browsed webpage 120-1 in a database (not shown) with the predicted category name PCN. The database is used to store the saved web pages and their category names.

In step S370, a preference information is identified according to the number of web pages of each category name in the database, and advertisements related to the preference information are recommended. The processor 110 selects the category name with the largest number of web pages as the preference information, and recommends advertisements related to the preference information. For example, in the database, the number of pages with the category name "Sports News" is the largest, and "Sports News" is used as the preferred information to recommend advertisements related to "Sports News" (such as the news about the opening game of Chinese Professional Baseball) ). In a In an embodiment, the database can distinguish stored web pages and their category names according to different users.

In this way, the automatic classification method of web pages proposed in this case can identify different preference information according to different users.

Please refer to Figures 10, 11 and 12. FIG. 10 is a block diagram of an automatic classification system 100 for web pages and a website 180 according to another embodiment of the present invention. FIG. 11 is a flowchart illustrating a method for automatically classifying webpages according to another embodiment of the present invention. FIG. 12 shows a schematic diagram of articles 180-11 according to an embodiment of the present invention. In this embodiment, the automatic classification system 100 of the webpage can determine whether an article with an article classification name is published in the webpages 180-1 and 180-2 of the website 180. The following is an example in which an article 180-11 with an article category name CN _180-11 is published in the web page 180-1 of the website 180. The web page 180-2 contains a plurality of articles 180-21, 180-22, 180-23.

Step S410, judging whether an article with an article category name is published. If yes, go to step S420; if no, end the process. The processor 110 determines that an article 180-11 with an article classification name CN _180-11 is published, and then executes step S420.

In step S420, when the article with the article category name is published, the application program interface is used to extract a plurality of keywords included in the article, and an identifier is given to each keyword included in the article. When the article 180-11 with the article classification name CN _180-11 is published, the processor 110 uses the API to retrieve the plurality of keywords KW ₁₈₀₁ , KW ₁₈₀₂ , . . . , KW ₁₈₀₆ contained in the article 180-11, An identifier is given to each keyword KW ₁₈₀₁ , KW ₁₈₀₂ , . . . , KW ₁₈₀₆ contained in the articles 180-11.

Step S430, based on the number of all articles in the website to which the article belongs, calculate the TF-IDF value of each keyword included in the article. The calculation of the TF-IDF value requires the definition of a matrix. In this example, the parent is all articles 180-11, 180-21, 180-22, 180-23 in website 180. The processor 110 calculates each keyword KW ₁₈₀₁ , KW ₁₈₀₂ ,  … , TF-IDF value of KW ₁₈₀₆ .

Step S440, generating a matrix according to the identifier ID of each keyword included in the article and the TF-IDF value of each keyword included in the article. The processor 110 is based on _{the identifier of each keyword KW 1801 ,} KW ₁₈₀₂ , _. The TF-IDF values of KW ₁₈₀₆ yield the matrix MX _180-11 .

Step S450, input the matrix into the webpage classification model to generate predicted classification names. The processor 110 inputs the matrix MX _180-11 to the webpage classification model 130 to generate a predicted classification name PCN _180-11 .

Step S460, when the article category name is different from the predicted category name, publish the article with the predicted category name. The processor 110 determines whether the article category name CN _180-11 is the same as the predicted category name PCN _180-11 , and when the article category name CN _180-11 is different from the predicted category name PCN _180-11 , the predicted category name PCN _{180 -11} Post article 180-11.

In this way, the automatic classification method of web pages proposed in this case can generate a matrix for the identifier corresponding to each keyword contained in the published article, that is, the TF-IDF value, and input it into the trained web page classification model to accurately Generates the category name of the published article.

Please refer to Figures 1 and 13. FIG. 13 is a flowchart illustrating a method for automatically classifying webpages according to another embodiment of the present invention. Steps S510 to S550 are respectively similar to steps S110 to S150 in FIG. 2 , and are not repeated here. After the processor 110 stores the webpage 120-1 with the predicted category name PCN, step S560 is executed.

Step S560, it is determined whether the predicted category name of the stored webpage has been changed. If yes, go to step S570; if no, end the process. The processor 110 determines that the predicted category name PCN of the stored webpage 120-1 has been changed, and then executes step S570.

In step S570, when the predicted category name of the stored webpage is changed, the webpage classification model is trained according to the matrix and the changed category name. When the predicted category name PCN of the stored webpage 120-1 is changed, indicating that the user is not satisfied with the predicted category name of the webpage classification model 130, the processor 110 trains the webpage classification model 130 according to the matrix MX and the changed category name. .

In this way, the automatic classification method of webpages proposed in this case can determine whether the predicted classification names have been changed, so as to optimize the webpage classification model.

To sum up, although the present invention has been disclosed by the above embodiments, it is not intended to limit the present invention. Those with ordinary knowledge in the technical field to which the present invention pertains, Various changes and modifications may be made without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the scope of the appended patent application.

S110, S120, S130, S140, S150: Steps

Claims (10)

An automatic classification method for web pages, comprising: using an application programming interface to retrieve a plurality of keywords contained in a web page in a website, and giving each of the keywords contained in the web page an identifier (ID); based on the The number of all the webpages in the website, calculate the TF-IDF value of each of the keywords contained in the webpage; according to the identifier of each of the keywords contained in the webpage and the generating a matrix of TF-IDF values; inputting the matrix into a webpage classification model to generate a predicted classification name; and storing the webpage with the predicted classification name; wherein a training method of the webpage classification model includes: using the application programming interface Retrieve the keywords and a webpage category name contained in the webpage of the website, and give each of the keywords contained in the webpage an identifier; based on the number of all webpages in the plurality of websites, calculate the the TF-IDF value of each of the keywords; generating the matrix according to the identifier of each of the keywords contained in the webpage and the TF-IDF value of each of the keywords contained in the webpage; according to the matrix and The webpage classification name trains the webpage classification model. The automatic classification method according to claim 1, wherein before using the application programming interface to retrieve the keywords included in the webpage of the website, the method further comprises: judging whether the webpage has been previously stored; When the webpage has not been previously stored, the automatic classification method is executed. The automatic classification method according to claim 1, further comprising: judging whether a browsed webpage has been saved; when the browsed webpage has not been saved, using the application program interface to retrieve the browsed webpage The webpage contains a plurality of keywords, and each of the keywords of the browsed webpage is given an identifier; based on the number of all webpages in the website to which the browsed webpage belongs, calculate the browsed webpage. TF-IDF value of each of the keywords of the webpage; according to the identifier of each of the keywords of the browsed webpage and the TF-IDF value of each of the keywords of the browsed webpage generating the matrix; inputting the matrix into the webpage classification model to generate the predicted category name; storing the visited webpage in a database with the predicted category name; according to the number of webpages of each category name in the database Identify a preference information and recommend advertisements related to the preference information. The automatic classification method according to claim 1, further comprising: judging whether an article with an article category name is published; when the article with the article category name is published, using the application program interface to retrieve the article A plurality of keywords included, and an identifier is given to each of the keywords included in the article; based on the number of all articles in the website to which the article belongs, calculate the TF-IDF of each of the keywords included in the article value; Generate the matrix according to the identifier ID of each of the keywords contained in the article and the TF-IDF value of each of the keywords contained in the article; input the matrix into the webpage classification model to generate the predicted category name ; When the article category name is different from the predicted category name, publish the article with the predicted category name. The automatic classification method according to claim 1, further comprising: judging whether the predicted classification name of the stored webpage has been changed; and when the stored predicted classification name of the webpage has been changed, according to the matrix and the changed category name to train the webpage classification model. An automatic classification system for web pages, comprising: a processor for retrieving a plurality of keywords contained in a web page in a website using an application programming interface, and giving each of the keywords contained in the web page an identifier (ID), based on the number of all webpages in the website, calculate the TF-IDF value of each of the keywords contained in the webpage, according to the identifier of each of the keywords contained in the webpage and each of the keywords contained in the webpage. A matrix is generated from the TF-IDF values of the keywords; and a webpage classification model is used to generate a predicted category name according to the matrix; the processor stores the webpage with the predicted category name; and wherein the processor further uses by: extracting the keywords and a page category name contained in the webpage of the website using the application programming interface, and giving each of the keywords contained in the webpage an identifier, based on the number of all webpages in the plurality of websites, Calculate the TF-IDF value of each of the keywords contained in the webpage, and generate the matrix according to the identifier of each of the keywords contained in the webpage and the TF-IDF value of each of the keywords contained in the webpage , and train the webpage classification model according to the matrix and the webpage classification name. The automatic classification system of claim 6, wherein the processor is further configured to determine whether the webpage has been previously stored. The automatic classification system of claim 6, wherein the processor is further configured to determine whether a browsed webpage has been stored, and when the browsed webpage has not been stored, the application program interface is used to retrieve the A plurality of keywords contained in the browsed webpage, and an identifier is given to each of the keywords in the browsed webpage. Based on the number of all webpages in the website to which the browsed webpage belongs, calculate the The TF-IDF value of each of the keywords of the browsed webpage, according to the identifier of each of the keywords of the browsed webpage and the TF of each of the keywords of the browsed webpage generating the matrix of IDF values, inputting the matrix to the webpage classification model to generate the predicted classification name, storing the viewed webpage in a database with the predicted classification name, and Identify a preference information according to the number of web pages of each category name in the database, and recommend advertisements related to the preference information. The automatic classification system of claim 6, wherein the processor is further configured to determine whether an article with an article category name is published, and when the article with the article category name is published, use the application program interface to retrieve Take a plurality of keywords contained in the article, and give each of the keywords contained in the article an identifier, and based on the number of all articles in the website to which the article belongs, calculate the number of keywords contained in the article. TF-IDF value, the matrix is generated according to the identifier ID of each of the keywords contained in the article and the TF-IDF value of each of the keywords contained in the article, and the matrix is input into the webpage classification model to The predicted category name is generated, and when the article category name is different from the predicted category name, the article is published under the predicted category name. The automatic classification system of claim 6, wherein the processor is further configured to determine whether the stored predicted classification name of the webpage has been changed, and when the stored predicted classification name of the webpage has been changed, according to The matrix and the changed category names train the webpage classification model.

Images