JP2010009254A - Feature word selection device, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a feature word selection device for reducing workload for allocating a category to a Web page by selecting feature words based on a predetermined rule. <P>SOLUTION: The feature word selection device is provided with: a feature word storage means 10 for storing feature words featuring a Web page to be distributed on a communication network N in association with the Web page; and a feature word selection means 11 for counting the hits showing matching frequency of the Web pages stored in association with the feature words, for each feature word stored in the feature word storage means 10, by retrieval using a search engine on the basis of the feature words, and for, when the number of hits becomes equal to or more than or less than a predetermined threshold, deleting the feature words from the feature word storage means 10, thereby selecting the feature words stored in the feature word storage means 10. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a feature word selection device, method, and program for selecting feature words for classifying web pages distributed on a communication network according to predetermined categories.

In recent years, innumerable web pages exist on the Internet, and it is not easy to search for web pages that meet the purpose. Therefore, although a search engine is provided, even if such a search engine is used, the target web page cannot always be searched.
Therefore, YAHOO! Some service providers such as JAPAN (registered trademark) provide a classification service for classifying web pages into categories, as shown in Non-Patent Document 1 below, in addition to search engines. It makes it easier to find the target web page.
[Online], April 21, 2008, YAHOO! JAPAN category, [Search April 23, 2008], Internet <URL: http: // dir. yahoo. co. jp />

In such a classification service for classifying by category, web pages are classified manually when the scale of the category is large (tens of thousands of nodes). When the category size is small (hundreds of nodes), many automatic classification processes have been attempted, and the words included in the web page existing on the Internet are compared with the pre-stored feature words. Based on the comparison result, categories corresponding to previously stored feature words are assigned.
When the size of the category is increased, the user can specify a classification category closer to the intention. However, if the category size is increased, the number of documents belonging to the category increases, the number of feature words handled during classification processing also increases, and the work of assigning a category that matches a web page becomes complicated and requires time. In addition, it is expensive to introduce a classification device that can withstand such work.

  The present invention is proposed in view of such circumstances, and by selecting pre-stored feature words based on a predetermined rule, it is possible to reduce the work burden of assigning categories to web pages. An object of the present invention is to provide a feature word selection device, method and program.

In order to achieve the above object, the present inventor has found a mechanism that can select previously stored feature words based on a predetermined rule, and has come up with the present invention.
The feature word selection device according to the present invention reduces the burden required for category classification processing based on a feature word stored in advance by deleting the feature word from the feature word database according to a predetermined rule. It is.

(1) Feature word storage means for storing a feature word characterizing a web page distributed over a communication network in association with the web page;
For each of the feature words stored in the feature word storage unit, the web page stored in the feature word storage unit in association with the feature word is searched by a search engine using the feature word as a search keyword. The number of hits indicating the frequency of matching is counted, and the feature word is stored in the feature word storage unit by deleting the feature word from the feature word storage unit when the number of hits is equal to or greater than a predetermined threshold value. A feature word selection means for selecting the feature words that are being used;
A feature word selection device comprising:

  According to the invention of (1), when the number of hits hitting the web page stored in the feature word storage unit in association with the feature word by the search by the search engine becomes a predetermined threshold value or less, Since the feature word is deleted from the feature word storage means, the following effects are produced.

  That is, when the number of hits exceeds a predetermined threshold, the feature word is deleted from the feature word storage means, so that the feature word is a common term, that is, a term that cannot characterize a web page. This makes it possible to make category classification based on the feature word impossible.

  In addition, when the number of hits falls below a predetermined threshold, the feature word is deleted from the feature word storage means, so the feature word is a term that is not associated with general users, that is, a term that cannot characterize a web page Therefore, it is possible to make the category classification based on the feature word impossible.

  Therefore, since only feature words that truly characterize the web page remain in the feature word storage means, category classification based on feature words can be performed efficiently, and the introduction cost of the category classification device can be kept low. Can do.

  (2) The feature word selection unit counts the appearance frequency at which the feature word appears in the web page stored in association with the feature word in the feature word storage unit, and the appearance frequency is predetermined. The feature word stored in the feature word storage means is selected by deleting the feature word from the feature word storage means when the threshold value is below a threshold value. Feature word sorting device.

  According to the invention of (2), when the appearance frequency appearing in the web page stored in association with the feature word is equal to or less than a predetermined threshold, the feature word is deleted from the feature word storage means. A feature word whose appearance frequency is less than or equal to a threshold is determined to be a term that is not associated with the creator of the web user, that is, a term that cannot characterize a web page, and categorization based on the feature word is not allowed. Can be possible.

  Therefore, since only feature words that truly characterize the web page remain in the feature word storage means, category classification based on feature words can be performed efficiently, and the introduction cost of the category classification device can be kept low. Can do.

  (3) The feature word selection means counts the appearance frequency of the feature word appearing in the title or text or summary of the web page stored in association with the feature word in the feature word storage means, The feature word stored in the feature word storage unit is selected by deleting the feature word from the feature word storage unit when the appearance frequency becomes a predetermined threshold value or less. The feature word sorting device according to (1).

  According to the invention of (3), the feature word is deleted from the feature word storage means when the appearance frequency appearing in the title of the web page stored in association with the feature word is equal to or less than a predetermined threshold. In general, a feature word whose appearance frequency in a title is less than or equal to a threshold value depends on the impression of the web page creator more than other items in the web page, and thus the appearance frequency in such a title is less than or equal to the threshold value. By deleting the feature word, only the feature word that characterizes the web page can be left in the feature word storage means.

(4) The computer
The web page stored in association with the feature word by a search by a search engine using the feature word as a search keyword for each feature word that is stored in the computer in advance and is distributed over a communication network. A feature word selection step of selecting the feature words by deleting the feature words when the hit count is equal to or greater than or less than a predetermined threshold. A method characterized by performing at least.

  (5) A program that causes a computer to execute the method according to (4).

  According to the present invention, since only feature words that truly characterize the web page remain in the feature word storage means, category classification based on the feature words can be performed efficiently, and the introduction cost of the category classification device Can be kept low.

  Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings. This is merely an example, and the technical scope of the present invention is not limited to this.

[Overall configuration of feature word sorter and related elements]
FIG. 1 is a diagram showing an overall configuration of a feature word selection device and related elements according to the present embodiment.

  The feature word selection device 1 according to the present embodiment is connected to a communication network N such as the Internet and a classification device 2. The search device 3 is connected to the communication network N and the feature word selection device 1.

  The classification device 2 classifies web pages on the communication network into categories. Various classification methods can be considered. For example, the following classification methods are possible.

  That is, the classification device 2 performs morphological analysis on the character data included in the web page, divides sentences (character data composed of a plurality of words) into words, and classifies the divided words into parts of speech. Extract only specific parts of speech (for example, nouns). Then, it is determined whether or not there is a matching feature word in the feature words stored in the feature word database 12 (see FIG. 2) to be described later. If there is a matching feature word, it is related to the matching feature word. When the average mutual information amount is equal to or greater than a predetermined value due to the average mutual information amount with the determined category, the category is set as the category of this web page.

ＭＩ（ｗ，ｃ）：形態素となる単語ｗとカテゴリｃの間の平均相互情報量
Ｐを確率として、Ｐ（Ｘ，Ｙ）は、「Ｘ」と「Ｙ」とが共起する確率であり、Ｐ（Ｘ）、Ｐ（Ｙ）は、それぞれ個別に生起する確率である。 For example, the following formula is used to calculate the average mutual information amount.

MI (w, c): P (X, Y) is a probability that “X” and “Y” co-occur with the average mutual information P between the word w as a morpheme and the category c as a probability. , P (X), and P (Y) are probabilities that occur individually.

  The search device 3 accepts a search request including a search keyword (hereinafter referred to as a query) from a user terminal (not shown) connected on the communication network N, and a communication address of a web page that matches the search keyword, Returns search results including at least title and summary.

  The search device 3 includes a query receiving unit 30 that receives a query from a user terminal connected on the communication network N, a query transmission unit 31 that transmits a query to the search processing unit 32, and a query transmitted from the query transmission unit 31. A search processing unit 32 that searches for a web page that matches the search keyword, and a search result generation unit 33 that generates a search result including at least the communication address, title, and summary of the web page searched by the search processing unit 32; In addition to the user terminal, a search result transmission unit 34 that transmits a search result to the feature word sorting device 1 and a search log recording unit 35 that records a search history are provided.

[Functional structure of feature word sorter]
FIG. 2 is a diagram illustrating an outline of a functional configuration of the feature word selection device 1 according to the present embodiment. The feature word selection device 1 according to the present embodiment includes a feature word storage unit 10 and a feature word selection unit 11. The feature word selection unit 11 further includes a feature word preparation processing unit 13 and a feature word selection processing unit 14.

  The feature word storage means 10 stores in advance feature words that are words that characterize web pages. Details of the feature word storage means 10 will be described later. The feature word sorting unit 11 sorts feature words based on a predetermined rule to be described later by processing in cooperation with the classification device 2. The feature word database 12 (feature word DB) constitutes a part of the feature word storage means 10 and stores the feature words in advance.

[Structure of feature word database and related elements]
FIG. 3 is a diagram showing an outline of the configuration of the feature word database 12 and related elements.
The feature word database 12 uses the constructed category structure data, the feature word ID table 121 in which the feature word ID and the feature word are stored in association with each other, and the communication address (document ID and document (web page)) in advance. For example, a document ID table 122 in which URLs are stored in association with each other, an ID correspondence table 123 in which feature word IDs and document IDs are stored in association with each other, and a frequency table 124 described later are provided.

  In the feature word ID table 121, feature word IDs are assigned to feature words and stored. In the document ID table 122, a document ID is assigned to a communication address (URL, etc.) of a web page and stored. The ID correspondence table 123 corresponds to a feature word ID assigned to a feature word, a document ID assigned to a web page, and an appearance position type representing an appearance position at which the feature word appears on the web page. Is remembered.

  In the frequency table 124, for each feature word ID, the number of hits hitting the web page stored in the document ID table 122 by the search based on the search keyword is stored as “frequency”. In the figure, “frequency” is stored in association with the type “search” in the sense that the frequency of web page hits based on the search keyword is stored as frequency.

  In the frequency table 124, for each feature word ID, the feature words stored in the feature word ID table 121 corresponding to the feature word ID are displayed in the “title” on the web page stored in the document ID table 122. "And the appearance frequency in the" text "are stored. In the figure, “frequency” is stored in association with the types “title” and “text” in the sense of storing the appearance frequency of feature words in the title and text of the web page.

  The feature word sorting device 1 extracts feature words that are targets for counting the number of hits and the appearance frequency with reference to the feature word ID table 121.

[Hardware configuration of feature word sorter]
FIG. 4 is a diagram illustrating a hardware configuration of the feature word selection device according to the present embodiment.
The feature word selection device 1 includes a CPU (Central Processing Unit) 310 (a plurality of CPUs such as a CPU 320 may be added in a multiprocessor configuration), a bus line 200, a communication I / F (I / F). An interface) 330, a main memory 340, a BIOS (Basic Input Output System) 350, an I / O controller 360, a hard disk 370, an optical disk drive 380, and a semiconductor memory 390. The hard disk 370, the optical disk drive 380, and the semiconductor memory 390 are collectively referred to as a storage device 410.

  The control unit 300 is a part that comprehensively controls the feature word selection device 1, and cooperates with the hardware described above by appropriately reading and executing various programs stored in the hard disk 370 (described later). Various functions according to the invention are realized.

  The communication I / F 330 is a network adapter when the feature word selection device 1 transmits / receives information to / from a user terminal or the like via the Internet N (FIG. 1). The communication I / F 330 may include a modem, a cable modem, and an Ethernet (registered trademark) adapter.

  The BIOS 350 records a boot program executed by the CPU 310 when the feature word sorting device 1 is started, a program that the feature word sorting device 1 depends on hardware, and the like.

  A storage device 410 such as a hard disk 370, an optical disk drive 380, and a semiconductor memory 390 can be connected to the I / O controller 360.

  The hard disk 370 stores various programs for causing the hardware to function as the feature word selection device 1, programs for executing the functions of the present invention, and tables and records to be described later. Note that the feature word selection device 1 can also use a hard disk (not shown) separately provided as an external storage device.

  As the optical disk drive 380, for example, a DVD-ROM drive, a CD-ROM drive, a DVD-RAM drive, or a CD-RAM drive can be used. In this case, the optical disk 400 corresponding to each drive is used. A program or data can be read from the optical disk 400 by the optical disk drive 380 and provided to the main memory 340 or the hard disk 370 via the I / O controller 360.

  Note that the computer in the present invention refers to an information processing device including a storage device, a control unit, and the like, and the feature word selection device 1 includes an information processing device including the storage device 410, the control unit 300, and the like. This information processing apparatus is included in the concept of the computer of the present invention.

  In the feature word selection device 1, the control unit 300 mainly corresponds to the feature word selection unit 11, and the storage device 410 mainly corresponds to the feature word storage unit 10.

[Hardware configuration of search device 3]
The classification device 2 and the search device 3 have the same configuration as the feature word selection device 1 described above.

[Flowchart According to Embodiment of the Present Invention]
[Preparation process for feature word selection]
FIG. 5 shows a flowchart of a preparation process for feature word selection according to the embodiment of the present invention.
S1: The classification device 2 sets the existing category data as a set D.
S2, S3: If the set D is not empty, the classification device 2 takes out the document d belonging to the set D for each category (class) c.
S4, S5: The classification device 2 generates a feature f (a feature word vector f) of the document d, and updates the feature word database 12.

[Classifier training process]
FIG. 6 shows a flowchart of the training process according to the embodiment of the present invention.
S6 to S9: The classification device 2 sets the existing category data as a set D. If the set D is not empty, the document d belonging to each category (class) c is extracted from the set D. The classification device 2 generates a feature f of the document d.
S10: The feature word selection device 1 performs a feature word selection process described later.
S11: The classification device 2 writes training data using the selected feature words.
S12: If the set D is empty in step S7, the model M is generated by performing training using the training data written in step S11.

[Classifier prediction processing]
FIG. 7 shows a flowchart of the prediction process according to the embodiment of the present invention.
S13, S14: The classification device 2 sets the input document as d and generates a feature f of the document d.
S15: The feature word selection device 1 performs a feature word selection process described later.
S16, S17: The classification device 2 predicts the category c to which it belongs by predicting the selected feature f ′ with the model M.

[Feature word selection process]
FIG. 8 is a flowchart showing the feature word selection process according to this embodiment.
S18: An element of the feature f is set as a set E.
S19, S20: If the set E is not empty, a feature element is extracted from the set E. In the figure, the place l indicates a place in the document where the feature element is located, as will be described later. The word t indicates a word in the document that can be a feature word.
S21: Select (l, t) with reference to the feature word DB.
S22: It is determined whether it is a deletion target. If it is a deletion target, the process returns to step S19. Otherwise, the process proceeds to step S23.
S23: (l, t) is output, and the process returns to step S19.

[Details of feature word selection processing]
Details of the feature word selection process will be described below.

There are the following types of selection conditions for (place l, word t).
(Condition a) The web search frequency of the word t is within a certain range.
(Condition b) In the existing category data, the frequency that the word t appears in the place l is within a certain range.
(Condition c) In the existing category data, the frequency that the word t appears in an arbitrary place l is within a certain range.
(Condition d) The web search frequency of the word t and the appearance frequency at a specific place in the existing category data satisfy a specific combination condition within a certain range.
Here, in the location l, there are types of text, title, meta description, and meta keyword.

  Hereinafter, with reference to FIG. 9 to FIG. 13, a specific processing flow regarding the preparation process and the selection process will be shown for the conditions a, b, and c.

[Feature word selection preparation process (condition a)]
FIG. 9 shows a feature word selection preparation process under condition a by the feature word preparation processing unit 13.
S30: An element of the feature f is set as a set E.
S31, S32: If the set E is not empty, a feature element (location l, word t) is extracted from E. If the set E is empty, the process ends.
S33, S34: If the search frequency for the word t has already been registered in the DB, the process returns to step S31.
S35, S36: When the number of hits from the search device 3 can be acquired, the feature word storage unit 10 registers the search frequency n of the word t in the DB. Otherwise, the process returns to step S31.

[Feature word selection process (condition a)]
FIG. 10 shows a feature word selection process under the condition a by the feature word selection processing unit 14.
S40: The search frequency of the word t is acquired from the feature word DB.
S41: If the frequency n can be acquired, the process proceeds to S42, and if not, the process ends.
S42, S44: When the frequency n is equal to or smaller than the lower limit value of the search frequency, a deletion determination is output. If not, the process proceeds to S43.
S43, S44: When the frequency n exceeds the search upper limit value, a deletion determination is output. If not, the process ends.

[Feature word selection preparation process (condition b)]
FIG. 11 shows a feature word selection preparation process under condition b by the feature word preparation processing unit 13.
S50: An element of the feature f is set as a set E.
S51, S52: If the set E is not empty, a feature element (location l, word t) is extracted from E. If the set E is empty, the process ends.
S53: The frequency of the place l of the word t is acquired from the feature word DB.
S54: If the frequency n can be acquired, the process proceeds to step S56. If the frequency n cannot be acquired, the process returns to step S55.
S55: The feature word storage means 10 registers in the DB with the frequency 1 of the place l of the word t, and returns to the process of step S51 again.
S56: Update the frequency of the location l of the word t in the DB. Then, the process proceeds to step S51.

[Feature word selection process (condition b)]
FIG. 12 shows a feature word selection process under the condition b by the feature word selection processing unit 14.
S60: The frequency n of the place l of the word t is acquired from the feature word DB.
S61: If the frequency n can be acquired, the process proceeds to step S62, and if not, the process ends.
S62, S64: If the frequency n is equal to or greater than the lower limit value of the place frequency, the process proceeds to step S63, and if not, the process proceeds to step S64, a deletion determination is output, and then the process ends.
S63, S64: If the frequency n does not exceed the upper limit value of the place frequency, the process is terminated. If not, the process proceeds to step S64, a deletion determination is output, and then the process is terminated.

[Feature word selection preparation process (condition c)]
The feature selection preparation process in the condition c by the feature word preparation processing unit 13 is the same as the condition b in FIG. The feature word preparation processing unit 13 executes the same processing only when the preparation processing of the condition b is not performed.

[Feature word selection process (condition c)]
FIG. 13 shows a feature word selection process under the condition c by the feature word selection processing unit 14.
S70: The frequencies at all locations of the word t are acquired from the feature word DB and summed up.
S71: If the total frequency na can be acquired, the process proceeds to step S72, and if not, the process ends.
S72: If the frequency total na is equal to or larger than the lower limit value of the place frequency total, the process proceeds to step S73, and otherwise, the process proceeds to step S74.
S73: If the frequency total na exceeds the upper limit value of the place frequency total, the process proceeds to step S74, and if not, the process ends.
S74: A deletion determination is output, and the process ends.

  As mentioned above, although this invention was demonstrated according to embodiment, the effect described in embodiment of this invention only enumerated the most suitable effect resulting from this invention, and the effect by this invention is the present invention. It is not limited to what was described in the Example.

It is a figure which shows the whole structure of the feature word selection apparatus which concerns on this embodiment, and a related element. It is a figure which shows the outline | summary of the function structure of the feature word selection apparatus which concerns on this embodiment. It is a figure which shows the outline | summary of a structure of a feature word database and a related element. It is a figure which shows the hardware constitutions of the feature word selection apparatus which concerns on this embodiment. The flowchart of the preparation process for the feature word selection which concerns on embodiment of this invention is shown. The flowchart of the training process which concerns on embodiment of this invention is shown. The flowchart of the prediction process which concerns on embodiment of this invention is shown. The flowchart showing the feature word selection process which concerns on this embodiment is shown. The flowchart of the feature word selection preparation process (condition a) which concerns on embodiment of this invention is shown. The flowchart of the feature word selection process (condition a) which concerns on embodiment of this invention is shown. The flowchart of the feature word selection preparation process (condition b) which concerns on embodiment of this invention is shown. The flowchart of the feature word selection process (condition b) which concerns on embodiment of this invention is shown. The flowchart of the feature word selection process (condition c) which concerns on embodiment of this invention is shown.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Feature word selection device 3 Retrieval device 10 Feature word storage means 11 Feature word selection means 12 Feature word database

Claims (5)

Feature word storage means for storing a feature word characterizing a web page distributed over a communication network in association with the web page;
For each of the feature words stored in the feature word storage unit, the web page stored in the feature word storage unit in association with the feature word is searched by a search engine using the feature word as a search keyword. The number of hits indicating the frequency of matching is counted, and the feature word is stored in the feature word storage unit by deleting the feature word from the feature word storage unit when the number of hits is equal to or greater than a predetermined threshold value. A feature word selection means for selecting the feature words that are being used;
A feature word selection device comprising: The feature word selection means includes:
The feature word counts the appearance frequency that appears in the web page stored in association with the feature word in the feature word storage means, and when the appearance frequency becomes a predetermined threshold value or less, the feature word 2. The feature word selection apparatus according to claim 1, wherein the feature words stored in the feature word storage means are selected by deleting words from the feature word storage means. The feature word selection means includes:
The frequency of appearance of the feature word appearing in the title, body, or summary of the web page stored in association with the feature word in the feature word storage means is counted, and the appearance frequency is equal to or less than a predetermined threshold. 2. The feature word selection according to claim 1, wherein the feature words stored in the feature word storage unit are selected by deleting the feature words from the feature word storage unit. apparatus. Computer
The web page stored in association with the feature word by a search by a search engine using the feature word as a search keyword for each feature word that is stored in the computer in advance and is distributed over a communication network. A feature word selection step of selecting the feature words by deleting the feature words when the hit count is equal to or greater than or less than a predetermined threshold. A method characterized by performing at least.   A program causing a computer to execute the method according to claim 4.

Images