JP2005250693A - Character information classification program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a classification technique that allows classifying user desired character information without registering words to be classified. <P>SOLUTION: Object character information prescribed as a processing object is extracted from character information disclosed on a network. Of the object character information, a character portion that satisfies predetermined requirements is considered a keyword. The keyword is extracted and subjected to keyword matching. Depending on whether or not there is the keyword, a keyword vector 104a is created from the character information. Based on the keyword vector 104a created, the similarity between keyword vectors within the character information is calculated. Based on the similarity, the keyword vectors are associated with one another. The result 107a of the association is outputted. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a character information classification technique.

  2. Description of the Related Art Conventionally, the information amount of electronic information including character information represented by so-called web pages or text information has increased. Of such electronic information including character information, a web page is stored in a server installed on a network including the Internet and is open to the user.

  The web page has various contents, and information requested by the user, such as the latest news information, is sequentially updated. It is important for the user to accurately acquire information in various contents that are sequentially updated in many web pages in the Internet.

In addition, the technique of notifying a user early about the information shown by the content of the web page updated sequentially as above-mentioned according to a user preference is considered. Moreover, the following is mentioned as a prior art regarding the classification | category of character information, for example.
JP 2001-265808 A JP 2003-248688 A JP 2002-41544 A

  In order to classify the contents of web pages, it was necessary to analyze sentence structures such as titles or text based on natural language processing. However, the data structure (design) of HTML (HyperText Markup Language) of web page contents is not uniform among individual web pages. For this reason, it has been difficult for the conventional character information classification technology to effectively acquire information for the contents of all web pages.

  Further, in the conventional web page content classification technology, it is necessary to provide a classification criterion in advance using keywords. For this reason, in the conventional web page content classification technology, it is necessary to create keywords in advance.

  Further, in the conventional web page content classification technology, keywords for classifying content are generated using a keyword extraction method called morphological analysis. The morphological analysis is to use a dictionary to decompose a natural sentence of text data into meaningful minimum units (words) and make them suitable for computer processing. For this reason, when performing keyword extraction using morphological analysis, it is necessary to register words in the dictionary, and it is difficult to analyze abbreviations and new words that are not registered in the dictionary, proper names, foreign words, etc. Met.

  The present invention has been made to solve such a problem, and an object thereof is to provide a technology for classifying content including character information effective in various information forms. Another object of the present invention is to provide a classification technique that can be performed on a content including character information desired by a user without registering a word to be classified.

In order to achieve the above object, the present invention has the following means.
That is, according to the present invention, target character information that is predetermined as a processing target is extracted from character information that is disclosed on the network, and a character portion that satisfies a predetermined condition in the target character information is regarded as a keyword. The keyword is extracted, keyword matching is performed on the keyword, a keyword vector is generated from the character information in accordance with the presence or absence of the keyword, and a keyword vector in the character information is generated based on the generated keyword vector Similarity is calculated with respect to each other, the keyword vectors are associated with each other based on the similarity, and a result of association between the keyword vectors is output.

In the present invention, target character information determined as a processing target is extracted from character information disclosed on the network, and classification processing is performed.
Therefore, according to the present invention, it is possible to provide a technology for classifying content including character information effective in various information forms.
In the present invention, the association (clustering) processing of the character information provided to the user is performed based on the keyword extracted from the character information.
Therefore, according to the present invention, it is possible to provide a classification technique that can be performed on a content including character information desired by a user without registering a word to be classified.
In the present invention, when extracting the keyword, a character portion regarded as a keyword is extracted based on the type of character information.
Further, the present invention is characterized in that when performing the keyword matching, keyword matching is performed using a regular expression.
Note that the present invention may be a program for realizing any of the above functions. In the present invention, such a program may be recorded on a computer-readable storage medium. Furthermore, the present invention may be an apparatus that realizes any one of the functions described above.

  According to the present invention, it is possible to provide a character information classification technique effective for various forms of information. Further, according to the present invention, it is possible to provide a classification technique capable of performing user-desired character information without registering a word to be classified.

Next, the best mode for carrying out the present invention will be described with reference to the drawings.
The character information classification device is configured using a personal computer (PC), a workstation (WS), a dedicated computer, or the like in order to perform the processing according to the present invention. This character information classification device includes, as hardware (not shown), a processing device (including a CPU, a main storage device (RAM, etc.), an input / output unit, a device driver, etc.), an input device (keyboard, mouse, etc.) It has a device (display device, printer, etc.) and a storage device (memory, hard disk, etc.). This computer storage device stores RSS (RDF Site Summary) bookmarks, RSS databases, keyword maps, keyword tables, vector tables, similarity tables, cluster tables, and the like.

  The RSS bookmark is a list composed of RSS URLs including character information to be acquired. The keyword map stores a keyword string obtained by a keyword extraction process, which will be described later, in association with all or part of the target character information. This keyword map has a map structure in which a corresponding keyword string can be extracted using all or part of the target character information as a key.

The keyword table stores all the keywords obtained by the keyword extraction process. In this keyword table, duplicate keywords are not stored. The vector table is a storage area for storing all or part of the target character information and a vector generated from the whole or part of the target character information in association with each other. The similarity table stores two vectors and the similarity between the vectors in association with each other. Furthermore, the cluster table stores information on clusters composed of one or more vectors.

  FIG. 1 shows an example of a character information classification device 100 in which a character information classification program according to the present invention is installed (installed) in a secondary storage such as a hard disk device of a computer. The character information classification device 100 causes a computer to execute a character information classification program, thereby performing a target character extraction unit 101, a keyword extraction unit 102, a keyword storage unit 103, a keyword vector generation unit 104, a similarity calculation unit 105, and a score recording unit. 106, a clustering unit 107, and an output unit 108 are realized.

Next, the operation of the character information classification device 100 configured as described above will be described.
The target character extraction unit 101 extracts target character information that is previously determined as a processing target from character information included in RSS published on the network. The keyword extraction unit 102 regards a character portion satisfying a predetermined condition in the target character information as a keyword, and extracts the keyword. The keyword storage unit 103 stores the keywords extracted by the keyword extraction unit 102 in a database. The keyword vector generation unit 104 performs keyword matching on the keyword, and generates a keyword vector 104a from the character information extracted by the keyword extraction unit 102 according to the presence or absence of the keyword.

  Based on the keyword vector 104a generated by the keyword vector generation unit 104, the similarity calculation unit 105 records the keywords in the RSS in the similarity table 105a, and is similar based on whether the keywords are similar to each other. The degree is calculated and scored. The similarity table 106 is a detailed view of the similarity table 105a. The clustering unit 107 performs keyword vector clustering (association) based on the high similarity calculated by the similarity calculation unit 105, and stores the clustering result in the database 107a. The output unit 108 refers to the database 107a and outputs it via an output device such as a display (not shown).

Next, character information classification processing by the character information classification device 100 will be described with reference to a flowchart.
FIG. 2 is a flowchart showing processing of the character information classification device 100 realized by causing a computer to execute the character information classification program of the present invention.
When the character information classification device 100 starts this processing, the target character extraction unit 101 acquires RSS (RDF Site Summary) from the network and stores this RSS in the storage device 109 such as a hard disk device (S101). At this time, the target character extracting unit 101 uses the WWW (World Wide Web) based on the RSS URL (Uniform Resource Locator) designated by the user of the device 100 from the input device 110 or the RSS URL stored in the RSS bookmark. ) Get RSS published on the server. Then, the target character extraction unit 101 transmits an HTTP (HyperText Transfer Protocol) request (request) to the designated URL, and acquires RSS from the WWW server.

  Thereafter, the target character extraction unit 101 extracts a target character string included in the acquired RSS.

  The keyword extraction unit 102 extracts keywords from the target character string based on a predetermined condition (S102). In the present embodiment, keyword extraction conditions are determined as follows. In other words, in the present embodiment, a portion in which any one of the alphabetic characters displayed in full-width or half-width and katakana displayed in full-width is continuous is extracted as a keyword. In the present embodiment, a portion where two or more kanji characters continue is extracted as a keyword.

  FIG. 3 shows an example of keyword extraction based on the above-described conditions by the keyword extraction unit 102. In FIG. 3, it is assumed that a character string “when using morphological analysis, the HTML BR tag is ...” is input to the keyword extraction unit 102. At this time, the keyword extraction unit 102 extracts character strings of “morpheme analysis”, “case”, and “HTML” as keywords based on the above-described conditions. Then, the keyword extraction unit 102 performs keyword extraction for all or part of the target character information included in the RSS.

The keyword storage unit 103 stores all the keywords extracted by the keyword extraction unit 102 in the keyword table generated in the storage device (S102).
The keyword vector generation unit 104 generates a keyword vector for all or a part of all the target character information of RSS stored in the keyword table (S103). In the present embodiment, the keyword vector is a vector format representing whether or not the entire target character information includes a keyword in the keyword table.

For example, a keyword extracted from all or a part of all target character information is represented by “K1 K2
If the character string to be classified is “S1 S2 S3”, whether or not each character string includes these keywords is represented by a numerical value of 0 or 1. And the determination method of whether a keyword is included is determined by the following keyword matching methods.

  FIG. 4 shows an example of a keyword vector generation method according to the present embodiment. In FIG. 4, the keyword K1 is “morphological analysis”. The keyword K2 is “case”. The keyword K3 is “HTML”. Further, the keyword K4 is “analysis”. The character strings are S1 “when morphological analysis is used, HTML tags become unknown words”, S2 “when morphological analysis and keyword analysis are used together”, and S3 “representing a line break in HTML tags is BR. ”.

  The character strings S1, S2, and S3 shown under the above conditions are expressed as the following keyword vectors V1, V2, and V3 by the keyword matching method of the present embodiment. That is, the keyword vector V1 is (1, 1, 1, 0) because the character string S1 includes the keywords K1, K2, and K3. The keyword vector V2 is (1, 1, 0, 0) because the character string S2 includes the keywords K1 and K2. The keyword vector V3 is (0, 0, 1, 0) because the character string S3 includes the keyword K3.

  Keyword matching is performed using regular expressions as follows. For example, a regular expression “X [^ Y] * Y [^ Z] * Z” is generated from a three-character keyword “XYZ” in all or part of the keyword extracted from the target character information. The symbol “*” indicates that “the preceding character is repeated zero or more characters”. The symbol “^ Y” indicates “not Y”. In other words, the regular expression for the keyword "public commission" is "public [^]] *" [^ committee] * ". An example of a character string that matches the regular expression is “trade commission”, “fair trade commission”, “fair trade commission”. After the regular expression is generated, the keyword with the whole or part of the keyword extracted from the target character information and the generated regular expression "X [^ Y] * Y [^ Z] * Z" Perform matching.

  After generating keyword vectors for all or part of all target character information, the similarity calculation unit 105 calculates the similarity between two vectors (similarity between two keyword vectors) based on this keyword vector. Then, the result of the calculated similarity is recorded in the similarity table 105a (S104). The similarity is calculated by combining the generated keyword vectors two by two and calculating the similarity for each other.

  Then, the similarity is calculated by the following equations (1) and (2) based on the number of keywords of both two vectors (vector 1 and vector 2) to be compared with the Hamming distance between the vectors.

(1): Hamming distance = number of keywords in vector 1 but not in vector 2 + number of keywords in vector 2 but not in vector 1 (2): similarity = 1−hamming distance / (keywords included in vector 1) Number + number of keywords in vector 2)

  When the Hamming distance and the similarity are calculated based on the equations (1) and (2), the clustering unit 107 collects all or part of the target character information similar based on the calculated similarity. Create Then, the clustering unit 107 records the cluster in the cluster table (S105). At this time, the clustering unit 107 sets vectors Vi and Vj that satisfy the following expression (3) as clustering targets. The clustering unit 107 forms a primary cluster composed of two vectors Vi and Vj that satisfy Expression (3).

  (3): S (Vi, Vj)> T (i ≠ j, S is a similarity, T is a threshold value)

  The clustering unit 107 compares the primary clusters and combines the clusters including one or more common vectors in the compared clusters to form a secondary cluster (S106). At this time, the clustering unit 107 deletes the combined primary cluster.

  The clustering unit 107 performs the above-described clustering process on all vectors. Further, the clustering unit 107 sets the primary cluster that remains without being combined by this processing as a secondary cluster. Then, the clustering unit 107 sets the generated secondary cluster as the final result of the clustering process.

  The output unit 108 outputs the final result of the clustering process via a display or the like (S107). In the present embodiment, the output of the final result of the clustering process from the output unit 108 is, for example, the following screen output provided in so-called agent application software (hereinafter referred to as agent software) that operates resident on the user's computer. Notify users by method.

  In the present embodiment, there are the following three times for the clustering process. That is, as the processing time, for example, (1) processing is performed every predetermined time, (2) processing is performed every designated date and time or day of the week, and (3) processing is performed according to the user's operation. A street is conceivable. In the present embodiment, the clustering process may be performed according to any of the above three times.

  In addition, the first screen output method of the agent software according to the present embodiment outputs the result of the clustering process updated every fixed time to a small pop-up window displayed on the computer screen. By clicking on the pop-up window with a pointing device such as a mouse, detailed information on the clustering processing result is output on the screen.

  When the first clustering processing result output method is performed, the output unit 108 of the character information classification device 100 sorts (sorts) the clusters in descending order of the number of articles included in each cluster. Then, the output unit 108 outputs to the pop-up window in order from the largest number of articles included.

  FIG. 5 is a diagram showing an example of a method in which the result of the clustering process generated by the character information classification device 100 according to this embodiment is output on the screen by agent software. The clustering process result is displayed on the output screen 10 of the computer by a pop-up window 11 (1).

  When the user clicks the article information displayed as text in the pop-up window 11 (detailed pop-up window 12 (2)) displayed on the output screen 10 with a pointing device, the article list display 14 (details) together with the agent character 13 is displayed. The displayed article list display 15) is displayed. The article list display 14 (detailed article list display 15) displays the subject of the RSS subjected to the clustering process (all or part of the target character information) based on the result of the clustering process (3). ). Further, when the user clicks the text of the article requested from the article list display 14, the character information classification device 100 displays the clicked article on the screen.

  The second screen output method according to the present embodiment outputs the result of the clustering process after completion of the clustering process (1) or according to the user's operation without using an output method such as a pop-up window. To do.

  FIG. 6 is an example of an output screen as a result of RSS clustering processing by the second screen output method. In FIG. 6, the clustering processing result screen 21 in which the clustering processing results are sorted based on the number of all or a part of the target character information belonging to the cluster, and the clustering processing results are designated by the user. An example of the clustering processing result screen 22 that searches for all or part of target character information including a keyword and displays a cluster to which the whole or part of the target character information belongs is shown. In this embodiment, when the clustering processing result is displayed by the first screen output method, the result is displayed after a predetermined time has elapsed, but when the clustering processing result is output by the second screen output method. As described above, regardless of the elapsed time after the processing, the result is displayed in response to the completion of the clustering processing or by the user's operation.

  FIG. 8 is an example of an RSS setting screen for clustering. In this setting screen 1, a text box 2 for inputting the URL of the RSS to be added, a button 3 for adding the URL inputted in the text box 2 to the URL list, a button 4 for deleting the URL from the URL list, and a currently registered URL. A list box 5 that displays a URL, a cancel button 6 that closes the setting screen 1 without reflecting the changed contents of the list box 5, and a button 7 that closes the setting screen 1 while reflecting the changed contents. That is, the setting screen 1 is a screen for editing (adding / deleting) a plurality of URLs recorded in the RSS bookmark. Then, the character information classification device 100 performs a clustering process on the RSS of the URL set by the user from the setting screen 1.

  FIG. 9 is an example of a screen for the user to instruct the execution of the clustering process in the present embodiment. In this case (1), a pop-up menu is displayed by right-clicking the agent character 13 displayed on the screen with a pointing device (not shown). Then, the user inputs an instruction for clustering processing and the like from this pop-up menu.

FIG. 9 shows another example of a screen in which the user instructs execution of clustering processing in the present embodiment. In this case (2), the user gives an instruction to the apparatus when the user clicks on the agent character 13 with a pointing device or from a menu window automatically displayed at a predetermined timing. At this time, menus related to various instructions necessary for the clustering process are displayed in this menu.

  FIG. 7 is an example of an output screen 31 as a result of the RSS clustering process by the second screen output method. When the cluster displayed on the ranking setting screen 21 or the keyword setting screen 22 is clicked with a pointing device, the character information classification device 100 displays an output screen 31. That is, the character information classification device 100 outputs a plurality of articles belonging to one cluster from the output screen 31. When the user finds and clicks on a desired article from the output screen 31, the browser application program for the existing web page is activated in the computer, and the web page of the clicked article is displayed on the computer output screen (display). The browser screen 32 is displayed.

1 shows an example of a character information classification device according to an embodiment of the present invention. It is a flowchart which shows the process of the character information classification device implement | achieved by making a computer run the character information classification program of this invention. An example of the keyword extraction based on the above-mentioned conditions by a keyword extraction part is shown. An example of a keyword vector generation method according to the present embodiment will be shown. It is a figure which shows an example of the method which output the screen of the result of the clustering process produced | generated by the character information classification device by agent software. It is an example of the output screen of the result of the RSS clustering process by the 2nd screen output method. It is an example of the output screen of the result of the RSS clustering process by the 2nd screen output method. It is an example of the setting screen of RSS of clustering. It is an example of the screen which a user instruct | indicates execution of clustering processing in this Embodiment.

Explanation of symbols

1 setting screen 2 text box 3 button 4 delete button 5 list box 6 cancel button 7 button 10 output screen 11 pop-up window 13 agent character 14 article list display 21 ranking setting screen 22 keyword setting screen 31 clustering output screen 32 web page browser screen 100 character information classification device 101 target character extraction unit 102 keyword extraction unit 103 keyword storage unit 104 keyword vector generation unit 105 similarity calculation unit 106 similarity table 107 clustering unit 108 output unit 109 storage device 110 input device

Claims (9)

A program executable by a computer,
Extracting target character information that is determined in advance from character information published on the network;
Extracting a keyword by considering a character part satisfying a predetermined condition in the target character information as a keyword;
Performing keyword matching on the keywords;
Generating a keyword vector from the character information according to the presence or absence of the keyword;
Calculating a degree of similarity for the keyword vectors in the character information based on the generated keyword vectors;
Associating the keyword vectors with each other based on the similarity;
And a step of outputting a result of associating the keyword vectors with each other. In the step of extracting the keyword,
The character information classification program according to claim 1, wherein a character portion regarded as a keyword is extracted based on a type of character information. In the keyword matching step,
The character information classification program according to claim 1, wherein keyword matching is performed using a regular expression. A target character extraction unit that extracts target character information that is predetermined as a processing target from character information that is disclosed on the network;
A keyword extraction unit that regards a character portion satisfying a predetermined condition in the target character information as a keyword and extracts the keyword;
A keyword vector generating unit that performs keyword matching on the keyword and generates a keyword vector from the character information according to the presence or absence of the keyword;
Based on the generated keyword vector, a similarity calculation unit that calculates a similarity between the keyword vectors in the character information;
A clustering unit for associating the keyword vectors based on the similarity,
An output unit that outputs a result of associating the keyword vectors with each other. The keyword extraction unit
The character information classification device according to claim 4, wherein a character portion regarded as a keyword is extracted based on the type of character information. The keyword vector generation unit
The character information classification device according to claim 4, wherein keyword matching is performed using a regular expression. Extracting target character information that is determined in advance from character information published on the network;
Extracting a keyword by considering a character part satisfying a predetermined condition in the target character information as a keyword;
Performing keyword matching on the keywords;
Generating a keyword vector from the character information according to the presence or absence of the keyword;
Calculating a degree of similarity for the keyword vectors in the character information based on the generated keyword vectors;
Associating the keyword vectors with each other based on the similarity;
A character information classification method in which a computer executes a step of outputting a result of associating the keyword vectors. In the step of extracting the keyword,
The character information classification method according to claim 7, wherein a character portion regarded as a keyword is extracted based on the type of character information. In the keyword matching step,
The character information classification method according to claim 7, wherein keyword matching is performed using a regular expression.

Images