JP2016105260A - Site consolidation method, site consolidation system, information processing device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To consolidate document information such as a news article and a comment distributed over a plurality of sites based on content.SOLUTION: Based on a connection sentence that connects the title with the summary sentence of sentence information including comments of the user of a terminal device, similar sentence information is clustered from site information distributed via a network. A representative comment is extracted from the comments. A summary page is prepared that carries at least one of the title and the summary sentence corresponding to the sentence information and the representative comment.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a site summarizing method, a site summarizing system, an information processing apparatus, and a program.

  In recent years, various technologies for collectively providing information distributed via a network have been developed (see, for example, Patent Documents 1 and 2).

  The invention described in Patent Document 1 is an invention related to an information display method in an information display device. Specifically, a user clip registration unit stores news headline data arbitrarily selected by a user from a news headline list screen. The user clip news headline is registered in the storage unit, and when an instruction to display the news headline clip screen is input from the input unit, the list screen display means reads the user clip news headline from the storage unit and reads the news headline clip It is displayed on the screen.

  The invention described in Patent Document 2 is an invention related to a summarizing system, a summarizing method thereof, and a recording medium on which a summarizing program is recorded. Specifically, a news article of a thread in which a plurality of news articles related to the same subject are collected. In a summary system that generates summary sentences of content, position importance determination is performed for determining the importance of position for each sentence of each news article in the thread, which is the importance of the sentence in relation to other sentences in the thread. A word importance level determination unit that determines the word importance level that is the importance level shown in the thread of each word used in each news article in the thread, and the average value and position importance of the word weight level for each sentence A sentence importance level determination unit that determines a sentence importance level that is an importance level of each sentence by adding a value of the degree according to a predetermined weight, and a sentence importance level is determined to be high. Remove the sentence, by appropriately positioned based on the settings defined the retrieved sentences advance, those with a summary creation unit that generates a thread summary of.

JP 2004-145724 A Japanese Patent Application Laid-Open No. 2002-039632

However, in the technologies described in Patent Documents 1 and 2 described above, information from only one site is handled, and no consideration is given to handling information collected from a plurality of sites. Since articles and comments for news of the same content are distributed over a plurality of sites, there is a problem as shown in FIG.
FIG. 10 is a state diagram showing the relationship between news and comments of a plurality of sites obtained via the network.
10, the figure of the holder represents a news article, and the balloon represents a comment on the news article / posting to SNS.

1. It takes time to check all the news articles.
2. With regard to the content of the news article, it is not possible to determine what the overall viewers think (positive, negative, emotional, etc.) with just one site.
3. Even if there are similar comments on multiple sites, it takes time to read them all (rising the risk of overlooking valuable comments).
In other words, because the same topic is distributed across multiple sites,
(A) It is not possible to determine whether a news article is highly recognizable with only one site.
(B) All of a plurality of sites must be viewed, which takes time.

  Therefore, an object of the present invention is to collect text information such as news articles and comments distributed on a plurality of sites on a content basis.

  In order to solve the above-mentioned problem, the invention described in claim 1 is characterized in that URL acquisition means for acquiring URL information described in the first sentence information posted so as to be viewable via a network, and the URL information Text acquisition means for acquiring second text information to be shown via a network, character string acquisition means for acquiring character string information of a title and a summary sentence from the second text information, and the second text information Based on a comment acquisition means for acquiring a comment and a connected sentence obtained by connecting the title and the summary sentence of the second sentence information, similar second sentence information among a plurality of the second sentence information is collected. First clustering means, second clustering means for extracting representative comments from the comments, and at least one title or summary sentence corresponding to the collected second sentence information And, characterized in that it is provided with a, and page creation means for creating a summary page to post and the representative comment.

  According to the present invention, text information such as news articles and comments distributed in a plurality of sites can be collected on a content basis.

It is an example of the block diagram which concerns on the summary site system of this invention. It is an example of a functional block diagram of the information processing apparatus 11 shown in FIG. It is an example of the flowchart for demonstrating operation | movement of the information processing apparatus 11 shown in FIG. It is the screen obtained by the summary site system shown in FIG. 1, and a comparative example. It is a conceptual diagram which shows the relationship before and behind the clustering process with respect to the news article and comment using the summary site system shown in FIG. It is explanatory drawing about the reputation analysis with respect to E disease using the summary site system shown in FIG. It is explanatory drawing about the totaling / display with respect to E disease using the summary site system shown in FIG. (A)-(c) is explanatory drawing at the time of calculating the distance between words using Numerical formula (2). It is explanatory drawing of the method of leaving one representative comment in a cluster. It is a state diagram which shows the relationship between the news of the site obtained via a network, and a comment.

<Configuration>
FIG. 1 is an example of a block diagram according to the summary site system of the present invention.
The summary site system 10 shown in FIG. 1 is a system composed of three information processing apparatuses 11, 12, and 13. The information processing apparatus 11 functions as a data acquisition server, the information processing apparatus 12 functions as a clustering server, and the information processing apparatus 13 functions as a display server.
The present invention is not limited to this, and the HDD (Hard Disk Drive) of one information processing apparatus is divided by partition so that it functions as a clustering server, data server, and display server for each partition. It may be configured.

  The information processing apparatus 11 includes a CPU (Central Processing Unit) 14, a ROM (Read Only Memory) 15, a RAM (Random Access Memory) 16, a display unit 17, an HDD 18, a LAN (Local Area Network) connection unit 19, an operation unit 20, A network connection unit 21 and a bus line 22 are included.

  The CPU 14 is a device that performs overall control of the information processing apparatus 11, and may be a microprocessor, for example. The ROM 15 is a device in which a control program is stored, and examples thereof include a mask ROM and an EPROM (Erasable Programmable Read Only Memory). The RAM 16 is a device that temporarily stores programs and data, and examples thereof include DRAM (Dynamic RAM) and flash memory. The display unit 17 is a device that displays information processed by the information processing device 11, an operation screen, and the like, and examples thereof include a liquid crystal display device. The HDD 18 stores OS (Operating System) and various application software, and stores text information such as news article information distributed via the network 24 and posted information flowing on SNS (Social Networking Site) Device. The LAN connection unit 19 is a device that the information processing device 11 connects to the information processing devices 12 and 13. The operation unit 20 is a device having a switch such as a power switch and a numeric keypad necessary for operating the information processing apparatus 11, and examples thereof include a keyboard and a mouse, but even a touch panel integrated with the display unit 17 Good. The network connection unit 21 connects the information processing device 11 to a terminal device 23a, 23b,... Or a news server that transmits news article information or an SNS server via a network 24 such as the Internet, and transmits and receives information. It is.

FIG. 2 is an example of a functional block diagram of the information processing apparatus 11 shown in FIG.
2 includes URL (Uniform Resource Locator) information acquisition means 31, HTML (Hyper Text Markup Language) page acquisition means 32, character string acquisition means 33, comment acquisition means 34, word vector creation means 35, Clustering means 36, database means 37, display means 38, and control means 39 are provided.

The URL information acquisition means 31 acquires the posting information flowing on the SNS using the API provided by the SNS server at a predetermined timing (for example, every 5 minutes), and the URL information included in the posting information Extract and acquire URLs, periodically acquire RSS (Really Simple Syndication) information sent by each website, and extract and acquire new URL information described in RSS. The URL information acquisition unit 31 is a unit that acquires URL information from each site, and is realized by the CPU 14, the ROM 15, the RAM 16, and the network connection unit 21, for example.
The HTML page acquisition means 32 uses the URL information (for example, a news server) acquired by the URL information acquisition means 31 based on the URL information included in the posted information or the new URL information described in the RSS. Access and get an HTML page (for example, a news article). The HTML page acquisition unit 32 is a unit that accesses a URL and acquires an HTML page, and is realized by the CPU 14, the ROM 15, the RAM 16, the HDD 18, and the network connection unit 21, for example.
The character string acquisition means 33 analyzes the HTML structure of the HTML page acquired by the HTML page acquisition means 32 and describes it using the <title> tag or <meta property = "og: title"> tag and their end tags. HTML page that obtains the character string such as the title between the tags and the summary sentence between the tags described using the <description> tag or the <meta property = "og: description"> tag and their end tags The title and summary sentence are stored in the database in association with the URL information. The character string obtaining unit 33 is a unit that obtains a title and a summary sentence from an HTML page, and is realized by the CPU 14, the ROM 15, the RAM 16, and the HDD 18, for example.
The comment acquisition unit 34 acquires post information including the URL information acquired by the URL information acquisition unit 31 as a comment, or an HTML page with a comment description column acquired by the HTML page acquisition unit 32 (for news articles, etc. On the Web page that allows users to enter comments, to obtain comments (determined by characters such as “comment” or a submit button tag) that exist between tags that contain the string “comment” Then, post information and comment information such as comments are associated with the URL information and stored in the database. The comment acquisition means 34 is means for acquiring comment information, and is realized by, for example, the CPU 14, the ROM 15, the RAM 16, and the HDD 18. Further, the comment acquisition means 34 acquires the number of pieces of post information for the same URL information such as news articles and the number of comments for HTML pages such as news articles, and stores them in the database in association with the URL information.
The word vector creation means 35 performs language analysis processing on the sentences obtained by concatenating the titles and summary sentences of each HTML page acquired by the character string acquisition means 33, deletes connected words, etc., and divides them into word units. The word is stored as a word vector in association with the URL information of the HTML page in the database. The word vector creating means 35 is a means for creating a word vector from the title and the summary sentence, and is realized by, for example, the CPU 14, the ROM 15, the RAM 16, and the HDD 18.

Here, in the database stored in the database means 37 configured in the HDD 18, the title of the HTML page such as a news article within a predetermined period, the summary text, the URL information of the HTML page (corresponding to the ID) The word vector extracted from the title and summary text, the comment of the viewer for the HTML page such as a news article, the number of comments for the news article, and the comment vector extracted from the comment text are stored. The database means 37 is realized by the HDD 18, for example.
Table 1 is an example of a database in which URL information of HTML pages, titles of HTML pages, summary sentences, and word vectors are stored in association with each other.
In addition, the title shown in the database of Table 1 is sandwiched between <title> tag <title> and tag </ title> in <title> A country suspected of being infected with E disease </ title> in the HTML page structure. It is the string “suspected E diseased in country A”, and the summary sentence is <description> the infected person suspected of E disease in country A ... </ description> tag <description> and tag < / description> is the character string “An infected person suspected of having E disease in country A ...”.

  Table 2 is an example of a database of URL information, comments, comment vectors, and the number of comments.

In the databases shown in Tables 1 and 2, data acquired at regular intervals (for example, every 5 minutes) is added and accumulated. When creating a summary page ranking of HTML pages such as news articles for 2 hours, URL information of the most recent 2 hours is extracted from the database in Table 2 (if the URL information is more than the default number (eg 500) In the extracted URL information, the total number of comments in the URL information unit is the default number (ex. 500 URL information is extracted), and the word corresponding to the extracted URL information in the database of Table 1 Extract vectors and organize HTML pages such as news articles (clustering). Also, refer to the database in Table 2, select representative comments from the comment vector corresponding to the extracted URL information, and refer to Table 2 for the number of comments corresponding to URL information corresponding to HTML pages such as news articles compiled These are used for ranking ranking between HTML pages such as summarized news articles and HTML pages such as other summarized news articles. Of the accumulated data, duplicate data and unnecessary data are deleted.
The clustering means 36 extracts URL information for a predetermined period (for example, the latest 2 hours, 1 day, etc.) from the database in Table 2 at a predetermined timing, and displays a word vector corresponding to the extracted URL information. This is a means for acquiring from one database and collecting together URL information of the same content on a content basis (clustering processing described later), and is realized by, for example, the CPU 14, the ROM 15, the RAM 16, and the HDD 18.

The display unit 38 is a unit that displays a summary page on which the text information compiled by the page creation unit 40 is posted, and is realized by the display unit 17, for example. However, when the information processing device 11 is a server device, it may not be provided.
The control unit 39 is a unit that performs overall control of each unit of the information processing apparatus 11 and delivers commands and data to each unit, and is realized by, for example, the CPU 14, the ROM 15, the RAM 16, the HDD 18, and the bus line 22.
Further, the clustering means 36 performs a clustering process on a plurality of comments associated with the URL information having the same contents and stored in the database means 37, deletes duplicate comments and similar comments, and displays representative comments. Leave the default number of top cases (example: top one).
The page creation means 40 and the clustering means 36 publish at least one title or summary sentence (for example, in chronological order or in descending order of comments) such as news articles with the same URL information compiled. Create a summary page that is a web page that has a text information display area and a comment display area that corresponds to the URL information at the bottom of the screen and displays the selected representative comments in the horizontal direction in the clustering means 36, It is made public via the network 24 to the terminal devices 23a and 23b. The page creation means 40 is a means for creating and publishing a summary page, and is realized by, for example, the CPU 14, the ROM 15, the RAM 16, the HDD 18, and the network connection unit 21. In addition, instead of or in addition to the page creation means 40, comment display area, a reputation analysis is performed on each comment corresponding to a news article of the same URL information that has been collected, and each classification (affirmation, denial) And a reputation display area for displaying a graph of the assigned category name and the frequency of the category may be created on the summary page. Thus, for example, one site can be classified as “scary” at another site, and another site can be classified as “surprise”. The number of “scary” and “surprise” can be graphed (it is a bar graph in the figure, but the type of graph is not limited).
A plurality of classifications (words such as adjectives or adverbs) such as “scary” and “surprise” based on reputation analysis may be displayed simultaneously for each comment.

  As shown in FIG. 1, using information processing apparatuses 11, 12, and 13 having the same hardware block configuration, for example, the information processing apparatus 11 is replaced with a URL information acquisition unit 31, an HTML page acquisition unit 32, and a character string acquisition unit 33. , Comment acquisition means 34, word vector creation means 35, data acquisition server having database means 37 for storing and creating a database, information processing apparatus 12, clustering server having clustering means 36, and information processing apparatus 13 as page creation means By functioning as a page display server having 40, a summary site system 10 may be configured using a plurality of devices, or a server device that functions each unit of the present invention only by the information processing device 11 may be used. . The database means 37 is configured to use a NAS (Network Attached Storage) and be connected to each information processing device 11, 12, 13 so as to be readable and writable as a device different from the information processing devices 11, 12, 13. Also good. Here, when the information processing device 11 is not a server device that provides a summary page to each terminal device 23 but is one of the terminal devices 23 operated by the user, the page creation means displays the created summary page. Instead of publishing to each terminal device 23, the created summary page is displayed on the display unit 17.

<Operation>
FIG. 3 is an example of a flowchart for explaining the operation of the information processing apparatus 11 shown in FIG.

First, the URL information acquisition means 31 acquires URL information from each site at a predetermined timing (step S1), and the HTML page acquisition means 32 is based on the URL information acquired by the URL information acquisition means 31. The URL is accessed, the HTML page is acquired (step S2), and the character string acquisition unit 33 acquires the title and summary sentence of the HTML page acquired by the HTML page acquisition unit 32 and stores them in the database shown in Table 1. (Step S3).
Here, the comment acquisition unit 34 acquires post information including the URL information acquired by the URL information acquisition unit 31 as a comment, and stores it in the database of Table 2. Further, the comment acquisition unit 34 determines whether or not there is a comment from the viewer in the HTML page acquired by the HTML page acquisition unit 32 (step S4). If there is a comment from the viewer (step S4 / Yes), an HTML page comment is acquired, stored in the database of Table 2, and the process proceeds to step S5 (step S5). If there is no comment from the user (step S4) / No), go to step S6.

The word vector creation means 35 converts the title and summary sentence acquired by the character string acquisition means 33 into a connected sentence, performs a language analysis process on the connected sentence, divides the connected sentence, and generates a word vector. Create and store in the database of Table 1 and proceed to step S7 (step S6).
The clustering means 36 extracts URL information for a predetermined period from the database shown in Table 2 at a predetermined timing, and uses the word vector created for the word vector creation means 35 corresponding to the extracted URL information. Then, clustering processing is performed to compile the URL information on a content basis (step S7), and the clustering means 36 performs clustering processing on a plurality of comments associated with the collected URL information to represent the default number of top cases. Comments are extracted (step S8).
The page creation means 40 publishes at least one title or summary sentence corresponding to the URL information compiled by the clustering means 36 with a link to the corresponding URL information, and puts it in the clustering means 36 at the bottom of the page. A summary page for rotating the representative comments extracted in step S9 is created (step S9). In step S9, instead of the representative comment at the bottom of the page, the page creation means 40 performs a reputation analysis on a plurality of comments associated with the URL information collected by the clustering means 36, and a plurality of comments A classification page indicating affirmation, denial, emotion, etc. may be assigned, and a summary page displaying a graph of the classification and reputation analysis result may be created. A summary page is created for each summary URL information (cluster). However, a plurality of summary URL information (cluster) may be posted on one summary page without creating a summary page for each summary URL information (cluster). In that case, a summary page is created that is posted in order for each group of second sentence information from the top in the total order of the number of comments corresponding to the collected URL information.

<Screen posting example>
FIG. 4 is a screen and a comparative example obtained by the summary site system shown in FIG.
In FIG. 4, the news ranking on the left is a comparative example, and the news ranking on the right is an example of a posting screen according to the present invention. The ranking standard of ranking is the number of comments for each news article in the news ranking diagram on the left side, and the total number of comments of news articles compiled by the news ranking on the right side.
HTML pages of news articles that contain many identical words (eg, Liberia, Ebola, etc.) are automatically compiled.
In the comparative example on the left, news articles of the baseball team H, news articles of the aquarium, news articles of Ebola fever, news articles of Idol A, and news articles of pizza shops are listed from the top.

  On the other hand, in the news ranking on the right side, Ebola-related news articles are aggregated so that the total number of comments is maximized and posted at the top, baseball team H news articles, idol A news articles, Aquarium news articles are listed in order. That is, the distributed information is collected for each content. Since the ranking process is based on the topic, the ranking changes.

FIG. 5 is a conceptual diagram showing the relationship before and after clustering processing for HTML pages and comments such as news articles using the summary site system shown in FIG.
Before the clustering process on the left side of FIG. 5, HTML pages and comments such as news articles of E illness, idol A, and baseball team H are scattered on the A site to the C site.
When such HTML pages and comments such as news articles are scattered, if clustering processing is performed, the contents of each HTML page, such as E disease, baseball team H, and idol, as shown on the right side of FIG. HTML pages such as news articles about A and comments will be collected.

FIG. 6 is an explanatory diagram of a reputation analysis for E disease using the summary site system shown in FIG.
The diagram on the left side of FIG. 6 is a diagram in which HTML pages such as news articles about E disease obtained by performing the clustering process are aggregated. Analyze each comment associated with an HTML page, such as four news articles. In the emotion dictionary for words that express emotions, “scary”, “scary” and “tremor” are classified as “scary”, and “surprised”, “shock” and “surprised” are classified as “surprised”. .

  Therefore, in the reputation analysis, when comments received from viewers of each terminal device are analyzed for HTML pages such as news articles about E illness, the comment on the HTML page of the A site is “I'm afraid of E illness”. Therefore, it is possible to assign a “scary” classification. In the comment on the HTML page of B site, “Illness is landing, surprised”, it is possible to assign “surprise” classification. Since the comment on the HTML page of the C site is “E terrible”, it is possible to assign a “scary” classification.

On the other hand, when the four news articles are linked in order of the latest time series or the number of comments, B site news articles, A site news articles, and C site news articles for HTML pages such as E disease news articles. A summary page is created that is posted in chronological order or number of comments. A summary page is created on the bottom of each site news article, with the amount of classification “Scared” and “Surprised” displayed in a graph.
In the lower right figure, the news articles whose contents match are deleted.

FIG. 7 is a conceptual diagram of clustering and display of comments for E disease using the summary site system shown in FIG.
Clustering is performed once again on the comments associated with the HTML pages of the four news articles about E disease summarized by the cluster link in the left figure. For each cluster of news articles, leave only the default number of top comments (1 in this example), and delete comments with similar duplicate contents. A method for leaving only the default number of upper comments will be described later.
Regarding the cluster of news articles for E disease, in the lower right figure, for E disease news articles, B site news articles, A site news articles, C site news articles, and D site news articles in chronological order or number of comments It will be posted on the summary page in order. A representative comment display area is set below the news article on the summary page, and a representative comment such as “E illness has landed, surprised” is rotated and displayed in the horizontal direction, for example, from left to right. A word representing emotions such as “surprise” may always be displayed in this rotation display column. Similarly, the summary pages of the news articles of other idols A and the news articles of the baseball team H compiled by clustering are created.

<Clustering method>
Next, the clustering method will be described.
(i) A linked sentence obtained by connecting titles and summary sentences extracted from HTML pages such as news articles is divided into words by language analysis.
For example, if the concatenated sentence of a news article x is “E disease in country A or suspected infected person in country A, E disease”, the vector x which is the word vector of news article x is “country A”, “ E disease, infected person, and suspicion. If the concatenated text of another news article y is "E illness, infected person in country A. Infected person appears in country A. The government is considering measures immediately." “E disease”, “country A”, “infected person”, “appear”, “government”, “urgent”, “countermeasure”, and “examination”.
(Ii) Measure the distance (all combinations) between HTML pages such as news articles.
The closer the vectors are, the more similar the HTML pages are.
Jaccard distance _J ^d is a kind of distance between vectors words found by Equation (1).

Using the example vector x and vector y above,
n ₁₁ = 3 (Country A, E disease, infected)
n ₁₀ = 1 (question)
n ₀₁ = 5 (appears, government, urgent, measures, study)
Table 3 shows the relationship between the vector x and the vector y.

In Table 3, 1 of the vector x or the vector y means “a word exists”, and 0 means “a word does not exist”. In addition, for example, n ₁₁ with the subscript “11” in Table 1 has three words “country A”, “E disease”, and “infected person” that exist in common in the vector x and the vector y. Therefore, the number of elements is “3”. Similarly, the number of elements is “1” because n ₁₀ exists in the vector x and the only word that does not exist in the vector y is “suspect”. Since n ₀₁ does not exist in the vector x and the words existing in the vector y “appear”, “government”, “urgent”, “countermeasure”, and “review”, the number of elements is “5”.
Therefore, the Jaccard distance _J ^d is
_J ^d = (1 + 5) / (3 + 1 + 5) = 2/3 = 0.667.

However, when Expression (1) is used, even if the vector y substantially includes the vector x, if there is a gap in the number of elements in the vectors x and y, the distance between words is increased and the accuracy is lowered. There is a problem.
Therefore, the present invention improves the accuracy by the mathematical formula (2) obtained by improving the mathematical formula (1).

  The difference between the formula (2) and the formula (1) is that, by weighting the formula (1) in consideration of the number of words (multiplying to reduce the weight according to the number of words), the vector y becomes the vector x. Even if there is a gap in the number of elements in the vectors x and y, the distance between words is not increased and the accuracy is improved. That is, the optimum distance can be obtained even if there is a difference in distance between the short sentence and the long sentence. Here, Wx is the number of words in vector x, and Wy is the number of words in vector y.

8A to 8C are explanatory diagrams when the distance between vectors corresponding to an HTML page such as a news article is calculated using Equation (2).
(Iii) The news articles (or clusters in which the news articles are gathered) that are the closest to each other are extracted, and if they are equal to or less than the threshold value dmin, they are regarded as news articles having the same content, and clusters are formed (summarizing those news articles). Only the distance to the newly formed cluster is recalculated to extract the minimum distance. Repeat as long as the threshold is below, and end if there are no news articles or clusters below the threshold.
In the example of FIG. 8, the distance from news article A to news article D is 0.2 for news article A to news article B, 0.8 for news article A to news article C, and news article A to news article D. Is 0.9, news article B to news article C is 0.6, news article B to news article D is 0.3, and news article C to news article D is 0.4. Assume that the threshold value dmin = 0.6. Therefore, since the news article A to the news article B is the smallest 0.2, the distance between the vectors is the shortest and the threshold value dmin = 0.6 or less, so the news article A and the news article B are similar. As a result, it is collected (FIG. 8A).
When news article A and news article B are collected as one new cluster (news article A + news article B: (A, B) in the figure) and the distance between news article C and news article D is determined, the news article The distance between the A + news article B (A, B) cluster and the news article C takes an average value, (0.8 + 0.6) /2=0.7, and the news article A + news article B (A, B) ) And the news article D is (0.9 + 0.3) /2=0.6. Therefore, since the distance between the news article C and the news article D is 0.4, the distance between the vectors is the shortest, and the threshold value dmin = 0.6 or less, the news article C and the news article D are grouped together. (FIG. 8B).
News article C and news article D become a new cluster (news article C + news article D). The distance between the news article A + news article B (A, B) cluster and the news article C + news article D (C, D) cluster is (0.8 + 0.6 + 0.9 + 0.3) /4=0.65. . Since there are no other news articles or clusters, the minimum distance = 0.65 exceeds the threshold value dmin = 0.6 at this point, and the calculation ends. From this cluster, it can be seen that news article A and news article B are news articles of the same content. News articles A and B and news articles C and D can be grouped together (FIG. 8C). Thereafter, the number of comments corresponding to each URL information of the news articles A and B and news articles C and D in the cluster is summed up for each cluster.

<How to leave a representative number of top comments (example: 1) in the cluster>
FIG. 9 is an explanatory diagram of a method for leaving one representative comment in a cluster.
(Α) When there are two comments in a cluster that is a set of comments for an HTML page such as a news article, the comment posting date and time, the number of characters in the comment, and the strength of the account of the sender who sent the comment (number of followers, friends The largest number of people (number of people) etc. is extracted as the default number of top cases (example: 1).

(Β) When there are three or more comments in a cluster, which is a set of comments for HTML pages such as news articles, etc. For all comments in the cluster, the sentence of the comment is linguistically analyzed, broken down into words, and a comment vector is generated Then, an average value of distances to other comment vectors is obtained, and a comment with a predetermined upper number is used as a representative comment in order from the smallest average value. In this example, it is described as the top one.
In the example shown in FIG. 9, when the distance from the vector of comment A to the vector of comment D is calculated using equation (2) as described above, comment A to comment B is 0.2, and comment A to comment C is 0.8, comment A to comment D is 0.9, comment B to comment C is 0.6, comment B to comment D is 0.3, and comment C to comment D is 0.4. Furthermore, the average value of the distance from the comment A to other comments is (0.8 + 0.9 + 0.2) /3=0.63, and the average value of the distance from the comment B to other comments is (0.2 + 0.6 + 0). .3) /3=0.37, average distance from comment C to other comments is (0.8 + 0.6 + 0.4) /3=0.6, average distance from comment D to other comments The value is (0.3 + 0.9 + 0.4) /3=0.53.
Accordingly, the average value 0.37 of the comment B is the minimum value and the top one, so the comment B is the representative comment. Here, in the case of the top two, in addition to the comment B, the comment D and the representative comment with the next lowest average value are also representative comments.
In the same manner as described above, one comment having the maximum comment posting date / time, the number of comments, the strength of the comment sender's account (number of followers, number of friends), and the like may be extracted.

<Program>
The information processing apparatus according to the present invention described above is realized by a program that causes a computer to execute processing. Examples of the computer include general-purpose computers such as a server device, a personal computer, and a workstation, but the present invention is not limited to this. Therefore, as an example, a case where the function of the present invention is realized by a program will be described below.

For example,
A computer readable program of an information processing device,
On the computer,
A URL acquisition step of acquiring URL information described in the first sentence information posted to be viewable via a network;
A sentence acquisition step of acquiring second sentence information indicated by the URL information via a network;
A character string acquisition step of acquiring character string information of the title and the summary sentence from the second sentence information;
A comment acquisition step of acquiring a comment for the second sentence information;
A first clustering step of grouping together similar second sentence information among a plurality of second sentence information, based on a linked sentence obtained by connecting the title and summary sentence of the second sentence information;
A second clustering step of extracting representative comments from the comments;
A page creation step for creating a summary page in which at least one title or summary sentence corresponding to the summarized second sentence information and a representative comment are posted;
A program for executing

  Such a program may be stored in a computer-readable storage medium.

<Storage medium>
Here, examples of the storage medium include computer-readable storage media such as CD-ROM, flexible disk (FD), and CD-R, semiconductor memories such as flash memory, RAM, ROM, and FeRAM, and HDD.

  CD-ROM is an abbreviation for Compact Disc Read Only Memory. Flexible disk means Flexible Disk (FD). CD-R is an abbreviation for CD Recordable. RAM is an abbreviation for Random-Access Memory, FeRAM is an abbreviation for Ferroelectric RAM, and means a ferroelectric memory.

<Effect>
According to the present invention, by collecting and aggregating information of web pages such as news articles from the information of sites distributed via the network, and generating pages classified on a content basis for each similar news article, You can see how much users are tweeting and how much they are commenting.
Further, according to the present invention, a news article including a comment is divided into a title and an outline, and the title and the summary sentence are divided, and the word vectors between the titles and the distance between the word vectors between the summary sentences are set as word vectors. A news article below a threshold value is regarded as a news article with the same content, and is classified and ranked on the basis of content, so that it can also be ranked in response. Even if the amount of processing is large, ranking processing can be performed with high accuracy. For example, news articles can be sorted and ranked in about a few seconds when the number of comments and news articles is about 100, and in about 10 seconds when the number of news articles is about 1,000. Moreover, it can be made easy to see by the rotation display.

  In the above-described embodiment, the news article has been described as an example. However, other than the news article, a blog, a column, a reservation site for accommodation facilities, a book or product, a service review site, an EC site, a Web store comparison site, etc. As long as it is a text page published on the network, it may be a page using a format other than an HTML page. The above-described embodiment shows an example of a preferred embodiment of the present invention, and the present invention is not limited thereto, and various modifications can be made without departing from the scope of the invention. is there.

11, 12, 13 Information processing device 14 CPU
15 ROM
16 RAM
17 Display unit 18 HDD
DESCRIPTION OF SYMBOLS 19 LAN connection part 20 Operation part 21 Network connection part 22 Bus line 23a, 23b, ... Terminal device 31 URL information acquisition means 32 HTML page acquisition means 33 Character string acquisition means 34 Comment acquisition means 35 Word vector creation means 36 Clustering means 37 Database Means 38 Display means 39 Control means

Claims (9)

URL acquisition means for acquiring URL information described in the first sentence information posted so as to be viewable via a network;
Text acquisition means for acquiring second text information indicated by the URL information via the network;
Character string acquisition means for acquiring character string information of the title and summary sentence from the second sentence information;
Comment acquisition means for acquiring a comment on the second sentence information;
First clustering means for grouping together similar second sentence information among a plurality of the second sentence information based on a connected sentence obtained by connecting the title and summary sentence of the second sentence information;
Second clustering means for extracting representative comments from the comments;
Page creation means for creating a summary page on which at least one title or summary sentence corresponding to the summarized second sentence information and the representative comment are posted;
An information processing apparatus comprising: The first clustering means divides the connected sentences, extracts words,
Based on the ratio of the number of words between two pieces of the second sentence information of the plurality of second sentence information and the ratio of the number of words included in one of the second sentence information, Calculate the similarity between the second sentence information,
2. The two second sentence information that are the most similar among the plurality of second sentence information and the similarity is equal to or less than a threshold value are grouped together. The information processing apparatus described in 1.   The second clustering means divides the comments, extracts words, and calculates a ratio of the number of words between two comments of the plurality of comments and a ratio of the number of words included in one of the comments. Based on the above, the similarity between the two comments is calculated, and the average value of the similarities between one comment and other comments is calculated among the plurality of comments, and the average value is the smallest The information processing apparatus according to claim 1, wherein a comment is the representative comment.   The information processing apparatus according to claim 1, wherein the page creation unit displays the representative comment in a lower area of the summary page.   The page creation means classifies the comment by reputation analysis, and posts the classified classification name and the frequency of each classification in a lower area of the summary page. The information processing apparatus according to one. The page creation means
2. A summary page is created in which a group of second sentence information is placed in order from the top in order of the total number of comments for the second sentence information compiled by the first clustering means. The information processing apparatus according to any one of 5. URL acquisition means for acquiring URL information described in the first sentence information posted so as to be viewable via a network;
Text acquisition means for acquiring second text information indicated by the URL information via a network;
Character string acquisition means for acquiring character string information of the title and summary sentence from the second sentence information;
Comment acquisition means for acquiring a comment on the second sentence information;
A data acquisition server with
First clustering means for grouping together similar second sentence information among a plurality of the second sentence information based on a connected sentence obtained by connecting the title and summary sentence of the second sentence information;
Second clustering means for extracting representative comments from the comments;
A clustering server with
Page creation means for creating a summary page on which at least one title or summary sentence corresponding to the summarized second sentence information and the representative comment are posted;
A display server with
A site summarization system characterized by comprising: A site summarization method performed by an information processing device,
A URL acquisition step of acquiring URL information described in the first sentence information posted to be viewable via a network;
A sentence acquisition step of acquiring second sentence information indicated by the URL information via the network;
A character string acquisition step of acquiring character string information of a title and a summary sentence from the second sentence information;
A comment acquisition step of acquiring a comment for the second sentence information;
A first clustering step of grouping together similar second sentence information among a plurality of the second sentence information based on a connected sentence obtained by connecting the title and summary sentence of the second sentence information;
A second clustering step of extracting representative comments from the comments;
A page creation step of creating a summary page on which at least one title or summary sentence corresponding to the summarized second sentence information and the representative comment are posted;
A site bundling method characterized by including a URL. The program for making a computer perform each step of Claim 8.

Images