KR101370831B1 - System and method for extracting condensed issue sentence - Google Patents

Abstract

An abbreviated issue sentence extraction system and method are disclosed. The system includes a clustering module for grouping related document materials into the same cluster from document information DBs for a plurality of document materials in which at least information about a title and a URL of a document are recorded; An issue sentence extraction module for extracting a valid issue sentence from which duplicates are removed from title information of individual document data belonging to the same cluster, and then generating an issue sentence sequence using a sentence as a unit string for each valid issue sentence. ; And an issue abbreviation generating module for generating a substring by extracting a common unit string from a plurality of issue sentence sequences belonging to the same cluster, wherein the substring generated by the issue contract generating module belongs to the same cluster. Characterized in that it provides an abbreviation for the plurality of document data.

Description

System and method for abbreviated issue sentence extraction {SYSTEM AND METHOD FOR EXTRACTING CONDENSED ISSUE SENTENCE}

The present invention is a technology related to a search service using a communication network such as the Internet, and more specifically, provides information summarized by issue with respect to document data including various news articles, books, documents, etc. provided through the Internet. It relates to an abbreviated issue sentence extraction system and method.

As the use of the Internet and the like increase, the type and number of documents provided are continuously increasing. The search service provided through a communication network such as the Internet crawls and indexes newly generated document data such as various news articles, books, and documents, and provides the basic data for the user's search based on the indexed information.

On the other hand, among the exponentially increasing document data, there are a lot of materials which are only slightly changed about the same issue (Issue), such as news articles. If a user wants to read a lot of documents one by one, it takes considerable time and effort because of documents with similar contents.
Prior Art Documents Korean Laid-Open Patent Publication No. 2012-0011662, "Document Ranking System and Method"

The present invention is to solve the problem of the search service using a communication network, such as the conventional Internet described above, grouping document data having a variety of issues by issue, and extracts a shortened issue sentence representing document groups having the same issue The purpose of this article is to provide an abbreviated issue sentence extraction system and method that can quickly deliver information summarized by issue to various users.

In addition, another object of the present invention, the shortened issue that can be provided to the user by ranking the shortened issue sentences for the grouped document data by analyzing the freshness of the grouped document data or the interest of other users. It is to provide a sentence extraction system and method.

An abbreviated issue sentence extraction system according to the present invention comprises: a clustering module for grouping document data related to each other from a document information DB for a plurality of document data in which information on at least a title and a URL of a document are recorded; An issue sentence extraction module for extracting a valid issue sentence from which duplicates are removed from title information of individual document data belonging to the same cluster, and then generating an issue sentence sequence using a sentence as a unit string for each valid issue sentence. ; And an issue abbreviation generating module for generating a substring by extracting a common unit string from a plurality of issue sentence sequences belonging to the same cluster, wherein the substring generated by the issue contract generating module belongs to the same cluster. Characterized in that it provides an abbreviation for the plurality of document data.

Here, the issue abbreviation generating module may generate the substring by extracting the longest common unit string from the plurality of issue sentence sequences belonging to the same cluster according to the longest common subsequence algorithm. Alternatively, the issue abbreviation generation module may be further configured to include a predetermined range of probability values based on a probability that each of the unit strings constituting each of the issue sentence sequences belonging to the same cluster appears through the entire plurality of issue sentence sequences. The substring consisting of the unit string may be generated.

The document information DB may further include creation time information of each of the document materials, and in this case, a rank for calculating a newness index based on the creation time information of each of a plurality of document materials belonging to the same cluster. The determination module may further include. Thereby, each of the issue contracts for a plurality of clusters may be provided by ranking the plurality of issue contracts based on the recency index.

On the other hand, it may further include a search log DB including at least a user identifier, URL information for each document data contained in the document information DB and the user's visit time information for each URL, in this case belonging to the same cluster The apparatus may further include a ranking determination module configured to calculate an index of interest of the user based on the number of visits of the user for the plurality of documents. Thereby, each of the issue contract statements for a plurality of clusters may be provided by ranking the plurality of issue contract statements based on the interest index.

In addition, the reduced issue sentence extraction method according to the present invention, the clustering step of grouping the document data related to each other from the document information DB for a plurality of document data recorded information on at least the title and URL of the document in the same cluster ; The issue sentence extraction step of extracting a valid issue sentence from which duplicates are removed from title information of individual document materials belonging to the same cluster, and generating an issue sentence sequence in which a sentence of a sentence is a unit string for each valid issue sentence. ; And an issue abbreviation generating step of generating a substring by extracting a common unit string from a plurality of issue sentence sequences belonging to the same cluster.

Here, in the issue contract generation step, the substring may be generated by extracting the longest common unit string from the plurality of issue sentence sequences belonging to the same cluster according to the longest common subsequence algorithm. Alternatively, in the issue abbreviation generation step, each of the unit strings constituting each of the issue sentence sequences belonging to the same cluster may fall within a preset probability value range based on a probability appeared throughout the plurality of issue sentence sequences. The substring consisting of the unit string may be generated.

On the other hand, when the document information DB includes the creation time information of each of the document data, by calculating the most recent index based on the creation time information of each of a plurality of document data belonging to the same cluster, each of the plurality of clusters The method may further include a ranking determining step of ranking the plurality of issue contracts based on the issue contract statement.

In addition, when using a search log DB that includes at least a user identifier, URL information for each document material included in the document information DB, and user visit time information for each URL, a plurality of document materials belonging to the same cluster. A ranking step of calculating a user's interest index by counting the number of visits of the user for the ranking, and ranking the plurality of issue contracts based on the interest index for each of the issue contracts for a plurality of clusters It may include.

According to the present invention, by grouping document data having various issues by issue and extracting and presenting abbreviated issue sentences representing document groups having the same issue, various document data can be quickly delivered to the user in the form of information summarized by issue. Can be.

In addition, according to the present invention, it is possible to analyze the recency of the grouped document data or the interests of other users to provide the user with a ranking of the abbreviated issue sentences for the grouped document data.

As a result, users who want to search for news articles or document data using a network such as the Internet can look at the shortened issue sentences that are summarized into the issues of interest to the latest or other users. User convenience is improved.

1 is a schematic diagram illustrating a network connection of an abbreviated issue sentence extraction system according to an embodiment of the present invention.
2 is a system configuration diagram of a reduced issue sentence extraction system according to an embodiment of the present invention.
3 is a diagram illustrating an example of a clustering DB according to an embodiment of the present invention, and shows an example of a record in which information about a title, a URL, and a creation time of document data belonging to each cluster ID is recorded.
FIG. 4 is a diagram illustrating an example of an issue sentence sequence DB according to an embodiment of the present invention, wherein each valid issue sentence extracted from title information of document data belonging to each cluster ID is divided into a unit string to form an issue sentence sequence. An example of the recorded record is shown.

Hereinafter, exemplary embodiments of the reduced issue sentence extraction system and method according to the present invention will be described in detail with reference to the accompanying drawings.

First, Figure 1 is a schematic diagram showing the network configuration of the reduced issue sentence extraction system according to an embodiment of the present invention. Users may access the search server 100a equipped with the abbreviated issue sentence extraction system 100 through the wired / wireless communication networks 120a and 120b using the user terminals 110a and 110b. That is, users access the search server 100a through the user terminals 110a and 110b to obtain information on various news articles, books, or documents that are currently an issue. At this time, the search server 100a provides information about the abbreviated issue sentence extracted from the issue sentence extraction system 100a to the user terminals 110a and 110b, but is special from the user terminals 110a and 110b as necessary. It can also be optionally provided only upon request. In addition, the abbreviated issue sentence extraction system 100 may be integrated and operated in a search server 100a that provides an Internet search service, or may be constructed as a separate system that is physically separated from the search server 100a. It may be operated by communicating through a communication network.

2 is a system configuration diagram of an abbreviated issue sentence extraction system 100 according to the present invention. As shown in FIG. 2, the abbreviated issue sentence extraction system 100 according to the present invention may include a clustering module 12, an issue sentence extraction module 14, and an issue abbreviation generation module 16. The ranking module 18 may further include. In addition, the clustering module 12, the issue sentence extraction module 14, the issue abbreviation generation module 16 and the ranking determination module 18 are controlled by the module control unit 10. In particular, the module controller 10 may appropriately control each of the modules 12, 14, 16, and 18 by an instruction to the search server 100a. In addition, although not shown in FIG. 2, when the issue shorthand extraction system 100 is built in a place physically separated from the search server 100a, a predetermined communication module capable of communicating with the search server 100a may be provided. It may further include further.

In addition, the reduced issue sentence extraction system 100 according to the present invention, the document information DB 21, the search log DB 22, the clustering DB 23, the issue sentence sequence controlled by the database management module 20 DB 24, issue abbreviation DB 25 and library 26 may be included.

Here, the document information DB 21 is a database containing document information on document data such as various news articles, books, documents, and the like provided through a communication network, and includes at least a title and a URL (Uniform Resource Locator; Information about "resource location indicators" where the type and location of a particular information resource is recorded. Further, the document information DB 21 includes creation time information for the individual document, where the "creation time" may mean not only the time when the document was actually created by the author but also the publication or publication time. The service operator can collect various document data provided on the Internet using a predetermined search engine and periodically database the document information on individual document data.

The clustering module 12 groups the document data related to each other into the same cluster based on the document information on the individual document data read from the document information DB 21 for the plurality of document data thus prepared in advance. do. Here, clustering refers to classifying data into groups based on concepts such as similarity, and for example, K-mean clustering algorithm may be used. Since clustering may be understood by those skilled in the art, various clustering techniques may be used, and thus a detailed description of the clustering scheme will be omitted herein.

In addition, each cluster grouped by clustering module 12 may be databased in clustering DB 23. The clustering DB 23, when the documents related to each other by the clustering module 12 is grouped into one cluster, a plurality of documents to which an identifier (hereinafter referred to as "cluster ID") for the cluster is assigned. A database in which information about is recorded, wherein titles, URLs, and creation times of a plurality of document materials are recorded according to the cluster ID. 3 shows an example of document information for nine document materials to which the cluster ID "32584206" is equally assigned.

Next, the issue sentence extraction module 14 extracts a valid issue sentence from which duplicates have been removed from title information of individual document materials belonging to the same cluster, and then, for each of the valid issue sentences, a sentence of a unit unit string. Create a sequence of issue statements. That is, the issue sentence extraction module 14 reads the clustering DB 23 and extracts title information of a plurality of document materials assigned the same cluster ID. The duplicate title is removed by comparing and analyzing the extracted plurality of title information. A plurality of title information without duplicates is adopted as a valid issue sentence, and special characters or punctuation marks included in the valid issue sentence are removed and converted into a pure sentence form. Then, the validity issue sentences are separated by the word unit of the sentence (eg, separated by the spacing unit). The separated words form a unit string, and as a result, one valid issue sentence may be divided into a plurality of unit strings. Therefore, one valid issue sentence is converted into an issue sentence sequence in which a plurality of unit strings are arranged in a sequence, and the generated issue sentence sequence is individually stored in the issue sentence sequence DB 24.

Meanwhile, in the issue sentence sequence DB 24, a plurality of issue sentence sequences generated for document data having the same cluster ID are databaseized, and a plurality of issue sentence sequences are recorded for each cluster ID. For example, the title information of each of the plurality of document data grouped into one cluster in FIG. 3 may include (1) valid issue sentence selection process by the issue sentence extraction module 14, (2) conversion into pure sentence form, It is recorded as shown in FIG. 4 through (3) a unit string classification process and (4) an issue sentence sequence generation process.

Next, the issue abbreviation generating module 16 extracts a common unit string from a plurality of issue sentence sequences belonging to the same cluster to generate a substring. That is, the issue abbreviation generation module 16 reads the issue sentence sequence DB 24 to read a plurality of issue sentence sequences stored for each cluster ID, analyzes the issue sentence sequence, and extracts a common unit string. Substring is stored in the issue abbreviation DB 25 for each cluster ID. In this case, the following method may be used as a method of extracting a common unit string by the issue contract generation module 16.

Firstly, a Longest Common Subsequence Algorithm (hereinafter, referred to as an "LCS algorithm") may be used. Here, the LCS algorithm refers to an algorithm that finds the longest among sequences that become subsequences of all of a plurality of sequences, and finds a subsequence having the longest common sequence in each sequence. Since it will be easily understood by those of ordinary skill in the art, a detailed description thereof will be omitted here. The issue abbreviation generating module 16 may generate a substring by extracting a longest common unit string from a plurality of issue sentence sequences belonging to the same cluster using the longest common subsequence algorithm. For example, a substring consisting of "ipani", "20kg", and "loss" is extracted from the issue sentence sequences illustrated in FIG. 4 according to the LCS algorithm.

On the other hand, as a second method, it is possible to use a probability-based substring generation method. That is, the issue abbreviation generation module 16 includes a range of probability values set in advance based on the probability that each unit string constituting each issue sentence sequence belonging to the same cluster appears through all the plurality of issue sentence sequences belonging to the same cluster. You can create substrings of unit strings belonging to. For example, the issue abbreviation generation module 16 reads each issue sentence sequence having the same cluster ID from the issue sentence sequence DB 24. In addition, the unit string constituting each issue sentence sequence is analyzed in order, and the number of times the unit string is included in the entire issue sentence sequence belonging to the same cluster is counted and stored in the library 26 together with the unit string. Then, the issue abbreviation generating module 16 calculates the appearance probability value P for the unit string based on each unit string and the number of appearances thereof by the following equation.

[Equation 1]

P = (number of occurrences of the corresponding unit string) / (number of all issue sentences) × 100

The service operator may set a reference probability value in advance, and select a unit string having a appearance probability value belonging to the reference probability value range. In the case of the LCS algorithm illustrated as the first method, only the unit string having the appearance probability value of 100% is selected to form a substring, but according to the probability-based substring generation method, the unit string having the appearance probability value is not 100%. You can construct a string. For example, when the reference probability value is set to 70% or more from the issue sentence sequences illustrated in FIG. 4, a substring consisting of "ipani", "birth", "20kg", and "loss" may be generated.

The abbreviated issue sentence extraction system according to the present invention uses the substring generated by the issue abbreviation generation module 16 using any one of the above-described methods for the plurality of document materials belonging to the same cluster. Provide it as an issue abbreviation. In particular, in the case of using the probability-based substring generation method, since the information about the unit string and the number of appearances is recorded in the library 26 for each cluster, the issue abbreviation may be generated based on the number of appearances of the unit string. For example, by arranging the unit strings constituting the issue abbreviation according to the number of appearances, it is possible to improve readability when the user reads the issue abbreviation.

Meanwhile, the system 100 may further include a ranking determination module 18. The ranking determination module, when the document information DB 21 includes the creation time information of each of the document materials, calculates a newness index based on the creation time information of each of the plurality of document materials belonging to the same cluster. Based on this recency index, it is possible to rank and provide the extracted issue contracts for each of the plurality of clusters. Here, the latest index may be calculated by normalizing as an index based on a time interval from, for example, a reference time to a creation time, based on URL and creation time information of individual document data among document information included in each cluster. According to the calculated freshness index, the issue contract for the cluster can be provided along with the ranking.

In addition, the ranking module 18 may rank each issue abbreviation for a plurality of clusters based on the degree of interest of the user. For example, the system 100 may further include a search log DB 22, and the search log DB 22 may include a number for identifying a corresponding user terminal when a user views individual document data through a search engine. Alternatively, the information may include information on a user identifier in text, URL information on a corresponding document, and information on a time read by the corresponding user terminal (hereinafter, referred to as “visit time”). The ranking module 18 may count the number of visits of the user for a plurality of document materials belonging to the same cluster through the search log DB 22, and calculate the user's interest index. For example, the number of URLs corresponding to document data belonging to the cluster may be counted, and the number of visits to each URL may be added to normalize the index of interest for the cluster.

The reduced issue sentence extraction method according to the present invention includes a clustering step of grouping related document data into the same cluster from document information DB for a plurality of document data in which at least information on a title and a URL of a document are recorded; An issue sentence extraction step of extracting a valid issue sentence having duplicates removed from title information of individual document materials belonging to the same cluster and generating an issue sentence sequence in which a sentence of a sentence is a unit string for each valid issue sentence; The method may further include generating an issue contract statement for generating a substring by extracting a common unit string from a plurality of issue sentence sequences belonging to the same cluster. The generated substring may be provided as an issue abbreviation for a plurality of document data belonging to the cluster.

In particular, in the issue abbreviation generation step, the substring may be generated by extracting the longest common unit string from the plurality of issue sentence sequences belonging to the same cluster according to the longest common subsequence algorithm. Alternatively, in the issue abbreviation generation step, each of the unit strings constituting each of the issue sentence sequences belonging to the same cluster may fall within a preset probability value range based on a probability appeared throughout the plurality of issue sentence sequences. The substring consisting of the unit string may be generated.

And, by using the document information DB containing the creation time information of each of a plurality of document data, by calculating the most recent index based on the creation time information of each of a plurality of document materials belonging to the same cluster, a plurality of clusters The method may further include a ranking determining step of ranking the plurality of issue contracts based on the issue index for each of the issue contracts.

Further, in the ranking determination step, a plurality of documents belonging to the same cluster from a search log DB including at least a user identifier, URL information for each document data included in the document information DB, and user visit time information for each URL. The interest index of the user may be calculated by counting the number of visits of the user to the data, and the plurality of issue contracts may be ranked based on the interest index for each issue contract for a plurality of clusters.

The above-mentioned abbreviated issue sentence extraction method may be performed by a general purpose computer device. For example, the computer device may include one or more processors or a central processing unit (CPU) connected to a main memory including a random access memory (RAM) and a read only memory (ROM). As is well known in the art, ROM is responsible for transferring data and instructions unidirectionally to the CPU, and RAM is typically used to transfer data and instructions bidirectionally. The RAM and ROM may comprise any suitable form of computer readable medium. The mass storage device is bi-directionally coupled to the processor to provide additional data storage capabilities and may be any of a computer-readable recording medium. The mass storage device is used to store programs, data, and the like, and may be a hard disk, which is usually slower than the main storage device, or an auxiliary storage device such as a CD or a DVD. And the processor may be connected to the wired or wireless communication network via a network interface. Through the network connection, the procedure of the above-described method can be performed. In addition, the reduced issue sentence extraction method according to the present invention may be provided as a computer-readable recording medium configured as one or more software programs and executable thereon.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the invention. It is therefore to be understood that the embodiments of the invention described herein are to be considered in all respects as illustrative and not restrictive, and the scope of the invention is indicated by the appended claims rather than by the foregoing description, Should be interpreted as being included in.

Claims (12)

A clustering module for grouping related document materials into the same cluster from document information DBs for a plurality of document materials in which at least information on a title and a URL of the document are recorded;
An issue sentence extraction module for extracting a valid issue sentence from which duplicates are removed from title information of individual document data belonging to the same cluster, and then generating an issue sentence sequence using a sentence as a unit string for each valid issue sentence. ; And
And an issue contract generation module for generating a substring by extracting a common unit string from a plurality of issue sentence sequences belonging to the same cluster.
And the substring generated by the issue contract generation module as an issue contract for the plurality of document materials belonging to the same cluster.
The method of claim 1,
The shortened issue sentence extracting system is configured to extract the longest common unit string from the plurality of issue sentence sequences belonging to the same cluster according to a longest common partial sequence algorithm to generate the partial string. .
The method of claim 1,
The issue contract generation module may include: the unit string belonging to a preset probability value range based on a probability that each of the unit strings constituting each of the issue sentence sequences belonging to the same cluster appears through the entire plurality of issue sentence sequences; Shortened issue sentence extraction system, characterized in that for generating the substring consisting of.
delete delete A retrieval system comprising the abbreviated issue sentence extraction system according to any one of claims 1 to 3.
A clustering step of grouping related document materials into the same cluster from document information DBs for a plurality of document materials in which at least information on a title and a URL of the document are recorded;
The issue sentence extraction step of extracting a valid issue sentence from which duplicates are removed from title information of individual document materials belonging to the same cluster, and generating an issue sentence sequence in which a sentence of a sentence is a unit string for each valid issue sentence. ; And
And a shortened issue sentence generating step of generating a substring by extracting a common unit string from a plurality of issue sentence sequences belonging to the same cluster.
The method of claim 7, wherein
The generating of the shortened issue sentence may include extracting the longest common unit string from the plurality of issue sentence sequences belonging to the same cluster according to a longest common partial sequence algorithm to generate the partial string. .
The method of claim 7, wherein
The generating of the issue contract may include generating the unit string belonging to a preset range of probability values based on a probability that each of the unit strings constituting the issue sentence sequence belonging to the same cluster is present through the entire plurality of issue sentence sequences. Shortened issue sentence extraction method, characterized in that for generating the substring consisting of.
delete delete A computer-readable recording medium containing a program for executing the abbreviated issue sentence extracting method according to any one of claims 7 to 9.

Images