JP2012018667A - Method, system and computer readable record medium for refining web document using text pattern extraction - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method, a system and a computer readable record medium for refining a web document using text pattern extraction.SOLUTION: A web document refinement method includes: a text pattern extraction stage in which a plural pieces of extract object data are analyzed by a predetermined criterion to extract the text pattern, and the plural pieces of extract object data are arranged and aligned based on the extracted text pattern; a normal equation extraction stage in which a normal equation is extracted from the plural pieces of extract object data arranged and aligned; and a web document refine stage in which a web document is refined and secondary data is generated using the extracted normal equation.

Description

  The present invention relates to a method, a system, and a computer-readable recording medium for refining a Web document using text pattern extraction, and more particularly, extracts a text pattern based on a Web document and uses the extracted text pattern. The present invention relates to a method, a system, and a computer-readable medium for extracting a regular expression that describes an entire pattern and then applying the extracted regular expression to a Web document for refinement.

  Various services using the Internet are provided due to the increase in the development and spread of the Internet, and a representative example is a search service. In such a search service, when a user inputs a word or a combination of words to be searched as a query, the search engine includes a search result text corresponding to the input query (for example, a search query input from the user). Web site, article, or an image having a file name including the corresponding search query).

  In this way, in order to appropriately provide the content that the user wants to search, the provider of the Internet search service collects such websites and articles in advance using a web crawler or input means provided separately. After that, keywords are extracted from collected websites and articles through morphological analysis, etc., indexed based on this, stored separately, and when the user enters a query, the search results are faster In general, it is implemented so as to be provided to the user.

  However, search results that have keywords that match the query entered by the user when keyword extraction and indexing operations using existing morpheme analysis are performed without processing the websites and articles collected by the prior art. It is determined that the search matches the other text, and the search is provided unconditionally. However, such search result text may include content that does not match the user's search intention.

  For example, when a user tries to search for an article about the actor “Takuya Kimura” and enters “Takuya Kimura” as a query, there is a reporter named “Takuya Kimura” in the collected articles. May be included in the body of the article. In this case, even if the content of the article is irrelevant to the content that the user wants to search, since there is a keyword that matches the query entered by the user, it is included in the text of the search result and provided to the user. It occurs. Referring to FIG. 5 a, when the user inputs the query “Takuya Kimura”, as shown in the red rectangle A part, the reporter name is completely unrelated to the actor “Takuya Kimura”. It can be seen that the content of the article “Takuya” is searched and displayed.

  Furthermore, unlike websites or articles created or edited directly by Internet search service providers, websites created by a third party and then collected by web crawlers, etc. must be tailored to a specific format. In order to re-categorize the search results according to another classification standard, such as the creator or the creation area, it is necessary to directly check and classify the search results manually. There have been difficulties in providing users with a separate classification standard such as creator or creation area.

  Therefore, in order to prevent such search errors and to provide more efficient search results, we will refine the collected websites and articles appropriately, and delete parts that are not related to the contents or separate them. There is a need for a technique for re-classification according to a standard and alignment with an index. Conventionally, there is no method for solving such a problem other than manual work by humans.

Korean Patent No. 0435442 Japanese Patent Laid-Open No. 2005-190074 Korean Open Patent 2002-0089677

  An object of the present invention is to solve the problems of the prior art described above.

  Another object of the present invention is to prevent errors that may be included in the search results by appropriately refining the collected websites and articles and deleting portions that are not related to the content that the user wants to search. And providing more accurate search results.

  Another object of the present invention is to extract contents for reclassifying collected websites and articles according to other criteria such as date, coverage, or reporter name through appropriate refinement. By searching or sorting by using as an index, more diverse and accurate search results can be provided.

  In order to achieve the object of the present invention as described above and achieve the following effects of the present invention, the characteristic configuration of the present invention is as follows.

  The Web document refinement method according to an embodiment of the present invention analyzes a plurality of extraction target materials according to a predetermined standard, extracts a text pattern from the plurality of extraction target materials, and based on the extracted text pattern, A text pattern extraction stage for ordering a plurality of extraction target materials, a regular expression extraction stage for extracting a regular expression from the ordered plurality of extraction target materials, and refining a Web document using the extracted regular expressions It is obtained by performing a Web document refinement step of generating secondary material.

  According to another embodiment of the present invention, the Web document refinement system analyzes a plurality of extraction target materials according to a predetermined standard, extracts a text pattern from the plurality of extraction target materials, and based on the extracted text pattern A text pattern extracting means for ordering the plurality of extraction target materials, a normal expression extracting means for extracting a regular expression from the ordered plurality of extraction target materials, and a Web document using the extracted regular expressions Includes regular expression application means for refinement and generation of secondary material.

  According to the embodiment of the present invention, it is possible to appropriately refine collected websites and articles, and to delete portions that are not related to the content that the user wants to search. And more accurate search results can be provided.

  In addition, according to the present invention, appropriate refinement of collected websites and articles, etc. is performed, and contents for reclassification according to other criteria such as date, coverage area or reporter name are extracted and used as an index. By searching or sorting, it is possible to provide more diverse and accurate search results.

The overall configuration of a search result providing system using a search database that is refined by applying a regular expression through text pattern extraction to a collected Web document according to an embodiment of the present invention FIG. It is a detailed block diagram of the search result provision system which concerns on one Embodiment of this invention. It is a detailed block diagram of the secondary process part in the search result provision system which concerns on one Embodiment of this invention. It is a graph which shows the PMI frequency in the frequency analysis means used with the Web document refinement system concerning one embodiment of the present invention. 3 is an exemplary screen illustrating a search result searched by a specific query in the prior art. 6 is an exemplary screen illustrating search results searched by a specific query using a Web document refinement system according to an embodiment of the present invention. 4 is an exemplary screen illustrating a result of being reclassified to a coverage area using a Web document refinement system according to an embodiment of the present invention. It is an exemplary screen illustrating a search result corresponding to a query by a reporter name after performing a search in the prior art. 6 is an exemplary screen illustrating search results reclassified by reporter name and provided using a Web document refinement system according to an embodiment of the present invention. It is a flowchart of the operation | movement in the secondary process part of the Web document refinement system which concerns on one Embodiment of this invention. It is the screen which showed one Example of the result of the gene analysis using a multiple sequence alignment (MSA, Multiple Sequence Alignment) technique.

  The following detailed description of the invention illustrates certain embodiments in which the invention can be practiced and refers to the accompanying drawings in which the embodiments are illustrated. These embodiments are described in detail to enable those skilled in the art to practice the invention. Various embodiments of the present invention are different from each other but need not be mutually exclusive. For example, the specific shapes, structures, and characteristics described herein can be implemented in other embodiments to the extent that they do not depart from the spirit and scope of the present invention with respect to one embodiment. In addition, the position or arrangement of individual components within each disclosed embodiment can be altered without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is, when properly described, what the claims claim and the equivalent scope of the appended claims Limited only by scope. In the drawings, like reference numbers indicate identical and similar functions throughout many aspects.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice the present invention.

  In the preferred embodiment of the present invention, the term “Web document” refers to manual or active, which can be browsed via the World Wide Web using a web browser program such as Internet Explorer, directly or indirectly. It is used in a broad sense that includes all text forms. HTML (HyperText Markup Language) is mainly used as a file format of a Web document, but is not necessarily limited to this, but XML (eXtensible Markup Language), SGML (Standard General Language), and SGML (Standard General Language) are included. Any text format that can be browsed directly or indirectly (including when connected to a plug-in or other separate program) using a web browser program corresponds to a web document. In order to browse a web document using a web browser program, generally, an address where the web document is located is input as a URL, and HTTP (HyperText Transfer Protocol) is often used as the address format, but this is not necessarily limited thereto. Not. The content of the Web document is not limited to a specific format, is not limited to general text, and can include various forms such as an image, music, a moving image, or a combination thereof. In addition, according to the specific classification according to the source of the Web document, the Web document may include general Web documents, advertisements, dictionaries, blogs, websites, news, clubs, images, specialized information, books, maps, videos, etc. Well, it is not limited to these. The “primary processed material” and “secondary processed material” obtained by processing the Web document having various sources and formats described above also have various sources and formats.

  In addition, the terms “refine” and “refinement” in the embodiments of the present invention are used in a broad sense including all operations for deriving a modified web document by performing predetermined processing on the web document. It is done. Here, as an embodiment of refinement, it may mean an operation of deleting a specific word, term, or part from an existing Web document, or a specific term or keyword inside an existing Web document is extracted and used. This may mean an operation of reconfiguring the database so that it can be arranged based on an indexing operation or a Web document selection operation, but is not necessarily limited thereto.

  In order to describe the preferred embodiment in the specification of the present invention, the content of the Web document is assumed to be a newspaper article. However, this does not mean that the Web document is limited to a newspaper article. It is obvious that the present invention can be applied to a Web document having various other contents.

Diagram 1 of the entire system, according to one embodiment of the invention, the collected Web documents, refined by applying the normal through the text pattern extraction equation using the search database constructed using this search It is drawing which shows schematically the whole structure of a result provision system.

  As shown in FIG. 1, in the entire system according to an embodiment of the present invention, a search result providing system 100 including a search database is connected to a plurality of user terminal devices 300 and a plurality of Web document servers 400 through a network 200. Has been.

  First, according to an embodiment of the present invention, the search result providing system 100 receives a search word, that is, a query from the user terminal device 300, and performs a search with reference to a search database (not shown) based on the search word. Thereafter, the search result derived from the result is transmitted to the user terminal device 300. In addition, the search result providing system 100 extracts a text pattern from a Web document collected from a plurality of Web document servers 400 through analysis based on a predetermined standard, and deletes it from the primary processing material or performs indexing as a separate classification standard. It also serves to generate a secondary processing material through a refinement process by searching for a necessary portion, generating a normal expression using text pattern extraction for the corresponding portion, and applying the generated normal formula to the primary processing material.

  In addition, according to an embodiment of the present invention, the network 200 may be configured regardless of communication modes such as wired and wireless, and may be a personal area network (PAN), a local area network (LAN), or a local area network (LAN). ), A metropolitan area network (MAN), a wide area network (WAN), and the like.

  On the other hand, the user terminal device 300 according to an embodiment of the present invention includes an input / output including a function for connecting a search result providing system 100 via the network 200 in order for a user to be provided with a search result for a predetermined query. Means a device, not only a desktop computer, but also a notebook computer, workstation, palmtop computer, personal digital assistant (PDA), web pad, mobile communication terminal including a smartphone, etc. Any digital device having a memory means, a microprocessor, and a computing capability may be included in the user terminal device 300 according to the present invention. Preferably, it is desirable that the web browser in the user terminal device 300 can be executed and used in order to connect to the search result providing system 100 and provide a search result by inputting a query. It is not limited.

  In addition, the Web document server 400 according to an embodiment of the present invention may be a Web server that includes Web documents that the search result providing system 100 collects through a predetermined method. The present invention is not limited to a server that handles content / format Web documents. Therefore, all web servers that can be accessed by the search result providing system 100 through the network 200 and collect web documents may be included in the web document server 400. Preferably, the web document handled by the web document server 400 may include a web document in which a news article is described.

Search Result Providing System FIG. 2 is a detailed block diagram of the search result providing system 100 according to an embodiment of the present invention.

  Referring to FIG. 2, the search result providing system 100 according to an exemplary embodiment of the present invention may include a transmission / reception unit 110, a search unit 120, a primary processing unit 130, a secondary processing unit 140, and a search database 150.

  The transmission / reception unit 110 receives a query from the user terminal device 300, transmits the query to the search unit 120, and transmits the search result extracted from the search unit 120 to the user terminal device 300.

  The search unit 120 searches the search database 150 in which the secondary processed material is stored for information that matches the query received from the transmission / reception unit 110. The search result extracted through the search is transmitted to the transmission / reception unit 110 for transmission to the user terminal device 300. In addition, the search unit 120 searches the secondary processing material stored in the search database 150 for information that matches the query, and then extracts information about the derived search result from the primary processing material stored in the database 150. The data may be transmitted to the transmission / reception unit 110.

  The primary processing unit 130 extracts a keyword from a Web document already collected from the Web document server 400 by morphological analysis or the like, performs an indexing operation based on the keyword, and searches for a primary processing material created by a search database. Save to 150. Here, the Web document may be collected by a known web crawler, and keyword extraction and indexing by morphological analysis may be performed by a known method.

  The secondary processing unit 140 extracts a portion where the frequency of appearance of a specific token is a specific numerical value or more for the primary processing material that has been subjected to keyword extraction and indexing work by the primary processing unit 130, A text pattern is extracted through analysis based on a predetermined standard for this, and aligned according to the extracted pattern, and a regular expression to be applied to the entire pattern is extracted based on the aligned contents. Further, the secondary processing unit 140 can store the result of refining the primary processing material based on the extracted regular formula as the secondary processing material in the search database 150. Detailed functions for each component of the secondary processing unit 140 will be described later.

  The search database 150 is a Web document that has already been collected from the Web document server 400, a primary processed material that has passed through the primary processing unit, a secondary processed material that has passed through the secondary processing unit, and a search result that has been searched and extracted by the search unit 120. Common name for the space to store. For simplicity of illustration of the present invention, only one search database 150 is shown in FIG. 2, but in other embodiments of the present invention, the various data may be partitioned into one or more physical partitions. It is obvious to those having ordinary knowledge in the field to which the present invention belongs that the data can be stored in the database. In addition, the primary processing material and the secondary processing material may exist separately, or the same contents of both materials may be related to each other, and the search unit 120 stores the secondary processing material stored in the search database 150. After performing a search through the processed material, the search result may be extracted and provided from the primary processed material related thereto.

  The transmission / reception unit 110, the search unit 120, the primary processing unit 130, and the secondary processing unit 140 in FIG. 2 may be physically implemented in one piece of hardware. It can be embodied in other hardware, and there can be a plurality of physically existing hardware performing the same function in parallel. As described above, it is not limited to the physical number and location of the hardware or database in which each component of the present invention is provided, and the fact that the design can be changed in various ways means that it is normal knowledge in the technical field to which the present invention belongs. It is obvious in those who have

Secondary Processing Unit The secondary processing unit 140 in the search result providing system 100 according to an embodiment of the present invention will be described in more detail with reference to FIG. 3. The secondary processing unit 140 includes a frequency analysis unit 141, a text pattern extraction. Means 142, regular expression extraction means 143, and regular expression application means 144 can be included.

  Here, the frequency analysis unit 141 according to the embodiment of the present invention targets primary processing data that has been extracted and indexed by the primary processing unit 130 and stored in the search database 150 or a separate database. Analyze which part of the primary processed data the frequency of appearance of a specific token (token, lexical analysis unit) is greater than a specific value. As a preferred embodiment of such a study criterion, after obtaining a transition of a PMI (Pointwise Manual Information) value corresponding to a predetermined token in a specific class using the following Equation 1, the PMI value is equal to or greater than a specific numerical value. The part can be analyzed.

Here, P (W) means the overall frequency of the specific token, and P (W | C) means the frequency of the specific token in the class. As an example, when a class is specified as a news article, a token that appears more frequently in a news article than a frequency of appearance in the entire Web document (for example, “newspaper” or “reporter”), a PMI value than in the case of other tokens. May be relatively high.

  As a more specific example, in order to extract a pattern corresponding to a reporter name when the class is a news article, a token that can exist near the reporter name, that is, the term “reporter” and inserted after the reporter name When the PMI value for the possible symbol “)” is obtained, it can be shown as in FIG. In the graph of FIG. 4, the horizontal axis represents a position in the news article, the vertical axis represents the PMI value for the term “reporter” and the symbol “)”, and the portion where the PMI value is equal to or greater than a specific numerical value is It is displayed with diagonal lines. That is, in the news article that is the primary processed material, the part where the PMI value for the term “reporter” and the symbol “)” is equal to or greater than a specific numerical value is displayed as A from the beginning of the news articles displayed with diagonal lines. It can be seen that this corresponds to the predetermined part, and from the predetermined part displayed in B to the end of the news article. Therefore, the text pattern can be extracted by the text pattern extraction unit 142 for the corresponding portion (hereinafter referred to as “extraction target portion”) displayed with diagonal lines in FIG. Since the role of the frequency analysis means 141 is to select an extraction target part necessary for extracting the text pattern from the primary processed material through the analysis, the text pattern extracting means 142 selects the specific part instead of the entire primary processed material. By making it possible to proceed with the work only for the text pattern, the load on the text pattern extraction means 142 can be greatly reduced.

Next, the text pattern extraction unit 142 according to an embodiment of the present invention uses a predetermined reference as an extraction target portion in which the frequency of the specific token selected from the frequency analysis unit 141 in the primary processing material is a specific numerical value or more. In this case, the text pattern is extracted by analysis, and based on this, the extraction target portions function in a one-dimensional manner. The function of the text pattern extraction unit 142 can be applied by introducing a Multiple Sequence Alignment (MSA) technique, which is a core technique for extracting a common sequence necessary for searching a genetic map using DNA, as its motif. The genetic map shows what gene is in which position in the chromosome in which the four alphabets of A, T, G, and C are arranged in a one-dimensional sequence, and is common using MSA technology. A plurality of DNAs having characteristics (for example, a person with blue eyes) are arranged, a common sequence is extracted from the arranged DNAs, and then genes for the common characteristics (blue eyes) are extracted. Referring to FIG. 8, which is an embodiment of a gene analysis result using the MSA technique, common features through a plurality of DNA sequences can be searched as illustrated in FIG. As a specific example of applying the MSA technique used for the genetic map search described above, the frequency analysis means 141 selects the term “reporter” and the part where the PMI value for the symbol “)” is equal to or greater than a specific value. Assume that the extracted part to be extracted is the same as in Table 1 below.

  The text pattern extraction unit 142 analyzes a plurality of extraction target portions corresponding to Table 1 with a predetermined standard, extracts a text pattern, and arranges and arranges the plurality of extraction target portions one-dimensionally based on this. More specifically, a morphological analysis may be used as a predetermined standard that can be used by the text pattern extraction unit 142. In addition, a Chart type, a Word type, a Byte type, a Word Type such as Korean, English, or date, a city, a time, An abstract concept such as an animal or a word unit, or a chunk or a page may be used, but is not limited thereto, and a plurality of the above-described criteria are combined and combined. Can be used. Further, a separate database in which specific words or terms are stored at the time of analysis may be referred to. For example, in Table 1 above, “Tokyo”, “Osaka”, “Yokohama”, and “Vancouver” are all words that correspond to “region”, and are stored in a separate database in advance, or the word “Vancouver”. Even if the number of characters is different from other words, it is analyzed by referring to the database that they are the same words corresponding to “region”, and all the extraction target parts corresponding to Table 1 are all represented by the symbol “(”. In the following, it is possible to grasp that the word corresponding to “Region” has a text pattern where the word is located, and as another example, the word corresponding to the reporter name is stored in a separate database. Even if it is not, it can be analyzed that it corresponds to a proper noun indicating someone's name.

  In addition, since all words cannot be stored in a separate database, the contents of the database may be added, changed, or deleted using a learning method (heuristic). As an example, if only “Tokyo”, “Osaka”, and “Yokohama” in Table 1 are stored in a separate database in advance as words corresponding to “Region”, among the extraction target portions of Table 1, In all three sentences, the character corresponding to the region is located after the symbol “(”, followed by the symbol “=”, so the same form, that is, the portion surrounded by the symbols “(” and “=” The character located in is analyzed when the region name comes out, and the word “Vancouver” in the fourth sentence is recognized as the region name even if it is not stored in the database, and the term “Vancouver” is added to the database In this case, “Vancouver” can be used for analysis of a new extraction target portion in the future.

  In the case of newspaper articles, the format (for example, an article creation format in which the reporter name is written before the reporter content, the title is written after the title, or the article is written at the end of the article) Therefore, after analyzing according to a predetermined standard, the text pattern is extracted, and the period of the extraction target part is limited by date or in units of weeks in order to align and align one-dimensionally. Various criteria may be applied, and the criteria used are not limited to the time period alone.

  As described above, the text pattern extraction unit 142 analyzes the extraction target portion corresponding to Table 1 according to a predetermined standard, extracts the pattern, and arranges the extraction target portion one-dimensionally based on the extracted pattern. The results can be shown as in Table 2 below.

Based on the result of the one-dimensional alignment shown in Table 2, the positions of symbols such as parentheses and equal signs located near the reporter name, the report area, and the part corresponding to the reporter name were all aligned by the text pattern. As can be seen, the word “TBC” is only included in some articles, so there is no part corresponding to other articles, and the word “TBC” is arranged in a separate position. It can be confirmed that the results shown in Table 2 are similar to the MSA results used in the genetic map shown in FIG.

  Next, in the regular expression extraction unit 143, a regular expression that is arranged one-dimensionally based on the text pattern via the text pattern extraction unit 142 and can be expressed by a generalized expression based on the aligned contents. Extract. Following the above example, regular expressions can be extracted as shown in Table 3 below based on the contents presented in Table 2.

In the regular expression disclosed in Table 3, first, the symbol “(” is located, and then a word consisting of 2 to 4 kanji characters or Japanese is located (regular expression <kanji, Japanese> {2, 4 }, Followed by a symbol “=”, followed by a word consisting of 4 kanji characters (displayed by a regular expression <kanji> {4}). ”May appear, but this may be indicated arbitrarily, that is, only in some articles, so that the regular expression“ (TBC)? ", Then the word" reporter "and the symbol") "will be located, after which no other content that can be normalized is shown, It is not related to the reporter name pattern to be extracted. Can be revealed. Further, referring to a separate database in the text pattern extracting means 142, 2 to 4 character kanji characters or Japanese words located after the symbol “(” indicate the coverage area, and 4 characters located after the symbol “=”. It may be determined that a word consisting of kanji indicates a reporter name, and the corresponding content may be included in the regular expression.Of course, the regular expression according to Table 3 is limited to the one-dimensionally arranged and aligned example in Table 2. Therefore, it should be understood that the regular expression can be changed in any number according to the contents arranged side by side according to the text pattern, and the expression method of the regular expression is not limited to Table 3. I must.

  Finally, the regular expression applying unit 144 according to an embodiment of the present invention generates a secondary processing material from the result of refinement by applying the normal formula extracted by the normal formula extracting unit 143 to the primary processing material. To do. As an example of refinement by applying a regular expression, the regular expression applying unit 144 may delete the reporter name from the primary processing material. When the normal formula shown in Table 3 is used, the normal formula applying unit 144 uses the primary processing material. Since it is determined that the word consisting of 4 kanji characters positioned immediately after the first symbol “=” corresponds to the reporter name, the 4 characters consisting of kanji existing at the corresponding position are recognized as the reporter name. Then, after deletion, it may be stored as a secondary processing material in the search database 150 or a separate database. Here, since the target for the regular expression applying unit 144 to refine by applying the regular expression is the primary processed material itself for which the keyword extraction and indexing work has been completed by the primary processing unit 130, the frequency analysis unit This is different from the extraction target portion selected from 141. As described above, in the embodiment in which the reporter name is deleted from the secondary processing material, when the user inputs the query “Takuya Kimura”, the search unit 120 searches from the secondary processing material from which the reporter name is deleted and the search result. 5b, as shown in FIG. 5b, only an accurate article including a query in the title or the article body can be provided, and a search result different from the search result of FIG. 5a according to the prior art is displayed. You can see that

  As another example of applying the regular expression, the regular expression applying means 144 reclassifies the primary processing material based on the reporter name or the report area, and uses the secondary processing material as an index to process the secondary processing material. It may be generated. When using the regular expressions in Table 3, the regular expression applying means 144 confirms the first symbol “(” in the primary processed material, followed by the reporting area where 2 or 4 characters are written in Kanji or Japanese, and the symbol after that. Confirm that “=” appears, reclassify based on the kanji or Japanese corresponding to the report area, and generate secondary processed data set as an index, or from the kanji that appears after the symbol “=” It is possible to re-classify based on this four-character reporter name, reclassify it as a standard, and generate a secondary processed material set as an index. When the secondary processing material is generated, the search unit 120 searches the secondary processing material for an article created from a specific coverage area, for example, “Kagoshima” by user selection or query input, and derives a search result. Because you can As shown in Fig. 6a, it is possible to accurately provide only articles whose coverage area is "Kagoshima" regardless of the title of the article and the content of the text. When the next processed material is generated, an article created by a specific reporter, for example, “Toshiyuki Tanaka”, is selected from the secondary processed material by the search unit 120 by the user's selection or query input, and the search result is derived. Therefore, as shown in FIG. 6c, it is possible to accurately provide only an article whose reporter name is “Toshiyuki Tanaka” regardless of the title of the article and the content of the text. This is shown in FIG. When the names of different names with the same name exist as keywords in the text or title, it can be seen that even if the reporter names are different, they are not included in the search results.

  The above two examples of regular expression application are only specific examples in which the regular expression is applied by the regular expression application unit 144, and are based on standard techniques in the technical field to which the present invention belongs and technical common knowledge of those skilled in the art. The examples of applying the regular expression can be used alone or in combination by various methods, or modified so that the same or similar functions can be performed.

  FIG. 7 is an operation flowchart in the secondary machining unit according to the embodiment of the present invention.

  According to the embodiment illustrated in FIG. 7, the frequency analysis unit 141 of the secondary processing unit may be located near a specific token, for example, a reporter name, among a plurality of Web documents to be refined, for example, news articles. The frequency of appearance of a specific token including “Reporter”, “”, etc., which are possible terms or symbols, is obtained (S100). Here, the frequency may be obtained using the PMI value described above.

  Next, the frequency analysis unit 141 selects a portion whose calculated frequency is equal to or higher than a predetermined numerical value as an extraction target material (S110).

  Thereafter, the text pattern extracting unit 142 of the secondary processing unit analyzes the selected extraction target material according to a predetermined standard including morphological analysis and extracts the text pattern, and based on this, extracts a plurality of extraction targets. The parts are arranged and aligned (S120). When analyzing the material to be extracted according to a predetermined standard, a separate database in which specific words or terms are stored may be referred to. Here, the separate database is a learning method that reflects the analysis result of the material to be extracted. It may be changed and updated by (heuristic).

  Thereafter, the regular expression extraction unit 143 of the secondary processing unit extracts a regular expression from a plurality of extraction target materials arranged and aligned (S130).

  Thereafter, the regular expression providing unit 144 of the secondary processing unit deletes a specific term or keyword expressed in the regular expression from a plurality of Web documents to be refined using the extracted regular expression, or The secondary document is generated through refinement such as reclassifying the Web document as a reference and setting it as an index (S140). The secondary material generated in this way may be a search target instead of the primary material when a query by the user is received.

  Embodiments according to the present invention may be embodied in a program instruction form that can be executed through various computer means, and may be recorded on a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc., alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field. Examples of the computer-readable recording medium include a hard disk, a floppy (registered trademark) disk, a magnetic medium such as a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, and a floppy disk. Specially configured to store and execute program instructions such as magneto-optical media such as optical disks and ROM, RAM, flash memory, etc. Hardware devices are included. Examples of program instructions include not only machine language codes created by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. A hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

  As described above, the present invention has been described with reference to specific embodiments of the invention such as specific components and limited embodiments and drawings, which are provided for a more general understanding of the present invention. Thus, the present invention is not limited to the above-described embodiments, and various modifications and variations of the present invention can be made from the above description by those who have ordinary knowledge in the technical field to which the present invention belongs.

  Therefore, the idea of the present invention should not be limited to the above-described embodiments, and not only the scope of the claims but also all modifications equivalent to and equivalent to the scope of the claims It can be said that it belongs to the category.

Claims (16)

A text pattern extraction step of analyzing a plurality of extraction target materials according to a predetermined standard, extracting the text pattern, and arranging the plurality of extraction target materials side by side based on the extracted text pattern;
A regular expression extraction step of extracting a regular expression from the plurality of extraction target materials arranged and aligned;
A Web document refinement method including a Web document refinement step of refining a Web document using the extracted regular expression to generate a secondary material. Before the text pattern extraction step,
2. The method according to claim 1, further comprising: a frequency analysis step of obtaining a frequency at which a specific token appears in the Web document and selecting a part having the frequency equal to or higher than a predetermined numerical value as the extraction target material. Web document refinement method.   The Web document refinement method according to claim 2, wherein a PMI value is used as the frequency.   4. The Web document refinement method according to claim 2, wherein the specific token includes a term or a symbol located in the vicinity of a reporter name in the Web document.   5. The Web document refinement method according to claim 1, wherein the predetermined criterion includes morphological analysis. In the text pattern extraction step,
The Web document refinement method according to any one of claims 1 to 5, wherein the extraction target material is analyzed with reference to a database in which specific words or terms are stored.   7. The Web document refinement method according to claim 6, wherein contents of the database are changed to reflect an analysis result of the extraction target material.   The refinement of the Web document includes deleting a specific term or keyword expressed in the regular expression from the Web document, or reclassifying the Web document based on the specific term or keyword and setting the index as an index. The Web document refinement method according to claim 1. After the Web document refinement step,
The Web document refinement method according to any one of claims 1 to 8, wherein a query is received from a user terminal device, and a search based on the query is performed on the secondary material. A text pattern extracting means for analyzing a plurality of extraction target materials according to a predetermined standard, extracting a text pattern thereof, and arranging the plurality of extraction target materials side by side based on the extracted text pattern;
A regular expression extraction means for extracting a regular expression from the plurality of the extraction target materials arranged and aligned;
A Web document refinement system, comprising: a normal expression application unit that refines a Web document using the extracted normal expression and generates secondary material.   The frequency analysis unit according to claim 10, further comprising: a frequency analysis unit that obtains a frequency at which a specific token appears in the Web document and selects a part having the frequency equal to or higher than a specific numerical value as the extraction target material. Web document refinement system. The text pattern extraction means includes
The Web document refinement system according to claim 10 or 11, wherein the extraction target material is analyzed with reference to a database in which specific words or terms are stored.   13. The Web document refinement system according to claim 12, wherein contents of the database are changed to reflect an analysis result of the extraction target material.   The refinement of the Web document includes deleting a specific term or keyword expressed in the regular expression from the Web document, or reclassifying the Web document based on the specific term or keyword and setting the index as an index. The Web document refinement system according to any one of claims 10 to 13.   The Web document refinement system according to any one of claims 10 to 14, further comprising a search unit that searches the secondary material by a query received from a user terminal device.   A computer-readable recording medium having recorded thereon a program for performing each step of the Web document refinement method according to any one of claims 1 to 9 on a computer.