KR102033416B1 - Method for generating data extracted from document and apparatus thereof - Google Patents

Abstract

Provided are a method and apparatus for extracting required data from an externally collected document and generating structured structured data. According to an embodiment of the present invention, a method of generating data may include obtaining the document from a document providing source, obtaining a data extraction parameter corresponding to the document providing source from a parameter database, and based on the data extraction parameter. Constructing a data generation rule that defines a rule for extracting data from the document and a format for storing the extracted data; and extracting structured data from the document according to the data generation rule.

Description

METHOD FOR GENERATING DATA EXTRACTED FROM DOCUMENT AND APPARATUS THEREOF}

The present invention relates to a method and apparatus for generating structured structured data from a document. More specifically, the present invention relates to a method for extracting data needed from documents collected from the outside and generating structured structured data.

With the development of communication and computing technologies, numerous documents are available over the Internet. Techniques such as crawling or scraping are used to obtain the necessary data from the documents provided over the communication network. Techniques such as crawling or scraping are techniques for obtaining a web document and extracting data from the obtained web document. For example, a search engine of a portal service may provide a search service by collecting and indexing information of the web. Here, the configuration of collecting information and extracting data of the web is referred to as a spider, a bot or an intelligent agent.

It is necessary to extract and database the desired data from numerous documents collected through the Internet. For example, security vulnerability information such as software products or network devices provided through various websites may need to be collected and organized into a database. To this end, the computing device collects documents through crawling, and an analysis device having a module (eg, a parser) for analyzing the documents extracts and analyzes the necessary data from the documents. Here, the module for analyzing the document extracts and analyzes the data according to the data presentation method of the document. For example, a module for analyzing a document may be configured to extract data located after the hyphen (-).

However, in the case of extracting data by using an analysis module dependent on the data presentation method of the document, whenever the data presentation method of the document is changed by receiving a document from another source or changing the format of the document provided by the source Redevelopment of the analysis module is inevitable. In addition, there is a problem in that the operation of the computing device that collects and analyzes the document must be terminated in order to apply the re-developed analysis module.

The technical problem to be solved by the present invention relates to a method and apparatus for minimizing redevelopment of a module for analyzing documents collected from various documents even if the form or presentation structure of the collected document is changed.

Technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above technical problem, a data generation method according to an embodiment of the present invention, obtaining the document from a document providing source, obtaining a data extraction parameter corresponding to the document providing source from a parameter database, Constructing a data generation rule defining a rule for extracting data from the document and a format for storing the extracted data based on the data extraction parameter and extracting structured data from the document according to the data generation rule It may include a step.

According to another embodiment, the rule for extracting data may include a selector indicating a location of data in the document.

According to another embodiment, the extracting may include selecting intermediate data from the document based on the selector, classifying the intermediate data, and generating the structured data based on the classification of the intermediate data. It may include a step.

According to another embodiment, the generating of the formal data may include classifying the intermediate data using at least one of a regular expression, a machine learning based classification model, and a text extraction algorithm. have.

According to another embodiment, the obtaining of the document may include obtaining a web document based on a Uniform Resource Locator (URL) of the document providing source, wherein the parameter database may include the extraction parameter and the URL. And storing the extracted parameters, wherein the obtaining of the extracted parameters may include retrieving the extracted parameters based on the URL.

According to another embodiment, the data generation method may further include outputting a rule list including the data generation rule and receiving an input for selecting the data generation rule from the list. The extracting may include extracting the structured data based on the selected data generation rule.

Also, according to another embodiment, the obtaining of the document may include crawling a web document from a preset URL.

Further, according to another embodiment, the step of acquiring the document is characterized in that to obtain a web document providing the vulnerability information, the vulnerability information is about the type of vulnerability, product name, vulnerability identifier and vulnerability date It may be characterized by including the information.

In order to solve the technical problem, a computing device according to an embodiment of the present invention, a crawler for obtaining a document obtained from an external document providing source, a rule for extracting data from the document and extracted data A parameter database for storing data extraction parameters which are setting values for a data generation rule defining a format to be stored; selecting the data extraction parameters based on the document providing source, and selecting the data generation rules based on the data extraction parameters And a processor configured to extract structured structured data from the document according to the data generation rule.

In order to solve the above technical problem, a computer program recorded in a non-transitory computer readable medium according to an embodiment of the present invention is a document that is executed when instructions of the computer program are executed by a processor of a server. Obtaining the document from a provisioning source, obtaining a data extraction parameter corresponding to the document provisioning source from a parameter database, a rule for extracting data from the document based on the data extraction parameter and storing the extracted data And forming a data generation rule defining a format and extracting structured data from the document according to the data generation rule.

1 and 2 illustrate examples of documents provided from different document providing sources.
FIG. 3 is a diagram for describing a structure for generating structured data from documents in which data is represented in different ways.
4 is a diagram illustrating an example of source code for web page analysis.
5 is a diagram for describing a structure of generating, by a computing device, structured data, according to an exemplary embodiment.
6 is a flowchart illustrating a process of generating structured data from a document by a computing device according to an embodiment.
7 is a flowchart illustrating a process of extracting structured data by a computing device according to an embodiment.
8 is a flowchart illustrating a process in which a computing device selects a data generation rule and extracts structured data according to another embodiment.
9 is a conceptual diagram illustrating an example for explaining a rule list and a data generation rule, according to an exemplary embodiment.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention, and methods for achieving them will be apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various forms. The embodiments of the present invention make the posting of the present invention complete and the general knowledge in the technical field to which the present invention belongs. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, the terms defined in the commonly used dictionaries are not ideally or excessively interpreted unless they are specifically defined clearly. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase.

Hereinafter, some embodiments of the present invention will be described with reference to the drawings.

1 and 2 illustrate examples of documents provided from different document providing sources.

1 is an example of a web page including Common Vulnerabilities and Exposures (CVE) information, which is vulnerability information provided by the Natinoal Vulnerability Database (NVD). The CVE information includes all or part of the CVE-ID 110, vulnerability overview 120, CVSS 130, CPE 140, CWE 150, and Reference 160. Vulnerability overview information (120) includes "place where a vulnerability was discovered", "(in) related software product names", "(when) conditions of the vulnerability occurrence", "(allow) attacker type", "(to) results of attack "," (via) means of attack "," (aka) vulnerability title in the reference site "," (a different vulnerability than) other CVE-IDs ", and the like.

2 is an example of vulnerability information provided through another web page 200. According to the example shown in FIG. 2, the vulnerability information 210 includes a vulnerability ID (Bugtraq ID; B-ID), a class of vulnerability, a CVE-ID (CVE), a remote / local information (Remote, Local), Date of publication, including vulnerable product name (Vulnerable). In addition, the webpage 200 may include title 260, discussion 220, exploit 230, solution 240, reference 250 as other vulnerability information. And the like may be further included.

In contrast to FIGS. 1 and 2, in both cases, even if the web page provides the vulnerability information, it can be confirmed that the configuration of the information or the locations where the information is displayed are different from each other. Therefore, when the parser extracting the information displayed after the CVE-ID from the web page shown in FIG. 1 as an identifier for the vulnerability is applied to the web page 200 shown in FIG. You won't be able to extract it.

Therefore, the structured data can be generated through the structure as shown in FIG. FIG. 3 is a diagram for describing a structure for generating structured data from documents in which data is represented in different ways.

The crawler 310 receives the first document 21, the second document 22, and the third document from the first document serving source 11, the second document serving source 12, and the third document serving source 13, respectively. (23) can be collected.

The computing device may generate the first structured data 31 by loading the obtained first document 21 into the first parser 321. In addition, the computing device may generate the second structured data 32 by loading the obtained second document 22 into the second parser 322. In addition, the computing device may generate the third structured data 33 by loading the third document 23 into the third parser 323. Therefore, in this case, parsers must be developed according to the data representation of each document. In addition, if the data representation of each document changes, the corresponding parser must also be re-developed.

4 is a diagram illustrating an example of the source code 400 for web page analysis. The source code 400 shown in FIG. 4 is configured such that the computing device extracts text existing at a predetermined position in the document through doc.select (). However, if the position of the element to extract the value is changed, the source code 400 must be modified. When the source code 400 is modified, the modified source code must be recompiled and applied to the computing device so that the computing device can extract desired data from the document in which the position of the element is changed. In addition, as the number of items of data to be extracted or the structure becomes more complicated, the difficulty of modifying the source code 400 increases.

5 is a diagram for describing a structure of the computing device 500 generating structured data, according to an exemplary embodiment. 5 is only for describing an embodiment, and the structure or the number of components of the computing device 500 may be changed according to an embodiment.

The computing device 500 according to an embodiment may include a crawler 510, a processor 520, and a parameter database 530. The crawler 510 may obtain the documents 21, 22, 23 from the document serving sources 11, 12, 13, respectively. According to an embodiment of the present disclosure, the crawler 510 may acquire the documents 21, 22, and 23 by accessing a predetermined Uniform Resource Locator (URL).

The parameter database 530 may store data extraction parameters. Here, the data extraction parameter means a setting value for a data generation rule that defines a rule for extracting data from a document and a format for storing the extracted data. Here, the data extraction parameter may be stored in match with information (eg, a URL) about a document providing source. That is, the rules for extracting data, the format for storing the extracted data, and information about the document providing source may be stored in the parameter database 530 as one data set.

The processor 520 may select data extraction parameters based on the document providing source. For example, when extracting data from the first document 21, the processor 520 queries the parameter database 530 for the URL for the first document providing source 11, and the first document providing source ( 11) You can select a data extraction parameter consisting of a URL and one data set.

The processor 520 may configure a data generation rule based on the selected data extraction parameter. For example, the processor 520 extracts data existing at any position of the first document 21 based on the data extraction parameter, and stores the extracted data in any item of the structured first structured data 31. You can configure data generation rules that define whether or not. The processor 520 may extract data from the first document 21 according to the configured data generation rule and generate the first structured data 31 structured according to the data generation rule.

As shown in FIG. 5, the parameters are separately stored in the parameter database, and the processor 520 configures a parser suitable for each document by using data generation rules, rather than a parser configured individually for each document. Can be generated. In addition, even when the data presentation method of the document provided from the document providing source is changed, the data extraction parameter stored in the parameter database using the parameter change program, without having to end the operation of the computing device 500 and perform a compilation again. By modifying only the computing device 500, the computing device 500 may extract data according to the changed data representation.

6 is a flowchart illustrating a process of generating structured data from a document by a computing device according to an embodiment.

First, the computing device may obtain a document from a document providing source (S610). Here, the document providing source may be a device external to the computing device. For example, the computing device may obtain a document by crawling a preset URL or by downloading a web document file.

Thereafter, the computing device may obtain a data extraction parameter corresponding to the obtained document providing source from the parameter database (S620). Here, the data extraction parameter may be stored by matching the URL of the document providing source with the data extraction parameter. Accordingly, the computing device may query and obtain the data extraction parameters matched thereto from the parameter database using the URL of the document providing source obtained from the document.

Thereafter, the computing device may configure a data generation rule that extracts data using the obtained parameters and defines a setting value for storing the extracted data in a predetermined format (S630). The computing device may extract the structured data by extracting the data according to the configured data generation rule and storing the extracted data in a preset item (S640).

7 is a flowchart illustrating a process of extracting structured data by a computing device according to an embodiment.

According to an embodiment of the present disclosure, the rule for extracting data may include a selector (eg, a CSS selector) indicating a position of data to be selected in the document. The computing device may first select intermediate data using the selector (S710).

Thereafter, the computing device may select data to be included in the structured data by classifying the intermediate data (S720). In operation S720, the computing device may classify the intermediate data by applying one or more of a regular expression, a classification model generated based on machine learning, and a text classification algorithm to the intermediate data.

Thereafter, the computing device may generate the structured data by storing the data in an item defined in the data generation rule according to the classification result of the intermediate data (S730). As shown in FIG. 7, by first selecting intermediate data using a selector and then classifying the intermediate data, the load generated by the computing device to classify all data included in the entire document may be reduced.

8 is a flowchart illustrating a process in which a computing device selects a data generation rule and extracts structured data according to another embodiment.

According to another embodiment, the computing device may configure a rule list based on the data extraction parameters stored in the parameter database and output the configured rule list (S810). The rule list may be a set of data generation rules configured using data extraction parameters. For example, as illustrated in FIG. 9, the computing device may output a rule list 900 including data generation rules through a display device. Referring to FIG. 9, each item 910 of the rule list 900 may represent one data set in which information 911 about the document providing source and data generation rule are matched. Here, the data generation rule may include a data extraction rule 912 and a data storage format 913.

Referring back to FIG. 8, the computing device may select a data generation rule (S820). According to an embodiment of the present disclosure, the computing device may receive an input for selecting a data generation rule from a rule list. That is, a data generation rule for extracting data from the rule list by the user and composing the structured data may be selected. According to another embodiment, the computing device may select a data generation rule based on the information of the information providing source that obtained the document.

Thereafter, the computing device may extract structured structured data from the document according to the selected data generation rule (S830).

The methods according to the embodiments of the present invention described so far may be performed by execution of a computer program implemented in computer readable code. The computer program may be transmitted to and installed on the second computing device from the first computing device via a network such as the Internet, and thus may be used in the second computing device. The first computing device and the second computing device include both a server device, a physical server belonging to a server pool for cloud services, and a stationary computing device such as a desktop PC.

The computer program may be stored in a recording medium such as a DVD-ROM or a flash memory device.

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it is to be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims (10)

A method for a computing device to generate structured data from a document, the method comprising:
Obtaining a plurality of documents formed in different structures according to different data presentation methods from a plurality of document providing sources;
Obtaining data extraction parameters corresponding to the plurality of document providing sources, respectively, from a parameter database;
Constructing data generation rules respectively defining a rule for extracting data from the document and a format for storing the extracted data based on the data extraction parameters respectively corresponding to the plurality of document providing sources; And
And extracting structured structured data having the same structure from the plurality of documents in the same data representation manner according to the configured data generation rules.
Each step of configuring the data generation rule,
Changing the data generation rule by modifying the data extraction parameter to correspond to a format of the plurality of documents by using a parameter change program when the plurality of documents have different structures;
Extracting the structured data,
And constructing a parser suitable for each of the plurality of documents formed in different structures using the data generation rule changed by the modified data extraction parameter.
How to generate data. The method of claim 1,
The rule for extracting the data is
And a selector indicative of the location of the data within the document.
How to generate data. The method of claim 2,
The extracting step,
Selecting intermediate data to be included in the structured structured data from the document based on the selector; And
And classifying the intermediate data according to the data generation rule and generating the structured data based on the classification of the intermediate data.
How to generate data. The method of claim 3,
Generating the structured data,
Characterized in that to classify the intermediate data using at least one of a regular expression, a machine learning based classification model, and a text extraction algorithm.
How to generate data. The method of claim 1,
Acquiring the document,
Obtaining a web document based on the Uniform Resource Locator (URL) of the document providing source,
The parameter database,
And matching and storing the extraction parameter and the URL,
Acquiring the extraction parameter,
Retrieving the extraction parameter based on the URL;
How to generate data. The method of claim 1,
The data generation method,
Outputting a rule list including the data generation rule; And
Receiving an input for selecting the data generation rule from the list;
Extracting the data,
And extracting the structured data based on the selected data generation rule.
How to generate data. The method of claim 1,
Acquiring the document,
Crawling a web document from a preset URL,
How to generate data. The method of claim 1,
Acquiring the document,
Obtaining a web document providing vulnerability information,
The vulnerability information,
Characterized by the type of vulnerability, product name of the vulnerability, the identifier of the vulnerability and the date the vulnerability occurred, characterized in that
How to generate data. In a computing device,
A crawler for obtaining a plurality of documents formed in different structures according to different data representation methods obtained from a plurality of external document providing sources;
A parameter database for storing a data extraction parameter, which is a setting value for a data generation rule for defining a rule for extracting data from the document and a format for storing the extracted data, for the plurality of documents; And
Respectively selecting the data extraction parameters based on the plurality of document providing sources, respectively configuring the data generation rules based on the respective selected data extraction parameters, and applying the same from the plurality of documents in accordance with the respective configured data generation rules. A processor configured to extract structured structured data having the same structure according to the data representation method;
The processor,
When the plurality of documents are formed in different structures, the data extraction parameter is modified by modifying the data extraction parameters so as to correspond to the format of the plurality of documents using a parameter change program, and by the modified data extraction parameters. By using the changed data generation rules to configure a parser suitable for each of the plurality of documents formed in different structures,
Computing device. A computer program recorded on a non-transitory computer readable medium, where instructions of the computer program are executed by a processor of a server,
Obtaining a plurality of documents formed in different structures according to different data presentation methods from a plurality of document providing sources;
Obtaining data extraction parameters respectively corresponding to the plurality of document providing sources from a parameter database;
Constructing data generation rules for defining a rule for extracting data from the document and a format for storing the extracted data for the plurality of documents based on the obtained data extraction parameters, respectively; And
And extracting the structured structured data having the same structure from the plurality of documents according to the same data expression scheme according to each of the configured data generation rules.
Each step of configuring the data generation rule,
Changing the data generation rule by modifying the data extraction parameter to correspond to a format of the plurality of documents by using a parameter change program when the plurality of documents have different structures;
Extracting the structured data,
And constructing a parser suitable for each of the plurality of documents formed in different structures using the data generation rule changed by the modified data extraction parameter.
Computer programs.

Images