KR101920683B1 - Method and Apparatus for Gathering Data Based on One Class - Google Patents

Abstract

In the present exemplary embodiment, the keyword included in the extracted document data is pre-processed as a vector, the preprocessed terms are learned based on the boundary parameter, and a single class that detects the boundary separating the labeling data area from the non- By extracting the valid data related to the keyword from the document data using the classification model, it is possible to collect a large amount of highly relevant data in relation to the keyword rather than the result of the search using the index word. A data collecting apparatus and method capable of extracting meaningful text data even with text data are provided.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a data-

The technical field to which this embodiment pertains is an apparatus and method for collecting text data.

The contents described in this section merely provide background information on the present embodiment and do not constitute the prior art.

When new technologies are developed in the technical field, authors may use similar terms for the same subject, and the same terms may be used differently for different authors. In order to solve the problem that the search results vary according to the terms included in the document, these terms are used to define specific concepts and terms in a consistent manner.

The data extracted from the retrieval based on the index words of the control word has a high reliability. However, since index-based data retrieval is performed manually by matching index words for each document, there is a problem that the number of retrieval results varies depending on the retrieval point. In addition, valid data exists among data that does not match the index word, but such data is missing in the process of searching with an index word.

In embodiments of the present invention, a keyword is searched and a term included in the extracted document data is preprocessed with a vector, a preprocessed term is learned based on a boundary parameter, and a boundary separating a labeling data area from an unlabeled data area is detected By extracting the valid data related to the keywords from the literature data using the single class classification model, it is possible to collect a large amount of highly relevant data in relation to the keyword, The main purpose of the invention is to extract meaningful text data even for corresponding text data.

Other and further objects, which are not to be described, may be further considered within the scope of the following detailed description and easily deduced from the effects thereof.

According to an aspect of the present invention, there is provided a method of collecting data by a computing device, the method comprising: extracting document data from a database by keyword, parsing the extracted document data, Generating a single class classification model that detects a boundary separating a labeling data area from a non-labeling data area; and extracting a labeling data area from the non- And extracting valid data related to the keyword from the document data using the single class classification model.

According to another aspect of the present invention, there is provided a document retrieval system including a document extracting unit that retrieves document data from a database and extracts document data, a document parsing unit that parses the extracted document data, A classifier model generation unit that learns the preprocessed terms based on boundary parameters and generates a single class classification model that detects a boundary separating a labeling data area from an unlabeled data area; And a valid data extracting unit for extracting valid data related to the keyword from the document data using the single class classification model.

According to yet another aspect of this embodiment, there is provided a computer program for data collection recorded in a non-transitory computer readable medium comprising computer program instructions executable by a processor, A step of searching for a keyword in a database and extracting document data, parsing the extracted document data, expressing the term as a vector on the basis of importance of terms contained in the document data, Generating a single class classification model that learns the pre-processed terms based on a boundary parameter and detects a boundary separating a labeling data region from an unlabeled data region; and generating a single class classification model using the single class classification model, remind And extracting valid data related to the keyword from the document data.

As described above, according to the embodiments of the present invention, words included in document data extracted from a keyword are pre-processed by a vector, pre-processed terms are learned based on boundary parameters, and labeling data By extracting the valid data related to the keywords from the literature data by using the single class classification model that detects the boundaries separating the regions, it is possible to collect a large amount of highly relevant data in relation to the keywords, , It is possible to extract text data meaningful even for text data corresponding to not only a specific category but also other categories.

Even if the effects are not expressly mentioned here, the effects described in the following specification which are expected by the technical characteristics of the present invention and their potential effects are handled as described in the specification of the present invention.

1 is a diagram illustrating data types processed by a data collection device according to an embodiment of the present invention.
2 and 3 are block diagrams illustrating a data acquisition device in accordance with an embodiment of the present invention.
4 and 5 are flowcharts illustrating a data collection method according to another embodiment of the present invention.
Figure 6 shows simulation results performed in accordance with embodiments of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Will be described in detail with reference to exemplary drawings.

1 is a diagram illustrating data types processed by a data collection device according to an embodiment of the present invention.

A text document is collected through a term search. For example, biomedical text documents can be collected through PubMed's MeSH term search. PubMed corresponds to a search engine and has access to the MEDLINE database. The MeSH term is an index and is administered by the National Library of Medicine (NLM). There are two major problems with MeSH terms. First, it takes a considerable amount of time for related experts to assign MeSH terms to the document. Next, there is a problem of missing valuable documents. That is, the document does not contain a specific MeSH term, but the document may contain valuable information about a particular MeSH term. Therefore, in the document collection in the field of text mining, there is a need for a data collection apparatus that does not omit relevant documents.

Referring to FIG. 1, data is divided into keyword dependent data 110 and keyword non-dependent data 120. A keyword is a term for reading a document from a specific database. The keyword dependent data 110 is a search result by a keyword, and the keyword non-dependent data 120 represents negative data related to a keyword. The keyword dependent data 110 is divided into indexer assignment data 130 and indexer unassigned data 140. The indexer assignment data 130 represents a document having an index word. The index unassigned data 140 represents a document for which the index word has not yet been matched. For example, an index term may be MeSH terms. The indexer assignment data 130 is divided into index keyword data 150 and index word non-keyword data 160. Index keyword data 150 represents a document having at least one index word associated with a keyword. The index word non-keyword data 160 indicates a document in which all of the index words included in the document are not related to the keyword. The keyword-related data 170 and the keyword-related data 180 represent data that are associated with or not related to the keyword, respectively.

The present embodiment can collect not only the index keyword data 150 but also the index keyword non-keyword data 160 and the index word unassignment data 140 as well as the index word keyword data 150 as the positive data. The related data 170 can be collected.

2 and 3 are block diagrams illustrating a data acquisition device in accordance with an embodiment of the present invention. 2, the data collecting apparatus 200 includes a document extracting unit 210, a document parsing unit 220, a term preprocessing unit 230, a classification model generating unit 240, and a valid data extracting unit 250). The data collection device 200 may omit some of the various components illustrated in FIG. 2 or may additionally include other components. For example, the data collecting apparatus 300 may further include an relevance evaluating unit 360. [

The document extracting units 210 and 310 retrieve document data from the database and extract document data.

The document parsing unit (220, 320) parses the extracted document data. The document parsing units 220 and 320 can classify document data into identifiers, document types, titles, and summaries.

The term preprocessing unit 230 or 330 preprocesses the term by expressing the term as a vector based on the importance of terms contained in the document data. For example, the term pre-processing units 230 and 330 may pre-process terms using TF-IDF Vectorization or ParagraphVector. TF-IDF is a weighted, statistical measure of how important a word is in a particular document in a document set of documents. TF (Term Frequency) is a value indicating how often a specific word appears in a document, and DF (Document Frequency) is a document frequency. TF-IDF is expressed by Equation (1).

With respect to the term t in document j, tf _{i, j} is the frequency of occurrence of the terms in the document, df _i is the number of documents containing the specific term, and N is the total number of documents.

ParagraphVector, unlike TF-IDF, groups vectors about synonyms. That is, semantic information of a word is provided. ParagraphVector differs from TF-IDF in that it takes into account word order and context.

The classification model generators 240 and 340 learn a pre-processed term based on a boundary parameter and generate a single class classification model that detects a boundary separating the labeling data area from the non-labeling data area. For example, the classification model generators 240 and 340 generate a single class classification model using a single class support vector machine (OCSVM).

OCSVM is expressed by Equation (2).

는 슬랙(Slack) 변수이고,

는 바이어스(Bias) 용어를 나타낸다. 즉,

는 분류오류(Misclassification)의 정도를 나타낸다.

는 아웃라이어(Outlier)의 분수(Fraction)의 상한을 조절하는 파라미터이다.

가 클수록 작고 엄격한 바운더리를 생성한다.In the equation, x _i is a data point, w is a normal vector of a hyperplane, n is the number of data points,

Is a slack variable,

Represents the Bias term. In other words,

Indicates the degree of misclassification.

Is a parameter for adjusting the upper limit of the fraction of the outlier.

The larger the size, the smaller and more rigid boundaries are created.

In this embodiment, the single class classification model learns positive data without learning negative data, and positive data includes (i) a document having an index word related to a keyword, (ii) a document having no index word related to a keyword, (ii) ) At least one of the documents having no index word can be targeted. An indexer forms a hierarchical relationship of terms into a tree structure. Associative terms correspond to abbreviations, singular forms, synonyms, or synonyms that have not been adopted as indexes. Cross-referenced between associative terms and index terms, and may be retrieved by replacing the corresponding index terms in the database with the corresponding index terms.

In this embodiment, the single class classification model can be expressed as Equation (3).

Unlike Equation (2), the parameter is represented by 1-RF in Equation (3). Here, a plurality of boundaries can be set by adjusting the RF (Relevance Factor), which is called RR (Relevance Rule). The boundary parameter adjusts the upper bound of the boundary of the single class classification model, and the boundary parameter is set to the first boundary parameter or the second boundary parameter. The first boundary parameter may be set to a value larger than the second boundary parameter.

The boundary parameter is set to the first boundary parameter when the title of the document data includes the keyword, and the boundary parameter can be set to the second boundary parameter if the title of the document data does not include the keyword. For example, the RF may be set to 0.9 if the title of the document includes a keyword, and the RF may be set to 0.3 if the title of the document does not include a keyword. It is to be understood that the present invention is not limited thereto and that appropriate values may be used depending on the design to be implemented.

The valid data extracting unit (250, 350) extracts valid data related to the keyword from the document data using a single class classification model.

The relevance evaluating unit 260 evaluates relevance between keywords and document data. The relevance evaluation unit 260 calculates the precision based on the positive data and the negative data to evaluate the relevance.

The relevance can be expressed as Equation (4).

Here, pr is a function for calculating the precision. Since this embodiment collects additional data, Precision is an important evaluation factor rather than a recall. Referring again to FIG. 1, s ₁ is input with the index keyword data 150 as positive data, and the index keyword non-keyword data 160 as negative data. s ₂ inputs the keyword keyword data 150 as positive data and the keyword non-dependent data 120 as negative data. The classification of the keyword data 150 and the keyword data 160 is highly reliable but not perfect. Since the index word non-keyword data 160 contains valuable information. The C-score is the harmonic mean of s ₁ and s ₁ .

In order to evaluate extraction of the keyword-related data 170 from the non-keyword non-keyword data 160, a frequency based approach is used. For example, it is determined whether the document includes a gene associated with a specific keyword.

Although the components included in the data collecting device are shown separately in FIGS. 2 and 3, a plurality of components may be combined with each other to form at least one module. The components are connected to a communication path connecting a software module or a hardware module inside the device and operate organically with each other. These components communicate using one or more communication buses or signal lines.

The data collection device may be implemented in logic circuitry by hardware, firmware, software, or a combination thereof, and may be implemented using a general purpose or special purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. Further, the device may be implemented as a System on Chip (SoC) including one or more processors and controllers.

The data collection device may be implemented as software, hardware, or a combination thereof on a computing device or server with hardware components. The computing device or server may be a communication device such as a communication modem for performing communication with various devices or wired / wireless communication networks, a memory for storing data for executing a program, a microprocessor for executing and calculating a program, May refer to a variety of devices including.

4 and 5 are flowcharts illustrating a data collection method according to another embodiment of the present invention. The data collection method may be performed by a computing device and operates in the same manner as the data collection device.

In step S410, the computing device searches the database for keywords and extracts literature data.

In step S420, the computing device parses the extracted document data. The step of parsing the document data can be divided into an identifier, a document type, a title, and a summary.

In step S430, the computing device preprocesses the term by expressing the term as a vector based on the importance of terms contained in the document data.

In step S440, the computing device learns the preprocessed terms based on the boundary parameters and creates a single class classification model that detects boundaries separating the labeling data areas from the unlabeled data areas.

The single class classification model learns positive data without learning negative data, and positive data includes (i) a document having an index word related to a keyword, (ii) a document having no index word related to a keyword, (ii) Or at least one of the documents that do not exist.

The indexer forms a hierarchical relationship of terms in a tree structure, and the association word corresponds to an abbreviated word, a singular plural form, a similar word, or a synonym not adopted as an index word, is cross-referenced between an association word and an index word, .

The boundary parameter adjusts the upper bound of the boundary of the single class classification model, the boundary parameter is set to the first boundary parameter or the second boundary parameter, and the first boundary parameter may be set to a value larger than the second boundary parameter.

The boundary parameter is set to the first boundary parameter when the title of the document data includes the keyword, and the boundary parameter can be set to the second boundary parameter if the title of the document data does not include the keyword.

In step S450, the computing device uses the single class classification model to extract valid data associated with the keyword from the literature data.

Referring to FIG. 5, the data collection method further includes a step of evaluating the relevance between the keyword and the document data. In step S560, the computing device calculates the precision based on the positive data and the negative data to evaluate the relevance.

Figure 6 shows simulation results performed in accordance with embodiments of the present invention. Referring to FIG. 6, it can be seen that the OCSVM to which the TF-IDF and the RR are applied is superior to the ParagraphVector and the OCSVM to which the RR is applied.

4 and 5 illustrate the sequential execution of the respective steps. However, those skilled in the art will appreciate that those skilled in the art will appreciate that various modifications, additions and substitutions are possible without departing from the essential characteristics of the embodiments of the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

The operations according to the present embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. A computer-readable medium represents any medium that participates in providing instructions to a processor for execution. The computer readable medium may include program instructions, data files, data structures, or a combination thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. The computer program may be distributed and distributed on a networked computer system so that computer readable code may be stored and executed in a distributed manner. Functional programs, codes, and code segments for implementing the present embodiment may be easily deduced by programmers of the technical field to which the present embodiment belongs.

The present embodiments are for explaining the technical idea of the present embodiment, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be construed according to the following claims, and all technical ideas within the scope of equivalents thereof should be construed as being included in the scope of the present invention.

100, 200: data collecting apparatus 110, 210: document extracting unit
120, 220: document parsing unit 130, 230: term preprocessing unit
140, 240: Classification model generation unit 150, 250: Valid data extraction unit
260: relevance evaluation unit

Claims (15)

A method of data collection by a computing device,
Searching the database for keywords and extracting document data;
Parsing the extracted document data;
Expressing the term by a vector on the basis of importance of terms included in the document data and performing preprocessing;
Generating a single class classification model that learns the pre-processed terms based on a boundary parameter and detects a boundary separating a labeling data region from an unlabeled data region; And
Extracting valid data related to the keyword from the document data using the single class classification model
/ RTI > The method according to claim 1,
Wherein the step of parsing the document data comprises classifying the document data into an identifier, a document type, a title, and a summary. The method according to claim 1,
Wherein the positive classifier classifies positive data belonging to a set of correct answers without learning negative data belonging to an incorrect set to classify the single class classification model, the positive data being (i) a document having an index word containing the meaning of the keyword, ii) a document that does not have an index word containing the meaning of the keyword, and (ii) a document that does not have the index word,
Wherein the keyword is at least one term included in the text content of the document data and the index word is a separate control word matched with the document data for the purpose of document search, Features that can be used to extract data, respectively Lt; / RTI > The method of claim 3,
Wherein the indexer forms a hierarchical relationship of the terms in a tree structure and the associative word corresponds to an abbreviated word, a singular plural form, a similar word, or a synonym not adopted as the index word, and is cross-referenced between the associative word and the index word, Is replaced with the index word corresponding to the keyword. The method according to claim 1,
Wherein the boundary parameter adjusts an upper limit of a boundary of the single class classification model, the boundary parameter is set to a first boundary parameter or a second boundary parameter, and the first boundary parameter is set to a value larger than the second boundary parameter Wherein the data collection method comprises the steps of: 6. The method of claim 5,
Wherein the boundary parameter is set to the first boundary parameter if the keyword is included in the title of the document data and the boundary parameter is set to the second boundary parameter if the title of the document data does not include a keyword. Lt; / RTI > The method according to claim 1,
Further comprising the step of evaluating the degree of relevance between the keyword and the document data and calculating the precision based on the positive data belonging to the correct answer set and the negative data belonging to the wrong answer set, Collection method. A document extracting unit for searching a database for keywords and extracting document data;
A document parsing unit parsing the extracted document data;
A preprocessing unit for preprocessing the term by expressing the term as a vector on the basis of importance of terms included in the document data;
A classification model generation unit that learns the preprocessed terms based on a boundary parameter and generates a single class classification model that detects a boundary separating a labeling data area from a non-labeling data area; And
Extracting valid data related to the keyword from the document data using the single class classification model,
Lt; / RTI > 9. The method of claim 8,
Wherein the document parsing unit classifies the document data into an identifier, a document type, a title, and a summary. 9. The method of claim 8,
Wherein the positive classifier classifies positive data belonging to a set of correct answers without learning negative data belonging to an incorrect set to classify the single class classification model, the positive data being (i) a document having an index word containing the meaning of the keyword, ii) a document that does not have an index word containing the meaning of the keyword, and (ii) a document that does not have the index word,
Wherein the keyword is at least one term included in the text content of the document data and the index word is a separate control word matched with the document data for the purpose of document search, Wherein the data collection device is capable of being used for extracting data, respectively. 11. The method of claim 10,
Wherein the indexer forms a hierarchical relationship of the terms in a tree structure and the associative word corresponds to an abbreviated word, a singular plural form, a similar word, or a synonym not adopted as the index word, and is cross-referenced between the associative word and the index word, Is replaced with the index word corresponding to the keyword. 9. The method of claim 8,
Wherein the boundary parameter adjusts an upper limit of a boundary of the single class classification model, the boundary parameter is set to a first boundary parameter or a second boundary parameter, and the first boundary parameter is set to a value larger than the second boundary parameter Wherein the data acquisition device is a data acquisition device. 13. The method of claim 12,
Wherein the boundary parameter is set to the first boundary parameter if the keyword is included in the title of the document data and the boundary parameter is set to the second boundary parameter if the title of the document data does not include a keyword. Lt; / RTI > 9. The method of claim 8,
Wherein the relevancy evaluating unit further comprises an relevancy evaluating unit for evaluating the relevance between the keyword and the document data, wherein the relevancy evaluating unit calculates the precision based on the positive data belonging to the correct answer set and the negative data belonging to the wrong answer set, And evaluating the data. A computer program for data collection recorded on a non-transitory computer readable medium comprising computer program instructions executable by a processor, the computer program instructions being executable by a processor of a computing device,
Searching the database for keywords and extracting document data;
Parsing the extracted document data;
Expressing the term by a vector on the basis of importance of terms included in the document data and performing preprocessing;
Generating a single class classification model that learns the pre-processed terms based on a boundary parameter and detects a boundary separating a labeling data region from an unlabeled data region; And
Extracting valid data related to the keyword from the document data using the single class classification model
A computer program for performing operations comprising:

Images