KR20190062848A - System of big data mining using incremental learning and a method thereof - Google Patents

Abstract

The present invention relates to a big data mining system using an incremental learning model, and to a method thereof. More specifically, the big data mining system distinguishes and processes typical data and atypical data, generates a unit classifier without depending on an existing feature reduction technique, uses a mass storage classifier generated in accordance with dynamic combination to freely learn a large amount of documents, processes the data in real time or in semi-real time by additionally reflecting only changing factors when additionally changing partial features, loads the data into distributed parallel frameworks by using modular algorithms, and provides an interface in consideration of the user convenience.

Description

TECHNICAL FIELD [0001] The present invention relates to a large data mining system using progressive learning models,

The present invention relates to a big data mining system and a method thereof using a progressive learning model, and more particularly, to a large data mining system using a progressive learning model, and more particularly to a method and apparatus for classifying and processing fixed data and unstructured data, It is possible to learn real time / semi real time processing by learning a large amount of documents freely by using the large capacity classifier generated and reflecting only the change elements when changing the partial characteristic addition, and it is possible to load in the distributed parallel framework by using the modularization algorithm And a big data mining system using a progressive learning model that provides an interface considering user's convenience and a method thereof.

With the expansion of new types of multimedia content, social network services (SNS), and the dissemination and use of smart devices, the size of data generated and distributed on the web is increasing exponentially. A tremendous amount of data that exists and is still growing on the Web can be used to interpret the world. This is the Big Data. Big data means a vast amount of information digitized. It can filter out unnecessary data from big data and extract and analyze only useful information to read people's thoughts, opinions and trends and to predict their behavior in advance. Big Data is one of the next generation information technology (IT) technology that is not only in Korea but also in the world because of its usefulness.

The domestic big data market is expected to reach 300 billion won in 2015 and grow to 1 trillion won in 2020. The domestic market related to Big Data is growing by more than 28% every year.

With the advent of the Big Data era, there is a growing demand for both quantitative and qualitative data analysis for time series analysis.

Korean Patent Laid-open Publication No. 10-2016-0075971 discloses a public-private large data collector module for collecting civil data provided from various sources in a public institution in real time on a web portal, SNS, intranet in a public institution, (HDFS), and real-time distributed parallel processing through the MapReduce framework to store and manage the data in a relational database. The real-time data management module stores real-time data mining And a big data management and system for a public service, including a public data large data analysis and visualization processor module that predicts classification, grouping, and civil movements.

The above-mentioned patent document is a conventional mining machine, and it is difficult to process big data produced by various digital media and sensors, and there is an inefficiency of re-analyzing the entire data when accumulating new data.

In order to perform data mining in an environment in which a large amount of data is frequently added as described above, there is a need for a technique of learning only incremental data and an incremental learning technique without repeating the entire data.

Korean Patent Publication [10-2016-0075971 (Publication date: June 30, 2016)

Mengle, S.S.R. and Goharian, N. 2009. "Ambiguity measure feature-selection algorithm." Journal of The American Society for Information Science and Technology. 60 (5): 1037-1050. Ko, Y., and J. Seo. 2004. "Using the feature projection technique based on a normalized voting method for text classification." Information Processing and Management. 40 (2): 191-208.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a method and apparatus for classifying and processing fixed data and unstructured data, It is possible to learn real time / semi real time processing by learning a large amount of documents freely by using the large capacity classifier generated and reflecting only the change elements when changing the partial characteristic addition, and it is possible to load in the distributed parallel framework by using the modularization algorithm , And a big data mining system using a gradual learning model that provides an interface considering user's convenience and a method therefor.

The objects of the embodiments of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description .

According to an aspect of the present invention, there is provided a big data mining system using a progressive learning model, comprising: a data input unit 101 for receiving fixed data and unstructured data; A data storage unit (database) 102 for storing the input data; A data preprocessing unit 103 for preprocessing and language processing the input data; A mass classifier (104) for processing data based on an incremental learning model that additionally reflects only partial change elements in input large data; A multiprocessing parallel processing unit 105 for distributing the tasks of the large capacity classifier; A performance measuring unit (106) for measuring the performance of the mass spectrometer; And an interface providing unit 107 for providing a web-based integrated interface of the progressive learning model-based big data mining system.

According to another aspect of the present invention, there is provided a big data mining method using a progressive learning model, the method comprising: inputting formatted data and unstructured data; A data storing step of storing the input data; A data preprocessing step for pre-processing and language processing of the input data; A large classification step of processing data based on an incremental learning model that additionally reflects only partial change elements in input large data; A multiprocessing parallel processing step of distributing the tasks of the large capacity classification step; A performance measurement step of measuring an operation performance of the large capacity classification step; Based interface of the Big Data Mining System based on the progressive learning model.

According to an embodiment of the present invention, there is provided a computer-readable recording medium storing a program for implementing a big data mining method using the progressive learning model.

According to an embodiment of the present invention, a program stored in a computer-readable recording medium is provided to implement a big data mining method using the progressive learning model.

According to the big data mining system and the method using the progressive learning model according to an embodiment of the present invention, it is possible to use a commercial-grade gradual learning model capable of processing large data that is rapidly growing, as well as atypical data, Since it is possible to classify sensor data, almost all types of data processing are possible, and it is expected that the effect will be significant in the whole industry.

According to the big data mining system and the method using the progressive learning model according to an embodiment of the present invention, fast and accurate big data analysis can be performed through progressive academic technology, and data storage efficiency can be increased, Matrix (large-scale sparse matrix) can be computed in a PC-class cluster environment, thereby achieving greater efficiency in a very high-performance computing environment.

Further, according to the big data mining system and the method using the progressive learning model according to an embodiment of the present invention, it is possible to broaden the application field of data analysis without discriminating the type of the fixed and unstructured data, and the utility of the system can be increased .

Further, according to the big data mining system and the method using the progressive learning model according to an embodiment of the present invention, it is possible to share data among users based on a web-based integrated data management system for effectively managing mining processes and results By easily reusing existing data processing results, it is possible to improve the recyclability of learning results, an important issue in large data analysis.

According to the big data mining system and the method using the progressive learning model according to an embodiment of the present invention, time-series big data mining technology is implemented for effective analysis of mass-produced data in real time, .

According to the big data mining system and method using the progressive learning model according to the embodiment of the present invention, the user tool that has been researched and developed is modularized into a core classification model, a driving module, and a user interface, Since it can be easily applied to a large-scale database, it can be easily installed in various distributed parallel frameworks and thus can be easily expanded.

Further, according to the big data mining system and method using the progressive learning model according to an embodiment of the present invention, there is an effect that can be applied to software for analyzing and predicting data with respect to future disaster and disaster countermeasures.

1 is a block diagram of a big data mining system using a progressive learning model according to an embodiment of the present invention;
FIG. 2 is an explanatory diagram of a large capacity classifier in a big data mining system using a progressive learning model according to an embodiment of the present invention; FIG.
3 is a configuration diagram of a large capacity classifier according to the present invention;
4 is a diagram of a web-based integrated interface screen in a big data mining system using a progressive learning model according to an embodiment of the present invention.
Figure 5 illustrates a work and resource management interface in Figure 4;
Figure 6 illustrates a mining parallel processing interface in Figure 4;
FIG. 7 illustrates a learning result performance evaluation interface in FIG. 4; FIG.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, .

On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the term " comprises " or " having ", etc. is intended to specify the presence of stated features, integers, steps, operations, elements, parts, or combinations thereof, And does not preclude the presence or addition of one or more other features, integers, integers, steps, operations, elements, components, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be construed as ideal or overly formal in meaning unless explicitly defined in the present application Do not.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings. Prior to this, terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary terms, and the inventor should appropriately interpret the concept of the term appropriately in order to describe its own invention in the best way. The present invention should be construed in accordance with the meaning and concept consistent with the technical idea of the present invention. Further, it is to be understood that, unless otherwise defined, technical terms and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Descriptions of known functions and configurations that may be unnecessarily blurred are omitted. The following drawings are provided by way of example so that those skilled in the art can fully understand the spirit of the present invention. Therefore, the present invention is not limited to the following drawings, but may be embodied in other forms. In addition, like reference numerals designate like elements throughout the specification. It is to be noted that the same elements among the drawings are denoted by the same reference numerals whenever possible.

1 is a block diagram of a big data mining system using a progressive learning model according to an embodiment of the present invention.

1, a big data mining system using a progressive learning model according to an embodiment of the present invention includes a data input unit 101, a data storage unit (DB) 102, a data preprocessing unit 103, a large capacity A classifying unit 104, a multiprocessing parallel processing unit 105, a performance measuring unit 106, and an interface providing unit 107.

The data input unit 101 receives the fixed data and the unstructured data.

The data storage unit (DB) 102 stores data input through the data input unit 101.

The data preprocessing unit 103 performs preprocessing and language processing of the input data.

The mass classifier 104 processes data based on an incremental learning model that additionally reflects only partial change elements in the input large amount of data.

The multiprocessing parallel processing unit 105 distributes the tasks of the large capacity classifier.

The performance measuring unit 106 measures the performance of the large capacity classifier.

The interface providing unit 107 provides a web-based integrated interface of the progressive learning model-based big data mining system. The interface providing unit 107 provides a task and resource management interface, a resource reuse interface, a parallel processing interface, and a learning result performance evaluation interface.

FIG. 2 is an explanatory diagram of a large capacity classifier in a big data mining system using a progressive learning model according to an embodiment of the present invention, and FIG. 3 is a configuration diagram of a large capacity classifier according to the present invention.

2, the large-capacity classifier 104 includes a plurality of unit classifiers 210-1, 210-2, and 210-n, and is generated by combining the plurality of unit classifiers. Each of the unit classifiers 210-1, 210-2 and 210-n includes qualities extraction units 211-1, 211-2 and 211-n for extracting qualities, (213-1, 213-2, and 213-n) for generating characteristic-quality matrices. The feature-quality-matrix extracting units 212-1, 212-2, and 212-

The large-capacity classifier 104 includes a matrix dynamic combining unit 221 for combining information on a plurality of feature-value matrices to be combined, and a matrix-dynamic weighting unit 221 for calculating a subject-weight vector from the integrated feature- Weighted vector generating unit 222 for generating a subject-weighted vector, and a multiple classification unit 223 for classifying the selected subject-weighted vector according to the voting type classification technique.

Hereinafter, a method of generating a large capacity classifier will be described with reference to FIGS. 2 and 3. FIG.

1. Unit classifier generation and dynamic combining method

When you want to apply automatic categorization technology to real services, you may need to learn and interpret millions of information resources in some cases. Generally, the feature selection technique is used for efficient document processing, which is known as a necessary process not only for reducing the amount of information but also for improving the performance. However, in the task of performing a large amount of document learning in a big data environment in which a large amount of data is poured in real time (or semi-real time), it is not free from the time and computing resource problem in analyzing the characteristics of the characteristics, The application of selection and reduction techniques also has its limitations.

The technique used in the progressive learning technique according to the present invention is to generate a large number of small capacity matrices and combine them dynamically without loss of information. FIG. 2 is an example of combining a plurality of small-scale learning results (classifiers) 210-1, 210-2, and 210-n for each database. A plurality of divided unit sorters 210-1, 210-2, and 210-n may be dynamically combined to generate a final large capacity learning model (classifier) 104. [

2. Creation process of unit classifier

In order to generate the unit classifier, a series of processes including the following preprocessing process is performed.

(1) The quality extracting units 211-1, 211-2, and 211-

The following two types can be considered for extracting qualities. When extracting information from titles, abstracts, etc., a feature set is generated through stemming (English) or morphological analysis (Korean). At this time, it is better to consider the process of removing low frequency features. Typically, a single occurrence of collection frequency = 1 in an entire document set takes up about 40-60% of the total.

① Keyword, descriptor

The descriptor field, which is a keyword field or a controlled vocabulary, is used.

② Info Extraction

Extracts key information including noun phrases from unstructured information such as title and abstract.

(2) Qualification information extraction sections 212-1, 212-2 and 212-n for each document,

A category code is assigned to individual qualities constituting a document.

The main creation field includes a document unique ID, a qualification, and a category code.

(3) The feature-property-matrix generating units 213-1, 213-2, and 213-

In the present invention, a core information matrix for generating individual unit classifiers is referred to as a " quality characteristic matrix ". Matrix information for computing the final feature vector is generated and loaded into a database (DB) or a binary file.

The main generated fields include the qualification unique ID, the qualification, the category code, TP, TN, FP, FN, CF, IDF and the like. <Table 1> relates to the feature-divisional appearance relation partition table, and some of the generation fields of the feature characteristic matrix are shown.

Category c _j belongs Category c _j Smile Appearance of qualities f _i TP (True-Positive) FN (False-Negative) No qualities f _i appear FP (False-Positive) TN (True-Negative)

3. Generation of mass classifier by combining unit classifier

The key to generating a large capacity classifier according to the present invention is to perform dynamic combining of a classifier using a method of combining characteristic feature matrices generated in step (3) during a unit classifier generation process. The unit classifier can automatically combine and dynamically combine and reproduce huge matrices if there are many documents to be studied.

(1) Matrix dynamic coupling unit 221

(1) First, a plurality of "target feature matrices" to be combined reside in the memory, and a distinct whole set of the feature values appearing in all the matrices is created.

(2) The information is retrieved by referring to the matrices to be combined with individual qualities. In this case, since the qualities do not appear in all the qualitative characteristics matrices, the main information such as TP, TN, FP, FN, IDF and CF is recalculated by dynamically calculating integrated information of each matrix such as the number of qualities and the total number of documents. This process means that the integrated matrix generation result of combining 10 learned classes of 10 classifiers and the numerical values of individual parameter elements in the classifier matrix obtained once at a time are exactly the same.

(2) a subject-weight vector generator 222,

Using the similarity measures such as distance coefficient, cosine and log odds ratio (LOR) from the integrated feature property matrix, a feature vector form suitable for the final voting classifier is created and loaded into a DB or binary file.

The feature vector using the LogTF * IDF * cosine coefficient can be expressed as Equation (1) below. Further, in the progressive learning model according to the present invention, a multiplication ratio (OR) and a semantic ambiguity resolution (AM) model are additionally applied to assign a quality weight.

&Quot; (1) &quot;

(3) The multiple classification unit 223

The classification is performed using the voting type classification technique as shown in Equation (2) using the feature vector generated in the integrated matrix. Feature Voting Classifier (FVC) is a probabilistic model that shows good classification performance and fast speed. After the generated feature vector resides in the memory, the input document is classified by performing high-speed multi-classification on a large number of input documents.

&Quot; (2) &quot;

Since the final generated classifier has a relatively small amount of data of the final computed vector, there is no restriction on the number of the qualities of the class because the memory resident capacity is small. Since the linear combination of each weight is performed, to be.

4 is a diagram of a web-based integrated interface screen in a big data mining system using a progressive learning model according to an embodiment of the present invention.

The system developed in this study has improved user accessibility through web based interface and can be operated in various environments such as Windows and Linux regardless of operating system. We also implemented an intuitive interface so that users without expert knowledge can easily classify them.

In order to facilitate the progressive learning function in the web browser such as the learning management interface, we developed a task and resource management function according to the user authority. Through the resource management function, users can easily load their own data into the database, and can perform and manage data classification based on the uploaded resources.

On the other hand, the user can create a new job using the uploaded data in the resource management, and visually check the progress of each learning process (intuitive learning). In the present module, learning is performed through a total of four steps. In the first and second steps of data preprocessing, the learning data is divided into a number designated by the user and the work is progressed to a multi-process. In the third step of the data integration process, the user selects and integrates the data completed up to the second step, and then the learning is completed through the four-step operation, which is an actual data learning step.

Users can reuse the results produced by each task with new work (reuse of resources and results). Through the work and resource management, it is possible to carry out the efficient classification experiment by implementing the function that does not perform the same operation again by sharing the user's own data as well as the work results shared by other users. Therefore, the user can gradually learn only the added data without re-learning the whole data continuously generated in the big data environment through the data reuse function.

Finally, the user can test the performance of the learned module automatically. After finishing the learning process (test result analysis), the performance of the learning result can be visually confirmed through the test button. In the test result, the performance index of each class of test data is calculated and output, and the performance evaluation result is visualized by a graph and displayed to the user. In addition, the result output function is implemented to save and manage the performance evaluation result of the job separately, and it is possible to analyze the analysis result by stating the analysis statistical information and the visual graph.

FIG. 5 is a diagram illustrating a task and resource management interface in FIG.

It is divided into work management which contains workflow of classification process and resource management which is used in classification process.

Task management is a parser that parses and registers resource data by parsing raw / unstructured raw data, preprocessing and language processing functions to refine the qualities used in classification using registered resources, merging results produced by other tasks and results of processing , A learning function for parallel processing of merged results in multiple processes, and a test function for classifying and measuring performance using a processed learning model. Resource management functions include sharing raw data, including raw data, so that the data used in the work can be reused. Task management and resource management provide a framework for sharing and collaborating tasks and resources of mutually privileged users.

FIG. 6 is a diagram illustrating a mining parallel processing interface in FIG.

In the big data mining environment according to the present invention, it is possible to perform parallel processing in units of processes for parallel processing of progressive learning. In addition, it is possible to directly select the number of processes to be parallelized in the user interface. In addition, since the progressive learning model can be implemented as an application program itself, it can be easily applied to various distributed parallel NoSql frameworks such as Hadoop and Hbase. FIG. 6 shows a process in which five jobs are multiprocessed.

FIG. 7 is a diagram showing a learning result performance evaluation interface in FIG.

Progressive learning adds to the existing learning outcomes without reworking only the incremental data of the existing learning results, while the general learning method re-learns all data every time data is added and changed. For re-learning, users can freely import previously processed data through resource management during job execution and merge with new data. This brilliant learning efficiency can be a good alternative to overcome data production speed, a key characteristic of big data-driven data processing.

After the learning process, you can simply evaluate the performance and check the results in the test process of the user interface. As shown in FIG. 7, classification statistics and performance indicators for each class of test data are calculated and output, and the performance evaluation results are visualized through graphs and displayed to the user. In addition, the result output function is implemented to save and manage the performance evaluation result of the job separately, and it is possible to analyze the analysis result by stating the analysis statistical information and the visual graph. The calculated learning result is stored in the job management menu, and the usability is ensured so that the user can inquire at any time.

Although the big data mining method using the progressive learning model according to an embodiment of the present invention has been described above, it is possible to use a computer readable recording medium and a progressive learning model in which a program for implementing a big data mining method using the progressive learning model is stored It is needless to say that a program stored in a computer-readable recording medium for implementing a big data mining method using the present invention can also be implemented.

That is, the big data mining method using the above-described progressive learning model can be easily understood by those skilled in the art that a program of instructions for implementing the big data mining method can be tangibly embodied and provided in a recording medium readable by a computer will be. In other words, it can be implemented in the form of a program command that can be executed through various computer means, and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination. The program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention or may be those known and available to those skilled in the computer software. Examples of the computer-readable medium include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs and DVDs, and optical disks such as floppy disks. Magneto-optical media and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, USB memory, and the like. The computer-readable recording medium may be a transmission medium such as a light or metal line, a wave guide, or the like, including a carrier wave for transmitting a signal designating a program command, a data structure, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

101: Data input unit 102: Data storage unit (DB)
103: Data preprocessing unit 104: Large capacity classifier
105: Multiprocessing parallel processing unit 106: Performance measurement unit
107: interface provisioning
210: unit learning model (unit sorter)
220: Large Capacity Learning Model (Mass Classifier)
211-1, 211-2, 211-n:
212-1, 212-2, and 212-n:
213-1, 213-2, and 213-n:
221: Matrix dynamic coupling unit
222: subject-weight vector generation unit
223:

Claims (3)

In a big data mining system using a progressive learning model,
A data input unit 101 for receiving the fixed data and the irregular data;
A data storage unit (database) 102 for storing the input data;
A data preprocessing unit 103 for preprocessing and language processing the input data;
A mass classifier (104) for processing data based on an incremental learning model that additionally reflects only partial change elements in input large data;
A multiprocessing parallel processing unit 105 for distributing the tasks of the large capacity classifier;
A performance measuring unit (106) for measuring the performance of the mass spectrometer; And
The interface providing unit 107 for providing a web-based integrated interface of the progressive learning model-based big data mining system,
Big data mining system using incremental learning model.
The method according to claim 1,
The mass classifier (104)
And includes a plurality of unit classifiers 210-1, 210-2, and 210-n, and is generated through combining the plurality of unit classifiers.
Each of the unit classifiers 210-1, 210-2, and 210-
Qualities extraction units (211-1, 211-2, 211-n) for extracting qualities;
A qualification information extraction unit (212-1, 212-2, 212-n) for each document for extracting qualification information per document;
Qualities characteristic matrix generation sections (213-1, 213-2, 213-n) for generating a qualification characteristic matrix;
/ RTI &gt;
The mass classifier (104)
A matrix dynamic joining unit 221 for integrating and combining the information of a plurality of feature characteristic matrices to be combined;
A subject-weight vector generation unit 222 for generating a subject-weight vector from the integrated feature-property matrix using a similar measure;
A multi-classifying unit 223 for classifying according to the voting type classification scheme using the generated topic-weighted vector,
Wherein the data mining system further comprises:
3. The method of claim 2,
The interface providing unit (107)
A task and resource management interface, a resource reuse interface, a parallel processing interface, and a learning result performance evaluation interface.

Images