KR20200010624A - Big Data Integrated Diagnosis Prediction System Using Machine Learning - Google Patents

Abstract

The present invention relates to a big data integrated diagnostic prediction system using machine learning and a method thereof. According to the present invention, the method comprises the steps of: (A) collecting data corresponding to a keyword set by a collection unit to a connected website or from an IoT device; (B) classifying, by a classification unit, the data to meet a preset value in order to analyze and learn the collected data; (C) generating analysis information by analyzing data stored in a database so as to correspond to analysis performance definition data by an analysis unit; (D) performing machine learning on the collected data or the classified data by a learning unit; and (e) generating, by a predictive diagnosis unit, a report including a predicted value and reliability of the collected data or classified data based on a result of the machine learning.

Description

Big Data Integrated Diagnosis Prediction System Using Machine Learning

The present invention relates to a big data integrated diagnostic prediction system and method using machine learning, and more specifically, to quantify and quantify evaluation items through classification and analysis on various data collected from a website or an IoT device. This study relates to a technique for learning accuracy and weights through deep learning techniques to derive reliability and error together to improve accuracy.

Big data refers to a large set of structured or unstructured data sets that go beyond the ability to collect, store, manage, and analyze data with existing database management tools and techniques for extracting value and analyzing the results from such data.

The development of big data technology, which is characterized by the creation, collection, analysis, and expression of various kinds of large-capacity data, makes it possible to predict and efficiently operate a diversified modern society more efficiently and provide customized information for each individual member of modern society. Allow management and analysis.

As such, big data shows the possibility of providing useful information to society and mankind in all areas such as politics, society, economy, culture, science and technology, and its importance is highlighted.

Big data analysis can involve data mining, machine learning, natural language processing, and pattern recognition, which were used in traditional statistics and computing. In particular, due to the increase in unstructured data, such as social media, text mining, opinion mining, social network analysis, cluster analysis, etc. are attracting attention.

However, in the conventional big data analysis, when the shape or feature of the data is changed, the model must be re-analyzed and designated again to execute the analysis program again.

In addition, the program extracts a value from a predetermined big data storage database and operates in a form in which the program is processed and analyzed.

In addition, the conventional big data analysis tool indexes only the data stored in the database and provides the analysis result. Therefore, it is difficult to check the analysis result during the big data collection, and it is troublesome to index and analyze the data collected from the database after the big data collection is completed. There is this.

Korean Registered Patent No. 1638986

The purpose of the present invention is to quantify evaluation items through classification and analysis of various data collected from websites or IoT devices, and to learn the biases and weights quantified through deep learning techniques to derive reliability and errors to improve accuracy. To improve.

An object of the present invention is to analyze the data to be collected in real time only by changing parameters without changing the logic without re-developing an analysis tool whenever the shape and model of the data to be collected change. Even if the table specification and communication protocol are changed, the analysis program does not need to be executed again, which is flexible, easy to maintain, and reduces the cost of redevelopment.

One embodiment of the present invention for achieving the technical problem is a big data integrated diagnostic prediction system using machine learning, the collection unit for collecting data corresponding to the basic data set from the device; A classification unit configured to store data classified in accordance with a predetermined value for analysis and learning of collected data in a database; An analysis unit for analyzing the data stored in the database to generate analysis information so as to correspond to the preset analysis performance definition data; A learning unit which performs machine learning on collected data or classified data; And a predictive diagnosis unit configured to generate a report including a predicted value of the collected data or classified data and a reliability thereof based on a machine learning result.

The collection unit Legacy DB module for storing the data collected from the website connected to the device in a database for big data analysis; And an IoT DB module for storing data collected from the IoT devices connected to the device in a database for big data analysis.

The basic data may include any one of a data collection method, a collected data storage path, table information, field information, a keyword, a size of collected data, a data collection time, and a collection target site.

The analysis unit includes a coordinator module for classifying to match a predetermined value for analysis and learning of the collected data; And an integrated DB for storing and managing data classified by the coordinator module on a memory basis.

The analysis execution definition data may include any one of an analysis execution method, an analysis execution cycle, an analysis target evaluation item, and a weight, and the analysis information may include data obtained by quantifying the frequency of each keyword.

The forecast value includes a numerical forecast value of any one of sales volume, attractiveness, growth rate, or demand for collected data or classified data, and categorical forecast value of any one of up / down, buy / sell, or crush / unlikeness. It is done.

In the big data integrated diagnosis prediction method using machine learning according to an embodiment of the present invention based on the above-described system, (a) collecting data corresponding to a keyword set to a website connected to a collection unit or from an IoT device; step; (B) classifying and classifying the classification unit so as to correspond to a predetermined value for analyzing and learning the collected data; (C) generating analysis information by analyzing the data stored in the database so as to correspond to the analysis performance definition data by the analysis unit; (D) performing machine learning on the collected data or the classified data by the learning unit; And (e) generating, by the predictive diagnosis unit, a report including predicted values of the collected data or classified data based on the machine learning result and reliability thereof.

Preferably, step (a) comprises: (a-1) determining whether the data collection target included in the basic data set by the collector is a website or an IoT device; (a-2) when the data collection target is a website as a result of the determination in step (a-1), storing the data collected from the website accessed by the collector in a database through a legacy DB module; And (a-3) storing the data collected from the connected IoT device in the database through the IoT DB module when the data collection target is the IoT device as a result of the determination in step (a-1). It features.

Step (b) includes the step of classifying the index set by the coordinator module (b-1); (B-2) classifying the collected data to match the set value indexed by the coordinator module; And (b-3) storing the classified data in the memory-based storage through the integrated DB.

(c) step (c-1) of analyzing, by the analysis unit, analyzing analysis definition data including any one of an analysis execution method, an analysis execution cycle, an analysis target evaluation item, and a weight; (C-2) determining, by the analysis unit, whether the data analysis target is a website or an IoT device; (c-3) analyzing the data stored in the Legacy DB module in accordance with the analysis performance definition data when the data analysis target is the website as a result of the determination in step (c-2); As a result of the determination in step (c-2), when the data analysis target is the IoT device, the analyzing unit analyzes the data stored in the IoT DB module to match the analysis performance definition data (c-4); And (c-5) generating the analysis information obtained by quantifying the frequency of each keyword of the data analyzed by the analysis unit.

In step (d), the learning unit receives a value of each of an input layer, a hidden layer, a learning model, and parameters for machine learning (d-1); (D-2) the learning unit indexing a function defined for performing machine learning; (D-3) the learning unit indexing a data mining technique for performing machine learning; And (d-4) generating the machine learning result information by executing the machine learning through the function and data mining technique indexed by the learning unit.

In step (e), the prediction diagnosis unit receives the machine learning result from the learning unit (e-1); (E-2) receiving the collected data or the classified data by the prediction diagnosis unit; And (e-3) generating, by the predictive diagnostic unit, a report including predicted values and reliability of the collected data or classified data based on the machine learning results.

According to the present invention as described above, the evaluation items are quantified through classification and analysis of various data collected from a website or an IoT device, and the deflections and weights are learned through deep learning techniques to derive reliability and error together. This has the effect of improving accuracy.

According to the present invention, it is possible to analyze the data to be collected in real time only by changing parameters without changing the logic without re-developing an analysis tool whenever the shape and model of the data to be collected are changed. Even if the specifications and communication protocols change, the analysis program does not need to be run again, which is flexible, easy to maintain, and reduces the cost of redevelopment.

1 is a block diagram showing a big data integrated diagnostic prediction system using machine learning in accordance with an embodiment of the present invention.
2 is a diagram showing a detailed configuration of a big data integrated diagnostic prediction system using machine learning according to an embodiment of the present invention.
3 is a flowchart illustrating an operation of a big data integrated diagnosis prediction system using machine learning according to an embodiment of the present invention.
4 is a diagram illustrating a framework of a big data integrated diagnostic prediction system using machine learning according to an embodiment of the present invention.
5 is a flowchart illustrating a method for predicting big data integrated diagnosis using machine learning according to an embodiment of the present invention.
6 is a flowchart illustrating step S100 of a method for predicting integrated big data diagnosis using machine learning according to an embodiment of the present invention.
7 is a flowchart illustrating step S200 of a method for predicting integrated big data diagnosis using machine learning according to an embodiment of the present invention.
8 is a flowchart illustrating step S300 of a method for predicting big data integrated diagnosis using machine learning according to an embodiment of the present invention.
9 is a flowchart illustrating step S400 of a method for predicting integrated big data diagnosis using machine learning according to an embodiment of the present invention.
10 is a flowchart illustrating step S500 of a method for predicting integrated big data diagnosis using machine learning according to an embodiment of the present invention.

Specific features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. Prior to this, terms or words used in the present specification and claims are defined in the technical spirit of the present invention on the basis of the principle that the inventor can appropriately define the concept of the term in order to explain the invention in the best way. It should be interpreted to mean meanings and concepts. In addition, when it is determined that the detailed description of the known function and the configuration related to the present invention may unnecessarily obscure the subject matter of the present invention, it should be noted that the detailed description is omitted.

1 is a block diagram showing a big data integrated diagnostic prediction system (S) using machine learning according to an embodiment of the present invention, Figure 2 is a big data integrated diagnostic using machine learning according to an embodiment of the present invention It is a figure which shows the detailed structure of the prediction system S. FIG.

Hereinafter, detailed description thereof will be omitted, but the big data integrated diagnosis prediction system S using machine learning according to an embodiment of the present invention may be any one of scikit-learn, tensorflow, or keras provided by Python. It is assumed that a framework that can be selected and applied for each module by setting a function and a function is installed.

As shown in Figure 1 and 2, the big data integrated diagnostic prediction system (S) using the machine learning according to an embodiment of the present invention, the collecting unit 100, the classification unit 200, the analysis unit 300 , The learning unit 400 and the predictive diagnosis unit 500 are configured.

First, the collection unit 100 collects data corresponding to the set keyword and the basic data, the Legacy DB module 102 for storing the collected data in a database for big data analysis, and the data collected from the IoT device big It is configured to include an IoT DB module 104 to store in the database for data analysis.

In this case, the basic data may include any one of a data collection method, a collected data storage path, table information, field information, a keyword, a size of collected data, a data collection time, and a collection target site.

In addition, the database for storing the big data may be composed of any one of Kasandra, hadoop or mongoDB, but the present invention is not limited thereto.

In addition, the data collected by the collection unit 100 may include any one of text, voice, or image, and collect comments based on tags and keywords.

The classification unit 200 classifies the coordinator module 202 for classifying and matching data with a predetermined value for analyzing and learning the collected data, and an integrated DB for storing and managing data classified by the coordinator module 202 on a memory basis. And 204.

In this case, the preset value for classifying the collected data may be changed by the administrator as a value for data classification and parameter setting.

According to the classification unit 200 according to an embodiment of the present invention, when the type of data to be collected, a table standard of a communication device, or a communication protocol is changed, the data or model about the data to be collected is changed by only changing parameters without changing logic. Data classification is possible.

The analysis unit 300 selects a database in which the analysis target data is stored for analyzing the collected data, and analyzes the data stored in the database to meet the analysis performance definition data to generate analysis information.

At this time, the database selection is any one of the Legacy DB module 102 or IoT DB module 104 of the collecting unit 100, the original data is stored, and the integrated DB 204 of the classification unit 200 stored the classified data. It is preferable to understand by selecting.

In addition, the analysis execution definition data includes any one of an analysis execution method, an analysis execution cycle, an analysis target evaluation item, or a weight, and the analysis information includes data obtained by quantifying a frequency for each keyword.

The learning unit 400 performs the learning through deep learning so as to correspond to the values of the input layer, the hidden layer, the learning model, and the parameters, respectively, by the data or classification unit 200 collected by the collecting unit 100. Machine learning execution module for defining the functions for performing the learning of the sorted data and machine learning execution module for generating machine learning result information by executing machine learning through the functions defined in the machine learning definition module 402 404.

At this time, the machine learning execution module 404 performs the learning by either data mining technique of supervised learning based on rule definition or unsupervised learning finding a rule. Settings can be changed.

Through the supervised learning of the learning unit 400 according to an embodiment of the present invention, an expert system embedded with a knowledge base and decision rules is supplemented and unsupervised learning. Complement the decision support system to support unstructured decision making.

The predictive diagnosis unit 500 generates a report including predicted values of the collected data or classified data and reliability thereof based on the machine learning result information received from the learner 400.

In this case, the predicted value may include any one of the numerical predictive value among the sales volume, the likelihood, the increase rate, or the demand quantity, and the categorical predictive value among the up / down, buy / sell, or crush / unfavorable values.

In addition, the report generated by the predictive diagnosis unit 500 includes any one of a data flow chart in the form of a time series chart, diagram, or video.

3 is a diagram illustrating an operation flow of a big data integrated diagnosis prediction system S using machine learning according to an embodiment of the present invention.

As shown in FIG. 3, first, the collecting unit 100 receives a front end and a back end framework for performing machine learning (step 1).

Subsequently, the collection unit 100 defines a data collection method, a database, table information, field information, and settings for the integrated database (step 2).

Subsequently, the classification unit 200 defines a method and a parameter for data analysis (step 3).

Subsequently, the analysis unit 300 analyzes the collected information according to a predetermined rule during the designated period (step 4).

Subsequently, the learner 400 defines an input layer, a hidden layer, a learning model, and parameters for machine learning (step 5).

Next, the learning unit 400 performs a machine learning to match the defined value (step 6).

Then, the predictive diagnosis unit 500 generates a report including the predicted value and the reliability thereof based on the learning result (step 7).

At this time, step 1 is executed once at first, and step 2, step 3, and step 5 can be changed by the user if there is a change in the model (suggestion of model change can be changed through automatic recommendation of the machine learning engine. Step 4 and step 6 are automatically executed periodically.

4 is a diagram illustrating a framework of a big data integrated diagnosis prediction system S using machine learning according to an embodiment of the present invention.

As shown in FIG. 4, Scikit-learn library, which is a Python library for machine learning, Numpy for numerical analysis, pandas for numerical analysis and visualization, matplotlib for visualization and seaborn are configured as common libraries from the bottom of the framework. use.

In addition, the backend can be selectively configured as tensorflow or theano according to the user's setting, and the frontend is configured as keras / pytorch.

Collectors for collecting information from various legacy databases and IoT devices were configured, and Oracle and MySQL were deployed for integration purposes, and Hadoop, MonogoDB, Redis, Cassandra, and elastic connnectors, which are existing big data storage tools, were deployed.

Data coordinator and integration DB were configured for data classification and analysis, ML student was in charge of M / L level analysis and learning, and DL student was in charge of D / L level analysis and learning. It was.

WAS automatically generates a report using various web applications and shows it to the user with the Election Viewer, so that information can be provided to various media such as mobile devices, web, and applications.

Hereinafter, a method for predicting big data integrated diagnosis using machine learning according to an embodiment of the present invention will be described with reference to FIG. 5.

First, the collection unit 100 collects data corresponding to a keyword set from the connected website or IoT device (S100).

Subsequently, the classification unit 200 classifies and stores the data in a database in accordance with a predetermined value for analyzing and learning the collected data (S200).

Subsequently, the analysis unit 300 analyzes the data stored in the database to match the analysis performance definition data to generate analysis information (S300).

Next, the learning unit 400 performs a machine learning on the collected data or classified data (S400).

Then, the predictive diagnosis unit 500 generates a report including the predicted value and reliability according to the data collected or classified based on the machine learning result (S500).

Hereinafter, referring to FIG. 6, the S100 step of the big data integrated diagnosis prediction method using machine learning according to an embodiment of the present invention will be described below.

First, the collection unit 100 determines whether the data collection target included in the set basic data is a website or an IoT device (S102).

As a result of the determination in step S102, when the data collection target is a website, the collection unit 100 stores the data collected from the connected website through the legacy DB module 102 in the database (S104).

As a result of the determination of step S102, when the data collection target is an IoT device, the data collected from the connected IoT device by the collector 100 is stored in a database through the IoT DB module 104 (S106).

Hereinafter, referring to FIG. 7, a detailed process of step S200 of the method for predicting big data integrated diagnosis using machine learning according to an embodiment of the present invention will be described below.

After operation S100, the classification unit 200 indexes the value set by the coordinator module 202 (S202). In this case, the set value is a value for data classification and parameter setting and may be changed by an administrator.

Subsequently, the classification unit 200 classifies the collected data to match the set value indexed by the coordinator module 202 (S204).

Then, the classification unit 200 stores the classified data in the memory-based storage through the integrated DB (204) (S206).

Hereinafter, referring to FIG. 8, a detailed process of step S300 of the big data integrated diagnosis prediction method using machine learning according to an embodiment of the present invention will be described below.

After operation S200, the analysis unit 300 indexes analysis execution definition data including any one of an analysis execution method, an analysis execution cycle, an analysis target evaluation item, and a weight (S302).

Next, the analysis unit 300 determines whether the data analysis target is a website or an IoT device (S304).

As a result of the determination in step S304, when the data analysis target is a website, the analysis unit 300 analyzes the data stored in the legacy DB module 102 to match the analysis performance definition data (S306).

As a result of the determination in step S304, when the data analysis target is the IoT device, the analysis unit 300 analyzes the data stored in the IoT DB module 104 to match the analysis performance definition data (S308).

Then, the analysis unit 300 generates analysis information obtained by quantifying the frequency for each of the keywords (S310).

Hereinafter, referring to FIG. 9, a detailed process of step S400 of the big data integrated diagnosis prediction method using machine learning according to an embodiment of the present invention will be described below.

After operation S300, the learner 400 receives values of each of an input layer, a hidden layer, a learning model, and parameters for machine learning (S402).

Next, the learning unit 400 indexes the function defined for performing the machine learning (S404).

Subsequently, the learning unit 400 indexes a data mining technique for performing machine learning (S406).

Then, the learning unit 400 executes the machine learning through the function and the data mining technique indexed to generate the machine learning result information (S408).

Hereinafter, the detailed process of step S500 of the big data integrated diagnostic prediction method using machine learning according to an embodiment of the present invention will be described with reference to FIG. 10.

After operation S400, the predictive diagnosis unit 500 receives a machine learning result from the learner 400 (S502).

Subsequently, the predicted diagnosis unit 500 receives the collected data or classified data (S504).

In operation S506, the predictive diagnosis unit 500 generates a report including predicted values of the collected data or the classified data based on the machine learning result and the reliability thereof.

As described above, according to one embodiment of the present invention, the analysis tool can be analyzed in real time by changing parameters without changing logic without re-developing an analysis tool whenever the shape and model of the data to be collected are changed. Even if the source database, the table specifications and communication protocols of the communication equipment change, the analysis program does not need to be run again, which is flexible, easy to maintain, and reduces the cost of redevelopment.

As described above and described with reference to a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as such, it is a deviation from the scope of the technical idea It will be understood by those skilled in the art that many changes and modifications can be made to the invention without departing from the scope of the invention. Accordingly, all such suitable changes, modifications, and equivalents should be considered to be within the scope of the present invention.

S: Big Data Integrated Diagnosis Prediction System Using Machine Learning
100: collector
102: Legacy DB Module
104: IoT DB module
200: classification
202: coordinator module
204: Integrated DB
300: analysis unit
400: learning unit
402: machine learning definition module
404: machine learning execution module
500: predictive diagnostic unit

Claims (12)

A collector configured to collect data corresponding to the basic data set from the device;
A classification unit configured to store data classified in accordance with a predetermined value for analysis and learning of collected data in a database;
An analysis unit for analyzing the data stored in the database to generate analysis information so as to correspond to preset analysis performance definition data;
A learning unit which performs machine learning on the collected data or classified data; And
Based on the machine learning results, the predictive diagnostic unit for generating a report containing the predicted value and the resulting reliability for the collected data or classified data
Big data integrated diagnostic prediction system using a machine learning, comprising. The method of claim 1,
The collection unit,
Legacy DB module for storing the data collected from the website connected to the device in a database for big data analysis; And
IoT DB module for storing the data collected from the IoT device connected to the device in a database for big data analysis
Big data integrated diagnostic prediction system using a machine learning, comprising. The method of claim 1,
The basic data is,
Big data integrated diagnostic prediction using machine learning, comprising any one of a data collection method, collected data storage path, table information, field information, keywords, size of collected data, data collection time or collection target site system. The method of claim 1,
The analysis unit,
Coordinator module for classifying to match the predetermined value for analysis and learning of the collected data; And
Integrated DB for storing and managing data classified by the coordinator module on a memory basis
Big data integrated diagnostic prediction system using a machine learning, comprising. The method of claim 1,
The analysis performance definition data includes any one of an analysis execution method, an analysis execution cycle, an analysis target evaluation item, or a weight,
The analysis information is a big data integrated diagnostic prediction system using machine learning, characterized in that it comprises data digitized the frequency for each keyword. The method of claim 1,
The predicted value is,
It includes any one of the numerical forecast value of the sales volume, favorability, increase rate or demand for the collected data or classified data, and the categorical prediction value of any one of up / down, buy / sell or crush / dislike Big Data Integrated Diagnostic Prediction System using Machine Learning. (a) collecting data corresponding to a keyword set by the collection unit to a connected website or from an IoT device;
(b) classifying the classification unit so as to correspond to a predetermined value for analyzing and learning the collected data and storing the same in a database;
(c) analyzing the data stored in the database to generate analysis information so as to correspond to analysis performance definition data;
(d) performing machine learning on data collected or classified by the learner; And
(e) the predictive diagnostics unit generating a report including predicted values and reliability of the collected data or classified data based on the machine learning results.
Big data integrated diagnostic prediction method using a machine learning comprising a. The method of claim 7, wherein
In step (a),
(a-1) determining whether the data collection target included in the basic data set by the collector is a website or an IoT device;
(a-2) when the data collection target is a website as a result of the determination in the step (a-1), storing the data collected from the website accessed by the collection unit in a database through a legacy DB module; And
(a-3) When the data collection target is the IoT device as a result of the determination in the step (a-1), storing the data collected from the IoT device connected to the collection unit in the database through the IoT DB module
Big data integrated diagnosis prediction method using a machine learning comprising a. The method of claim 7, wherein
Step (b),
(b-1) the classification unit indexing the value set by the coordinator module;
(b-2) classifying, by the classification unit, classifying the collected data to match the set value indexed by the coordinator module; And
(b-3) storing the classified data in the memory-based storage through the integrated DB.
Big data integrated diagnosis prediction method using a machine learning comprising a. The method of claim 7, wherein
In step (c),
(c-1) indexing, by the analysis unit, analysis execution definition data including any one of an analysis execution method, an analysis execution cycle, an analysis target evaluation item, and a weight;
(c-2) the analyzing unit determining whether the data analysis target is a website or an IoT device;
(c-3) analyzing the data stored in the Legacy DB module in accordance with the analysis performance definition data when the data analysis target is a website as a result of the determination of step (c-2);
(c-4) analyzing the data stored in the IoT DB module in accordance with analysis performance definition data when the data analysis target is the IoT device as a result of the determination of step (c-2); And
(c-5) generating analysis information quantifying the frequency of each keyword of the data analyzed by the analysis unit.
Big data integrated diagnosis prediction method using a machine learning comprising a. The method of claim 7, wherein
In step (d),
(d-1) the learning unit receiving values of each of the input layer, the hidden layer, the learning model, and the parameters for machine learning;
(d-2) the learning unit indexing a function defined for performing machine learning;
(d-3) the learning unit indexing a data mining technique for performing machine learning; And
(d-4) the step of generating machine learning result information by executing machine learning through the function and data mining technique indexed by the learning department.
Big data integrated diagnosis prediction method using a machine learning comprising a. The method of claim 7, wherein
In step (e),
(e-1) the predictive diagnosis unit receiving the machine learning result from the learner;
(e-2) receiving the collected data or the classified data by the predictive diagnosis unit; And
(e-3) the predictive diagnosis unit generates a report including the predicted values and the reliability of the data collected or classified based on the machine learning results.
Big data integrated diagnostic prediction method using a machine learning comprising a.

Images