KR102393109B1 - Big data platform for collaboration among heterogeneous organizations and big data learning method using common learning model - Google Patents

Abstract

The big data learning method using the global learning model of the present invention comprises: distributing the learning model to at least two or more participating companies; performing pre-learning with the learning model distributed by each participating company; transmitting the weight of the learning model determined as optimal in the pre-learning from each participating company to an integrated platform; updating the learning model by combining the weights transmitted by the participating companies in the integrated platform; distributing the updated learning model to the participating companies; and performing fine-tuning learning for the updated learning model in each participating company.

Description

BIG DATA PLATFORM FOR COLLABORATION AMONG HETEROGENEOUS ORGANIZATIONS AND BIG DATA LEARNING METHOD USING COMMON LEARNING MODEL

The present invention is an AI learning model based on a data parallel-based model distributed learning method to create and share a global optimal mode that can be used jointly in the same industry without sharing data of each participating company participating in the big data platform. , to a big data platform for collaboration between heterogeneous institutions and a big data learning method using a common learning model.

As cloud systems are widely used, a large amount of data is accumulated, and as the need for analysis of the data emerges, the size of the market for big data analysis is also increasing. Domestic cloud platforms are still centered on infrastructure provision (IaaS), lacking enterprise application software that users can use, and big data analysis platforms are either not available or weak.

In the existing big data platform, inter-platform interworking is not considered, and it is common to analyze using data stored in its own system. Therefore, if additional data is needed during the analysis process, it must be processed into a form that can be additionally collected and utilized.

Although many companies disclose data according to government policy, most of them do not disclose important data sensitive to the company's operations and profits. Accordingly, it is practically difficult to expect synergy from data sharing. However, since the undisclosed data has a direct relationship to the profits of each company, the disclosure of non-public data cannot be forced.

As the importance of big data rises and data assetization takes place, individual companies do not share their data with other companies and institutions. Even if they share data, they are reluctant to disclose raw data. However, in order to create an optimal global learning model that can be applied in various situations, data from many companies in the same industry must be used.

Even in the power/generation field, a lot of data exceeding several hundred GB per day is accumulated by the individually built big data platform, but when preprocessing for data analysis, there is very little data to be actually used. If data is shared between heterogeneous companies in the same industry and the data that makes the algorithm is doubled, the performance of the algorithm can be improved.

However, since data is not shared between other companies due to data assetization and data security issues, the possibility of building and using a big data platform composed of heterogeneous companies is very low.

Even if a single big data platform is used between heterogeneous companies, most architectures are client-server structures. It is a form of developing and distributing .

When using a single platform, network costs and I/O costs occur as a lot of data is transmitted to one place. When data is managed in one place, a bottleneck occurs, which arises from the network cost, which is the time to transmit data, and the I/O cost for one server to receive and store data. As the number of platform participating organizations (enterprises) increases, network costs and I/O costs increase, resulting in increased inefficiency.

If data beyond the processing range of one platform is received, a server overload problem may occur, and the overload may lead to system failure. That is, the stability of the system may be weakened. If data synchronization is not performed within a given time due to system overload, a synchronization problem between platform data and actual data may occur. For example, if company A's data is not updated due to platform overload, an algorithm that does not reflect the latest situation of company A may be created and applied.

Therefore, there is a need for a big data platform that creates and shares a Global Optimal Model that can be used jointly in the same industry without sharing data of each company. In addition, the need for a platform that can generate and utilize the highest performance model by creating a final optimization model for each company by adding each company's data to the generated global optimal model is emerging.

Republic of Korea Registration No. 10-1971790

The present invention provides a big data platform and a big data learning method using a global learning model that can create/share/apply a global optimal mode that can be used jointly without sharing data unique to each participating company (institution) want to

A big data learning method using a global learning model according to an aspect of the present invention includes: distributing the learning model to at least two or more participating companies; performing pre-learning with the learning model distributed by each participating company; transmitting the weight of the learning model determined as optimal in the pre-learning from each participating company to an integrated platform; updating the learning model by combining the weights transmitted by the participating companies in the integrated platform; distributing the updated learning model to the participating companies; and performing fine-tuning learning for the updated learning model in each participating company.

Here, the learning model may be a narrow deep learning-based learning model, the pre-learning may be Supervised Pre-Training, and the fine-tuned learning may be Fine-tuned Training.

Here, in the step of performing the pre-learning, a global learning model in which pre-learning is performed using private data of each participating company may be used.

Here, in the step of performing the fine-tuning learning, only parameters of some layers close to the output layer may be applied to the learning.

Here, in the step of performing the pre-learning, when the learning cycle is long, learning can be done by the synchronous learning method, and when real-time is important, learning can be done by the non-synchronous learning method.

A big data platform for collaboration between heterogeneous organizations according to another aspect of the present invention includes: an integrated platform for collecting big data and distributing a big data learning model written to use the big data, and an updated learning model; and a big data participation platform provided in each participating company and supporting the unique business of each participating company using the big data,

The big data participation platform may receive the created big data learning model, perform pre-learning using the private data of each participating company, receive the updated learning model, and perform fine-tuning learning.

Here, the big data participation platform may transmit the weight of the learning model calculated as a result of performing the pre-learning to the integrated platform.

Here, the big data participation platform can learn by a synchronous learning method when the learning cycle is long, and can learn by a non-synchronous learning method when real-time is important.

Here, the big data participation platform may apply to learning only some layers (parameters) close to the output layer during the fine-tuning learning.

Here, the integrated platform may be based on SaaS (Software as a Services) that provides a service that can commonly utilize application software based on a marketplace.

When the big data learning method using the big data platform and/or global learning model for collaboration between heterogeneous institutions according to the idea of the present invention according to the above configuration is implemented, the big data platform participating companies (institutions) are jointly used without sharing data It has the advantage of being able to create/share/apply a possible global optimal mode.

The big data learning method using the big data platform and/or the global learning model for collaboration between heterogeneous organizations of the present invention protects sensitive data of each company through data isolation of individual companies using the platform, and is common to each company. It has the advantage of creating collaborative synergy by collaboratively creating and learning AI models to be applied.

The big data learning method using the big data platform and/or global learning model for collaboration between heterogeneous institutions of the present invention has the advantage that it can be utilized by creating a model optimized for individual companies by additionally learning individual company data in a general-purpose model. .

1 is a conceptual diagram illustrating a data parallel-based distributed learning architecture.
2 is a conceptual diagram illustrating a learning structure of a CNN (Convolutional Neural Networks) model.
3 is a conceptual diagram illustrating a structure of overlap training (Training) consisting of prior learning and fine-tuning learning.
4 is a conceptual diagram illustrating a SaaS integration platform structure.
5 is a block diagram illustrating a distributed learning structure of a big data platform for collaboration between heterogeneous organizations according to the spirit of the present invention.
6 is a flowchart illustrating a big data learning method using a global learning model according to the spirit of the present invention.
7 to 10 are block diagrams showing each step of the big data learning method using the global learning model of FIG. 6 in the big data platform for collaboration between heterogeneous organizations of FIG. 5 .

In describing the present invention, terms such as first, second, etc. may be used to describe various components, but the components may not be limited by the terms. The terms are only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When a component is referred to as being connected or connected to another component, it may be directly connected or connected to the other component, but it can be understood that other components may exist in between. .

The terminology used herein is used only to describe specific embodiments, and is not intended to limit the present invention. The singular expression may include the plural expression unless the context clearly dictates otherwise.

In this specification, the terms include or include are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and includes one or more other features or numbers, It may be understood that the existence or addition of steps, operations, components, parts, or combinations thereof is not precluded in advance.

In addition, shapes and sizes of elements in the drawings may be exaggerated for clearer description.

First, let's look at the distributed learning architecture that can be applied to the present invention.

1 is a conceptual diagram illustrating a data parallel-based distributed learning architecture.

Data parallel-based model distributed learning has a replica of the same model on each platform server of a heterogeneous company in the same industry using an integrated platform, and each server has a different subset of the data (Subset of the entire training dataset, generated by each company) between training. It is a method of merging the results while in charge of the collected data). A Global Optimal Model can be trained without data sharing.

In the synchronous parameter averaging method described in the drawings, parameters calculated by each device are transmitted to a device serving as a parameter server to average the parameters. The averaged parameter is sent back to the worker device, and each worker device performs learning through the data stored in each device. Repeat this process to make the final model. That is, the parameters are improved (reflected learning) by averaging the parameters that are the results of learning by each participating worker device.

In the case of synchronous learning per epoch, it guarantees the same degree of accuracy in theory as that of learning on a single device.

In the case of the gradient update method, the change value of the parameter is transmitted instead of transmitting the parameter to the parameter server. The synchronous gradient update method is the same as the parameter average method.

Each worker becomes a big data platform for heterogeneous companies in the same industry, and the parameter server can become an integrated platform.

Next, as a detailed learning method that can be applied to the aforementioned distributed learning architecture, we will look at pre-training.

2 shows a learning structure of a Convolutional Neural Networks (CNN) model.

Convolutional Neural Networks (CNN) models belong to the deep learning series of narrow learning models. In order to distinguish it from the broad concept of deep learning that means all machine learning, it is specified that it is deep learning in a narrow sense.

A lot of data is needed to train the models of the deep learning series of narrow-mindedness, including the Convolutional Neural Networks (CNN) model, and high computing power and long learning time are required to learn. Even if there is a lot of data generated by each company, it is not possible to have all the data for various environments, device types, and special situations. More than a certain amount of data is required for each case, and the more data is reflected in the model training process, the better the model can be created. Recently, numerous large data sets have been published on the Internet, and models learned from the data are being shared. For example, it is often the case that a model is created to classify 1,000 image types, pre-trained, and then the final model is shared. The reason for providing such a pre-trained model is that it is difficult to collect large amounts of data, and there are few environments with computing power capable of processing large amounts of data.

In addition, in the case of private or sensitive data, there is a problem in that it is difficult to disclose the data to the outside.

However, when you want to share the model that is the finished product, you can share the model with high accuracy as the final product with others while preventing data leakage by sharing only the pre-trained model. Building models using more data, even between peers and peers, can lead to higher-performing pre-trained models. However, when the model is trained in the existing method, data is shared, so if the model is created using the data parallel-based model distributed learning method described above, a pre-trained model with high accuracy can be created without data leakage.

Next, we will look at fine-tuned training.

3 shows a structure of superimposed training (Training) consisting of prior learning and fine-tuning learning.

Deep learning-based learning models, including CNN models, generally boast high accuracy for all data when only pre-training is performed. However, in the case of practical application, that is, when applied to each heterogeneous participating company, it is necessary to modify or fit the model to fit the situation and data of the company. By additionally learning data to be applied to the pre-trained model learned from data of various participating companies, it is possible to create a model suitable for the field to be applied in practice. In this process, the entire model can be trained, and when there is little data or the properties are similar to the data used for pre-learning, most of the model's layer (parameter) values are fixed, and some layers (parameters) close to the output layer ) can be applied to learning. When applied to the context of the present invention, each participating company creates a universally applicable public pre-trained model based on a data parallel-based model distributed learning technique. After that, it is possible to create a model optimized for each company by additionally learning each company's data in the pre-trained model.

Next, let's take a look at the platform.

4 is a conceptual diagram illustrating a SaaS integration platform structure.

Software as a Services (SaaS) refers to a platform 100 that provides a service that can commonly utilize application software based on a marketplace. SaaS provides software that can be used independently by each user in an isolated environment (Multi Tenant), independently guarantees system availability without interference from other user groups, and supports privilege management functions.

SaaS can solve the problem of inefficient reuse of developed software and models as they are shared in package units. Unlike the package unit sharing method, which is difficult to maintain and it is difficult to synchronize the shared model according to software and model changes, when using the SaaS platform 100, not only the company that developed the software but also other stakeholders such as other companies suffer from the above problems. can be used without In the case of the present invention, the pre-learning model created in cooperation with each company is shared with the participation platform 1000 of each company through the SaaS-based platform 100 and can be jointly utilized.

5 shows a distributed learning structure of a big data platform for collaboration between heterogeneous organizations according to the spirit of the present invention.

The big data platform for collaboration between heterogeneous institutions of the illustrated distributed learning structure is,

an integrated platform 100 for collecting big data and distributing a big data learning model written to use the big data, and an updated learning model; and big data participation platforms 1100 to 1400 provided in each participating company and supporting the unique business of each participating company using the big data.

In order to achieve the idea of the present invention, the learning model is for a global optimal model, the integrated platform 100 serves as a parameter server for the learning model, and the big data participation platform 1100 to 1400 receives the written big data learning model, performs pre-learning using the private data of each participating company, receives the updated learning model, and performs fine-tuning learning.

With the distributed learning structure shown, a big data learning method using a learning model (abbreviated as a global learning model) for reaching a global optimal model according to the spirit of the present invention can be performed. In other words, the illustrated distributed learning structure supports a shared AI algorithm through distributed learning that does not share the sensitive data of each participating company.

Using the data parallel-based model distributed learning method, it is possible to create a Global Optimal Model by sharing only the weights and weight changes of the AI model without sharing data of each participating company.

Because the optimal power model can be periodically retrained without data sharing by participating companies, the optimal model can be periodically generated without risk of data leakage and data sharing procedures by participating companies. The latest optimal AI models show higher prediction accuracy.

Even if weights and weight change values are leaked during network communication, there is no need to worry about information leakage during network communication because any information about the original data and AI model cannot be inferred or restored.

In an improved implementation, the big data participation platform 1100 to 1400 pre-trains a learning model with the unique data of each participating company, and as a result transmits a weight to the integrated platform 100 . This improved implementation can significantly reduce the amount of data transmitted because only the weights of the model are transmitted instead of the data of each participating company. Accordingly, by reducing network cost and I/O cost, inefficiency due to data transmission can be eliminated and model update time can be shortened.

Depending on the company/institution, the MDMS system generates more than 100GB of data per day, and there is data in TB units per table in most system databases. When the above-mentioned large-capacity data is transmitted, it may cause a network failure as well as an increase in transmission time due to network overload. In addition, since the integrated platform must store all data collected from participating companies, huge I/O costs are incurred. Excessive I/O cost not only slows down storage, but also increases the probability of system failure. Therefore, the method based on the data parallel-based model distributed learning method proposed in the present invention is superior in terms of efficiency and stability.

A predictive deep learning model with the same structure is trained with data from each participating institution. In this case, the model update is repeated every epoch in order to prevent falling into the local optimization in the distributed learning process.

The big data participation platforms 1100 to 1400 may apply a synchronous learning method and an asynchronous learning method, respectively, depending on the application of the AI model. That is, a synchronous learning method or an asynchronous learning method may be applied to each big data participation platform 1100 to 1400. The synchronous learning method takes longer to learn, but boasts higher accuracy. Therefore, when the model is updated at a specific time or when the learning cycle is long, the synchronous model generation method is more suitable.

On the other hand, for short-term real-time models such as 5-minute units, the asynchronous model generation method is more suitable. The asynchronous model creation method is suitable when real-time performance is important because model learning proceeds without waiting for the learning results of other participating organizations for model update.

Therefore, in the case of a real-time prediction model, a method of periodically learning the model through asynchronous learning with a fast learning time is required.

If a participating company does not participate in the global optimal model creation process, the completed model does not consider the situation of the participating company. When the model completed in this way is applied to a company that has not participated in the model creation process, the performance of the company may deteriorate when the model is applied because it does not reflect the situation of the company. Therefore, the platform of the present invention provides sufficient incentives for participating companies to periodically participate in the process of generating a predictive model.

The big data participation platforms 1100 to 1400 can contribute to the creation of an optimal AI algorithm for each company through fine-tuned training.

High performance can be seen even if each company directly applies the global optimal model generated through the above-described data parallel-based model distributed learning method. However, if the data of each company is additionally learned in the global optimal model, a model that better reflects the characteristics of the company can be created. In this case, when there is little data, many layer parameter values of the model are fixed and only a small number of layer parameters are trained.

That is, each of the big data participation platforms 1100 to 1400 can complete the optimal AI algorithm for each company by adjusting the degree of the model parameter to be additionally learned according to the amount of data of the participating company.

Preferably, the integrated platform 100 may be implemented as a SaaS-based global optimal model sharing platform, but is not limited thereto.

The integrated platform 100 not only serves as a parameter server in the process of learning the above-described global optimal model, but also functions as a sharing platform for sharing the completed global optimal AI model so that participating companies can use it. Each participating company can create and use multiple global optimal AI models (M _g ).

That is, each big data participation platform (1100 ~ 1400) of each participating company can always use the latest global optimal model. A global optimal model can be used.

In addition, the integrated platform 100 may serve as a hub (HUB) for storing and providing data and external data that can be shared by all participating companies.

6 is a flowchart illustrating a big data learning method using a global learning model according to the spirit of the present invention.

The big data learning method using the illustrated global learning model includes: distributing the learning model to at least two or more participating companies (S110); performing pre-learning with the learning model distributed by each participating company (S120); transmitting the weight of the learning model determined as optimal in the pre-learning from each participating company to the integrated platform (S130); updating the learning model by combining the weights transmitted by the participating companies in the integrated platform (S140); distributing the updated learning model to the participating companies (S150); and performing fine-tuning learning on the updated learning model in each participating company (S200).

For example, the learning model is a narrow deep learning-based learning model such as a Convolutional Neural Networks (CNN) model that performs learning to reach a global optimal model, and the pre-learning is Supervised Pre-Training of the CNN model, and the fine-tuning The training may be fine-tuned training of the CNN model.

7 to 10 show each step of the big data learning method using the global learning model of FIG. 6 being performed in the big data platform for collaboration between heterogeneous organizations of FIG. 5 .

As shown in FIG. 7 , in the step S120 of performing the pre-learning, each participating company learns the same AI model from the data of each company by using the big data participation platform. For example, each participating company can learn the AI model for 1 epoch based on the data of each facility. At this time, the same AI model is learned using the data of individual companies, so data sharing between participating companies does not occur.

As shown in FIG. 8, in the step of transmitting the weight of the learning model (S130), the big data participation platforms 1100 to 1400 of each participating company transmit the weight of the learned model to the integrated platform, and the learning model In the updating step (S140), the integrated platform 100 collects the model weights of each participating company and then aggregates them to update the model.

For example, the big data participation platforms 1100 to 1400 of each participating company can transmit the weight change value of the model trained by 1 epoch to the integrated platform. In this case, the transmission amount is significantly reduced compared to transmitting all data. can While the original data is in TB units, the model weight may be around 500 MB.

9, in step S150, the weight of the model updated in the integrated platform 100 is transmitted to the big data participation platforms 1100 to 1400 of each participating company.

Next, steps S120 to S150 may be repeated until the optimal model is completed (S160). Accordingly, when the final model is completed, each company can utilize the global optimal model in its own big data platform.

As shown in Figure 10, in the step (S200) of performing the fine-tuning learning, the data of each participating company is additionally learned in the global optimal model completed in the big data participation platform (1100 ~ 1400) of each participating company, You can create an AI model optimized for your company.

Table 1 below is a pseudocode showing the principle of operation of a big data platform for collaboration between heterogeneous organizations according to the spirit of the present invention.

As shown in the table above, the idea of the present invention can be easily applied to the necessary fields, and the technology of the present invention has high versatility, and can be used in all companies and institutions that utilize big data as well as the electric power and power generation industries specified in the detailed description.

When the proposed technology of the present invention is used, it is possible to lay the foundation for maximizing synergy effects through collaboration while protecting the data of each company.

Those skilled in the art to which the present invention pertains should understand that the present invention may be embodied in other specific forms without changing the technical spirit or essential characteristics thereof, so the embodiments described above are illustrative in all respects and not restrictive. only do The scope of the present invention is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

100: integrated platform
1100, 1200, 1300, 1400: Big data participation platform

Claims (10)

In a big data learning method using a global learning model in which each step is performed by a big data platform,
distributing the learning model to at least two or more participating companies;
performing pre-learning with the learning model distributed by each participating company;
transmitting the weight of the learning model determined as optimal in the pre-learning from each participating company to an integrated platform;
updating the learning model by combining the weights transmitted by the participating companies in the integrated platform;
distributing the updated learning model to the participating companies; and
Performing fine-tuning learning for the updated learning model in each participating company
including,
In the step of performing the prior learning, the synchronous learning method and the asynchronous learning method are respectively applied according to the application of the AI model,
A big data learning method using a global learning model that applies a synchronous model learning method to a model that updates a model at a specific time, and an asynchronous model learning method to a short-term real-time model.
According to claim 1,
The learning model is a deep learning-based learning model in a narrow sense,
The pre-learning is Supervised Pre-Training,
The fine-tuning learning is a big data learning method using a global learning model that is fine-tuned training.
According to claim 1,
In the step of performing the pre-learning,
A big data learning method using a global learning model that performs pre-learning using the private data of each participating company.
According to claim 1,
In the step of performing the fine-tuning learning,
A big data learning method using a global learning model that applies only the parameters of some layers close to the output layer to training.
delete an integrated platform for collecting big data and distributing a big data learning model written to use the big data, and an updated learning model; and
A big data participation platform provided in each participating company and supporting the unique business of each participating company using the big data,
The big data participation platform is
Receive the big data learning model created above and perform pre-learning using the private data of each participating company,
Receive the updated learning model, perform fine-tuning learning,
The synchronous learning method and the asynchronous learning method are respectively applied depending on the application of the AI model, but in the case of a model that updates the model at a specific time, the synchronous model learning method is applied, and in the case of a short-term real-time model, the asynchronous model learning method is applied.
A big data platform for collaboration between heterogeneous organizations.
7. The method of claim 6,
The big data participation platform is
A big data platform for collaboration between heterogeneous organizations that transmits the weight of the learning model calculated as a result of performing the pre-learning to the integrated platform.
delete 7. The method of claim 6,
The big data participation platform is
A big data platform for collaboration between heterogeneous institutions that applies only some layers (parameters) close to the output layer in the fine-tuning learning.
7. The method of claim 6,
The integrated platform is a big data platform for collaboration between heterogeneous organizations based on SaaS (Software as a Services) that provides services that can commonly utilize application software based on a marketplace.

Images