KR20190143519A - Connected data architecture of datalake framework - Google Patents

Abstract

The present invention relates to a connected data architecture of a data lake framework, which logically assembles a plurality of physically divided storages by software to stably transmit data for each interface while managing the data systematically and structurally. The present invention includes: a data lake (100) storing the data; a cloud storage (200) receiving the data from the data lake (100) to perform a backup operation; a data center (300) receiving or transmitting the data from or to the data lake (100); and a plurality of micro storages (400) previously storing identification information for receiving the data from the data center (300).

Description

CONNECTED DATA ARCHITECTURE OF DATALAKE FRAMEWORK}

The present invention relates to a connected data architecture of the datalake framework, and more particularly, a plurality of physically divided storages can be logically bundled by software to manage data systematically and structurally while stably transferring data for each interface. It relates to the connected data architecture of the Datalake framework.

Recently, due to the development of IT technology, the use of the Internet, etc. in the enterprise increases, and a large amount of data is produced and consumed. Increasingly, corporate value is shifting to data.

As a result, companies are recognizing the necessity of corporate data building and analysis systems to store and manage large amounts of data, and are building data warehouses or data silos.

The data warehouse is a practical application methodology that effectively integrates, coordinates and manages individual database management systems distributed across a large organization, and provides a foundation for efficient decision-making systems. ㆍ Consists of conversion and sorting tools, database marketing system, metadata, end user access and utilization tools.

Such a data warehouse includes an ODS (Operational Data Store) that purifies some of a plurality of source data, and the data of the ODS, as described in Korean Patent No. 10-1543506 (Registration Date: June 04, 2015). A data warehouse (DW) that integrates and creates a reference relationship between related data, and a plurality of data that generate a multidimensional model for each analysis subject on the data of the ODS or the DW based on a predetermined business rule. And a table report for generating a plurality of tables for each subject for one of the marts and one of the data marts, wherein the table report includes different tables using raw and adjusted results based on reference performance. Create a performance evaluation table and a balance sheet performance table; Generates quality information management table, the adjusted results can be yisugwan formed by considering both the customer reclassification, negative margin, tacheo, like accounts, handwriting adjustment, the old attribute bad MOU and staff performance tuning.

However, data warehouses have a risk of failure due to the enormous amount and complexity of data volume, which requires a huge amount of money and time to be invested, and data that is processed in an enterprise can be processed efficiently as it is massed in various forms. More and more companies are using data lakes.

Instead of defining a typical database structure first, then a data lake stores all kinds of data and then makes that data available when needed, instead of filling it with data that fits the structure.

Data lakes enable you to collect and store any type of data at any cost, at any cost, to secure data and prevent unauthorized access, to catalog, search and discover relevant data from a central repository, and to perform new types of data analysis. can do.

In addition, as data processed in the enterprise is mass-produced in various forms, use cases using big data are increasing.

A use case is a big function of a company or the like. It is a unique functional unit provided to a class or a system according to the importance of a message exchanged between systems, and one or more things outside of mutual actors are executed by the system. do.

In order to reliably transmit a large amount of data stored in such a data lake, there is a demand for a description of a use case for each interface.

The present invention has been proposed to solve the above-mentioned problems, in which a large amount of data is stored in a large capacity in a company or the like and logically bundled with a plurality of physically partitioned storage by software to manage data systematically and structurally while stably maintaining data for each interface. The purpose is to provide a connection data architecture of the datalake framework that can be transmitted.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The connection data architecture of the data lake framework according to the present invention for achieving the above object is a data lake 100 for storing data, and cloud storage for receiving and backing up data from the data lake 100 ( 200, a data center 300 for receiving or transmitting data from the data lake 100, and a plurality of micro storage 400 storing identification information in advance for receiving data from the data center 300. The data lake 100 includes a data lake communication unit 101 for communicating with the cloud storage 200, the data center 300, and the micro storage 400, and stores data previously input. Data storage unit 102 for management, the cloud storage 200, the data center 300 and the micro storage 400 to the day A transfer function generator 103 for generating a transfer function corresponding to the cloud storage 200 or the data center 300 or the micro storage 400 to transmit a transfer function, and the transfer function generator 103 A first data transmission unit 104 for transmitting data stored in the data storage unit 102 to the cloud storage 200 or the data center 300 by using a method overriding a transmission function generated in A second for extracting and transmitting data corresponding to the identification information of each microstorage 400 from the data storage unit 102 by using a method that overloaded the transfer function generated by the transfer function generator 103 It is configured to include a data transmission unit 105.

As described above, according to the present invention, a large amount of data is stored in a company and physically partitioned by software to logically bundle data by systematically and systematically managing data while stably transferring data for each interface, thereby creating a data space. This can be extended and backed up as well as improving the ease of uploading and coding.

In addition, there is an effect that can efficiently transfer data by utilizing the override and overload for each interface.

1 is a connection data architecture of a data lake framework according to an embodiment of the present invention;
2 is a data lake of the connection data architecture of the data lake framework according to an embodiment of the present invention;
3 is an Abyss Storage Cluster of a data lake of the connection data architecture of the data lake framework according to an embodiment of the present invention;
4 is a diagram illustrating cloud bursting and cloud spanning of a data lake of a connection data architecture of a data lake framework according to an embodiment of the present invention;
5 is a cloud storage of a connected data architecture of a data lake framework according to an embodiment of the present invention;
6 is a data center of a connected data architecture of a data lake framework according to an embodiment of the present invention;
7 is a microstorage of the connection data architecture of the datalake framework in accordance with one embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings will be described in detail the connection data architecture of the data lake framework according to the present invention.

1 is a connection data architecture of the data lake framework according to an embodiment of the present invention, Figure 2 is a data lake of the connection data architecture of the data lake framework according to an embodiment of the present invention, Figure 3 Abyss Storage Cluster of the data lake of the connected data architecture of the data lake framework according to an embodiment of the present invention, Figure 4 is a cloud bursting of the data lake of the connected data architecture of the data lake framework according to an embodiment of the present invention Cloud Spanning.

5 is a cloud storage of the connection data architecture of the data lake framework according to an embodiment of the present invention, Figure 6 is a data center of the connection data architecture of the data lake framework according to an embodiment of the present invention, 7 is a microstorage of the connection data architecture of the datalake framework according to an embodiment of the present invention.

In adding the reference numerals to the components of the drawings, the same components are to have the same reference numerals as possible even if displayed on different drawings, and known functions that are determined to unnecessarily obscure the subject matter of the present invention Detailed description of the configuration will be omitted. Also, directional terms such as 'top', 'bottom', 'front', 'back', 'tip', 'front', 'back' and the like are used in connection with the orientation of the disclosed figure (s). Since the components of the embodiments of the present invention may be positioned in various orientations, the directional terminology is used for the purpose of illustration and not limitation.

In the connection data architecture of the data lake framework according to an exemplary embodiment of the present invention, as shown in FIG. 1, a data lake 100 for storing data and data received from the data lake 100 are provided. Cloud storage 200 for receiving and backing up, a data center 300 for receiving or transmitting data from the data lake 100, and a plurality of identification information stored in advance for receiving data from the data center 300. It comprises a micro storage 400 of.

The connected data architecture of the data lake framework defines and designs an interface between the data lake 100 and the cloud storage 200 and an interface between the data lake 100 and the data center 300, and the data lake. Define and design an interface between the 100 and the microstorage 400.

As shown in FIG. 2, the data lake 100 includes a data lake communication unit 101 for communicating with the cloud storage 200, the data center 300, and the micro storage 400, in advance. Data storage unit 102 for storing and managing the input data, the cloud storage 200 or the cloud storage 200 to transmit data to the cloud storage 200, the data center 300 and the micro storage 400 or Using a transfer function generation unit 103 for generating a transfer function corresponding to the data center 300 or the micro storage 400 and a transfer function generated by the transfer function generation unit 103 A first data transmitter 104 and the transfer function for transmitting data stored in the data storage 102 to the cloud storage 200 or the data center 300. A second data transmission unit for extracting and transmitting data corresponding to the identification information of each microstorage 400 from the data storage unit 102 by using the method that overloaded the transfer function generated in the unit 103; 105).

In addition, the data lake 100 is formed based on the large-capacity Abyss Storage Cluster for SMB configured as shown in Figure 3, it is possible to actually develop the H / W prototype of the Abyss Storage Cluster and mass production of the product . In addition, to improve the performance of Abyss Storage, the company has completed performance tests for each disk medium of the storage and bonding and network traffic tests for both domestic and international networks.

In addition, the data lake 100 may include a physical layer and a distributed storage layer to establish an enterprise-wide data lake to capture, process, and analyze a large amount of data and provide it to a user or a system consuming data. (Distributed Storage Layer), Security Layer, Data Acquisition Layer, Messaging Layer, Ingestion Layer, Lambda Architecture, Services It may be configured to include a layer (Serving Layer).

In this case, the datalake 100 provides cloud bursting and cloud spanning by the relationship between the metadata of the lambda architecture and the content layer, as shown in FIG. 4.

The cloud bursting is an application distribution model used in a hybrid cloud (mixed cloud) environment, and when the computing capacity of the work device 100 is exceeded, it is automatically transmitted to the public cloud due to the excess demand, so that the application can continue to run. Make sure

The cloud spanning is a delivery model for distributing application components that require a large amount of computing resources in multiple cloud environments at the same time.

The data lake communication unit 101 may use a wireless network to communicate with the cloud storage 200, the data center 300, and the micro storage 400. The wireless network may use Wi-Fi, Long Term Evolution (LTE), and the like.

The data storage unit 102 is for storing and managing data, and a user may input and store data in advance.

In addition, the data storage unit 102 may input and store data transmitted from any one of the cloud storage 200, the data center 300, and the micro storage 400.

The transfer function generator 103 generates a transfer function corresponding to the cloud storage 200, the data center 300, or the micro storage 400 so as to transmit data. That is, the transmission function generator 103 generates a transmission function for transmitting data according to the cloud storage 200 or the data center 300 or the micro storage 400 connected to receive or transmit the data.

The first data transmission unit 104 is for transmitting data to the cloud storage 200 or the data center 300, the cloud storage 200 or the data in the transfer function generation unit 103 In order to transmit data to the center 300, a method overriding the generated transmission function is used.

Here, the override is to redefine the methods in the parent class in the child class.

When transmitting data in connection with the cloud storage 200 through the first data transmission unit 104, it is necessary to expand the data space, secure data transmission, and data security. In particular, it can support a variety of services, such as intelligent analysis or prediction using the resources of the cloud storage 200.

When transmitting data by being connected to the data center 300 through the first data transmission unit 104, expansion and backup of a data space, secure data transmission, and security are necessary.

The second data transmitter 105 is connected to the micro storage 400 to transmit data.

At this time, the second data transmission unit 105 extracts data from the data storage unit 102 according to the identification information of each micro storage 400, and transfers the transfer function generated by the transmission function generation unit 103. The extracted data is transmitted to the micro storage 400 by using the overloaded method.

Here, the overload can define a method of the same name using different parameters, the type and number of parameters can be changed.

Through the second data transmission unit 105, it is possible to transmit different data for each micro storage.

As shown in FIG. 5, the cloud storage 200 is received from the data lake 100 through the cloud communication unit 201 and the cloud communication unit 201 for communicating with the data lake 100. And a backup unit 202 for backing up the received data.

The cloud storage 200 is a data storage model in which digital data is stored in a logical pool, and a physical environment is generally owned and managed by a hosting company while a physical storage spans a plurality of servers. These cloud providers are responsible for ensuring that data is always available and accessible, and that the physical environment runs protected. An individual or organization purchases or leases storage volumes from a provider to store user, organization, and application data. Accordingly, the cloud storage 200 mainly stores a large amount of data, and can be accessed anytime and anywhere through an internet connection.

The cloud communication unit 201 may use a wireless network such as Wi-Fi, the Internet, etc. to communicate with the data lake 100.

The backup unit 202 backs up the data received from the data lake 100. That is, the backup unit 202 copies and backs up the data received from the data lake 100 through the cloud communication unit 201, and then stores and manages the data.

As illustrated in FIG. 6, the data center 300 is received from the data lake 100 through a center communication unit 301 for communicating with the data lake 100 and the center communication unit 301. It includes a data management unit 302 for storing and managing the received data, and a data transfer unit 303 for delivering new data previously input to the data lake 100.

The data center 300 stores and manages big data, which is a plurality of data mainly used in corporations and the like.

The center communication unit 301 communicates with the data lake 100 through a wireless network such as Wi-Fi, the Internet, or the like.

When receiving data from the data lake 100 through the center communication unit 301 is transmitted to the data management unit 302.

The data manager 302 stores and manages data received from the data lake 100.

In this case, when the user newly inputs data, the data manager 302 may further store and manage new data newly input by the user.

The data transfer unit 303 transfers previously inputted new data to the data lake 100 through the center communication unit 301.

As illustrated in FIG. 7, the micro storage 400 may include a micro communication unit 401 for communicating with the data lake 100 and for generating identification information to be distinguishable from other micro storage 400. An identification information unit 402, an identification information transmission unit 403 for transmitting identification information to receive data from the data lake 100, and the data receiving the identification information of the identification information transmission unit 403. And a data accumulator 404 for receiving and storing data corresponding to the identification information from the rake 100.

The micro storage 400 may be an electronic device such as a smartphone, a security device, a medical device, a navigation device, an IoT device, and the like.

The micro communication unit 401 communicates with the data lake 100 using a wireless network such as Wi-Fi.

The identification information unit 402 inputs and generates identification information for identifying the micro storage 400. In this case, the identification information may be generated including the type of micro storage, user information, and the like.

The generated identification information is transmitted to the data lake 100 through the identification information transmission unit 403. That is, the identification information transmitting unit 403 transmits the generated identification information to the data lake 100 to request data.

The identification information transmission unit 403 transmits the identification information to the data lake 100 so that the data necessary for the micro storage 400 can be received.

The data accumulator 404 accumulates and stores the data received from the data lake 100. For example, when the micro storage 400 is navigation, the identification information generated by the identification information unit 402 including navigation information is transferred to the data lake 100 by the identification information transmission unit 403. The data is received and stored in the data lake 100 such as a map corresponding to the navigation of the identification information.

 The connected data architecture of the data lake framework configured as described above can logically bundle various physically divided storages by software, and transmit or receive data for each interface between the data lake, cloud storage, data center, and micro storage. It becomes possible.

In other words, the connected data architecture of the datalake framework transmits data per interface stably while systematically and systematically managing data by logically tying a plurality of storages in which a large amount of data is stored and physically partitioned by software in an enterprise or the like. By doing so, the data space can be expanded and backed up, as well as to facilitate easy uploading and coding.

The embodiments of the present invention described above and illustrated in the drawings should not be construed as limiting the technical spirit of the present invention. The protection scope of the present invention is limited only by the matters described in the claims, and those skilled in the art can change and change the technical idea of the present invention in various forms. Therefore, such improvements and modifications will fall within the protection scope of the present invention if it is obvious to those skilled in the art.

100: data lake 101: data lake communication unit
102: data storage unit 103: transfer function generation unit
104: first data transmission unit 105: second data transmission unit
200: cloud storage 201: cloud communication
202: backup unit 300: data center
301: center communication unit 302: data management unit
303: data transfer unit 400: microstorage
401: micro communication unit 402: identification information unit
403: identification information transmission unit 404: data accumulation unit

Claims (1)

A data lake 100 for storing data, a cloud storage 200 for receiving and backing up data from the data lake 100, and a data center for receiving or transmitting data from the data lake 100 ( 300, and a plurality of micro storage 400 is stored in advance identification information for receiving data from the data center 300,
The data lake 100, the data storage communication unit 101 for communicating with the cloud storage 200, the data center 300 and the micro storage 400, and for storing and managing pre-input data Data storage unit 102, the cloud storage 200, the data center 300 and the micro storage 400 for transmitting data to the cloud storage 200 or the data center 300 or the micro Transfer function generator 103 for generating a transfer function to correspond to the storage 400 and a method that overrides the transfer function generated by the transfer function generator 103 to the data storage 102. A transfer function generated by the first data transmitter 104 and the transfer function generator 103 for transmitting the stored data to the cloud storage 200 or the data center 300. A data lake frame including a second data transmitter 105 for extracting and transmitting data corresponding to the identification information of each microstorage 400 from the data storage 102 using the overloaded method. Workpiece's connection data architecture.

Images