KR20180059269A - Bigdata platform interlock apparatus and method thereof - Google Patents

Abstract

A big data platform linking apparatus and a linking method thereof are disclosed. In the apparatus, a metadata processing unit receives metadata from a first database which provides the metadata that is setting information corresponding to a user, transmits the received metadata to a distributed file system in the big data platform, and directly stores the metadata in the distributed file system. A result transmitting unit directly reads and obtains analysis result data after a query analysis by the big data platform, from the distributed file system, and transmits the obtained analysis result data to a second database. A scheduler controls the time and order of tasks performed by the metadata processing unit and the result transmitting unit. Accordingly, the present invention can save a resource and provide fast performance.

Description

BIGDATA PLATFORM INTERLOCK APPARATUS AND METHOD THEREOF FIELD OF THE INVENTION [0001]

The present invention relates to a Big Data Platform Interworking Device and its interworking method.

Recently, interest in big data technology that gives meaningful value to massive data such as stereotyped or unstructured data is increasing.

Many application services are required to produce accurate and fast results through big data.

On the other hand, the Big Data Platform used for processing Big Data analyzes Big Data in conjunction with an external Relational Database Management System (RDBMS). At this time, RDBMS technology is fast because it has evolved for decades, but the Big Data platform is mostly open-sourced and has not yet been optimized, so using the existing RDBMS connectivity technology on the Big Data platform will be very slow.

For example, Hadoop, which has recently become one of the big data platforms, is a technology that processes data in parallel on several computers in order to deal with data that is difficult to process by the existing RDBMS method, Technology.

In Hadoop, it works with existing RDBMS through sqoop. Here, sqoop is a technology designed to efficiently convert large-capacity data between RDBMS and Hadoop.

For example, the Hadoop cluster receives meta data from the RDBMS providing metadata through a scan, performs big data analysis using the metadata according to a query from the user, It is provided as an external RDBMS through interworking technologies such as SQL, Java Database Connectivity (JDBC), and Open Database Connectivity (ODBC). Here, the metadata is setting information used for analyzing the log to be analyzed.

As described above, when the existing connection technology is used, resources required to be used for analysis are consumed because resources such as Hive servers associated with Hadoop must be used for each required process. Therefore, analysis processing in Hadoop cluster There is a problem that the speed is delayed.

SUMMARY OF THE INVENTION The present invention provides a large data platform interworking apparatus and method for interworking between a big data platform and an external RDBMS.

According to an aspect of the present invention,

A metadata processing unit that receives metadata from a first database that provides metadata, which is setting information corresponding to a user, and transfers the received metadata to a distributed file system in the big data platform to directly store the received metadata; A result transmitting unit for reading and acquiring analysis result data after the query analysis is performed by the big data platform directly from the distributed file system and for transmitting the obtained analysis result data to a second database; And a scheduler for performing time and order control of operations performed by the metadata processing unit and the result transmitting unit.

Here, the metadata processing unit performs consistency checking on the metadata received from the first database.

In addition, the metadata processing unit extracts only the changed information from the metadata received from the first database, and stores the extracted information in the distributed file system.

In addition, if the number of jobs to be processed by one process generated by the meta data processor is to be processed using a multi-thread, the scheduler can process the meta data processor using the multi-thread .

A query receiver for receiving a query registered by a user or receiving a user query registered as the big data platform from the big data platform; And a query execution unit operable to perform an analysis of a query received by the query receiving unit in association with the big data platform at a cycle and an execution time set by the user, The user query is read and control is performed so that no conflict occurs between the components during the entire operation from execution of the query to transmission of the analysis result data.

In addition, the query execution unit simultaneously performs an analysis query including a UDF (User Defined Function) set in the Big Data Platform at a cycle and a execution time set by the user.

In addition, the big data platform is a Hadoop system, the distributed file system is a HDFS (Hadoop Distributed File System), and the first database and the second database are RDBMS (Relational Database Management System).

According to another aspect of the present invention,

1. A method for a Big Data Platform Interlocking Device to interact with a Big Data Platform and an external first and second databases,

Receiving metadata, which is setting information corresponding to a user, from the first database, transferring the received metadata to a distributed file system in the big data platform and directly storing the received metadata; Determining whether analysis result data is stored in the distributed file system after the query analysis is performed by the big data platform; And directly reading and acquiring the analysis result data from the distributed file system when it is determined that the analysis result data is stored in the distributed file system, and transferring the analysis result data obtained to the second database .

Here, the directly storing step may include: receiving metadata, which is setting information corresponding to a user, from the first database; Performing a consistency check on the metadata; Extracting only the changed information from the metadata if the matching is confirmed; And directly storing the extracted metadata in the distributed file system.

Also, in the direct storing step, when the number of jobs to be processed by one process needs to be processed using the multi-thread, the multi-thread is generated and processed.

The method may further include controlling the analysis of the query in cooperation with the big data platform at a cycle and a execution time set by the user between the direct storing step and the determining step.

According to another aspect of the present invention,

Wherein the communication unit performs data transmission and reception so that the interlocking device can perform an interlocking operation between the big data platform and a first external database and a second external database, And a program for causing the processor to perform a data transfer with the first and second databases, wherein the processor calls a program stored in the memory, and reads the metadata read from the first database via the communication unit Data is directly transferred to and stored in the distributed file system of the big data platform, and analysis result data is directly read from the distributed file system and transferred to the second database.

Here, the communication unit further stores a program for receiving a query registered by a user or receiving a user query from the big data platform registered as the big data platform, the memory executing a query, and the processor The program stored in the memory is called to perform an operation for performing an analysis on a query received through the communication unit in association with the big data platform at a cycle and execution time set by the user.

Further, the memory further stores a program for performing an operation using a multi-thread, and the processor calls a program stored in the memory so that the number of operations that a single process has to process is multi- If you need to do so, create a multithread and process it.

In addition, the processor performs control so that collision does not occur during an operation that is performed using a multi-thread.

A computer-readable recording medium according to an aspect of the present invention,

A computer readable recording medium storing a program for performing a method of interworking with a big data platform, the computer readable medium having stored thereon metadata that is setting information corresponding to a user from the first database, Transferring them directly to the system and storing them directly; A function of determining whether analysis result data is stored in the distributed file system after the query analysis is performed by the big data platform; And directly reading and acquiring the analysis result data from the distributed file system when it is determined that the analysis result data is stored in the distributed file system, and transferring the analysis result data obtained to the second database .

According to the present invention, since an interworking device that interworks between a big data platform and an external RDBMS directly records metadata in a distributed file system and is implemented to fetch analysis results stored in a distributed file system, You will not consume resources.

In addition, the Big Data Platform Interoperability can provide fast performance in terms of speed by directly reading and writing to the distributed file system.

In addition, the number of tasks to be processed by a single process created in the Big Data Platform Interworking Device can be checked and processed using as many multithreads as necessary, thereby saving resources allocated to the process.

1 is a diagram schematically illustrating an interworking method between a general Hadoop system and an RDBMS-based system.
FIG. 2 is a diagram schematically illustrating a flow of data transfer between the Hadoop system and the RDBMS-based system shown in FIG.
3 is a diagram schematically illustrating a method of interfacing a Big Data Platform according to an embodiment of the present invention.
4 is a diagram schematically illustrating a Hadoop system using a Big Data Platform Interworking Device according to an embodiment of the present invention.
FIG. 5 is a diagram schematically illustrating a job and a data flow when using the Big Data Platform Interlocking Device according to the embodiment of the present invention.
6 is a view showing another configuration of the interlocking device according to the embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise. Also, the terms " part, "" module," and " module ", etc. in the specification mean a unit for processing at least one function or operation and may be implemented by hardware or software or a combination of hardware and software have.

First, a method of interworking with an external system of a general big data platform will be described.

In the embodiment of the present invention, a Hadoop system is used as a big data platform for convenience of explanation, but the technical scope of the present invention is not limited to this and can be applied to other types of big data platforms.

FIG. 1 is a diagram schematically illustrating an interworking method between a general Hadoop system and an RDBMS-based system, and FIG. 2 is a diagram schematically illustrating a data transfer flow between the Hadoop system and the RDBMS-based system shown in FIG.

Referring to FIG. 1, the Hadoop system 10 is provided with metadata, which is configuration information, through an RDBMS 20 providing setup information and a step-and-click interface.

When the Hadoop system 10 provides the large data analysis result using the metadata to the external RDBMS 30, i.e., the RDBMS 30 utilizing the analysis result, the Hadoop system 10 performs the Spark / Pig interworking method 51, the JDBC / ODBM interworking The method 52 or the step 53 is used.

2, when the big data collected in the HDFS (Hadoop Distributed File System) 11 of the Hadoop system 10 is stored, the Hive Server 12 then stores the large data The analyzing unit 13 receives the metadata from the RDBMS 20 and analyzes the result of the analysis of the big data by the analyzing unit 13 through the interworking techniques 51, To the RDBMS 30.

In this way, when the Hadoop system and the external RDBMS are connected using the interworking technology (40, 51, 52, 53) such as a scan in the general technology, resources such as a hive server associated with Hadoop must be used for each required process Therefore, it is necessary to use separate scheduling for each process and separate development every time the business logic is added. In addition, .

Hereinafter, embodiments of the present invention for solving the above problems will be described.

3 is a diagram schematically illustrating a method of interfacing a Big Data Platform according to an embodiment of the present invention. In the embodiment of the present invention, the Hadoop system, which is a specific system, will be described as an example, but the present invention is not limited thereto.

In general, the data inflow rate of Hadoop system 100, which is a big data platform, is remarkably faster than that of Hadoop Distributed File System (HDFS) 110, which stores large amounts of data in a distributed manner in Hadoop system 100, (HDFS Read & Write), and large-capacity SQL (Structured Query Language).

Therefore, the interlocking device 400, which manages interworking between the Hadoop system 100 and the external RDBMSs 200 and 300 according to the embodiment of the present invention, combines the HDFS Read & Write and the large- The system 100 and the RDBMSs 200 and 300 are efficiently linked and resources for analysis are not used.

The interworking device 400 according to the embodiment of the present invention checks and extracts change information from the RDBMS 200 that provides metadata corresponding to the setting information of the log analysis information in a single process and outputs the change information to the HDFS 110 Add the data directly. At this time, the user may automatically recognize the desired metadata and generate the metadata if the metadata is new.

Also, in order to ensure the consistency of the setting information, the hive query operation is performed so that the primary key is recognized and sorted based on the primary key to take only the latest data.

In addition, the interworking apparatus 400 according to the embodiment of the present invention can recognize and perform the scheduling and performance management for the INSERT, OVERWRITE type user query, read the position of the designated HDFS 100, Bulk Insert can be performed.

Meanwhile, the interworking device 400 according to the embodiment of the present invention can perform data transmission using the JDBC connection function between the Hadoop system 100 and the RDBMSs 200 and 300.

4 is a diagram schematically illustrating a Hadoop system using a Big Data Platform Interworking Device according to an embodiment of the present invention.

4, the interlocking apparatus 400 according to the embodiment of the present invention includes a metadata processing unit 410, a query receiving unit 420, a query executing unit 430, a result transmitting unit 440, and a scheduler 450).

The metadata processing unit 410 receives the metadata about the user from the RDBMS 200 and directly stores the metadata in the HDFS 110 of the Hadoop system 100. [

Alternatively, the metadata processing unit 410 may perform consistency checking on the metadata received from the RDBMS 200, extract only the changed information, and directly store the extracted information in the HDFS 110. [

The query receiving unit 420 receives a query directly registered by the user or receives a user query registered in the Hadoop system 100 from the Hadoop system 100.

The query execution unit 430 analyzes the query received by the query receiving unit 420 in conjunction with the Hive Server 120 of the Hadoop system 100 at the cycle and execution time set by the user . For example, the query execution unit 430 simultaneously executes an analysis query including a hive UDF (User Defined Function) set in the hive server 120 at a cycle and a execution time set by the user.

The result transmitting unit 440 reads the analysis result data directly from the HDFS 110 when the analysis result in which the big data base analysis is completed for the query in the Hadoop system 100 is stored in the HDFS 110, And transmits the data to the RDBMS 300 utilizing the analysis result. In this case, the result transmitting unit 440 should be able to read the analysis result data directly from the HDFS 110 and transmit the analysis result data to the corresponding RDBMS 300 by recognizing the location of the HDFS 110, the target RDBMS 300, and the like.

The scheduler 450 controls the metadata processing unit 410, the query receiving unit 420, the query executing unit 430 and the result transmitting unit 440 so that the Hadoop system 100 and the external RDBMSs 200 and 300 Interworking is performed.

Specifically, the scheduler 450 checks the number of tasks to be processed by one process generated by the meta data processor 410 and processes the tasks using as many multithreads as necessary. You can save allocated resources.

In addition, the scheduler 450 reads the setting information of the user and the query, and performs time and order control so that collision does not occur during operation of each component from the execution of the query to the transmission of the analysis result. For example, when the multithread is created and operated by the meta data processor 410, control is performed so that no conflict occurs with each other.

FIG. 5 is a diagram schematically illustrating a job and a data flow when using the Big Data Platform Interlocking Device according to the embodiment of the present invention.

Referring to FIG. 5, first, data collected by an external data collection server (not shown) is stored in the HDFS 110 of the Hadoop system 100. These data are collected periodically by external data collection servers or at the time of event occurrence, and stored in the HDFS 110 to form big data.

The scheduler 450 of the interworking device 400 according to the embodiment of the present invention recognizes that the query registration is performed from the user through the query receiving unit 420 in step S100, From the external RDBMS 200 (S110).

Then, the metadata processing unit 410 confirms the consistency of the read metadata, directly transfers the metadata to the HDFS 110 and stores it (HDFS Write) (S120). Here, the metadata processing unit 410 recognizes a primary key to ensure the consistency of the metadata.

Alternatively, the metadata processing unit 410 may perform a query operation so as to sort only the latest data by sorting the metadata whose consistency has been confirmed.

The query execution unit 430 then analyzes the query received by the query receiving unit 420 in conjunction with the hive server 120 of the Hadoop system 100 at the cycle and execution time set by the user (S130). For example, the query execution unit 430 simultaneously executes an analysis query including a hive UDF set in the hive server 120 at a cycle and a execution time set by the user.

Next, the analysis performing unit 130 of the Hadoop system 100 performs analysis corresponding to the user query using the metadata and analysis data stored in the HDFS 110 (S140). Since the characteristic according to the embodiment of the present invention is not related to the process of performing the analysis in the Hadoop system 100, a detailed description of the process of performing the analysis corresponding to the query is omitted here Will be readily apparent to those skilled in the art.

Thereafter, the analysis performing unit 140 stores the analysis result data, which is the result of the analysis, in the HDFS 110 (S150).

If it is recognized by the scheduler 450 of the interworking device 400 that the analysis result data performed by the Hadoop system 100 is stored in the HDFS 110, (HDFS Read) (S160), and then transmits the read analysis result data to the RDBMS 300 using the analysis result (S170). At this time, the result transmitting unit 440 recognizes the location of the HDFS 110, the target RDBMS 300, and the like, reads the analysis result data directly from the HDFS 110, and transmits the data to the corresponding RDBMS 300.

As described above, in the embodiment of the present invention, the interworking device 400 interworking between the Hadoop system 100 and the external RDBMSs 200 and 300 directly records the metadata in the HDFS 110, Because it is implemented to retrieve stored analysis results, it does not consume resources that should be used for analysis.

In addition, since the interlock 400 directly reads and writes to the HDFS 110, it is possible to provide fast performance in terms of speed.

In the present invention, the number of tasks to be processed by one process generated by the meta data processing unit 410 in the interworking device 400 is checked and processed using as many multithreads as necessary, You can save money.

6 is a view showing another configuration of the interlocking device according to the embodiment of the present invention.

Referring to FIG. 6, an interlocking device 600 according to an embodiment of the present invention includes a communication unit 610, a memory 620, and a processor 630.

The communication unit 610 performs data transmission and reception so that the interlocking device 600 can perform an interlocking operation between the Hadoop system 100 and the RDBMSs 200 and 300 according to an embodiment of the present invention. In particular, in the case of the Hadoop system 100, direct data transmission / reception with the HDFS 110 of the Hadoop system 100 is possible.

Specifically, the communication unit 610 receives the metadata about the user from the RDBMS 200 and directly transmits the received metadata to the HDFS 110 of the Hadoop system 100. [

The communication unit 620 receives a query directly registered by the user or receives a user query registered in the Hadoop system 100 from the Hadoop system 100. [

The communication unit 620 receives analysis result data directly from the HDFS 110 and transmits the analysis result data to the RDBMS 300. At this time, the communication unit 620 must be able to recognize the location of the HDFS 110, the target RDBMS 300, and the like, receive the analysis result data directly from the HDFS 110, and transmit the data to the corresponding RDBMS 300.

Meanwhile, the memory 620 stores a program for executing data transmission with the Hadoop system 100 and the RDBMSs 200 and 300, a program for executing a query, And a program that performs control so as not to occur.

The processor 630 may be configured to invoke a program stored in the memory 620 to implement the procedures and / or methods proposed in embodiments of the present invention as described with reference to Fig. That is, the processor 630 directly transmits the metadata read from the RDBMS 200 to the HDFS 110 of the Hadoop system 100 through the communication unit 610 and stores the metadata in the period and the execution time set by the user An analysis of the user query is performed in cooperation with the hive server 120 of the Hadoop system 100 and the analysis result data is read directly from the HDFS 110 and is transferred to the RDBMS 300

Such a processor 630 may also be referred to as a controller, a microcontroller, a microprocessor, a microcomputer, or the like. The processor 630 may also be implemented by hardware or firmware, software, or a combination thereof.

The processor 630 performs a consistency check on the metadata received from the RDBMS 200 by the communication unit 610 and extracts only the changed information and stores the extracted information in the HDFS 110 directly.

The embodiments of the present invention described above are not implemented only by the apparatus and method, but may be implemented through a program for realizing the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

Claims (16)

As an interlocking device for the Big Data Platform,
A metadata processing unit that receives metadata from a first database that provides metadata, which is setting information corresponding to a user, and transfers the received metadata to a distributed file system in the big data platform to directly store the received metadata;
A result transmitting unit for reading and acquiring analysis result data after the query analysis is performed by the big data platform directly from the distributed file system and for transmitting the obtained analysis result data to a second database; And
A scheduler for performing time and order control of a job performed by the metadata processing unit and the result transmitting unit;
The Big Data Platform Interworking Device. The method according to claim 1,
Wherein the metadata processing unit performs a consistency check on the metadata received from the first database,
Big Data Platform Interlock. 3. The method of claim 2,
Wherein the metadata processing unit extracts the changed information from the metadata received from the first database and stores the extracted information in the distributed file system,
Big Data Platform Interlock. The method according to claim 1,
The scheduler controls the meta data processor to perform processing using the multi-thread when the number of tasks to be processed by one process generated by the meta data processor is to be processed using a multi-thread doing,
Big Data Platform Interlock. The method according to claim 1,
A query receiver for receiving a query registered by a user or receiving a user query registered as the big data platform from the big data platform; And
A query execution unit operable to perform analysis on a query received by the query receiving unit in association with the big data platform at a cycle and execution time set by a user,
Further comprising:
Wherein the scheduler reads a user query through the query receiver and performs control so as to prevent a collision between components during a whole operation from execution of a query to transmission of analysis result data,
Big Data Platform Interlock. 6. The method of claim 5,
Wherein the query execution unit concurrently performs an analysis query including a UDF (User Defined Function) set in the big data platform at a cycle and a execution time set by the user,
Big Data Platform Interlock. 7. The method according to any one of claims 1 to 6,
Wherein the big data platform is a Hadoop system, the distributed file system is a HDFS (Hadoop Distributed File System), the first database and the second database are relational database management systems (RDBMS)
Big Data Platform Interlock. 1. A method for a Big Data Platform Interlocking Device to interact with a Big Data Platform and an external first and second databases,
Receiving metadata, which is setting information corresponding to a user, from the first database, transferring the received metadata to a distributed file system in the big data platform and directly storing the received metadata;
Determining whether analysis result data is stored in the distributed file system after the query analysis is performed by the big data platform; And
Directly reading and acquiring the analysis result data from the distributed file system when it is determined that the analysis result data is stored in the distributed file system, and delivering the obtained analysis result data to the second database
The method comprising: 9. The method of claim 8,
Wherein the direct storing comprises:
Receiving metadata, which is setting information corresponding to a user, from the first database;
Performing a consistency check on the metadata;
Extracting only the changed information from the metadata if the matching is confirmed; And
Directly storing the extracted metadata in the distributed file system
The method comprising the steps of: 9. The method of claim 8,
In the direct storing step,
If the number of tasks that a single process needs to handle is multithreaded,
Big Data Platform Interworking Method. 9. The method of claim 8,
Between the direct storing step and the determining step,
Further comprising controlling the analysis of the query to be performed in cooperation with the big data platform at a cycle and a execution time set by the user,
Big Data Platform Interworking Method. As an interlocking device for the Big Data Platform,
A communication unit, a memory, and a processor,
The communication unit performs data transmission and reception so that the interlocking device can perform an interlocking operation between the big data platform and the external first and second databases,
The memory stores a program for causing the communication unit to perform data transmission with the big data platform and the first and second databases,
Wherein the processor calls the program stored in the memory to directly transfer the metadata read from the first database through the communication unit to the distributed file system of the big data platform and store the analysis result directly from the distributed file system Reading data from the first database and transferring the data to the second database,
Big Data Platform Interlock. 13. The method of claim 12,
Wherein the communication unit receives a query registered by a user or receives a user query registered with the big data platform from the big data platform,
The memory further stores a program for causing a query to be executed,
Wherein the processor invokes a program stored in the memory to perform an operation in cooperation with the big data platform at an interval and an execution time set by a user so that an analysis of a query received through the communication unit can be performed,
Big Data Platform Interlock. 14. The method of claim 13,
The memory further stores a program for performing an operation using a multi-thread,
Wherein the processor calls a program stored in the memory and generates a multi-thread when the number of tasks to be processed by one process needs to be processed using the multi-thread,
Big Data Platform Interlock. 15. The method of claim 14,
Wherein the processor is configured to perform control to prevent collision during an operation that is performed using the multi-
Big Data Platform Interlock. A computer-readable recording medium storing a program for performing an interworking method of a big data platform,
Receiving metadata, which is setting information corresponding to a user, from the first database, transmitting the received metadata to a distributed file system in the big data platform, and storing the metadata directly;
A function of determining whether analysis result data is stored in the distributed file system after the query analysis is performed by the big data platform; And
A step of directly reading and acquiring the analysis result data from the distributed file system when it is determined that the analysis result data is stored in the distributed file system and transferring the analysis result data obtained to the second database
The computer-readable recording medium according to claim 1,

Images