KR102338653B1 - Method of performing distributed grouping processing for each node to minimize shuffling in cluster environment of large data and apparatus thereof - Google Patents

Abstract

A method performed by a device including a cluster consisting of a plurality of nodes comprises the steps of: running a first application; distributing input data and inputting the distributed input data into a plurality of nodes included in the cluster; generating grouping data by performing aggregation on the input data in the plurality of nodes; outputting the grouping data from the cluster including the plurality of nodes in response to the input data; and storing the grouping data in a queue or a buffer associated with the cluster. The step of performing the aggregation on the input data includes a step of generating the grouping data by grouping the input data based on a specific condition within each of the plurality of nodes. Therefore, the present invention can process data efficiently.

Description

Method of performing distributed grouping processing for each node to minimize shuffling in a cluster environment of large data and a device supporting it

The present invention relates to processing of large-capacity data using a clustering technique, and more particularly, to a grouping method in memory for processing large-capacity data in a cluster environment.

In data processing, when data is distributed and processed for each server and then grouped again, a data exchange process between each server called shuffling occurs.

Shuffling a small amount of data is not unreasonable, but if shuffling continuously occurs for a large amount of data, a significant network and memory load occurs.

In order to solve the above problems, an efficient data processing method in a cluster environment is required.

Laid-open Patent Publication No. 10-2020-0178550, 2020.12.18

In order to solve the above problems, the present invention is to provide a method of processing by dividing the steps through grouping (grouping operation) in the memory within each node.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to an embodiment of the present invention for solving the above problems, it is possible to provide a method performed by an apparatus including a cluster consisting of a plurality of nodes.

The method may include executing a first application; distributing input data and inputting it as a plurality of partial data to each of a plurality of nodes included in the cluster; generating grouping data by each of the plurality of nodes performing aggregation on the plurality of partial data; outputting the grouping data from the cluster including the plurality of nodes in response to the input data; and storing the grouping data in a queue or buffer connected to the cluster.

The generating of the grouping data may further include aggregating the plurality of partial data based on a specific condition within each of the plurality of nodes.

The step of aggregating the plurality of partial data may include comparing data values required for grouping among the input data based on the specific condition and grouping them into data having a common value. .

The grouping of data having a common value may include: checking metadata in a header of each data; and when the attribute values of the metadata are the same, determining the grouping target data.

In the grouping of data having a common value, when the properties of the metadata are not the same, the data generation subject, whether to grant the modification authority, the data security level, data processing details, and shuffle according to a predetermined similarity criterion determining whether or not the ring is performed and whether data is similar to the number of times of performing the shuffling; and grouping the plurality of partial data when the similarity score according to the similarity determination exceeds a reference value.

The method includes executing a second application; distributing the stored grouping data and inputting it to a plurality of nodes included in the cluster; and processing the data input to each of the plurality of nodes.

The processing of the data input to each of the plurality of nodes may include omitting the exchange of data between the plurality of nodes.

According to an embodiment of the present invention, there is provided a method performed by an apparatus including a database and a plurality of nodes, the method comprising: determining a configuration environment of the database; selecting a stored procedure if the database is a relational database; and generating grouping data by merging data of each of the plurality of nodes when the database is not a relational database.

The generating of the grouping data may include: generating mapping data having different key values for each data; and collecting data using the mapping data.

The grouping data may include distributed processing without shuffling in the cluster including the node.

According to an embodiment of the present invention, it is possible to provide an apparatus including a cluster consisting of a plurality of nodes.

An apparatus comprising a cluster consisting of the plurality of nodes includes: a memory configured to store data; and one or more processors connected to the memory.

The one or more processors are configured to: execute a first application; distributed input data and input as a plurality of partial data to each of a plurality of nodes included in the cluster; generating grouping data by each of the plurality of nodes performing aggregation on the plurality of partial data; output the grouping data from the cluster including the plurality of nodes in response to the input data; and store the grouping data in a queue or a buffer associated with the cluster, wherein when the processor generates the grouping data, the plurality of partial data is stored in each of the plurality of nodes based on a specific condition. It may be configured to include drawing.

According to an embodiment of the present invention, it is possible to provide an apparatus including a database and a plurality of nodes.

The apparatus includes a memory configured to store data; and one or more processors connected to the memory.

The one or more processors are configured to: determine a configuration environment of the database; if the database is a relational database, select a stored procedure; and when the database is not a relational database, each of the plurality of nodes may be configured to merge data to generate grouping data.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention, data can be efficiently processed by solving a problem in which performance deteriorates due to a large amount of shuffling in the cluster.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide various embodiments and, together with the detailed description, explain technical features of the various embodiments.
1 is a diagram showing cluster distribution processing of data.
FIG. 2 is a diagram illustrating data gathering from multiple nodes to one node due to shuffling occurring in a cluster.
3 is a diagram illustrating a process in which data subjected to cluster distribution processing according to the present invention becomes self-aggregation.
4 is a diagram illustrating a conceptual diagram of a cluster and self-aggregation according to the present invention.
5 is a conceptual diagram illustrating a process of data processing according to the present invention.
6 is a diagram illustrating an example of a data processing process according to a DB configuration environment according to the present invention.
7 is a diagram illustrating an example of an apparatus in which the present invention can be implemented.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe the correlation between a component and other components. Spatially relative terms should be understood as terms including different directions of components during use or operation in addition to the directions shown in the drawings. For example, when a component shown in the drawing is turned over, a component described as “beneath” or “beneath” of another component may be placed “above” of the other component. can Accordingly, the exemplary term “below” may include both directions below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary embodiments of various embodiments, and is not intended to represent the only embodiments.

1 is a diagram illustrating cluster distributed parallel processing of data.

A cluster (or computer cluster) is a concept that connects multiple computers through a network so that they can be used as one computer. Computer clusters are used in various fields such as computer operating systems, computer hardware, and statistical data. The components of a computer cluster are usually connected by a high-speed local area network. Clusters generally boast better performance and reliability than single machines, and are much more efficient than single machines.

In FIG. 1 , each computing device (eg, a computer) is represented as one node, and the storage system includes a database (DB). 1 illustrates a cluster distributed processing process of data by storing data in each node of a cluster through a command called INPUT.

If you compare the amount of data (or information) that one server can process with the amount that multiple servers can process at the same time, the latter will usually be more. However, it is not always efficient to configure and apply a cluster environment to all tasks processed by the server.

FIG. 2 is a diagram illustrating data gathering from multiple nodes to one node due to shuffling occurring in a cluster.

Shuffling refers to data exchange between nodes. For example, an aggregate function (aggregation) operation such as 'group by' in Select SQL of DB (database) corresponds to this. After all, shuffling is also an operation for grouping data.

Data stored in each node of FIG. 2 is collected into one node through shuffling. Specifically, data is individually input to each node and stored as one node through shuffling, which is output through an OUTPUT command and enters the storage system. For example, referring to FIG. 2 , data DATA distributed and stored in each of the first node Node1, the second node Node2, and the N-th node NodeN is exchanged between nodes through a SHUFFLLING operation. may be performed, and as a result, data may be aggregated into the first note Node1.

A node cannot complete an operation with only the data it has, but completes the operation by adding the data of other nodes. In order to combine data of each node, data may be aggregated in one node by exchanging data between nodes through a network. A series of operations may be performed at the node receiving the data, such as the need to add the received data back to the memory.

This operation collects and calculates data that has been distributed across multiple nodes in one node again, and may consume a lot of resources such as network and memory. In other words, shuffling occurs in the cluster and data is gathered from multiple nodes to one node, causing a bottle-neck, which prevents efficient use of computing power and causes cluster performance degradation.

Therefore, there is a need for a method to solve the problem of performance degradation due to a lot of shuffling as described above.

3 is a diagram illustrating a process in which data subjected to cluster distribution processing according to the present invention becomes self-aggregation.

When shuffling occurs a lot, it is difficult for one application to process all tasks at once. This is because the cluster processes data distributed troop processing, so the next operation in the cluster is performed only after the data exchange operation including shuffling is finished. Any delay caused by this will affect the overall operation. In order to avoid shuffling as much as possible, it is much more efficient to divide the data into steps through grouping operations that can be done within each node's memory.

Referring to FIG. 3 , the data is grouped first with logic in the data. This grouped data is stored (or referred to as a STACK) in a queue (or buffer/database) as it is. This process is called self-aggregation.

Here, the queue is one of the basic data structures of the computer, and refers to a format in which data inserted first is stored in a FIFO (First In First Out) structure. Queue is not a cluster environment, and grouped data is stored separately.

Types of databases (databases, DBs) include relational databases (RDBs) and memory DBs. When selecting a DB procedure, only RDB is used, and other DB types are used when a node loads or merges grouped data.

4 is a diagram illustrating a conceptual diagram of a cluster and self-aggregation according to the present invention.

Data is input from the storage system to the first cluster by the input command. Data input to the first cluster is distributed and input to each node, and each node performs a primary grouping operation on the data it owns in its own memory.

After performing the first grouping operation by each node itself, the grouped data is stored in a queue or buffer.

A merge operation of reading and summing the first grouped data stored in the queue or buffer is performed.

The merged data is distributed and inputted to each node again in the second cluster. Therefore, since the effect of shuffling within the node is generated and not shuffling for each cluster, it is possible to prevent performance degradation due to the shuffling for each cluster.

5 is a conceptual diagram showing a process of data processing according to the present invention.

The device including the cluster consisting of a plurality of nodes executes a first application (S501).

The device distributes input data and inputs it to a plurality of nodes included in the cluster (S503). In this case, the input data is input as a plurality of partial data to each of the plurality of nodes.

In response to the input data, the data output from the cluster is grouped to generate grouping data (S505). Specifically, each of the plurality of nodes generates grouping data by performing aggregation on the plurality of partial data. In this case, each of a plurality of nodes may aggregate a plurality of partial data based on a specific condition.

The step of aggregating the plurality of partial data may include comparing data values that need to be grouped among the input data based on the specific condition and grouping them into data having a common value.

The grouping into data having a common value may include checking metadata in a header of each data.

As a result of checking the metadata, if the attribute values of the metadata are the same, each node determines data having the same attribute values of the metadata as grouping target data.

As a result of checking the metadata, if the properties of the metadata are not the same, each node determines the subject of data creation, whether to grant the right to modify, the security level of the data, whether to perform data processing, whether to perform shuffling, and whether to perform shuffling according to a predetermined similarity criterion. It is determined whether the data is similar to the number of times. In addition, when a similarity score according to a similarity determination of each node exceeds a reference value, each node groups a plurality of partial data.

In response to the input data, grouping data is output from the cluster including a plurality of nodes and stored in a queue or a buffer (S507). The stored data may be output immediately, or may be distributed and inputted to and processed by a plurality of nodes included in the cluster by executing the second application (S509).

According to the above process, since shuffling (data exchange between nodes) does not occur between a plurality of nodes, the efficiency of distributed parallel processing of data is improved.

Grouping data (primary grouping data) stored in the queue or buffer is immediately output or processed while the second application is executed. The second application execution step will be described in detail below with reference to FIG. 6 .

6 is a diagram illustrating an example of a data processing process according to a DB configuration environment according to the present invention, and is a diagram specifically illustrating the second application execution step of FIG. 5 .

FIG. 6 shows a case in which a DB configuration environment is a relational database (RDB) or a case where the DB configuration environment is not an RDB (else).

The device including the DB executes the second application (S601), and determines whether the DB configuration environment is the RDB (S602).

If the DB configuration environment is not RDB, the node loads the primary grouping data and merges the data (S603).

Specifically, in order to perform data merging, mapping data having different key values for each data is generated (S604). If a collection such as a map with keys does not overlap, the data loaded in the memory of each node can be grouped by itself. Here, the map collection has a structure that stores an Entry object composed of a key and a value. Both keys and values are objects and, as noted above, keys cannot be duplicated.

Data is merged by being collected based on different mapping data (S605). In this case, since data is merged for each node, distributed processing without shuffling is possible.

When the DB configuration environment is RDB, a DB procedure is performed (S606).

RDB is a simple principle computerized information database that tabulates simple relationships between keys and values, and stores data in the form of tables.

A DB procedure (or stored procedure) stores a procedure that organizes a series of operations on a database in a relational database management system, also called a persistent storage module.

According to the present invention, the following data processing process is proposed.

1. Separate grouping processing for data to be shuffled within a node. Each node performs primary grouping in its own memory only for the data it has.

2. Prepare the queue/buffer for the next step. After completing the primary grouping for each node, the primary grouped data is stored in the queue/buffer.

3-1. In the second application, read and merge the first grouped data from the queue/buffer. This work can be done as a separate application, or if the queue/buffer is used as a DB, it can be processed through the 'Group By' function (set function operation) of the DB.

3-2. Alternatively, the primary grouped data stored in the queue/buffer may be output.

4. Divide the merged data for each node again, and repeat the above process.

By grouping data in each node's own memory and storing the grouped data in a queue/buffer, shuffling occurs in the cluster, and data is collected from multiple nodes to one node, thereby reducing the bottleneck. can reduce what happens. This allows efficient use of computing power and prevents cluster performance degradation.

7 is a diagram illustrating an example of an apparatus in which the present invention can be implemented.

Referring to FIG. 7 , the device may include an input/output unit 310 , a communication unit 320 , a sensing unit 330 , a database 340 , and a processor 350 .

The input/output unit 310 may be various interfaces or connection ports that receive user input or output information to the user. The input/output unit 310 may be divided into an input module and an output module, and the input module receives a user input from a user. The user input may be made in various forms including a key input, a touch input, and a voice input. Examples of input modules that can receive such user input include a traditional keypad, keyboard, and mouse, as well as a touch sensor that detects a user's touch, a microphone that receives a voice signal, a camera that recognizes gestures through image recognition, A proximity sensor composed of an illuminance sensor or infrared sensor that detects a user's approach, a motion sensor that recognizes a user's motion through an acceleration sensor or a gyro sensor, and other various types of input means that detect or receive various types of user input It is a comprehensive concept that includes all Here, the touch sensor may be implemented as a piezoelectric or capacitive touch sensor for detecting a touch through a touch panel or a touch film attached to the display panel, an optical touch sensor for detecting a touch by an optical method, and the like. In addition, the input module may be implemented in the form of an input interface (USB port, PS/2 port, etc.) that connects an external input device that receives a user input instead of a device that detects a user input by itself. In addition, the output module can output various information and provide it to the user. The output module is a comprehensive concept that includes a display that outputs an image, a speaker that outputs a sound (and/or an amplifier connected thereto), a haptic device that generates vibration, and other various types of output means. In addition, the output module may be implemented in the form of a port-type output interface for connecting the above-described individual output means.

For example, the display-type output module may display text, still images, and moving images. The display includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flat panel display (FPD), and a transparent display. display), a curved display, a flexible display, a three-dimensional display, a holographic display, a projector, and various types of devices capable of performing other image output functions. It is a concept meaning an image display device in a broad sense that includes all. Such a display may be in the form of a touch display integrally formed with the touch sensor of the input module.

The communication unit 320 may communicate with an external device. Accordingly, the device (device) may transmit/receive information to and from an external device through the communication unit. For example, the device may communicate with an external device so that information stored and generated in the illegal parking/stop warning system is shared using the communication unit.

Here, communication, that is, transmission and reception of data may be performed by wire or wirelessly. To this end, the communication unit includes a wired communication module that accesses the Internet through a local area network (LAN), a mobile communication module that accesses a mobile communication network through a mobile communication base station and transmits and receives data, and a wireless local area network (WLAN) such as Wi-Fi. A short-distance communication module using an area network communication method or a wireless personal area network (WPAN) communication method such as Bluetooth or Zigbee, or a global navigation satellite system (GNSS) such as GPS (Global Positioning System) ) using a satellite communication module or a combination thereof. A wireless communication technology used for communication may include a Narrowband Internet of Things (NB-IoT) for low-power communication. In this case, for example, NB-IoT technology may be an example of LPWAN (Low Power Wide Area Network) technology, and may be implemented in standards such as LTE Cat (category) NB1 and/or LTE Cat NB2, It is not limited. Additionally or alternatively, a wireless communication technology implemented in a wireless device according to various embodiments may perform communication based on LTE-M technology. In this case, as an example, the LTE-M technology may be an example of an LPWAN technology, and may be called various names such as enhanced machine type communication (eMTC). For example, LTE-M technology is 1) LTE CAT 0, 2) LTE Cat M1, 3) LTE Cat M2, 4) LTE non-BL (non-Bandwidth Limited), 5) LTE-MTC, 6) LTE Machine Type Communication, and/or 7) may be implemented in at least one of various standards such as LTE M, and is not limited to the above-described name. Additionally or alternatively, a wireless communication technology implemented in a wireless device according to various embodiments may include at least one of ZigBee, Bluetooth, and Low Power Wide Area Network (LPWAN) in consideration of low power communication. may include, and is not limited to the above-mentioned names. For example, the ZigBee technology can create PAN (personal area networks) related to small/low-power digital communication based on various standards such as IEEE 802.15.4, and can be called by various names.

The identification unit 330 includes a camera that recognizes an object through image recognition, a detection sensor that detects an object approach, and various types of identification/sensing means that detect or receive various types of external inputs. could be a concept. The identification unit may be understood to be the same as the input module in the input/output unit 310 and/or may be understood to be separate from the input module. The identification unit 330 includes a geomagnetic sensor, an acceleration sensor, a temperature/humidity sensor, an infrared sensor, a gyroscope sensor, a location sensor (eg, GPS), a barometric pressure sensor, a proximity sensor, and an RGB sensor (illuminance). sensor), a lidar sensor, an illuminance sensor, and a current sensor may further include, but is not limited thereto. Since a function of each sensor can be intuitively inferred from the name of a person skilled in the art, a detailed description thereof will be omitted.

The database 340 may store various types of information. A database can store data temporarily or semi-permanently. For example, the database includes an operating program (OS) for driving the first device and/or the second device, a program or application for generating data or Braille for hosting a website (eg, a web application) ) may be stored. In addition, the database may store the modules in the form of computer code as described above.

Examples of the database 340 include a hard disk (HDD), a solid state drive (SSD), a flash memory, a read-only memory (ROM), a random access memory (RAM), and the like. This can be. Such a database may be provided in a built-in type or a detachable type.

The processor 350 controls the overall operation of the device (device). To this end, the processor 350 may perform calculation and processing of various types of information and may control operations of components of the first device and/or the second device. For example, the processor 350 may execute a program or application for a method of performing distributed grouping processing for each node in order to minimize shuffling in a cluster environment of large data. The processor 350 may be implemented as a computer or a similar device according to hardware software or a combination thereof. In hardware, the processor 350 may be provided in the form of an electronic circuit that processes electrical signals to perform a control function, and in software, it may be provided in the form of a program that drives the processor 240 in hardware. Meanwhile, in the following description, unless otherwise specified, the operations of the first device and/or the second device may be interpreted as being performed under the control of the processor 350 . That is, when the modules required for the method of performing distributed grouping processing for each node in order to minimize shuffling in a cluster environment of large data are executed, the modules are executed by the processor 350 as the first device and/or the second device. It can be interpreted as controlling to perform the operations of

In summary, the present invention may be implemented through various means. For example, various embodiments may be implemented by hardware, firmware, software, or a combination thereof.

In case of implementation by hardware, the method according to the present invention may include one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays), a processor, a controller, a microcontroller, a microprocessor, and the like.

In the case of implementation by firmware or software, the method according to the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described below. For example, the software code may be stored in a memory and driven by a processor. The memory may be located inside or outside the processor, and data may be exchanged with the processor by various known means. The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

According to the present invention, data stored in the database 340 is input to each of the plurality of nodes included in the cluster. The processor 350 is configured to generate grouping data by performing aggregation on the input data in a plurality of nodes. Specifically, the processor 350 performs self-aggregation by generating the grouping data by grouping input data based on a specific condition in each of the plurality of nodes. The processor 350 outputs grouping data generated in the plurality of nodes, and the output grouping data is stored in the database 340 connected to the cluster.

The processor 350 may determine whether the database 340 configuration environment is an RDB. When the database configuration environment is not the RDB, the data of the database 340 is loaded into the node, and the processor 350 merges the loaded data. Only when the database 340 configuration environment is RDB, the processor 350 executes the DB procedure.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains know that the present invention may be embodied in other specific forms without changing the technical spirit or essential features thereof. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

310: input / output unit
320: communication unit
330: identification unit
340: database
350: processor

Claims (11)

In the method performed by an apparatus including a cluster consisting of a plurality of nodes,
running a first application;
distributing input data and inputting it as a plurality of partial data to each of a plurality of nodes included in the cluster;
aggregating the plurality of partial data based on a specific condition within each of the plurality of nodes;
generating grouping data based on the aggregation result;
outputting the grouping data from the cluster including the plurality of nodes in response to the input data; and
Storing the grouping data in a queue or buffer associated with the cluster,
The step of aggregating the plurality of partial data comprises:
Comprising the step of grouping data having a common value by comparing the data values (values) required to be grouped among the input data based on the specific condition,
The step of grouping the data having the common value is,
Check the metadata in the header of each data, and group them in different ways depending on whether the attribute values of the metadata are the same,
When the attribute values of the metadata are the same, it is determined as grouping target data,
If the attribute values of the metadata are not the same, whether the data is similar to the data creation subject, whether to grant the modification authority, the security level of the data, data processing details, whether to perform shuffling, and whether to perform the shuffling according to a predetermined similarity criterion and grouping the plurality of partial data when a similarity score according to the similarity determination exceeds a reference value. delete delete delete According to claim 1,
running a second application;
distributing the stored grouping data and inputting it to a plurality of nodes included in the cluster; and
The method further comprising the step of processing the data input to each of the plurality of nodes. 6. The method of claim 5,
The method, further characterized in that the step of processing the data input to each of the plurality of nodes comprises omitting the exchange of data between the plurality of nodes. According to claim 1,
determining a configuration environment of a database included in the device;
selecting a stored procedure if the database is a relational database; and
If the database is not a relational database, each of the plurality of nodes merging data further comprising generating grouping data. 8. The method of claim 7,
The step of generating the grouping data includes:
generating mapping data having different key values for each data; and
The method further comprising the step of aggregating data using the mapping data. 9. The method of claim 8,
The grouping data is characterized in that distributed processing without shuffling in the cluster including the node. In the device comprising a cluster (cluster) consisting of a plurality of nodes,
a memory configured to store data; and
one or more processors coupled to the memory;
The one or more processors include:
run a first application;
distributed input data and input as a plurality of partial data to each of a plurality of nodes included in the cluster;
aggregating the plurality of partial data based on a specific condition within each of the plurality of nodes;
generate grouping data based on the aggregation result;
output the grouping data from the cluster including the plurality of nodes in response to the input data; and
Storing the grouping data in a queue or buffer associated with the cluster,
When the processor aggregates the plurality of partial data, data having a common value is grouped by comparing data values that require grouping among the data input based on the specific condition,
When the processor groups the data having the common value, it checks the metadata in the header of each data, and groups in different ways depending on whether the attribute values of the metadata are the same,
If the attribute values of the metadata are the same, it is determined as grouping target data,
If the attribute values of the metadata are not the same, whether the data is similar to the data creation subject, whether to grant the modification authority, the security level of the data, data processing details, whether to perform shuffling, and whether to perform the shuffling according to a predetermined similarity criterion , and grouping the plurality of partial data when a similarity score according to the similarity determination exceeds a reference value. 11. The method of claim 10,
The processor is
determining a configuration environment of a database included in the device;
if the database is a relational database, select a stored procedure; and
If the database is not a relational database, each of the plurality of nodes is configured to merge data to generate grouping data.

Images