KR20230099450A - Algorithm for Distributed Parallel Processing of Multi-grid DC distribution Big Data - Google Patents

Abstract

A big data analysis and processing apparatus using the Algorithm for Distributed Parallel Processing of Multi-grid DC distribution Big Data, which divides the population data into a plurality of input subpopulations ( subpopulation) is assigned, respectively, and an output subpopulation including offspring data is obtained by performing a distributed parallel processing algorithm operation based on a fitness value for a predetermined problem for the assigned input subpopulation, a plurality of map modules configured to repeatedly perform a distributed parallel processing algorithm operation a plurality of times using an output subpopulation as the input subpopulation; Reduce configured to receive the output subpopulation from each of the plurality of map modules after the repeated execution of the distributed parallel processing algorithm operation is finished, and to generate output data based on a fitness value from the received output subpopulation ( reduce) module; and an output module configured to output the output data.

Description

Algorithm for Distributed Parallel Processing of Multi-grid DC distribution Big Data}

Embodiments relate to a data analysis and processing device and a data analysis and processing method using an algorithm for distributed parallel processing of multi-grid DC distribution big data (Algorithm for Distributed Parallel Processing of Multi-grid DC distribution Big Data). It is about a data analysis and processing device and method based on a map-reduce distributed parallel processing algorithm with improved processing and analysis performance by reducing data input and output by improving the map and reduce process.

Big data refers to large-scale data that cannot be processed, collected, stored, explored, or analyzed using commonly used software tools. Big data analysis is becoming a concern not only in engineering and science, but also in all fields that collect and manage data, such as business and government. The main issue in big data is to effectively cope with data scale, variety, and processing speed. Hadoop is a well-known open platform for storing, managing, processing, and analyzing big data. Processing speed is a very important factor in big data, and Hadoop's MapReduce technique is used in various forms as parallel distributed processing programming to efficiently process, search, and analyze big data.

On the other hand, Distributed Parallel Processing Algorithm is an optimal solution by repeatedly performing a parallel operator consisting of selection, intersection, and mutation targeting potential solutions to solve a specific mathematical and/or logical problem. is an algorithm to obtain Distributed parallel processing algorithm is an optimization technique that finds an optimal solution based on the principle of survival of the fittest, which is the basis of the evolutionary process in the natural world. According to the distributed parallel processing algorithm, it is possible to find a global optimal solution rather than a local optimal solution because it simultaneously searches in a group of multiple points rather than a single point. Because of these advantages, parallel algorithms are applied to various fields in combination with fuzzy systems and neural networks.

According to one aspect of the present invention, the performance of a conventional Distributed Parallel Processing Algorithm using MapReduce is improved, and large-scale distributed processing is performed for multi-circuit DC distribution network big data. ) can provide a big data analysis and processing device and method using a new Distributed Parallel Processing Algorithm.

An apparatus for analyzing and processing multi-circuit DC distribution network big data using a Distributed Parallel Processing Algorithm according to an embodiment is assigned a plurality of input subpopulations obtained by dividing population data, respectively , Obtain an output subpopulation including offspring data by performing a distributed parallel processing algorithm operation based on a fitness value for a predetermined problem on the assigned input subpopulation, and converting the output subpopulation to the input subpopulation. a plurality of map modules configured to repeatedly perform a distributed parallel processing algorithm operation a plurality of times using a subpopulation; Receive the output subpopulation from each of the plurality of map modules after the repeated execution of the distributed parallel processing algorithm operation is finished, and generate output data based on a fitness value from the received output subpopulation. reduce module; and an output module configured to output the output data.

A data analysis and processing method using MRPGA according to an embodiment includes dividing population data into a plurality of input subpopulations; assigning the plurality of input subpopulations to a plurality of map modules, respectively; obtaining an output subpopulation including descendant data by performing, in each of the plurality of map modules, a distributed parallel processing algorithm operation based on a fitness value for a predetermined problem with respect to the input subpopulation; receiving the output subpopulation obtained from each of the plurality of map modules in a reduce module, and determining output data from the output subpopulation based on the fitness value; and outputting the output data, wherein the obtaining of the output subpopulation may be repeatedly performed a plurality of times by using the output subpopulation obtained from each map module as the input subpopulation.

A computer-readable storage medium according to an embodiment may store instructions for executing the data analysis and processing method using the MRPGA by the computer by being executed by the computer.

A data analysis and processing apparatus and a data analysis and processing method according to an aspect of the present invention improve the map and reduce process of a Distributed Parallel Processing Algorithm based on MapReduce, Compared to the simple Distributed Parallel Algorithm, it has the advantage of fast convergence and efficient parallel processing. In addition, the data analysis and processing device and data analysis and processing method can quickly find an excellent optimal solution according to the number of repetitions of sub-generations, and have excellent performance in processing big data. .

1 is a schematic diagram showing an operational flow of a Hadoop distributed file system.
2 is a schematic diagram illustrating data processing by a MapReduce technique.
3 is a schematic diagram showing a data flow of a simple algorithm (SG A) using a conventional MapReduce.
4 is a schematic block diagram of a data analysis and processing apparatus using a Distributed Parallel Processing Algorithm (MRPGA) according to an embodiment.

A data analysis and processing apparatus and a data analysis and processing method according to embodiments of the present invention apply the MapReduce technique of the Hadoop environment to a distributed parallel algorithm (PGA) in a new way. It is configured to perform analysis and processing of data using a MapReduce Distributed Parallel Processing Algorithm (MRPGA).

Hadoop is an open source software middleware based on MapReduce, a cloud computing platform used by many companies such as Yahoo, Facebook, Amazon, IBM, and NexR. It consists of HDFS (Hadoop Distributed File System), a distributed file system to support a distributed computing framework, and MapReduce, a distributed programming model.

The application of MapReduce in the Hadoop environment consists of jobs, which are units of work performed by clients. A job consists of input data, a MapReduce program, and configuration information. Jobs are also divided into map tasks and reduce tasks for execution. For control of a job execution process, one job tracker and a plurality of task trackers may be used. The job tracker coordinates the execution of all jobs throughout the system by scheduling the tasks to be performed by the task trackers. The task tracker performs the task and sends the overall results of each job to the job tracker. If the task fails at this time, the job tracker reschedules it on another task tracker.

1 is a schematic diagram showing the operational flow of a Hadoop Distributed File System (HDFS) used in embodiments of the present invention.

Referring to FIG. 1, in HDFS, a namenode maintains metadata of data blocks constituting a file and provides block location information to a client through block mapping. For example, the metadata may include file information such as a storage location (eg, home/foo/data) of a file stored in a datanode or the number of copies. A client can store metadata information in the NameNode or request a specific data block from the NameNode. Afterwards, the client accesses the block of a specific data note included in the rack of the datanode to perform the actual operation on the file, and handles reading, writing, copying, etc. of the data block.

The composition of Hadoop cluster is divided into a master node that integrates namenode and job tracker, and a slave node that includes task tracker and datanode. In Hadoop, RPC (Remote Procedure Call) protocol is used for control signals between master and slave nodes, and RPC is also used for communication between master and client. And datanode and client transfer data through TCP socket. Finally, in the MapReduce process, the task tracker delivers the result of the map task to the reduce task, and Hadoop uses HTTP for communication between them.

A distributed file system such as HDFS is a client/server based file system in which a client side can access data stored on a server and process it as if it were stored on its own. Distributed file systems, unlike existing distributed file systems, must be huge and perform well. Through this, it is necessary to efficiently prepare for continuous data increase, and a mechanism that can cope with frequent failures is needed.

A map reduce technique used in MRPGA according to embodiments is a technique for distributing and processing large amounts of data in a widely configured cluster environment such as HDFS. Parallel processing models such as the existing MPI (Message Passing Interface) are suitable for fields requiring high-performance computing, but have many problems when applied to large-scale data processing. MapReduce was created in consideration of requirements such as scalability according to the amount of data and minimization of network traffic when moving data between nodes, and is currently widely used in the field of large-scale data processing. Embodiments perform a wide range of distributed processing using MapReduce, so that a big data environment, such as a data warehouse suitable for large-scale data processing, data mining, and information search, can be used. It can be usefully used in the field.

2 is a schematic diagram showing data processing by the map reduce technique.

Referring to FIG. 2, the input data of NameNote can be divided into a plurality of divided data through a map function and distributed to a plurality of data nodes. For example, each DataNote could be local storage on each individual computer. After sorting the intermediate data distributed to each computer, it can be collected through a reduce function and displayed as output data. Map and reduce functions process and store data in the form of (key, value), which is a pair of key and value similar to a hash data structure. Accordingly, it is easy to distribute and collect data, and data can be sorted into groups using the same key.

3 is a schematic diagram showing the data flow of a simple algorithm (SGA) using a conventional map reducer, which is based on a selectorecombinative algorithm.

In the conventional SGA using MapReduce, an initial model of an algorithm modified to fit Map and Reduce was presented. Referring to FIG. 3, first, the input data is divided into a plurality of map functions and allocated, and each map function calculates the fitness value of the data for a specific problem to obtain data with the best fit (key, value). output in the format of For example, a key output from the map function may be a key of an individual data set, and a value may be a fitness value for each data of the individual data set. Next, offspring data is created by mixing the key-value pairs obtained from each map function by selection and intersection using a parallel operator. The reduce function creates output data by aggregating data with the same key according to the key value of descendant data.

In this way, once the MapReduce process is completed, one generation is completed, and the output data, which is the result of the reduction, becomes the input data of the next generation. The map-reduce process is repeated for a set number of generations (i.e., the number of repetitions). Every time the generation changes, data input/output to the HDFS-based database (DB) is generated by the map and reduce functions.

The SGA using the conventional map reduce described above has a structure in which the map function receives data, calculates the fitness value, stores only the best global solution, and distributes parallel algorithm operation using the reduce function. are using However, according to this method, there is a disadvantage in that data input/output occurs for each generation. That is, if the map function receives population data from the HDFS-based database, processes it, and passes it to the reduce function, the data obtained by performing work in the reduce function (here, data refers to individual values constituting the population) ) is recorded in the HDFS-based DB again. Moving on to the next generation, the map function receives and processes the data group stored in the HDFS-based DB again.

In one aspect of the present invention, an MRPGA technique, a new PGA technique using map reduce, is proposed to improve this point. 4 is a schematic block diagram of a data analysis and processing device using an MRPGA according to an embodiment.

Referring to FIG. 4, the data analysis and processing apparatus according to the present embodiment includes an HDFS-based database (DB) 10, a plurality of map modules 21, 22, 23, ..., 2n, a reduce module ( 30) and an output module 40. In this specification, terms such as "database", "unit", "module", "device" or "system" are intended to refer to a combination of hardware and software driven by the hardware. . For example, the hardware may be a data processing device including a CPU or other processor. Also, software driven by hardware may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

In the block diagram of FIG. 4, each of the DB 10, the map modules 21, 22, 23, ..., 2n, the reduce module 30, and the output module 40 is shown as a separate rectangle separated from each other. However, this is merely functionally dividing the device according to the operation performed in the data analysis and processing device according to one embodiment. That is, some or all of the DB 10, the map module 21, 22, 23, ..., 2n, the reduce module 30, and the output module 40 may be integrated and implemented as a single device, Alternatively, the DB 10, the map modules 21, 22, 23, 2n, the reduce module 30, and the output module 40 are implemented as physically distinct individual devices and operated through communication through a network. may be implemented.

The MRPGA used in this embodiment proposes a map structure in the form of a local search linking selection and intersection by applying and improving the selection-recombination parallel algorithm used in the conventional SGA method described above with reference to FIG. 3 And while passing the excellent individual data to the reduce module 30, the mutation using the map reduce technique was applied. The detailed process of MRPGA according to this embodiment is as follows.

First, as an initialization process, population data is generated by configuring the data to be analyzed as a solution to a given problem, and input data is configured by calculating the fitness value for the given problem for individual solutions included in the population to form an HDFS-based DB (10 ) can be stored. Here, the solution refers to data or a set of data that can be an answer to a given mathematical and/or logical problem, and the specific content or form of the problem and solution related to data analysis according to the embodiments may be limited to any one. No, and may vary from case to case.

Next, the population may be divided into a plurality of subpopulations and assigned to a plurality of map modules 21, 22, 23, ...., 2n as input subpopulations, respectively. That is, the total size of the population is equal to the value obtained by multiplying the size of each subpopulation by the number of map modules 21, 22, 23, 2n. Each map module (21, 22, 23, , 2n) sets each solution as a key in the assigned input subpopulation and uses the fitness value for each solution as a value to input subpopulations. Parallel algorithmic operations can be performed on

In this specification, distributed parallel processing algorithm calculation refers to a calculation process for finding a solution having an optimal fitness value for a given problem while performing selection, intersection, and mutation between each solution based on the survival of the fittest principle. In a parallel algorithm, selection means to select a solution with a relatively high fitness value for a given problem, and crossover means to mutually replace part of the data included in individual solutions between solutions. Since a specific method of calculating a distributed parallel processing algorithm including selection and intersection is well known in the art, a detailed description thereof will be omitted. For example, in one embodiment, a tournament selection method may be used as a selection method and uniform crossover may be used as a crossover method, but is not limited thereto.

In each of the map modules 21, 22, 23, 2n, a solution having a relatively high fitness value for a given problem is selected from among all solutions of the assigned subpopulation, and crossover may be performed on the selected solutions. . Through this selection and intersection process, descendant data is generated, and descendant data can replace existing data in a subpopulation. Next, the map modules 21, 22, 23, 2n may recalculate only the fitness values of descendant data after selection and replacement.

In the embodiments, the map modules 21, 22, 23, 2n perform the above process of selecting a solution having a relatively high fitness value for the input subpopulation, performing intersection, and calculating the fitness value again, It can be repeated as many times as the number of sub-generations. That is, as a result of selection and crossing by the map module 21, 22, 23, 2n, the output subpopulation generated including descendant data is returned to the map module 21, 22, 23, 2n. By becoming an input subpopulation for a subpopulation, an operation for one subgeneration is performed. At this time, the output subpopulation may be input again as an input subpopulation for the map module (21, 22, 23, 2n) from which the output subpopulation was generated, or the map module (21) from which the corresponding output subpopulation was generated. , 22, 23, , 2n) and other map modules (21, 22, 23, 2n) may be input as input subpopulations.

The inventors of the present invention conducted experiments to evaluate the MRPGA method according to the embodiments. The Hadoop version used in the experiment was version 120, and the operating system of the hardware was Ubuntu version 1204. Intel Core (TM) i5 650 (320 GHz) was used as the hardware CPU, and 3 GB of RAM was installed.

The Hadoop mode used in the experiment performed parallel processing by utilizing a pseudo-distribution mode that can configure a virtual distributed environment on a single computing device.

일 때 적합 도 값이 최대인 값을 찾는 문제로, 적합도 값 F(x)는 다음 수학식 1과 같이 산출된다.For the problem to find the optimal solution using the distributed parallel processing algorithm, the OneMax Problem, which is a simple optimization problem that is widely used, was used. this problem is the problem

As a problem of finding the value with the maximum fitness value when , the fitness value F(x) is calculated as in Equation 1 below.

The data analysis and processing apparatus and data analysis and processing method using MRPGA according to the embodiments of the present invention described above are a method of implementing a PGA through a map reduce technique by constructing a distributed environment from a Hadoop environment. It is a new map-reduce-based PGA method that improves the disadvantages of the method. According to the data analysis and processing apparatus and method using the MRPGA according to the embodiments, the speed of the PGA can be improved by reducing the number of data inputs and outputs to the HDFS DB, and the integration and distribution of search areas can be achieved through a single reduce process. You can maintain the diversity of your data with maps. In addition, the performance of the PGA was improved by local search by repeatedly performing the distributed parallel processing algorithm operation for each subpopulation, and efficient variation was implemented using the characteristics of MapReduce.

Data analysis and processing methods according to the embodiments described above may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. The computer-readable recording medium on which the program for implementing the data analysis and processing method according to the embodiments is recorded includes all kinds of recording devices that store data readable by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and carrier wave (for example, transmission through the Internet). It also includes what is implemented in the form of. In addition, computer-readable recording media may be distributed to computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner.

The present invention reviewed above has been described with reference to the embodiments shown in the drawings, but this is only exemplary, and those skilled in the art will understand that various modifications and variations of the embodiments are possible therefrom.

However, such modifications should be considered within the technical protection scope of the present invention.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

10: HDFS DB
21: map module 1
22: map module 2
23: map module 3
2n: map module 2n
30: reduce module
40: output module

Claims (7)

Each of a plurality of input subpopulations obtained by dividing the population data is allocated, and an output subpopulation including descendant data is obtained by performing a distributed parallel processing algorithm operation based on a fitness value for a predetermined problem for the allocated input subpopulations. a plurality of map modules configured to repeatedly perform a distributed parallel processing algorithm operation a plurality of times by using the output subpopulation as the input subpopulation;
Reduce configured to receive the output subpopulation from each of the plurality of map modules after the repeated execution of the distributed parallel processing algorithm operation is finished, and to generate output data based on a fitness value from the received output subpopulation module; and an output module configured to output the output data.
The method of claim 1 , wherein the output module is configured to output output data generated by repeating a distributed parallel processing algorithm operation a predetermined number of times by inputting the output data as the population data to the plurality of map modules. A data analysis and processing device to be used.
2. The data analysis and processing apparatus according to claim 1, further comprising a database based on a Hadoop distributed file system configured to store the population data and the output data.
partitioning the population data into a plurality of input subpopulations;
assigning the plurality of input subpopulations to a plurality of map modules, respectively;
obtaining an output subpopulation including descendant data by performing, in each of the plurality of map modules, a distributed parallel processing algorithm operation based on a fitness value for a predetermined problem with respect to the input subpopulation;
receiving the output subpopulation obtained from each of the plurality of map modules in a reduce module, and determining output data from the output subpopulation based on the fitness value; and outputting the output data, wherein the step of obtaining the output subpopulation is repeatedly performed a plurality of times using the output subpopulation obtained from each map module as the input subpopulation Data analysis, characterized in that and processing methods.
The method of claim 4, wherein the dividing of the population data into a plurality of input subpopulations comprises data based on a Hadoop distributed file system.
6. The method of claim 5, further comprising recording the output data as the population data in the database, wherein the outputting the output data comprises performing a distributed parallel processing algorithm operation by using the output data as the population data. A data analysis and processing method comprising the step of outputting output data generated by repeating a predetermined number of times.
performing by a computing device, wherein the computing device divides the population data into a plurality of input subpopulations;
assigning each of the plurality of input subpopulations to a plurality of map modules of the computing device;
obtaining an output subpopulation including descendant data by performing, in each of the plurality of map modules, a distributed parallel processing algorithm operation based on a fitness value for a predetermined problem with respect to the input subpopulation;
receiving the output subpopulation obtained from each of the plurality of map modules in a reduce module of the computing device, and determining output data from the output subpopulation based on the fitness value; and outputting the output data, wherein the obtaining of the output subpopulation comprises a data analysis and processing method repeatedly performed a plurality of times using the output subpopulation obtained from each map module as the input subpopulation. A computer-readable recording medium, characterized in that stored instructions for performing.

Images