KR101622810B1 - System and method for distributing big data - Google Patents

Abstract

A big data distribution system according to one aspect of the present invention which can improve the processing speed of a job includes a plurality of data nodes in which data is stored; A plurality of job management systems coupled to at least one data node to store collected data in a data node; (N > 1) among the plurality of data nodes as a first storage data node for storing the collected data when a collecting operation for the collected data is requested, And a work distribution system for distributing the collection job to the job management system connected to each of the N first storage data nodes.

Description

[0001] SYSTEM AND METHOD FOR DISTRIBUTING BIG DATA [0002]

The present invention relates to a data distribution system, and more particularly, to a big data distribution system and method for efficiently collecting and analyzing a large amount of data.

In recent years, data has grown exponentially as the use of social services and the volume of smart devices have increased. As a result, there is an increasing interest in large data technology for collecting and analyzing large amounts of data, extracting meaningful data, and analyzing the results.

1 is a diagram showing a conventional big data system.

Referring to FIG. 1, a conventional Big Data System 100 includes a collection system 110, an analysis system 120, a task management system 130, and a plurality of storage systems 140.

This conventional large data system 100 largely performs an acquisition task of storing data in the storage system 140 and an analysis task of analyzing the data stored in the storage system 140.

First, the collection operation is performed through the collection system 110, the job management system 130, and the storage system 140.

More specifically, the collection system 110 collects data and transmits the collected data to the task management system 130. [

When the collected data is received from the collection system 110, the task management system 130 selects one of the plurality of storage systems 140 and stores the collected data in the selected storage system 140.

On the other hand, the analysis work is performed through the analysis system 120, the work management system 130, and the storage system 140.

More specifically, the analysis system 120 requests the analysis data required for analysis to the work management system 130, and analyzes the received analysis data upon receiving the analysis data requested from the work management system 130. Then, the analysis system 120 transmits the analysis result data to the job management system 130.

The task management system 130 requests data from the storage system 140 in which the analysis data is stored when the analysis data request is received from the analysis system 120. [ When the work management system 130 receives the analysis data from the storage system 140, the work management system 130 transmits the received analysis data to the analysis system 120.

In addition, when the task management system 130 receives analysis result data from the analysis system 120, it selects one of the plurality of storage systems 140 and stores the analysis data in the selected storage system 140.

The conventional big data distribution system 100 performs both the collection operation and the analysis operation in one job management system 130 as described above. Accordingly, in the conventional big data distribution system 100, there is a problem that the load on the job management system 130 increases and the job processing speed drops in proportion to the amount of data.

In addition, since the conventional large data distribution system 100 sets the minimum storage units of the plurality of storage systems 140 to be the same regardless of the data size, there is another problem that the storage space is wasted.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a big data distribution system and method capable of improving the work processing speed of a job management system.

It is another object of the present invention to provide a big data distribution system and method capable of efficiently using a storage system.

According to an aspect of the present invention, there is provided a big data distribution system comprising: a plurality of data nodes storing data; A plurality of job management systems coupled to at least one data node to store collected data in a data node; And when a collecting job for collected data is requested, selecting N among the plurality of data nodes as a first storing node for storing the collected data, and selecting, among the plurality of job management systems, And a task distribution system for distributing the collection task to at least one task management system connected to each of the nodes.

According to another aspect of the present invention, there is provided a method of distributing big data, comprising: receiving a collected data collecting operation request; Selecting N of the plurality of data nodes based on at least one of the data type and size of the collected data and determining the selected N data nodes as a first storage node to store the collected data; And distributing the collection task by selecting at least one of the job management systems connected to each of the N first storage nodes.

According to another aspect of the present invention, there is provided a method of distributing big data comprising: receiving an analysis data request and analysis job request from an analysis system; Searching at least one storage node among the plurality of data nodes for storing the analysis data and selecting at least one of the search management systems connected to the at least one storage node to distribute the analysis task; And receiving the analysis data from the at least one job management system and collecting the received analysis data and transmitting the collected analysis data to the analysis system.

According to another aspect of the present invention, there is provided a method for distributing big data, comprising: receiving an analysis data request and analysis job request; Searching for M second storage nodes storing the analysis data among the plurality of data nodes and distributing the analysis task by selecting at least one of the job management systems connected to the M second storage nodes; And receiving the analysis data from the at least one job management system and collecting the received analysis data and transmitting the collected analysis data to the analysis system.

According to the present invention, by distributing the collection job and the analysis job to a plurality of job management systems, it is possible to prevent a job from concentrating on one job management system, and to improve a job processing speed by processing a plurality of jobs in parallel .

In addition, according to the present invention, a plurality of job management systems connected to at least one storage system have another effect that the system expansion is facilitated as the amount of data increases.

Further, according to the present invention, there is another effect that data can be prevented from occupying unnecessarily large storage space by determining the number of data to be stored according to the data type.

In addition, according to the present invention, there is another effect that efficiency can be improved by minimizing the empty space in the storage space by determining the storage unit according to the data size.

1 is a diagram showing a conventional big data system.
2 is a diagram schematically illustrating a big data distribution system according to an embodiment of the present invention.
3 is a block diagram illustrating the task distribution system of FIG.
FIG. 4 is a block diagram for explaining the collection task management unit of FIG. 3. FIG.
5 is a diagram for explaining an example of distributing a collection job to at least one job management system.
FIG. 6 is a block diagram for explaining the analysis task management unit of FIG. 3. FIG.
7 is a view for explaining an example of distributing analysis work to at least one work management system.
8 is a flowchart illustrating a method of distributing big data according to an embodiment of the present invention.
Figure 9 is a flow chart illustrating one embodiment of a method for determining a storage node to store collected data.
10 is a flow chart illustrating another embodiment of a method for determining a storage node to store collected data.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a diagram schematically illustrating a big data distribution system according to an embodiment of the present invention.

2, a big data distribution system 200 according to an embodiment of the present invention includes a collection system 210, an analysis system 220, a task management system 230, a storage system 240, (250).

Such a big data distribution system 200 performs collection work and analysis work. The collection task is a series of tasks for collecting and storing data and is performed through the collection system 210, the task distribution system 250, the task management system 230, and the storage system 240.

Meanwhile, the analysis job is a series of operations for analyzing the stored data and storing the analysis results. The analyzing system 220, the work distribution system 250, the work management system 230, and the storage system 240 Lt; / RTI >

Hereinafter, the components for performing the collection operation or the analysis operation will be described in detail.

First, the acquisition system 210 collects data. At this time, the data to be collected includes regular data, semi-regular data and unstructured data.

Formal data represents data stored in fixed fields, such as relational databases, spreadsheets, and the like. Unstructured data refers to data that is not stored in a fixed field, such as text documents, images, moving images, voice data, and the like. Semi-structured data refers to data that is not stored in a fixed field but contains metadata or schemas, such as XML, HTML, and text.

This collection system 210 requests the job while transferring collected data collected from various collectors (not shown) to the job distribution system 250.

The analysis system 220 then analyzes the data. The analysis system 220 reads the data stored in the storage system 240 and performs data analysis. To this end, the analysis system 220 requests the work distribution system 250 for analysis data and work to be analyzed.

Next, the job management system 230 performs a collection job or an analysis job. More specifically, the job management system 230 stores the data collected by the collection system 210 in the storage system 240 when the job distribution system 250 is assigned the collection job.

On the other hand, when the task management system 230 is assigned an analysis task by the task distribution system 250, the analysis system 220 requests the analysis data requested by the analysis system 220 to the storage system 240. When the task management system 230 receives the analysis data from the storage system 240, the task management system 230 transmits the received analysis data to the task distribution system 250.

As shown in FIG. 2, the task management system 230 includes a plurality of tasks, and each task management system 230 is connected to at least one storage system 240. At this time, the plurality of job management systems 230 may share one or more storage systems 240.

For example, the first job management system 231 may be connected to the first through third storage systems 241, 242, and 243. In addition, the second job management system 232 may be connected to the second through fourth storage systems 242, 243, and 244. Accordingly, the first and second job management systems 231 and 232 share the second and third storage systems 242 and 243, respectively.

Meanwhile, in one embodiment, the job management system 230 may perform both the collection job and the analysis job, but in another embodiment, the job management system 230 may perform the collection job, and the analysis A work management system may be included. Hereinafter, it is assumed that one job management system 230 can perform both a collection job and an analysis job for convenience of explanation.

Next, the storage system 240 includes a plurality of data nodes in which data is stored. At this time, the data node includes a plurality of stores for physically storing data, and a server for managing the plurality of stores.

Hereinafter, a data node and a storage data node are mixed. At this time, the storage data node means a data node storing the collected data or a data node storing analyzed data.

The task distribution system 250 then selects at least one of the plurality of task management systems 230 to distribute the task. Here, the tasks include the collection tasks requested by the collection system 210, and the analysis tasks requested by the analysis system 220.

Hereinafter, the job distribution system 250 will be described in more detail with reference to FIGS. 3 to 7. FIG.

3 is a block diagram illustrating the task distribution system of FIG.

3, the job distribution system 250 includes a job type determination unit 310, a collection job management unit 320, a storage location database 330, and an analysis job management unit 340, And may further include a monitoring unit 350.

First, the storage location database 330 includes location information of the data stored in the storage system 240. At this time, the location information includes at least one of the storage system identification information, the data node identification information, the storage identification information, and the storage location information.

Next, the operation type determination unit 310 determines the operation type. At this time, the operation type is classified into a collection operation and an analysis operation.

More specifically, when the job request is received together with the collected data from the collecting system 210, the job type determining unit 310 determines that the job is a collecting job, and transmits the collected data to the collecting job managing unit 320 .

On the other hand, when the analysis data request and the work request are received from the analysis system 220, the work type determination unit 310 determines the work as an analysis work and requests the analysis work management unit 320 for analysis data.

Next, the collection job management unit 320 determines the storage number and storage location for the collected data. The collection task management unit 320 selects at least one task management system 230 based on the determined storage number and storage position, and distributes the collected task to at least one selected task management system 230.

Hereinafter, the collection job management unit 320 will be described in more detail with reference to FIG.

FIG. 4 is a block diagram for explaining the collection task management unit of FIG. 3. FIG.

4, the collection task management unit 320 includes a data type setting unit 410, a data type determination unit 420, a storage number setting unit 430, a storage number determination unit 440, a data size determination unit 450, a storage unit setting unit 460, a storage position determining unit 470, and a collecting task distributor 480.

First, the data type setting unit 410 sets a data type. Data types are distinguished using at least one of the type, importance, and utilization of data. For example, the data type may be set using a data type table such as the following [Table 1].

Type of data Importance of data Usage of data Orthopedic (100) (10) Phase (1) The semi- (20) (2) Atypical (300) (30) (3)

Referring to Table 1, the data type setting unit 410 may divide the data type into a regular type, a semi-regular type, and an irregular type and set 100, 200, and 300 for each. In addition, the importance of data can be divided into upper, middle, and lower, and 10, 20, and 30 can be set for each. Then, the utilization of data can be divided into upper, middle, and lower, and 1, 2, and 3 can be set for each.

Meanwhile, the data type setting unit 410 may set the sum of the set values to the data type. For example, the data type is 111, the data type is regular, the importance is high, and the utilization rate is data type 111.

Next, the data type determination unit 420 determines the data type for the collected data. More specifically, the data type determination unit 420 determines the type, importance, and utilization of the collected data, and confirms the set values for each element in the data type table. Then, the data type determination unit 420 determines that the value obtained by summing the determined setting values is a data type.

 Next, the storage number setting unit 430 sets the storage number according to the data type. Here, the number of storage means the number of data nodes for storing the corresponding data, and indicates the number of the same data stored in the storage system 240.

For example, the number of storage can be set as shown in Table 2 below.

Data type Number of storage (number of data nodes) 111,112,121,122,211,221,311,321 5 113,123, 212,213,222,223,312,313,322 4 131, 132, 133, 231, 232, 233, 323, 331, 332, 333 3

Referring to Table 2, the storage number setting unit 430 may set the storage number to 5 for the data of which the data type is 111, 211, 311 and whose importance and utilization are high. On the other hand, the storage number setting unit 430 can set the storage number to 3 for the data having the importance and utilization degree such as 133, 233, and 333.

Accordingly, data having high importance and utilization is stored in a relatively large number of data nodes, and data having low importance and utilization is stored in a relatively small number of data nodes, so that the storage space can be efficiently used.

Next, the storage number determination unit 440 determines the storage number of collected data. More specifically, when the data type for the collected data is determined, the stored number determining unit 440 determines the stored number based on the determined data type.

Next, the data size determination unit 450 determines the size of the collected data.

Next, the storage unit setting unit 460 sets a minimum storage unit according to the size of the data. Herein, the minimum storage unit refers to a fixed block unit of the storage space, and represents a storage unit of the data node.

For example, the minimum storage unit may be set as shown in Table 3 below.

Data Size Data node group Minimum storage unit Node number 0MB ≤ X <10MB A 1MB 1, 2, 3, 4, ... 10MB ≤ X <50MB B 10MB 11, 12, 13, ... 50MB ≤ X <100MB C 50MB 21, 22, 23, ... 100MB ≤ X <500MB D 100MB 31, 32, 33, ... 500MB≤X E 500MB 41, 42, 43, ...

Referring to Table 3, the minimum storage units can be classified into five according to the data size. If the data size is greater than 0 MB and less than 10 MB, set the minimum storage unit to 1 MB, and if the data size is greater than or equal to 500 MB, set the minimum storage unit to 500 MB.

Meanwhile, the data node group may include a plurality of data nodes having the same minimum storage unit. For example, the data node group A may include a plurality of data nodes 1, 2, 3, ... with a minimum storage unit of 1 MB.

Next, the storage location determination unit 470 determines a storage location based on the size of the collection data determined by the data size determination unit 450 and the storage number of the collection data determined by the storage number determination unit 440 . At this time, the storage location refers to the location where the collected data is stored, and represents a specific data node.

More specifically, the storage position determination unit 470 determines the data node group based on the size of the collected data. Then, the storage location determination unit 470 selects at least one of the plurality of data nodes included in the determined data node group, and determines the selected at least one data node as the storage data node.

At this time, the storage position determination unit 470 may select at least one data node based on the number of stored data.

In one embodiment, the storage location determiner 470 may select as many storage nodes as the number of data nodes included in the data node group. For example, it is assumed that the storage number determination unit 440 determines the storage number of the collection data having the data size of 25 MB to be three. At this time, the storage location determination unit 470 can select three of the plurality of data nodes included in the data node group B.

In another embodiment, the storage location determiner 470 may select at least one data node based on the free space and the number of stores of the data node.

More specifically, the storage location determination unit 470 identifies a free space for each of the plurality of data nodes included in the data node group. Then, the storage position determination unit 470 extracts spare nodes having a spare space larger than the collected data size, and compares the extracted spare nodes with the stored number.

At this time, if the number of spare nodes is equal to or greater than the storage number, the storage location determination unit 470 can select as many storage nodes as the number of spare nodes. On the other hand, if the number of spare nodes is smaller than the storage number, the storage location determiner 470 can select all spare nodes.

Meanwhile, the storage location determination unit 470 may select at least one of the plurality of data nodes based on the free space of the data node. In one embodiment, the storage location determiner 470 may select at least one data node in order of increasing free space.

The storage location determination unit 470 stores the storage data node information in the storage location database 330 when the storage data node to store the collected data is determined. At this time, the stored data node information may include storage system identification information, data node identification information, storage identification information, and storage location information.

Next, the collection task distributor 480 replicates the collection data to generate collection data of the number of stores -1, and distributes the collection task for each collection data to at least one job management system 230. [

More specifically, the collecting task distributor 480 replicates the collected data and generates collected data of the stored number -1.

In addition, the collection task distributor 480 identifies the task management system 230 connected to the storage data node determined by the storage location determiner 470. Then, the collection task distributor 480 transmits the collection task request together with the collected data and the stored data node information to the identified task management system 230.

At this time, if there are a plurality of job management systems 230 connected to specific data nodes, the collection job distribution unit 480 selects one of the plurality of job management systems 230 to distribute the collection job.

On the other hand, the collection task distributor 480 can select one of the plurality of job management systems 230 based on the status information on the job management system 230. At this time, the status information may include at least one of the number of collected or waiting collection tasks, the number of analysis tasks being processed or pending, the CPU utilization, the memory utilization, and the estimated completion time of the task completion.

In one embodiment, the collection task distributor 480 may distribute the collection task to the task management system 230 that has the lowest CPU utilization and memory utilization.

In another embodiment, the collection task distributor 480 may distribute the collection task to the task management system 230, which has the smallest sum of the number of collection tasks and the number of analysis tasks.

In another embodiment, the collection task distributor 480 may distribute the collection task to the task management system 230 having the earliest estimated task completion time.

Hereinafter, an example in which the collection task is distributed to at least one task management system 230 in the collection task management unit 320 will be described with reference to FIG.

5 is a diagram for explaining an example of distributing a collection job to at least one job management system.

First, the collection system 210 collects the data 510 and sends the collected data 510 to the job distribution system 250 to request a job.

The job distribution system 250 then determines the requested job as a collection job from the collection system 210 and determines the number of stores for the collection data 510. In addition, the work distribution system 250 selects at least one of the plurality of data nodes based on the number of storage and the size of the collection data 510 to determine the storage location.

For example, assume that the data type table is the same as Table 1, and the minimum storage unit is set as shown in Table 2.

The work distribution system 250 may determine the number of stored data 510 to be four, if the type of the collected data 510 is regular, of high importance, and is determined to be utilized. Then, if the size of the collected data 510 is 15 MB, the task distribution system 250 selects four of the data nodes included in the data node group B and transmits the selected data nodes to the storage data nodes 521, 522, 523 , &Lt; / RTI &gt; 524).

Next, the work distribution system 250 generates three replicated data 511, 512, 513 replicating the collected data 510. Then, the collection task for the collection data 510 and the replication data 511, 512, and 513 is distributed to at least one task management system 250.

At this time, the job distribution system 250 can identify the job management system 250 connected to each storage data node and distribute the collection job for the corresponding storage data node to the confirmed job management system 250.

5, the task distribution system 250 confirms that the first storage data node 521 is associated with the first task management system 231, and the collection data 510 and the first storage data It is possible to transmit the collection job request together with the node information to the first job management system 231. [

Meanwhile, the work distribution system 250 may be configured to determine that the second storage data node 522 is connected to the first and second job management systems 231 and 232, 231, and 232 may be selected. At this time, the work distribution system 250 can select based on the status information of the job management system.

If the first task management system 231 is idle and the second task management system 232 is in operation then the task distribution system 250 may collect the first duplicate data 511 and the second stored data node information And send the job request to the first job management system 231. [

The work distribution system 250 may also select and distribute the collection job to the third and fourth storage data nodes 523 and 524, respectively, as described above.

Meanwhile, the job management system 230 stores the received data in the storage data node when the collecting job is requested.

3, the analysis job management unit 340 searches the storage location database 330 for the stored data nodes storing the analysis data, and the at least one job management system 230 connected to the retrieved stored data nodes Distribute the analysis work.

Hereinafter, the analysis task manager 340 will be described in detail with reference to FIG.

FIG. 6 is a block diagram for explaining the analysis task management unit of FIG. 3. FIG.

6, the analysis job management unit 340 includes a storage data node search unit 610, an analysis job distribution unit 620, and an analysis data collection unit 630.

First, the storage data node searching unit 610 searches the storage location database 330 for a storage data node storing analysis data. If there are a plurality of stored stored data nodes, the stored data node searching unit 610 selects one of the plurality of stored data nodes.

Next, the analysis task distributor 620 confirms the task management system 230 connected to the stored data node, and sends an analysis task request together with the stored data node information and analysis data information to the confirmed task management system 230 .

At this time, if there are a plurality of job management systems 230 connected to the storage data nodes, the analysis job distribution unit 620 selects one of the plurality of job management systems 230 and distributes the analysis job.

On the other hand, the analysis task distributor 620 can select one of the plurality of job management systems 230 based on the status information on the job management system 230. At this time, the status information may include at least one of the number of collected or waiting collection tasks, the number of analysis tasks being processed or pending, the CPU utilization, the memory utilization, and the estimated completion time of the task completion.

In one embodiment, the analysis task distributor 620 may distribute analysis tasks to the task management system 230 having the lowest CPU utilization and memory utilization.

In another embodiment, the analysis task distributor 620 may distribute the analysis task to the task management system 230 having the smallest sum of the number of collection tasks and the number of analysis tasks.

In yet another embodiment, the analysis task distributor 620 may distribute the analysis task to the task management system 230 that has the fastest task completion time.

Next, the analysis data collection unit 630 receives data from at least one job management system 230, collects the received data, and transmits the collected data to the analysis system 220. [

Hereinafter, an example in which the analysis job is distributed to at least one job management system 230 in the analysis job management unit 340 will be described with reference to FIG.

7 is a view for explaining an example of distributing analysis work to at least one work management system.

First, the analysis system 220 requests the work distribution system 250 for analysis data to be analyzed.

The job distribution system 250 determines the requested job as an analysis job from the analysis system 220 and retrieves the stored data node in which the requested analysis data is stored.

7, when the first to fourth storage data nodes 521, 522, 523, and 524 are retrieved, the task distribution system 250 searches the task management system 250 ). The work distribution system 250 may then distribute the analysis work for that storage node to the identified work management system 250.

The task distribution system 250 may determine that the first storage data node 521 is associated with the first task management system 231 and send the analysis task request along with the first stored data node information to the first task management system 231).

Meanwhile, the work distribution system 250 may be configured to determine that the second storage data node 522 is connected to the first and second job management systems 231 and 232, 231, and 232 may be selected. At this time, the work distribution system 250 can select based on the status information of the job management system.

If the first task management system 231 is in the idle state and the second task management system 232 is in operation, the task distribution system 250 sends the analysis task request along with the second stored data node information to the first task management system (231).

The work distribution system 250 may also select and distribute the analysis job to the third and fourth storage data nodes 523 and 524, respectively, as described above.

Next, the job management system 230 requests analysis data from the storage system 240 when the analysis job is requested. When the requested data is received from the storage system 240, the received data is transferred to the work distribution system 250.

At this time, if a plurality of pieces of data are received as in the first task management system 231, the task management system 230 first collects a plurality of pieces of data and transfers them to the task distribution system 250.

Next, the job management system 230 collects the data received from the plurality of job management systems 230 and transfers the collected data to the analysis system 220.

Referring again to FIG. 3, the monitoring unit 350 monitors the plurality of job management systems 230 and generates status information for each job management system 230. At this time, the status information may include at least one of the number of collected or waiting collection tasks, the number of analysis tasks being processed or pending, the CPU utilization, the memory utilization, and the estimated completion time of the task completion.

The generated status information is provided to the collection job management unit 320 and the analysis job management unit 340.

8 is a flowchart illustrating a method of distributing big data according to an embodiment of the present invention.

Referring to FIG. 8, when the job distribution system 250 receives the job request (S801), it determines the type of the requested job (S802).

When the job distribution request is received with the collected data from the collection system 210, the job distribution system 250 determines the type of job as a collection job.

On the other hand, when the analysis data request and the work request are received from the analysis system 220, the work distribution system 250 determines the type of work as an analysis work.

When the collection job is requested, the job distribution system 250 determines a first storage data node to store the collected data (S803). A method of determining the first storage data node will be described in more detail with reference to FIGS. 9 and 10. FIG.

9 is a flow chart illustrating one embodiment of a method for determining stored data nodes to store collected data.

Referring to FIG. 9, first, the work distribution system 250 determines the type of collected data (S901). More specifically, the task distribution system 250 determines the data type based on at least one of the type, importance, and utilization of the collected data. To determine the data type, the task distribution system 250 may pre-set the data type table as shown in Table 1.

Next, the work distribution system 250 determines the number of stored data for the collected data (S902). More specifically, the workload distribution system 250 may determine the number of stores based on the type of collected data. To this end, the work distribution system 250 may preset the number of storage as shown in Table 2.

Next, the work distribution system 250 determines the size of the collected data (S903), and determines the data node group based on the size (S904). At this time, the data node group includes a plurality of data nodes having the same minimum storage unit.

 Next, the work distribution system 250 selects data nodes as many as the number of data nodes included in the determined data node group (S905). At this point, the work distribution system 250 determines the selected data node as the stored data node to store the collected data.

Next, the work distribution system 250 stores the stored data node information for the collected data in the storage location database 330 (S906). At this time, the stored data node information includes at least one of the storage system identification information, the data node identification information, the storage identification information, and the storage location information.

10 is a flow chart illustrating another embodiment of a method for determining stored data nodes to store collected data.

S1001 to S1004 shown in Fig. 10 are the same as S901 to S904, and a description thereof will be omitted.

The work distribution system 250 determines the free space of each of the data nodes included in the determined data node group when the data node group is determined (S1005).

Next, the work distribution system 250 extracts a spare node whose free space is larger than the collected data size, and compares the extracted number of spare nodes with the stored number (S1006).

At this time, if the number of spare nodes is equal to or greater than the storage number, the work distribution system 250 selects data nodes corresponding to the number of spare nodes (S1007). On the other hand, if the number of spare nodes is less than the number of storage, the work distribution system 250 selects all spare nodes (S1008).

Although shown in FIG. 10, in another embodiment, the task distribution system 250 may select data nodes included in another data node group if the number of spare nodes is less than the number of storage nodes.

For example, assume that the job distribution system 250 has selected the first data node group based on the size of the collected data.

If there are three spare nodes included in the first data node group and four stored numbers, the task distribution system 250 allocates three spare nodes of the first data node group and one spare node of the second data node group You can choose. At this time, the second data node group may include a data node having a smaller minimum storage unit than the first data node group.

Meanwhile, the work distribution system 250 determines the selected data node as the stored data node to store the collected data.

Next, the work distribution system 250 stores the first storage data node information for the collected data in the storage location database 330 (S1009).

Referring again to FIG. 8, the task distribution system 250 determines at least one of the plurality of job management systems 230 (S804). The task distribution system 250 selects a task management system 230 to distribute the collection task to each of the plurality of first storage data nodes.

More specifically, the task distribution system 250 identifies the task management system 230 associated with a particular first storage data node and selects the identified task management system 230. If there are a plurality of job management systems 230 connected to a specific first storage data node, the job distribution system 250 may manage a plurality of job management systems 230 based on status information about each job management system 230 ).

Next, the task distribution system 250 transmits the stored data node information, the collection data, and the collection task request to at least one task management system 230 (S805).

In operation S806, the job distribution system 250 searches the storage location database 330 for the second storage data node in which the analysis data is stored when the type of the requested job is determined to be an analysis job.

Next, the job distribution system 250 determines at least one of the plurality of job management systems 230 (S807). The task distribution system 250 selects a task management system 230 that will distribute analysis tasks for each of the at least one second storage data node.

More specifically, the task distribution system 250 identifies the task management system 230 associated with a particular second storage data node and selects the identified task management system 230. If there are a plurality of job management systems 230 connected to a specific second storage data node, the job distribution system 250 may manage a plurality of job management systems 230 based on status information about each job management system 230 ).

Next, the work distribution system 250 transmits the second stored data node information, the analysis data request, and the analysis work request to at least one work management system 230 (S808).

Next, when the analysis data is received from the at least one job management system 230 to which the analysis job is distributed, the job distribution system 250 collects the received analysis data and transmits it to the analysis system 220 (S809 and S810)

It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the following claims It can be understood that

Claims (14)

A plurality of data nodes in which data is stored;
A plurality of job management systems connected to at least one data node to store collected data in data nodes connected to each other; And
(N > 1) among the plurality of data nodes is selected as a first storage data node for storing the collected data when a collecting operation for collected data is requested, A task distribution system for distributing the collection task to a task management system connected to each of the N first storage data nodes,
Wherein the job distribution system checks an expected completion time of the plurality of job management systems if a plurality of job management systems are connected to any one of the first storage data nodes configured by the selected N data nodes, And a collecting job distributing unit for distributing the collecting job to the work management system having the earliest estimated time to complete the job. 2. The system of claim 1,
A data type determination unit that determines a data type based on at least one of the type, importance, and utilization of the collected data; And
And a storage number determination unit for determining storage number N of the collection data set in advance according to the determined data type. The method according to claim 1,
Each data node is divided into a minimum storage unit and is included in one of the data node groups divided by the same minimum storage unit,
Wherein the job distribution system determines a data node group set in a minimum storage unit in advance according to the size of the collected data and selects N of the data nodes included in the determined data node group to be determined as a first stored data node And a storage location determination unit for determining a location of the large data distribution system. The image processing apparatus according to claim 1,
A plurality of job management systems, each of the plurality of job management systems, based on status information about each of the plurality of job management systems, including at least one of a number of job operations being processed or queued, a CPU utilization rate, a memory utilization rate, And distributing the collected data by selecting one of the plurality of distributed data. delete A plurality of data nodes in which data is stored;
A plurality of job management systems connected to at least one data node to store collected data in data nodes connected to each other; And
(N > 1) among the plurality of data nodes is selected as a first storage data node for storing the collected data when a collecting operation for collected data is requested, A task distribution system for distributing the collection task to a task management system connected to each of the N first storage data nodes,
(M > 1) second stored data node, in which the analysis data is stored, among a plurality of data nodes, when the analysis work on the analysis data is requested, Further comprising an analysis task manager for selecting one of the stored data nodes and distributing the analysis task to a task management system connected to the selected second storage data node. 7. The system according to claim 6,
If one of the plurality of job management systems connected to the selected second storage data node is selected, distributing the analysis job by selecting one of the plurality of job management systems based on status information about each of the plurality of job management systems And an analysis job distribution unit. Receiving a job request from the job distribution system and determining a type of the requested job;
Wherein when the collecting data collecting job request is received in the step of determining the type of the requested job, the job distribution system selects N of the plurality of data nodes based on at least one of the data type and the size of the collected data Determining the selected N data nodes as a first storage data node to store the collected data; And
And distributing the collection task by selecting at least one of the task management systems connected to each of the N first storage data nodes,
When the analysis job analysis request is received in the step of determining the type of the requested job, the job distribution system searches for M second storage data nodes storing the analysis data among the plurality of data nodes, And distributing the analysis job by selecting at least one of the job management systems connected to the M second storage data nodes. 9. The method of claim 8,
Determining the data type based on at least one of the type, importance and utilization of the collected data, and determining a stored number N of the collected data in accordance with the determined data type Wherein the method comprises the steps of: 9. The method of claim 8,
Wherein the job distribution system determines a data node group set in advance in a minimum storage unit according to the size of the collected data and selecting a data node as many as the stored number of data nodes included in the determined data node group ,
Wherein the data node group includes one or more data nodes having the same minimum storage unit. 9. The method of claim 8,
Identifying the work management system associated with each first storage data node; And
The job distribution system selects one of the plurality of job management systems, and if the information about the corresponding storage node is included in the selected job management system, And sending a collection data and a collection job request. Receiving an analysis data request and analysis job request from a work distribution system;
Wherein the job distribution system searches for M second storage data nodes storing the analysis data among a plurality of data nodes and selects at least one of the job management systems connected to the retrieved M second storage data nodes, Distributing tasks; And
And when the analysis data is received from the at least one task management system, the task distribution system collects the received analysis data and sends it to the analysis system. 13. The method of claim 12,
Identifying the work management system coupled to each second storage data node; And
If the plurality of job management systems connected to one second storage data node are selected, the job distribution system selects one of the plurality of job management systems, and the information about the second storage data node And transmitting an analysis data request and analysis job request. 14. The job management system according to claim 11 or 13,
Wherein the task management system is one of the plurality of task management systems having the highest expected task completion time.

Images