KR20190097844A - File system program and method for controlling data cener using it - Google Patents

Abstract

The present invention relates to a file system program in which a data generator is mounted. The file system program includes instructions executing the steps of: checking the number of data centers, which will process data requested by a data generator; determining expected work time of each data center using a data transmission time, data storage time, and data analysis time of each data center linked to the data generator; and determining the number of data centers, which will process data, among the data centers linked to the data generator as much as the number of the data centers requested by the data generator using the expected work time.

Description

FILE SYSTEM PROGRAM AND METHOD FOR CONTROLLING DATA CENER USING IT}

The present invention relates to a file system program description.

Data production is exploding in recent years. 90% of the data produced so far has been produced in the last two years and is petabytes in size. Specifically, weather companies generate more than 20 terabytes of data each day to store temperature readings, wind speeds, barometric pressure, and satellite images from around the world, and many US Department of Energy research organizations generate hundreds of petabytes of simulation data annually. Furthermore, the data to be produced by 2018 is expected to exceed 1 exabyte.

As the volume of data grows exponentially, many organizations and organizations build and control geo-distributed data centers to improve data distribution, effective collaboration, and analytics. Implement a file system program. Through the implemented file system program, it is possible to share and analyze data existing in different areas, perform simulations and verify the results.

Existing file system programs are designed to handle such data sharing and collaboration needs, but have the following problems.

First, existing file system programs choose the closest data center to migrate data generated from the data generator, which provides resources such as storage bandwidth and network connectivity between the data center and the data generator. There is a problem in that effective utilization of resources in geographically dispersed integrated data centers cannot be guaranteed without considering data storage time that depends on availability. In addition, simply considering the distance between the data source and the data center does not consider post-storage processing such as analysis of stored data, and uses a single data center to complete a task.

Second, existing file system programs use a unified namespace to aggregate data sets stored in two different data centers into a single data center, which increases the integration performance overhead as the frequency of using unified namespaces increases. .

The problem to be solved by the present invention is to provide a file system program for determining a data center to process data based on the requested type of work and the resources of each of the data centers and a data center control method thereby.

Another problem to be solved by the present invention is to provide a file system program for controlling direct data transfer between data centers without the integrated namespace when work completion time can be reduced.

Checking the number of data centers to process the data requested by the data generation source, the file system program mounted on the data generation source according to an embodiment of the present invention, each of the data centers linked to the data generation source Determining an estimated working time of each data center using data transmission time, data storage time, and data analysis time, and processing data among data centers linked to the data generation source using the estimated working time Instructions for executing the step of determining the number of data centers to perform.

A file system program mounted on the data generation source compares the data size allocated to the data centers to process the data with the available storage capacity of the data centers, and the allocated data size is larger than the storage capacity; In the case of the data centers to process the data, further comprising instructions for executing the step of determining the data center determined in consideration of the expected working time instead of the data center having the smallest storage capacity as the data center to process the data.

The file system program mounted in the data generation source further includes instructions for executing the step of confirming the job type requested by the data generation source, and determining the number of data centers to process the data by the number. Further considering the type of work, determine the number of data centers to process the data.

The requested job type includes at least one of a data batch request or a data analysis request.

The file system program mounted in the data generator according to an embodiment of the present invention may be configured to: identify a job type requested by the data generator, and based on the requested job type, interlocked with the data generator. Instructions for executing the step of determining one or more data centers of the data centers to process data.

The file system program mounted in the data generation source further includes instructions for executing the step of determining an available resource of each of the data centers linked to the data generation source, and determining one or more data centers to process the data. The step of further considering the resource, determines one or more data centers to process the data.

Determining one or more data centers to process the data comprises using the requested job type and the resource to determine a candidate group of one or more data centers to process the data, wherein each data center included in the candidate group is selected by the data center. Determine an expected working time required for processing data and determine one or more data centers in the candidate group to process the data based on the estimated working time.

The estimated working time is determined according to at least one of data transmission time, data storage time, or data analysis time of each data center included in the candidate group.

Determining one or more data centers to process the data includes identifying a number of data centers to process the data requested by the data generator, and determining the number of data centers in the candidate group to process the data. The above data centers are determined.

A file system program mounted in a data center according to an embodiment of the present invention may include requesting transmission of data existing in another data center to the data center, and using the local namespace of the data center. Instructions for executing the step of directly receiving the data from the other data center.

Receiving the data directly from the other data center receives the data directly from the other data center using a virtual direct channel established in accordance with the local namespace between the data center and the other data center.

According to the present invention, the data center to process the data is determined in consideration of the requested operation type and the resources of each data center, so that the resources of the data center can be effectively used.

In addition, according to the present invention, direct data transfer between data centers is possible, thereby reducing the performance overhead caused by frequent use of the integrated namespace of file system programs.

1 is a diagram illustrating an environment in which a file system program is implemented, according to an exemplary embodiment.
2 is a view for explaining a method of transmitting data in a network under the control of a file system program mounted in a data generating source.
3 is a flowchart illustrating a method of controlling a data center by a file system program mounted on a data generator, according to an exemplary embodiment.
4 is a flowchart illustrating another method of controlling a data center by a file system program mounted on a data generator, according to an exemplary embodiment.
5 is a flowchart illustrating another method of controlling a direct transfer between data centers by a file system program mounted in data centers according to an embodiment.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise.

Hereinafter, a file system program and a data center control method using the same will be described with reference to the accompanying drawings.

1 is a diagram illustrating an environment in which a file system program is implemented, according to an exemplary embodiment.

Referring to FIG. 1, an environment 1000 in which a file system program is implemented includes a data generation source 100, file system programs 210 to 240, and data centers 300.

The data generator 100 refers to a device that generates data for a specific purpose, and the data generated is not limited to either type.

For example, the data generator 100 may be a satellite device that collects satellite images to observe the weather, and stores or collects the collected satellite images in any of the data centers 300. File system program 210 may request that any of the data centers 300 analyze satellite images.

Although the data generation source 100 is illustrated in FIG. 1 singularly, according to various embodiments, each of the plurality of data generation sources may be independently linked to a file system program, and the data centers 300 may process data. Of course you can ask.

The file system programs 210 to 240 are programs mounted in the data generation source 100 or the plurality of data centers 310 to 330, and interoperate with the data generation source 100 and the data centers 300. A data center to process data is determined among the data centers 300 based on the data processing request by the generation source 100.

File system program 210 integrates data centers 300 under an integrated namespace, thereby accessing data centers 300.

The data centers 300 refer to a cluster of devices in which a computing system, a communication device, a storage device, and the like are implemented, and perform functions such as data placement and analysis.

In the present invention, the data centers 300 refer to a cluster of geographically dispersed data centers, and among the data centers 300, the data center determined by the file system program 210 receives data from the data generation source 100. Receive and place or store the received data.

It is assumed that resources of the data centers 310 to 330 constituting the data centers 300 are independent of each other. Specifically, each data center 310 to 330 independently has storage capacity, storage bandwidth, computational performance, and network bandwidth.

2 is a view for explaining a method of transmitting data in a network under the control of a file system program mounted in a data generating source.

Referring to FIG. 2, the network 2000 includes a data generator 100, data centers 300, and a VPN switch 400. As described in FIG. 1, it is assumed that the file system program 210 is mounted in the data generation source 100 to control data transmission.

The data generator 100 continuously generates a large amount of data and is connected to the data center 300 through the VPN switch 400.

Although the data generator 100 is illustrated as a single unit in FIG. 2, a plurality of data generators may exist in the network 2000 to independently perform the following data transmission method for each data generator. to be.

를 결정하고, 데이터 생성원(100)에 탑재된 파일 시스템 프로그램(210)은

를 확인한다. 예를 들면, 데이터 생성원(100)이 데이터를 처리할 데이터 센터들의 수

를 "1"로 결정한 경우, 데이터는 단일 데이터 센터에 저장되고 분석된다.If the data generated by the data generator 100 is allocated to all the data centers 300, the minimum job execution time is achieved, but the overhead of data migration between the data centers 300 after data storage and analysis is Will increase. Thus, the data generation source 100 can determine the number of data centers to process the data.

And the file system program 210 mounted on the data generation source 100

Check. For example, the number of data centers for which data generation 100 will process data.

If is determined to be "1", the data is stored and analyzed in a single data center.

The file system program 210 determines the transmission time, storage time and analysis time of each data center linked to the data generation source 100 to determine one or more data centers among the data centers 300 that will actually process the data. The total data transfer time, the total data storage time and the total data analysis time are determined.

만큼의 데이터를 데이터 센터들(300) 중 임의의 데이터 센터로 전송하는 경우 데이터 조각(slice)의 크기를

, 데이터 생성원(100)에서 임의의 데이터 센터로 전송하는데 필요한 조각수를

라고 가정하면, 전송되는 총 데이터 양

이다. 데이터 생성원(100)에서

번째 데이터 센터로 한 조각의 데이터를 전송하는데 걸리는 시간을

로 표시하면, 데이터 생성원(100)에서 데이터 센터들(300)에 데이터가 전송되는데 소요되는 전체 데이터 전송 시간

은 수학식 1과 같다.Specifically, the data generation source 100

If you send as much data to any of the data centers 300 as the size of the data (slice)

, The number of pieces needed to transfer from the data source 100 to any data center

, The total amount of data transferred

to be. In the data generator 100

The amount of time it takes to transfer a piece of data to the first data center.

In this case, the total data transmission time required for data transmission from the data generation source 100 to the data centers 300.

Is the same as Equation 1.

번째 데이터 센터가 저장하는데 걸리는 시간을 라고 하면, 데이터 센터들(300)이 소모한 전체 데이터 저장 시간

는 수학식 2와 같다.A piece of data transmitted from the data generator 100

The time it takes for the first data center to store , The total data storage time consumed by the data centers 300

Is the same as Equation 2.

번째 데이터 센터가 한 조각의 데이터를 분석하는데 소요되는 시간을

라 하고, 분석해야 하는 조각수를

라 하면, 데이터 센터들(300)이 소모한 전체 데이터 분석 시간

은 수학식 3과 같다.

Time for the first data center to analyze a piece of data

The number of pieces to analyze

That is, the total data analysis time consumed by the data centers 300

Is the same as Equation 3.

,

및

에 따라

번째 데이터 센터의 예상 작업 시간

을 결정할 수 있으며,

는 수학식 4와 같다.Also,

,

And

Depending on the

Work hours in the first data center

Can be determined,

Is the same as Equation 4.

는 데이터 배치 시간

을 의미한다. 데이터 처리에서는 조각을 단위로 사용하기 때문에, 데이터를 수신한 데이터 센터는 첫번째 조각을 수신하는 즉시 작업을 시작한다. 따라서, 시간 중복을 피하기 위해 최대 전송 시간과 저장 시간을 사용하여 데이터 배치 시간

을 결정하며,

는

번째 데이터 센터의 데이터 배치 시간을 의미한다.In Equation 4,

Data placement time

Means. Because data processing uses pieces in units, the data center that receives the data starts working as soon as it receives the first piece. Therefore, data placement time using maximum transfer time and storage time to avoid time duplication

To determine

Is

Data placement time of the first data center.

개 결정하고, 결정된 데이터 센터들로 구성된 제1 세트를 결정한다.If the requested job type includes both a data batch request and a data analysis request, the file system program 210 opens the data center in the order of the smallest estimated work time.

Determine a first set of determined data centers.

The file system program 210 then distributes the data to each data center based on the estimated working time of each data center included in the first set, and the overall estimates required for the data centers included in the first set to process the data. Calculate your work time.

번째로 작은 데이터 센터에 할당하고,

를 다시 계산한다. 이후, 파일 시스템 프로그램(210)은 제1 세트에 포함된 데이터 센터들 중 사용 가능한 스토리지 용량이 적은 데이터 센터 대신 예상 작업 시간을 고려하여 결정된 데이터 센터를 추가하여 제2 세트를 구성한다.If the data size allocated to the data centers included in the first set is less than the available storage capacity of the data centers, the file system program 210 may exceed the estimated working time.

To the first smallest data center,

Recalculate. Thereafter, the file system program 210 configures the second set by adding the data center determined in consideration of the expected working time instead of the data center having less available storage capacity among the data centers included in the first set.

번째로 작은 데이터 센터를 추가하여 제2 세트를 구성할 수 있다.For example, the file system program 210 may have an estimated working time except for the data centers having the least available storage capacity among the data centers included in the first set.

The second smallest data center can be added to form a second set.

The file system program 210 calculates the total estimated working time required for data centers included in the second set to process data, and the data size allocated to the data centers included in the second set is the available storage of the data centers. If less than the dose is repeated.

Conversely, if the data size allocated to the data centers included in the second set is larger than the available storage capacity of the data centers, the file system program 210 causes the data centers included in the second set to locate and locate the data. Have them analyze.

If the requested job type is a data batch request, the file system program 210 determines the data center having the smallest data batch time by the number of data centers to process the data received from the data generator 100, and determines the determined data center. Let data centers place data

If the type of work requested is a data analysis request, it is advantageous to perform the analysis in the same data center unless the analysis time improves when migrating data to another data center. Therefore, if the data analysis time of the data center to which the data is transmitted is greater than the sum of the transfer time to the other data center and the data analysis time of the other data center, the file system program 210 is connected between the data centers 300. Allow data transfer.

3 is a flowchart illustrating a method of controlling a data center by a file system program mounted on a data generator, according to an exemplary embodiment.

Referring to FIG. 3, the file system program 210 checks a job type requested by the data generator 100 (S100).

The requested job type includes at least one of a data batch request or a data analysis request. Specifically, a data placement request may refer to a request relating to the transfer of data to any data center or to the movement of data, such as staging, space allocation, metadata registration, data retrieval, and the like. In addition, a data analysis request may refer to a data analysis request at any data center.

The file system program 210 determines available resources of each of the data centers 300 (S110).

Specifically, the file system program 210 determines at least one of available storage capacity, storage bandwidth, computational performance, or network bandwidth of each of the data centers 300 at the time of receiving the data processing request.

In this case, the file system program 210 may receive resource information of the data center from the file system programs 220 to 240 installed in each of the data centers 300, or metadata about the data centers 300. (meta data) can be received, through which resources in each data center can be determined.

The file system program 210 determines one or more data centers to process data among the data centers 300 linked to the data generation source 100 (S120).

In one embodiment, the data center determiner 230 may determine one or more data centers to process data among the data centers 300 based on the requested job type.

For example, if the requested job type is a data analysis request, the file system program 210 determines among the data centers 300 data centers whose computational performance is greater than or equal to a certain value as the data center to process the data, or This high specific number of data centers can be determined as the data center to process the data.

Similarly, if the requested job type is a data batch request, the file system program 210 determines the data centers of which the storage performance is greater than or equal to a certain value among the data centers 300 as data centers to process the data, or This high specific number of data centers can be determined as the data center to process the data.

In another embodiment, file system program 210 may determine one or more data centers to process data among data centers 300 by further considering the available resources of each of data centers 300.

In particular, the file system program 210 determines the candidate group of one or more data centers to process the data using the job type and the resource. For example, the file system program 210 may identify the type of work requested by the data generator 100 and determine the total resources required to perform the requested work type. Thereafter, the file system program 210 may determine data centers having available resources more than the determined total resources as candidate groups.

The file system program 210 then determines the estimated work time required for each data center included in the candidate group to process the data.

In detail, the file system program 210 determines at least one of data transmission time, data storage time, or data analysis time of each data center based on the resource.

Here, the data transmission time refers to the time at which data is transmitted from the data generation source 100 to a specific data center, the data storage time refers to the time at which the data center stores the transmitted data, and the data analysis time refers to the data center. Refers to the time to analyze the transmitted data.

Thereafter, the file system program 210 determines the expected working time of the data center by using at least one of the data transmission time, data storage time, or data analysis time determined for each data center.

For example, the file system program 210 may calculate the estimated working time required for the first data center 310 included in the candidate group to process the data placement request, and the data transfer time and data storage time of the first data center 310. You can decide by the time.

Thereafter, the file system program 210 determines one or more data centers to process data among the data centers included in the candidate group based on the expected working time.

In one embodiment, file system program 210 may determine a particular number of data centers for which the estimated work time is small to one or more data centers to process the data.

In another embodiment, the file system program 210 identifies the number of data centers to process the data requested by the data generation source 100, and determines the number of data centers determined from among the data centers included in the candidate group. May be determined as one or more data centers to be processed.

For example, when the number of data centers to process data requested by the data generator 100 is "3", the file system program 210 determines that the estimated work time is smallest in the candidate group. Three data centers can be determined.

According to the present invention, the data center to process the data is determined in consideration of the requested work type and the resources of each data center, so that the resources of the linked data centers can be effectively used.

If one or more data centers to process the data are determined, the file system program 210 transmits and processes the data to the determined data center from the data generation source 100 that has transmitted the data processing request (S130). In this case, the data is transferred through the integrated namespace of the file system program.

4 is a flowchart illustrating another method of controlling a data center by a file system program mounted on a data generator, according to an exemplary embodiment.

Referring to FIG. 4, the file system program 210 checks a job type requested by the data generator 100 (S200). In this case, the job type includes at least one of a data batch request or a data analysis request.

The file system program 220 determines one or more data centers to process the data based on the identified job type.

First, the file system program 210 confirms the number of data centers to process data requested by the data generation source 100 (S210).

The file system program 210 determines at least one of a data transmission time, a data storage time, or a data analysis time of each of the data centers 300 linked to the data generation source 100 (S220).

Specifically, the file system program 210 determines the data transfer time, data storage time or data analysis time using at least one of available storage capacity, storage bandwidth, computational performance, or network bandwidth of each of the data centers 300. do.

For example, the file system program 210 may determine data transmission time using network bandwidth, determine data storage time using storage capacity and storage bandwidth, and determine data analysis time using arithmetic performance. Can be.

The file system program 210 determines the number of data centers determined in step S210 as one or more data centers to process the data, based on the resource and task type determined for each of the data centers 300.

In detail, the file system program 210 determines the importance of at least one of the data transmission time, the data storage time, or the data analysis time according to the job type (S230).

For example, if the job type is a data batch request, a data center with a high storage bandwidth can complete the job quickly, regardless of the data analysis capability, so the file system program 210 may be able to store data storage time and data rather than data analysis time. High importance can be set for data transfer time.

Conversely, if the job type is a data analysis request, the data center with high data analysis capability can complete the job quickly, so that the file system program 210 can set high importance on the data analysis time and the data storage time rather than the data storage time. have.

In addition, if the job type requires both data placement and data analysis, then both the data center's storage and data processing capabilities must be considered. In this case, the file system program 210 may set the order of importance of data storage time, data analysis time, and data transmission time in consideration of the data capacity allocated to each of the data centers 300.

The file system program 210 determines an expected working time for each of the data centers 300 using the determined importance and at least one of data transmission time, data storage time, or data analysis time (S240).

For example, when the job type is a data batch request and a low importance is set for the data analysis time, the file system program 210 may determine the sum of the data storage time and the data transfer time as the expected work time.

Thereafter, the file system program 210 determines the number of data centers determined in step S210 among the data centers 300 as one or more data centers to process the data, based on the determined working time (S250).

For example, when it is determined in step S210 that the number of data centers to process data is three, the file system program 210 transfers three data centers having the least working time to one or more data centers to process the data. You can decide.

The file system program 210 allocates data to each of the data centers (S260). Data centers assigned data process the assigned data according to the type of work.

5 is a flowchart illustrating another method of controlling a direct transfer between data centers by a file system program mounted in data centers according to an embodiment.

In FIG. 5, it is assumed that an analysis operation is required in the first data center 310 for data existing in the second data center 320.

Referring to FIG. 5, the file system program 220 mounted in the first data center 310 requests data transmission to the second data center 320 for data existing in the second data center 320 ( S300).

Upon receiving the request, the second data center 320 transmits data to the first data center 310, and the first data center 310 receives data using a local namespace (S310).

If the first data center 310 receives the data through the integrated interface of the file system program in step S310, FUSE overhead and NFS overhead occurs.

Accordingly, the first data center 310 receives data using a virtual direct channel established according to a local namespace between the first data center 310 and the second data center 320.

For example, the file system program 220 mounted in the first data center 310 generates a directory in which data can be transmitted and received through the virtual direct channel in the first data center 310, and the second data center ( The data transmitted from 320 may be stored in the generated directory, and the first data center 310 may be accessed to access the file stored in the directory.

The first data center 310 analyzes the received data (S320).

According to the present invention, direct data transfer between data centers is possible, thereby reducing the integration performance overhead due to FUSE and NFS caused by frequent access to the integrated namespace of file system programs.

The file system program according to the above-described embodiments of the present invention may be implemented as a program (or an application) to be combined with a computer, which is hardware, and stored in a computer-readable recording medium.

The computer-readable recording medium refers to a medium that stores data semi-permanently, not a medium for storing data for a short time such as a register, a cache, a memory, and the like, and is readable by a device. Examples of the recording medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. That is, the file system program according to the embodiment of the present invention may be stored in various recording media on various servers accessible by a computer or various recording media on the computer of the user.

According to the present invention, the data center to process the data is determined in consideration of the requested operation type and the resources of each data center, so that the resources of the data center can be effectively used.

In addition, according to the present invention, direct data transfer between data centers is possible, thereby reducing integration performance overhead caused by frequent access of the integration namespace of the file system program.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (11)

As a file system program mounted in the data generator,
Identifying the number of data centers to process the data requested by the data generator,
Determining an expected working time of each data center by using data transmission time, data storage time and data analysis time of each data center linked to the data generation source, and
Determining the number of data centers to process data from among data centers linked to the data generation source using the estimated working time
A file system program containing instructions for executing a. In claim 1,
Comparing the data size allocated to the data centers to process the data with the available storage capacity of the data centers, and
If the allocated data size is larger than the storage capacity, determining the determined data center as a data center to process data instead of the data center having the smallest storage capacity among the data centers to process the data.
File system program further comprising instructions for executing. In claim 1,
Instructions for executing a step of confirming a job type requested by the data generator;
Determining the number of data centers to process the data is
File system program for determining the number of data centers to process the data, further considering the type of operation. In claim 3,
The requested job type is
A file system program comprising at least one of a data batch request or a data analysis request. As a file system program mounted in the data generator,
Identifying a job type requested by the data generator, and
Based on the requested job type, determining one or more data centers to process data among data centers linked to the data generation source;
A file system program containing instructions for executing a. In claim 5,
Further comprising instructions for determining an available resource of each of the data centers associated with the data generation source,
Determining one or more data centers to process the data
And further considering the resource, to determine one or more data centers to process the data. In claim 6,
Determining one or more data centers to process the data
Determine a candidate group of one or more data centers to process the data using the requested job type and the resource, determine an expected working time required for each data center included in the candidate group to process the data, A file system program for determining one or more data centers to process the data in the candidate group based on working time. In claim 7,
And the expected working time is determined according to at least one of data transmission time, data storage time, or data analysis time of each data center included in the candidate group. In claim 7,
Determining one or more data centers to process the data
Identifying a number of data centers to process data requested by the data generation source, and determining the number of data centers in the candidate group as one or more data centers to process the data. As a file system program installed in the data center,
Requesting transmission to the data center for data residing in another data center, and
Receiving the data directly from the other data center using the local namespace of the data center
A file system program containing instructions for executing a. In claim 10,
Receiving the data directly from the other data center
A file system program for receiving said data directly from said other data center using a virtual direct channel established in accordance with said local namespace between said data center and said other data center.

Images