KR20200072128A - Distributed file system and file managing method for live service - Google Patents

Abstract

Provided are a distributed file system and a file management method for a live service. According to one embodiment of the present invention, a computing device included in the distributed file system for a live service may set an expiration time for each of the files requested to be stored in association with a live service, group the files requested to be stored into a plurality of directories identified by a deletion time and store the files using the set expiration time, and delete all files stored in a directory selected based on a current time and the deletion time.

Description

DISTRIBUTED FILE SYSTEM AND FILE MANAGING METHOD FOR LIVE SERVICE}

The following description relates to a distributed file system and a file management method for a live service.

Distributed file systems are generally used for the purpose of permanently storing files on a network. Depending on the purpose, it has various functions and features. For example, Hadoop Distributed File System (HDFS) divides and stores large files into several pieces for big data processing, and stores and distributes files. Gives Ceph supports erasure coding, which is effective in reducing capacity, and SeaweedFS features fast processing of small files.

On the other hand, the most widely used HLS and MPEG-DASH protocols for live services divide video data into TS segments. When 2Mbps video is live for 4 hours, about 4800 TS files of 750KB capacity are generated. And, these files should be deleted because they are no longer needed when the live service ends.

Accordingly, there is a need for a live dedicated distributed file system optimized for these characteristics of a live service, that is, real-time processing and automatic deletion of a large number of small files.

A distributed file system capable of limiting a target file to be moved to a newer file when scaling a distributed file system in consideration of volatility of a file stored in a live service, a computer device included in the distributed file system, and the computer device It provides a computer program stored in a computer-readable recording medium and a recording medium to execute the file management method in combination with a computer device and a file management method performed by the computer device.

A distributed file system capable of optimizing the deletion of files through a directory structure in which files are grouped and stored for each survival time in consideration of volatility of files stored in a live service, a computer device included in the distributed file system, and the computer device A file management method to be performed, and a computer program stored in a computer-readable recording medium to execute the file management method in combination with a computer device, and a recording medium therefor.

A distributed file system capable of more quickly processing small and many files of a live service by storing relatively small and many files in a live service through a relatively large parallel volume file, and a computer device included in the distributed file system , A file management method performed by the computer device, and a computer program stored in a computer-readable recording medium for execution of the file management method on the computer device in combination with the computer device and a recording medium therefor.

A computer device included in a distributed file system for a live service, comprising: at least one processor implemented to execute readable instructions on the computer device, and stored by the at least one processor in association with the live service Set an expiration time for each of the requested files, and use the set expiration time to group and store the files requested to be stored into a plurality of directories identified by a deletion time, based on the current time and the deletion time It provides a computer device characterized in that the files stored in the directory selected by the batch deletion.

A data processing method performed by a computer device included in a distributed file system for a live service, the expiration time for each of the files requested to be stored in association with the live service by at least one processor included in the computer device. Setting it; Grouping and storing, by the at least one processor, the files requested to be stored into a plurality of directories identified by a deletion time using the set expiration time; And collectively deleting files stored in a directory selected based on the current time and the deletion time by the at least one processor.

In combination with a computer device, a computer program stored in a computer-readable recording medium is provided to execute the file management method on the computer device.

A computer-readable recording medium in which a program for executing the file management method is executed is provided.

In consideration of the volatility of a file stored in a live service, a target file to be moved during scaling of a distributed file system can be limited to the latest file.

In consideration of the volatility of the files stored in the live service, the deletion of the files can be optimized through a directory structure in which the files are grouped and stored for each survival time.

By storing relatively small and many files in a live service through a relatively large sized parallel volume file, the small and many files that the live service has can be processed more quickly.

1 is a diagram showing an example of a configuration of a distributed file system according to an embodiment of the present invention.
2 is a diagram showing an example of the role of servers in an embodiment of the present invention.
3 is a flowchart illustrating an example of a file storage process in an embodiment of the present invention.
4 is an example for explaining a warm-up process in an embodiment of the present invention.
5 is a view for explaining a direct method, a relay method and a redirect method in one embodiment of the present invention.
6 is a diagram showing an example of information stored and shared by the main keeper in an embodiment of the present invention.
7 is a diagram illustrating an example of storing a plurality of files in one volume file in one embodiment of the present invention.
8 is a diagram illustrating an example of deleting a group whose expiration time has expired in one embodiment of the present invention.
9 is a diagram showing an example of a group-only redis in an embodiment of the present invention.
10 is a diagram illustrating an example of a health check session between servers in an embodiment of the present invention.
12 is a diagram showing an example of information searched through CLUSTER in an embodiment of the present invention.
13 is a diagram showing an example of components of a distributed file system in an embodiment of the present invention.
14 is a diagram showing an example of a thread model in an embodiment of the present invention.
15 is a block diagram showing an example of a computer device according to an embodiment of the present invention.
16 is a flowchart illustrating an example of a file management method according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The distributed file system according to embodiments of the present invention is specialized for a live service, but has the same problem situation as a general distributed file system. In other words, the server model for file retrieval (centralized, decentralized), file replication (replication, erasure coding), failure response, scalability, etc. should be considered, and additionally, high performance for real-time processing of live services should be possible. do.

The basic design philosophy of a distributed file system according to one embodiment is simplicity and performance. It was based on a fully distributed model considering scalability as a distributed file system, and the simpler one was chosen for many options to keep the overall structure and concept simple. Network IO (Input/Output), which can be a performance bottleneck on the machine, is a multi-threaded Proactor Pattern, and disk IO can be implemented as a parallel volume file that is effective for processing small and large files.

The distributed file system according to embodiments of the present invention may have the following features.

A. Decentralized

The distributed file system according to embodiments of the present invention may be implemented as a distributed file system of a fully distributed model such that no central server exists. Therefore, theoretically, infinite expansion is possible, and there is no bottleneck, so performance degradation does not occur due to expansion, and failure of some servers does not lead to total failure (No SPOF: No Single Point of Failure).

B. Simple & seamless scaling

System expansion in a fully distributed model can lead to large-scale file movement, and file movement has a significant effect on system performance. Therefore, in general, expansion is performed after the system is completely stopped. Also, system expansion and thus file movement can complicate the system structure.

The distributed file system according to the embodiments of the present invention may be used exclusively for a live service, and in the live service, since most of the access is concentrated on the latest file, the target of movement may be limited to the latest file. Predictable and uninterrupted expansion can be performed by applying a warm-up method to scaling according to embodiments of the present invention.

C. Practical & effective failover

In general, in a distributed file system, files are copied in case of failure, so even if some servers fail, the system is almost always available (High Availability). Disorders are very rare, but must be addressed. However, the process of handling a fault is very difficult and complicated, and can sometimes affect the normal operation and performance.

The distributed file system according to embodiments of the present invention may be implemented not to recover from a failure. However, it provides a simple way to access the file at the time the file is requested, and this method can be implemented simply without affecting normal operation and performance.

D. Optimized for small, many & volatile files

The distributed file system according to embodiments of the present invention is small and can introduce a parallel volume file to quickly process a large number of files, and may have a directory structure optimized for deletion of files.

E. Minimal RESTful Application Program Interfaces (APIs)

In a distributed file system according to embodiments of the present invention, a client software development kit (SDK) having distribution and compatibility issues may not be supported. Instead, it can support restful APIs that are not dependent on the development language and are easy to use. Only a minimally refined 4 APIs (PUT, GET, DEL, CLUSTER) can be efficiently used in a distributed file system according to embodiments of the present invention.

1. System Architecture

A. Cluster

1 is a diagram showing an example of a configuration of a distributed file system according to an embodiment of the present invention. 1 shows that the distributed file system 100 may include N groups. Also, each of the groups may include a plurality (eg 3 or more) of machines. For example, in FIG. 1, group 1 110 may include three machines 111, 112, 113, and group N 120 also includes three machines 121, 122, 123. It shows an example. Other groups included in the distributed file system 100 may be similarly implemented. Also, each machine can implement a server. In FIG. 1, machines 1-1 (111) included in group 1 110 include servers 1-1 (111-2), machines 1-2 (112) represent servers 1-2 (112-2), and machines. 1-3(113) shows an example of implementing the server 1-3(113-2), respectively. Similarly, in FIG. 1, the machine N-1 (121) included in the group N 120 is the server N-1 (121-2), and the machine N-2 (122) is the server N-2 (122-2). In the example, the machine N-3 123 implements the servers N-3 123-2, respectively. Depending on the embodiment, each of the machines may further implement a cache server. In FIG. 1, machines 111, 112, 113, 121, 122, and 123 implement cache servers 111-1, 112-1, 113-1, 121-1, 122-1, and 123-1. It shows an example.

Also, one Redis may be implemented in three groups over three machines. In FIG. 1, an example in which the redis sentinel 1 114 is implemented for the three machines 111, 112, and 113 of the group 1 110, the redis sentinel for the three machines in the group N 120 Each of the examples of N 124 is shown. Redis Sentinel may be implemented in each of the other groups included in the distributed file system 100.

Further, a ZooKeeper 130 may be implemented across N groups.

Components and operations of the distributed file system 100 will be described in more detail later.

1) Group

In the distributed file system according to embodiments of the present invention, instead of storing the location of a file using a "jump-consistent-hash (jc-hash)" algorithm to implement a decentralized model, the location of the file is determined. Compute at runtime. The jc-hash algorithm is very useful for maintaining minimal key movement and server-to-server balance against changes in the number of servers.

Group represents a logical location where files are stored in a fully distributed model, and servers in the same group are subject to replication.

To store files under the same directory in the same group, get the hash value using the "fnv1a" hash algorithm (a lightweight hash algorithm that generates a 64-bit hash value) using the directory name excluding the file name, and calculate the hash value and the number of groups. By executing the jc-hash algorithm as input, we finally get the group index to store the file. The process of obtaining the group index can be expressed as "Jc-hash(fnv1a(filedir), N) => 0~N-1". Here, "filedir" may mean a directory name excluding a file name, "fnv1a()" may mean a "fnv1a" hash algorithm, and "Jc-hash()" may mean a jc-hash algorithm, respectively, for N groups. One of the group indices from 0 to N-1 can be obtained.

2) Role

There can be three servers in the group, and they can be targeted for replication. It was determined that the number of copies exceeding 3 was not meaningful, and was fixed to 3 for simplicity, but is not limited thereto.

2 is a diagram showing an example of the role of servers in an embodiment of the present invention. In FIG. 2, M may refer to a master server, S to a slave server, and F to a slave server set as a feeder. At this time, "up" indicates uploading of files, "down" indicates downloading of files, and "change" indicates roles that can be changed between servers. For example, a feeder can change its role to a master server, and a slave server can change its role to a master server or feeder. In addition, the role of the master server as a feeder can be changed. At this time, the three servers perform different roles at the same time, and the master server (Master, M) in charge of writing in the group, and the other two servers can store replicas as slave servers (Slave, S). In addition, a slave server performing data migration during resharding (or rebalancing) may be defined as a feeder (F).

Role determination and switching of servers can be safely performed using ZooKeeper.

1) Replication

If there is a request to save a file in the group, the group's master can process the request. The file is stored on the local disk, and for quick access, metadata such as the location and size of the file can be managed in memory.

3 is a flowchart illustrating an example of a file storage process in an embodiment of the present invention. In response to a request to save a file from a client, the master server can save the file to local storage, and then request that slaves (feeders and slave servers) store replicas (replicas of files), and one or more replicas are successfully saved. If it does, the client's request to save the file can be confirmed as successful by storing the metadata. If even one replica cannot be stored, the request to save the file may be treated as a failure, and at this time, the metadata is not stored, and the file may be discarded.

One) Resharding (or rebalancing)

In a distributed file system, the key relocation operation according to the expansion and contraction of a cluster is called resharding (or rebalancing). Resharding is a part that can affect the performance and complexity of a distributed file system. In the distributed file system according to embodiments of the present invention, resharding is designed to maintain high performance and not be complicated in consideration of characteristics of a live service. Became.

Since the live service is a real-time service, after a certain period of time, the corresponding live data is no longer needed and should be deleted (volatility of the live file). Considering the live time machine function (a function that can be played by moving to the past time in live mode, also called the DVR (Digital Video Recorder) function), the storage time of a live file is about 1 to 4 hours, and is subject to migration when resharding The file can be viewed in the last 4 hours.

4 is an example for explaining a warm-up process in an embodiment of the present invention. 4 illustrates a situation in which new groups G5 and G6 are added to service groups G1, G2, G3, and G4 of the distributed file system according to an embodiment of the present invention. At this time, in the distributed file system, migration can be performed first and then the cluster can be changed. Here, the migration process is defined as warming-up, and the feeder in the group can feed the replication request. Feeding may refer to a process of transferring a file to be relocated after warming up to a new group. The warm-up may be performed for a set time in the distributed file system, and as described above, this time may be set to about 4 hours according to an embodiment. When the warm-up is over and the cluster is changed, relocation of all necessary files may be completed.

Warm-up can be applied when monitoring the load of the cluster and predicting the expansion. If you have to inevitably expand the cluster, you can reduce or skip the warm-up time. In this case, since the relocation of files is not complete, some files may not be accessible. However, since the live service mostly accesses the latest files, the file access problem affects only the time machine service. In short, when cluster expansion is more urgent than a live time machine service, warm-up can be omitted.

2) Relay & Redirect

5 is a view for explaining a direct method, a relay method and a redirect method in one embodiment of the present invention. In the direct method, the client and the group G1 directly send and receive requests and files, and in the relay method, the client and the group G2 send and receive requests and files through relays of other groups G1. The receiving method and the redirect method may mean a method in which the client directly sends and receives a request and a file with the corresponding group G2 when the group G1 informs the client of the other group G2 in which the necessary file is stored.

If a user request (PUT, GET, etc.) is delivered directly to a target group (a group that manages the file), it will be processed immediately, but when a request comes to another group, the request must be delivered to the target group , At this time, a relay (Relay) and a redirect (Redirect) method can be used, the distributed file system according to embodiments of the present invention can support both methods.

The redirect method can be used only when the user can directly access the changed server address. On the other hand, the relay method can be used in a situation where a user cannot directly access a specific server.

Typically, commercial servers are configured as a private network for security, and it is common to connect through an L4 switch from the outside. Therefore, the general user can use the relay method. If there are servers for other services using a distributed file system, these servers can be configured on the same network as the distributed file system, and in this case, the redirect method may be a better option in terms of performance.

2) ZooKeeper

The main keeper can be used to share information such as cluster configuration, group-specific roles, server execution status, and settings. The main keeper itself is a central server model implemented in a leader/follower pattern. In the distributed file system according to embodiments of the present invention, only a very small amount of fixed information is shared, so the main keeper is the bottleneck. Does not work.

6 is a diagram showing an example of information stored and shared by the main keeper in an embodiment of the present invention. As illustrated in FIG. 6, the main keeper may store information about a cluster, a configuration, a machine, and a runtime.

(a) cluster

The main keeper may store group information constituting the cluster, machine information constituting the group, and whether resharding is in progress.

(b) config

The main keeper can save each module (common, coordinator, file manager, group) settings.

(c) machine

The main keeper records whether the server for the distributed file system is running and can use a machine for the purpose of preventing duplicate execution.

(d) Runtime

The master keeper can determine the master, feeder, and slave at runtime in the group, and can be used to share the information between groups.

2. File Management

As a feature of the live service, we mentioned a small number of files and volatile. In a distributed file system according to embodiments of the present invention, a parallel volume file can be introduced to efficiently process small and many files, and a time concept can be introduced to the directory structure for automatic deletion of expired files. have.

1) Volume File

If the number of files exceeds a certain level, the performance of the operating system and distributed file system may deteriorate. In the distributed file system according to embodiments of the present invention, by storing a plurality of individual files in one volume file, the number of files can be reduced to prevent a performance degradation problem due to the number of files.

7 is a diagram illustrating an example of storing a plurality of files in one volume file in one embodiment of the present invention. As described above, embodiments of the present invention may introduce a parallel volume file for quickly processing small files. For example, in the case of 720p live stream (2Mbps) in the HTTP Live Streaming (HLS) protocol most widely used for live service, when the TS duration is set to 3 seconds, the size of the TS file is approximately 750 KB, and the volume file When the default value of is 1 GB, about 1300 individual TS files can be stored in one volume file. 7 shows an example of a volume file in which a plurality of files are stored. At this time, data movement between servers included in the distributed file system may be performed in units of such volume files. For example, as the data migration for resharding described above is performed in units of such volume files, performance degradation due to the number of files can be prevented.

2) Expiration

In the distributed file system according to embodiments of the present invention, the maximum expiration time may be set and utilized, and all files may be automatically deleted before the maximum expiration time after creation. If the expiration time (TTL) is set in the file storage request, the file can be deleted after the TTL, and if the TTL is not set, the file can be deleted after the maximum expiration time. As described above, the deletion time of each file may be determined according to the maximum expiration time and expiration time for the file. At this time, in order to efficiently perform automatic deletion, files may be grouped and stored in a directory by survival time (deletion time), and files of a group having expired time may be deleted at once.

8 is a diagram illustrating an example of deleting a group whose expiration time has expired in one embodiment of the present invention. 8 shows an example of deleting files grouped in units of directories in units of directories at once (deleting files in a directory having an expiration time before the current time at a time).

3) Meta Data

The directory may be determined according to the expiration time, and may be stored at a specific offset of the volume file allocated at the time of storage. In order to access a file, a physical storage location needs to be managed, and this information is called meta data. Metadata can be managed in memory for fast file retrieval. However, in order to overcome the obstacle, the master stores the metadata in Redis, and the servers in the group can refer to the metadata stored in Redis as needed.

4) Failover

9 is a diagram showing an example of a group-only redis in an embodiment of the present invention. Servers belonging to the same group must keep the file information the same in all situations under normal and failure conditions. The file information (meta data) is stored in the corresponding redis for the group, and the file information is kept up-to-date through the redis when a failure is recovered. Storing file information on Redis can be done by the group's master. FIG. 9 shows an example in which the server 1-1 (111-2), which is the master server, stores metadata, which is file information of the group 1 110, in the redis sentinel 1 (114).

The distributed file system according to embodiments of the present invention does not actively recover files and metadata when a failure is overcome. Instead, if there is a request to read a file, you can query Redis to find the master that created the file and recover files and metadata from this server. This policy takes into account the fact that in live services, most requests are concentrated on new files. In FIG. 9, the slave servers Server 1-2 (112-2) and Server 1-3 (113-2) can use the metadata stored in the Redis Sentinel 1 (114). For example, the server 1-2 (112-2) and the server 1-3 (113-2) use the metadata stored in the redis sentinel 1 (114) in the event of a failure, such as the master server server 1-1 (111-). 2) can find and recover files and metadata from server 1-1 (111-2).

Since Redis is used only for groups, a separate Redis Instance may exist for each group. Redis Sentinel, which can perform better than Redis Cluster at a small scale, is applied because Redis Cluster can store metadata of a group. Can be. Leathers Sentinel applies the leader/follower pattern, so the leader can handle all write/read requests, and can be implemented to elect a server among the followers as a new leader with the agreement of the centers when the leader fails. have.

The master can select the replication target by sensing the health of slaves. On the other hand, the slave can determine whether it is isolated by detecting the robustness of the master, and if it is isolated, all metadata managed in its memory can be discarded to prevent metadata mismatch with the master. At this time, the slave reads the latest information recorded in the redis and can prevent data mismatch between servers. Masters and slaves may have a health check session with Transmission Control Protocol (TCP) to detect each other's robustness as quickly as possible. 10 is a diagram illustrating an example of a health check session between servers in an embodiment of the present invention. As shown in FIG. 10, the master server can detect the robustness of other servers by establishing a health check session while exchanging pings with a transmission control protocol (TCP) with each of the slave servers and feeders of the same group.

5) Chunked Transfer

The distributed file system according to embodiments of the present invention may support "HTTP chunked transfer". The chunked transfer is a method of transmitting a file being created, that is, a file whose size is not yet determined, as much as it is created. Therefore, it is possible to transfer the file relatively quickly after the file is completely generated. Latency is important in live service, so it is possible to reduce the delay part by chunk transmission.

11 is a diagram illustrating an example of a volume file for chunk transmission in an embodiment of the present invention. When allocating a volume file for file storage, a fixed-size file knows the start and end offsets from the volume file, so one of the available volume files can be allocated. When multiple fixed size files are requested to be stored, file writing may be simultaneously performed on one volume file. On the other hand, since the size of the file transmitted by the chunk transmission is unknown, other files cannot be stored in the allocated volume file until the corresponding chunk transmission is completed. If another file storage request is made, another volume file, or a new volume file, in which chunk transmission is not in progress may be allocated.

3. Interface

Most distributed file systems provide a client SDK (or library) for each programming language, and users can develop programs to read and write files using the SDK. Some distributed file systems also provide rest pool APIs, which have a separate server in charge of rest pool API processing, and access the distributed file system using a client SDK on this server. This is useful because the client SDK handles the complex parts of the distributed file system instead. However, when using the client SDK, it is necessary to consider distribution issues and compatibility issues, and sometimes these issues become even bigger. Here, the distribution and compatibility issues cannot refer to the SDK version in the system because the subject applying the modified SDK is a user, so it can mean an issue that must support all the deployed versions.

The distributed file system according to embodiments of the present invention can be easily used with only the restful API. General users can fully utilize the distributed file system by using PUT, GET, and DELETE APIs. In addition, by using the CLUSTER API, cluster information of a distributed file system can be queried, and a distributed file system can be used more efficiently by directly accessing a server that manages files.

1) APIs

(a) PUT / PUT2

PUT / PUT2 may be an API for storing files in a distributed file system. As described above, if a TTL option is specified for a file to be stored, the corresponding file may be automatically deleted from the distributed file system after the specified TTL time has elapsed. PUT may be in relay mode, and PUT2 may be in redirect mode.

(b) GET / GET2

GET / GET2 may be an API for reading a file stored in a distributed file system. GET may be in relay mode, GET2 may be in redirect mode.

(c) DEL

DEL may be an API for deleting a file stored in a distributed file system. As an example, the user can delete individual files using DEL.

(d) CLUSTER

CLUSTER may be an API for querying cluster information of a distributed file system. The user can access the server directly by obtaining the location (group index) of the file using the jc-hash algorithm and the "fnv1a" hash algorithm. For example, among "groups" fields searched through CLUSTER, master server information may be recorded in "m1", feeder server information in "m2", and slave server information in "m3". 12 is a diagram showing an example of information searched through CLUSTER in an embodiment of the present invention.

2) HTTP communication

The distributed file system according to embodiments of the present invention may communicate with an external client through an HTTP (eg, v1.0, v1.1) protocol. Distributed file systems can also use HTTP between servers, and can also use separate private commands (replication, feeding, health check, etc.).

3. Components

1) Overall View

13 is a diagram showing an example of components of a distributed file system in an embodiment of the present invention.

The coordinator analyzes user requests (or internal commands of the distributed file system) transmitted through HTTP, and can perform defined operations in consideration of cluster information and roles in groups. You can delegate to the Manager (FileManager).

The file manager can process low-level file storage and read commands, and the actual file input/output can be performed through the volume manager (VolumeManager), and the metadata can be managed in the metadata manager using Redis. .

2) Thread Model

14 is a diagram showing an example of a thread model in an embodiment of the present invention. To get the most out of the 10G NIC's performance, you need to utilize all the cores of the processor and minimize the wait time for network IO and disk IO. In the distributed file system according to embodiments of the present invention, a multi-threaded proactor pattern may be applied, and a thread pool is placed for each module (HTTP, coordinator, file manager) to perform IO It can be implemented to perform asynchronously so that there is no blocking.

15 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the machines (for example, machines 111, 112, 113, 121, 122, and 123) included in the distributed file system for a live service according to the present embodiment is illustrated in FIG. 15. It may correspond to (1500), it may be implemented through the server server (1500). Also, a file management method according to an embodiment may be performed by the computer device 1500 illustrated through FIG. 15.

15, the computer device 1500 may include a memory 1510, a processor 1520, a communication interface 1530, and an input/output interface 1540. The memory 1510 is a computer-readable recording medium, and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, a non-destructive large-capacity recording device such as a ROM and a disk drive may be included in the computer device 1500 as a separate permanent storage device separate from the memory 1510. In addition, an operating system and at least one program code may be stored in the memory 1510. These software components may be loaded into the memory 1510 from a computer-readable recording medium separate from the memory 1510. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD/CD-ROM drive, and memory card. In other embodiments, software components may be loaded into memory 1510 through communication interface 1530 rather than a computer-readable recording medium. For example, software components may be loaded into memory 1510 of computer device 1500 based on a computer program installed by files received over network 1560.

The processor 1520 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 1520 by memory 1510 or communication interface 1530. For example, the processor 1520 may be configured to execute a received command according to program code stored in a recording device such as the memory 1510.

The communication interface 1530 may provide a function for the computer device 1500 to communicate with other devices (eg, the above-described storage devices) through the network 1560. For example, requests, commands, data, files, etc. generated by the processor 1520 of the computer device 1500 according to program codes stored in a recording device such as the memory 1510 may be transmitted through a network ( 1560). Conversely, signals, commands, data, files, and the like from other devices may be received through the network 1560 to the computer device 1500 through the communication interface 1530 of the computer device 1500. Signals, instructions, data, etc. received through the communication interface 1530 may be transferred to the processor 1520 or the memory 1510, and files and the like may be further stored by the computer device 1500 (described above) Permanent storage device).

The input/output interface 1540 may be a means for interfacing with the input/output device 1550. For example, the input device may include a device such as a microphone, keyboard or mouse, and the output device may include a device such as a display or speaker. As another example, the input/output interface 1540 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 1550 may be configured as a computer device 1500 and one device.

Further, in other embodiments, the computer device 1500 may include fewer or more components than the components in FIG. 15. However, there is no need to clearly show most prior art components. For example, the computer device 1500 may be implemented to include at least a portion of the input/output device 1550 described above, or may further include other components such as a transceiver, database, and the like.

The communication method is not limited, and a communication method that utilizes a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, a broadcast network) that the network 1560 may include, as well as Bluetooth or NFC (Near Field Communication). Short-range wireless communications such as can also be included. For example, the network 1560 includes a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , Any one or more of the networks such as the Internet. Further, the network 1560 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. It is not limited.

16 is a flowchart illustrating an example of a file management method according to an embodiment of the present invention. The file management method according to the present embodiment may be performed by the computer device 1500 described above as an example. For example, the processor 1520 of the computer device 1500 may be implemented to execute control instructions according to code of an operating system included in the memory 1510 or code of at least one program. Here, the processor 1520 is a computer device 1500 so that the computer device 1500 performs steps 1610 to 1630 included in the method of FIG. 16 according to a control command provided by a code stored in the computer device 1500. Can be controlled. The computer device 1500 may be included in a distributed file system.

In step 1610, the computer device 1500 may set an expiration time for each of the files requested to be stored in association with the live service. In one embodiment, the computer device 1500 may set the expiration time of the first file requested to be stored, to the expiration time set in the storage request corresponding to the first file, or to the maximum expiration time set in the distributed file system. have. For example, when an expiration time is set in a request to store the first file, the expiration time of the first file may be set to an expiration time set in the storage request. On the other hand, when the expiration time is not set in the storage request, the expiration time of the first file may be set as the maximum expiration time that is previously set in the distributed file system. As described above, the expiration time can be used to determine the deletion time of the corresponding file. For example, if the storage time of the file is 08: 00 and the expiration time set for the file is 3 hours and 30 minutes, the deletion time of the corresponding file may be 11:30.

In operation 1620, the computer device 1500 may group and store files requested to be stored into a plurality of directories identified by a deletion time using the set expiration time. In one embodiment, the computer device 1500 determines the deletion time of each of the files according to the expiration time set for each of the files requested to be stored, and the files having the same deletion time or a deletion time within a preset time range are the same directory. Can be saved as. For example, files with a deletion time of 11:00 may be stored in the same directory. At this time, the directory can be identified by the deletion time "11:00." As another example, files included in a time range in which the deletion time is "between 11:01 and 11:30" may be stored in the same directory. In this case, the directory may be identified by the time range described above.

At this time, grouping and storing files into a directory may mean logical storage of files, and physical storage of files may be performed through steps 1621 to 1624 described later. Step 1625 describes a process for retrieving physically stored files.

In operation 1621, the computer device 1500 may generate and store a plurality of volume files. For example, a volume file in which about 1300 individual TS files are stored has already been described. The computer device 1500 may generate and store a plurality of volume files, each of which can store multiple files.

In step 1622, the computer device 1500 may allocate a volume file for each of the files requested to be stored.

In operation 1623, the computer device 1500 may store each of the files requested to be stored at a specific offset of the assigned volume file.

For logical storage of files, a directory may be determined according to an expiration time, while volume file allocation for physical storage of files may be performed in various ways. For example, files requested to be stored may be sequentially stored in one volume file or may be stored in a randomly selected volume file. As another example, volume files to be allocated to files to be stored may be sequentially selected.

In operation 1624, the computer device 1500 may store metadata including a specific offset in the memory 1510.

In operation 1625, the computer device 1500 may search for files stored in a plurality of volume files using metadata stored in the memory 1510.

As such, metadata stored in the memory 1510 may enable fast searching for files stored in the volume file.

Meanwhile, data movement between servers included in the distributed file system may be performed in units of such volume files. As described above, due to the characteristics of the live service, the files are relatively small, many, and contain volatile characteristics. As described above, in the distributed file system, when the number of files exceeds a certain level, the performance of the operating system and the distributed file system may deteriorate. Therefore, as in the present embodiment, it is possible to reduce the number of files by storing a plurality of individual files in a single volume file, and it is possible to prevent a performance degradation problem due to the number of files by moving data in units of volume files.

In operation 1630, the computer device 1500 may collectively delete files stored in the selected directory based on the current time and the deletion time. For example, the computer device 1500 may collectively delete files stored in a directory in which the deletion time is the current time or after the current time.

As described above, according to embodiments of the present invention, in consideration of the volatility of a file stored in a live service, a target file to be moved during scaling of a distributed file system may be limited to the latest file. Also, considering the volatility of the files stored in the live service, the deletion of the files can be optimized through a directory structure in which the files are grouped and stored by survival time. In addition, by storing relatively small and many files in a live service through a relatively large sized parallel volume file, it is possible to process small and many files that the live service has more quickly.

The system or device described above may be implemented as a hardware component, or a combination of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable gate arrays (FPGAs). , A programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and/or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The medium may be a computer that continuously stores executable programs or may be temporarily stored for execution or download. In addition, the medium may be various recording means or storage means in the form of a combination of single or several hardware, and is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks, And program instructions including ROM, RAM, flash memory, and the like. In addition, examples of other media include an application store for distributing applications, a site for distributing or distributing various software, and a recording medium or storage medium managed by a server. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

A computer device included in a distributed file system for a live service,
At least one processor implemented to execute readable instructions on the computer device
Including,
By the at least one processor,
Set an expiration time for each of the files requested to be stored in association with the live service,
Using the set expiration time, the files requested to be stored are grouped into a plurality of directories identified by a deletion time and stored,
Batch deletion of files stored in the selected directory based on the current time and the deletion time
Computer device characterized in that. According to claim 1,
By the at least one processor,
Setting the expiration time of the first file requested to be stored to an expiration time set in the storage request corresponding to the first file, or to a maximum expiration time preset in the distributed file system.
Computer device characterized in that. According to claim 1,
By the at least one processor,
Determining the deletion time of each of the files according to the expiration time set for each of the files requested to be stored, and storing files having the same deletion time or a deletion time within a preset time range in the same directory
Computer device characterized in that. According to claim 1,
By the at least one processor,
Batch deletion of files stored in a directory whose deletion time is the current time or after the current time
Computer device characterized in that. According to claim 1,
By the at least one processor,
Create and store multiple volume files,
Allocate a volume file for each of the files requested to be stored,
Each of the files requested to be stored is stored at a specific offset of the allocated volume file,
Storing metadata including the specific offset in a memory further included by the computer device
Computer device characterized in that. The method of claim 5,
By the at least one processor,
Retrieving files stored in the plurality of volume files using metadata stored in the memory
Computer device characterized in that. The method of claim 5,
Data movement between servers included in the distributed file system is performed in units of the volume file
Computer device characterized in that. A data processing method performed by a computer device included in a distributed file system for a live service,
Setting an expiration time for each of the files requested to be stored in association with the live service by at least one processor included in the computer device;
Storing, by the at least one processor, the files requested to be stored into a plurality of directories identified by a deletion time using the set expiration time; And
Batch deleting files stored in a directory selected based on the current time and the deletion time by the at least one processor
File management method comprising a. The method of claim 8,
The step of setting the expiration time,
Setting the expiration time of the first file requested to be stored to an expiration time set in the storage request corresponding to the first file, or to a maximum expiration time preset in the distributed file system.
File management method characterized by. The method of claim 8,
Grouping and storing the plurality of directories,
Determining the deletion time of each of the files according to the expiration time set for each of the files requested to be stored, and storing files having the same deletion time or a deletion time within a preset time range in the same directory
File management method characterized by. The method of claim 8,
Creating and storing a plurality of volume files;
Allocating a volume file for each of the files requested to be stored;
Storing each of the files requested to be stored at a specific offset of the allocated volume file; And
Storing metadata including the specific offset in a memory further included by the computer device
File management method further comprising a. The method of claim 11,
Data movement between servers included in the distributed file system is performed in units of the volume file
File management method characterized by. A computer program stored in a computer readable recording medium in combination with a computer device for executing the method of claim 8 on a computer device. A computer-readable recording medium on which a computer program for executing the method of any one of claims 8 to 12 is executed on a computer device.

Images