KR101559901B1 - System for storing distributed file in HDFS(Hadoop Distributed File System) and providing method thereof - Google Patents

Abstract

The present invention relates to an HDFS dispersion storage system and a providing method thereof and, more particularly, to a system for distributing and storing a file in a hadoop distributed file system (HDFS), a system for performing map-reduce operations for a moving picture stored in the HDFS, and a providing method thereof. The HDFS distribution storage system for distributing and storing the file in the HDFS according to one aspect of the present invention includes an obtaining module which obtains a replication factor of the HDFS, a division module which generates M division file blocks, and a control module.

Description

[0001] The present invention relates to an HDFS distributed storage system and a method for providing the HDFS distributed storage system.

The present invention relates to a system for distributedly storing files in a Hadoop Distributed File System (HDFS), a system for performing mapping tasks for moving images stored in an HDFS, and a method for providing the same.

The present invention has been produced as a result of research on the multi-channel AVB broadcasting and lighting convergence control cloud technology development of GRRC (GRRC aviation 2013-B02).

Cloud computing is a grouping of multiple computers into a cloud, which is a virtual computing group. It is not the responsibility of individual computers to store or manipulate data, but the concept of delegating to the cloud, a collection of computers. It means. Cloud computing has been used in many fields, and the field of big data has recently been highlighted. Big data refers to a field of research on methodologies for storing and processing data over petabytes (or more), beyond a large number of terabytes (Tera Bytes). Considering the definition of big data, broadcast media data such as video is the most representative big data. Because such large amounts of data can not be processed by a single computer, cloud computing is seen as a basic platform for big data processing.

Apache's Hadoop system, a multi-computer cloud environment based on the Java language, is one of the cloud computing environments for large data in the form of open software.

Since the data that Hadoop normally processes is at least a few hundred gigabytes in size, the data is not stored on a single computer, but rather is divided into blocks and stored on multiple computers. Therefore, Hadoop includes the Hadoop Distributed File System (HDFS), which allows input data to be divided and processed. Distributed data is stored in a Map-reduce process developed for high-speed parallel processing of large volumes of data in a cluster environment. Lt; / RTI > In addition, Hadoop supports data multiplexing to maintain data even in the event that some computers in the cloud fail, and even if some of the computers are down during the operation, other computers can take over the task and complete the task. (fault-tolerant) structure.

HDFS provides an environment for distributing file systems across a large number of computers in a cloud. When a file system is distributed over several computers, if one of the computers fails, the whole file system may be destroyed. To prevent this, the file system is stored in multiple computers by multiplexing each block. More specifically, in HDFS, a file is divided into a plurality of blocks and distributed to a plurality of computers. The size of each file block is basically 64 MB, which is a very large size considering that the number of blocks of a general file is several kB . In addition, one data block is redundantly stored by a predetermined number of replication factors, and each block is distributed and stored in triplicate as a basic set value. HDFS has a number of other features than the normal file system, one of which is that once each file is written, it can not be modified. Therefore, in order to change the contents of a file, the entire file must be rewritten. This is a hard work that allows files distributed over several nodes to maintain consistent contents.

HDFS has a master / slave structure. The HDFS cluster consists of a name node, which is a master server, and a number of data nodes, which distribute and store the contents of a file on a block basis. The data node reads and writes the requested file from the client, and performs block creation, deletion and replication according to the instruction of the name node. There is only one name node in a single HDFS cluster, and it manages the file system namespace, the file access control by the client, the meta information of various files stored in HDFS (HDFS namespace, Data node mapping information), and performs file system namespace operations such as opening / closing / renaming of files and directories. Also, the current state and the list of held data blocks are periodically received from each data node (Heartbeat).

When a client wants to create a file in HDFS, it first creates a file on its local file system. If the generation of the file is completed or the size of the data to be written to the file becomes the size of the preset data block (for example, 64M), the data node and the block ID to be stored in the data block are received from the name node do. Then, the file block is stored in the data node. If the client has more data to write to the file, repeat the above process on a block-by-block basis. On the other hand, in order to read data stored in the HDFS, the client acquires a data block location list (block ID and a list of data nodes storing the corresponding block) of the file from the name node, and stores the data Directly communicates with the node to read the data block.

On the other hand, MapReduce is a framework that supports parallel processing in a distributed environment to analyze large amount of data stored in HDFS, and is known as the most effective method among cloud computing parallelization methods so far known.

A maple deuce job that is processed in the MapReduce consists of a plurality of (n) map tasks and a plurality of (m) reduce tasks. In the map task, the raw data (input file) is divided into independent units of operation, processed and sorted into related data of the form <key, value>, and the task of reduction And the final result is generated.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram briefly showing a method of performing MapReduce in Hadoop. FIG.

Referring to FIG. 1, in Hadoop, an input file is divided into a plurality of splits and allocated to a map task in order to perform map de-misses on the input files stored in the HDFS. At this time, each split consists of a map entry which is an independent processing unit.

Each map task server analyzes the allocated splits on a map-entry-by-map basis to generate <key, value> data, and the server that has processed all of the allocated splits further allocates and processes splits. Because computers today have a large number of physical / logical cores on one CPU, several servers can be running on one computer.

The result data generated by the mapping process is shuffled, classified according to the key into a reduction group, and transmitted to the server for performing the reduction task. The reduction task is divided into the reduction group in the reduction entry unit And the final result (output file) is generated by processing. At this time, the generated output file is stored in HDFS.

On the other hand, the map reduction system consists of a number of task trackers and a job tracker. Job trackers are masters that manage task tasks (map tasks and redisplay tasks) performed by task trackers, and are generally located together on the server where the HDFS name node is running. Task tracker performs tasks requested by users on a task-by-task basis and is usually located on a server that acts as a data node.

When a client requests a MapReduce operation to a job tracker, the job tracker constructs a task list for each task tracker. The task tracker periodically sends a heartbeat message, and the job tracker returns the task ID to be processed in the return value of this message. The task tracker that receives the task ID fetches the related task information and the program to be executed from the HDFS, and executes the map task and / or the reuse task using the fork command. (Task tracker can perform multiple tasks at the same time). If Task Tracker fails or a new Task Tracker is added, Job Tracker recognizes and adds / removes it (the task of the failed Task Tracker is reassigned to another Task Tracker). On the other hand, in order to maximize the data locality of the data, the job tracker tries to find a server that is already in the input file block or a server in such a rack when searching for a server to perform the map task .

On the other hand, in Hadoop, several computers (data nodes) need to cooperate in using a file, so that even if one of them fails, the file operation can not be performed. In HDFS, each file block is stored in multiple data nodes in a redundant manner in order to solve this problem. At this time, the number of redundant storage is called a replication facror (replication factor). Typically, the replication factor is set to 3, but it is a tunable value.

On the other hand, video is the most representative big data field. One HD (High Definition) image has a large size ranging from several to several tens of gigabytes, and video shooting using a smart phone has become commonplace, and a considerable amount of video is being shared on the Internet. Although the video is the representative big data, the basic concept of cloud storage that collects them in a single virtual environment is used, and there is not much research on cloud environment for analyzing media such as video.

The present invention provides an HDFS distributed storage system capable of storing data stored in a file as much as a replication factor on an HDFS, and also capable of distributing the file to an HDFS in order to efficiently perform mapping on the file, and a method thereof .

In particular, the present invention provides a system and method for minimizing network traffic between a server that performs mapping tasks for the moving image file and a server that actually stores the moving image file in a distributed storage of moving image files in HDFS.

It is another object of the present invention to provide a system and method for efficiently mapping and distributing a video file distributed and stored as described above.

According to an aspect of the present invention, there is provided an HDFS distributed storage system for distributedly storing files in a HDFS (Hadoop Distribution File System), the division module for generating M divided file blocks corresponding to the files, M is a number of blocks when the file is divided into S / R sizes, R is a replication factor of the HDFS, S is a predetermined default block size, (1? I? M-R + 1) among the M divided file blocks is a file of the S / R of the file among the files (M-R + 2 &lt; = j &lt; M &gt;) among the M divided file blocks, and the data included in an area corresponding to the basic block size from a start point of an i- ), The HDFS distributed storage system from the start of the i-th block when the file hayeoteul divided by the size of the S / R includes data included in the area to the end point of the file is provided.

In one embodiment, the jth divided file block (M-R + 2? J? M) among the M divided file blocks is divided into R - &lt; / RTI &gt; (M-j + 1) th block.

In one embodiment, the file may be a video file.

In one embodiment, the control module may further store a header block including header information of the moving picture file in all the data nodes distributedly storing the M divided file blocks.

According to another aspect of the present invention, there is provided a method for performing Map-Reduce on a moving image file stored by the above-described HDFS distributed storage system, and a job tracker server and a plurality of task tracker servers Wherein the job tracker server divides the video file into unit splits with reference to a key frame, and splits the divided unit splits into corresponding unit split of the plurality of task tracker servers To a map task executed in a task tracker server storing a divided file block to be executed by the task tracker server, and the task tracker server performs a map task on data included in the unit split allocated to the task tracker server .

According to another aspect of the present invention, there is provided a method of providing an HDFS distributed storage system for distributing files in HDFS, the method comprising: generating M divided file blocks corresponding to the files, Wherein R is the replication factor of the HDFS and S is a predetermined basic block size and distributing the M divided file blocks to the data nodes on the HDFS, , And an i-th divided file block (1? I? M-R + 1) among the M divided file blocks is divided into a plurality of blocks from a start point of an i-th block when the file is divided into S / (M-R + 2? J? M) among the M divided file blocks includes data included in an area corresponding to a basic block size, I &lt; th &gt; block From this starting point HDFS distributed storage system provides a way to include the data contained in the area to the end point of the file is provided.

In one embodiment, the jth divided file block (M-R + 2? J? M) among the M divided file blocks is divided into R - &lt; / RTI &gt; (M-j + 1) th block.

In one embodiment, the file is a moving picture file, and the method for providing the HDFS distributed storage system stores a header block including header information of the moving picture file in all the data nodes distributedly storing the M divided file blocks Step &lt; / RTI &gt;

According to another aspect of the present invention, there is provided a computer-readable recording medium on which a program for performing the above-described method is recorded.

According to another aspect of the present invention there is provided an HDFS distributed storage system comprising a processor and a memory for storing a computer program executed by the processor, wherein the computer program, when executed by the processor, There is provided an HDFS distributed storage system for allowing a storage system to perform the above-described method.

According to another aspect of the present invention, there is provided a method for providing a moving picture deuce system that performs a map deuce on a moving picture file stored by the HFDS distributed storage system and includes a job tracker server and a plurality of task tracker servers, ) The job tracker server divides the moving image file into unit splits based on a key frame, (b) the job tracker server divides each divided unit split into a corresponding unit split of the plurality of task tracker servers (C) assigning the divided unit split to the task tracker server; and (c) assigning the unit split to the task tracker server. Providing a video maple deuce system including performing a map task on data This method is provided.

According to an embodiment of the present invention, it is possible to provide a Hadoop system that can store large-capacity files and perform parallel analysis and analysis at high speed. That is, according to an embodiment of the present invention, data included in the file can be stored redundantly as much as a replication factor on the HDFS, thereby increasing the robustness of the file.

In addition, when the file stored in the HDFS is a moving image, the amount of communication required to perform the mapping process on the moving image can be minimized while enhancing the parallelism.

In addition, the header information necessary for the analysis and processing of the moving image may be stored in all the servers that distribute and store the moving image, so that each server that maps the moving image can refer to the header information stored in its local storage have. Therefore, there is an effect that the network overhead that may occur when the header information is read from a remote location is reduced.

BRIEF DESCRIPTION OF THE DRAWINGS A brief description of each drawing is provided to more fully understand the drawings recited in the description of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram briefly showing a method of performing MapReduce in Hadoop. FIG.
2 is a block diagram schematically illustrating an HDFS distributed storage system according to an embodiment of the present invention.
3A to 3C are views for comparing a divided file block generated in the HDFS distributed storage system according to an embodiment of the present invention with a file block generated by the conventional HDFS.
4 is a diagram for explaining a problem that may occur when a moving image file is divided into file blocks of a predetermined size.
FIG. 5 is a diagram for explaining a maple deuces system according to an embodiment of the present invention.
FIG. 6 is a diagram for explaining an example in which a moving picture file is divided into splits and a divided split is allocated to a task tracker server by a job tracker server according to a maple deuces system according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise.

In this specification, the terms "comprises" or "having" and the like refer to the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, But do not preclude the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof.

Also, in this specification, when any one element 'transmits' data to another element, the element may transmit the data directly to the other element, or may be transmitted through at least one other element And may transmit the data to the other component. Conversely, when one element 'directly transmits' data to another element, it means that the data is transmitted to the other element without passing through another element in the element.

Hereinafter, the present invention will be described in detail with reference to the embodiments of the present invention with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements.

2 is a block diagram schematically illustrating an HDFS distributed storage system according to an embodiment of the present invention.

Referring to FIG. 2, an HDFS distributed storage system 100 may be provided to implement a method of providing an HDFS distributed storage system according to an embodiment of the present invention.

When the client 200 requests the HDFS 300 to store a file to be stored on the HDFS 300, the HDFS distributed storage system 100 generates a plurality of divided file blocks corresponding to the requested file, A plurality of divided file blocks may be distributedly stored in the data nodes (for example, 320-1 to 320-3) constituting the HDFS 300. [

Meanwhile, the file may be a moving picture file, and the moving picture file that can be handled by the present invention is not limited to the storage format of the moving picture such as avi, mov, mp4, wmv format, and is widely known as divx, mpeg, Lt; RTI ID = 0.0 &gt; codecs. &Lt; / RTI &gt;

Meanwhile, the moving picture file may include a plurality of frames (still images). Some of the frames included in the moving picture may be key frames. The key frame may include a frame having all of the image data at the corresponding time point and having no dependency on the before / after frame. In one embodiment of the present invention, a key frame may include a frame that has all of the image data at the corresponding point in time and does not receive the reference of the previous frame.

In addition, the moving picture file may include header information. The HDFS distributed storage system 100 may divide the moving image into a plurality of divided file blocks, and the HDFS distributed storage system 100 may divide the moving image into a plurality of divided file blocks, for example, The location information of the key frame may be obtained from the header information.

In order to store each of the generated plurality of divided file blocks, the HDFS distributed storage system 100 stores a data node in which the corresponding block is stored for each file block, to a name node 310 included in the HDFS You can query. In addition, if the name node 310 transmits information (for example, ID of a data node) about a data node to which the corresponding block is actually stored in response to a query, the corresponding block may be transmitted to the data node for storage .

 Meanwhile, the HDFS distributed storage system 100 may include a partition module 110 and a control module 120. According to an embodiment of the present invention, some of the above-mentioned components may not necessarily correspond to components necessary for the implementation of the present invention, and in accordance with an embodiment, the distributed storage system 100 It goes without saying that more components may be included.

The distributed storage system 100 may include hardware resources and / or software required to implement the technical idea of the present invention, and it is understood that one means a physical component or one device no. That is, the distributed storage system 100 may mean a logical combination of hardware and / or software provided to implement the technical idea of the present invention. If necessary, the distributed storage system 100 may be installed in a separate apparatus, The present invention may be embodied as a set of logical structures for realizing the technical idea of the present invention. Also, the distributed storage system 100 may mean a set of configurations separately implemented for each function or role for implementing the technical idea of the present invention. For example, the partitioning module 110 and / or the control module 120 may be located in different physical devices, or may be located in the same physical device. In addition, according to an embodiment, elements constituting each individual module such as the acquisition module 110, the partitioning module 110 and / or the control module 120 are also located in different physical devices, The elements located on top of each other may be combined with each other to realize the function performed by each individual module.

In this specification, a module may mean a functional and structural combination of hardware for carrying out the technical idea of the present invention and software for driving the hardware. For example, the module may mean a logical unit of a predetermined code and a hardware resource for executing the predetermined code, and it does not necessarily mean a physically connected code or a kind of hardware. Can be easily deduced to the average expert in the field of &lt; / RTI &gt;

The control module 120 may control the functions and / or resources of other configurations (e.g., the partitioning module 110, etc.) included in the distributed storage system 100 according to an embodiment of the present invention.

The partitioning module 110 may be divided into a plurality of divided file blocks corresponding to a target file stored in the HDFS.

As described above, in a conventional conventional HDFS, a large file can be divided into a predetermined size (i.e., a basic block size, for example, 64M, 96MB, etc.) and the refreshment factor of the HDFS , 3). Therefore, in the conventional HDFS, there may exist the same file block as the replication factor.

On the other hand, the plurality of divided file blocks generated by the partitioning module 110 according to an embodiment of the present invention may be different from each other, and data included in the target file is divided into a plurality of divided file blocks, As shown in FIG. Meanwhile, the number of the plurality of divided file blocks generated by the partitioning module may be the number of blocks (M) when the target file is divided into S / R sizes (where R is a replication factor of HDFS, S is the basic block size).

Hereinafter, with reference to FIG. 3A to FIG. 3C, a description will be made of a specific method of generating the M divided file blocks corresponding to the target file by the partitioning module 110. FIG.

FIG. 3A illustrates a target file to be stored by the HDFS distributed storage system according to an exemplary embodiment of the present invention. FIG. 3B illustrates a divided file block generated in the conventional HDFS. FIG. FIG. 5 is a diagram illustrating a split file block generated by the HDFS distributed storage system according to an exemplary embodiment. In the example of FIGS. 3A to 3C, M is 9, the basic block size is 96 MB, and the replication factor of the HDFS is 3.

Referring to FIG. 3B, in the conventional HDFS, the target file 1 shown in FIG. 3B can be divided into predetermined basic block size units to generate three file blocks 11, 12, and 13 . Meanwhile, according to the conventional HDFS, the three file blocks 11, 12 and 13 can be redundantly stored by the replication factors (i.e., 3). Therefore, three file blocks 11 to 13 may be redundantly present on the conventional HDFS.

Meanwhile, according to an embodiment of the present invention, the partitioning module 110 may generate M different divided file blocks corresponding to the target file. Here, M is the number of blocks when the target file is divided into S / R size, R is the replication factor of the HDFS, and S is the basic block size.

In the first embodiment of the present invention, the data included in the M divided file blocks generated by the division module 110 may be as follows.

1) the i-th divided file block (1? I? M-R + 1) among the M divided file blocks starts from the start point of the i-th block when the target file is divided into S / And may include data included in an area corresponding to the basic block size.

2) The j-th divided file block (M-R + 2? J? M) among the M divided file blocks is divided from the starting point of the i-th block when the target file is divided into S / And data included in an area up to an end point of the target file.

Meanwhile, according to the second embodiment of the present invention, in the case 2), the jth divided file block is divided into R- (M -j + 1) &lt; th &gt; block.

FIG. 3C is a diagram showing a divided file block corresponding to a target file as shown in FIG. 3A generated by the dividing module 110 according to the second embodiment. The split file blocks 21 to 27 are the file blocks generated by the 1) rule, and the split file blocks 28 and 29 are the file blocks generated by the 2) rule.

When each block when the target file is divided into S / R sizes is referred to as a sub-block, the fifth divided file block 25 in FIG. 3C is applied to the starting point of the fifth sub-block 5 (5), (6), and (7) which are areas corresponding to the basic block size (which is equal to the size of the sub-block R).

In the meantime, the eighth divided file block 28 in FIG. 3C is a block included in the area from the start point of the eighth sub-block 8 to the end point of the target file (that is, the end point of the sub-block 9) (M-j + 1) = 3- (9-8 + 1) = 1 th sub-block (i.e., sub-block 1) from the start point of the target file And may include data included in an area up to the end point of the &lt; RTI ID = 0.0 &gt; As a result, the eighth divided file block 28 may include data included in the subblocks 8, 9, and 1.

Referring back to FIG. 2, the control module 120 may distribute the M divided file blocks to data nodes on the HDFS.

The control module 120 requests the ID of the data node to be stored / duplicated in the corresponding divided file block to the name node 310 included in the HDFS 300 whenever the divided file block is generated or determined, The segmented file block may be stored in the data node (e.g., 320-1) based on the information that the node 310 returns.

Meanwhile, each of the divided file blocks generated by the partitioning module 110 is not duplicated and stored as redundant as the replication factor. However, since each of the divided file blocks generated by the partitioning module 110 already includes duplicated data as much as the replication factor, the data constituting the target file can be redundantly stored in the HDFS as much as the replication factor .

There is no limitation on the type (or type) of files that can be stored in the HDFS by the HDFS distributed storage system 100. [

On the other hand, when the file stored in the HDFS is a moving image file, a header area having meta information about the moving image needs to be specifically managed. All the servers that process the video must process the header before processing the video in earnest to be able to process the appropriate video. That is, each task tracker server must know the location information of each frame, which is a map entry constituting the split, in order to perform map ridding for the allocated split. To check position information of each frame, You need to reference the file. Therefore, if all the task tracker servers that process the split request data to the data node storing the header portion, not only the overhead for the data node storing the header is increased but also the performance is decreased due to the network delay and the like . Therefore, the control module 120 may further store a separate header block including header information of the moving picture file in all the data nodes that distribute and store the plurality of divided file blocks. Then, the task tracker server can then read the header information from its local storage, thereby solving the above-described problem.

As described above, in the MapReduce method, a large-capacity file is divided into map entry units, which are independent operation units, and the map step is performed. In order to improve the speed, , And one split is allocated to one server (i.e., a map task tracker server). The map task tracker server sequentially processes the map entries included in the allocated split. The performance of the MapReduce method relies heavily on how to divide the input large files into map entries, which are the minimum processing units, and group them into splits. It is better to keep the size of the map entry small considering the simplification of the algorithm and the flexibility of the split size. It is better to keep the split size of the job allocation unit allocated to the server small enough so that parallelization can be properly performed. However, It is good enough to be large enough to do so. In short, it is important to find an appropriate harmonization point for the size of the split.

Since an embodiment of the present invention is particularly directed to a moving picture file and a smaller number of map entries, it is preferable that each frame of the moving picture can be configured as a unit of a map entry in one embodiment of the present invention, .

On the other hand, one problem that occurs when each frame of a moving picture is composed of one map entry is that the majority of the frames included in the moving picture are compressed depending on the preceding or following frame on the basis of the compression technique. Since it is possible to increase the compression rate of the current frame by assuming that the moving image is a sequence of still pictures having continuous motion, it is possible to increase the compression rate of the current frame. . Therefore, in order to process any one frame (i.e., one map entry), unless the frame is a key frame (a frame that does not depend on the preceding and succeeding frames but has all of the image data and can be independently analyzed) You must reference the frame. In other words, when each frame of the moving picture is made into a single map entry, the part of each Hadoop maple deuce pattern assumes that each map entry is independent. However, if MapleDesus treats the map entries contained in one split in order in the file, even if each frame of the moving image is made into a single map entry, as long as each split does not have an image dependent on another split, There is no problem in processing each frame in the moving image.

As described above, each split must be completely independent of each other, since it must be able to be processed in a separate server. Therefore, each split should be separated by key frames that are not dependent on other frames. This may mean that all splits must start at the beginning of one key frame and end at a point just before the start of another key frame.

However, if the moving image file block is uniformly divided into basic block size units and stored in the HDFS, the boundaries of the blocks stored in the respective data nodes and the split frames based on the key frames may be different from each other. That is, as shown in FIG. 4, when the split is divided based on the key frame, the phenomenon that the boundary is slightly shifted due to the problem that the boundary between the key frame boundary and the file block do not coincide is shown as shown in FIG. 4 I have. Therefore, even if you assign a job to process a split that corresponds to a server that owns a file block, the server must read the spoofed file fragments beyond the file block from the outside.

The moving picture deuce system according to an embodiment of the present invention divides the moving picture file into respective splits with reference to the key frame, and by using the HDFS distributed storage system according to the embodiment of the present invention, Can be solved. In the example of FIG. 4, each split is set as close as possible to the size of the file block. However, in the moving picture deuce system according to the embodiment of the present invention, the moving picture file is split It may be divided.

FIG. 5 is a diagram for explaining a mapping task system 400 according to an embodiment of the present invention. FIG. 6 illustrates a case where a moving picture file is divided into splits by the mapping task system 400, FIG. 4 is a diagram for explaining an example in which the task tracker server is allocated to the task tracker server.

As described above, the moving picture file v stored in the HDFS may be processed by the mapping system 400 according to an embodiment of the present invention. The mapping system 400 may include one job tracker server 410 and a plurality of task tracker servers (e.g., 420-1 through 420-2) as in a conventional Hadoop system, The server can act as a data node. That is, each of the task tracker servers 420-1 to 420-2 may store at least a part of the divided file blocks generated from the moving picture file in its local storage 421-1 to 421-4 . On the other hand, the job tracker server 410 may allocate a split to each of the tracker servers 410-1 to 420-2, and the task tracker servers 410-1 to 420-2 may include a split The map task can be performed on the data.

As described above, since the splits processed by the task tracker servers 420-1 to 420-2 must be independent from each other, the job tracker server 410 divides the video file to be processed on the basis of the key frame , Which may mean that the boundary of each split should be split so that it is the starting point of the key frame. As shown in FIG. 6, the job tracker server 410 divides a motion picture file 2 to be subjected to mapping to a unit split (S1 to S9) corresponding to the number of key frames on the basis of a key frame .

Meanwhile, the job tracker server 410 may allocate the divided unit split to the task tracker server storing the split file block including the split.

6, the job tracker server 410 transmits a unit split S1 to the task tracker server storing any one of the split file blocks 21, 28, and 29 including the unit split S1 Can be assigned.

Further, the unit split S1 can be allocated to the task tracker server storing the split file block 21 including the unit split S2.

Further, the unit split S3 can be allocated to the task tracker server storing any one of the split file blocks 22 and 24 including the unit split S3. However, since the split file blocks 21 and 24 include only a part of the unit split S3, the unit split S3 may not be allocated to the task tracker server storing the split file blocks 21 and 24. [

In addition, the unit split S1 can be allocated to the task tracker server storing any of the split file blocks 23 and 24 including the unit split S4.

The job tracker server may allocate the unit split to the task tracker server storing the split file block including the unit split in the same manner for the remaining unit split (S5 to S9).

In this manner, the task tracker server 410 allocates a split to each of the task tracker servers 420-1 to 420-4 so that the task tracker server, which performs the map task for each split, You only need to access the file block, and you may not need to retrieve the data remotely from another server to process the splits allocated to it.

Meanwhile, according to an embodiment, the HDFS distributed storage system 100 may include a processor and a memory for storing a program executed by the processor. The processor may include a single-core CPU or a multi-core CPU. The memory may include high speed random access memory and may include non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices. Access to the memory by the processor and other components can be controlled by the memory controller. Here, if the program is executed by a processor, the distributed storage system 100 according to the present embodiment may be configured to perform a method of providing a system for distributed storage in the above-described HDFS.

Meanwhile, the HDFS distributed storage system providing method and the video map devise system providing method according to an embodiment of the present invention may be implemented as computer-readable program instructions and stored in a computer-readable recording medium. The control program and the target program according to the embodiment can also be stored in a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored.

Program instructions to be recorded on a recording medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of software.

Examples of the computer-readable recording medium include magnetic media such as a hard disk, a floppy disk and a magnetic tape, optical media such as CD-ROM and DVD, a floptical disk, And hardware devices that are specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. The above-mentioned medium may also be a transmission medium such as a light or metal wire, wave guide, etc., including a carrier wave for transmitting a signal designating a program command, a data structure and the like. The computer readable recording medium may also be distributed over a networked computer system so that computer readable code can be stored and executed in a distributed manner.

Examples of program instructions include machine language code such as those produced by a compiler, as well as devices for processing information electronically using an interpreter or the like, for example, a high-level language code that can be executed by a computer.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

It is intended that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents. .

Claims (11)

An HDFS distributed storage system that distributes files to a Hadoop Distribution File System (HDFS)
Wherein M is a number of blocks when the file is divided into S / R sizes, R is a replication factor of the HDFS, and S is a predetermined number A default block size; And
And a control module for distributively storing the M divided file blocks on data nodes on the HDFS,
Wherein the i-th divided file block (1? I? M-R + 1)
The data included in an area corresponding to the basic block size from a start point of an i-th block when the file is divided into S / R sizes,
The M-th divided file block (M-R + 2? J? M)
An HDFS distributed storage system including data included in an area from a start point of an i-th block to an end point of the file when the file is divided into S / R sizes.
The method according to claim 1,
The M-th divided file block (M-R + 2? J? M) of the M divided file blocks is divided into R- (M-j +1) th block of the HDFS file system.
The method according to claim 1,
Wherein the file is a moving image file.
The method of claim 3,
The control module includes:
And a header block including header information of the moving picture file is further stored in all the data nodes distributedly storing the M divided file blocks.
4. A moving picture deuce system for performing map-reduction on a moving picture file stored by the system according to claim 3 and including a job tracker server and a plurality of task tracker servers,
The job tracker server includes:
A map executed by a task tracker server that divides the video file into unit splits with reference to a key frame and stores split divided unit blocks into divided file blocks including corresponding unit splits among the plurality of task tracker servers; Task,
The task tracker server,
A video maple deuces system that performs a map task on data contained in a unit split assigned to itself.
A method for providing an HDFS distributed storage system for distributing files to HDFS,
Wherein M is a number of blocks when the file is divided into S / R sizes, R is a replication factor of the HDFS, S is a predetermined number of blocks Basic block size; And
Distributing the M divided file blocks to data nodes on the HDFS,
Wherein the i-th divided file block (1? I? M-R + 1)
The data included in an area corresponding to the basic block size from a start point of an i-th block when the file is divided into S / R sizes,
The M-th divided file block (M-R + 2? J? M)
And data included in an area from a start point of an i-th block to an end point of the file when the file is divided into S / R sizes among the files.
The method according to claim 6,
The M-th divided file block (M-R + 2? J? M) of the M divided file blocks is divided into R- (M-j +1) th block of the HDFS file system.
The method according to claim 6,
The file is a video file,
The HDFS distributed storage system providing method includes:
Storing a header block including header information of the moving image file in all the data nodes distributed and storing the M divided file blocks.
9. A computer-readable recording medium on which a program for carrying out the method according to any one of claims 6 to 8 is recorded.
As an HDFS distributed storage system,
A processor; And
A memory for storing a computer program executed by the processor,
The computer program causes the distributed storage system to perform the method of any one of claims 6 to 8 in the HDFS when executed by the processor.
A method for providing a moving picture deuce system that performs a map deuce on a moving picture file stored by the system of claim 3 and includes a job tracker server and a plurality of task tracker servers,
(a) the job tracker server dividing the moving image file into unit splits based on a key frame;
(b) the job tracker server allocating each divided unit split to a map task executed in a task tracker server storing a divided file block including a corresponding unit split among the plurality of task tracker servers; And
and (c) each of the task tracker servers allocated with the split unit split performs a map task on data included in the unit split allocated to the task tracker server.

Images