KR20170013587A - Low power strorage device based on hadoop and managin module thereof - Google Patents

Low power strorage device based on hadoop and managin module thereof Download PDF

Info

Publication number
KR20170013587A
KR20170013587A KR1020150106433A KR20150106433A KR20170013587A KR 20170013587 A KR20170013587 A KR 20170013587A KR 1020150106433 A KR1020150106433 A KR 1020150106433A KR 20150106433 A KR20150106433 A KR 20150106433A KR 20170013587 A KR20170013587 A KR 20170013587A
Authority
KR
South Korea
Prior art keywords
hadoop
module
power storage
storage device
present
Prior art date
Application number
KR1020150106433A
Other languages
Korean (ko)
Inventor
이현화
한재용
임동일
Original Assignee
디포커스 (주)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 디포커스 (주) filed Critical 디포커스 (주)
Priority to KR1020150106433A priority Critical patent/KR20170013587A/en
Publication of KR20170013587A publication Critical patent/KR20170013587A/en

Links

Images

Classifications

    • G06F17/30218—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/26—Power supply means, e.g. regulation thereof
    • G06F1/32—Means for saving power
    • G06F1/3203—Power management, i.e. event-based initiation of a power-saving mode
    • G06F1/3234—Power saving characterised by the action undertaken
    • G06F1/325—Power saving in peripheral device
    • G06F1/3268—Power saving in hard disk drive
    • G06F17/3007—

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Power Sources (AREA)

Abstract

The present invention relates to a low-power storage device based on a hadoop and a low-power storage device management module based on a hadoop. The low low-power storage device based on a hadoop and the low-power storage device management module based on a hadoop according to the present invention may include a storage control part including a control module and a hadoop processor high-speed accelerator; a backplane; and a storage expander. Also, according to the present invention, the low-power storage device management module based on a hadoop includes a kernel module including a file system module; a hadoop module; and a hadoop control module. So, it is possible to provide a low-power storage device based on a hadoop to an appliance.

Description

≪ Desc / Clms Page number 1 > HOWOAD-BASED LOW POWER STORAGE DEVICE AND MANAGEMENT MODULE < RTI ID = 0.0 >

The present invention provides a Hadoop-based low-power storage device and a Hadoop-based low-power storage device management module.

Recently, Hadoop software has been used as a way to effectively distribute Big Data.

However, even with Hadoop software, as the amount of data increases, a greater number of appliances are required at the server end for data storage and analysis. The increase in these appliances results in high power consumption for server operation and high cost of management of appliances.

Nevertheless, Hadoop-based low-power storage devices are not available. Also, a management module for controlling the Hadoop-based low-power storage device is not provided.

In addition, existing Hadoop-based servers use the Name Node redundancy scheme, which causes a problem of high cost due to having a shared storage.

It is an object of the present invention to provide a Hadoop-based low-power storage device in an appliance.

Another object of the present invention is to provide a Hadoop-based low-power storage device management module for providing a Hadoop-based low-power storage device in an appliance, and a Hadoop- A management module for operating the low-power storage device of FIG.

In addition, the present invention provides a management module for operating the Hadoop-based low-power storage device.

More particularly, the present invention is to provide a device that minimizes the power consumed in distributing and analyzing big data by controlling the Hadoop-based low-power storage device through the management module.

Also, the present invention provides a management module optimized for managing an appliance to which the Hadoop-based low-power storage device and the management module are applied.

A Hadoop-based low-power storage apparatus according to an embodiment of the present invention includes a storage controller including a control module and a Hadoop processor accelerator; Backplane; And a storage expander.

The Hadoop-based low-power storage device may further include a solid state disk (SSD) cache.

The Hadoop processor accelerator may include an LZ4 algorithm operation circuit.

The size of the block processed by the LZ4 algorithm operation circuit may be 256 KB.

The storage controller may include four input / output lines.

A Hadoop-based low-power storage device management module according to another embodiment of the present invention includes a kernel module including a file system module; Hadoop module; And a Hadoop control module.

The Hadoop-based low-power storage device management module may further include a Hadoop processor accelerator module.

The Hadoop processor accelerator module may be compressing and decompressing data with the LZ4 algorithm.

The block size of the data may be 256 KB.

The Hadoop-based low-power storage device management module may further include a common information model management module.

The low-power storage apparatus according to the present invention can process a large amount of data at a high speed using a Hadoop-based file system, but consumes less power than a conventional server.

1 is an illustration of an Hadoop-based appliance management system in accordance with the present invention;
2 is a conceptual diagram of a Hadoop-based low-power storage device related to the present invention;
3 is a conceptual diagram of a Hadoop-based low-power storage device management module related to the present invention;

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. The terms " comprises "and / or" comprising "used in the specification do not exclude the presence or addition of one or more other elements in addition to the stated element.

1 is an exemplary diagram of a Hadoop-based appliance management system in accordance with the present invention.

Referring to FIG. 1, the Hadoop-based appliance management system may include a Hadoop-based appliance 10 and a Hadoop storage appliance server 100.

In this specification, an appliance means hardware such as a server or storage. The appliance may be an information device that is pre-installed with software and sold in a state optimized for a specific task. The user can use the appliance by connecting the power supply at the time of purchase without installing a separate program such as installation or setting of the integrated equipment operating system or application software.

In particular, the Hadoop-based appliance 10 refers to an appliance having Hadoop software installed therein. Hereinafter, the Hadoop-based appliance 10 may be abbreviated as an appliance 10.

The appliance 10 may include a management server 20, a master node 30, and a data node 40.

The management server 20 can connect the appliances in network units. At this time, the management server 20 can transmit the data received from the Hadoop storage appliance server 100 to the necessary node without transmitting the data to all the nodes. In addition, the management server 20 can manage and control the master node 30 in operation of the Hadoop distributed file system (HDFS).

The master node 30 stores a large amount of data in the Hadoop distributed file system and performs parallel computation through Map Reduce. In addition, the master node 30 may store meta information of files and directories in the Hadoop distributed file system. The meta information may include a structure of a directory, information on a file, and physical location information on which a file is stored. In addition, the meta information may include block mapping information to the data node 40.

The data node 40 reads and stores the requested file. In addition, the data node 40 may receive a command from the master node 30 and perform a role of creating, deleting, or duplicating a block.

The master node 30 and the data node 40 may be programs that perform the functions described above. Alternatively, the master node 30 and the data node 40 may be part or all of the hardware operated by the program performing the function. Or the master node 30 and the data node 40 may be hardware in which the program is stored.

The Hadoop storage appliance server 100 transmits a command to the appliance 10 to read or store data. It is also possible to receive the result of the operation of the appliance 10.

The Hadoop storage appliance server 100 may include a low-power storage device 110, a device management module 120, and an appliance management module 130.

The low-power storage device 110 is a device for operating the Hadoop-based appliance management system of the present invention at low power.

The device management module 120 collectively controls each component of the Hadoop storage appliance server 100 and each component of the appliance 10. The device management module 140 may execute one or more programs or applications for operation of the Hadoop-based appliance management system according to the present invention. In particular, the device management module 120 may control the functions and operations of the low-power storage device 110. Accordingly, the Hadoop-based appliance management system including the low-power storage device 110 can be operated with low power.

The device management module 120 may be implemented as a central processing unit (CPU), a microprocessor unit (MPU), a microcontroller unit (MCU), an application processor (AP), or any type of processor well known in the art And the like.

The appliance management module 130 can generate various statistical data on the storage result of the appliance 10 and the big data. To this end, the appliance management module 130 may monitor the resource allocation result of the Hadoop-based appliance management system through the Hadoop storage appliance server 100 and accordingly availability. That is, the appliance management module 130 manages the performance of the Hadoop-based appliance management system of the present invention. In addition, the appliance management module 130 may provide the user with information about the performance, information on the statistics, and information on the security status. To this end, the appliance management module 130 may provide the user with an interface for informing the user of the information and receiving various commands from the user.

The appliance 10 and the Hadoop storage appliance server 100 store various data, commands, and / or information. The appliance 10 and the Hadoop storage appliance server 100 may store one or more applications for operating the Hadoop-based appliance management system. The appliance 10 and the Hadoop storage appliance server 100 may temporarily or non-temporarily store data or the like transmitted from an external device or a server. The appliance 10 and the Hadoop storage appliance server 100 may be implemented in any form of computer readable media including nonvolatile memory such as ROM, EPROM, EEPROM, flash memory, etc., hard disk, removable disk, or any form of computer readable storage medium known in the art And the like.

1 and 2 are classified according to functions or operations, and thus may be classified according to other criteria. In addition, since the illustrated elements are not essential elements, they may not include some elements or may further include additional elements.

According to an embodiment of the present invention, the operation method of the Hadoop-based appliance management system and the operation of various modules can be implemented as a code readable by a processor on a medium on which the program is recorded. Examples of the medium that can be read by the processor include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc., and may be implemented in the form of a carrier wave (e.g., transmission over the Internet) .

The above-described Hadoop-based appliance management system is not limited to the configuration and method of the embodiments described above, but the embodiments may be modified so that all or some of the embodiments are selectively And may be configured in combination.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. , And are not intended to limit the scope of the present invention. It will be apparent to those skilled in the art that other modifications based on the technical idea of the present invention may be practiced without departing from the scope of the invention disclosed herein.

Hereinafter, the Hadoop-based low-power storage device 200 according to the present invention will be described in detail with reference to FIG.

Backplane 210,

In the present invention, the backplane 210 is a circuit board, a part of the low-power storage device is connected to a connector, and the backplane has a bus as a data exchange path. Storage may be connected to the backplane 210.

Storage Expander (220)

In the present invention, the storage expander 220 increases the processing allowable capacity of the storage connected to the backplane 210.

In one embodiment of the present invention, the processing capacity of the storage is greater than 48 terabytes (TB).

The storage controller 230,

The Hadoop-based low-power storage device according to the present invention includes a control module 240 and a Hadoop processor accelerator 250.

In one embodiment of the present invention, the storage controller 230 includes four input / output lines. The four input / output lines may be an interface of a bus of the backplane 210. The interface of the bus may be a PCI-EXPRESS 2.0 X4 Lane.

The control module 240,

The storage controller 230 according to the present invention includes a control module 240. The control module may be a low power consumption central processing unit (CPU).

In one embodiment according to the present invention, the low power consumption central processing unit may be a silver-in-order system, and may be an Atom processor, specifically a product name of Intel Corporation.

Hadoop processor acceleration accelerator (250)

The storage controller 230 according to the present invention includes a Hadoop processor accelerator 250. The Hadoop processor accelerator accelerator 250 accelerates the processing speed of compression and decompression of processing data, thereby increasing the data processing speed.

In one embodiment of the present invention, the Hadoop processor accelerator 250 includes an LZ4 algorithm operation circuit.

In one embodiment of the present invention, the size of a block processed by the LZ4 algorithm operation circuit is 256 KB (kilobyte).

Solid State Disk (SSD) cache

In one embodiment of the present invention, a solid state disk (SSD) cache is further included. The SSD cache can read the necessary data blocks in advance in Hadoop, so that it can cache the blocks at the time of HeartBeat, thereby improving the data read performance.

Hereinafter, the Hadoop-based low-power storage device management module 300 according to the present invention will be described in detail with reference to FIG.

Hardware device 310,

The Hadoop-based low-power storage device management module according to the present invention may further include a hardware device 210, which may be the Hadoop-based low-power storage device described above. All of the modules included in the Hadoop-based low-power storage device management module according to the present invention may be connected to the hardware device 210. [

The kernel module 330,

The Hadoop-based low-power storage device management module according to the present invention includes a kernel module 330. The kernel module 330 may input data to the hardware device 310, receive data from the hardware device 310, or input a command for a slave device included in the hardware device 310. [

The file system module 320,

In the present invention, the kernel module 330 includes a file system module 320. The file system module 320 supports a Hadoop Distribution File System (HDFS).

Hadoop Processor Accelerator Module (340)

In one embodiment of the present invention, the Hadoop-based low-power storage device management module further includes a Hadoop processor accelerator module 340. The Hadoop processor accelerator module 340 can increase the data processing speed by accelerating the processing speed of decompressing and decompressing the processing data using the Hadoop processor accelerator accelerator 250.

In one embodiment of the present invention, the Hadoop processor accelerator module 340 compresses and releases data with the LZ4 algorithm.

In one embodiment of the present invention, the block size of the data is 256 KB.

The virtual machine module 350,

In one embodiment of the present invention, the Hadoop-based low-power storage device management module includes a virtual machine module 350. The virtual machine module 350 is not limited as long as it guarantees a certain operation for the same bytecode regardless of the hardware, but may be preferably a Java Virtual Machine of Oracle Corporation.

The Hadoop module (360)

The Hadoop based low power storage device management module according to the present invention includes a Hadoop module 360. Hadoop is a mapreduce framework and the Hadoop module is a file system module that supports the Hadoop distributed file system The Hadoop processor accelerator module 320 and the Hadoop processor accelerator module 340 can implement parallel processing in a cluster environment composed of untrusted computers of petabytes or more of mass data.

In one embodiment of the present invention, the Hadoop module may be operated on the virtual machine module.

Common Information Model Management Module

In one embodiment of the present invention, the Hadoop-based low-power storage device management module includes a common information model (CIM) module. The common information model can manage the information of each module included in the Hadoop-based low-power storage device management module as an object. Due to the common information model, information on modules related to the Hadoop module 360 can be processed.

Hadoop control module 370

The Hadoop-based low-power storage device management module according to the present invention includes a Hadoop control module 370. The Hadoop control module 360 can transfer the status and command related to data processing to the Hadoop module 360 through the common information model management module.

Claims (7)

A storage controller including a control module and a Hadoop processor accelerator;
Backplane; And
A storage expander; and a Hadoop-based low-power storage device. The method according to claim 1,
And a solid state disk (SSD) cache.
Hadoop-based low-power storage devices. The method according to claim 1,
The Hadoop processor accelerated accelerator includes an LZ4 algorithm operation circuit
Hadoop-based low-power storage devices. 5. The method of claim 4,
The size of the block processed by the LZ4 algorithm operation circuit is 256 KB
Hadoop-based low-power storage devices. The method according to claim 1,
The storage control unit includes four input / output lines
Hadoop-based low-power storage devices. A kernel module including a file system module;
Hadoop module; And
Including the Hadoop control module
Hadoop-based low-power storage device management module. The method according to claim 6,
And a common information model management module
Hadoop-based low-power storage device management module.
KR1020150106433A 2015-07-28 2015-07-28 Low power strorage device based on hadoop and managin module thereof KR20170013587A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020150106433A KR20170013587A (en) 2015-07-28 2015-07-28 Low power strorage device based on hadoop and managin module thereof

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020150106433A KR20170013587A (en) 2015-07-28 2015-07-28 Low power strorage device based on hadoop and managin module thereof

Publications (1)

Publication Number Publication Date
KR20170013587A true KR20170013587A (en) 2017-02-07

Family

ID=58107971

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020150106433A KR20170013587A (en) 2015-07-28 2015-07-28 Low power strorage device based on hadoop and managin module thereof

Country Status (1)

Country Link
KR (1) KR20170013587A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019067929A1 (en) * 2017-09-28 2019-04-04 Intel Corporation Multi-criteria power management scheme for pooled accelerator architectures

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019067929A1 (en) * 2017-09-28 2019-04-04 Intel Corporation Multi-criteria power management scheme for pooled accelerator architectures
US10444813B2 (en) 2017-09-28 2019-10-15 Intel Corporation Multi-criteria power management scheme for pooled accelerator architectures

Similar Documents

Publication Publication Date Title
EP3188449B1 (en) Method and system for sharing storage resource
US9639459B2 (en) I/O latency and IOPs performance in thin provisioned volumes
US10248175B2 (en) Off-line affinity-aware parallel zeroing of memory in non-uniform memory access (NUMA) servers
WO2014019428A1 (en) Method and system for allocating fpga resources
JP2016539399A (en) Data write request processing method and storage array
US10089013B2 (en) System and method for managing a non-volatile storage resource as a shared resource in a distributed system
US9755986B1 (en) Techniques for tightly-integrating an enterprise storage array into a distributed virtualized computing environment
US20190026142A1 (en) Information processing apparatus, control method and storage medium
US20130166797A1 (en) Storage apparatus and method for controlling same
US20160182620A1 (en) Data distribution apparatus, data distribution method, and data distribution program for parallel computing processing system
CN107528871B (en) Data analysis in storage systems
US10169062B2 (en) Parallel mapping of client partition memory to multiple physical adapters
US20220121359A1 (en) System and method to utilize a composite block of data during compression of data blocks of fixed size
JP6199782B2 (en) Computer system
CN104954452A (en) Dynamic cipher card resource control method in virtualization environment
US10846265B2 (en) Method and apparatus for accessing file, and storage system
KR20170013587A (en) Low power strorage device based on hadoop and managin module thereof
JP2018010698A (en) Data writing request processing method and storage array
CN107273188B (en) Virtual machine Central Processing Unit (CPU) binding method and device
WO2013065151A1 (en) Computer system, data transmission method, and data transmission program
US11829798B2 (en) System and method to improve data compression ratios for fixed block sizes in a smart data accelerator interface device
CN117076409B (en) File sharing method, device, system, electronic equipment and storage medium
KR20170013592A (en) Appliance managing system based on hadoop
An et al. Design and implement of pre-loading ssd cache data using split file on hadoop mapreduce
CN104503821A (en) Method for calculating virtual disc IO (input/output) speed