CN103617293A

CN103617293A - Key-Value storage method oriented towards storage system of mass small files

Info

Publication number: CN103617293A
Application number: CN201310688270.1A
Authority: CN
Inventors: 王雷; 王振; 王平
Original assignee: Beihang University
Current assignee: Beihang University
Priority date: 2013-12-16
Filing date: 2013-12-16
Publication date: 2014-03-05

Abstract

The invention provides a Key-Value storage method oriented towards a storage system of mass small files. The application of the storage system of the small files can be met through the storage method. Compared with the frequently-used Redis and LevelDB, the storage method has the advantages that pay expenses are lower, delaying is shorter, and the lightweight-class persistent Key-Value storage method is oriented towards the storage system of the small files. A storage structure comprises two parts, and as shown in the figure 1, the two parts include a HashTable structure in an internal storage and a persistent storage Store on a disk.

Description

A kind of Key-Value storage means towards mass small documents storage system

Technical field

The present invention relates to a minute mass small documents field of storage, be specifically related to a kind of Key-Value storage means towards mass small documents storage system.

Background technology

In recent years, social network sites is as rapid in network service development such as Facebook, Twitter, Renren Network and e-commerce website e-Bay, Alibaba, and this class service all needs to store the small documents such as a large amount of pictures, short text.Small documents is often referred to the file that file size is less than 64M, the picture that often need to store such as internet, applications, mail, e-book, music file, microblogging, content of text etc.

Small documents storage has caused some concerns in academia and industry member gradually.Social network sites Facebook has stored 2,600 hundred million pictures, and capacity surpasses 20PB, and these file overwhelming majority are all less than 64MB.In supercomputer field, for example, the application program on ORNL ' s CrayXT5 cluster (18688 nodes, 12 processors of each node) can, periodically by application state writing in files, cause system to accumulate a large amount of small documents.In concrete scientific research computing environment, for example, during some biology calculates, may produce 3,000 ten thousand files, and its mean size only has 190KB.U.S.'s Pacific Northwest National Laboratory is published in a research report data of 2007 and shows, in the system that this laboratory is used, stored 1,200 ten thousand files, the file that is wherein less than 64MB accounts for 94% of total number of files, and what be less than 64KB accounts for 58%.The huge whale net of music site has been included the music file of 3,600,000 MP3 format.Various data all show, the data of accessing on internet mostly are the small documents of high access frequency.

The retrieving information of small documents is realized by preserving Key-Value key-value pair, conventionally uses NoSQL database (as Redis and LevelDB).Yet the main flow such as LevelDB and Redis Key-Value storage engines is bad memory scan information timeliness fruit.Reason is as follows:

1. the value data layout of small documents storage system record is fixed, size is unified, with respect to Redis, be applicable to various data structures, small documents storage system can be for application-specific reduced data structure, reduced the operation that data structure is packed and resolved, and avoid realizing communication protocol, thus save CPU expense, reduce and postpone.

2.Redis system is mainly for the application in internal memory by data buffer storage.Redis can configure persistence function, but small documents storage system can, for its application-specific, can be supported persistence better.

3.LevelDB is standalone version persistence Key-Value storage engines, but LevelDB reduces and writes fashionable delay by the MemTable in internal memory and the Level blocks of files on disk, but for read request, LevelDB may need repeatedly disk operating (will read at most above disk file 6 times), so LevelDB is very suitable for writing the situation that mutiread is few.And in real internet, applications, as commodity photo in the photograph album on Facebook website or Taobao, its application characteristic is write-once, repeatedly read, so LevelDB is also not suitable for preserving retrieving information in small documents storage system.

Summary of the invention

The present invention proposes the Key-Value storage engines KVDB that a kind of lightweight towards small documents storage system can persistence.

KVDB structural design is as Fig. 1.KVDB comprises two parts, the HashTable structure in internal memory and the persistent storage Store on disk.

Figure is as Fig. 2 for HashTable class.DictEntry is used for representing an entry of the Hash table in upper figure.Use chained list method to solve hash-collision.HashTable has preserved key for retrieving and value in the position of Store.

Store is used for depositing value content, and Store can write a value content, and returns to this value content in the side-play amount of Store inside, or a given reference position, and Store can read out a value content.The class figure of Store is as Fig. 3.In KVDB, all these type of data structure that will preserve, all must be able to carry out serializing and unserializing (serialize and deserialize), so that Store class leaves this data structure in disk, carry out persistence.The corresponding disk file of Store class, such comprises handle and some statistical informations of pointing to this disk file, and all data structures that will preserve all write in this disk, and Store provides read-write interface function simultaneously.For power interruption recovering function is provided, Store also provides an interface function loading from disk.For improving reading speed, in Store for the file of depositing value content when loading, by mmap function, file content is mapped in internal memory.Like this, during disk file corresponding to access Store, conventionally can in internal memory, have access to.

The class figure of KVDB is as Fig. 4.The disclosed interface of KVDB comprises constructed fuction, initialization function init, and format output function format, writes a key-value key-value pair put function, reads a key-value key-value pair get function.

Accompanying drawing explanation

Fig. 1 is the structural drawing of KVDB

Fig. 2 is the HashTable class figure of KVDB

Fig. 3 is the Store class figure in KVDB

Fig. 4 is KVDB class figure

Embodiment

KVDB carries out key-value storage by the HashTable structure in internal memory and the persistent storage Store on disk to data structure, for example, when for retrieving information RetrieveInfo, key is this small documents filename (comspec), and value is retrieving information RetrieveInfo.

While writing small documents, retrieving information RetrieveInfo corresponding to this small documents that small documents is generated writes KVDB<RetrieveInfo>.While reading small documents, just from KVDB<RetrieveInfo>, read out the retrieving information RetrieveInfo that this small documents filename (complete trails filename) is corresponding.

Claims

1. the present invention has designed a kind of Key-Value storage means towards mass small documents storage system, is a kind of application-specific that can either meet small documents storage system, also can reduce the storage means that the lightweight of expense, delay can persistence simultaneously.