CN104618484A - Key value pair based transverse expansion high-performance storing method - Google Patents
Key value pair based transverse expansion high-performance storing method Download PDFInfo
- Publication number
- CN104618484A CN104618484A CN201510060118.8A CN201510060118A CN104618484A CN 104618484 A CN104618484 A CN 104618484A CN 201510060118 A CN201510060118 A CN 201510060118A CN 104618484 A CN104618484 A CN 104618484A
- Authority
- CN
- China
- Prior art keywords
- data
- performance
- client
- key
- storage method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/563—Data redirection of data network streams
Abstract
The invention discloses a key value pair based transverse expansion high-performance storing method, and aims at achieving the transverse expansion high-performance object storage service based on a key/ value object storage and a ceph open source file system serving as a bottom storage. The high-performance storing method involves the devices including a client, a server and a storage side, wherein the server receives information from the client and feeds back to the client and synchronously processes the client information and stores to the storage side. The method mainly comprises the steps of receiving and feeding back data; processing data; arraying data. The high-performance storing method is novel in design, and simple to perform; the frequent network access and frequent data update can be handled in a message array form; therefore, the problems of data migration and load balance can be handled well; the system storage performance can be greatly improved.
Description
Technical field
The present invention relates to Computer Storage field, be specifically related to distributed storage, Storage Virtualization, asynchronous network communication, main flow objects services agreement, is exactly a kind of high-performance storage method extending transversely based on key-value pair specifically.
Background technology
Along with developing rapidly of network application, network information data amount is increasing, and the mass data storage of PB rank becomes more and more important.The file system of traditional sense can not meet the requirement such as Large Copacity, high reliability, high-performance of existing application, and for meeting these new demands, distributed file system application and object storage obtain extensive attention.
Ceph file system is a distributed file system having superior function, high reliability, extensibility, and is a Linux PB level distributed file system.It has following several advantage: can expand to PB level capacity easily, have high-performance, high reliability to multiple operating load.Object storage can be provided, block stores and file system stores three kinds of functions.
Existing storage mainly block storage and file stores these two kinds storages, is also the storage class of two kinds of main flows.Block stores main based on SAN, and it is the network of gigabit rate, relies on optical-fibre channel (Fibre Channel) to provide higher handling capacity, support that farther Distance geometry is communicated with more reliably for the connection between server and memory device; File stores main based on NAS, and NAS network connection storage-be connected on existing network by storage facilities, provides data and file service.Object storage system (Object-Based Storage System) is the advantage combining NAS and SAN, the high speed simultaneously with SAN is directly accessed and the advantages such as the data sharing of NAS, provides the storage architecture of the data sharing of high reliability, professional platform independence and safety.
Libevent is the network library that an event triggers.Ibevent comprises incident management, cache management, DNS, HTTP, a few major part of buffer memory event.Incident management comprises various IO(socket), the event such as timer, signal; Cache management refers to evbuffer function; DNS is the asynchronous DNS query function that libevent provides; HTTP is that a lightweight http of libevent realizes, and comprises server and client side.
Summary of the invention
For the current condition of current technical development, the invention provides a kind of high-performance storage method extending transversely based on key-value pair.
A kind of high-performance storage method extending transversely based on key-value pair of the present invention, the technical scheme solving the problems of the technologies described above employing is as follows: this high-performance storage method is the high-performance storage method extending transversely based on key/value; The equipment used comprises: client, server and storage end, and its key step comprises: receive and feedback data, deal with data and queue data; When operating data persistence, selecting laterally extendible storage system, forming a high-performance object stores service extending transversely.
Preferably, this high-performance storage method to be increased income file system based on ceph, allows it as bottom storage system.
Preferably, described high-performance object stores service supports the stores service of http protocol, comprises the increase storing data, revises, checks and delete.
Preferably, described server is the HTTP server based on libevent storehouse, primary recipient from client information and feed back to client, and be responsible for carrying out basic handling to data and feeding back the storage information of data.
Preferably, described server sets up an event trigger mechanism, and receives the information from client based on HTTP GET/POST agreement and feed back to client.
Preferably, described server process from the information of client, the information becoming a kind of object to store the information processing from client.
Preferably, described object stores and adopts key/value form.
Preferably, this high-performance storage method utilizes the data transmission of message queue asynchronous process and storage; When continually to data inserting in database, when submitting data to search engine continually, then take the asynchronous insertion of message queue.
A kind of beneficial effect compared with prior art had based on the high-performance storage method extending transversely of key-value pair of the present invention is: this high-performance storage method is novel in design, employing be all the storehouse of lightweight, implement simple; Access to netwoks frequently can be processed with message queue, and Data Update frequently; The problem such as Data Migration, load balance of generation can be processed well, greatly improve system storage performance.
Accompanying drawing explanation
What accompanying drawing 1 was described high-performance storage method realizes block diagram.
Embodiment
For making the object, technical solutions and advantages of the present invention clearly understand, below in conjunction with specific embodiment, and with reference to accompanying drawing, a kind of high-performance storage method extending transversely based on key-value pair of the present invention is further described.
A kind of high-performance storage method extending transversely based on key-value pair of the present invention is the high-performance storage method extending transversely based on key/value.The equipment that this high-performance storage method uses comprises: client, server and storage end, and described server receives the information from client and feeds back to client, and stores end by being stored into after client-side information process; The key step of the method comprises: receive and feedback data, deal with data and queue data; As shown in Figure 1.This high-performance storage method, when operating data persistence, selects laterally extendible storage system, designs and Implements a high-performance object stores service extending transversely.The feature of this high-performance storage method is, object store and extending transversely, each object adopts key/value to define storage, support http protocol and energy extending transversely.
Embodiment:
A kind of high-performance storage method extending transversely based on key-value pair described in the present embodiment, to increase income file system based on ceph, it is allowed to store as bottom, design and Implement a high-performance object stores service extending transversely, this high-performance object stores service supports the stores service of http protocol, comprises the increase storing data, revises, checks and delete.
Receive in this high-performance storage method and comprise with the step of feedback data: described server is the HTTP server based on libevent storehouse, primary recipient from client information and feed back to client, and be responsible for carrying out basic handling to data and feeding back the storage information of data.Described server mainly relies on libevent storehouse to realize, and sets up an event trigger mechanism; Described server accepts the information from client based on HTTP GET/POST agreement and feeds back to client.
In high-performance storage method described in the present embodiment, the server that its service routine runs remotely can receive data (from the address field of browser or utilize crul order) from client, and this is exactly mainly the remote interaction of server and client side.Described server provides external interface service, if server can not receive data, then needs to return certain value and what is illustrated as.Be that it is the agreement of a lightweight based on HTTP GET/POST agreement herein, support new functional requirement.
In this high-performance storage method, the step of deal with data comprises: described server process is from the information of client, the information becoming a kind of object to store the information processing from client, object storage provide there is high-performance, storage architecture that high reliability, cross-platform and secure data are shared, solve the high access in I/O and data throughput, make the memory load of cluster balanced.In high-performance storage method described in the present embodiment, described object stores and adopts key/value form, can have better access speed like this.The input of this high-performance storage method should follow simple principle.Therefore when perform PUT operation time, only need input operation accord with, key and value just can carry out stored in; When performing GET(or DELETE) time, only need input key and just can obtain (or deletion) data.
In this high-performance storage method, the step of queue data comprises: utilize the message queue transmission of asynchronous process data and storage well, when continually to data inserting in database, submit data to search engine continually time, then can take the asynchronous insertion of message queue.In addition, by slower processing logic, the processing logic having concurrent restricted number, background process can also be placed on by message queue.
In addition, high-performance storage method described in the present embodiment, bottom stores increases income file system as the management stored using ceph.Ceph file system of increasing income is dynamic in storage size, and constantly can expand along with the growth of business scale, this is very favourable to the growth of current memory capacity explosion type; Secondly, it is change on a storage device, for the system be made up of thousands of nodes, node failure and replacing will inevitably occur along with operation maintenance, and ceph increase income file system can process well with these problems produce the problem such as Data Migration, load balance, suitable intellectuality, greatly reduces maintenance cost; Finally, ceph increases income file system in data variation, usually used as the storage system in internet, applications, can deal with the continuous write of data, the frequent updating of data with existing for so large-scale system, amendment and deletion.
From the specific implementation process of high-performance storage method described above, this high-performance storage method has following features:
1) based on HTTP GET/POST agreement, PHP, Java, Perl, Shell, Python, Ruby etc. support that the programming language of http protocol all can call;
2) a kind of message queue is provided, can the transmission of asynchronous process data and storage well, when continually to data inserting in database, when submitting data to search engine continually, the asynchronous insertion of this message queue just can be taked;
3) storage of key/value form; Object storage provide there is high-performance, storage architecture that high reliability, cross-platform and secure data are shared.
Above-mentioned embodiment is only concrete case of the present invention; scope of patent protection of the present invention includes but not limited to above-mentioned embodiment; any claims according to the invention and any person of an ordinary skill in the technical field to its suitable change done or replacement, all should fall into scope of patent protection of the present invention.
Claims (8)
1. the high-performance storage method extending transversely based on key-value pair, it is characterized in that, the method is the high-performance storage method extending transversely based on key/value, the equipment used comprises: client, server and storage end, described server receives the information from client and feeds back to client, and stores end by being stored into after client-side information process; The key step of this high-performance storage method comprises: receive and feedback data, deal with data and queue data; When operating data persistence, selecting laterally extendible storage system, forming a high-performance object stores service extending transversely.
2. a kind of high-performance storage method extending transversely based on key-value pair according to claim 1, it is characterized in that, this high-performance storage method increases income file system as bottom storage system based on ceph.
3. a kind of high-performance storage method extending transversely based on key-value pair according to claim 1 or 2, it is characterized in that, described server is the HTTP server based on libevent storehouse, and is responsible for carrying out basic handling to data and feeding back the storage information of data.
4. a kind of high-performance storage method extending transversely based on key-value pair according to claim 3, is characterized in that, described high-performance object stores service supports the stores service of http protocol, comprises the increase storing data, revises, checks and delete.
5. a kind of high-performance storage method extending transversely based on key-value pair according to claim 4, is characterized in that, described server receives the information from client based on HTTP GET/POST agreement and feeds back to client.
6. a kind of high-performance storage method extending transversely based on key-value pair according to claim 3, is characterized in that, described server process from the information of client, the information becoming a kind of object to store the information processing from client.
7. a kind of high-performance storage method extending transversely based on key-value pair according to claim 7, is characterized in that, described object stores and adopts key/value form.
8. a kind of high-performance storage method extending transversely based on key-value pair according to claim 7, it is characterized in that, this high-performance storage method utilizes the data transmission of message queue asynchronous process and storage; When continually to data inserting in database, when submitting data to search engine continually, then take the asynchronous insertion of message queue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510060118.8A CN104618484A (en) | 2015-02-05 | 2015-02-05 | Key value pair based transverse expansion high-performance storing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510060118.8A CN104618484A (en) | 2015-02-05 | 2015-02-05 | Key value pair based transverse expansion high-performance storing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104618484A true CN104618484A (en) | 2015-05-13 |
Family
ID=53152759
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510060118.8A Pending CN104618484A (en) | 2015-02-05 | 2015-02-05 | Key value pair based transverse expansion high-performance storing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104618484A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106155586A (en) * | 2016-05-31 | 2016-11-23 | 华为技术有限公司 | A kind of storage method, server and storage control |
| CN109783013A (en) * | 2017-11-15 | 2019-05-21 | 三星电子株式会社 | Configure and access the method and system of expansible object storage |
| CN112035563A (en) * | 2020-08-28 | 2020-12-04 | 南京华盾电力信息安全测评有限公司 | Real-time database system based on shared storage |
-
2015
- 2015-02-05 CN CN201510060118.8A patent/CN104618484A/en active Pending
Non-Patent Citations (4)
| Title |
|---|
| 严志敏: ""TOF基础服务高性能缓存系统的设计与实现"", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 张聪萍: ""钱塘分布式文件存储系统性能优化"", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 李翔: ""Ceph分布式文件系统的研究及性能测试"", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 胡昊: ""Key-Value数据存储加速技术研究"", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106155586A (en) * | 2016-05-31 | 2016-11-23 | 华为技术有限公司 | A kind of storage method, server and storage control |
| CN109783013A (en) * | 2017-11-15 | 2019-05-21 | 三星电子株式会社 | Configure and access the method and system of expansible object storage |
| CN109783013B (en) * | 2017-11-15 | 2021-06-01 | 三星电子株式会社 | Method and system for configuring and accessing extensible object stores |
| US11269520B2 (en) | 2017-11-15 | 2022-03-08 | Samsung Electronics Co., Ltd. | Methods to configure and access scalable object stores using KV-SSDS and hybrid backend storage tiers of KV-SSDS, NVME-SSDS and other flash devices |
| CN112035563A (en) * | 2020-08-28 | 2020-12-04 | 南京华盾电力信息安全测评有限公司 | Real-time database system based on shared storage |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10817195B2 (en) | Key-value based message oriented middleware | |
| US10581957B2 (en) | Multi-level data staging for low latency data access | |
| CN109074377B (en) | Managed function execution for real-time processing of data streams | |
| Fu et al. | A fair comparison of message queuing systems | |
| CN108280522B (en) | Plug-in distributed machine learning calculation framework and data processing method thereof | |
| AU2014212780B2 (en) | Data stream splitting for low-latency data access | |
| CN106815338A (en) | A kind of real-time storage of big data, treatment and inquiry system | |
| Mai et al. | Optimizing network performance in distributed machine learning | |
| TWI735545B (en) | Model training method and device | |
| CN104580226A (en) | Session data sharing system and method | |
| CN105138615A (en) | Method and system for building big data distributed log | |
| US11429566B2 (en) | Approach for a controllable trade-off between cost and availability of indexed data in a cloud log aggregation solution such as splunk or sumo | |
| US20210173714A1 (en) | High-throughput parallel data transmission | |
| CN109815026A (en) | Electric power time series database based on distributed component | |
| CN105740248B (en) | A kind of method of data synchronization, apparatus and system | |
| CN107343021A (en) | A kind of Log Administration System based on big data applied in state's net cloud | |
| CN111090674A (en) | Search engine system based on hot words and cache | |
| CN103559229A (en) | Small file management service (SFMS) system based on MapFile and use method thereof | |
| Pal et al. | Big data real time ingestion and machine learning | |
| CN109739435A (en) | File storage and update method and device | |
| CN104618484A (en) | Key value pair based transverse expansion high-performance storing method | |
| Senger et al. | BSP cost and scalability analysis for MapReduce operations | |
| US20170060922A1 (en) | Method and device for data search | |
| CN102567419A (en) | Mass data storage device and method based on tree structure | |
| CN109597903A (en) | Image file processing apparatus and method, document storage system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150513 |