CN103902665A - Storage virtualization system implementation method - Google Patents
Storage virtualization system implementation method Download PDFInfo
- Publication number
- CN103902665A CN103902665A CN201410088599.9A CN201410088599A CN103902665A CN 103902665 A CN103902665 A CN 103902665A CN 201410088599 A CN201410088599 A CN 201410088599A CN 103902665 A CN103902665 A CN 103902665A
- Authority
- CN
- China
- Prior art keywords
- server
- master
- block
- storage
- mfs
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
- G06F3/0667—Virtualisation aspects at data level, e.g. file, record or object virtualisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/1435—Saving, restoring, recovering or retrying at system level using file system or storage system metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Library & Information Science (AREA)
- Quality & Reliability (AREA)
- Hardware Redundancy (AREA)
Abstract
The invention discloses a storage virtualization system implementation method. The method relates to a main server, a backup server and block servers, wherein the main server is used for managing nodes and mainly used for storing metadata relevant to data files; the backup server is a backup system of the main server, and rapidly takes over services of the main server once the main service crashes; the block servers are in charge of specific storage work and used for storing data. Through the method, performance bottlenecks generated in the traditional storage process can be effectively eliminated, purchase cost of users is reduced, the problem of single-point faults generated in a traditional storage solution is solved, reliability of a storage system is improved, and the storage system high in performance and stability is built with low cost.
Description
?
Technical field
The present invention relates to Storage Virtualization field, be specifically related to the method that memory virtualization system is realized.
Technical background
On internet, addressable information content increases sharply at present, and Internet firm has reached EB level to the demand of storage, and traditional storage solution price is high, and performance bottleneck is large, and is prone to Single Point of Faliure.Certainly, each large-scale website is also all being stored the data of magnanimity, and the data of these magnanimity are effectively storage how, is the problem that the architect of each large-scale website must solve.The implementation method of this Storage Virtualization can effectively solve the problem that conventional store solution produces, needn't use professional storage system, use storage server at a low price can realize the construction of memory virtualization system, and can by copy elimination Single Point of Faliure is set, guarantee data security.
Summary of the invention
The technical problem to be solved in the present invention is: in view of with above-mentioned existing problem, the object of this invention is to provide a kind of method that memory virtualization system is realized, the performance bottleneck that can effectively provide conventional store to run into by the method, reduce user's purchase cost, and eliminate conventional store solution and run into ground Single Point of Faliure problem, promote the reliability of storage system, with the storage system of a set of high-performance of low cost construction and high stability.
The technical solution adopted in the present invention is:
The method that Storage Virtualization is realized, comprising: master server, standby server, block server, wherein,
Master server: management node, the main storage metadata relevant to data file, metadata comprises: NameSpace (Name Space), the namely bibliographic structure of whole file system, 64 label mappings are arrived the position of data block and the form of composing document thereof by an energy, and primary server joint can periodically receive the renewal (" Heart-beat ") coming from each block server node allows metadata keep last state;
Standby server: the standby system of master server, once there is the machine of delaying, the business that standby server is taken over rapidly master server when master server;
Block server: be responsible for concrete storage work, be used for storing data, memory virtualization system carries out piecemeal by file according to fixed size, each piece is called a data block, each data block is divided take Block as unit, size is 64KB, and each data block has 64 unique labels; It is fault-tolerant that memory virtualization system adopts the mode of copy to realize, and each data block has multiple stored copies (being defaulted as three).The number of block server can have multiple.
Described method performing step is as follows:
1) prepare physical server installing operating system configuration of IP;
2) configuration master server, set up applications;
3) configuration backup server;
4) configuration block server;
5) test storage virtualization system availability.
The layoutprocedure of described server is as follows: be connected to server OS command Window by ssh
Carry out and start as issued orders to install:
chmod?+x?./?ics-1.1--2011-××-××-××-××-××.bin
./?ics-1.1--2011-××-××-××-××-××.bin
Comprise master server, block server and three assemblies of client are installed, select suitable assembly to select to install.
Described master server installation procedure is as follows:
1) space bar selects to install Master and client program;
2) fill in following information:
A) LOCAL_ADPT=eth0, the network interface card title (all nodes are relevant) of use
B) LOCAL_HOST=..., the IP address (all nodes are relevant) that above-mentioned network interface card is corresponding
C) MASTER_HOST=..., configuration Floating IP address and above-mentioned IP are positioned at the same network segment, and master server and all backup servers need to configure identical Floating IP address (all nodes are irrelevant)
3) start stores service:
/etc/init.d/ics-failover?start
/etc/init.d/ics-daemon?start
4) initialization storage resources:
mkdir?–p?/mnt/mfs
Mfsmount/mnt/mfs – H $ { MASTER_HOST} note: (the machine IP address)
mkdir?/mnt/mfs/root
umount?/mnt
Described block server installation procedure is as follows:
1) Chunk program is selected to install in space;
2) fill in following information:
A) LOCAL_ADPT=eth0, the network interface card title (all nodes are relevant) of use
B) LOCAL_HOST=..., the IP address (all nodes are relevant) that above-mentioned network interface card is corresponding
C) MASTER_HOST=..., configuration Floating IP address and above-mentioned IP are positioned at the same network segment, and master server and all backup servers need to configure identical Floating IP address (all nodes are irrelevant)
D) EXPORT_PATH=/mfsdata, the data directory of configuration block server (only relevant with chunk node) note: above configuration is actual is to write in mfs.cfg file, whether be positioned at/etc/mfs/mfs.cfg of file, can check configuration successful.
3) restart block server.
Block server multipath program is set as follows:
On any block server, except setting/mfsdata is store path, can also increase other path.For example, when block server has increased a hard disk, under be mounted to/mfsdata1 of this piece dish path, directly revise the configuration file/etc/mfs/mfs.cfg on this block server, one (attention quotation marks) of many increases on EXPORT_PATH:
EXPORT_PATH=”/mfsdata?/mfsdata1”
Described test storage virtualization system availability comprises: verification system service:
Mfs-failover service:
The fault-tolerant process of master server, is used Floating IP address configuration master server, operates on master server and backup server;
Mfs-daemon service:
Master server finger daemon, switches master server and backup server, comprises jre(1.6.23), tomcat(5.5.31) etc., operate on master server and backup server;
MFS-chunks service:
Piece service processes, operates on block server.
Storage all dispose good after, can input URL by browser is the operation working condition that the ip of master server adds port numbers 9425 and checks memory virtualization system, for example: http: // 192.168.0.100:9425.
Beneficial effect of the present invention is:
Adopt the inventive method to effectively raise reliability, extensibility and the high performance demand of data storage and can be user and reduce storage space purchase cost, guarantee high-performance, high reliability, the low cost of customer data storage system.This method can allow user build rapidly memory virtualization system, and puts in actual production environment, and can externally expand according to the physical memory space, the demand of data-storage system being reached the standard grade rapidly, efficiently being moved to reach client.
Accompanying drawing explanation
Fig. 1 is memory virtualization system network topological diagram of the present invention.
Embodiment
With reference to the accompanying drawings, by embodiment, the present invention is further described:
Master server: management node, the main storage metadata relevant to data file, metadata comprises: NameSpace (Name Space), the namely bibliographic structure of whole file system, 64 label mappings are arrived the position of data block and the form of composing document thereof by an energy, and primary server joint can periodically receive the renewal (" Heart-beat ") coming from each block server node allows metadata keep last state;
Standby server: the standby system of master server, once there is the machine of delaying, the business that standby server is taken over rapidly master server when master server;
Block server: be responsible for concrete storage work, be used for storing data, memory virtualization system carries out piecemeal by file according to fixed size, each piece is called a data block, each data block is divided take Block as unit, size is 64KB, and each data block has 64 unique labels; It is fault-tolerant that memory virtualization system adopts the mode of copy to realize, and each data block has multiple stored copies (being defaulted as three).The number of block server can have multiple.
Described method performing step is as follows:
1) prepare four physical server installing operating systems configuration of IP;
2) configuration master server, set up applications;
3) configuration backup server;
4) configuration block server;
5) test storage virtualization system availability.
The layoutprocedure of described server is as follows: be connected to server OS command Window by ssh, carry out and start as issued orders to install:
chmod?+x?./?ics-1.1--2011-××-××-××-××-××.bin
./?ics-1.1--2011-××-××-××-××-××.bin
Comprise master server, block server and three assemblies of client are installed, select suitable assembly to select to install.
Described master server installation procedure is as follows:
1) space bar selects to install Master and client program;
2) fill in following information:
A) LOCAL_ADPT=eth0, the network interface card title (all nodes are relevant) of use
B) LOCAL_HOST=..., the IP address (all nodes are relevant) that above-mentioned network interface card is corresponding
C) MASTER_HOST=..., configuration Floating IP address and above-mentioned IP are positioned at the same network segment, and master server and all backup servers need to configure identical Floating IP address (all nodes are irrelevant)
3) start stores service:
/etc/init.d/ics-failover?start
/etc/init.d/ics-daemon?start
4) initialization storage resources:
mkdir?–p?/mnt/mfs
Mfsmount/mnt/mfs – H $ { MASTER_HOST} note: (the machine IP address)
mkdir?/mnt/mfs/root
umount?/mnt
Described block server installation procedure is as follows:
1) Chunk program is selected to install in space;
2) fill in following information:
A) LOCAL_ADPT=eth0, the network interface card title (all nodes are relevant) of use
B) LOCAL_HOST=..., the IP address (all nodes are relevant) that above-mentioned network interface card is corresponding
C) MASTER_HOST=..., configuration Floating IP address and above-mentioned IP are positioned at the same network segment, and master server and all backup servers need to configure identical Floating IP address (all nodes are irrelevant)
D) EXPORT_PATH=/mfsdata, the data directory of configuration block server (only relevant with chunk node) note: above configuration is actual is to write in mfs.cfg file, whether be positioned at/etc/mfs/mfs.cfg of file, can check configuration successful.
3) restart block server.
Block server multipath program is set as follows:
On any block server, except setting/mfsdata is store path, can also increase other path.For example, when block server has increased a hard disk, under be mounted to/mfsdata1 of this piece dish path, directly revise the configuration file/etc/mfs/mfs.cfg on this block server, one (attention quotation marks) of many increases on EXPORT_PATH:
EXPORT_PATH=”/mfsdata?/mfsdata1”
Described test storage virtualization system availability comprises: verification system service:
Mfs-failover service:
The fault-tolerant process of master server, is used Floating IP address configuration master server, operates on master server and backup server;
Mfs-daemon service:
Master server finger daemon, switches master server and backup server, comprises jre(1.6.23), tomcat(5.5.31) etc., operate on master server and backup server;
MFS-chunks service:
Piece service processes, operates on block server.
Storage all dispose good after, can input URL by browser is the operation working condition that the ip of master server adds port numbers 9425 and checks memory virtualization system, for example: http: // 192.168.0.100:9425.
Claims (9)
1. the method that Storage Virtualization is realized, comprising: master server, standby server, block server, is characterized in that:
Master server: management node, the main storage metadata relevant to data file, metadata comprises: NameSpace, it is the bibliographic structure of whole file system, 64 label mappings are arrived the position of data block and the form of composing document thereof by an energy, and primary server joint can periodically receive more newly arriving of coming from each block server node allows metadata keep last state;
Standby server: the standby system of master server, once there is the machine of delaying, the business that standby server is taken over rapidly master server when master server;
Block server: be responsible for concrete storage work, be used for storing data, memory virtualization system carries out piecemeal by file according to fixed size, each piece is called a data block, each data block is divided take Block as unit, size is 64KB, and each data block has 64 unique labels.
2. the method that a kind of Storage Virtualization according to claim 1 is realized, is characterized in that: described method performing step is as follows:
1) prepare physical server installing operating system configuration of IP;
2) configuration master server, set up applications;
3) configuration backup server;
4) configuration block server;
5) test storage virtualization system availability.
3. the method that a kind of Storage Virtualization according to claim 2 is realized, is characterized in that, the layoutprocedure of described server is as follows: be connected to server OS command Window by ssh, carry out and start as issued orders to install:
chmod?+x?./?ics-1.1--2011-××-××-××-××-××.bin
./?ics-1.1--2011-××-××-××-××-××.bin
Comprise master server, block server and three assemblies of client are installed, select suitable assembly to select to install.
4. the method that a kind of Storage Virtualization according to claim 3 is realized, is characterized in that: described master server installation procedure is as follows:
1) space bar selects to install Master and client program;
2) fill in following information:
A) LOCAL_ADPT=eth0, the network interface card title of use
B) LOCAL_HOST=..., the IP address that above-mentioned network interface card is corresponding
C) MASTER_HOST=..., configuration Floating IP address and above-mentioned IP are positioned at the same network segment, and master server and all backup servers need to configure identical Floating IP address
3) start stores service:
/etc/init.d/ics-failover?start
/etc/init.d/ics-daemon?start
4) initialization storage resources:
mkdir?–p?/mnt/mfs
Mfsmount/mnt/mfs – H $ MASTER_HOST} note:
mkdir?/mnt/mfs/root
umount?/mnt
。
5. the method that a kind of Storage Virtualization according to claim 3 is realized, is characterized in that: described block server installation procedure is as follows:
1) Chunk program is selected to install in space;
2) fill in following information:
A) LOCAL_ADPT=eth0, the network interface card title of use
B) LOCAL_HOST=..., the IP address that above-mentioned network interface card is corresponding
C) MASTER_HOST=..., configuration Floating IP address and above-mentioned IP are positioned at the same network segment, and master server and all backup servers need to configure identical Floating IP address
D) EXPORT_PATH=/mfsdata, the data directory of configuration block server
3) restart block server.
6. the method realizing according to a kind of Storage Virtualization described in claim 4 or 5, is characterized in that: block server multipath program is set as follows:
When block server has increased a hard disk, under be mounted to/mfsdata1 of this piece dish path, directly revise the configuration file/etc/mfs/mfs.cfg on this block server, one of many increase on EXPORT_PATH:
EXPORT_PATH=”/mfsdata?/mfsdata1”
。
7. the method that a kind of Storage Virtualization according to claim 2 is realized, is characterized in that: described test storage virtualization system availability comprises verification system service:
Mfs-failover service:
The fault-tolerant process of master server, is used Floating IP address configuration master server, operates on master server and backup server;
Mfs-daemon service:
Master server finger daemon, switches master server and backup server, operates on master server and backup server;
MFS-chunks service:
Piece service processes, operates on block server.
8. the method that a kind of Storage Virtualization according to claim 1 is realized, is characterized in that: described block server is multiple.
9. according to the memory virtualization system access method described in above arbitrary claim, it is characterized in that: storage all dispose good after, inputting URL by browser is the operation working condition that the ip of master server adds port numbers 9425 and checks memory virtualization system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410088599.9A CN103902665A (en) | 2014-03-11 | 2014-03-11 | Storage virtualization system implementation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410088599.9A CN103902665A (en) | 2014-03-11 | 2014-03-11 | Storage virtualization system implementation method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103902665A true CN103902665A (en) | 2014-07-02 |
Family
ID=50993987
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410088599.9A Pending CN103902665A (en) | 2014-03-11 | 2014-03-11 | Storage virtualization system implementation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103902665A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104407918A (en) * | 2014-11-29 | 2015-03-11 | 中国航空工业集团公司第六三一研究所 | Storage and recovery method for avionics system task data |
| CN106897027A (en) * | 2017-01-18 | 2017-06-27 | 成都电科致远网络科技有限公司 | A kind of distributed storage service system and method based on desktop virtualization |
| CN111245641A (en) * | 2019-12-27 | 2020-06-05 | 广东睿江云计算股份有限公司 | Cloud host CPU and memory modification method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1464396A (en) * | 2002-06-27 | 2003-12-31 | 深圳市中兴通讯股份有限公司 | Method for realizing backup between servers |
| US20070174569A1 (en) * | 2006-01-26 | 2007-07-26 | Infortrend Technology, Inc. | Method of managing data snapshot images in a storage system |
| JP2009151772A (en) * | 2007-12-20 | 2009-07-09 | Internatl Business Mach Corp <Ibm> | Method, system and computer program for managing storage virtualization in storage infrastructure |
| CN102063473A (en) * | 2010-12-21 | 2011-05-18 | 南京壹进制信息技术有限公司 | Method for expanding storage space on line |
| CN102915362A (en) * | 2012-10-18 | 2013-02-06 | 浪潮(北京)电子信息产业有限公司 | Method and system for managing resource information in memory virtualization system |
| CN103268252A (en) * | 2013-05-12 | 2013-08-28 | 南京载玄信息科技有限公司 | Virtualization platform system based on distributed storage and achieving method thereof |
-
2014
- 2014-03-11 CN CN201410088599.9A patent/CN103902665A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1464396A (en) * | 2002-06-27 | 2003-12-31 | 深圳市中兴通讯股份有限公司 | Method for realizing backup between servers |
| US20070174569A1 (en) * | 2006-01-26 | 2007-07-26 | Infortrend Technology, Inc. | Method of managing data snapshot images in a storage system |
| JP2009151772A (en) * | 2007-12-20 | 2009-07-09 | Internatl Business Mach Corp <Ibm> | Method, system and computer program for managing storage virtualization in storage infrastructure |
| CN102063473A (en) * | 2010-12-21 | 2011-05-18 | 南京壹进制信息技术有限公司 | Method for expanding storage space on line |
| CN102915362A (en) * | 2012-10-18 | 2013-02-06 | 浪潮(北京)电子信息产业有限公司 | Method and system for managing resource information in memory virtualization system |
| CN103268252A (en) * | 2013-05-12 | 2013-08-28 | 南京载玄信息科技有限公司 | Virtualization platform system based on distributed storage and achieving method thereof |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104407918A (en) * | 2014-11-29 | 2015-03-11 | 中国航空工业集团公司第六三一研究所 | Storage and recovery method for avionics system task data |
| CN104407918B (en) * | 2014-11-29 | 2017-06-27 | 中国航空工业集团公司第六三一研究所 | A kind of preservation of avionics system task data and restoration methods |
| CN106897027A (en) * | 2017-01-18 | 2017-06-27 | 成都电科致远网络科技有限公司 | A kind of distributed storage service system and method based on desktop virtualization |
| CN106897027B (en) * | 2017-01-18 | 2020-03-24 | 成都电科致远网络科技有限公司 | Distributed storage service system and method based on desktop virtualization |
| CN111245641A (en) * | 2019-12-27 | 2020-06-05 | 广东睿江云计算股份有限公司 | Cloud host CPU and memory modification method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11829256B2 (en) | Data storage management system for holistic protection of cloud-based serverless applications in single cloud and across multi-cloud computing environments | |
| US11294768B2 (en) | Live browsing of backed up data residing on cloned disks | |
| US20210011812A1 (en) | Preparing containerized applications for backup using a backup services container and a backup services container-orchestration pod | |
| US20210397522A1 (en) | High reliability interactive document management system implemented as software-as-a-service (saas) | |
| US9727273B1 (en) | Scalable clusterwide de-duplication | |
| Rao et al. | Performance issues of heterogeneous hadoop clusters in cloud computing | |
| US20170206140A1 (en) | System and method for building a point-in-time snapshot of an eventually-consistent data store | |
| US8706694B2 (en) | Continuous data protection of files stored on a remote storage device | |
| US10983908B1 (en) | Method and system for garbage collection of data protection virtual machines in cloud computing networks | |
| US10353872B2 (en) | Method and apparatus for conversion of virtual machine formats utilizing deduplication metadata | |
| US20130173771A1 (en) | Automated policy management in a virtual machine environment | |
| US11392458B2 (en) | Reconstructing lost data objects by generating virtual user files from available nodes within a cluster | |
| CN103929500A (en) | Method for data fragmentation of distributed storage system | |
| CN105531675A (en) | Decentralized distributed computing system | |
| CN102495774B (en) | Method and system for realizing system recovery of computer | |
| US8615488B2 (en) | Physical replication of database subset to improve availability and reduce resource cost in a cloud environment | |
| US20150154221A1 (en) | Deduplication method and deduplication system using data association information | |
| CN104239166A (en) | Method for realizing file backup for running virtual machine | |
| US8438130B2 (en) | Method and system for replicating data | |
| CN105302922B (en) | A kind of distributed file system snapshot implementing method | |
| CN102142032A (en) | Method and system for reading and writing data of distributed file system | |
| CN103902665A (en) | Storage virtualization system implementation method | |
| US9183208B1 (en) | Fileshot management | |
| US11079960B2 (en) | Object storage system with priority meta object replication | |
| US20130173780A1 (en) | Resource throttling and automated policy management in a virtual machine enviromnent |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140702 |