WO2013170504A1

WO2013170504A1 - Large data storage system

Info

Publication number: WO2013170504A1
Application number: PCT/CN2012/076516
Authority: WO
Inventors: 王东临; 金友兵
Original assignee: 天津书生投资有限公司
Priority date: 2012-05-16
Filing date: 2012-06-06
Publication date: 2013-11-21
Also published as: CN103428232B; CN103428232A

Abstract

Provided is a large data storage system having a high-performance and low-investment large data storage architecture. The system comprises a plurality of virtual machines running on a first physical server, and a first storage disk. The first physical server is directly connected to the first storage disk. The first storage disk is used for providing data storage. One of the virtual machines is used for supporting storage sharing functions. The other virtual machines are connected through an internal bus to the virtual machine supporting the storage sharing functions, and are used for receiving user requests and reading, according to such user requests, data in the first storage disk by means of the virtual machine supporting the storage sharing function, and presenting the data from the first storage disk to the user.

Description

A big data storage system

The present invention relates to the field of data storage, and in particular, to a big data storage system. Background technique

There are a variety of big data storage systems in the prior art, and Figure 1 shows a big data storage system commonly used in the prior art. As shown in Figure 1, the big data storage in the prior art is usually in the form of a SAN and a fiber switch, which is very expensive. The cloud storage technology represented by Hadoop uses a large number of inexpensive servers to form a large amount of storage capacity, which greatly reduces the cost compared with the SAN. However, each storage device still needs to be equipped with a corresponding storage server, which requires high network bandwidth and often needs With expensive network equipment, and the Name Node still has a single point of failure risk, cost, performance and reliability are still not ideal.

To this end, it is necessary to provide a high-performance, low-cost big data storage architecture capable of storing big data. Summary of the invention

The embodiment of the invention provides a big data storage system to provide a high performance, low input, high reliability big data storage architecture.

A large data storage system, which is included in the embodiment of the present invention, includes a plurality of virtual machines running on a first physical server, and a first storage disk, wherein the first physical server is directly connected to the first direct storage. Disk connection; where

The first directly connected storage disk is configured to provide data storage;

One of the multiple virtual machines for supporting a storage sharing function;

The other one of the multiple virtual machines is connected to the virtual machine supporting the storage sharing function through an internal bus, and is configured to receive a request from the user, and read the first by the virtual machine supporting the storage sharing function according to the user request. Directly connect the data of the storage disk to present the data on the first directly connected storage disk to the user. With the big data storage system provided by the embodiment of the present invention, the directly connected storage disk is directly connected to the physical server, and the access efficiency is higher than that of the network connection, and multiple physical machines are run on one physical server, so that one physical server is replaced. The functions of multiple physical servers in the prior art are flexible and inexpensive, and the access speed is fast because multiple virtual machines are connected through an internal bus. Therefore, the data storage system provided by the embodiments of the present invention has the advantages of high performance and low cost. DRAWINGS

FIG. 1 is a structural block diagram of a large data storage system commonly used in the prior art.

FIG. 2 is a structural block diagram of a big data storage system according to an embodiment of the present invention.

FIG. 3 is a structural block diagram of a big data storage system according to an embodiment of the present invention.

FIG. 4 is a structural block diagram of a big data storage system according to another embodiment of the present invention.

FIG. 5 is a structural block diagram of a big data storage system according to another embodiment of the present invention. detailed description

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

FIG. 2 is a structural block diagram of a big data storage system according to an embodiment of the present invention. As shown in FIG. 1, the physical server 100 is directly connected to the direct-attached storage 200, wherein the virtual server 101 runs on a plurality of virtual machines 101 to 104, wherein the virtual machine 104 has a storage sharing function; the virtual machines 101 to 103 and the virtual machine 104 Connected via an internal bus.

The virtual machines 101 to 103 are configured to receive a request from the user, and read the data of the direct-attached storage 200 through the virtual machine 104 according to the user request, and present the data on the direct-attached storage 200 to the user.

Direct Connect Storage 200 is used to provide data storage.

Those skilled in the art can understand that the number of virtual machines on the physical server is not limited to the number of illustrations, and the type and number of the virtual machines may be increased or decreased according to the performance of the physical server and the needs of the actual application. Presenting the data to the user is also one of the uses of the present invention. In practical applications, other applications for processing the data are also included in the solution of the present invention. In an embodiment of the invention, each direct storage may be formed by a disk array. In an embodiment of the invention, the disk array may adopt a RAID mode to improve reliability. You can increase the capacity by increasing the number of disks in the disk array. In an embodiment of the invention, the direct-attached storage 200 may also be cascaded by a plurality of disk arrays by means such as SAS lines.

The multi-virtual machine in the embodiment of the present invention is equivalent to the server cluster in the prior art. The scalable DAS in the embodiment of the present invention is compared with the San in the prior art, but the technical solution provided by the embodiment of the present invention may be used. The need for prior art storage servers and expensive fiber optic network systems is no longer required, and the cost is greatly reduced. In addition, in the prior art, when data is read, the data needs to be read to the storage server, and then through the network switch, and finally to the application server, and when the data is read by using the technical solution of the embodiment of the present invention, The data is directly read from the shared virtual machine, and then the application virtual machine is transmitted through the internal bus. It can be seen that the data access efficiency of the technical solution provided by the embodiment of the present invention is better.

In an embodiment of the present invention, multiple groups of application service groups can be deployed in a single physical application server to improve system service performance. FIG. 3 is a structural block diagram of a specific big data storage system according to an embodiment of the present invention. As shown in FIG. 3, two sets of application service groups are established in one physical server, and each set of application service groups includes three application servers with different functions, as shown in the figure, wherein each group of application service groups includes a post-web server vml or Vm4 (corresponds to the web server in the pre-server, for security reasons, the pre-server is usually located in another independent physical server, as shown in Figure 4), the application server vm2 or vm5 (used to provide different users) Applications such as mail servers, file servers, etc., upload server vm3 or vm6 (for receiving and processing user upload requests and data); the physical server further includes a virtual machine vm7, which has storage sharing capabilities With this virtual machine vm7, multiple virtual machines can access one Das device at the same time. The virtual machine vml-vm6 is connected to the virtual machine vm7 through the internal bus of the physical server, and directly connected to the DAS through the virtual machine vm7. In an embodiment of the invention, the virtual machine vml-vm6 is connected to the virtual machine vm7 through the NFS protocol. In an embodiment of the present invention, the application service group may further include a database server; each application service group may also include different types and different numbers of virtual servers, for example, the first application service group may include two application services. The second application service group can contain no application server or only one application server, but a database server. In addition, the number of virtual machines included in the two is not limited to the number shown in FIG. 2.

A person skilled in the art can understand that the type and number of application service groups of a virtual machine on a single physical server are not limited to the number of illustrated, and the number of application service groups can be increased or decreased according to the performance of the physical server and the needs of the actual application. .

FIG. 4 is a structural diagram of a big data storage system according to another embodiment of the present invention. As shown in Figure 4, the big data storage system is based on the big data storage system shown in Figures 2 and 3, and is further extended. If the physical server 100 and the direct attached storage disk 200 shown in FIG. 2 are referred to as a storage subsystem, the big data storage system shown in FIG. 4 includes at least N subsystems (N is an integer greater than or equal to 1, in a large In the case of data storage, N is usually a very large number). Each subsystem processes and stores data for different users, that is, stores different user data in different subsystems according to the user ID. In an example, each subsystem may store 10000 user data, store user data with ID 0-9999 in DAS1 of the first subsystem, and store user data with ID 10000-19999 in the second sub-item. System DAS2, and so on.

The system shown in FIG. 4 further includes: a pre-server for receiving a request of the user, and directing the request of the user to the corresponding subsystem according to the correspondence between each user and the subsystem recorded in the index database. , is processed and stored by different subsystems; an index database is used to record the correspondence between the user ID and the subsystem (the correspondence is not necessarily the foregoing sequential relationship, and it is possible that the user of ID1000 is in subsystem one, ID1001 The user in subsystem two, ID1002 user is in subsystem one). In an embodiment of the invention, the pre-database and the index database may be in the same physical server.

When the system rapidly expands the subsystem, it only needs to add the correspondence between the user ID and the subsystem in the index database. When the subsequent user accesses, the pre-server serves as a unified user portal, and the user request can be imported into the corresponding subsystem.

In an embodiment of the present invention, if user A shares a document to another user B, and user A's data is located in the first subsystem, and user B's request is handled by the second subsystem, then When user B desires to access the shared document, the processing flow is: the pre-server directs the request of user B to the physical server in the second subsystem, and the physical server of the second subsystem finds that the requested document is located in the first subsystem. After that, the physical server requesting the first subsystem provides the shared document to it. After receiving the request from the second subsystem, the physical server of the first subsystem first verifies the validity of the request (ie, verifies whether the user B has the right), and then obtains the shared document from the Dasl of the first subsystem, and Return it to the physical server of the first subsystem.

The system further includes a Nas system as a backup for each DAS. Once Das is damaged, the virtual server in the subsystem can directly read backup data from the NAS to provide services to users. Since the NAS is only used for backup, the performance requirements of the Nas are not high, so the cost can be greatly reduced. In addition, this figure illustrates only one Nas disk, but in one embodiment, any number of Nas may be used as a backup system.

In an embodiment of the invention, the system further includes an offline backup server for backing up data on the Nas. The dual backup of Nas backup and offline backup further ensures the security of the system.

Those skilled in the art will appreciate that each physical server in the diagram of Figure 4 omits the shared server virtual machine.

FIG. 5 is a structural diagram of a big data storage system according to another embodiment of the present invention. As shown in FIG. 5, physical servers 100 and 300 are directly connected to direct-attached storages 200 and 400, respectively, and further include a monitoring server 500.

Normally, the virtual servers 101 to 103 read the data of the direct-attached storage 200 through the virtual machine 104, and present the data on the direct-attached storage 200 to the user; the virtual servers 301 to 303 read the data of the direct-attached storage 400, The data on the direct attached storage 400 is presented to the user. However, once the monitoring server 500 monitors that the physical server 300 stops working, the user request responded to by the original physical server 300 is directed to the physical server 100, and the virtual machine (may be the virtual machines 101 to 103 may be The newly added virtual machines 105 to 107) present the data on the direct attached storage 400 to the user. On the contrary, once the monitoring server 500 monitors that the physical server 100 stops working, the user request responded by the original physical server 100 is directed to the physical service 300. The virtual machine on the physical server 300 presents the data on the direct attached storage 200 to the user. Specifically, after the monitoring server 500 monitors that the physical server 300 stops working, the information is returned to the pre-server and the index database, and the index database updates the correspondence between the user ID and the subsystem, and the subsequent pre-server will be the original The user request directed to the physical server 300 is directed to the physical server 100.

In another embodiment of the present invention, the virtual server 101-104 image on the physical server 100 is stored in the direct-attached storage 200. After the physical server 100 stops working, the physical server 300 can invoke the virtual machine 101 on the direct-attached storage 200. A mirror of 104 runs a new virtual machine to access data on the directly connected storage 200.

In another embodiment of the present invention, the SSD hard disk and memory can be built in the server 100 and/or 300 as a buffer to further improve performance.

Those skilled in the art can understand that the entire big data storage system can be extended by expanding the number of storage subsystems. For example, a big data storage system can contain 4000 storage subsystems, and each physical server can be directly connected with some or all of them. The storage connection, so that once the monitoring system detects that the physical server of a certain subsystem stops working, the user request originally connected to the physical server is imported to other physical servers directly connected to the subsystem, and through other physical The server accesses the direct attached storage of the subsystem.

It is also understood by those skilled in the art that the technical solutions described in the embodiments of the present invention can also be variously combined, and the combined big data storage system is also within the scope of the present disclosure. For example, at present, only one set of application service groups is listed in each physical server shown in FIG. 4, but it is obvious that the internal components of the physical servers can be as shown in FIG. 2 or FIG. 3. For another example, the subsystems of Figure 4 can be grouped in pairs, and the technical solutions shown in Figure 5 are adopted in each group to ensure redundancy.

With the embodiment of the present invention, there is no single point of failure, so the security is better.

The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalents, improvements, etc., which are included in the spirit and scope of the present invention, should be included in the present invention. Within the scope of protection.

Claims

Rights request

1. A big data storage system, characterized in that it includes multiple virtual machines running on a first physical server, and a first storage disk, wherein the first physical server is directly connected to the first storage disk ; in,

The first storage disk is used to provide data storage;

One of the multiple virtual machines is used to support the storage sharing function;

Other stations in the multiple virtual machines are connected to the virtual machine that supports the storage sharing function through an internal bus, and are used to receive the user's request. According to the user's request, read the first data through the virtual machine that supports the storage sharing function. Store the data on the disk, and present the data on the first storage disk to the user.

2. The system of claim 1, wherein the multiple virtual machines running on the first physical server are divided into at least two service groups, and each service group uses the virtual machine supporting the storage sharing function. Read data from the first storage disk.

3. The system of claim 1, wherein when the first physical server and the first storage disk are combined into a subsystem, the system further includes:

At least one subsystem for processing and storing data of different users;

The front-end server is used to receive user requests, and according to the corresponding relationship between each user and subsystem, direct the user's request to the corresponding subsystem, and then process and store it by different subsystems.

4. The system of claim 3, further comprising:

The index database is used to record and store the correspondence between user IDs and subsystems for call by the front-end server.

5. The system of claim 3, wherein the at least one subsystem includes multiple virtual machines running on a second physical server and a second storage disk;

The first physical server and the second physical server are directly connected to the first storage disk and the second storage disk respectively;

The multiple virtual machines on the second physical server are further used to: when the first physical server When the server fails to work properly, the data on the first storage disk is accessed.

6. The system of claim 5, wherein the multiple virtual machines on the second physical server that access data on the first storage disk are original service groups on the second physical server, or a new service group.

7. The system of claim 5, wherein the first storage disk is further used to store multiple virtual machine images of the first physical server;

The second physical server is further configured to call the multi-virtual machine image of the first physical server in the first storage disk when the first physical service cannot work normally, through the first physical server. The multiple virtual machine images access the data of the first directly connected storage disk.

8. The system according to claim 5, 6 or 7, further comprising: a monitoring server, configured to monitor the working status of the first server and the second physical server.

9. The system according to any one of claims 1 to 5, further comprising: NAS, used to back up data on the first storage disk, and when the first disk is damaged, NAS is used to backup the data on the first storage disk. Multiple virtual machines provide user data directly.

10. The system according to any one of claims 1 to 5, characterized in that the direct-connected storage is composed of one or a group of cascaded disk arrays.