CN107766343B - Data storage method and device and storage server - Google Patents
Data storage method and device and storage server Download PDFInfo
- Publication number
- CN107766343B CN107766343B CN201610671219.3A CN201610671219A CN107766343B CN 107766343 B CN107766343 B CN 107766343B CN 201610671219 A CN201610671219 A CN 201610671219A CN 107766343 B CN107766343 B CN 107766343B
- Authority
- CN
- China
- Prior art keywords
- storage
- storage resource
- data
- resource
- resources
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data storage method, a data storage device and a data storage server. When a data storage request is acquired, the SLA service requirement of the data storage request is determined, the storage resources meeting the SLA service requirement are matched in the existing storage resources according to a pre-established storage resource table, and data storage is performed by using the matched storage resources, so that the data can be adaptively stored in the corresponding storage resources according to the SLA service requirement of the data storage request.
Description
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a data storage method and apparatus, and a storage server.
Background
With the development of big data storage and application scenarios, the storage and access of massive heterogeneous data are problems often faced by big data applications and products. For example, in a video recommendation scene, it is generally necessary to store data of all movies (including information of director, actors, genre, year, and the like), comment data of a user on a movie, real-time portraits of a user (including data of movie genre preferred by a user and the like), a recommendation list of each user (a list of movies recommended for each user), and the like.
Massive heterogeneous data generally has different data types, including relational data, Wide-table (Wide-Column) data, Key Value pair (KV), files, and the like. For example, the comment data of the user on the movie is long text, the data information of the movie is a semi-structured wide table or KV, and the real-time representation of the user is the wide table or KV. Each data type may have various storage modes and storage media according to the requirements of a service or a user. For example, unstructured files may be stored on a Distributed computing (Hadoop) Distributed File System (HDFS), MongoDB (a database written in the C + + language based on Distributed File storage), broad-list or KV data may be stored on HDFS, HBase (a broad-list data storage tool), Cassandra (a broad-list data storage tool), Redis (a memory-based KV data storage tool), aerofile (a KV data storage tool), and so on. The HDFS file storage generally uses a magnetic disk as a main storage medium, and has low supportable access performance. Hbases typically use disk-stored wide-table data to support highly concurrent write operations and to support point-and-click (find data by row key), and Structured Query Language (SQL) queries can be supported with tools such as Phoenix (an SQL Query tool on hbases). Redis is memory-based KV storage that can provide high performance checkpointing. The cost of different storage media is different, for example, the memory database needs a large amount of memory and is high in cost, the cost of the SSD is medium, and the cost of the disk is low.
However, for different services and users, there are different access modes (click through, SQL query, etc.) and performance requirements for each data type, and these requirements determine the storage mode of data. For example, user comment data may only need to support low-concurrency, hourly-level queries, and only need to be stored on the HDFS; movie data only need to support minute or second-level click and need to be stored in HBase; real-time portraits of users need to support highly concurrent checkpointing, need to be stored on Redis, and the like. Therefore, data with different access modes and access performance requirements should be stored on corresponding storage tools and storage media as required, so that the performance can reach the standard and the cost is optimal.
Therefore, how to realize the adaptive data storage to the corresponding storage tool and storage medium according to different access modes and access performance requirements needed by the service and the user is an urgent problem to be solved.
Disclosure of Invention
Embodiments of the present invention provide a data storage method, an apparatus, and a storage server, so as to implement adaptively storing data in a corresponding storage resource according to a Service-Level agent (SLA) Service requirement of a data storage request.
In a first aspect, a data storage method is provided, where a storage resource table is created in advance, and a correspondence between storage resources and SLA service qualities that can be provided by the storage resources is stored in the storage resource table. When a data storage request is acquired, the SLA service requirement of the data storage request is determined, the storage resources meeting the SLA service requirement are matched in the existing storage resources according to a pre-established storage resource table, and data storage is performed by using the matched storage resources, so that the data can be adaptively stored in the corresponding storage resources according to the SLA service requirement of the data storage request.
The pre-created storage resource table may be created according to a data type, an access method, an access performance, and a storage resource location that are supported by a storage resource included in a registration request of the storage resource.
In a possible design, if at least two storage resources meeting the SLA service requirements are matched in existing storage resources according to a pre-created storage resource table, one storage resource is selected from the at least two storage resources meeting the SLA service requirements as a finally matched storage resource.
The storage resource with the largest remaining capacity of the storage resource can be selected from the at least two storage resources meeting the SLA service requirements as the finally matched storage resource, and the storage resource with the fastest response time can also be selected from the at least two storage resources meeting the SLA service requirements as the finally matched storage resource.
In another possible design, the SLA service quality that can be provided by the storage resources in the pre-created storage resource table can be adjusted according to the access performance of each storage resource, so that when the storage resources are subsequently matched, the storage resources with the optimal storage performance and meeting the SLA service requirements can be matched in the existing storage resources according to the pre-created storage resource table.
In another possible design, if it is determined that a storage resource for stopping providing the data storage service exists in the pre-created storage resource table, the storage resource for stopping providing the data storage service is deleted from the pre-created storage resource table.
A second aspect provides a storage server, which includes an application program interface, a storage resource manager, and a resource matcher, where the application program interface is configured to obtain a data storage request and determine a service access level agreement SLA service requirement of the data storage request. The storage resource manager is used for creating and storing a storage resource table in advance, and the storage resource table stores the corresponding relation between the storage resources and the SLA service quality which can be provided by the storage resources. The resource matcher is used for matching out storage resources meeting the SLA service requirement acquired by the application program interface from the existing storage resources according to a storage resource table stored by the storage resource manager, and storing data by using the matched storage resources, so that the data can be adaptively stored in the corresponding storage resources according to the SLA service requirement of the data storage request.
The storage resource manager creates the storage resource table in advance according to the data type, the access mode, the access performance and the storage resource position supported by the storage resource in the registration request of the storage resource.
In one possible design, if the resource matcher matches at least two storage resources meeting the SLA service requirements in existing storage resources according to a pre-created storage resource table, one storage resource is selected from the at least two storage resources meeting the SLA service requirements as a finally matched storage resource.
In another possible design, the storage resource manager is further configured to: and according to the pre-created storage resource table, before the storage resources meeting the SLA service requirements are matched from the existing storage resources, the SLA service quality which can be provided by the storage resources in the pre-created storage resource table is adjusted according to the access performance of the storage resources.
In another possible design, if the storage resource manager determines that a storage resource stopping providing the data storage service exists in a pre-created storage resource table, the storage resource manager deletes the storage resource stopping providing the data storage service in the storage resource table.
In a third aspect, a data storage apparatus is provided, which has a function of implementing the behavior of the network device in the first aspect. The functions can be realized by hardware, and the functions can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above-described functions.
In one possible design, the data storage apparatus includes a fetching unit and a processing unit, where the fetching unit is configured to fetch a data storage request and determine a service access level agreement, SLA, service requirement of the data storage request. The processing unit is configured to create a storage resource table in advance, where a correspondence between a storage resource and an SLA service quality that can be provided by the storage resource is stored in the storage resource table. The processing unit matches the storage resources meeting the SLA service requirements acquired by the acquisition unit from the existing storage resources according to a pre-created storage resource table, and performs data storage by using the matched storage resources, so that the SLA service requirements of the data storage request can be met, and data can be adaptively stored in the corresponding storage resources.
The processing unit creates the storage resource table in advance according to the data type, the access mode, the access performance and the storage resource position supported by the storage resource in the registration request of the storage resource.
The processing unit specifically matches, in the existing storage resources, the storage resources that meet the SLA service requirement acquired by the acquiring unit according to a pre-created storage resource table in the following manner: and if at least two storage resources meeting the SLA service requirements are matched in the existing storage resources according to a pre-established storage resource table, selecting one storage resource from the at least two storage resources meeting the SLA service requirements as the finally matched storage resource.
The processing unit is further configured to: and according to the pre-created storage resource table, before the storage resources meeting the SLA service requirements are matched from the existing storage resources, the SLA service quality which can be provided by the storage resources in the pre-created storage resource table is adjusted according to the access performance of the storage resources.
If the processing unit determines that the storage resource for stopping providing the data storage service exists in the pre-created storage resource table, the processing unit deletes the storage resource for stopping providing the data storage service in the storage resource table.
In another possible design, the data storage device includes a processor configured to support the data storage device to perform the corresponding functions of the method of the first aspect, and an I/O interface configured to support communication between the data storage device and an application server or the like.
Further, the network device may further include a storage, a network adapter, a memory, a storage adapter, and the like.
The data storage device 200 may be a storage server.
In a fourth aspect, a computer storage medium is provided for storing computer software instructions for the data storage device, which includes a program designed to execute the data storage method according to the first aspect.
Drawings
FIG. 1 is a system architecture for implementing the present invention;
FIG. 2 is a flow chart of a data storage method according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a storage server according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a storage server according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of interaction among an application program, a data and access middle layer and a storage tool provided by the embodiment of the invention;
FIG. 6 is a diagram illustrating a storage resource registration process according to an embodiment of the present invention;
FIG. 7 is a diagram illustrating a logout procedure of a storage resource according to an embodiment of the present invention;
FIG. 8 is a diagram illustrating a process for creating metadata according to an embodiment of the present invention;
FIG. 9 is a diagram illustrating a process for deleting metadata according to an embodiment of the present invention;
FIG. 10 is a diagram illustrating a process for storing data according to an embodiment of the present invention;
FIG. 11 is a diagram illustrating a process of querying data according to an embodiment of the present invention;
FIG. 12 is a diagram illustrating a storage resource replacement process according to an embodiment of the present invention;
FIG. 13 is a schematic structural diagram of a data storage device according to an embodiment of the present invention;
fig. 14 is a schematic structural diagram of a data storage device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described in detail below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only some embodiments, and not all embodiments, of the present invention.
Fig. 1 shows a system architecture applied in an embodiment of the present invention, and as shown in fig. 1, an external host is connected to a Storage system through a Storage Area Network (SAN), and an application server and a Storage server in the Storage system are connected through a low-latency high-throughput Network (e.g., a fibre channel FC, a 10G ethernet) to form a cluster, so as to provide a data Storage service to the outside. The application server is provided with a network adapter connected with an external host; the storage server includes local storage devices such as a disk and a Solid State Drive (SSD), and is configured to store data. A typical data storage process is: an external host initiates a data storage request, which is then sent to an application server over a network, such as a SAN, and then routed to a particular storage server via a protocol internal to the storage system.
The embodiment of the invention provides a storage server, which is characterized in that a storage resource table is created in advance, and the corresponding relation between storage resources and SLA service quality which can be provided by the storage resources is stored in the storage resource table. When the storage server obtains a data storage request, the SLA service requirement of the data storage request is determined, the storage resources meeting the SLA service requirement are matched in the existing storage resources according to a pre-established storage resource table, and the matched storage resources are utilized for data storage, so that the data can be adaptively stored in the corresponding storage resources according to the SLA service requirement of the data storage request.
Fig. 2 is a flowchart illustrating an implementation method of implementing data storage by a storage server, as shown in fig. 2, including:
s101: the storage server acquires a data storage request and determines service access level agreement (SLA) service requirements of the data storage request.
Wherein the data storage request may be sent by an application program in an application server. The SLA service requirements comprise requirements of data types, access modes, access performances and the like of data needing to be stored.
S102: and the storage server matches the storage resources meeting the SLA service requirements in the existing storage resources according to a pre-established storage resource table.
In the embodiment of the invention, the corresponding relation between the storage resources and the SLA service quality which can be provided by the storage resources is stored in the pre-established storage resource table. The pre-created storage resource table may be created according to a data type, an access method, an access performance, and a storage resource location that are supported by a storage resource included in a registration request of the storage resource.
After the storage server acquires the data storage request, based on the access mode and the performance requirement in the SLA service requirement of the acquired data storage request, and based on a preset rule, according to a pre-created storage resource table, the storage resource meeting the access mode and the performance requirement can be matched in the existing storage resources.
S103: and storing data by using the matched storage resources.
Optionally, in the embodiment of the present invention, if at least two storage resources meeting the SLA service requirement are matched in existing storage resources according to a pre-created storage resource table, one storage resource is selected from the at least two storage resources meeting the SLA service requirement as a finally matched storage resource. The embodiment of the present invention is not limited, for example, the storage resource with the largest remaining capacity of the storage resource may be selected, and the storage resource with the fastest response time may also be selected.
Optionally, in the embodiment of the present invention, the SLA service quality that can be provided by the storage resources in the pre-created storage resource table may also be adjusted according to the access performance of each storage resource, so that when the storage resources are subsequently matched, the storage resources with the optimal storage performance and meeting the SLA service requirements may be matched in the existing storage resources according to the pre-created storage resource table.
Optionally, if it is determined that a storage resource stopping providing the data storage service exists in the pre-created storage resource table, deleting the storage resource stopping providing the data storage service in the pre-created storage resource table.
In the embodiment of the invention, the storage server matches the storage resources meeting the SLA service requirements in the existing storage resources according to the pre-established storage resource table, and performs data storage by using the matched storage resources, so that the data can be adaptively stored in the corresponding storage resources according to the SLA service requirements of the data storage request. In order to implement the above functions, the storage server in the embodiments of the present invention includes a hardware structure and/or a software module corresponding to each function. The elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein may be embodied in hardware or in a combination of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present teachings.
The embodiment of the present invention may perform the division of the functional units on the storage server according to the above method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. It should be noted that the division of the unit in the embodiment of the present invention is schematic, and is only a logic function division, and there may be another division manner in actual implementation.
Based on the data storage method, the embodiment of the invention provides a storage server. Fig. 3 is a schematic diagram illustrating a possible structure of a storage server according to an embodiment of the present invention. In fig. 3, the storage server 100 includes an Application Programming Interface (API) 101, a storage resource manager 102, and a resource matcher 103.
The application program interface 101 is configured to obtain a data storage request and determine an SLA service requirement of the data storage request. Wherein the data storage request may be sent by an application program in an application server. The SLA service requirements comprise requirements of data types, access modes, access performances and the like of data needing to be stored.
Optionally, the application program interface 101 may provide operations such as resource registration/deregistration, metadata creation/deletion, data storage/modification, data query, etc., for example, an application program in an application server may also send a data query request to the storage server 100 through the application program interface 101.
The storage resource manager 102 is configured to create and store a storage resource table in advance.
In the embodiment of the present invention, a storage resource table created in advance by the storage resource manager 102 stores a correspondence between a storage resource and an SLA service quality that can be provided by the storage resource. The storage resource refers to a storage tool and a storage medium on the storage server 100, which can provide a data storage function, for example, HDFS, HBase, Cassandra, Redis, Riak, Aerospike, and the like. The SLA service quality which can be provided by the storage resource refers to the data type, the access mode, the access performance, the storage resource position and the like which are supported by the storage resource. The corresponding relationship between the storage resources in the storage resource table and the SLA service quality that can be provided by the storage resources in the embodiment of the present invention can be shown in table1 below:
| wide-column,key-value,20ms | HBase1@location1 |
| wide-column,SQL,50ms | Phoenix1@location2 |
| wide-column,key-value,10ms | HBase3@location3 |
| KV,key-value,5ms | Redis1@location4 |
TABLE1
In table1, the HBase1 cluster at location (location)1 can provide a 20ms response time point-check (key-value) for wide table data (wide-column), the Phoenix1 cluster at location2 (an HBase cluster with Phoenix query tool installed) can provide a 50ms response time SQL query for wide table data (wide-column), the HBase3 cluster at location3 may use SSD as storage, can provide a 10ms response time point-check (key-value) for wide table data (wide-column), and the Redis1 cluster at location4 can provide a 5ms response time point-check (key-value) for KV data.
The resource matcher 103 is configured to match, according to a storage resource table stored by the storage resource manager 102, a storage resource that meets the SLA service requirement acquired by the application program interface 101 among existing storage resources, and store data by using the matched storage resource.
In the embodiment of the present invention, the resource matcher 103 may match a storage resource that meets the access mode and the performance requirement in an existing storage resource based on a preset rule based on the access mode and the performance requirement in the SLA service requirement of the data storage request acquired by the application program interface 101. For example, if the data storage request of the application indicates that wide table data needs to be stored, a click is needed in the future, and the performance requires 15ms, the matched storage resource can be determined to be HBase3@ location3 according to the storage resource table stored in table 1. Embodiments of the present invention do not limit the specific matching rules, which may be just matching the SLA service requirements of the data storage request, or may be higher than the SLA service requirements of the data storage request, for example, a high-end user may provide a storage tool with higher performance than the service required by the high-end user for matching.
Optionally, in the embodiment of the present invention, if the resource matcher 103 matches at least two storage resources meeting the SLA service requirement from existing storage resources according to a pre-created storage resource table, one storage resource is selected from the at least two storage resources meeting the SLA service requirement as a finally matched storage resource. The embodiment of the present invention is not limited, for example, the storage resource with the largest remaining capacity of the storage resource may be selected, and the storage resource with the fastest response time may also be selected.
Optionally, in this embodiment of the present invention, the storage resource manager 102 may further adjust, according to the access performance of each storage resource, the SLA service quality that can be provided by the storage resource in the pre-created storage resource table, so that when the subsequent resource matcher 103 matches the storage resource, the storage resource with the optimal storage performance and meeting the SLA service requirement may be matched in the existing storage resource according to the pre-created storage resource table.
Optionally, if it is determined that a storage resource stopping providing the data storage service exists in the pre-created storage resource table, the storage resource manager 103 deletes the storage resource stopping providing the data storage service in the pre-created storage resource table.
The storage server provided by the embodiment of the present invention is described in detail below with reference to practical applications.
Optionally, in the embodiment of the present invention, the storage server 100 may further include a storage metadata manager 104, as shown in fig. 4. The metadata manager 104 is configured to manage data tables on all storage resources on the storage server 100 that can provide a data storage function, and receive a data packet storage location query request sent by a resource matcher to query a specific location of the data table when the application program interface 101 obtains the data storage request. In this embodiment of the present invention, a table structure information (Schema) metadata table including a data table may be stored in the metadata manager 104, where the Schema metadata table may be as shown in table 2:
| table1 | HBase1@location1 |
| table2 | HBase1@location1 |
| table3 | Phoenix1@location2 |
| table4 | HBase3@location3 |
TABLE2
In Table2, data table1 is on HBase1 at location1, data table2 is on HBase1 at location1, data table3 is on Phoenix1 at location2, and data table4 is on HBase3 at location 3.
It should be noted that the structure of the storage server 100 shown in fig. 3 and fig. 4 in the embodiment of the present invention is only schematically illustrated, and is a simplified design of the storage server 100, and does not limit the structure of the storage server 100 provided in the embodiment of the present invention. In practical applications, the metadata manager 104, the storage resource manager 102, the resource matcher, and the like may be independent components, or all or part of them may be integrated into one component. In addition, in practical applications, the storage server may further include a network adapter, an input/output O interface, a memory, a storage adapter, and the like. All storage servers that can implement embodiments of the present invention are within the scope of the embodiments of the present invention.
It should be noted that the application program interface 101, the storage resource manager 102, the resource matcher 103, and the metadata manager 104 in the embodiment of the present invention may be understood as an intermediate layer of data storage and access. A storage resource may be understood as a storage facility on which an application performs data storage or data query via the data storage and access middle layer, as shown in fig. 5.
For convenience of description, the following description of the embodiments of the present invention will be made with reference to the structure shown in fig. 5 to describe processes of registering/deregistering, creating/deleting metadata, storing/modifying data, querying data, and the like of storage resources involved in data storage and access processes.
Storage resource registration
Storage resources can be used to store data in the future, first requiring registration with a storage resource manager and a metadata manager on a data storage and access middle tier.
FIG. 6 shows a schematic diagram of a storage resource registration process. The storage resource initiates a registration request to the storage resource manager, where the registration request includes information such as a data type, an access mode, an access performance, and a storage resource location supported by the storage resource, and for example, if the storage resource initiates a command { wide-column, Key Value, 20ms, register, HBase1@ location1} to the storage resource manager, it indicates that the storage resource initiates the registration request, the registered storage resource address is HBase1@ location1, the data type supporting storage is wide-table data (wide-column), the access mode is a click-Value, and the access performance is a response time of 20 ms. After receiving the registration request, the storage resource manager adds a corresponding relationship between a storage resource and the SLA service quality that the storage resource can provide, for example, to a pre-created storage resource table according to the data type, access mode, access performance, and storage resource location supported by the storage resource included in the registration request of the storage resource, for example, adding a corresponding relationship between a storage resource and the SLA service quality that the storage resource can provide to the storage resource table
| wide-column,key-value,20ms | HBase1@location1 |
Optionally, if data is stored on the storage resource, the storage resource further needs to register Schema metadata information and location provided by the storage resource with the metadata manager, for example, the storage resource sends a command { table1, register, HBase1@ location1} to the metadata manager, which indicates that the storage resource initiates a registration request to the metadata manager, registers the storage resource with the storage resource address of HBase1@ location1, and stores a data table1 on the storage resource with the storage resource address of HBase1@ location 1.
Second, logging off storage resources
FIG. 7 illustrates a schematic diagram of a storage resource deregistration process. When the storage resource stops providing the data storage service, a logout request needs to be initiated to the storage resource manager. For example, when the storage resource HBase stops providing data storage service, a logout request command { unregister, HBase1@ location1} is initiated to the storage resource manager. And after receiving the logout request, the storage resource manager determines that the storage resource stopping providing the data storage service exists in a pre-created storage resource table, and deletes the storage resource HBase stopping providing the data storage service in the storage resource table. Meanwhile, a storage resource which stops providing the data storage service also needs to log out Schema metadata provided by the storage resource to the metadata manager, for example, when the storage resource HBase stops providing the data storage service, a log-out request command { table1, unregister, HBase1@ location1} is initiated to the metadata manager.
Third, create metadata
Fig. 8 shows a schematic diagram of a process of creating metadata. The application sends a data table creation request to the application program interface, as in (r) of fig. 8. The data table creating request comprises SLA service requirements of storage and access, the SLA service requirements comprise data types, access modes, access performance and Schema metadata of data to be created, for example, if a data table creating command sent by the application program is { wide-column, Key Value, 20ms, create, table1}, it is indicated that the data types are wide-table data (wide-column), the access modes are point-check (Key-Value), and the access performance is data table1 of 20 ms. After the application program interface receives the data table creation request, the resource matcher obtains the data table creation request, and inquires storage resources meeting the storage and access requirements from a storage resource manager, for example, fig. 8. The storage resource manager matches storage resources meeting storage and access requirements in the existing storage resources according to a pre-stored storage resource table, and feeds back the matched storage resources (possibly a plurality of storage resources) to the resource matcher. The resource matcher selects one storage resource from the storage resources fed back by the storage resource manager according to a rule (e.g., a rule such as a high remaining capacity of the storage resource), for example, selects HBase1@ location1, and feeds back the selected storage resource to the application program interface, as shown in fig. 8. The application program interface calls an application program interface of the corresponding storage resource HBase1@ location1, and creates a data table1 of Schema metadata on the selected storage resource HBase1@ location1, such as the data table ((r) in fig. 8). The application program interface sends the data table1 of Schema metadata and the corresponding storage location HBase1@ location1 to the metadata manager, e.g., in fig. 8. The metadata manager stores the correspondence between said data table1 and the corresponding storage location HBase1@ location 1.
Fourthly, deleting metadata
Fig. 9 shows a schematic diagram of a process of deleting metadata. The application sends a data table deletion request to the application program interface. For example, if the data table deletion request command sent by the application is { drop, table1}, it indicates that the data table1 needs to be deleted, as shown in fig. 9. The application program interface queries the metadata manager for the storage location of the data table1, shown as ② in FIG. 9. The metadata manager returns the location HBase1@ location1 of the storage resource where the data table1 resides to the application program interface, shown in fig. 9 c. The application program interface deletes the data table1 from the storage resource HBase1@ location1 at the location through the application program interface HBase API corresponding to the storage resource HBase1, as shown in (r) of fig. 9. The application program interface updates the metadata information to the metadata manager and deletes the metadata (data table1), as indicated by 9 in the figure.
Fifthly, storing/modifying data:
FIG. 10 shows a schematic diagram of a process for storing data. The application program sends a data storage request to the application program interface, wherein the data storage request is used for indicating data to be written in the data table. For example, the data storage request command sent by the application is { table1, put, data }, which indicates that data needs to be written in the data table1, such as (r) in fig. 10. The application program interface queries the metadata manager for the storage location of the data table1, shown as ② in FIG. 10. The metadata manager queries the storage location of the data table1 in the stored Schema metadata table, and feeds back the location HBase1@ location1 of the storage resource where the data table1 is located to the application program interface, as shown in fig. 10. The application program interface writes data into the storage resource at the position HBase1@ location1 through the application program interface HBase API corresponding to the storage resource HBase1, as shown in (r) of fig. 10.
The data in the data table is modified in a similar manner, and is not described herein again.
Sixthly, inquiring data
FIG. 11 shows a schematic diagram of a process for querying data. The application sends a data table query request to the application program interface, as shown in fig. 11, the application sends a data table query request command { table1, get, data } to the application program interface, which indicates that data needs to be queried in the data table 1. The application program interface queries the metadata manager for the location of the data table1, shown as ② in FIG. 11. The metadata manager queries the storage location of the data table1 in the stored Schema metadata table, and feeds back the location HBase1@ location1 of the storage resource where the data table1 is located to the application program interface, as shown in fig. 11. And performing data query on the storage resource with the position HBase1@ location1 through an application program interface HBase API corresponding to the storage resource HBase1 by using a program programming interface, such as the fourth step in FIG. 11. Of course, in the embodiment of the present invention, if there is an associated operation across data sources, the application program interface calls the associated interface across data sources provided by the data storage and access middle layer to perform an operation). The application program interface returns the data to the application program, e.g., # in fig. 11.
In practical applications, storage resource replacement may also occur, and fig. 12 shows a schematic diagram of a storage resource replacement process. If the HBase1 at the position of location1 needs to be replaced by HBase3, data on the cluster to be replaced HBase1 is led into the replacement cluster HBase3 as required, such as the step (r) in FIG. 12. The storage resource registration is performed by the replacement cluster HBase3, and the specific registration process may refer to the resource registration process according to the foregoing embodiment. The storage resource manager updates the stored storage resource table, and the metadata manager updates the stored Schema metadata table, as shown in fig. 12. The resource logout is performed by the cluster to be replaced HBase1, and the specific logout process may refer to the resource logout process according to the above embodiment, where the storage resource manager deletes the relevant record in the storage resource table, and the metadata manager deletes the record in the Schema metadata table, as shown in fig. 12.
Based on the data storage method provided by the above embodiment, the embodiment of the invention also provides a data storage device. In the case of using an integrated unit, fig. 13 shows a schematic structural diagram of a data storage device 200 according to an embodiment of the present invention. As shown in fig. 13, the data storage transpose 200 includes a fetching unit 201 and a processing unit 202, wherein the fetching unit 201 is configured to fetch a data storage request and determine a service access level agreement SLA service requirement of the data storage request. The processing unit 202 is configured to create a storage resource table in advance, where a corresponding relationship between a storage resource and an SLA service quality that can be provided by the storage resource is stored in the storage resource table. The processing unit 202 matches a storage resource meeting the SLA service requirement acquired by the acquiring unit 201 in an existing storage resource according to a pre-created storage resource table, and stores data by using the matched storage resource.
The processing unit 202 specifically matches, in the following manner, a storage resource meeting the SLA service requirement acquired by the acquiring unit 201 in an existing storage resource according to a pre-created storage resource table: and if at least two storage resources meeting the SLA service requirements are matched in the existing storage resources according to a pre-established storage resource table, selecting one storage resource from the at least two storage resources meeting the SLA service requirements as the finally matched storage resource.
Optionally, the processing unit 202 is further configured to: and according to the pre-created storage resource table, before the storage resources meeting the SLA service requirements are matched from the existing storage resources, the SLA service quality which can be provided by the storage resources in the pre-created storage resource table is adjusted according to the access performance of the storage resources.
The processing unit 202 creates the storage resource table in advance according to the data type, the access mode, the access performance, and the storage resource location supported by the storage resource included in the registration request of the storage resource.
Optionally, if it is determined that a storage resource stopping providing the data storage service exists in the pre-created storage resource table, the processing unit 202 is further configured to delete the storage resource stopping providing the data storage service in the storage resource table.
In the embodiment of the present invention, the processing unit 202 of the data storage device 200 may be a processor. The fetch unit 201 of the data storage device 200 may be an I/O interface, wherein the I/O interface is a generic term and may comprise one or more interfaces.
When the processing unit 202 is a processor and the obtaining unit is an I/O interface, the data storage device provided by the embodiment of the present invention may have a structure as shown in fig. 14.
Fig. 14 shows another possible structure diagram of the data storage apparatus 200 according to the embodiment of the present invention, and as shown in fig. 14, the data storage apparatus 200 includes a processor configured to support the data storage apparatus 200 to perform the corresponding functions in the above method, and an I/O interface configured to support communication between the data storage apparatus and an application server or the like.
Further, the network device may further include a storage, a network adapter, a memory, a storage adapter, and the like, as shown in fig. 14.
The data storage device 200 may be a storage server.
It should be noted that the data storage device 200 provided in the embodiment of the present invention can be used to implement the data storage method related to the foregoing embodiment, and has the functions of the storage server 100 related to the foregoing embodiment, and for a place that is not described in detail in the embodiment of the present invention, reference may be made to the relevant description of the foregoing embodiment, and no further description is given to the embodiment of the present invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (15)
1. A method of storing data, comprising:
acquiring a data storage request and determining service access level agreement (SLA) service requirements of the data storage request;
matching out storage resources meeting the SLA service requirements from existing storage resources according to a pre-created storage resource table, wherein the storage resource table stores the corresponding relation between the storage resources and the SLA service quality which can be provided by the storage resources;
and storing data by using the matched storage resources.
2. The method of claim 1, wherein matching out storage resources satisfying the SLA service requirements among existing storage resources according to a pre-created storage resource table comprises:
and if at least two storage resources meeting the SLA service requirements are matched in the existing storage resources according to a pre-established storage resource table, selecting one storage resource from the at least two storage resources meeting the SLA service requirements as the finally matched storage resource.
3. The method according to claim 1 or 2, wherein before matching out the storage resources satisfying the SLA service requirements from the existing storage resources according to a pre-created storage resource table, the method further comprises:
and adjusting the SLA service quality which can be provided by the storage resources in the pre-created storage resource table according to the access performance of the storage resources.
4. The method according to any one of claims 1 to 3, wherein the pre-created storage resource table is created according to data types supported by the storage resource, access ways, access performances and storage resource locations included in the registration request of the storage resource.
5. The method of any of claims 1 to 4, further comprising:
and if the storage resource table created in advance is determined to have the storage resource stopping providing the data storage service, deleting the storage resource stopping providing the data storage service from the storage resource table.
6. A storage server comprising an application program interface, a storage resource manager and a resource matcher, wherein,
the application program interface is used for acquiring a data storage request and determining the service access level agreement SLA service requirement of the data storage request;
the storage resource manager is used for creating and storing a storage resource table in advance, and the storage resource table stores a corresponding relation between storage resources and SLA service quality which can be provided by the storage resources;
and the resource matcher is used for matching out storage resources meeting the SLA service requirement acquired by the application program interface from the existing storage resources according to a storage resource table stored by the storage resource manager, and storing data by using the matched storage resources.
7. The storage server according to claim 6, wherein the resource matcher matches a storage resource satisfying the SLA service requirement acquired by the application program interface among existing storage resources according to a storage resource table stored by the storage resource manager, in the following manner:
and if at least two storage resources meeting the SLA service requirements are matched in the existing storage resources according to a pre-established storage resource table, selecting one storage resource from the at least two storage resources meeting the SLA service requirements as the finally matched storage resource.
8. The storage server of claim 6 or 7, wherein the storage resource manager is further to:
and according to the pre-created storage resource table, before the storage resources meeting the SLA service requirements are matched from the existing storage resources, the SLA service quality which can be provided by the storage resources in the pre-created storage resource table is adjusted according to the access performance of the storage resources.
9. The storage server according to any one of claims 6 to 8, wherein the storage resource manager creates the storage resource table in advance according to a data type supported by the storage resource, an access method, an access performance, and a storage resource location included in the registration request of the storage resource.
10. The storage server of any of claims 6 to 9, wherein the storage resource manager is further to:
and if the storage resource table created in advance is determined to have the storage resource stopping providing the data storage service, deleting the storage resource stopping providing the data storage service from the storage resource table.
11. A data storage device comprising an acquisition unit and a processing unit, wherein,
the acquiring unit is used for acquiring a data storage request and determining the service access level agreement SLA service requirement of the data storage request;
the processing unit is used for creating a storage resource table in advance, matching storage resources meeting the SLA service requirements acquired by the acquisition unit in the existing storage resources according to the created storage resource table in advance, and storing data by using the matched storage resources;
wherein, the storage resource table stores the corresponding relationship between the storage resource and the SLA service quality which can be provided by the storage resource.
12. The data storage device according to claim 11, wherein the processing unit matches, in accordance with a pre-created storage resource table, storage resources that satisfy the SLA service requirement acquired by the acquiring unit among existing storage resources, specifically in the following manner:
and if at least two storage resources meeting the SLA service requirements are matched in the existing storage resources according to a pre-established storage resource table, selecting one storage resource from the at least two storage resources meeting the SLA service requirements as the finally matched storage resource.
13. The data storage device of claim 11 or 12, wherein the processing unit is further to:
and according to the pre-created storage resource table, before the storage resources meeting the SLA service requirements are matched from the existing storage resources, the SLA service quality which can be provided by the storage resources in the pre-created storage resource table is adjusted according to the access performance of the storage resources.
14. The data storage device according to any one of claims 11 to 13, wherein the processing unit creates the storage resource table in advance according to a data type supported by the storage resource, an access method, an access performance, and a storage resource location included in the registration request of the storage resource.
15. The data storage device of any of claims 11 to 14, wherein the processing unit is further to:
and if the storage resource table created in advance is determined to have the storage resource stopping providing the data storage service, deleting the storage resource stopping providing the data storage service from the storage resource table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610671219.3A CN107766343B (en) | 2016-08-15 | 2016-08-15 | Data storage method and device and storage server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610671219.3A CN107766343B (en) | 2016-08-15 | 2016-08-15 | Data storage method and device and storage server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107766343A CN107766343A (en) | 2018-03-06 |
| CN107766343B true CN107766343B (en) | 2022-02-25 |
Family
ID=61260039
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610671219.3A Active CN107766343B (en) | 2016-08-15 | 2016-08-15 | Data storage method and device and storage server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107766343B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110389710A (en) * | 2018-04-20 | 2019-10-29 | 华为技术有限公司 | The method and apparatus for distributing storage resource |
| CN108803542B (en) * | 2018-08-01 | 2020-09-18 | 武汉索雅信息技术有限公司 | Industrial production data management system |
| CN109245949B (en) * | 2018-10-31 | 2022-03-01 | 新华三技术有限公司 | Information processing method and device |
| CN109726202B (en) * | 2018-12-18 | 2020-11-17 | 北京新唐思创教育科技有限公司 | Block chain data storage method and computer storage medium |
| CN110347675A (en) * | 2019-06-05 | 2019-10-18 | 阿里巴巴集团控股有限公司 | A kind of date storage method and device |
| US10970309B2 (en) | 2019-06-05 | 2021-04-06 | Advanced New Technologies Co., Ltd. | Data storage method and apparatus |
| CN111209310B (en) * | 2020-04-23 | 2020-07-31 | 太平金融科技服务（上海）有限公司 | Service data processing method and device based on stream computing and computer equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103338241A (en) * | 2013-06-19 | 2013-10-02 | 合肥工业大学 | Novel public cloud architecture and virtualized resource self-adaption configuration method thereof |
| CN105808341A (en) * | 2014-12-29 | 2016-07-27 | 中国移动通信集团公司 | Method, apparatus and system for scheduling resources |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100918384B1 (en) * | 2006-12-05 | 2009-09-24 | 한국전자통신연구원 | SLA application Apparatus and Method for Internet Leased Line Service |
| US7856499B2 (en) * | 2008-03-20 | 2010-12-21 | Sap Ag | Autonomic provisioning of hosted applications with level of isolation terms |
| US8189486B2 (en) * | 2008-03-28 | 2012-05-29 | Verizon Patent And Licensing Inc. | Method and system for providing holistic, iterative, rule-based traffic management |
| US8621476B2 (en) * | 2008-12-15 | 2013-12-31 | Korea Advanced Institute Of Science And Technology | Method and apparatus for resource management in grid computing systems |
| CN101969475A (en) * | 2010-11-15 | 2011-02-09 | 张军 | Business data controllable distribution and fusion application system based on cloud computing |
| CN102137014B (en) * | 2011-03-11 | 2013-12-04 | 华为技术有限公司 | Resource management method, system and resource manager |
| GB2502337A (en) * | 2012-05-25 | 2013-11-27 | Ibm | System providing storage as a service |
-
2016
- 2016-08-15 CN CN201610671219.3A patent/CN107766343B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103338241A (en) * | 2013-06-19 | 2013-10-02 | 合肥工业大学 | Novel public cloud architecture and virtualized resource self-adaption configuration method thereof |
| CN105808341A (en) * | 2014-12-29 | 2016-07-27 | 中国移动通信集团公司 | Method, apparatus and system for scheduling resources |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107766343A (en) | 2018-03-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107766343B (en) | Data storage method and device and storage server | |
| US20190068690A1 (en) | Automated management of resource attributes across network-based services | |
| WO2020223268A1 (en) | A declarative and reactive data layer for component-based user interfaces | |
| US20160149766A1 (en) | Cloud based management of storage systems | |
| US10592470B2 (en) | Discovery of calling application for control of file hydration behavior | |
| CN102082800A (en) | User request processing method and server | |
| US11182406B2 (en) | Increased data availability during replication | |
| US10394775B2 (en) | Order constraint for transaction processing with snapshot isolation on non-transactional NoSQL servers | |
| CN111324606B (en) | Data slicing method and device | |
| US11308066B1 (en) | Optimized database partitioning | |
| US9456036B2 (en) | Switch-based data tiering | |
| US10262024B1 (en) | Providing consistent access to data objects transcending storage limitations in a non-relational data store | |
| US11500833B2 (en) | Archiving accelerator-only database tables | |
| US20230164208A1 (en) | Method and system for enforcing governance across multiple content repositories using a content broker | |
| US10853349B2 (en) | Event based analytics database synchronization | |
| US10671636B2 (en) | In-memory DB connection support type scheduling method and system for real-time big data analysis in distributed computing environment | |
| US11321374B2 (en) | External storage of unstructured database objects | |
| US11093477B1 (en) | Multiple source database system consolidation | |
| US20150370649A1 (en) | Sending a Request to a Management Service | |
| CN110347654B (en) | Method and device for online cluster characteristics | |
| US20230077698A1 (en) | Interspersed message batching in a database system | |
| US9537941B2 (en) | Method and system for verifying quality of server | |
| US20220374810A1 (en) | Accelerating outlier prediction of performance metrics in performance managers deployed in new computing environments | |
| US10628416B2 (en) | Enhanced database query processing | |
| US11336739B1 (en) | Intent-based allocation of database connections |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200207 Address after: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Applicant after: HUAWEI TECHNOLOGIES Co.,Ltd. Address before: 210012 HUAWEI Nanjing base, 101 software Avenue, Yuhuatai District, Jiangsu, Nanjing Applicant before: Huawei Technologies Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |