CN108234551A - A kind of data processing method and device - Google Patents
A kind of data processing method and device Download PDFInfo
- Publication number
- CN108234551A CN108234551A CN201611159311.8A CN201611159311A CN108234551A CN 108234551 A CN108234551 A CN 108234551A CN 201611159311 A CN201611159311 A CN 201611159311A CN 108234551 A CN108234551 A CN 108234551A
- Authority
- CN
- China
- Prior art keywords
- storage
- cluster
- data
- resource
- identity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/146—Markers for unambiguous identification of a particular session, e.g. session cookie or URL-encoding
Abstract
The embodiment of the invention discloses a kind of data processing method and device, the method includes：Acquisition currently needs the storage performance grade of the first data stored；According to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established, corresponding first cluster identity of storage performance grade of the first data is obtained；Storage resource is dispatched in the corresponding storage cluster of the first cluster identity, for storing the first data, the stock number of storage resource and the data volume of the first data match storage resource.Using the embodiment of the present invention, can meet the needs of different data is to storage performance, improve the reliability of data storage.
Description
Technical field
The present invention relates to Internet technical field more particularly to a kind of data processing method and devices.
Background technology
Cloud storage is one and is stored and managed the cloud computing system for core with data, and can be specifically should by cluster
With functions such as, network technology or distributed file systems, the storage cluster in network is gathered into collaboration by application software
Work, the common system that data storage and Operational Visit function are externally provided.But a cloud computing system usually manages
A variety of data, different data are different to the performance requirement of storage resource, and traditional cloud storage system will not store
Cluster is divided according to the difference of storage performance, can not meet performance requirement of the different data to storage resource, reduces storage
Reliability.
Invention content
Technical problem to be solved of the embodiment of the present invention is, provides a kind of data processing method and device, can meet
Demand of the different data to storage performance improves the reliability of data storage.
In order to solve the above-mentioned technical problem, an embodiment of the present invention provides a kind of data processing method, the method includes：
Acquisition currently needs the storage performance grade of the first data stored；
According to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established, described first is obtained
Corresponding first cluster identity of storage performance grade of data；
Storage resource is dispatched in the corresponding storage cluster of first cluster identity, the storage resource is used to store institute
The first data are stated, the stock number of the storage resource and the data volume of first data match.
Correspondingly, the embodiment of the present invention additionally provides a kind of data processing equipment, and described device includes：
Performance rate acquiring unit, for obtaining the storage performance grade of the first data for currently needing to store；
Cluster identity acquiring unit, for according to the storage performance grade and the cluster identity of storage cluster pre-established
Correspondence obtains corresponding first cluster identity of storage performance grade of first data；
Storage resource scheduling unit, for dispatching storage resource in the corresponding storage cluster of first cluster identity,
The storage resource is used to store first data, the data volume phase of the stock number of the storage resource and first data
Matching.
Implement the embodiment of the present invention, the storage performance grade of the first data stored is currently needed by obtaining, according to pre-
The storage performance grade and the correspondence of the cluster identity of storage cluster first established obtain the storage performance grade of the first data
Corresponding first cluster identity, dispatches storage resource in the corresponding storage cluster of the first cluster identity, and storage resource is used to deposit
The first data are stored up, can meet the needs of different data is to storage performance, improve the reliability of data storage.
Description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention, for those of ordinary skill in the art, without creative efforts, can be with
Other attached drawings are obtained according to these attached drawings；
Fig. 1 is the configuration diagram of a kind of data processing system provided in the embodiment of the present invention；
Fig. 2 is the flow diagram of a kind of data processing method provided in the embodiment of the present invention；
Fig. 3 is the structure diagram of a kind of data processing equipment provided in the embodiment of the present invention；
Fig. 4 is the structure diagram of a kind of controller provided in the embodiment of the present invention；
Fig. 5 is the schematic diagram of a kind of configuration file provided in the embodiment of the present invention；
Fig. 6 is the schematic diagram of a kind of configuration file provided in another embodiment of the present invention；
Fig. 7 is the schematic diagram of the details of a kind of configuration storage rear end provided in the embodiment of the present invention；
Fig. 8 is the schematic diagram of a kind of configuration volume type provided in the embodiment of the present invention；
Fig. 9 a are a kind of schematic diagrames provided in the embodiment of the present invention；
Fig. 9 b are a kind of schematic diagrames provided in another embodiment of the present invention；
Figure 10 is the schematic diagram of a kind of establishment volume provided in the embodiment of the present invention；
Figure 11 is that provided in the embodiment of the present invention a kind of is wrapping with the schematic diagram for being downloaded to virtual machine.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other without creative efforts
Embodiment shall fall within the protection scope of the present invention.
An embodiment of the present invention provides a kind of data processing method, controller can obtain the first number for currently needing to store
According to storage performance grade, according to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established, obtain
Corresponding first cluster identity of storage performance grade of the first data is taken, is dispatched in the corresponding storage cluster of the first cluster identity
Storage resource, storage resource can meet the needs of different data is to storage performance, improve data storage for storing the first data
Reliability.
Illustratively, controller can be the cloud control in the basic clouds of OpenStack (the cloud computing management platform increased income)
Device or cluster controller etc., are not limited specifically by the embodiment of the present application.
Based on above-mentioned principle, the embodiment of the invention discloses a kind of configuration diagrams of data processing system shown in FIG. 1.
As shown in Figure 1, the data processing system can include at least one controller, Ceph (expansible memory space) cluster 1, Ceph
Cluster 2 and virtual machine (Kernel-based Virtual Machine, KVM) cluster based on kernel, controller can divide
Not and Ceph clusters 1, Ceph clusters 2 and KVM clusters establish communication connection, and Ceph clusters 1 can be established with KVM clusters and be communicated
Connection, Ceph clusters 2 can be established with KVM clusters and be communicated to connect.
Wherein, a KVM cluster and at least two Ceph clusters can be included in the Region of an OpenStack.One
A Ceph clusters can include at least one memory node, and KVM clusters can include at least one virtual machine.Controller can be right
Said one KVM clusters and at least two Ceph clusters are managed.Controller can be respectively in Ceph clusters 1 and Ceph clusters
The volume of cloud storage is created in 2, volume refers to the storage resource that stock number is specified stock number, the volume created in Ceph clusters 1
With the volume that is created in Ceph clusters 2 can simultaneously carry in KVM clusters on same virtual machine, can also carry give KVM collection
Different virtual machine in group.
Based on above-mentioned principle, the embodiment of the invention discloses a kind of configuration diagrams of data processing method shown in Fig. 2.
As shown in Fig. 2, the data processing method at least may comprise steps of：
S201, acquisition currently need the storage performance grade of the first data stored.
Controller can obtain the storage performance grade of the first data for currently needing to store, such as the first data are regarded for sound
Frequency during either instant communication data according to since audio, video data or instant communication data are higher to real-time demand, then should
First data demand high-performance stores, then the storage performance of the first data is higher ranked, when for another example the first data are web data,
Since web data is relatively low to real-time demand, which can carry out the storage of generic storage, then first data
Energy grade is relatively low.Illustratively, controller can be controller shown in FIG. 1.
Wherein, storage performance grade includes at least：For first storage performance with the first storage speed storage data etc.
Grade and the second storage performance grade for storing data with the second storage speed.It should be noted that the embodiment of the present invention
Middle storage performance grade is including but not limited to two storage performance grades, such as controller is based on demand of the data to real-time
The data type of data is configured to the first data type, the second data type and third data type, the first data type
Data it is higher to real-time demand, then the storage performance grade of the data of the first data type can be for first storage
Speed stores the first storage performance grade of data；The data of second data type with respect to the first data type data to real-time
Property demand it is relatively low, then the storage performance grade of the data of the second data type can be for the second storage speed store data
The second storage performance grade；The data of third data type are relatively low to real-time demand with respect to the data of the second data type,
Then the storage performance grade of the data of third data type can be to be stored for storing the third of data with third storage speed
Performance rate, wherein the first storage speed can be more than the second storage speed, the second storage speed can be more than third storage speed
Degree.
Optionally, controller can receive the storage resource acquisition request of virtual machine transmission, and storage resource obtains request and takes
Storage performance grade with the first data, according to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established
Relationship obtains corresponding first cluster identity of storage performance grade of the first data, in the corresponding storage collection of the first cluster identity
Storage resource is dispatched, and storage resource is sent to virtual machine in group, so that virtual machine stores the first data using storage resource.
Illustratively, storage cluster can be the Ceph clusters in Fig. 1, such as Ceph clusters 1 or Ceph clusters 2.
In the specific implementation, controller can pre-establish storage performance grade and the corresponding of the cluster identity of storage cluster is closed
System, such as controller are configured with two storage clusters, respectively the first storage cluster and the second storage cluster, wherein the first storage
The storage performance of cluster is higher than the storage performance of the second storage cluster, then controller can establish the first storage performance grade and the
The correspondence of the cluster identity of one storage cluster, and establish the second storage performance grade and the cluster identity of the second storage cluster
Correspondence.Wherein, the first storage performance grade is used to store data with the first storage speed, and the second storage performance grade is used
In storing data with the second storage speed, the first storage speed is more than the second storage speed, then the first storage performance grade is higher than
Second storage performance grade.It should be noted that controller can wrap each storage cluster according to the needs of application scenarios
The control node contained is updated, so as to change the storage performance grade of each storage cluster, specifically not by the embodiment of the present invention
Limitation.The cluster identity of storage cluster can be used for the unique mark storage cluster, and cluster identity can include storage cluster
Cluster name or the cluster number etc. to storage cluster configuration, such as the cluster identity of the first storage cluster is Ceph clusters
1, the cluster identity of the second storage cluster is Ceph clusters 2.
In addition, when there are the first data for needing to store in a KVM in KVM clusters, which can obtain the first number
According to storage performance grade, and generate about the first data storage resource obtain request, wherein the storage resource obtain request
The storage performance grade of the first data is carried, which can obtain request by the storage resource and be sent to controller.Controller root
According to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established, the storage performance of the first data is obtained
Corresponding first cluster identity of grade dispatches storage resource in the corresponding storage cluster of the first cluster identity, and storage is provided
Source is sent to the virtual machine, so that the virtual machine stores the first data using the storage resource that above-mentioned scheduling obtains.
S202 according to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established, obtains the
Corresponding first cluster identity of storage performance grade of one data.
Specifically, after controller acquisition currently needs the storage performance grade of the first data stored, it can be according to pre-
The storage performance grade and the correspondence of the cluster identity of storage cluster first established obtain the storage performance grade of the first data
Corresponding first cluster identity.For example, the first storage performance grade corresponds to the cluster identity of the first storage cluster, the second storage
Can grade correspond to the cluster identity of the second storage cluster, then controller gets the storage performance grades of the first data and is deposited for first
After storing up performance rate, corresponding first cluster identity of storage performance grade that can obtain the first data is the first storage cluster
Cluster identity；Or controller gets the storage performance grades of the first data as after the second storage performance grade, it can be with
Corresponding first cluster identity of storage performance grade for obtaining the first data is the cluster identity of the second storage cluster.
Optionally, controller is closed according to the storage performance grade pre-established and the corresponding of the cluster identity of storage cluster
It is that before corresponding first cluster identity of storage performance grade for obtaining the first data, matching for each storage cluster can be generated
File is put, configuration file can include the cluster identity and its attribute information of storage cluster, and attribute information includes each user couple
The storage that the facility information and storage cluster for the storage device that the control authority of storage cluster, storage cluster are included are included
Resource summation.
Illustratively, when there are when two storage clusters (such as Ceph clusters 1 and Ceph clusters 2), controller can match
The catalogue ceph1 of Ceph clusters 1 is put, which is used to place the configuration file of Ceph clusters 1.Catalogue ceph1 can following institute
Show：
[root@DEVSSTEST-CON1~] #cd/etc/ceph
ceph1/cephbak/
In addition, the configuration file of Ceph clusters 1 can be with as shown in figure 5, wherein cinder, glance or root can be with tables
Show user ,-rw------- or-rw-r--r-- can represent the permission of different user.For example, the corresponding permissions of user root
For-rw-------, for representing that the configuration file of Ceph clusters 1 is readable and writeable, i.e. user for user root
Root can read the configuration file of Ceph clusters 1, can also change the configuration file of Ceph clusters 1.For another example, user cinder
Corresponding permission be-rw-r--r--, for represent the configuration file of Ceph clusters 1 be for user cinder it is readable and
Not writeable, i.e. user cinder can read the configuration file of Ceph clusters 1, but cannot change the configuration of Ceph clusters 1
File.For another example, the corresponding permissions of user glance are-rw-r--r--, for represent the configuration files of Ceph clusters 1 for
It is readable and not writeable for the glance of family, i.e., user glance can read the configuration file of Ceph clusters 1, but cannot
Change the configuration file of Ceph clusters 1.Controller is necessary to ensure that in configuration file user and its to configuration in the embodiment of the present invention
The permission of file is correct, correctly to run Ceph clusters 1.Illustratively, mon_host=10.125.224.82,
10.125.224.83,10.125.224.84 can represent the IP address of Ceph clusters 1 for 10.125.224.82,
10.125.224.83 10.125.224.84.Fsid=26a79278-9104-4e72-9343-9a369395a7d5 can be with
Represent the cluster identity of Ceph clusters 1.
Further, the catalogue ceph2 of Ceph clusters 2 can also be configured in controller, which is used to place Ceph clusters 2
Configuration file.Catalogue ceph2 can be as follows：
[root@DEVSSTEST-CON1~] #cd/etc/ceph
Ceph2/cephbak/
In addition, the configuration file of Ceph clusters 2 can be with as shown in fig. 6, wherein cinder, glance or root can be with tables
Show user ,-rw------- or-rw-r--r-- can represent the permission of different user.For example, the corresponding permissions of user root
For-rw-------, for representing that the configuration file of Ceph clusters 2 is readable and writeable, i.e. user for user root
Root can read the configuration file of Ceph clusters 2, can also change the configuration file of Ceph clusters 2.For another example, user cinder
Corresponding permission be-rw-r--r--, for represent the configuration file of Ceph clusters 2 be for user cinder it is readable and
Not writeable, i.e. user cinder can read the configuration file of Ceph clusters 2, but cannot change the configuration of Ceph clusters 2
File.For another example, the corresponding permissions of user glance are-rw-r--r--, for represent the configuration files of Ceph clusters 2 for
It is readable and not writeable for the glance of family, i.e., user glance can read the configuration file of Ceph clusters 2, but cannot
Change the configuration file of Ceph clusters 2.Controller is necessary to ensure that in configuration file user and its to configuration in the embodiment of the present invention
The permission of file is correct, correctly to run Ceph clusters 2.Illustratively, mon_host=10.125.224.13,
10.125.224.19,10.125.224.21 can represent the IP address of Ceph clusters 2 for 10.125.224.13,
10.125.224.19 10.125.224.21.Fsid=6260a31d-137f-4e54-bb51-a67065510d8f can be with
Represent the cluster identity of Ceph clusters 2.
Optionally, controller scheduling resource amount can provide in the first cluster for the first storage of the first default stock number
Source, and scheduling resource amount is the second storage resource of the second default stock number in the second cluster, determines the first storage resource
Resource type is matched with the cluster identity of the first cluster, the resource type of the second storage resource and the cluster identity of the second cluster
Match, the first storage resource and the second storage resource are sent to virtual machine, so that virtual machine obtains the storage performance of the first data
Grade, virtual machine obtain first according to the correspondence of the storage performance grade pre-established and the cluster identity of storage cluster
Corresponding first cluster identity of storage performance grade of data, virtual machine determine to match with the first cluster identity resources-type
Type, virtual machine store the first data using the storage resource indicated by resource type.
For example, cinder.conf files can be configured in controller, and cinder.conf files are used to that each deposit to be configured
The corresponding storage device of accumulation, such as enabled_backends=cephdriver-1, cephdriver-2.Wherein, it deposits
It can be storage rear end, the i.e. corresponding entity device of storage cluster to store up equipment.The detailed of each storage device can be configured in controller
Thin information, such as the quantity of memory node that is included of the device identification of storage device, storage device or each memory node
Node identification etc., illustratively, can be as follows：
volume_driver、volume_backend_name、rbd_pool、rbd_ceph_conf、rbd_flatten_
volume_from_snapshot、rbd_max_clone_depth、rbd_store_chunk_size、rados_connect_
timeout、glance_api_version、rbd_user、rbd_secret_uuid.Controller configuration storage device it is detailed
Thin information can be as shown with 7.
In addition, controller can state the type of volume in block storage, as shown in figure 8, controller is in the first storage cluster
The type of middle statement volume is ssd, and the type that volume is stated in the second storage cluster is all.Due to the storage of the first storage cluster
Performance is higher than the storage performance of the second storage cluster, then the internal memory performance for the volume that type is ssd depositing higher than the volume that type is all
Store up performance.
In addition, controller can also generate the correspondence of the device identification of the type and storage device of volume, the type of volume
Correspondence with the device identification of storage device can be as shown in Fig. 9 a and 9b.
Further, controller scheduling resource amount can provide in the first cluster for the first storage of the first default stock number
Source, and scheduling resource amount is the second storage resource of the second default stock number in the second cluster.With schematic diagram shown in Fig. 10
For, controller can create the first volume in Ceph clusters 1, and volume Two, the wherein first volume and are created in Ceph clusters 2
The type of volume two differs.By taking the schematic diagram shown in Figure 11 as an example, the first volume that controller can also obtain above-mentioned establishment and
Volume Two is mounted to same KVM, so that the KVM can use the number of the different storage performance demands of different types of volume storage
According to meet demand of the data of different storage performance demands to storage resource.
S203 dispatches storage resource in the corresponding storage cluster of the first cluster identity.
Controller obtains the according to the correspondence of the storage performance grade pre-established and the cluster identity of storage cluster
It, can be in the corresponding storage cluster of the first cluster identity after corresponding first cluster identity of storage performance grade of one data
Storage resource is dispatched, which can be used for storing the first data, and the wherein stock number of storage resource can be with the first number
According to data it is flux matched, such as storage resource stock number can be more than or equal to the first data data volume, so as to complete
Store the first data.
Optionally, controller can also obtain the storage performance grade of the second data for currently needing to read, according to advance
The correspondence of the storage performance grade of foundation and the cluster identity of storage cluster obtains the storage performance grade pair of the second data
The second cluster identity answered reads the second data in the corresponding storage cluster of the second cluster identity.For example, controller needs to read
When taking the second data, controller can determine the storage cluster of the second data of storage according to the storage performance grade of the second data,
I.e. controller can obtain second according to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established
Corresponding second cluster identity of storage performance grade of data reads the second number in the corresponding storage cluster of the second cluster identity
According to.
In the embodiment of the present invention, acquisition currently needs the storage performance grade of the first data stored, according to pre-establishing
Storage performance grade and storage cluster cluster identity correspondence, obtain the first data storage performance grade it is corresponding
First cluster identity dispatches storage resource in the corresponding storage cluster of the first cluster identity, and storage resource is for storage first
Data can meet the needs of different data is to storage performance, improve the reliability of data storage.
Fig. 3 is referred to, Fig. 3 is the structure diagram of a kind of data processing equipment provided in the embodiment of the present invention, described
Data processing equipment can be used for implementing to combine the part or all of step in embodiment of the method shown in Fig. 2, as shown in the figure originally
Data processing equipment in embodiment can at least include performance rate acquiring unit 301, cluster identity acquiring unit 302 and
Storage resource scheduling unit 303, wherein：
Performance rate acquiring unit 301, for obtaining the storage performance grade of the first data for currently needing to store.
Cluster identity acquiring unit 302, for according to the storage performance grade and the cluster mark of storage cluster pre-established
The correspondence of knowledge obtains corresponding first cluster identity of storage performance grade of first data.
Storage resource scheduling unit 303, for the scheduling storage money in the corresponding storage cluster of first cluster identity
Source, the storage resource are used to store first data, the data of the stock number of the storage resource and first data
Amount matches.
Optionally, the storage performance grade includes at least：For with the first storage of the first storage speed storage data
Performance rate and the second storage performance grade for storing data with the second storage speed.
Optionally, the performance rate acquiring unit 301 is additionally operable to obtain the storage of the second data for currently needing to read
Performance rate.
The cluster identity acquiring unit 302 is additionally operable to according to the storage performance grade and storage cluster pre-established
The correspondence of cluster identity obtains corresponding second cluster identity of storage performance grade of second data.
Further, the data processing equipment in the embodiment of the present invention can also include：
Data-reading unit 304, for reading second number in the corresponding storage cluster of second cluster identity
According to.
Optionally, the data processing equipment in the embodiment of the present invention can also include：
Configuration file generation unit 305, for the cluster identity acquiring unit 302 according to the storage performance pre-established
The correspondence of grade and the cluster identity of storage cluster obtains corresponding first collection of storage performance grade of first data
Before group identification, the configuration file of each storage cluster is generated, the configuration file includes the cluster of the storage cluster
Mark and its attribute information, the attribute information include control authority of each user to the storage cluster, the storage collection
The storage resource summation that the facility information and the storage cluster for the storage device that group is included are included.
Optionally, the performance rate acquiring unit 301, the storage resource for receiving virtual machine transmission obtain request,
The storage resource obtains the storage performance grade that request carries first data.
Further, the data processing equipment in the embodiment of the present invention can also include：
Storage resource transmitting element 306 corresponds to for the storage resource scheduling unit 303 in first cluster identity
Storage cluster in scheduling storage resource after, the storage resource is sent to the virtual machine, so that the virtual machine makes
First data are stored with the storage resource.
Optionally, it is default for first to be additionally operable to the scheduling resource amount in the first cluster for the storage resource scheduling unit 303
First storage resource of stock number, and scheduling resource amount is the second storage resource of the second default stock number in the second cluster.
Further, the data processing equipment in the embodiment of the present invention can also include：
Determination unit 307, for determining the cluster mark of the resource type of first storage resource and first cluster
Know matching, the resource type of second storage resource is matched with the cluster identity of second cluster.
Storage resource transmitting element 306, for first storage resource and second storage resource to be sent to void
Plan machine, so that the virtual machine obtains the storage performance grade of first data, the virtual machine is deposited according to what is pre-established
The correspondence of the cluster identity of performance rate and storage cluster is stored up, the storage performance grade for obtaining first data is corresponding
First cluster identity, the virtual machine determine the resource type to match with first cluster identity, and the virtual machine uses
Storage resource indicated by the resource type stores first data.
In the embodiment of the present invention, performance rate acquiring unit 301 obtains the storage of the first data for currently needing to store
Can grade, cluster identity acquiring unit 302 is according to pair of the storage performance grade pre-established and the cluster identity of storage cluster
It should be related to, obtain corresponding first cluster identity of storage performance grade of the first data, storage resource scheduling unit 303 is first
Storage resource is dispatched in the corresponding storage cluster of cluster identity, can meet the needs of different data is to storage performance, improves data
The reliability of storage.
Fig. 4 is referred to, Fig. 4 is the structure diagram of a kind of controller provided in the embodiment of the present invention, and the present invention is implemented
The controller that example provides can be used for implementing the method that the above-mentioned embodiment of the present invention shown in Fig. 2 is realized, for convenience of description, only
It shows and is not disclosed with the relevant part of the embodiment of the present invention, particular technique details, please refer to the present invention shown in Fig. 2 and implement
Example.
As shown in figure 4, the controller includes：At least one processor 401, such as CPU, at least one input unit 403,
At least one output device 404, memory 405, at least one communication bus 402.Wherein, communication bus 402 is used to implement this
Connection communication between a little components.Wherein, input unit 403 is specifically as follows network interface etc., between storage cluster
It interacts.Wherein, output device 404 is specifically as follows network interface etc., for being interacted between storage cluster.Its
In, memory 405 may include high-speed RAM memory, it is also possible to further include non-labile memory, for example, at least a magnetic
Disk storage, specifically for correspondence of cluster identity of storage storage performance grade and storage cluster etc..Memory 405 can
Choosing can include at least one storage device for being located remotely from aforementioned processor 401.Batch processing generation is stored in memory 405
Code, and processor 401, input unit 403 and output device 404 call the program code stored in memory 405, for holding
Row is following to be operated：
Processor 401 obtains the storage performance grade of the first data for currently needing to store.
Processor 401 is obtained according to the correspondence of the storage performance grade pre-established and the cluster identity of storage cluster
Take corresponding first cluster identity of storage performance grade of first data.
Processor 401 dispatches storage resource, the storage resource in the corresponding storage cluster of first cluster identity
For storing first data, the stock number of the storage resource and the data volume of first data match.
Optionally, the storage performance grade includes at least：For with the first storage of the first storage speed storage data
Performance rate and the second storage performance grade for storing data with the second storage speed.
Optionally, following operation can also be performed in processor 401：
Processor 401 obtains the storage performance grade of the second data for currently needing to read.
Processor 401 is obtained according to the correspondence of the storage performance grade pre-established and the cluster identity of storage cluster
Take corresponding second cluster identity of storage performance grade of second data.
Processor 401 reads second data in the corresponding storage cluster of second cluster identity.
Optionally, processor 401 is according to the storage performance grade pre-established and the correspondence of the cluster identity of storage cluster
Before corresponding first cluster identity of storage performance grade for obtaining first data, following operation can also be performed in relationship：
The configuration file of each storage cluster is generated, the configuration file includes the cluster identity of the storage cluster
And its attribute information, the attribute information include control authority of each user to the storage cluster, storage cluster institute
Comprising the facility information of storage device and the storage resource summation that is included of the storage cluster.
Optionally, processor 401 obtains the storage performance grade of the first data for currently needing to store, and is specifically as follows：
Input unit 403 receives the storage resource that virtual machine is sent and obtains request, and the storage resource obtains request and carries
The storage performance grade of first data.
After processor 401 dispatches storage resource in the corresponding storage cluster of first cluster identity, it can also hold
Row is following to be operated：
The storage resource is sent to the virtual machine by output device 404, so that the virtual machine uses the storage
Resource stores first data.
Optionally, following operation can also be performed in processor 401：
The scheduling resource amount in the first cluster of processor 401 is the first storage resource of the first default stock number, and the
Scheduling resource amount is the second storage resource of the second default stock number in two clusters.
Processor 401 determines that the resource type of first storage resource is matched with the cluster identity of first cluster,
The resource type of second storage resource is matched with the cluster identity of second cluster.
First storage resource and second storage resource are sent to virtual machine by output device 404, so that described
Virtual machine obtains the storage performance grade of first data, and the virtual machine is according to the storage performance grade pre-established and deposits
The correspondence of the cluster identity of accumulation obtains corresponding first cluster identity of storage performance grade of first data,
The virtual machine determines the resource type to match with first cluster identity, and the virtual machine uses the resource type institute
The storage resource of instruction stores first data.
Specifically, the controller introduced in the embodiment of the present invention can to implement the present invention combine Fig. 2 introductions method
Part or all of flow in embodiment.
One of ordinary skill in the art will appreciate that realizing all or part of flow in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the program can be stored in computer read/write memory medium
In, the program is when being executed, it may include such as the flow of the embodiment of above-mentioned each method.Wherein, the storage medium can be magnetic
Dish, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access
Memory, RAM) etc..
The above disclosure is only the preferred embodiments of the present invention, cannot limit the right model of the present invention with this certainly
It encloses, therefore equivalent variations made according to the claims of the present invention, is still within the scope of the present invention.
Claims (12)
1. a kind of data processing method, which is characterized in that the method includes：
Acquisition currently needs the storage performance grade of the first data stored；
According to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established, first data are obtained
Corresponding first cluster identity of storage performance grade；
Dispatch storage resource in the corresponding storage cluster of first cluster identity, the storage resource is for storing described the
One data, the stock number of the storage resource and the data volume of first data match.
2. the method as described in claim 1, which is characterized in that the storage performance grade includes at least：For being deposited with first
Store up the first storage performance grade of speed storage data and for the second storage performance of the second storage speed storage data
Grade.
3. the method as described in claim 1, which is characterized in that the method further includes：
Acquisition currently needs the storage performance grade of the second data read；
According to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established, second data are obtained
Corresponding second cluster identity of storage performance grade；
Second data are read in the corresponding storage cluster of second cluster identity.
4. the method as described in claim 1, which is characterized in that the storage performance grade and storage that the basis pre-establishes collect
The correspondence of the cluster identity of group, before corresponding first cluster identity of storage performance grade for obtaining first data,
It further includes：
Generate the configuration file of each storage cluster, the configuration file include the storage cluster cluster identity and its
Attribute information, the attribute information include each user and the control authority of the storage cluster, the storage cluster are included
The facility information of storage device and the storage resource summation that is included of the storage cluster.
5. the method as described in claim 1, which is characterized in that the acquisition currently needs the storage of the first data stored
Energy grade, including：
It receives the storage resource that virtual machine is sent and obtains request, the storage resource obtains request and carries depositing for first data
Store up performance rate；
It is described after scheduling storage resource, to be further included in the corresponding storage cluster of first cluster identity：
The storage resource is sent to the virtual machine, so that the virtual machine stores described first using the storage resource
Data.
6. the method as described in claim 1, which is characterized in that the method further includes：
Scheduling resource amount is the first storage resource of the first default stock number in the first cluster, and money is dispatched in the second cluster
Second storage resource of the source amount for the second default stock number；
Determine that the resource type of first storage resource is matched with the cluster identity of first cluster, the second storage money
The resource type in source is matched with the cluster identity of second cluster；
First storage resource and second storage resource are sent to virtual machine, so that the virtual machine obtains described the
The storage performance grade of one data, the virtual machine is according to the storage performance grade pre-established and the cluster identity of storage cluster
Correspondence, obtain corresponding first cluster identities of storage performance grade of first data, the virtual machine determine with
The resource type that first cluster identity matches, the virtual machine are deposited using the storage resource indicated by the resource type
Store up first data.
7. a kind of data processing equipment, which is characterized in that described device includes：
Performance rate acquiring unit, for obtaining the storage performance grade of the first data for currently needing to store；
Cluster identity acquiring unit, for according to the storage performance grade and the correspondence of the cluster identity of storage cluster pre-established
Relationship obtains corresponding first cluster identity of storage performance grade of first data；
Storage resource scheduling unit, it is described for dispatching storage resource in the corresponding storage cluster of first cluster identity
Storage resource is used to store first data, the data volume phase of the stock number of the storage resource and first data
Match.
8. device as claimed in claim 7, which is characterized in that the storage performance grade includes at least：For being deposited with first
Store up the first storage performance grade of speed storage data and for the second storage performance of the second storage speed storage data
Grade.
9. device as claimed in claim 7, which is characterized in that
The performance rate acquiring unit is additionally operable to obtain the storage performance grade of the second data for currently needing to read；
The cluster identity acquiring unit is additionally operable to according to the storage performance grade and the cluster identity of storage cluster pre-established
Correspondence, obtain corresponding second cluster identities of storage performance grade of second data；
Described device further includes：
Data-reading unit, for reading second data in the corresponding storage cluster of second cluster identity.
10. device as claimed in claim 7, which is characterized in that described device further includes：
Configuration file generation unit, for the cluster identity acquiring unit according to the storage performance grade pre-established and storage
The correspondence of the cluster identity of cluster, obtain first data corresponding first cluster identity of storage performance grade it
Before, generate the configuration file of each storage cluster, the configuration file include the storage cluster cluster identity and its
Attribute information, the attribute information include each user and the control authority of the storage cluster, the storage cluster are included
The facility information of storage device and the storage resource summation that is included of the storage cluster.
11. device as claimed in claim 7, which is characterized in that
The performance rate acquiring unit, the storage resource for receiving virtual machine transmission obtain request, and the storage resource obtains
Request is taken to carry the storage performance grade of first data；
Described device further includes：
Storage resource transmitting element, for the storage resource scheduling unit in the corresponding storage cluster of first cluster identity
After middle scheduling storage resource, the storage resource is sent to the virtual machine, so that the virtual machine uses the storage
Resource stores first data.
12. device as claimed in claim 7, which is characterized in that
The storage resource scheduling unit, the scheduling resource amount in the first cluster that is additionally operable to are deposited for the first of the first default stock number
Resource is stored up, and scheduling resource amount is the second storage resource of the second default stock number in the second cluster；
Described device further includes：
Determination unit, the resource type for determining first storage resource are matched with the cluster identity of first cluster,
The resource type of second storage resource is matched with the cluster identity of second cluster；
Storage resource transmitting element, for first storage resource and second storage resource to be sent to virtual machine, with
The virtual machine is made to obtain the storage performance grade of first data, the virtual machine is according to storage performance pre-established etc.
The correspondence of grade and the cluster identity of storage cluster obtains corresponding first cluster of storage performance grade of first data
Mark, the virtual machine determine the resource type to match with first cluster identity, and the virtual machine uses the resource
Storage resource indicated by type stores first data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611159311.8A CN108234551B (en) | 2016-12-15 | 2016-12-15 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611159311.8A CN108234551B (en) | 2016-12-15 | 2016-12-15 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108234551A true CN108234551A (en) | 2018-06-29 |
| CN108234551B CN108234551B (en) | 2021-06-25 |
Family
ID=62651353
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611159311.8A Active CN108234551B (en) | 2016-12-15 | 2016-12-15 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108234551B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110166429A (en) * | 2019-04-12 | 2019-08-23 | 深圳壹账通智能科技有限公司 | Data package processing method, device, computer readable storage medium and server |
| CN110333931A (en) * | 2019-05-27 | 2019-10-15 | 北京迈格威科技有限公司 | The system of shared storage for training pattern |
| CN111290712A (en) * | 2020-01-22 | 2020-06-16 | 腾讯科技（深圳）有限公司 | Block device creating method and device, cloud computing management system and storage medium |
| CN114185488A (en) * | 2021-11-29 | 2022-03-15 | 广东财经大学 | Storage optimization method and system for big data cluster |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521063A (en) * | 2011-11-30 | 2012-06-27 | 广东电子工业研究院有限公司 | Shared storage method suitable for migration and fault tolerance of virtual machine |
| CN103500147A (en) * | 2013-09-27 | 2014-01-08 | 浪潮电子信息产业股份有限公司 | Embedded and layered storage method of PB-class cluster storage system |
| CN104283959A (en) * | 2014-10-14 | 2015-01-14 | 国云科技股份有限公司 | Performance-grading-based storage mechanism suitable for cloud platform |
| CN104536909A (en) * | 2014-12-09 | 2015-04-22 | 华为技术有限公司 | Memory management method, memory management device and memory device |
| EP2754078A4 (en) * | 2011-04-14 | 2015-12-02 | Complete Genomics Inc | Processing and analysis of complex nucleic acid sequence data |
| CN106095531A (en) * | 2016-06-08 | 2016-11-09 | 北京航空航天大学 | A kind of dispatching method of virtual machine loaded based on grade and physical machine in cloud platform |
-
2016
- 2016-12-15 CN CN201611159311.8A patent/CN108234551B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2754078A4 (en) * | 2011-04-14 | 2015-12-02 | Complete Genomics Inc | Processing and analysis of complex nucleic acid sequence data |
| CN102521063A (en) * | 2011-11-30 | 2012-06-27 | 广东电子工业研究院有限公司 | Shared storage method suitable for migration and fault tolerance of virtual machine |
| CN103500147A (en) * | 2013-09-27 | 2014-01-08 | 浪潮电子信息产业股份有限公司 | Embedded and layered storage method of PB-class cluster storage system |
| CN104283959A (en) * | 2014-10-14 | 2015-01-14 | 国云科技股份有限公司 | Performance-grading-based storage mechanism suitable for cloud platform |
| CN104536909A (en) * | 2014-12-09 | 2015-04-22 | 华为技术有限公司 | Memory management method, memory management device and memory device |
| CN106095531A (en) * | 2016-06-08 | 2016-11-09 | 北京航空航天大学 | A kind of dispatching method of virtual machine loaded based on grade and physical machine in cloud platform |
Non-Patent Citations (1)
| Title |
|---|
| 陆平: "《云计算基础架构及关键应用》", 30 April 2016 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110166429A (en) * | 2019-04-12 | 2019-08-23 | 深圳壹账通智能科技有限公司 | Data package processing method, device, computer readable storage medium and server |
| CN110166429B (en) * | 2019-04-12 | 2022-03-22 | 深圳壹账通智能科技有限公司 | Data packet processing method and device, computer readable storage medium and server |
| CN110333931A (en) * | 2019-05-27 | 2019-10-15 | 北京迈格威科技有限公司 | The system of shared storage for training pattern |
| CN111290712A (en) * | 2020-01-22 | 2020-06-16 | 腾讯科技（深圳）有限公司 | Block device creating method and device, cloud computing management system and storage medium |
| CN111290712B (en) * | 2020-01-22 | 2021-06-18 | 腾讯科技（深圳）有限公司 | Block device creating method and device, cloud computing management system and storage medium |
| CN114185488A (en) * | 2021-11-29 | 2022-03-15 | 广东财经大学 | Storage optimization method and system for big data cluster |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108234551B (en) | 2021-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110032447B (en) | Method and apparatus for allocating resources | |
| CN102255962B (en) | Distributive storage method, device and system | |
| TW202002587A (en) | Block chain-based data processing method and device | |
| US20160092119A1 (en) | Data migration between different types of storage systems | |
| US10235047B2 (en) | Memory management method, apparatus, and system | |
| US9405579B2 (en) | Seamless extension of local computing power | |
| US9459897B2 (en) | System and method for providing data analysis service in cloud environment | |
| CN104283959B (en) | A kind of memory mechanism based on grading performance suitable for cloud platform | |
| CN106201566A (en) | The rich big special hot upgrade method of software of profit and equipment | |
| CN108234551A (en) | A kind of data processing method and device | |
| CN107580011B (en) | Data sharing method and desktop cloud server | |
| CN107707622A (en) | A kind of method, apparatus and desktop cloud controller for accessing desktop cloud virtual machine | |
| CN106598692A (en) | Method for creating mirror image file in logical volume and starting virtual machine, and server | |
| US20170153909A1 (en) | Methods and Devices for Acquiring Data Using Virtual Machine and Host Machine | |
| US11307900B2 (en) | Adjustment of the number of central processing units to meet performance requirements of an I/O resource | |
| CN102611754A (en) | Management method of client in cloud storage based on iSCSI (internet small computer system interface) | |
| CN104601666A (en) | Log service method and cloud platform | |
| CN102664894B (en) | System and method for software provision based on cloud computing | |
| CN105224541B (en) | Uniqueness control method, information storage means and the device of data | |
| CN107908365A (en) | The method, apparatus and equipment of User space memory system data interaction | |
| CN113051102A (en) | File backup method, device, system, storage medium and computer equipment | |
| CN105426125B (en) | A kind of date storage method and device | |
| CN102833319A (en) | Web-based virtual box real-time migration method | |
| CN109088913A (en) | The method and load-balanced server of request data | |
| CN110703997A (en) | Cloud storage system, data storage method, device, equipment and computer medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |