CN112347036B - Inter-cloud migration method and device of cloud storage system - Google Patents
Inter-cloud migration method and device of cloud storage system Download PDFInfo
- Publication number
- CN112347036B CN112347036B CN201910735036.7A CN201910735036A CN112347036B CN 112347036 B CN112347036 B CN 112347036B CN 201910735036 A CN201910735036 A CN 201910735036A CN 112347036 B CN112347036 B CN 112347036B
- Authority
- CN
- China
- Prior art keywords
- migration
- scheduling service
- user
- user directory
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The CDM receives a migration request to generate an incidence relation between user directories before and after migration, searches the corresponding incidence relation between the user directories before and after migration according to the user directories carried by the main scheduling service request after receiving the main scheduling service request sent by a data sender, and returns the main scheduling service corresponding to CLUs before and after migration and the user directories before and after migration to the data sender according to the incidence relation between the user directories before and after migration. And the data sender carries the user directories before and after the migration respectively, requests the data nodes from the main scheduling service of the CLUs before and after the migration, and writes data to the data nodes after the migration. The invention only migrates the metadata, does not migrate the video data, can not produce the video reading and writing flow of large area because of migrating, store more flexibly, not limited by the storage node, extend the expansion capacity upper limit of the cabinet, hard disk, can not increase the hardware, save the space of the stander, computer lab and economic cost.
Description
Technical Field
The invention belongs to the technical field of cloud storage, and particularly relates to an inter-cloud migration method and device of a cloud storage system.
Background
With the rapid development of network technology and the gradual maturity of digitization, networking, high-definition and intellectualization of video monitoring, the video acquisition technology of a video monitoring front-end network camera (IPC) is also rapidly developed, from 10 ten thousand pixels at the early stage to 200 ten thousand and 500 ten thousand pixels at present, the definition of video is higher and higher, the information content in a video picture is richer and richer, and the requirement on storage capacity is higher and higher. At present, the most basic high definition is 720P, and video recording per hour can be compressed to about 3GB, and about 17T (17000G) in one month. Hundreds of paths or even thousands of paths of high-definition monitoring are performed in the monitoring system, huge video files generated by the monitoring system are rushed to a memory, and the storage problem of the security industry is increasingly highlighted under the conditions that the definition of security videos is continuously improved and the number of security monitoring paths is continuously increased.
Just as video monitoring is applied more and more widely, the Storage capacity is larger and larger, and the Storage mode is also shifted from the original centralized Storage to the current popular Cloud Direct Storage (CDS). As shown in fig. 1, in a CDS system,
with the continuous expansion of the monitoring service scale, for example, the storage time of videos and pictures is prolonged or the service access is increased by adding new IPCs, the capacity expansion of the existing cloud storage resources is needed at this time. The current storage capacity expansion mode includes increasing hard disks, expanding cabinets, and expanding Cloud Direct Virtual (CDV) storage nodes in a cluster unit (CLU), but when the number of the CDV storage nodes in the CLU reaches the upper limit of the system capacity, storage of key IPC point locations cannot be increased in the CLU, or capacity expansion cannot be performed on other key IPC point locations.
Under the above circumstances, one processing method in the prior art is to delete the original configuration and reconfigure the original configuration, but this processing will cause the original video to be lost, and the operation is cumbersome and not acceptable by the user. The other processing mode is to store another copy of the original video in a backup mode, so as to make the storage space free.
Disclosure of Invention
The invention aims to provide an inter-cloud migration method and an inter-cloud migration device for a cloud storage system, which are used for realizing the purpose of increasing the storage of key IPC user directories or expanding other key IPC user directories by dynamically migrating among cluster groups CLUs and overcoming the problems in the background art.
In order to achieve the purpose, the technical scheme of the invention is as follows:
an inter-cloud migration method of a cloud storage system, the cloud storage system comprising a metadata server CDM and a cluster group CLU, the inter-cloud migration method comprising:
CDM receives the migration request, generates the association relationship of the user directories before and after migration according to the migration request, and returns the user directories after migration to the data sender;
CDM receives a main scheduling service request sent by a data sender, and searches the corresponding user directory incidence relation before and after migration according to the user directory carried by the main scheduling service request;
and when the CDM judges that the user directory carried by the main scheduling service request is the migration, the main scheduling service corresponding to the CLUs before and after the migration and the user directories before and after the migration are returned to the data sender according to the incidence relation of the user directories before and after the migration, so that the data sender can conveniently carry the user directories before and after the migration respectively, the data node is requested to the main scheduling service of the CLUs before and after the migration, the data node returned by the CLU after the migration is set to be in a readable and writable state, and the data sender writes data to the readable and writable data node.
Preferably, the migration request includes an identifier of a data sender and a cloud storage user after migration, and the CDM receives the migration request and generates a user directory association relationship before and after migration according to the migration request, including:
receiving a migration request, searching a CLU after migration according to the cloud storage user after migration in the migration request, generating a user directory after migration, setting the state attribute of the user directory before migration to be a second state, setting the state attribute of the user directory after migration to be a third state, and generating an association relation between the user directory before migration and the user directory after migration.
Further, the CDM may determine that the user directory carried by the primary scheduling service request is migration, including:
and if the state attribute corresponding to the user directory carried by the main scheduling service request is a third state, judging that the user directory is migrated.
Further, the inter-cloud migration method of the cloud storage system further includes:
for the newly configured user directory, CDM generates user directory metadata, and the state attribute of the user directory is set to be a first state;
and if the state attribute corresponding to the user directory carried by the main scheduling service request is in the first state, judging that the state attribute is in a conventional working state, and directly returning the corresponding main scheduling service to the data sender.
Further, the inter-cloud migration method of the cloud storage system further includes:
CDM receives a main scheduling service request sent by a client, and searches the corresponding user directory incidence relation before and after migration according to a user directory carried by the main scheduling service request;
and the CDM returns the main scheduling service corresponding to the CLUs before and after the migration and the user directories before and after the migration to the client according to the incidence relation of the user directories before and after the migration, so that the client can conveniently carry the user directories before and after the migration respectively, request data nodes from the main scheduling service of the CLUs before and after the migration, and acquire data from the data nodes obtained by the request.
The invention also provides an inter-cloud migration device of a cloud storage system, wherein the cloud storage system comprises a metadata server CDM and a cluster CLU, and the inter-cloud migration device of the cloud storage system is applied to the CDM and comprises the following components:
the association module is used for receiving the migration request, generating the association relation of the user directories before and after the migration according to the migration request, and returning the user directories after the migration to the data sender;
the scheduling service module is used for receiving a main scheduling service request sent by a data sender, searching corresponding association relations between user directories before and after migration according to the user directories carried by the main scheduling service request, and when the user directories carried by the main scheduling service request are judged to be migration, returning the main scheduling service corresponding to CLUs before and after migration and the user directories before and after migration to the data sender according to the association relations between the user directories before and after migration, so that the data sender can conveniently carry the user directories before and after migration respectively, request data nodes to the main scheduling service of the CLUs before and after migration, set the data nodes returned by the CLUs after migration to be in a readable and writable state, and set the data nodes returned by the CLUs before migration to be in a readable and writable state, thereby writing data to the readable and writable data nodes by the data sender.
Preferably, the migration request includes an identifier of a data sender and a cloud storage user after migration, the association module receives the migration request, generates a user directory association relationship before and after migration according to the migration request, and executes the following operations:
receiving a migration request, searching a CLU after migration according to the cloud storage user after migration in the migration request, generating a user directory after migration, setting the state attribute of the user directory before migration to be a second state, setting the state attribute of the user directory after migration to be a third state, and generating an association relation between the user directory before migration and the user directory after migration.
Further, the scheduling service module judges that the user directory carried by the main scheduling service request is migration, and executes the following operations:
and if the state attribute corresponding to the user directory carried by the main scheduling service request is a third state, judging that the user directory is migrated.
Further, the association module is further configured to generate user directory metadata for the newly configured user directory, and set a state attribute of the user directory to be a first state;
the dispatch service module further performs the following operations:
and if the state attribute corresponding to the user directory carried by the main scheduling service request is in the first state, judging that the state attribute is in a conventional working state, and directly returning the corresponding main scheduling service to the data sender.
Further, the dispatch service module is further configured to:
receiving a main scheduling service request sent by a client, and searching a corresponding user directory incidence relation before and after migration according to a user directory carried by the main scheduling service request;
and returning the main scheduling service corresponding to the CLUs before and after the migration and the user directories before and after the migration to the client according to the association relationship of the user directories before and after the migration, so that the client can conveniently carry the user directories before and after the migration respectively, request data nodes from the main scheduling service of the CLUs before and after the migration, and acquire data from the data nodes obtained by the request.
According to the cloud migration method and device of the cloud storage system, the incidence relation between the user directories before and after migration is generated through the migration request, CDM searches the corresponding incidence relation between the user directories before and after migration according to the user directory carried by the main scheduling service request after receiving the main scheduling service request sent by the data sender, and returns the main scheduling service corresponding to the CLU before and after migration and the user directory before and after migration to the data sender according to the incidence relation between the user directories before and after migration when the user directory carried by the main scheduling service request is judged to be migrated. Therefore, a data sender carries user directories before and after migration respectively, requests data nodes from the main scheduling service of the CLUs before and after migration, sets the data nodes returned by the CLUs after migration to be in a readable and writable state, and sets the data nodes returned by the CLUs before migration to be in a readable and non-writable state, so that data can be written to the readable and writable data nodes. When the method is used for migration, only the metadata is migrated, the video data is not migrated, and large-area video read-write flow caused by migration is avoided. After migration, space is released, and key point location expansion and new point location storage can be immediately provided. The storage is more flexible, the capacity expansion capacity of the storage nodes, the expansion cabinet and the hard disk is not limited to the upper limit, hardware is not increased, and the space of the rack and the machine room and the economic cost are saved.
Drawings
FIG. 1 is a schematic structural diagram of a video surveillance cloud storage CDS system;
FIG. 2 is a schematic diagram of user directory storage in a CLU of the CDS system;
figure 3 is a schematic diagram of a prior art CDS conventional data storage process;
fig. 4 is a flowchart of an inter-cloud migration method of a cloud storage system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
The cloud migration method of the cloud storage system can be applied to the application environment shown in fig. 1. Fig. 1 illustrates an example of cloud storage of a video monitoring system, where a monitoring device on a video monitoring system side includes a network camera IPC, a video management server VM and a client, and a cloud storage CDS side includes a metadata server CDM, storage nodes CDV, and cluster groups CLU, each CLU including multiple CDVs.
The VM is a video management server and provides equipment access management, such as IPC (IPC access management (registration), live service, retrieval playback and other service function scheduling; CDM provides metadata cluster management service and received stream scheduling, mainly takes charge of electing a main scheduling service (Clu) in a cluster CLU, and for a standard protocol IPC, CDM is also responsible for scheduling received stream CDV and receiving IPC code stream; and the storage node CDV is deployed with a data node DN, provides scheduling service in the cluster service and is mainly responsible for functions of resource management, service access service and data storage. The cluster CLU represents a physical partition, and a plurality of CDVs in the CLU constitute one cluster group.
Although fig. 1 illustrates cloud storage of a video monitoring system, the application is also applicable to data storage of other business systems, and the business systems may also be banking business systems, business systems of e-commerce platforms, and the like.
In a cloud storage system CDS, a cluster group CLU is typically striped into individual user spaces, which are provided for final users USR, and a user directory is a basic unit for users divided from the USR. When the IPC is configured with cloud storage, CDM in CDS correspondingly generates a user directory, which is similar to the file folder of the IPC, and the video data of the IPC is written to the user directory. The user directory has a capacity attribute that is consistent with the size of the user-configured capacity. When IPC writes video, the video data are distributed on each CDV in the CLU in a balanced way, and the user directory capacity is shared on each CDV. As shown in fig. 2, for example, the CLU-1 includes CDV1, CDV2, and CDV3, and the user directory 1, the user directory 2, and the user directory 3 allocated to the user a are used and may respectively correspond to the IPC1, IPC2, and IPC3 of the user a.
As shown in fig. 3, the conventional normal data storage process of IPC in the video monitoring system is as follows:
(1) and CDM selects a scheduling service of CDV from the CLU cluster group as a main scheduling service.
And the master scheduling service (master Clu) is responsible for scheduling DN under the CLU cluster, and DN is responsible for selecting storage resource to write and record data. This embodiment CDM selects Clu1 as the primary Clu.
(2) And according to the configured storage resources, after the storage plan is started, the IPC carries the user directory and acquires the master scheduling service from the CDM.
In the CDS, CDM selects a scheduling service of a CDV from a CLU cluster group as a primary scheduling service, where the primary scheduling service is responsible for scheduling DN data nodes in CLU clusters to which the primary scheduling service belongs, and the DN is responsible for selecting storage resources to write and record data.
The present embodiment assumes that CDM selects the scheduling service Clu1 of CDV1 as the primary scheduling service. CDM analyzes user directory, then finds out the association relation between the user directory and CLU cluster group, and after finding out the corresponding relation, CDM feeds back the main scheduling service in the cluster group to IPC. The CDM embodiment returns Clu1 as the primary scheduling service.
(3) And reporting the capacity of the data node DN of each CDV in the cluster CLU corresponding to the user directory to the master scheduling service.
The data node DN of each CDV of the CLU cluster group reports the capacity to the master scheduling service, and the main purpose is to tell the master scheduling service that the DN is in an activated state and can be used. The main scheduling service Clu1 stores these DN lists in memory for subsequent IPC selection.
(4) IPC requests the data node DN from the master dispatch service for writing the video data.
(5) IPC sends the video stream storage directly to the selected DN.
Because the main scheduling service memory contains all data node DNs in the CLU cluster group, the main scheduling service selects the DNs according to the principle of storage path load balance and round-robin storage on the DNs. The selection of the DN by the master scheduling service according to the principle of load balancing of the number of storage paths on each DN and round robin storage is a mature technology in the field and is not described herein again.
The general idea of the application is that when the CDS storage capacity is insufficient, all or part of the user directory of a non-critical user in a CLU that needs capacity expansion can be migrated to other CLUs, for example, when the CLU-1 has insufficient capacity, and the CLU-2 has less storage data, the data storage of the IPC1 of the user a can be migrated to the CLU-2 for storage.
In an embodiment, as shown in fig. 4, a method for inter-cloud migration of a cloud storage system according to the present application includes:
In a video surveillance system, after IPC cloud storage is configured, CDM generates a user directory metadata, such as Usrdir, as shown in the following table:
| user directory | State attributes | Pre-migration user catalog | Post-migration user directory |
| Usrdir | 0 |
TABLE 1
For the newly-created user directory, the state attribute defaults to the first state, namely, the new state (0).
When a user directory is migrated from one CLU (CLU-1) to another CLU (CLU-2), a migration request needs to include the user directory before migration and the CLU after migration, i.e., it needs to know to which CLU the user directory before migration is migrated.
In this embodiment, the migration request includes an identifier of a data sender and a cloud storage user after migration, and a user directory association relationship before and after migration is generated in CDM, where the generation process is as follows:
receiving a migration request, searching a CLU after migration according to the cloud storage user after migration in the migration request, generating a user directory after migration, setting the state attribute of the user directory before migration to be a second state, setting the state attribute of the user directory after migration to be a third state, and generating an association relation between the user directory before migration and the user directory after migration.
In the migration of this embodiment, an administrator is required to perform migration configuration, configure a user directory to be migrated and a CLU to be migrated, where the user directory to be migrated is a user directory before migration, and the CLU to be migrated is a CLU after migration. The technical scheme includes that a CLU with insufficient capacity is migrated, a non-key user directory in the CLU is migrated to other CLUs with the surplus of storage space, generally speaking, the user directory before migration corresponds to non-key users in the CLU before migration, the storage space of the CLU before migration can be released after the user directory before migration is migrated, and the requirements of the key users or newly added users in the CLU before migration are met. The user directories before and after migration correspond to different CLUs.
When the user directory before migration is configured, setting can be directly performed for a data sender, for example, IPC is set, and since the user directory is similar to a folder of the IPC, if the identification of the IPC is known, the corresponding user directory can be known. When the migrated CLU is configured, only one cloud storage user corresponding to the migrated CLU needs to be known, and the administrator may determine the migrated CLU by selecting a cloud storage user corresponding to the migrated CLU. Therefore, the migration request comprises the identification of the data sender and the cloud storage user after migration, wherein the identification of the data sender implicitly comprises the cloud storage user before migration.
The application is not limited to the information specifically included in the migration request, and may include, for example, an identifier of a data sender, and a post-migration cloud storage user; the user directory before migration and the CLU after migration can also be directly included. Since the administrator is not easy to know which CLU has the storage capacity, but is easy to know which cloud storage users are not key users and can occupy the capacity, the method and the system determine the specific pre-migration user directory by including the identification of the data sender in the migration request, and determine the post-migration CLU by including the post-migration cloud storage users, so that intuitive configuration is facilitated.
In this embodiment, the cloud storage user after migration in the migration request may find the to-be-migrated CLU (post-migration CLU), and generate a user directory after migration, such as Usrdir', in the to-be-migrated CLU. Setting the state attribute of the user directory before migration to be a second state, namely a migration state, such as 2; and setting the state attribute of the user directory after migration to be a third state, namely a migration state, such as 1. The generated association relationship between the user directories before and after migration is shown in table 2.
| User directory | State attributes | Pre-migration user directory | Post-migration user directory |
| Usrdir | 2 | Usrdir’ | |
| Usrdir’ | 1 | Usrdir |
TABLE 2
In the user directory incidence relation before and after the migration, the user directories and the relation between the user directories before and after the migration, and the corresponding state attributes of the user directories are recorded. And after the same camera is migrated for multiple times, only the latest two times of migrated metadata are recorded.
It will be readily appreciated that including the pre-migration user directory and post-migration CLU directly in the migration request may also implement the above process. The generated user directory incidence relation before and after the migration contains the state attribute corresponding to the user directory, so that the user directory is convenient to distinguish whether the user directory is newly built or before and after the migration.
It should be noted that, in the application environment as shown in fig. 1, a system administrator configures the migrated IPC and the migrated cloud storage user on the user interface of the VM, and sends a migration request to the CDM. After receiving the migration request, CDM generates the association relationship between the user directories before and after the migration, sends a response message to VM, at least notifies the VM of the user directory after the migration, and the VM sends the user directory after the migration to IPC. In the present application, CDM is only required to notify the post-migration user directory to IPC through VM, and for different service systems, the interfaces with CDM are different, and the present application is not limited to interfacing with CDM through VM. It will be readily appreciated that CDM interfaces with a particular business system, and typically with a VM when the business system is a video surveillance system. For convenience of description, in the present application, a CDM is used to describe the interaction with a data sender, where the data sender not only refers to the data sender itself in a service system, but also includes a management server and the like interacting with the CDM in the service system, and details are not described below.
In this embodiment, after receiving the association relationship between the migrated user directories, the IPC may send a master scheduling service request to the CDM while carrying the migrated user directories. In order to distinguish whether the main scheduling service request sent by the IPC is a normal main scheduling service request or a migration main scheduling service request, in this embodiment, a corresponding user directory association relationship before and after migration is first searched through a user directory carried in the main scheduling service request.
For example, table 2 is found according to the user directory Usrdir ', and the state attribute corresponding to the user directory Usrdir' in table 2 is 1, which indicates that the user directory is migrated, that is, the migrated user directory, and it may be determined that migration is required at this time.
For another example, the table 1 is searched according to the user directory Usrdir, and the state attribute corresponding to the user directory Usrdir in the table 1 is 0, which indicates the normal working state of the user directory, and it may be determined that the migration is not needed at this time.
In this embodiment, a state attribute of a user directory is set in a user directory association relationship table before and after migration, and when the state attribute is 0, it indicates that a user directory is newly created; when the state attribute is 1, indicating that the user directory is immigrated; when the status attribute is 2, this indicates that the user directory is migrated.
It should be noted that, the method can determine whether the user directory carried in the primary scheduling service request is migrated without modifying the original processing flow of CDM. The method and the device can also carry different identifiers in the main scheduling service request message to distinguish whether the message is in a normal working state or in a transition state, for example, when the identifier is a normal, the message is in the normal working state; when a transfer is carried, it is a transfer.
And step 403, when the CDM judges that the user directory carried by the main scheduling service request is the migration, returning the main scheduling service corresponding to the CLU before and after the migration and the user directory before and after the migration to the data sender according to the association relationship between the user directories before and after the migration.
After receiving the primary scheduling service request, according to table 2, when the status attribute of the post-migration user directory is 1, the CDM searches the user directories before and after migration in the user directory association relationship table before and after migration. In this embodiment, CDM finds corresponding user directories before and after migration according to table 2, so as to determine CLUs before and after migration, learn the primary scheduling services corresponding to the CLUs before and after migration, and return the respective corresponding primary scheduling services to the data sender.
For example, if the CLU before migration is CLU-1, the CLU after migration is CLU-2, clu1 is the master scheduling service of CLU-1 before migration, and Clu2 is the master scheduling service of CLU-2 after migration, then Clu1 and Clu2 are returned to the IPC. And meanwhile, returning the user directories before and after the migration to the IPC.
If the state attribute corresponding to the user directory carried by the master scheduling service request is in the first state, that is, as shown in table 1, it is determined as the normal working state, and the corresponding master scheduling service is directly returned to the data sender, and the subsequent write data is described in steps 101 to 103, which is not described herein again.
In this embodiment, the IPC requests the main scheduling service CLU of the migrating out and migrating in cluster CLU to write the data node DN list with the user directories before and after migration, respectively.
How the CLU selects the master scheduling service and how the master scheduling service selects the DN are well-established technologies and will not be described herein.
In this embodiment, the IPC sets DN lists returned by the main scheduling service of the CLU before the migration and the main scheduling service of the CLU after the migration to readable, unwritable, and readable, writable states, that is, sets a data node returned by the CLU after the migration to a readable, writable state, and sets a data node returned by the CLU before the migration to a readable, unwritable state.
After determining the state of the DN, the IPC of this embodiment creates a video file to the readable and writable DN, and writes the video data, thereby completing the migration of the user.
By the method, only the metadata and the video data are migrated during migration, and large-area video read-write flow caused by migration is avoided. And after the migration, the space is released, and the key point location expansion and new point location storage can be immediately supplied. The storage is more flexible, the capacity expansion capacity of the storage nodes, the expansion cabinet and the hard disk is not limited to the upper limit, hardware is not increased, and the space of the rack and the machine room and the economic cost are saved.
In another embodiment, the reading of data is also involved, corresponding to the video surveillance system retrieving the playback. The inter-cloud migration method of the cloud storage system further includes:
CDM receives a main scheduling service request sent by a client, and searches the corresponding user directory incidence relation before and after migration according to a user directory carried by the main scheduling service request;
and the CDM returns the main scheduling service corresponding to the CLUs before and after the migration and the user directories before and after the migration to the client according to the incidence relation of the user directories before and after the migration, so that the client can conveniently carry the user directories before and after the migration respectively, request data nodes from the main scheduling service of the CLUs before and after the migration, and acquire data from the data nodes obtained by the request.
Similarly, taking video monitoring as an example, if the client performs retrieval and playback of the video, a main scheduling service request is sent to CDM, where the request carries the user directory. And a CDM lookup table 2 for returning the main scheduling service corresponding to the CLU before and after the migration and the user directory before and after the migration to the client. The client carries the user directories before and after the migration, requests data nodes from the main scheduling service of the CLUs before and after the migration, and acquires the video data from the data nodes obtained by the request, so that the video data can be played back. The process of reading data in the present application is similar to the process of requesting the master scheduling service and acquiring the data node by the data sender, and is not described herein again.
In one embodiment, the present application further provides an inter-cloud migration apparatus of a cloud storage system, where the cloud storage system includes a metadata server CDM and a cluster group CLU, and the inter-cloud migration apparatus of the cloud storage system, applied to the CDM, includes:
the association module is used for receiving the migration request, generating the association relation of the user directories before and after the migration according to the migration request, and returning the user directories after the migration to the data sender;
the scheduling service module is used for receiving a main scheduling service request sent by a data sender, searching a corresponding user directory incidence relation before and after migration according to a user directory carried by the main scheduling service request, and when the user directory carried by the main scheduling service request is judged to be migration, returning the main scheduling service corresponding to CLUs before and after migration and the user directory before and after migration to the data sender according to the user directory incidence relation before and after migration, so that the data sender respectively carries the user directories before and after migration, requests a data node to the main scheduling service of the CLUs before and after migration, sets the data node returned by the CLU after migration to be in a readable and writable state, and sets the data node returned by the CLU before migration to be in a readable and writable state, thereby the data sender writes data to the readable and writable data node.
For specific limitations of the inter-cloud migration apparatus of the cloud storage system, refer to the above limitations on the inter-cloud migration method of the cloud storage system, and are not described herein again. All or part of the modules in the inter-cloud migration apparatus of the cloud storage system may be implemented by software, hardware, or a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, the migration request includes an identifier of a data sender and a post-migration cloud storage user, the association module receives the migration request, generates a user directory association relationship before and after migration according to the migration request, and executes the following operations:
receiving a migration request, searching a CLU after migration according to the cloud storage user after migration in the migration request, generating a user directory after migration, setting the state attribute of the user directory before migration to be a second state, setting the state attribute of the user directory after migration to be a third state, and generating an association relation between the user directory before migration and the user directory after migration.
In one embodiment, the scheduling service module determines that a user directory carried by the main scheduling service request is migration, and executes the following operations:
and if the state attribute corresponding to the user directory carried by the main scheduling service request is a third state, judging that the user directory is migrated.
In an embodiment, the association module is further configured to generate user directory metadata for the newly configured user directory, and set a state attribute of the user directory to be a first state;
the dispatch service module further performs the following operations:
and if the state attribute corresponding to the user directory carried by the main scheduling service request is in the first state, judging that the state attribute is in a conventional working state, and directly returning the corresponding main scheduling service to the data sender.
In one embodiment, the dispatch service module is further configured to:
receiving a main scheduling service request sent by a client, and searching a corresponding user directory incidence relation before and after migration according to a user directory carried by the main scheduling service request;
and returning the main scheduling service corresponding to the CLU before and after the migration and the user directories before and after the migration to the client according to the association relationship of the user directories before and after the migration, so that the client can conveniently carry the user directories before and after the migration respectively, request data nodes from the main scheduling service of the CLU before and after the migration, and acquire data from the requested data nodes.
Although the CDM is used in the present embodiment, in the cloud storage system, both the data sender and the client of the service system need to cooperate with the CDM, and the description of the present application using the CDM as an example is also applicable to protection of the data sender and the client.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, and these are all within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (8)
1. An inter-cloud migration method of a cloud storage system, wherein the cloud storage system comprises a metadata server CDM and a cluster CLU, and the inter-cloud migration method of the cloud storage system comprises the following steps:
for the newly configured user directory, CDM generates user directory metadata, and the state attribute of the user directory is set as a first state;
CDM receives the migration request, generates the incidence relation of the user directories before and after migration according to the migration request, sets the state attribute of the user directory before migration to be a second state, sets the state attribute of the user directory after migration to be a third state, and returns the user directory after migration to the data sender;
CDM receives a main scheduling service request sent by a data sender, and searches the corresponding user directory incidence relation before and after migration according to the user directory carried by the main scheduling service request;
and when the CDM judges that the user directory carried by the main scheduling service request is the migration, the main scheduling service corresponding to the CLUs before and after the migration and the user directories before and after the migration are returned to the data sender according to the incidence relation of the user directories before and after the migration, so that the data sender can conveniently carry the user directories before and after the migration respectively, the data node is requested to the main scheduling service of the CLUs before and after the migration, the data node returned by the CLU after the migration is set to be in a readable and writable state, and the data sender writes data to the readable and writable data node.
2. The method for migrating between clouds in a cloud storage system according to claim 1, wherein the CDM, which determines that a user directory carried by a primary scheduling service request is a migration, includes:
and if the state attribute corresponding to the user directory carried by the main scheduling service request is a third state, judging that the user directory is migrated.
3. The method for migrating between clouds in a cloud storage system according to claim 1, wherein the method for migrating between clouds in a cloud storage system further includes:
and if the state attribute corresponding to the user directory carried by the main scheduling service request is in the first state, judging that the state attribute is in a conventional working state, and directly returning the corresponding main scheduling service to the data sender.
4. The method for migrating between clouds in a cloud storage system according to claim 1, wherein the method for migrating between clouds in a cloud storage system further includes:
CDM receives a main scheduling service request sent by a client, and searches a corresponding user directory association relation before and after migration according to a user directory carried by the main scheduling service request;
and the CDM returns the main scheduling service corresponding to the CLUs before and after the migration and the user directories before and after the migration to the client according to the incidence relation of the user directories before and after the migration, so that the client can conveniently carry the user directories before and after the migration respectively, request data nodes from the main scheduling service of the CLUs before and after the migration, and acquire data from the data nodes obtained by the request.
5. An inter-cloud migration apparatus of a cloud storage system, the cloud storage system including a metadata server CDM and a cluster group CLU, the inter-cloud migration apparatus of the cloud storage system being applied to the CDM, and comprising:
the correlation module is used for generating user directory metadata by CDM for the newly configured user directories and setting the state attribute of the user directories to be a first state; the system comprises a data sender, a migration request receiving module, a migration module and a migration module, wherein the data sender is used for receiving the migration request, generating the incidence relation of user directories before and after migration according to the migration request, setting the state attribute of the user directory before migration to be a second state, setting the state attribute of the user directory after migration to be a third state and returning the user directory after migration to the data sender;
the scheduling service module is used for receiving a main scheduling service request sent by a data sender, searching a corresponding user directory incidence relation before and after migration according to a user directory carried by the main scheduling service request, and when the user directory carried by the main scheduling service request is judged to be migration, returning the main scheduling service corresponding to CLUs before and after migration and the user directory before and after migration to the data sender according to the user directory incidence relation before and after migration, so that the data sender respectively carries the user directories before and after migration, requests a data node to the main scheduling service of the CLUs before and after migration, sets the data node returned by the CLU after migration to be in a readable and writable state, and sets the data node returned by the CLU before migration to be in a readable and writable state, thereby the data sender writes data to the readable and writable data node.
6. The device for migrating between clouds of cloud storage system according to claim 5, wherein the scheduling service module determines that the user directory carried by the main scheduling service request is migrated, and executes the following operations:
and if the state attribute corresponding to the user directory carried by the main scheduling service request is a third state, judging that the user directory is migrated.
7. The device for migrating between clouds in a cloud storage system according to claim 5, wherein the scheduling service module further performs the following operations:
and if the state attribute corresponding to the user directory carried by the main scheduling service request is in the first state, judging that the state attribute is in a conventional working state, and directly returning the corresponding main scheduling service to the data sender.
8. The device for migrating between clouds in a cloud storage system according to claim 5, wherein the scheduling service module is further configured to:
receiving a main scheduling service request sent by a client, and searching a corresponding user directory incidence relation before and after migration according to a user directory carried by the main scheduling service request;
and returning the main scheduling service corresponding to the CLU before and after the migration and the user directories before and after the migration to the client according to the association relationship of the user directories before and after the migration, so that the client can conveniently carry the user directories before and after the migration respectively, request data nodes from the main scheduling service of the CLU before and after the migration, and acquire data from the requested data nodes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910735036.7A CN112347036B (en) | 2019-08-09 | 2019-08-09 | Inter-cloud migration method and device of cloud storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910735036.7A CN112347036B (en) | 2019-08-09 | 2019-08-09 | Inter-cloud migration method and device of cloud storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112347036A CN112347036A (en) | 2021-02-09 |
| CN112347036B true CN112347036B (en) | 2023-03-14 |
Family
ID=74367827
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910735036.7A Active CN112347036B (en) | 2019-08-09 | 2019-08-09 | Inter-cloud migration method and device of cloud storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112347036B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117155759B (en) * | 2023-10-27 | 2024-01-05 | 腾讯科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106034080A (en) * | 2015-03-10 | 2016-10-19 | 中兴通讯股份有限公司 | Metadata migration method and metadata migration device in distributed system |
| CN106953893A (en) * | 2017-02-15 | 2017-07-14 | 北京百度网讯科技有限公司 | Data Migration between cloud storage system |
| US9727588B1 (en) * | 2010-03-29 | 2017-08-08 | EMC IP Holding Company LLC | Applying XAM processes |
| CN108509275A (en) * | 2018-03-29 | 2018-09-07 | 新华三技术有限公司 | A kind of catalogue moving method and metadata load-balancing method |
-
2019
- 2019-08-09 CN CN201910735036.7A patent/CN112347036B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9727588B1 (en) * | 2010-03-29 | 2017-08-08 | EMC IP Holding Company LLC | Applying XAM processes |
| CN106034080A (en) * | 2015-03-10 | 2016-10-19 | 中兴通讯股份有限公司 | Metadata migration method and metadata migration device in distributed system |
| CN106953893A (en) * | 2017-02-15 | 2017-07-14 | 北京百度网讯科技有限公司 | Data Migration between cloud storage system |
| CN108509275A (en) * | 2018-03-29 | 2018-09-07 | 新华三技术有限公司 | A kind of catalogue moving method and metadata load-balancing method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112347036A (en) | 2021-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6510112B2 (en) | Datastream Capture and Persistence Policy | |
| US10691716B2 (en) | Dynamic partitioning techniques for data streams | |
| US9794135B2 (en) | Managed service for acquisition, storage and consumption of large-scale data streams | |
| US9858322B2 (en) | Data stream ingestion and persistence techniques | |
| US9276959B2 (en) | Client-configurable security options for data streams | |
| US11093468B1 (en) | Advanced metadata management | |
| JP6450756B2 (en) | Partition-based data stream processing framework | |
| US7966470B2 (en) | Apparatus and method for managing logical volume in distributed storage systems | |
| CN108924244B (en) | Distributed system and flow distribution method and device for same | |
| US20120259901A1 (en) | Distributed storage of data in a cloud storage system | |
| US20160364407A1 (en) | Method and Device for Responding to Request, and Distributed File System | |
| US8930501B2 (en) | Distributed data storage system and method | |
| US20180188990A1 (en) | Method, apparatus and system for inserting disk | |
| CN103631924A (en) | Application method and system for distributive database platform | |
| CN113806300B (en) | Data storage method, system, device, equipment and storage medium | |
| CN112416892A (en) | Emergency video data cloud storage system | |
| CN113946276A (en) | Disk management method and device in cluster and server | |
| CN112347036B (en) | Inter-cloud migration method and device of cloud storage system | |
| WO2020134786A1 (en) | Server expansion method and device, server and storage medium | |
| CN111722783B (en) | Data storage method and device | |
| CN113905252A (en) | Data storage method and device for live broadcast room, electronic equipment and storage medium | |
| CN109343959B (en) | Multi-user computing and IO intensive SaaS system and application method | |
| CN109343928B (en) | Virtual memory file redirection method and system for virtual machine in virtualization cluster | |
| CN107168645B (en) | Storage control method and system of distributed system | |
| CN111399753A (en) | Method and device for writing pictures |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |