CN113282246B

CN113282246B - Data processing method and device

Info

Publication number: CN113282246B
Application number: CN202110662084.5A
Authority: CN
Inventors: 通景根; 王士青
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2021-06-15
Filing date: 2021-06-15
Publication date: 2023-07-04
Anticipated expiration: 2041-06-15
Also published as: CN113282246A

Abstract

The embodiment of the application provides a data processing method and device, wherein the method comprises the following steps: and when the storage server meets the first condition, starting a drainage program in the storage server, and determining a first client having a mapping relation with the storage server through the drainage program. Setting a drainage device, registering a disguised data transmission path between the drainage device and the first client, and initializing a drainage pipeline corresponding to the drainage device. The original data transmission path between the storage server and the first client is replaced with a disguised data transmission path. And controlling the client data to be stored to the drainage device through the disguised data transmission path through the drainage pipeline. When the storage server cannot read and write data, a disguised data transmission path between the drainage device and the first client is adopted to replace an original data transmission path between the storage server and the first client, so that the data of the first client can be stored in the drainage device, and the running stability of the client is effectively ensured.

Description

Data processing method and device

Technical Field

Embodiments of the present disclosure relate to computer technology, and in particular, to a data processing method and apparatus.

Background

In the current business process, the client typically depends on a server, for example, a storage unit in the server may store data of the client.

In general, in order to ensure the operation of the upper layer service of the client, the server needs to work for a long time under high load, and there is a strong dependency between the server and the client, when a storage unit in the server fails, the upper layer service in the client is affected, and in the related art, the operation of the upper layer service needs to be closed when the storage unit is repaired or self-checked.

In the related art, the operation of the client needs to be stopped, and the client can be restarted after the repair or self-check of the storage unit is completed, which may cause the operation of the client to lack stability.

Disclosure of Invention

The embodiment of the application provides a data processing method and device, which are used for solving the problem of lack of stability of operation of a client.

In a first aspect, an embodiment of the present application provides a data processing method, including:

when a storage server meets a first condition, starting a drainage program in the storage server, and determining a first client having a mapping relation with the storage server through the drainage program;

Setting a drainage device, registering a disguised data transmission path between the drainage device and the first client, and initializing a drainage pipeline corresponding to the drainage device;

replacing an original data transmission path between the storage server and the first client with the disguised data transmission path, wherein a path identifier of the disguised data transmission path is the same as a path identifier of the original data transmission path;

and controlling to store client data to the drainage device through the disguised data transmission path through the drainage pipeline, wherein the client data is data received from the first client.

In one possible design, after registering the disguised data transmission path between the streaming device and the first client, the method further comprises:

setting the storage state of the storage server to be a drainage state;

controlling, by the tapping pipe, storage of client data to the tapping device via the camouflage data transmission path, comprising:

acquiring a storage state of the storage server;

and when the storage state of the storage server is determined to be the drainage state, controlling to store the client data to the drainage equipment through the disguised data transmission path through the drainage pipeline.

In one possible design, controlling, through the diversion pipe, storage of client data to the diversion apparatus through the disguised data transmission path includes:

determining a plurality of data units in the client data;

storing the plurality of data units to the drainage device according to a preset data rate;

generating and recording log information corresponding to each data unit, wherein the log information comprises at least one of the following: a starting memory address in the reclamation device, a memory offset in the reclamation device, and a rollback flag to indicate whether the data unit has been rollback to the storage server.

In one possible design, initializing the drainage tube corresponding to the drainage device includes:

loading a target drive for data drainage, and unloading or disabling an original drive corresponding to the original data transmission path;

initializing the drainage pipeline according to the drainage program and the target drive.

In one possible design, after initializing the drainage tube according to the drainage procedure and the target drive, further comprising:

And sending notification information to the first client, wherein the notification information is used for indicating that a port corresponding to the disguised data transmission path is available, so that the first client switches among a plurality of ports according to the notification information until the first client switches to the port corresponding to the disguised data transmission path.

In one possible design, the storage server includes at least one candidate storage device therein; setting up drainage equipment, include:

determining the equipment state of each storage equipment to be selected, wherein the equipment state is an idle state or an occupied state;

determining a device in an idle state in the at least one storage device to be selected as the drainage device;

setting a port corresponding to the disguised data transmission path as an available state, wherein the port corresponding to the disguised data transmission path is a port for accessing the drainage equipment;

and dividing a metadata area corresponding to the first client in the drainage equipment, wherein the metadata area is used for storing log information corresponding to data received from the first client.

In one possible design, at least one pluggable device is plugged into the storage server; setting up drainage equipment, include:

Generating an analog port corresponding to the drainage equipment, and sending an identification of the analog port to the first client;

setting the priority of the disguised data transmission path higher than the priority of the original data transmission path;

In one possible design, after setting the storage state of the storage server to the drain state, the method further includes:

performing preset processing, wherein the preset processing comprises at least one of the following steps: self-checking treatment, self-repairing treatment and upgrading treatment.

In one possible design, the method further comprises:

determining that the storage server is restored to an available state through the drainage program;

setting a storage state of the storage server to a normal-copy state by the streaming program, the normal-copy state being used to instruct storage of the client data to the storage server;

and starting a drainage data migration process, and returning the data stored in the drainage equipment to the storage server through the drainage data migration process.

In one possible design, the migrating the data stored in the draining device back to the storage server includes:

determining the storage position and the state of the transition of each data unit in the drainage equipment according to the log information corresponding to each data unit in the drainage equipment;

returning the data units which are not returned in the returning state in the drainage equipment to the storage server according to the storage position and the returning state of each data unit in the drainage equipment;

setting the state of the data unit which is successfully returned to the returned state in the log information.

In one possible design, after setting the storage state of the storage server to the normal-copy state by the migration program, the method further includes:

receiving data to be processed sent by the first client;

and when the storage state of the storage server is determined to be the normal-copy state, storing the data to be processed to the storage server through the original data transmission path.

In one possible design, after setting the storage state of the storage server to the drainage state, the method further includes:

Starting a timer;

and when the timing time of the timer is longer than or equal to the preset time, sending prompt information to the target equipment, wherein the prompt information is used for indicating that the drainage time is longer than or equal to the preset time.

In one possible design, the first condition includes at least one of:

the storage server fails;

the storage server receives an overhaul instruction;

the storage server receives an upgrade instruction, wherein the upgrade instruction is used for indicating to upgrade the storage server.

In a second aspect, an embodiment of the present application provides a data processing apparatus, including:

the system comprises a determining module, a storage server and a storage server, wherein the determining module is used for starting a drainage program in the storage server when the storage server meets a first condition, and determining a first client having a mapping relation with the storage server through the drainage program;

the processing module is used for setting drainage equipment, registering a disguised data transmission path between the drainage equipment and the first client, and initializing a drainage pipeline corresponding to the drainage equipment;

a replacing module, configured to replace an original data transmission path between the storage server and the first client with the disguised data transmission path, where a path identifier of the disguised data transmission path is the same as a path identifier of the original data transmission path;

And the storage module is used for controlling the client data to be stored to the drainage device through the disguised data transmission path through the drainage pipeline, wherein the client data is the data received from the first client.

In one possible design, the processing module is further configured to:

setting a storage state of the storage server to a streaming state after registering a disguised data transmission path between the streaming device and the first client;

the storage module is specifically used for:

acquiring a storage state of the storage server;

In one possible design, the memory module is specifically configured to:

determining a plurality of data units in the client data;

In one possible design, the processing module is specifically configured to:

In one possible design, the processing module is further configured to:

and after initializing the drainage pipeline according to the drainage program and the target drive, sending notification information to the first client, wherein the notification information is used for indicating that a port corresponding to the disguised data transmission path is available, so that the first client switches among a plurality of ports according to the notification information until switching to the port corresponding to the disguised data transmission path.

In one possible design, the storage server includes at least one candidate storage device therein; the processing module is specifically configured to:

In one possible design, at least one pluggable device is plugged into the storage server; the processing module is specifically configured to:

In one possible design, the processing module is further configured to:

after setting the storage state of the storage server to be a drainage state, performing preset processing, wherein the preset processing comprises at least one of the following steps: self-checking treatment, self-repairing treatment and upgrading treatment.

In one possible design, the processing module is further configured to:

In one possible design, the processing module is specifically configured to:

In one possible design, the memory module is further configured to:

after the storage state of the storage server is set to a normal-copy state through the drainage program, receiving data to be processed sent by the first client;

In one possible design, the processing module is further configured to:

after setting the storage state of the storage server to a drainage state, starting a timer;

In one possible design, the first condition includes at least one of:

the storage server fails;

the storage server receives an overhaul instruction;

In a third aspect, an embodiment of the present application provides a data processing apparatus, including:

a memory for storing a program;

a processor for executing the program stored by the memory, the processor being adapted to perform the method of the first aspect and any of the various possible designs of the first aspect as described above when the program is executed.

In a fourth aspect, embodiments of the present application provide a computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of the first aspect above and any of the various possible designs of the first aspect.

In a fifth aspect, embodiments of the present application provide a computer program product comprising a computer program which, when executed by a processor, implements a method as described in the first aspect and any of the various possible designs of the first aspect.

The embodiment of the application provides a data processing method and device, wherein the method comprises the following steps: and when the storage server meets the first condition, starting a drainage program in the storage server, and determining a first client having a mapping relation with the storage server through the drainage program. Setting a drainage device, registering a disguised data transmission path between the drainage device and the first client, and initializing a drainage pipeline corresponding to the drainage device. And replacing the original data transmission path between the storage server and the first client by using the disguised data transmission path, wherein the path identifier of the disguised data transmission path is the same as the path identifier of the original data transmission path. The client data is stored to the streaming device through the disguised data transmission path by the streaming pipe, the client data being data received from the first client. When the storage server can not normally provide the read-write service, the first client side with the mapping relation with the storage server is determined, the drainage device is arranged, then a disguised data transmission path between the drainage device and the first client side is adopted to replace an original data transmission path between the storage server and the first client side, so that the data of the first client side can be stored in the drainage device, and the phenomenon that the client side stops running due to the fact that the storage server can not realize the data read-write is avoided, so that the running stability of the client side is effectively ensured.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the following description will briefly describe the drawings that are required to be used in the embodiments or the related technical descriptions, and it is obvious that, in the following description, the drawings are some embodiments of the present application, and other drawings may be obtained according to the drawings without any inventive effort to those skilled in the art.

FIG. 1 is a schematic diagram of a system for a data processing method according to an embodiment of the present application;

FIG. 2 is a flowchart of a data processing method according to an embodiment of the present application;

FIG. 3 is a flowchart of a data processing method according to an embodiment of the present disclosure;

fig. 4 is a schematic implementation diagram of a determining drainage device according to an embodiment of the present application;

fig. 5 is a schematic implementation diagram of log information provided in an embodiment of the present application;

FIG. 6 is a third flowchart of a data processing method according to an embodiment of the present application;

FIG. 7 is a schematic diagram illustrating implementation of data migration according to an embodiment of the present disclosure;

FIG. 8 is a flow chart of a data processing method according to an embodiment of the present disclosure;

FIG. 9 is a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure;

Fig. 10 is a schematic hardware structure of a data processing device according to an embodiment of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

For a better understanding of the technical solutions of the present application, the following further details are provided for the background art related to the present application:

with the importance of the industry on data, different redundancy mechanisms are provided for each storage manufacturer to ensure that the data is safe and not lost. Most commonly there are redundant arrays of independent disks (Redundant Arrays of Independent Disks, raid) and ceph (a distributed file system). For example, a distributed storage can be built to provide services to the outside based on the complete cluster environment matched with the corresponding technology.

Based on the above description, an internet protocol storage area network (Internet Protocol Storage Area Network, ipsec an) is one of the most common applications based on the two backend raids and ceph.

The IPSAN is based on a very mature Ethernet technology, and because of the simple configuration technology, the configuration of equipment can comprise the configuration of equipment such as a router, an optical fiber switch, a multiport repeater and the like, and the characteristics of low cost of the IPSAN are quite obvious, and a common server or a PC only needs to be provided with a network card, so that a large-capacity storage space can be shared and used. The user can manage a storage space of the back end like a local disk, so that the method is widely applied.

Taking ceph (distributed file system) as an example, a storage server needs to provide cluster service, and the ceph distributed file system pools physical disks to provide a logical volume with redundancy. The logical volume is then mapped to the client by the IPSAN. After the client finds and connects the logical volume, the mapped logical volume can be used like a local disk.

For example, the implementation process of data reading may be understood in conjunction with fig. 1, and fig. 1 is a schematic system diagram of a data processing method according to an embodiment of the present application.

As shown in fig. 1, there are currently a client and a storage server, where an upper layer service may run in the client, the storage server may include a storage unit, the storage unit may implement storage of data, and the storage server may further include a processing unit, where the processing unit may be configured to implement control logic of the storage server.

Specifically, the client may write the related data of the upper layer service into the storage server to realize the storage of the data, and the client may also read the related data from the storage server so as to realize the corresponding upper layer service, so that the service of the lower layer storage may be provided in the storage server.

In the actual implementation process, the specific upper layer service implemented by the client and the specific implementation of the storage server may be selected according to the actual requirement, which is not particularly limited in this embodiment.

Based on the above description, it can be determined that, due to the strong dependency of the operation of the upper layer service on the storage server in the client, each module in the actual application and the cluster environment is completely bound, and any link has a problem and may directly affect the upper layer service.

However, repair of each service module also often requires shutting down operation of the upper layer service. This is often undesirable in the field of storage applications, especially video surveillance.

Furthermore, the storage server at the back end is often operated continuously for a long time under a high load. Even though the throughput of the upper layer traffic is low, the backend still needs to ensure that all services are uninterrupted. This may result in that all storage servers may not be self-checking or patrol, upgrade, etc. Creating a significant potential for data security. In case of a real failure of the storage server at the back end, even if the problem can be repaired soon, the client needs to stop the upper layer service to wait for the storage server to be normal, and then restart.

In summary, because the coupling degree between the client and the storage server is high, the storage server cannot realize operations such as self-checking, inspection, upgrading and the like in order to meet the operation requirement of the client; in addition, when the storage server fails, the client needs to stop the upper layer service and wait for the storage server to recover, which causes the running of the client to lack stability.

Therefore, how to realize the short-time isolation of the upper layer service and the storage server at the rear end, not only can ensure that the upper layer service is not interrupted, but also strives for the self-checking self-repairing and inspection or upgrading time for the storage server at the rear end, the importance and the necessity of the self-checking self-repairing and inspection or upgrading time are self-evident, and aiming at the problems in the related technology, the application provides the following technical concept: when the storage server fails or the storage server needs to perform self-checking, self-repairing, upgrading and other operations, the server determines drainage equipment corresponding to the client, the drainage equipment is used for storing client data during the period that the storage server pauses data storage, a data transmission path between the client and the drainage equipment is generated, the data of the client are stored in the drainage equipment, so that the server can have time to perform self-checking and fault maintenance on a storage unit, meanwhile, the data storage of the client is not affected, and therefore the operations of fault maintenance, self-checking, self-repairing, upgrading and the like of the server can be effectively realized while the stable operation of the client is ensured.

The data processing method provided in the present application is described below in conjunction with specific embodiments, and fig. 2 is a flowchart of the data processing method provided in the embodiment of the present application.

As shown in fig. 2, the method includes:

s201, when the storage server meets a first condition, starting a drainage program in the storage server, and determining a first client having a mapping relation with the storage server through the drainage program.

In this embodiment, the first condition is a condition for indicating that the storage server cannot realize reading and writing of data, and it is understood that when the storage server works normally, the storage server can normally realize reading and writing of data, but when the storage server satisfies the first condition, the storage server cannot normally realize reading and writing of data.

In one possible implementation, the first condition may include, for example, at least one of: the storage server fails, the storage server receives the overhaul instruction, and the storage server receives the upgrade instruction, wherein the upgrade instruction is used for indicating the upgrade of the storage server.

When the storage server fails, the storage server needs to be subjected to corresponding maintenance treatment, so that data cannot be read and written normally; when the storage server receives the overhaul instruction and the upgrade instruction, the storage server needs to perform corresponding self-checking, self-repairing, upgrading and other operations on the storage unit, so that corresponding data reading and writing cannot be performed, wherein the overhaul instruction and the upgrade instruction can be issued in real time manually or can be an instruction triggered automatically at fixed time, and the embodiment is not particularly limited.

The specific implementation manner of the first condition is not particularly limited in this embodiment, and besides the implementation manner described above, the specific implementation manner of the first condition may also be selected and set according to actual requirements, so long as the first condition is a condition for indicating that the storage server cannot normally perform data reading and writing.

On the basis of the above description, when it is determined that the storage server meets the first condition, it indicates that the storage server cannot perform normal data read-write operation currently, and corresponding processing needs to be performed on the first client corresponding to the storage server, so as to ensure that the part of the first client is not affected by the storage server and keeps normal operation.

When the first client performs corresponding processing on the first client, the first client corresponding to the storage server needs to be determined first, and in one possible implementation manner, a drainage program in the storage server can be started, and the first client having a mapping relation with the storage server is determined through the drainage program.

For example, the storage server may include a master node, where the master node may actively enable a drainage procedure, and then determine, through the drainage procedure, that a first client of a mapping relationship exists with the storage server currently meeting the first condition, where each client may have a mapped storage server, where the mapped storage server is used to instruct the current client to perform data reading and writing in the storage server corresponding to the mapping relationship.

For example, there are currently a client 1 and a client 2, and a storage server a, wherein the client 1 reads and writes data from and to the storage server a, and the client 2 reads and writes data from and to the storage server a, respectively, so that there is a mapping relationship between the client 1 and the storage server a, and there is a mapping relationship between the client 2 and the storage server a.

In the actual implementation process, the specific setting of the mapping relationship can be selected according to the actual requirement, and the implementation of the mapping relationship is not limited in this embodiment.

And in one possible implementation manner, the client storable servers may be in a many-to-one relationship, for example, so that when determining that the first client has a mapping relationship with the storage server, for example, at least one first client may be determined, and the specific implementation of the first client is not limited in this embodiment, and in particular, which client and the type of client may be selected according to the actual requirement.

S202, setting drainage equipment, registering a disguised data transmission path between the drainage equipment and the first client, and initializing a drainage pipeline corresponding to the drainage equipment.

After determining the first client, in order to ensure the normal operation of the part of the first client, in this embodiment, for example, a drainage device may be set, where the drainage device is used to replace the storage server to store the data of the first client during the period that the storage server cannot normally perform data reading and writing.

In one possible implementation, assuming that a plurality of first clients are currently determined, for example, the same drainage device may be set for the plurality of first clients, that is to say, the data generated by the plurality of first clients are all stored in the same drainage device; or, the corresponding drainage devices may be set for each first client, that is, the data generated by the plurality of first clients are stored in different drainage devices, which is not limited in this embodiment, and the specific implementation of the drainage devices may be selected according to the actual requirement.

The present embodiment may register a masquerading data transmission path between the streaming device and the first client after setting the streaming device, so that the data of the first client is stored in the streaming device through the masquerading data transmission path later.

After the camouflage data transmission path is registered, the drainage pipeline corresponding to the drainage device can be initialized, wherein the drainage pipeline can be understood as a virtual object, and the purpose of the drainage pipeline is to drain the client data to the camouflage data transmission path and then store the client data in the drainage device.

It will be appreciated that in a drainage pipe, drainage is an action, and a flow refers to a data flow, that is, an IO (Input/Output) of a client, the drainage pipe may guide the data flow into a camouflage data transmission path. The pipeline is realized by bypassing the original modules by a pipeline-like mode, and the data flow reaches the drainage equipment introduced in the embodiment.

In one possible implementation, the diversion program may, for example, initialize a diversion object, which is the diversion pipe described in the current embodiment, along which the read-write data of the client may flow to a particular diversion device.

S203, replacing an original data transmission path between the storage server and the first client by using the disguised data transmission path, wherein the path identification of the disguised data transmission path is the same as the path identification of the original data transmission path.

It will be appreciated that there is an original data transmission path between the storage server and the first client through which data from the client may be stored in the storage server during normal operation of the storage server.

However, when the storage server satisfies the first condition, the storage server cannot perform normal data reading and writing, and in this embodiment, a disguised data transmission path between the streaming device and the first client is registered, so that in order to ensure normal reading and writing of data of the client, the original data transmission path may be replaced by the disguised data transmission path, so that data of the client may be stored in the streaming device through the disguised data transmission path.

In one possible implementation manner, the disguised data transmission path in this embodiment may be understood as a pseudo path, where the pseudo path refers to a path in a multipath input/output (Multi Path Input Output, MPIO) to which the original client is connected, where the path has exactly the same identifier as the path to which the original client is connected, that is, the path identifier of the disguised data transmission path is the same as the path identifier of the original data transmission path, and the client cannot distinguish between them.

Therefore, for the client, the client does not perceive that the storage device is switched into the drainage device from the storage server, and the stable operation of the client can be effectively ensured.

It should be noted that, the drainage device in this embodiment may be a device capable of implementing data storage, such as a disk, a pluggable device, and other storage servers, and the implementation of a specific drainage device may be selected according to actual requirements, which is not limited in this embodiment.

S204, controlling to store client data to the drainage device through a disguised data transmission path through a drainage pipeline, wherein the client data is data received from a first client.

Based on the above description, it may be determined that, in this embodiment, the drainage pipe corresponding to the drainage device is initialized, where the drainage pipe may be used to control the data flow of the client to the drainage device, so after the setting of the drainage pipe, the disguised data transmission path, and the drainage device is implemented, the data sent by the client may be stored in the drainage device.

In one possible implementation manner, the client data may be controlled to be stored to the drainage device through the disguised data transmission path through the set drainage pipe, where the client data is data received from the first client, and the specific content and implementation of the client data are not limited in this embodiment.

It can be understood that when the storage server meets the first condition, the storage server is only unable to perform the read-write operation of the data normally, but the storage server may perform the normal processing operation, which may be understood that the storage unit of the storage server needs to be overhauled, but the control unit may still operate normally, for example, the storage server may control the data sent by the client to be stored in the drainage device.

In this embodiment, when the storage server cannot normally read and write data, a disguised data transmission path between the first client and the drainage device is established, and the original data transmission path between the storage server and the first client is replaced by the disguised data transmission path, so that data sent by the client can be controlled to be stored into the drainage device through the drainage pipeline through the disguised data transmission path, thereby effectively ensuring normal operation of the client, without interrupting the function of waiting for the storage server to recover normal data reading and writing, and effectively ensuring stable operation of the client.

The data processing method provided by the embodiment of the application comprises the following steps: and when the storage server meets the first condition, starting a drainage program in the storage server, and determining a first client having a mapping relation with the storage server through the drainage program. Setting a drainage device, registering a disguised data transmission path between the drainage device and the first client, and initializing a drainage pipeline corresponding to the drainage device. And replacing the original data transmission path between the storage server and the first client by using the disguised data transmission path, wherein the path identifier of the disguised data transmission path is the same as the path identifier of the original data transmission path. The client data is stored to the streaming device through the disguised data transmission path by the streaming pipe, the client data being data received from the first client. When the storage server can not normally provide the read-write service, the first client side with the mapping relation with the storage server is determined, the drainage device is arranged, then a disguised data transmission path between the drainage device and the first client side is adopted to replace an original data transmission path between the storage server and the first client side, so that the data of the first client side can be stored in the drainage device, and the phenomenon that the client side stops running due to the fact that the storage server can not realize the data read-write is avoided, so that the running stability of the client side is effectively ensured.

On the basis of the foregoing embodiments, the data processing method provided in the present application is described in further detail below with reference to fig. 3 to 5, fig. 3 is a flowchart of the data processing method provided in the embodiment of the present application, fig. 4 is an implementation schematic diagram of the determining drainage device provided in the embodiment of the present application, and fig. 5 is an implementation schematic diagram of log information provided in the embodiment of the present application.

As shown in fig. 3, the method includes:

s301, when a storage server meets a first condition, starting a drainage program in the storage server, and determining a first client having a mapping relation with the storage server through the drainage program.

The implementation of S301 is the same as that of S201 described above, and for example, the mapping relationship therein may be further understood in conjunction with fig. 4.

Specifically, when the storage server satisfies the first condition, the storage server may, for example, actively enable the migration program, and then, the storage server may, for example, obtain, through the migration program, a mapping relationship between the current storage server and a small computer system interface (Internet Small Computer System Interface, iSCSI) of the client, thereby determining a first client having a mapping relationship with the storage server that currently satisfies the first condition.

For example, it can be understood with reference to fig. 4 that a storage server a is currently present, and a client 1 and a client 2 are present, wherein a mapping relationship exists between both the client 1 and the client 2 and the storage server a.

The specific implementation manner may refer to the description of the foregoing embodiments, and will not be repeated herein.

S302, setting a drainage device, and registering a disguised data transmission path between the drainage device and the first client.

In this embodiment, a drainage device needs to be set, so that when a storage server cannot realize normal data reading and writing, client data is stored in the drainage device, and a possible implementation manner of setting the drainage device is described below.

In one possible implementation, the storage server may include at least one storage device to be selected, where the storage device to be selected may be, for example, a spare disk set in the storage server in advance, and the disk may be any available block device, and then the drainage device is determined in the at least one storage device to be selected, that is, the drainage device is a device inside the storage server.

Specifically, the device state of each storage device to be selected can be determined, the device state is an idle state or an occupied state, and the device in the idle state in at least one storage device to be selected is determined to be a drainage device.

That is, the to-be-selected storage device in the idle state in the to-be-selected storage device is determined as the drainage device, if there are multiple devices in the idle state in the to-be-selected storage device, for example, one may be selected randomly as the drainage device, or may be selected according to the identifier, the priority, etc. of the to-be-selected storage device to determine the drainage device, which is not limited in this embodiment; alternatively, if there is no storage device to be selected in the storage server in the idle state, a new storage device to be selected, such as a new disk, may be inserted on the storage server, for example.

After the drainage device is determined, the drainage device is required to be set correspondingly, for example, a port corresponding to the disguised data transmission path can be set to be in an available state, and the port corresponding to the disguised data transmission path is a port for accessing the drainage device; and dividing a metadata area corresponding to the first client in the drainage equipment, wherein the metadata area is used for storing log information corresponding to data received from the first client.

Specifically, when the client performs IO delivery, it is generally required to perform port selection, for example, the client may switch between ports in the port group provided by the storage server until switching to an available port.

Therefore, in order to ensure that the client data can be stored in the drainage device through the disguised data transmission path, a port corresponding to the disguised data transmission path needs to be set to be in an available state, so that the client can select the port and then the data can be stored in the drainage device, wherein the port corresponding to the disguised data transmission path is a port for accessing the drainage device.

And, in this embodiment, a metadata area corresponding to the first client may be further divided in the streaming device, where the metadata area may be used to store log information corresponding to data received from the first client, where, for example, a storage location, a rollback flag, a storage time, and the like of each data of the first client may be recorded in the log information.

In one possible implementation manner, the metadata area may be understood as a specific area in the drainage device for storing log information, for example, may be divided according to a size of the drainage device, where a size occupied by log information corresponding to each piece of data may be a fixed size, for example, 16 bits, or other sizes, which is not limited in this embodiment.

The above describes one possible implementation of the provision of a tapping device, in another possible implementation, at least one pluggable device being plugged into the storage server, the pluggable device can be determined, for example, as a tapping device, i.e. the tapping device is a device external to the storage server.

In this implementation manner, when the drainage device is set, for example, an analog port corresponding to the drainage device may be generated, and an identifier of the analog port may be sent to the first client; the priority of the masquerade data transmission path is set higher than the priority of the original data transmission path.

It will be appreciated that in order for the client to be able to store data in the sink device, a corresponding interface needs to be provided to the client, whereas in current implementations the sink device is a pluggable device, that is to say a storage medium that is required to sink the client data to the outside, so that a pseudo port can be provided to the client by means of an analog port, that is to say an analog port corresponding to the sink device is generated, and an identification of the analog port is sent to the first client, so that the client can achieve access to the analog port and attempt to transfer data through the analog port, and such an implementation can achieve complete isolation of the client from the storage server.

Meanwhile, in the current implementation manner, since the drainage device is an external storage medium, in order to enable the data of the client to be correctly stored in the drainage device, in this embodiment, the priority of the disguised data transmission path corresponding to the drainage device needs to be set, in one possible implementation manner, for example, the priority of the link issued by the IO of the client may be changed through Asynchronous Logic Unit Access (ALUA), so that the priority of the disguised data transmission path is higher than the priority of the original data transmission path, thereby implementing the drainage of the client data into the drainage device.

Meanwhile, the metadata area corresponding to the first client may be divided in the drainage device, where the metadata area is used to store log information corresponding to the data received from the first client, and an implementation manner of the metadata area is similar to that of the above-described alternative storage device and is not described herein again.

That is, when the drainage device is set, for example, the drainage device may be determined by a storage device to be selected in the storage server, and then the drainage device is set correspondingly, or the drainage device may be determined by at least one pluggable device plugged in the storage server, which is not limited in this embodiment, and may be selected and set according to actual requirements.

For example, as shown in fig. 4, it may be understood that the drainage device set for the first client may be used in conjunction with fig. 4, and assuming that the drainage device X and the drainage device Y are currently determined, for example, the drainage device X is determined as a drainage device corresponding to the client 1, the drainage device Y is determined as a drainage device corresponding to the client 2, then a disguised data transmission path between the drainage device X and the client 1 may be registered, and a data transmission path between the drainage device Y and the client 2 may be registered, so as to drain data of the client 1 and the client 2 into the respective corresponding drainage devices.

After the registration data transmission path between the drainage device and the first client is established, since the data of the client is stored by flowing to the drainage device through the registration data transmission path, and the data is not stored depending on the storage server, each storage unit in the storage server can be considered to be decoupled from the client, the storage server can perform preset operations, such as maintenance of faults of the storage unit, or corresponding operations of self-checking, self-repairing, upgrading and the like.

Therefore, the short-term effective isolation between the client and the storage server can be effectively realized in the embodiment, so that the operations of maintenance, self-checking, self-repairing and upgrading are performed on the storage server under the condition that the normal operation of the client is not affected.

S303, setting the storage state of the storage server to be a drainage state.

After the disguised data transmission path registration is completed, the storage server may set a storage state of the storage server to a drainage state, for example, in the drainage state, the storage server may preferentially stream data sent by the first client to the drainage device, so as to store the data sent by the client to the drainage device.

Meanwhile, the storage server can also open the timer to record the drainage time for guiding the data of the client to the drainage device, and it can be understood that the drainage device is only the device temporarily applied for maintenance, self-checking, self-repairing, upgrading and other operations of the storage server, and is not a long-term storage device, so in this embodiment, by timing the drainage time, when the timing time of the timer is greater than or equal to the preset time, prompt information can be sent to the target device, and the prompt information is used for indicating that the drainage time is greater than or equal to the preset time, thereby avoiding the long-term dependence of the drainage device on data storage, and resulting in excessive data stored in the drainage device.

In addition, in one possible implementation manner, after the storage state of the storage server is set to the drainage state, the client data does not flow to the storage server, and then the storage server can perform a preset process, where the preset process may include at least one of the following: self-checking processing, self-repairing processing and upgrading processing, so that maintenance and upgrading of the storage server can be realized under the condition that upper-layer business is not affected in the embodiment.

S304, loading a target drive for data drainage, and unloading or disabling an original drive corresponding to the original data transmission path.

After the storage state of the storage server is set to the drainage state, the initialization process of the drainage pipeline is performed rapidly, specifically, the storage server can load a TARGET (TARGET) drive specially used for data drainage, wherein the original drive (original TARGET drive) corresponding to the original data transmission path can be unloaded or disabled, so that port reuse is avoided.

S305, initializing a drainage pipeline according to a drainage program and a target drive.

After the target drive is loaded, the drainage pipe may be initialized according to the drainage program and the target drive, and in one possible implementation, the drainage program initializes a drainage object, where the drainage object may be understood as a pipe corresponding to the current storage server.

The data of the client may flow to a specific disk along the stream-guiding pipeline, that is, the stream-guiding device mentioned in this embodiment, so that the initialization operation may be completed by a stream-guiding program in the storage server, where one stream-guiding program may process a plurality of stream-guiding tasks, and the plurality of stream-guiding tasks may be implemented by different metadata areas.

And S306, sending notification information to the first client, wherein the notification information is used for indicating that a port corresponding to the disguised data transmission path is available, so that the first client switches among a plurality of ports according to the notification information until the first client switches to the port corresponding to the disguised data transmission path.

After the initialization of the drainage pipe is completed, the setting related to the drainage device is completed, and at this time, the relevant condition that the client data is stored in the drainage device is already provided, in order to enable the data of the first client to be stored in the drainage device, notification information needs to be sent to the first client, where the notification information is used to indicate that a port corresponding to the disguised data transmission path is available, so that the first client switches among a plurality of ports according to the notification information until switching to the port corresponding to the disguised data transmission path.

It can be understood that during the process of registering the disguised data transmission path between the drainage device and the first client, the IO of the first client is briefly blocked, and during this process, the first client may continuously retry issuing, but the duration is very short, and after the initialization of the drainage pipe is completed, the storage server may notify the client that the current port is normal, so that the client may continue issuing data.

Here, the policy used by the client is a failover policy only, and when the storage is abnormal, the client IO switches each port in the port group provided by the storage server until switching to an available port. Therefore, the selection process of the drainage port is active selection of the client, and external intervention is not needed, so that the retry process in the embodiment does not cause IO interruption, and the running stability of the client can be effectively ensured.

S307, replacing the original data transmission path between the storage server and the first client by using the disguised data transmission path, wherein the path identification of the disguised data transmission path is the same as the path identification of the original data transmission path.

The implementation of S307 is similar to that of S203, and will not be described here again.

S308, acquiring the storage state of the storage server.

And in this embodiment, the storage location of the data of the client depends on the storage state of the storage server, where when the storage state of the storage server is the streaming state, the client data is stored in the streaming device through the disguised data transmission path, and when the storage state of the storage server is not the streaming state, the client data is stored in the storage server through the original data transmission path, so in this embodiment, the storage state of the storage server needs to be acquired.

S309, when the storage state of the storage server is determined to be the drainage state, determining a plurality of data units in the client data.

When the storage state of the storage server is determined to be the streaming state, the client data may be stored in the streaming device.

In one possible implementation manner, when storing the client data in this embodiment, the client data may be divided into a plurality of data units, where the size of each data unit and the specific data included may be selected according to the actual requirement.

The data unit may be understood as, for example, data in a sliding window, and in this embodiment, in order to implement flow control of data storage, for example, data storage may be performed by a mechanism of the sliding window, where the data in a sliding window may be used as a data unit, and when data storage is performed, for example, data in a sliding window may be stored each time.

S310, storing a plurality of data units to the drainage device according to a preset data rate.

In the process of storing each data unit, for example, the data unit may be stored according to a preset data rate, and in one possible implementation manner, the storage server may determine the preset data rate, where the purpose of determining the preset data rate is to control the data amount flowing to the drainage device, and the preset data rate may be understood to be flow control, specifically, when the IO is issued through the disguised data transmission path, the IO rate may be kept at a reasonable rate through a sliding window manner, so that normal operation of the client service is ensured, and normal processing of the IO by the disk may be ensured.

And the flow control is performed by setting the preset data rate, so that guarantee can be provided for the abnormal recovery or data migration efficiency in the future.

The storage server sequentially stores each data unit in the drainage device through the disguised data transmission path according to the determined preset data rate and the determined preset data rate by aiming at the data issued by the client, and the storage device is replaced by the original storage server to be the directional drainage device without any change of the IO mode aiming at the client.

In one possible implementation manner, the data stored in the drainage device can be backed up again, so that the data redundancy is ensured, and the safety of data storage is effectively ensured.

S311, generating and recording log information corresponding to each data unit, wherein the log information comprises at least one of the following: a starting memory address in the reclamation device, a memory offset in the reclamation device, and a migration flag to indicate whether the data unit has migrated back to the storage server.

In the process of storing data, the storage server may further generate and record the log information corresponding to each data unit, and based on the description, it may be determined that, for each first client, a metadata area is divided in the corresponding drainage device and used for storing the log information, and then the log information in this embodiment may be stored in the metadata area corresponding to the drainage device, for example.

For example, as can be understood with reference to fig. 5, assuming that the metadata area corresponding to the client 1 is currently divided in the streaming device X, log information corresponding to each data unit of the client 1 may be stored in the metadata area as shown in fig. 5.

The log information may be recorded with a storage location of each data unit, for example, where the storage location may be determined, for example, by using a stored starting location and a stored offset, and according to the stored starting location and the stored offset, it may be quickly and effectively determined at which location of the drainage device the current data unit is stored.

And, a return flag may be further included in the log information, because after the storage server is restored to normal, the data stored in the current drainage device needs to be returned to the storage server, so the return flag may be used to indicate whether the current data unit is returned to the storage server, for example, an initial value of the return flag may be 0, which indicates that the data is not returned, and after the data is returned, the return flag may be set to 1, which indicates that the data is returned, where setting of the return flag may effectively ensure security of the data and ordering of the data return.

In the actual implementation process, the log information may further include any information related to unit storage, for example, storage time of each data unit, size of the data unit, etc., and the specific implementation of the log information is not particularly limited in this embodiment, and may be selected according to actual requirements.

The data processing method provided by the embodiment of the application comprises the following steps: the metadata area corresponding to the first client is divided in the drainage device so as to store log information of each data unit of the client, and the storage position and the return state of each data can be effectively determined based on the log information through storing the log information, so that the ordering of data storage and return can be effectively ensured.

On the basis of the above embodiment, after the maintenance or upgrading of the storage server is completed, the above described drainage process may be further subjected to ending processing, and the data in the drainage device may be migrated back to the storage server to ensure the integrity of the data stored in the storage server, where the ending processing of the drainage process and the implementation manner of the data migration are described in combination with the specific embodiment, fig. 6 is a flowchart three of a data processing method of an embodiment of the present application, and fig. 7 is a schematic diagram of the implementation manner of the data migration provided by the embodiment of the present application.

As shown in fig. 6, the method includes:

s601, determining that the storage server is restored to an available state through a drainage program.

In one possible implementation, after the storage server overhauls or upgrades are completed, the storage server may notify the migration program, for example, that the storage server has been restored to normal, so that in this embodiment it may be determined by the migration program that the storage server is restored to an available state.

S602, setting the storage state of the storage server to be a normal-copy state through a drainage program.

After determining that the storage server is restored to the usable state, the storage state of the storage server may be set to a normal-copy state by changing the storage state of the storage server through a streaming program.

In one possible implementation, the storage state of the storage server may be set, for example, by the following five-call procedure.

The implementation process of five calls can be as follows:

1. the storage server tells the drainage process: i get good;

2. the drainage process receives the storage recovery message and enters into ending process;

3. the drainage process finishes ending, like storing and sending a drainage information message;

4. storing the received message, preparing to receive the client IO, and informing that the drainage process is ready;

5. the drainage process modifies the drainage state of the storage server into a normal-copy state, and starts the drainage data migration process to migrate the data blocks.

In the ending and ending process of the second step, the above-described camouflage data transmission path and the drainage pipe may be processed.

The received client data is stored in the storage server through the original data transmission path after the drainage state is set to the normal-copy state for the camouflage data transmission path, so that the camouflage data transmission path can exist continuously or can be destroyed without influence, and therefore, for the camouflage data transmission path, the camouflage data transmission path can be selected to be cancelled or not processed, and the embodiment does not limit the method.

In addition, for drainage lines, the initialized drainage line is a virtual object, which can be destroyed or destroyed after the drainage process has ended.

S603, starting a drainage data migration process.

After setting the storage state of the storage server to the normal-copy state, the data in the migration device may be migrated, and in one possible implementation, the process of data migration may be implemented, for example, by a migration process of the data, so that the migration process of the data may be started.

S604, determining the storage position and the state of the transition of each data unit in the drainage device according to the log information corresponding to each data unit in the drainage device.

In the process of data migration, the storage server can process the data through a drainage data migration program, and specifically, the drainage data migration program can determine the migration state of each data unit in the drainage device according to the log information corresponding to each data unit, namely, whether the data is migrated to the storage server; and also a storage location for each data unit, wherein the storage location may be determined, for example, by a stored starting address and a stored offset.

S605, returning the data units which are not returned in the returning state in the drainage equipment to the storage server according to the storage positions and the returning states of the data units in the drainage equipment.

According to the state of the migration, the data units which are not migrated in the drainage device can be determined, the storage starting positions and the storage offset of the data units are determined according to the storage positions of the data units, and then the data units which are not migrated in the state of the drainage device are sequentially migrated to the storage server.

In a possible implementation manner, in the process of performing data migration, in order to ensure the correctness of the sequence of data storage, for example, a preset storage space may be reserved in the storage server, for example, as shown in fig. 7, a preset storage space is provided and is specially used for storing the migration data of the drainage device, and the client data newly received by the client at the current moment is stored after the preset storage space.

The size of the preset storage space can be determined according to the size of the data amount stored in the drainage device, for example, and the starting position of the preset storage space can be the ending position of the data storage before overhaul, for example.

In the actual implementation process, the setting of the preset storage space in the storage server can be selected and set according to actual requirements, so long as the preset storage space can be ensured to accommodate data in the drainage device, and the data stored in the preset storage space and the rest data are stored in sequence, and the specific setting position and setting size are not limited in this embodiment.

S606, setting the returning state of the data unit which is successfully returned to the returned state in the log information.

After the data unit completes the migration, the migration state of the part of the data unit may be set to be the migrated state, for example, the migrated state may be denoted by "1" to indicate that the current data unit has been migrated to the storage server, and in the actual implementation process, the specific expression manner of the migrated state may be selected according to the actual requirement, which is not limited in this embodiment.

In one possible implementation manner, in the migration process, if a request for data reading of a client is just received, for example, information comparison and judgment can be first performed in a metadata area in the drainage device according to the read request to determine whether the content to be read is in the drainage device at present and whether a migration flag corresponding to the data is in a non-migration state, if so, it can be determined that the current data has not been migrated yet, and then the corresponding data is returned to the client directly according to the data in the drainage device.

Or if the data corresponding to the data is in the state of being returned, the data is indicated to be returned to the storage server, and the corresponding data is required to be queried in the storage server, and the data is returned to the client.

In addition, after the storage state of the storage server is set to the normal-copy state, it indicates that the current overhaul or upgrade of the storage server has been completed, and the storage server may perform normal data read-write processing, so that the storage server may receive the data to be processed sent by the first client, and store the data to be processed to the storage server through the original data transmission path when it is determined that the storage state of the storage server is the normal-copy state.

For example, as can be understood from fig. 7, when the storage server returns to normal after the overhaul or the upgrade is completed, the storage server may directly store the data sent by the client in the storage server through the original data transmission path when receiving the data to be processed sent by the first client.

According to the data processing method, when the storage server is restored to normal, the data stored in the drainage device are returned to the storage server, so that the integrity of the data stored in the storage server can be effectively guaranteed, and in a specific returning process, the returning is sequentially carried out according to the storage position and the returning mark of the data unit, so that the accuracy and the order of the returning of the data can be effectively guaranteed, data transmission errors are avoided, and the consistency of the data is guaranteed.

Based on the foregoing embodiments, a system description is provided for a data processing method provided in the present application in conjunction with a detailed flowchart, and fig. 8 is a schematic flow chart of the data processing method provided in the embodiment of the present application.

As shown in fig. 8, the client may issue an IO to the server, and when the storage server fails or needs to be overhauled, upgraded, or the like, the storage server cannot complete normal data storage, the storage server may start a drainage program, determine a first client having a mapping relationship with the storage server through the drainage program, and register a disguised data transmission path between the drainage device and the first client.

After the disguised data transmission path registration is completed, the storage state of the storage server may be set to a drainage state, and a drainage pipe corresponding to the drainage device may be initialized,

referring to fig. 8, after the current state of drainage is recorded, it is determined that the received IO needs to be drained, and then the storage server transfers the client IO to a drainage pipe, where the drainage pipe is a pipe corresponding to the drainage device in a direction opposite to the drainage device, in one possible implementation manner, for example, the storage of the client data to the drainage device through the disguised data transmission path may be controlled through the drainage pipe.

In this embodiment, after determining the drainage device, a specific metadata area may also be determined in the drainage device, where the metadata area is used to store information such as a storage location, a state of a transition of each data unit of the client, and specific partition information of the metadata area may be, for example, drainage information in fig. 8.

After the setting of the drainage device is finished, the data of the client can be drained to the drainage device for storage, the length of the stored data and the drainage time can be recorded in the storage process, the data storage of the client can be subjected to flow control, so that excessive data is prevented from being stored in the drainage device, and log information of each data storage can be recorded.

At this time, the data stream of the client is already realized, the io issued by the client directly reaches the stream guiding device through the stream guiding pipeline, and the io mode is not changed for the client. While the back-end device becomes the directional drainage device in the bottom layer of the storage and slowly writes the disk through the flow control.

All services of the storage server are separated from the upper layer service IO at the moment, and no interference is generated on the upper layer IO in the storage equipment no matter the storage area network (Storage Area Network, SAN), CEPH, cluster and other service operations, so that the operations of service self-checking, repairing, service restarting, upgrading and the like of the storage server can be realized.

And, after determining that the storage server is restored to the usable state by the migration program, and setting the storage state of the storage server to the normal-copy state by the migration program, a data migration may be initiated, the data stored in the migration device may be migrated back into the storage server, for example, may be migrated back into the data unit shown in fig. 8.

In one possible implementation, for example, after completing five calls, data migration may be initiated according to the IO tail information, where the IO tail information may be, for example, a location of each data store, a state of migration, and so on.

And it is worth noting that in the illustration of FIG. 8, both data storage and reading in the storage server is accomplished based on logical unit number LUN (Logical Unit Number), where in a SAN, a LUN is a number that is used to identify a logical unit that is a device addressed by SCSI. In other words, the storage system partitions the physical hard disk into portions having logical addresses, and thus allows the host to access, and such a partition is called a LUN. A LUN in general also refers to a logical disk created on SAN storage.

In summary, according to the data processing method provided by the embodiment of the application, the drainage device is arranged to drain the data of the client, so that the short-term isolation between each module of the client and each module of the server can be realized, and the client can keep running continuously while the server can perform operations such as overhauling and upgrading.

In the implementation manner, the client supports the multipath configuration manner, so that the automatic transfer of io is realized by adopting a pseudo path manner, the conventional client does not need to be changed, and the automatic transfer can be realized and completed automatically without manual intervention.

Meanwhile, in this embodiment, services are provided in a similar component or module manner, and the storage server may be a hard disk directly, so that distributed storage backend services such as raid or ceph are bypassed. Even a cluster environment is not needed, the communication of a single storage to a client is ensured, the normal issuing of IO for a period of time can be ensured, and the processing time of self-checking and self-repairing is strived for each module under the client

And after the back end resumes normal storage, the storage server will self-initiate the data migration flow. Automatically calculating the length of the data written into the drainage device, and flowing out of a corresponding space at the rear end for migration. Meanwhile, in the migration process, a new issuing io can automatically write from the reserved migration space. Thereby ensuring the io seamless connection of the whole process without interruption.

Meanwhile, in the embodiment, the data can be sequentially migrated according to the storage position and the migration mark of the data in the drainage equipment, so that the IO before drainage and the IO after drainage are ensured to be free from any loss, and the consistency of the data can be effectively ensured.

Fig. 9 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 9, the apparatus 90 includes: a determination module 901, a processing module 902, a replacement module 903, and a storage module 904.

A determining module 901, configured to, when a storage server meets a first condition, start a drainage program in the storage server, and determine, by using the drainage program, a first client having a mapping relationship with the storage server;

a processing module 902, configured to set a drainage device, register a disguised data transmission path between the drainage device and the first client, and initialize a drainage pipe corresponding to the drainage device;

a replacing module 903, configured to replace an original data transmission path between the storage server and the first client with the disguised data transmission path, where a path identifier of the disguised data transmission path is the same as a path identifier of the original data transmission path;

A storage module 904, configured to control, through the streaming pipe, storage of client data to the streaming device through the disguised data transmission path, where the client data is data received from the first client.

In one possible design, the processing module 902 is further configured to:

the storage module 904 is specifically configured to:

acquiring a storage state of the storage server;

In one possible design, the memory module 904 is specifically configured to:

determining a plurality of data units in the client data;

In one possible design, the processing module 902 is specifically configured to:

In one possible design, the processing module 902 is further configured to:

In one possible design, the storage server includes at least one candidate storage device therein; the processing module 902 is specifically configured to:

In one possible design, at least one pluggable device is plugged into the storage server; the processing module 902 is specifically configured to:

In one possible design, the processing module 902 is further configured to:

setting a storage state of the storage server to a normal-copy state by the migration program;

In one possible design, the memory module 904 is further configured to:

In one possible design, the processing module 902 is further configured to:

In one possible design, the first condition includes at least one of:

the storage server fails;

the storage server receives an overhaul instruction;

The device provided in this embodiment may be used to implement the technical solution of the foregoing method embodiment, and its implementation principle and technical effects are similar, and this embodiment will not be described herein again.

Fig. 10 is a schematic hardware structure of a data processing apparatus according to an embodiment of the present application, and as shown in fig. 10, a data processing apparatus 100 according to the present embodiment includes: a processor 1001 and a memory 1002; wherein the method comprises the steps of

Memory 1002 for storing computer-executable instructions;

the processor 1001 is configured to execute computer-executable instructions stored in a memory to implement the steps executed by the data processing method in the above embodiment. Reference may be made in particular to the relevant description of the embodiments of the method described above.

Alternatively, the memory 1002 may be separate or integrated with the processor 1001.

When the memory 1002 is provided separately, the data processing device further comprises a bus 1003 for connecting said memory 1002 to the processor 1001.

The embodiment of the application also provides a computer readable storage medium, wherein computer execution instructions are stored in the computer readable storage medium, and when a processor executes the computer execution instructions, the data processing method executed by the data processing device is realized.

Embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements a data processing method as performed by a data processing device as described above.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.

The integrated modules, which are implemented in the form of software functional modules, may be stored in a computer readable storage medium. The software functional module is stored in a storage medium, and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (english: processor) to perform some of the steps of the methods described in the embodiments of the present application.

It should be understood that the above processor may be a central processing unit (english: central Processing Unit, abbreviated as CPU), or may be other general purpose processors, digital signal processors (english: digital Signal Processor, abbreviated as DSP), application specific integrated circuits (english: application Specific Integrated Circuit, abbreviated as ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.

The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile memory NVM, such as at least one magnetic disk memory, and may also be a U-disk, a removable hard disk, a read-only memory, a magnetic disk or optical disk, etc.

The bus may be an industry standard architecture (Industry Standard Architecture, ISA) bus, an external device interconnect (Peripheral Component, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, the buses in the drawings of the present application are not limited to only one bus or one type of bus.

The storage medium may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: various media that can store program code, such as ROM, RAM, magnetic or optical disks.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of data processing, comprising:

setting a drainage device, registering a disguised data transmission path between the drainage device and the first client, and initializing a drainage pipeline corresponding to the drainage device, wherein the drainage pipeline is used for guiding data of the first client to the disguised data transmission path and storing the data in the drainage device;

Controlling, by the streaming pipe, client data to be stored to the streaming apparatus through the disguised data transmission path, the client data being data received from the first client;

the storage server comprises at least one storage device to be selected; setting up drainage equipment, include:

dividing a metadata area corresponding to the first client in the drainage equipment, wherein the metadata area is used for storing log information corresponding to data received from the first client;

or alternatively, the process may be performed,

at least one pluggable device is inserted on the storage server; setting up drainage equipment, include:

2. The method of claim 1, further comprising, after registering a disguised data transmission path between a streaming device and the first client:

setting the storage state of the storage server to be a drainage state;

acquiring a storage state of the storage server;

3. The method of claim 2, wherein controlling, via the diversion pipe, storage of client data to the diversion apparatus via the disguised data transmission path comprises:

Determining a plurality of data units in the client data;

4. A method according to any one of claims 1-3, wherein initializing the corresponding drainage tube of the drainage device comprises:

5. The method of claim 4, further comprising, after initializing the drainage tube according to the drainage procedure and the target drive:

6. A method according to any one of claims 1-3, wherein after setting the storage state of the storage server to the drain state, the method further comprises:

7. A method according to any one of claims 1-3, wherein the method further comprises:

8. The method of claim 7, wherein the migrating the data stored in the streaming device back into the storage server comprises:

9. The method of claim 7, further comprising, after setting the storage state of the storage server to the normal-copy state by the migration program:

receiving data to be processed sent by the first client;

10. The method of claim 2, further comprising, after setting the storage state of the storage server to the drain state:

starting a timer;

11. The method of any one of claims 1-3, 5, 8, 9, 10, wherein the first condition comprises at least one of:

The storage server fails;

the storage server receives an overhaul instruction;

12. A data processing apparatus, comprising:

the processing module is used for setting a drainage device, registering a disguised data transmission path between the drainage device and the first client, initializing a drainage pipeline corresponding to the drainage device, and storing the drainage pipeline in the drainage device, wherein the drainage pipeline is used for guiding the data of the first client to the disguised data transmission path;

A storage module, configured to control, through the drainage pipe, to store client data to the drainage device through the disguised data transmission path, where the client data is data received from the first client;

the storage server comprises at least one storage device to be selected; the processing module is specifically configured to:

or alternatively, the process may be performed,

at least one pluggable device is inserted on the storage server; the processing module is specifically configured to:

13. The apparatus of claim 12, wherein the processing module is further configured to:

the storage module is specifically used for:

acquiring a storage state of the storage server;

when the storage state of the storage server is determined to be the drainage state, controlling to store client data to the drainage device through the disguised data transmission path through the drainage pipeline;

the storage module is specifically used for:

determining a plurality of data units in the client data;

generating and recording log information corresponding to each data unit, wherein the log information comprises at least one of the following: a starting storage address in the reclamation device, a storage offset in the reclamation device, and a rollback flag to indicate whether the data unit has been rollback to the storage server;

The processing module is specifically configured to:

initializing the drainage pipeline according to the drainage program and the target drive;

the processing module is further configured to:

after initializing the drainage pipeline according to the drainage program and the target drive, sending notification information to the first client, wherein the notification information is used for indicating that a port corresponding to the disguised data transmission path is available, so that the first client switches among a plurality of ports according to the notification information until switching to the port corresponding to the disguised data transmission path;

the processing module is further configured to:

after setting the storage state of the storage server to be a drainage state, performing preset processing, wherein the preset processing comprises at least one of the following steps: self-checking treatment, self-repairing treatment and upgrading treatment;

the processing module is further configured to:

Starting a drainage data migration process, and returning the data stored in the drainage equipment to the storage server through the drainage data migration process;

the processing module is specifically configured to:

setting the state of returning the successfully returned data unit to be the state of returned data unit in the log information;

the storage module is further configured to:

storing the data to be processed to the storage server through the original data transmission path when the storage state of the storage server is determined to be the normal-copy state;

the processing module is further configured to:

When the timing time of the timer is longer than or equal to the preset time, sending prompt information to target equipment, wherein the prompt information is used for indicating that the drainage time is longer than or equal to the preset time;

the first condition includes at least one of:

the storage server fails;

the storage server receives an overhaul instruction;

14. A data processing apparatus, comprising:

a memory for storing a program;

a processor for executing the program stored by the memory, the processor being for performing the method of any one of claims 1 to 11 when the program is executed.

15. A computer readable storage medium comprising instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 11.