WO2017050283A1

WO2017050283A1 - Data processing method and device

Info

Publication number: WO2017050283A1
Application number: PCT/CN2016/099936
Authority: WO
Inventors: 朱炫鹏; 牛克强
Original assignee: 中兴通讯股份有限公司
Priority date: 2015-09-25
Filing date: 2016-09-23
Publication date: 2017-03-30
Also published as: CN106559444A

Abstract

Disclosed are a data processing method and device. The method comprises: acquiring the length of data to be transmitted to a standby storage apparatus; judging whether to compress the data according to the length of the data; and in the case where a judgement result is yes, compressing the data. The invention solves the problem in the related art that the time for transmitting some data is relatively long due to the fact that all transmission data are compressed, so that whether data compression is conducted can be determined separately for each transmission, thereby enabling the total transmission time for data transmission to be the shortest.

Description

Data processing method and device

Technical field

The present invention relates to the field of communications, and in particular to a data processing method and apparatus.

Background technique

In computer network storage systems, remote replication technology is introduced to prevent user data loss or downtime of critical business systems due to various disasters. The remote replication technology synchronizes the data on the local primary storage device mirrored volume (referred to as the source mirrored volume) to the mirrored volume of the remote storage device (referred to as the target mirrored volume). When the data of the local storage device is lost, data can be recovered from the remote storage device, thereby ensuring high reliability of data in the storage device.

In order to improve the efficiency of network transmission, some storage devices use data compression technology. The primary storage device compresses the data before the synchronization, and then transfers the compressed data to the backup storage device. The storage device decompresses the data and writes the data to the target mirrored volume. In general, data compression can reduce transmission time, but on a high-speed Fibre Channel (FC) network, if the amount of compressed data is small, the total transmission time may increase. For example, a piece of 500-byte data is compressed into 400 bytes, which is reduced by 100 bytes. On an 8Gbps FC network, the 100-byte transmission time is 125ns, which means that it can save 125ns of transmission time. However, the compression time of the primary storage device plus the decompression time of the standby storage device generally exceeds 125 ns, so that the total transmission time is increased instead of direct compression without compression.

It can be seen that during the entire remote replication process, some data compression can save time, and some data compression can not save time. In the prior art, the general storage device specifically negotiates and enables data compression before the remote replication starts, and then all The data is compressed by the transmission, and it is impossible to determine whether to enable data compression by each transmission.

An effective solution has not been proposed for the problem that the compression of all transmission data in the related art leads to the transmission of certain data for a long time.

Summary of the invention

The embodiments of the present invention provide a data processing method and apparatus, so as to at least solve the problem that the compression of all transmission data in the related art causes a certain data to be transmitted for a long time.

According to an aspect of the embodiments of the present invention, a data processing method includes: obtaining a length of data to be transmitted to a backup storage device; determining whether to compress the data according to a length of the data; In the case of YES, the data is compressed.

Further, determining whether to compress the data according to the length of the data comprises: determining, according to the length of the data, whether a sum of a time for compressing the data and a time for transmitting the compressed data is smaller than a direct transmission The time of the data.

Further, the method further includes: receiving a frame that is sent by the backup storage device to support compression of data; and determining, according to the received frame, that the standby storage device supports compression of data.

Further, the method further includes: notifying, by the reserved bits of the Fibre Channel FC frame, whether the data is compressed by the reserved storage device.

Further, after performing compression processing on the data to be transmitted, the method further includes: transmitting the compressed data to the standby storage device.

According to another aspect of the present invention, a data processing method includes: receiving data sent by a primary storage device, wherein the data is that the primary storage device determines a location according to the length of the acquired data. When the data is compressed, the data after the data is compressed, and the received data is decompressed.

Further, before receiving the data sent by the primary storage device, the method further includes: sending, to the primary storage device, a frame that supports compression of data, wherein the frame is used by the primary storage device to determine The storage device supports compression of data.

According to another aspect of the present invention, a data processing apparatus is provided, including: an obtaining module, configured to acquire a length of data to be transmitted to a backup storage device; and a determining module, configured to determine according to the length of the data Whether to compress the data; and the compression module is configured to perform compression processing on the data if the determination result is YES.

Further, the determining module includes: a determining unit, configured to be long according to the data The degree is judged whether the sum of the time at which the data is compressed and the time at which the compressed data is transmitted is less than the time at which the data is directly transmitted.

According to still another aspect of the present invention, a data processing apparatus is provided, including: a receiving module, configured to receive data sent by a primary storage device, where the data is the primary storage device according to the acquired The length of the data determines the data after the data is compressed, and the decompressing module is configured to decompress the received data.

According to still another embodiment of the present invention, a storage medium is also provided. The storage medium is arranged to store program code for performing the following steps:

Obtaining a length of data to be transmitted to the backup storage device; determining whether to compress the data according to the length of the data; and if the determination result is yes, performing compression processing on the data.

According to the embodiment of the present invention, the length of the data to be transmitted to the standby storage device is acquired; whether the data is compressed according to the length of the data; and if the determination result is yes, the data is compressed. The invention solves the problem that all the transmission data is compressed in the related art and causes some data to be transmitted for a long time, and it is possible to separately determine whether to perform data compression for each transmission, so that the total transmission time of the transmission data is the shortest.

DRAWINGS

The drawings described herein are intended to provide a further understanding of the invention, and are intended to be a part of the invention. In the drawing:

1 is a flow chart 1 of a data processing method according to an embodiment of the present invention;

2 is a second flowchart of a data processing method according to an embodiment of the present invention;

3 is a block diagram 1 of a data processing apparatus according to an embodiment of the present invention;

4 is a block diagram of a data processing apparatus in accordance with a preferred embodiment of the present invention;

FIG. 5 is a block diagram 2 of a data processing apparatus according to an embodiment of the present invention; FIG.

6 is a structural block diagram of a primary storage device and a standby storage device in a remote replication system according to an embodiment of the present invention;

7 is a schematic diagram of a data compression flag bit according to an embodiment of the present invention;

8 is a flowchart of a method for enabling data compression of a primary storage device during data transmission according to an embodiment of the present invention;

9 is a flow chart of a method for enabling data storage during data transmission by a standby storage device in accordance with an embodiment of the present invention.

detailed description

The invention will be described in detail below with reference to the drawings in conjunction with the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments may be combined with each other without conflict.

The embodiment of the present invention provides a data processing method. FIG. 1 is a flowchart 1 of a data processing method according to an embodiment of the present invention. As shown in FIG. 1, the method includes:

Step S102: Acquire a length of data to be transmitted to the backup storage device.

Step S104, determining whether to compress the data according to the length of the data;

In step S106, if the result of the determination is YES, the data is subjected to compression processing.

The length of the data to be transmitted to the backup storage device is obtained through the foregoing steps; whether the data is compressed according to the length of the data; and if the determination result is yes, the data is compressed, and the related technology is solved. The problem of compressing all transmitted data leads to a problem of transmitting some data for a long time, and it is possible to individually determine whether or not to perform data compression for each transmission, so that the total transmission time of the transmitted data is the shortest.

Further, determining whether to compress the data according to the length of the data may include: determining, according to the length of the data, whether a sum of a time for compressing the data and a time for transmitting the compressed data is less than a time for directly transmitting the data .

Further, receiving, by the standby storage device, a frame that supports compression of the data; determining, according to the received frame, that the standby storage device supports compression of the data.

Further, the standby storage device is advertised to the data storage device by the reserved bits of the Fibre Channel FC frame.

Further, after the data to be transmitted is compressed, the compressed data is transmitted to the standby storage device.

The embodiment of the present invention further provides a data processing method, and FIG. 2 is a second flowchart of the data processing method according to the embodiment of the present invention. As shown in FIG. 2, the method includes:

Step S202, receiving data sent by the primary storage device, where the data is data after the primary storage device determines to compress the data according to the length of the acquired data, and compresses the data;

Step S204, performing decompression processing on the received data.

Further, before receiving the data sent by the primary storage device, sending a frame supporting compression of the data to the primary storage device, wherein the frame is used by the primary storage device to determine that the standby storage device supports compression of data.

The embodiment of the present invention provides a data processing apparatus. FIG. 3 is a block diagram of a data processing apparatus according to an embodiment of the present invention. As shown in FIG. 3, the method includes an obtaining module 32, a determining module 34, and a compressing module 36. Each module is briefly described.

The obtaining module 32 is configured to acquire a length of data to be transmitted to the standby storage device;

The determining module 34 is configured to determine whether to compress the data according to the length of the data;

The compression module 36 is arranged to perform compression processing on the data if the determination result is YES.

4 is a block diagram of a data processing apparatus in accordance with a preferred embodiment of the present invention. As shown in FIG. 4, the determination module 34 includes:

The determining unit 42 is configured to determine whether the sum of the time for compressing the data and the time for transmitting the compressed data is smaller than the time for directly transmitting the data according to the length of the data.

The embodiment of the present invention further provides a data processing apparatus. FIG. 5 is a block diagram 2 of a data processing apparatus according to an embodiment of the present invention. As shown in FIG. 5, the receiving module 52 and the decompression module are included. 54, a brief description of each module below.

The receiving module 52 is configured to receive data sent by the primary storage device, where the data is data after the primary storage device determines to compress the data according to the length of the acquired data, and compresses the data;

The decompression module 54 is configured to decompress the received data.

The embodiments of the present invention are further described below in conjunction with specific embodiments.

The embodiment of the present invention provides a storage system for remote replication that determines whether to enable data compression for each transmission. FIG. 6 is a structural block diagram of a primary storage device and a standby storage device in a remote replication system according to an embodiment of the present invention. As shown in FIG. 6, the remote replication system includes a local primary storage device and a remote standby storage device, and the structures of the two are the same, which are further described below.

The main storage device 61 is implemented by the above-mentioned obtaining module 32, the judging module 34 and the compressing module 36, and the main device for storing data is the originating end of the remote copying. The data compression module 602, the data decompression module 604, the data transmission module 606, and the source mirroring volume 608 are connected to the backup storage device 62 through the FC network 612;

The storage device 62 is functionally implemented by the receiving module 52 and the decompression module 54 described above, and is a device for storing backup data, which is a receiving end of remote replication. The data compression module 602, the data decompression module 604, the data transmission module 606, and the target mirror volume 610 are connected to the main storage device 61 through the FC network 612;

The data compression module 602 performs a compression operation on the transmitted data.

The data decompression module 604 performs a decompression operation on the transmitted data;

a data transmission module 606 for transmitting data on the FC network;

a source mirrored volume 608 for storing a volume of source data;

a target mirrored volume 610 for storing a volume of backup data;

FC network 612, a network for transmitting data.

The embodiment of the present invention further provides a storage method for remote replication that determines whether to enable data compression separately for each transmission, and the method includes the following steps:

Step A, the primary storage device 61 initiates a data transmission, and the data transmission module 606 sends a FCP_CMND frame with a write command to the backup storage device 62, where the Fibre Channel Protocol for SCSI (FCP) is carried;

Step B, the data storage module 606 of the storage device 62 receives the write command, constructs the FCP_XFER_RDY frame, and sends the selected reserved position 1 to the primary storage device 61;

Step C, the data transfer module 606 of the primary storage device 61 receives the FCP_XFER_RDY frame, and checks the selected reserved bit. If it is 1, go to step D, otherwise go to step G;

Step D, the primary storage device 61 determines whether it is necessary to enable data compression according to the data length and the like, if necessary, go to step E; otherwise, go to step G;

Step E, the main storage device 61 reads data from the mirrored volume, sends it to the data compression module 602 for compression, and sends the compressed data to the data transmission module 606;

Step F, the data storage module 606 of the primary storage device 61 uses the compressed data to construct the FCP_DATA frame, the selected reserved position 1, is sent to the backup storage device 62, step I;

Step G, the primary storage device 61 reads data from the mirrored volume, and sends it to the data transmission module 606;

Step H, the data storage module 606 of the primary storage device 61 uses the data to construct the FCP_DATA frame, and sends the selected reserved location 0 to the standby storage device 62;

Step I: The data transmission module 606 of the backup storage device 62 receives the FCP_DATA frame, and checks the selected reserved bit. If it is 1, go to step J, otherwise go to step K;

Step J, the data storage module 606 of the storage device 62 sends the data to the data decompression module 604, the decompressed data is stored in the mirrored volume of the storage device 62, step L;

Step K: The data transmission module 606 of the backup storage device 62 stores the data in the mirrored volume of the standby storage device 62;

Step L: The data transmission module 606 of the backup storage device 62 sends an FCP_RSP frame to the primary storage device 61.

Step M: The data transmission module 606 of the primary storage device 61 receives the FCP_RSP frame, and the current data transmission process ends.

FIG. 7 is a schematic diagram of a data compression flag according to an embodiment of the present invention. As shown in FIG. 7, the data compression flag of the embodiment of the present invention is: BIT18 (RSV 703) of the word 3WORD3 of the FC frame header 701 is located in the DF_CTL 702 field. In the FC protocol, the bit is a reserved bit, and the embodiment of the present invention uses it as a flag to announce whether data compression is enabled.

The following is a detailed description from both sides of the primary storage device and the standby storage device.

FIG. 8 is a flowchart of a method for enabling data compression of a primary storage device during data transmission according to an embodiment of the present invention. As shown in FIG. 8, the method includes the following steps:

Step S802, sending an FCP_CMND frame with a write command to the backup storage device 62, and the data transmission module 606 constructs and issues an FCP_CMND frame, indicating that the data is ready to be written to the target mirrored volume 610;

Step S804, the data transmission module 606 receives the FCP_XFER_RDY frame sent by the backup storage device 62, indicating that the backup storage device 62 is ready to receive the written data.

Step S806, the data transmission module 606 checks whether the BIT18 of the WORD3 of the FCP_XFER_RDY frame is 1, if it is 1, it indicates that the backup storage device 62 supports data compression, and proceeds to step S808; otherwise, it indicates that the standby storage device 62 does not support data compression, and the process proceeds to step S818. ;

Step S808, according to the data length and other information to determine whether to enable data compression, if enabled, go to step S810, otherwise, go to step S818;

Step S810, reading data from the source mirror volume 608;

Step S812, the data is sent to the data compression module 602 for data compression;

Step S814, the data transmission module 606 constructs the FCP_DATA frame using the compressed data;

Step S816, the data transmission module 606 sets the BIT18 of the WORD3 of the FCP_DATA frame to 1, indicating that the carried data is compressed data, and proceeds to step S824;

Step S818, reading data from the source mirror volume 608;

Step S820, the data transmission module 606 constructs an FCP_DATA frame using the data;

Step S822, the data transmission module 606 sets the BIT18 of the WORD3 of the FCP_DATA frame to 0, indicating that the carried data is uncompressed data;

Step S824, the data transmission module 606 sends an FCP_DATA frame to the backup storage device 62.

Step S826, the data transmission module 606 receives the FCP_RSP frame sent by the standby storage device 62, indicating that the write operation is completed.

FIG. 9 is a flowchart of a method for enabling data storage during data transmission by a standby storage device according to an embodiment of the present invention. As shown in FIG. 9, the method includes the following steps:

Step S902, the data transmission module 606 receives the FCP_CMND frame with the write command sent by the primary storage device 61, indicating that the primary storage device 61 is ready to write data to the target mirrored volume 610;

Step S904, the data transmission module 606 constructs an FCP_XFER_RDY frame;

Step S906, the data transmission module 606 sets the BIT18 of the WORD3 of the FCP_XFER_RDY frame to 1, indicating that the local end supports data compression;

Step S908, the data transmission module 606 sends an FCP_XFER_RDY frame to the primary storage device 61, indicating that the data is ready to be received;

Step S910, the data transmission module 606 receives the FCP_DATA frame sent by the primary storage device 61;

Step S912, the data transmission module 606 checks whether the BIT18 of the WORD3 of the FCP_DATA frame is 1, if it is 1, it means that the carried data is compressed data, go to step S914; otherwise, it means that the carried data is uncompressed data, go to step S916;

Step S914, the compressed data is sent to the data decompression module 604 for data decompression;

Step S916, the decompressed data or uncompressed data is written to the target mirror volume 610;

In step S918, the data transmission module 606 sends an FCP_RSP frame to the primary storage device 61, indicating that the write operation is completed.

According to the embodiment of the present invention, at each data transmission, the judgment is made according to the data length and the like, and if the compression can save the total transmission time, the compression is performed; otherwise, the compression is not performed, so that the total transmission time can be minimized. The invention uses the reserved bits of the FC frame to notify the peer storage device whether to enable data compression, does not require additional negotiation messages and processes, does not occupy additional network bandwidth, and does not reduce the efficiency of remote replication.

Embodiments of the present invention also provide a storage medium. Optionally, in the embodiment, the foregoing storage medium may be configured to store program code for performing the following steps:

Step S1: Obtain a length of data to be transmitted to the backup storage device.

Step S2, determining whether to compress the data according to the length of the data;

In step S3, if the result of the determination is YES, the data is subjected to compression processing.

Optionally, in this embodiment, the foregoing storage medium may include, but not limited to, a USB flash drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, and a magnetic memory. A variety of media that can store program code, such as a disc or a disc.

It will be apparent to those skilled in the art that the various modules or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device such that they may be stored in the storage device by the computing device and, in some cases, may be different from the order herein. The steps shown or described are performed, or they are separately fabricated into individual integrated circuit modules, or a plurality of modules or steps thereof are fabricated as a single integrated circuit module. Thus, the invention is not limited to any specific combination of hardware and software.

The above description is only the preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes can be made to the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and scope of the present invention are intended to be included within the scope of the present invention.

Industrial applicability

The embodiment of the present invention is applied to the field of communications, and solves the problem that all the transmission data is compressed in the related art and causes some data to be transmitted for a long time. It is possible to separately determine whether to perform data compression for each transmission, so as to transmit the total data. The transmission time is the shortest.

Claims

A data processing method comprising:

Obtaining the length of data to be transmitted to the standby storage device;

Determining whether to compress the data according to the length of the data;

When the result of the determination is YES, the data is subjected to compression processing.
The method of claim 1, wherein determining whether to compress the data based on the length of the data comprises:

Whether the sum of the time for compressing the data and the time for transmitting the compressed data is less than the time for directly transmitting the data is determined according to the length of the data.
The method of claim 2, wherein the method further comprises:

Receiving a frame sent by the backup storage device to support compression of data;

Determining, according to the received frame, that the standby storage device supports compression of data.
The method of claim 3, wherein the method further comprises:

Whether the data is compressed is advertised to the standby storage device by a reserved bit of the Fibre Channel FC frame.
The method according to any one of claims 1 to 4, wherein after the data to be transmitted is subjected to compression processing, the method further comprises:

Transmitting the compressed data to the standby storage device.
A data processing method comprising:

Receiving data sent by the primary storage device, where the data is determined by the primary storage device to compress the data according to the length of the acquired data, Data after the data is compressed;

Decompressing the received data.
The method of claim 6, wherein the method further comprises: before receiving the data sent by the primary storage device:

Sending to the primary storage device a frame supporting compression of data, wherein the frame is used by the primary storage device to determine that the standby storage device supports compression of data.
A data processing device comprising:

Obtaining a module, configured to acquire a length of data to be transmitted to the standby storage device;

a determining module, configured to determine whether to compress the data according to the length of the data;

The compression module is configured to perform compression processing on the data if the determination result is YES.
The apparatus of claim 8 wherein said determining module comprises:

The determining unit is configured to determine, according to the length of the data, whether a sum of a time for compressing the data and a time for transmitting the compressed data is less than a time for directly transmitting the data.
A data processing device, comprising:

The receiving module is configured to receive data sent by the primary storage device, where the data is compressed by the primary storage device when the data is compressed according to the length of the acquired data. After the data;

And a decompression module, configured to decompress the received data.