CN111723024B

CN111723024B - Data processing method, device and system, client server and storage server

Info

Publication number: CN111723024B
Application number: CN201910203852.3A
Authority: CN
Inventors: 牛克强; 刘瑞芳
Original assignee: Alibaba Cloud Computing Ltd
Current assignee: Alibaba Cloud Computing Ltd
Priority date: 2019-03-18
Filing date: 2019-03-18
Publication date: 2024-05-07
Anticipated expiration: 2039-03-18
Also published as: CN111723024A

Abstract

The embodiment of the application provides a data processing method, a system device, a client server and a storage server. According to the technical scheme, the data storage request sent by the target application program is received through the virtual storage equipment in the client server. Wherein the virtual storage device is created by the client server based on the target application. The first logical storage address is obtained by a first logical storage device associated with the virtual storage device. The first logical storage device is generated by the first storage server corresponding to the target application. And sending the data storage request to the first storage server based on the first logic storage address so as to store the data to be stored into at least one physical storage device corresponding to the first logic storage address through the first storage server. The embodiment of the application can realize seamless connection between the application program of the enterprise user in the traditional centralized storage system and the existing distributed storage system.

Description

Data processing method, device and system, client server and storage server

Technical Field

The embodiment of the application relates to the technical field of networks, in particular to a data processing method, a device and a system, a client server and a storage server.

Background

The distributed storage system dispersedly stores data on a plurality of independent devices, and a plurality of storage servers (namely storage nodes) share the storage load together by adopting an expandable distributed system structure, so that the elastic expansion of the storage capacity and the performance can be realized. And the balance of system load can be realized through the node migration of the data, so that the condition of single-point overheating is avoided. When storage nodes are added to the cluster, the overall capacity and performance of the system also expands linearly with it. Therefore, the system gradually replaces the traditional centralized storage system by the advantages of high reliability, low network time delay, high access efficiency, easiness in expansion and the like.

In the distributed storage system, the application program is located in the client server, and the data storage service of the application program is provided by the storage server. Because the storage server can manage a plurality of physical storage devices, the actual storage position of the data may migrate, or the physical storage address may change in a scattered storage condition. The application program needs to implement network connection with any storage node by establishing an association relationship with a logical storage device generated by any storage node based on a plurality of physical storage device mappings, where the storage node determines an actual physical storage address corresponding to the logical storage address based on the logical storage address of the logical storage device.

However, in the conventional centralized storage system, since data is stored in the same physical storage device, its physical storage address is fixed and only the storage capability of two nodes is provided. In a conventional centralized storage system, an application program of an enterprise user acquires a data storage service by establishing network links with two storage nodes respectively. The application program firstly establishes network connection with the main node, and when the main node fails and can not provide storage service for the application program, the application program switches the network connection to the standby node, so that the application program of the enterprise user only supports a double-node storage model, but the double-node storage model limits the lateral expansion capacity of the storage node and can not adapt to the distributed cluster storage environment of the existing multiple storage nodes.

Therefore, how to adapt an application program of an enterprise user in a traditional centralized storage system to a current distributed storage system, so that the enterprise user experiences the storage service of the distributed storage system, the complexity of enterprise user migration is reduced, and the seamless expansion of storage nodes is a problem to be solved.

Disclosure of Invention

The embodiment of the application provides a data processing method, a device, a system, a client server and a storage server, so that an application program of an enterprise user in a traditional centralized storage system can realize seamless connection with the existing distributed storage system, and the user can obtain better storage service.

In a first aspect, an embodiment of the present application provides a data processing method, including

Receiving a data storage request sent by a target application program by virtual storage equipment in a client server; wherein the virtual storage device is created by a client server based on the target application;

Acquiring a first logical storage address through a first logical storage device associated with the virtual storage device; the first logic storage device is generated by a first storage server corresponding to the target application program;

And sending the data storage request to the first storage server based on the first logical storage address so as to store the data to be stored into at least one physical storage device corresponding to the first logical storage address through the first storage server.

In a second aspect, an embodiment of the present application provides a data processing method, including:

Receiving a data storage request sent by virtual storage equipment based on a first logic storage address; wherein the first logical address is obtained by the virtual storage device from a first logical storage device based on the data storage request; the data storage request is sent to the virtual storage device for a target application program;

Determining a first physical storage address corresponding to the first logical storage address based on a preset address mapping relation;

determining at least one physical storage device corresponding to the first physical storage address;

and storing the data to be stored into at least one physical storage device corresponding to the first physical storage address.

In a third aspect, an embodiment of the present application provides a data processing apparatus, including:

The first receiving module is used for receiving a data storage request sent by a target application program by virtual storage equipment in the client server; wherein the virtual storage device is created by a client server based on the target application;

The first acquisition module is used for acquiring a first logic storage address through a first logic storage device associated with the virtual storage device; the first logic storage device is generated by a first storage server corresponding to the target application program;

And the first sending module is used for sending the data storage request to the first storage server based on the first logic storage address so as to store the data to be stored into at least one physical storage device corresponding to the first logic storage address through the first storage server.

In a fourth aspect, an embodiment of the present application provides a data processing apparatus, including:

The storage request receiving module is used for receiving a data storage request sent by the virtual storage device based on the first logic storage address; wherein the first logical address is obtained by the virtual storage device from a first logical storage device based on the data storage request; the data storage request is sent to the virtual storage device for a target application program;

The first storage address determining module is used for determining a first physical storage address corresponding to the first logical storage address based on a preset address mapping relation;

the first storage device determining module is used for determining at least one physical storage device corresponding to the first physical storage address;

And the storage module is used for storing the data to be stored into at least one physical storage device corresponding to the first physical storage address.

In a fifth aspect, in an embodiment of the present application, a client server is provided, including a processing component and a storage component; the storage component is used for storing one or more computer instructions, wherein the one or more computer instructions are used for being called by the processing component for execution;

The processing assembly is configured to:

In a sixth aspect, in an embodiment of the present application, a storage server is provided, including a processing component and a storage component; the storage component is used for storing one or more computer instructions, wherein the one or more computer instructions are used for being called by the processing component for execution;

The processing assembly is configured to:

Compared with the prior art, the application can obtain the following technical effects:

The embodiment of the application provides a data processing method, a system device, a client server and a storage server. According to the technical scheme, the data storage request sent by the target application program is received through the virtual storage equipment in the client server. By establishing the storage connection between the application program and the virtual storage device, the target application program can be enabled to not need to modify the I/O configuration, and the target application program can be enabled to perceive that the storage process is completed only through the virtual storage device. And by establishing the association relationship between the virtual storage and the first logic storage device, the network connection between the virtual storage device and the first storage server is established, so that the data storage request can be sent to the first storage server based on the first logic storage address acquired from the first logic storage device. Thereby realizing that the data to be stored is stored into at least one physical storage device corresponding to the first logical storage address through the first storage server. Therefore, the embodiment of the application can enable the application program only supporting the dual-node storage model to be adapted to the distributed storage system of multi-node storage, realize seamless connection between the target application program and the existing distributed storage system, greatly reduce the complexity of storage service migration of enterprise users and enable the enterprise users to obtain better storage service experience.

These and other aspects of the application will be more readily apparent from the following description of the embodiments.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow diagram of one embodiment of a data processing method according to the present application;

FIG. 2 is a flow chart of yet another embodiment of a data processing method according to the present application;

FIG. 3 is a schematic diagram of a network connection provided in accordance with the present application for establishing a connection with the new first storage server;

FIG. 4 is a flow diagram illustrating one embodiment of a data processing method provided in accordance with the present application;

FIG. 5 is a schematic diagram illustrating one embodiment of a data processing system provided in accordance with the present application;

FIG. 6 is a schematic diagram of a data processing system based on ISCSI distributed storage in accordance with the present application;

FIG. 7 is a schematic diagram illustrating the construction of one embodiment of a data processing apparatus provided in accordance with the present application;

FIG. 8 is a schematic diagram of another embodiment of a data processing apparatus according to the present application;

FIG. 9 illustrates a schematic diagram of one embodiment of a client server provided in accordance with the present application;

Fig. 10 is a schematic structural diagram of an embodiment of a storage server according to the present application.

Detailed Description

In order to enable those skilled in the art to better understand the present application, the following description will make clear and complete descriptions of the technical solutions according to the embodiments of the present application with reference to the accompanying drawings.

In some of the flows described in the specification and claims of the present application and in the foregoing figures, a plurality of operations occurring in a particular order are included, but it should be understood that the operations may be performed out of order or performed in parallel, with the order of operations such as 101, 102, etc., being merely used to distinguish between the various operations, the order of the operations themselves not representing any order of execution. In addition, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first" and "second" herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, and are not limited to the "first" and the "second" being different types.

As described in the background art, in order to adapt an application program of an enterprise user of a conventional centralized storage system to an existing distributed storage system, the inventor proposes a technical solution through a series of researches. By establishing a storage connection between the target application program and the virtual storage device, the application program can not only avoid modifying the I/O configuration, but also ensure that the target application program perceives the storage process to be completed only through the virtual storage device. And by establishing the association relationship between the virtual storage and the first logic storage device, the network connection between the virtual storage device and the first storage server is established, so that the data storage request can be sent to the first storage server based on the first logic storage address acquired from the first logic storage device. Thereby realizing that the data to be stored is stored into at least one physical storage device corresponding to the first logical storage address through the first storage server. Therefore, the embodiment of the application can enable the target application program only supporting the dual-node storage model to be adapted to the distributed storage system of multi-node storage, realize seamless connection between the target application program and the existing distributed storage system, greatly reduce the complexity of storage service migration of enterprise users and enable the users to obtain better storage service experience.

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to fall within the scope of the application.

Fig. 1 is a schematic flow chart of an embodiment of a data processing method according to an embodiment of the present application. The method is suitable for the client server and can comprise the following steps:

101: a virtual storage device in the client server receives a data storage request sent by a target application.

Wherein the virtual storage device is created by a client server based on the target application.

In practical application, the client server creates a virtual storage device based on a target application program in advance, and obtains a logical storage device generated by the first storage server by establishing a network connection with the first storage server. When the first storage server generates the first logic storage device, a preset address mapping relation between a virtual storage area of the first logic storage device and a physical storage address of at least one physical storage device is stored.

Thus, the target application may consider the storage service to be provided by the virtual storage device, and store the data to be stored in the virtual storage device by sending a storage request to the virtual storage device.

Optionally, a virtual storage drive is run in the client server, and the virtual storage drive component may generate a virtual storage device through sys node registration, so that the target application may perform a read operation by accessing the virtual storage device.

102: And acquiring a first logic storage address through a first logic storage device associated with the virtual storage device.

Wherein the first logical storage device is generated by a first storage server corresponding to the target application.

103: And sending the data storage request to the first storage server based on the first logical storage address so as to store the data to be stored into at least one physical storage device corresponding to the first logical storage address through the first storage server.

The virtual storage device does not actually have a storage address, and thus it is necessary to acquire the storage address through the first logical storage device with which the association relationship is established. However, since the first logical storage device is obtained by mapping the actual physical storage device through a logical partition, the storage address is a logical storage address, but the actual physical storage address may be determined by the storage server based on the logical storage address, so that the data to be stored is stored in at least one physical device.

In practical applications, when the data storage/fetching operation is performed on the first logic storage device in the distributed storage system, the storage/fetching operation on the corresponding physical storage area can be implemented. When the storage operation of the first logic storage device is executed, a first logic storage address of data to be stored is obtained, and the first storage server determines at least one physical storage address corresponding to the first logic storage address according to the preset address mapping relation list, so that the data to be stored is stored in the corresponding at least one physical storage device.

Each physical storage device has a unique device identifier, and each logical storage address in the logical storage devices carries the device identifier of the corresponding physical storage device. When the data to be stored is stored in any one of the physical storage devices, the device identification of the actual storage position of the data to be stored is recorded based on the association relation established between the virtual storage device and the first logic storage device. Even if the first storage server is in fault restart or node migration, when the storage service is acquired through other storage servers, the virtual storage device can reestablish the association relationship between the recorded device identifier and the logic storage device carrying the device identifier. Thus, even if the logical storage device changes, the target application can still read the data through the logical storage device associated with the virtual storage device, and the process application cannot sense the change of the logical storage device.

As an achievable real-time manner, the method may further include:

and the virtual storage equipment receives a data reading request sent by the target application program.

And determining a second logic storage address of the data to be read through the first logic storage device.

And sending the data reading request to the first storage server based on the second logic storage address so as to acquire the data to be read from at least one physical storage device corresponding to the second logic storage address through the first storage server.

And sending the data to be read to the target application program.

Similar to the aforementioned data storage process, the data reading process still requires determining, by the virtual storage device, a second logical storage address to store the data to be read based on the first logical storage device. The first storage server determines at least one physical storage address corresponding to the second logical storage address based on a preset address mapping relation list. And the virtual storage equipment reads the data to be read from the at least one physical storage address, and sends the data to be read to the target application program to complete the data reading process.

In the embodiment of the application, the virtual storage device created based on the target application program and the first storage server are used for generating the first logic storage device to establish the association relation, so that the network connection between the target application program and the storage server is not only established, but also the target application program can indirectly acquire the access/fetch service of the first storage server through the virtual storage device. The association relation between the virtual storage device and the logic storage device is established, the connection between the distributed storage system of the multi-storage node architecture and the target application program of the dual-node architecture is actually established, the seamless connection between the target application program and the existing distributed storage system is realized, the complexity of enterprise user storage service migration is greatly reduced, and therefore data access with high reliability, high storage efficiency and low network delay of the target application program is realized, and higher-quality data storage service is obtained.

In order to implement the data reading operation of the target application program, an association relationship between the virtual storage device and the first logic storage device needs to be established in advance.

Fig. 2 is a schematic flow chart of another embodiment of a data processing method according to an embodiment of the present application. The method may comprise the steps of:

201: a first storage server corresponding to the target application is determined from a plurality of storage servers.

202: And acquiring a first logic storage device generated by the first storage server corresponding to the target application program.

203: And establishing an association relation between the first logic storage device and the virtual storage device.

The embodiment of the application can be applied to a distributed storage structure of a block storage technology and a distributed storage structure of a file storage technology, and is not particularly limited.

Taking ISCSI (INTERNET SMALL Computer SYSTEM INTERFACE, an internet small Computer system interface) and also called an IP-SAN storage technology as an example, in order to implement data interaction between a client server and a storage server, a target device is installed in the storage server, and an initiator is installed in the client server to implement data transmission based on SCSI protocol at an upper layer of IP protocol.

From the foregoing, it is known that a virtual storage device in a client service can be created through a virtual storage driver, and in addition, a target application program can also establish a connection with a first storage server through the virtual storage driver. Specifically, the virtual storage drive obtains the address of the target through the initiator, logs in the target based on the address of the target, the user name and other information, and obtains ISCSI block equipment generated by the target, namely logic storage equipment, wherein the block equipment is standard kernel mode equipment, and then establishes network connection between the target application program and the first storage server through establishing the association relation between the block equipment and the virtual storage equipment. So that the data storage service of the first storage server can be obtained through the virtual storage device.

As an alternative embodiment, the determining, from the plurality of storage servers, the first storage server corresponding to the target application may include:

Acquiring storage demand information of a target application program;

acquiring storage service information of each of the plurality of storage servers;

Determining the service priority of each storage server according to the matching degree of the storage demand information and the storage service information of each storage server;

And selecting the storage server with the highest service priority as the first storage server corresponding to the target application program.

Optionally, in some embodiments, the monitoring, by the monitoring device, the service status of the first storage server may include:

The monitoring device is controlled to send a test request to the first storage server every first preset time;

If the response generated by the first storage server for the test request is not received within a second preset time, determining that the storage service of the first storage server is abnormal; wherein the second preset time is less than or equal to the first preset time.

Optionally, if the monitoring does not receive the response generated by the first storage server for the test request within the second preset time, determining that the storage service of the first storage server is abnormal may include:

If the response generated by the first storage server for the test request is not received within a second preset time, judging whether the network connection with the first storage server is abnormal or not;

and if the network connection is abnormal, determining that the storage service of the first storage server is abnormal.

In practical application, the monitoring device may send a test request to the first storage server every 5 seconds, and if no response generated by the first storage server is received within 1 second, it may determine that a service abnormality occurs, which may be caused by network congestion or network connection terminals, or may be caused by network jitter.

Further, whether to switch the storage node may be determined by judging whether the network connection is normal, and if the network connection terminal is connected to the first storage server, the node switching operation is performed, and the specific operation procedure is as follows.

In practical applications, the distributed storage system may further include a management server for managing the plurality of storage servers. The management server may learn storage service information of each storage server, which may be, for example, network bandwidth, geographic location, number of switch hops, etc.

The client server may obtain storage service information of the plurality of storage servers through the management and control server, and obtain storage requirement information of the target application program, where the storage requirement information may include a network bandwidth requirement, a geographic location of the client server, a network delay requirement, and the like.

Optionally, in some embodiments, the storage service information includes at least one storage service sub-information, and the storage requirement information includes at least one storage requirement sub-information.

The determining the service priority of each storage server according to the matching degree of the vehicle demand information and the storage service information of each storage server may include:

determining a corresponding matching value of at least one matching parameter of each storage server;

determining a weight value of each matching parameter;

and carrying out weighted summation based on the respective weight value of the at least one matching parameter and the corresponding matching value, and calculating the matching degree value of the vehicle demand information and the storage service information of each storage server respectively.

Optionally, the determining the corresponding matching value of the at least one matching parameter of each storage server may include:

and determining the matching value of the storage service sub-information corresponding to each of the at least one matching parameter and the storage requirement sub-information corresponding to each storage server.

For example, the at least one matching parameter may include a bandwidth matching parameter, a zone matching parameter, a latency matching parameter; the matching value of the bandwidth matching parameter can be obtained by calculating the matching degree of the bandwidth information of the target application program and the network bandwidth of each storage server, and the closer the matching value is, the higher the matching value is, and the lower the matching value is, vice versa.

The matching value of the region matching parameter can be obtained by calculation based on the degree of matching between the geographic position of the calculation target application program and the geographic position of each storage server, and the closer the geographic is to the matching value, the higher the geographic is, and vice versa, the lower is.

The matching value of the delay matching parameter can be obtained by calculating the matching degree of the network delay requirement of the target application program and the hop count of the switch of each storage server, wherein the more the switch number is, the lower the delay is, the higher the matching value is, and otherwise, the higher the delay is and the lower the matching value is.

For example, the weight values corresponding to the bandwidth matching parameter, the region matching parameter and the time delay matching parameter are respectively 60, 20 and 20.

Then, for any storage server, the matching degree value is obtained by carrying out weighted summation on the matching value of at least one matching parameter corresponding to the storage server.

That is, the matching degree value of any one storage server and the target application program can be expressed as:

S _i＝A_i*0.6+B_i*0.2+C_i x 0.2; wherein a _i represents a matching value of a bandwidth matching parameter of any one storage server, and B _i represents a matching value of a region matching parameter of any one storage server; c _i represents the matching value of the latency matching parameter for any one storage server.

Wherein, the higher the matching degree value of the storage server is, the higher the service priority of the storage server is, and the priorities of the storage servers are determined according to the order of the matching degree value from large to small.

Optionally, in some embodiments, after determining the service priority of each storage server according to the matching degree between the storage requirement information and the storage service information of each storage server, the method further includes:

monitoring the service state of the first storage server through a monitoring device;

If the storage service of the first storage service is monitored to be abnormal, reducing the service priority of the first storage server;

And if the storage service of the first storage service is monitored to be normal, gradually increasing the service priority of the first storage server based on the matching degree with the target application program.

For example, the matching degree values of the first storage server, the second storage server and the third storage server are 100, 80 and 40, respectively.

In order to avoid the situation that the priority of the storage node changes more severely, so that the node is switched too fast to cause switching jitter, the priority of the first storage server can be adjusted in a slow-down and slow-up mode.

When the first storage server is abnormal, the matching degree value of the first storage server can be reduced to be half of the original value, namely, 50, and the priority is reduced to be the second level. In order to ensure that the priority is not the highest, judging whether the reduced matching degree value is the largest or not, if so, continuing to reduce to half of the current matching degree value; if not maximum, no further adjustment may be necessary.

If the first storage server is restored to normal, the service priority of the first storage server is gradually increased based on the original matching degree with the application program. The matching degree needs to be gradually and slowly increased, and the matching degree can be increased by 0.5 times according to the original mode, namely, the matching degree is increased from 50 to 75 and then is increased from 75 to the highest value of 100.

Optionally, the method may further include:

Monitoring whether the priority of the first storage server changes.

And if the priority of the first storage server is reduced, re-selecting the storage server with the highest priority from the plurality of storage servers as a new first storage server.

Establishing a network connection with the new first storage server.

In order to ensure that the service is not interrupted and the optimal service quality is obtained, the storage server with the highest priority is always selected to provide the data storage service for the target application program.

The establishing of the network connection with the new first storage server may include:

And replacing the first logic storage device associated with the virtual storage device with a second logic storage device generated by the new first storage server corresponding to the target application program.

FIG. 3 is a schematic diagram of establishing a network connection with the new first storage server. Where Host represents the target application, VDISK DRIVER represents the virtual storage drive, vdisk represents the virtual storage device, sdx1 represents the first logical storage device, sdx2 represents the second logical storage device.

In practical application, the standby virtual storage device can be set based on the dual-node structure of the target application program, and the standby virtual storage device can be used for executing the node switching request triggered and generated at the user to be switched from the virtual storage device to the standby virtual storage device. In practice, the client server is arranged in the monitoring device, so that the monitoring result can be automatically changed into the storage server, and the storage failure switching actual target application program is not perceived, so that the user node switching request can trigger the target application program to generate according to the service requirement or the actual switching requirement.

The backup logical storage device may be generated by the first storage server and may also be provided by any other connectable storage server, which is not specifically limited herein.

The first logic storage device carries the device identification of at least one physical storage device corresponding to the logic storage address.

As an optional implementation manner, the establishing the association relationship between the first logical storage device and the virtual storage device may include:

And establishing an association relation between the equipment identification of the at least one physical storage equipment and the virtual storage equipment.

Since the logical partitions of the storage servers to the logical storage devices are not fixed, each storage server may be different to the logical partition of the cloud storage device at the back end. Therefore, the client server and the storage server are disconnected from the network, and if the network connection is re-established with the disconnected storage server, the physical storage area corresponding to the generated logical storage device is also changed. Thus, to ensure that the target application can still read and retrieve stored data after a logical storage device change. Setting a unique device identifier for each physical storage device; by establishing the association relation between the equipment identifier and the virtual storage equipment, after logging in the first storage server each time, the first storage server can be matched with the logic storage equipment carrying the equipment identifier based on the equipment identifier to the target application program, so that a foundation is laid for realizing the transverse expansion of the storage node.

In some embodiments, the replacing the first logical storage device associated with the virtual storage device with the second logical storage device generated by the new first storage server based on the target application may include:

Determining the equipment identifier of at least one physical storage equipment carried by the first logic storage equipment;

acquiring a second logic storage device which is generated by the new first storage server and carries the device identifier;

And replacing the first logic storage device with the second logic storage device carrying the device identifier.

In an actual application, the replacing the first logical storage device with the second logical storage device carrying the device identifier may include:

and cutting off the association relation between the virtual storage device and the first logic storage device and reestablishing the association relation between the virtual storage device and the second logic storage device.

In the embodiment of the application, in order to ensure that the service processing of accessing the target storage application into the distributed storage system is not affected, a virtual storage driving device is introduced, and the association relationship between the virtual storage device created by the virtual driving device and the logic storage device is actually the association relationship with the device identifier of the actual physical storage device, thereby providing possibility for linear expansion of the target node.

In addition, in order to ensure the storage service quality, a service priority is set for each storage server, the storage server with the highest service priority is used as the first storage server, and the data storage service is always provided for the target application program through the storage server with the highest service priority by monitoring the service priority of the first storage server, so that the reading efficiency and the system performance are improved.

FIG. 4 is a flow chart illustrating an embodiment of a data processing method according to the present application. The method is suitable for a storage server and can comprise the following steps:

401: a data storage request sent by a virtual storage device based on a first logical storage address is received.

Wherein the first logical address is obtained by the virtual storage device from a first logical storage device based on the data storage request; the data storage request is sent to the virtual storage device for a target application.

402: Determining a first physical storage address corresponding to the first logical storage address based on a preset address mapping relation;

403: determining at least one physical storage device corresponding to the first physical storage address;

404: and storing the data to be stored into at least one physical storage device corresponding to the first physical storage address.

The embodiment of the application comprises a plurality of storage servers which are deployed based on a distributed architecture, and lays a foundation for realizing linear expansion of storage nodes. When storage nodes are added to the cluster, the overall capacity and performance of the system also expands linearly with it. Therefore, the system has high reliability, low network time delay, high access efficiency and easy expansion performance, and provides better data storage service for target application programs.

Optionally, as an embodiment, the method may further include:

and receiving a data reading request sent by the virtual storage device based on the second logic storage address.

Wherein the second logical address is obtained from the logical storage device by the virtual storage device based on the data read request; the data reading request is sent to the virtual storage device for a target application program.

Determining a second physical storage address corresponding to the second logical storage address based on a preset address mapping relation;

determining at least one physical storage device corresponding to the second physical storage address;

Acquiring data to be read from at least one physical storage device corresponding to the second physical storage address;

And sending the data to be read to the virtual storage device so that the target application program can acquire the data to be read from the virtual storage device.

Optionally, as an embodiment, the method may further include:

receiving a network connection request sent by a target application program in the client server;

Configuring at least one physical storage device for the target application based on the network connection request;

Generating a first logical storage device based on a physical storage address of the at least one physical storage device;

Transmitting a first logic storage device carrying a device identifier of the at least one physical storage device to the client server, so that the client server establishes an association relationship between the virtual storage device and the first logic storage device; wherein the virtual storage device is created by the client server based on the target application.

Optionally, the method of configuring at least one physical storage device for the target application program based on the network connection request may include:

acquiring storage demand information of the target application program based on the network connection request;

at least one physical storage device is configured for the target application based on the storage requirements.

In a practical application, the generating the first logical storage device based on the physical storage address of the at least one physical storage device may include:

Determining a logical storage address corresponding to the physical storage address of the at least one physical storage device based on the preset address mapping relationship;

The first logical storage device is generated based on the logical storage address and the device identification of the at least one physical storage device.

The foregoing details of the implementation of the embodiments of the present application have been described in detail, and are not repeated herein.

FIG. 5 is a schematic diagram illustrating an embodiment of a data processing system according to the present application. The system may include a client server 501 and a plurality of storage servers 502 deployed in a distributed manner; wherein the client server 501 includes a target application 5011;

The client server 501 is configured to determine, from the plurality of storage servers 502, a first storage server 5021 corresponding to the target application 5011; acquiring a first logic storage device 5012 generated by the first storage server 5021 corresponding to the target application program; creating a virtual storage device 5013 based on the target application program and establishing an association of the first logical storage device 5012 with the virtual storage device 5013.

The first storage server 5021 is configured to receive a network connection request sent by the target application 5011; configuring at least one physical storage device for the target application 5011 based on the network connection request; generating a first logical storage device 5012 based on the physical storage address of the at least one physical storage device; the first logical storage device 5012 carrying the device identification of the at least one physical storage device is sent to the client server 501.

The application passes through in real time, in order to ensure that the service processing of accessing the target storage application into the distributed storage system is not influenced, a virtual storage driving device is introduced, the association relation between the virtual storage equipment created by the virtual driving device and the logic storage equipment is actually the association relation established with the equipment identification of the actual physical storage equipment, thereby providing possibility for linear expansion of the target node and laying foundation for the data storage service based on multiple nodes.

Optionally, as an implementation manner, the target application program is configured to send a data storage request to the virtual storage device.

The virtual storage device is used for acquiring a first logic storage address through the first logic storage device based on the data request; and sending the data storage request to the first storage server based on the first logical storage address.

The first storage server is further configured to store data to be stored in at least one physical storage device corresponding to the first logical storage address.

Optionally, as an implementation manner, the target application program is further configured to send a data reading request to the virtual storage device; and receiving the data to be read sent by the virtual storage equipment.

The virtual storage device is further configured to determine, through the first logical storage device, a second logical storage address of data to be read based on the data read request; transmitting the data read request to the first storage server based on the second logical storage address; and receiving the read data of the tape sent by the first storage server and sending the data to be read to the target application program.

The first storage server is further configured to obtain the data to be read from at least one physical storage device corresponding to the second logical storage address; and sending the data to be read to the virtual storage device.

FIG. 6 is a schematic diagram of a data processing system based on ISCSI distributed storage, in accordance with the present application. The system may include, in addition to the client server 501, the plurality of storage servers 502, the target application 5011, the first storage server 5021, and the first logical storage device 5012 and the virtual storage device 5013, an ISCSI initiator 5014 in the client server 501 and ISCSI targets 5022 in the plurality of storage servers 502.

The ISCSI initiator 5014 is configured to establish a network connection with the ISCSI target 5022.

The ISCSI target 5022 is configured to perform a data store/read operation based on the target application generating a data store/read request.

Fig. 7 is a schematic diagram showing a structure of an embodiment of a data processing apparatus according to the present application. The device application client service may include:

The first receiving module 701 is configured to receive, by using a virtual storage device in a client server, a data storage request sent by a target application program.

The first obtaining module 702 is configured to obtain, through a first logical storage device associated with the virtual storage device, a first logical storage address.

A first sending module 703, configured to send the data storage request to the first storage server based on the first logical storage address, so that the data to be stored is stored in at least one physical storage device corresponding to the first logical storage address by the first storage server.

As an achievable real-time manner, the apparatus may further include:

And the second receiving module is used for receiving the data reading request sent by the target application program by the virtual storage equipment.

And the second acquisition module is used for determining a second logic storage address of the data to be read through the first logic storage device.

And the second sending module is used for sending the data reading request to the first storage server based on the second logic storage address so as to acquire the data to be read from at least one physical storage device corresponding to the second logic storage address through the first storage server.

And the third sending module is used for sending the data to be read to the target application program.

As an achievable real-time approach, the apparatus may comprise:

And the first determining module is used for determining a first storage server corresponding to the target application program from a plurality of storage servers.

And the third acquisition module is used for acquiring the first logic storage device generated by the first storage server corresponding to the target application program.

And the association establishing module is used for establishing the association relation between the first logic storage device and the virtual storage device.

As an alternative embodiment, the first determining module may include:

The storage demand acquisition unit is used for acquiring storage demand information of the target application program;

a storage service acquisition unit configured to acquire storage service information of each of the plurality of storage servers;

And the priority determining unit is used for determining the service priority of each storage server according to the matching degree of the storage demand information and the storage service information of each storage server.

And the selecting unit is used for selecting the storage server with the highest service priority as the first storage server corresponding to the target application program.

Optionally, in some embodiments, the priority determining unit may further include:

And the monitoring unit is used for monitoring the service state of the first storage server through the monitoring device. Triggering a degradation unit if the storage service of the first storage service is monitored to be abnormal; and triggering an upgrading unit if the storage service of the first storage service is monitored to be recovered to be normal.

A downgrade unit, configured to reduce a service priority of the first storage server;

and the upgrading unit is used for gradually improving the service priority of the first storage server based on the matching degree with the application program.

Optionally, in some embodiments, the monitoring unit may specifically be configured to:

Optionally, if the monitoring does not receive the response generated by the first storage server for the test request within the second preset time, determining that the storage service of the first storage server is abnormal may be specifically used to:

Optionally, the apparatus may further include:

and the priority detection module is used for monitoring whether the priority of the first storage server changes. Triggering a reselection unit if the priority of the first storage server decreases.

And the reselection unit is used for reselecting the storage server with the highest priority from the plurality of storage servers as a new first storage server.

And the connection establishment module is used for establishing network connection with the new first storage server.

The connection establishment module may be specifically configured to:

The first logic storage device carries the device identification of at least one physical storage device corresponding to the logic storage address. As an optional implementation manner, the establishing the association relationship between the first logical storage device and the virtual storage device may specifically be used to:

In some embodiments, the replacing the first logical storage device associated with the virtual storage device with the second logical storage device generated by the new first storage server based on the target application may be specifically configured to:

In an actual application, the replacing the first logical storage device with the second logical storage device carrying the device identifier may specifically be used to:

Fig. 8 shows a schematic structural diagram of another embodiment of a data processing apparatus according to the present application. The device is applied to a storage server and can comprise:

the storage request receiving module 801 is configured to receive a data storage request sent by a virtual storage device based on a first logical storage address.

The first storage address determining module 802 is configured to determine, based on a preset address mapping relationship, a first physical storage address corresponding to the first logical storage address.

A first storage device determining module 803, configured to determine at least one physical storage device corresponding to the first physical storage address.

A storage module 804, configured to store data to be stored in at least one physical storage device corresponding to the first physical storage address.

Optionally, as an embodiment, the apparatus may further include:

And the read request receiving module is used for receiving a data read request sent by the virtual storage device based on the second logic storage address.

The second storage address determining module is used for determining a second physical storage address corresponding to the second logical storage address based on a preset address mapping relation.

The second storage device determining module is used for determining at least one physical storage device corresponding to the second physical storage address;

The data reading module is used for acquiring data to be read from at least one physical storage device corresponding to the second physical storage address;

and the data to be read sending module is used for sending the data to be read to the virtual storage device so that the target application program can acquire the data to be read from the virtual storage device.

Optionally, as an embodiment, the apparatus may further include:

And the connection request receiving module is used for receiving a network connection request sent by the target application program in the client server.

And the configuration module is used for configuring at least one physical storage device for the target application program based on the network connection request.

And the logic storage device generation module is used for generating a first logic storage device based on the physical storage address of the at least one physical storage device.

And the logic storage device sending module is used for sending the first logic storage device carrying the device identifier of the at least one physical storage device to the client server so that the client server establishes the association relationship between the virtual storage device and the first logic storage device.

Wherein the virtual storage device is created by the client server based on the target application.

Optionally, the configuration module may specifically be configured to:

In practical applications, the logic storage device generating module may specifically be configured to:

Fig. 9 is a schematic structural diagram of an embodiment of a client server according to an embodiment of the present application. The client server may include a processing component 901 and a storage component 902.

The storage component 902 is configured to store one or more computer instructions; the one or more computer instructions are configured to be invoked for execution by the processing component 901.

The processing assembly 901 may be configured to:

Wherein the processing component 901 may include one or more processors to execute computer instructions to perform all or part of the steps in the methods described above. Of course, the processing component may also be implemented as one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic elements for executing the methods described above.

The storage component 902 is configured to store various types of data to support operations in a server. The memory component may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

Of course, the client server may necessarily also include other components, such as input/output interfaces, communication components, and the like.

The input/output interface provides an interface between the processing component and a peripheral interface module, which may be an output device, an input device, etc.

The communication component is configured to facilitate communication between the client server and other devices, either wired or wireless, such as communication with a terminal.

The embodiment of the application also provides a computer readable storage medium storing a computer program, which when executed by a computer, can implement the data processing method of the embodiment shown in fig. 1 and 2.

Fig. 10 is a schematic structural diagram of an embodiment of a storage server according to an embodiment of the present application. The computer device may include a processing component 1001 and a storage component 1002.

The storage component 1002 is configured to store one or more computer instructions; the one or more computer instructions are configured to be invoked by the processing component 1001 for execution.

The processing component 1001 may be configured to:

Wherein the processing component 1001 may include one or more processors to execute computer instructions to perform all or part of the steps in the methods described above. Of course, the processing component may also be implemented as one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic elements for executing the methods described above.

The storage component 1002 is configured to store various types of data to support operations in a server. The memory component may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

Of course, the storage server may necessarily include other components, such as input/output interfaces, communication components, and the like.

The communication component is configured to facilitate communication between the storage server and other devices, either wired or wireless, such as communication with a terminal.

The embodiment of the application also provides a computer readable storage medium storing a computer program, which when executed by a computer can implement the data processing method of the embodiment shown in fig. 4.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of data processing, comprising:

Transmitting the data storage request to the first storage server based on the first logical storage address so as to store data to be stored into at least one physical storage device corresponding to the first logical storage address through the first storage server;

wherein the virtual storage device establishes an association with the first logical storage device as follows: acquiring storage demand information of a target application program; acquiring storage service information of each of a plurality of storage servers; determining the service priority of each storage server according to the matching degree of the storage demand information and the storage service information of each storage server; selecting a storage server with the highest service priority as a first storage server corresponding to the target application program; acquiring a first logic storage device generated by the first storage server corresponding to the target application program; and establishing an association relation between the first logic storage device and the virtual storage device.

2. The method according to claim 1, wherein the method further comprises:

The virtual storage device receives a data reading request sent by the target application program;

determining a second logic storage address of data to be read through the first logic storage device;

Transmitting the data reading request to the first storage server based on the second logical storage address so as to acquire the data to be read from at least one physical storage device corresponding to the second logical storage address through the first storage server;

and sending the data to be read to the target application program.

3. The method of claim 1, wherein the first logical storage device carries a device identifier of at least one physical storage device corresponding to a logical storage address of the first logical storage device;

The establishing the association relationship between the first logic storage device and the virtual storage device includes:

4. The method according to claim 1, wherein after determining the service priority of each storage server according to the matching degree between the storage requirement information and the storage service information of each storage server, the method further comprises:

5. The method of claim 4, wherein the monitoring, by the monitoring device, of the service status of the first storage server comprises:

6. The method of claim 5, wherein determining that an exception occurred to the storage service of the first storage server if the response generated by the first storage server to the test request was not received within a second preset time is monitored comprising:

7. The method of claim 5, wherein the method further comprises:

Monitoring whether the priority of the first storage server changes;

if the priority of the first storage server is reduced, the storage server with the highest priority is reselected from the plurality of storage servers to serve as a new first storage server;

establishing a network connection with the new first storage server.

8. The method of claim 7, wherein the establishing a network connection with the new first storage server comprises:

9. The method of claim 8, wherein replacing the first logical storage device associated with the virtual storage device with a second logical storage device generated by the new first storage server based on the target application comprises:

10. The method of claim 9, wherein the replacing the first logical storage device with the second logical storage device carrying the device identification comprises:

11. A data processing method, applied to a storage server, comprising:

Receiving a data storage request sent by virtual storage equipment based on a first logic storage address; wherein the first logical storage address is obtained by the virtual storage device from the first logical storage device based on the data storage request; the data storage request is sent to the virtual storage device for a target application program;

storing data to be stored into at least one physical storage device corresponding to the first physical storage address;

12. The method as recited in claim 11, further comprising:

Receiving a data reading request sent by the virtual storage device based on a second logic storage address; wherein the second logical storage address is obtained from the logical storage device by the virtual storage device based on the data read request; the data reading request is sent to the virtual storage device for the target application program;

Determining a second physical storage address corresponding to the second logical storage address based on the preset address mapping relation;

13. The method of claim 11, wherein the method further comprises:

Receiving a network connection request sent by a target application program in a client server;

14. The method of claim 13, wherein the configuring at least one physical storage device for the target application based on the network connection request comprises:

15. The method of claim 13, wherein the generating a first logical storage device based on the physical storage address of the at least one physical storage device comprises:

16. A data processing system comprising a client server and a plurality of storage servers in a distributed deployment; wherein the client server comprises a target application program;

The client server is used for determining a first storage server corresponding to the target application program from the plurality of storage servers; acquiring a first logic storage device generated by the first storage server corresponding to the target application program; creating a virtual storage device based on the target application program and establishing an association relation between the first logic storage device and the virtual storage device; wherein the virtual storage device establishes an association with the first logical storage device as follows: acquiring storage demand information of a target application program; acquiring storage service information of each of the plurality of storage servers; determining the service priority of each storage server according to the matching degree of the storage demand information and the storage service information of each storage server; selecting a storage server with the highest service priority as a first storage server corresponding to the target application program; acquiring a first logic storage device generated by the first storage server corresponding to the target application program; establishing an association relationship between the first logic storage device and the virtual storage device;

The first storage server is used for receiving a network connection request sent by the target application program; configuring at least one physical storage device for the target application based on the network connection request; generating a first logical storage device based on a physical storage address of the at least one physical storage device; and sending a first logic storage device carrying the device identification of the at least one physical storage device to the client server.

17. The system of claim 16, wherein the system further comprises a controller configured to control the controller,

The target application program is used for sending a data storage request to the virtual storage device;

The virtual storage device is used for acquiring a first logic storage address through the first logic storage device based on the data storage request; transmitting the data storage request to the first storage server based on the first logical storage address;

18. The system of claim 16, wherein the system further comprises a controller configured to control the controller,

The target application program is further used for sending a data reading request to the virtual storage device; receiving data to be read sent by the virtual storage equipment;

The virtual storage device is further configured to determine, through the first logical storage device, a second logical storage address of data to be read based on the data read request; transmitting the data read request to the first storage server based on the second logical storage address; receiving the data to be read sent by the first storage server and sending the data to be read to the target application program;

19. A data processing apparatus, comprising:

The first acquisition module is used for acquiring a first logic storage address through a first logic storage device associated with the virtual storage device; the first logic storage device is generated by a first storage server corresponding to the target application program; the virtual storage device establishes an association with the first logical storage device as follows: acquiring storage demand information of a target application program; acquiring storage service information of each of a plurality of storage servers; determining the service priority of each storage server according to the matching degree of the storage demand information and the storage service information of each storage server; selecting a storage server with the highest service priority as a first storage server corresponding to the target application program; acquiring a first logic storage device generated by the first storage server corresponding to the target application program; establishing an association relationship between the first logic storage device and the virtual storage device;

20. A data processing apparatus, comprising:

The storage request receiving module is used for receiving a data storage request sent by the virtual storage device based on the first logic storage address; wherein the first logical storage address is obtained by the virtual storage device from the first logical storage device based on the data storage request; the data storage request is sent to the virtual storage device for a target application program; the virtual storage device establishes an association with the first logical storage device as follows: acquiring storage demand information of a target application program; acquiring storage service information of each of a plurality of storage servers; determining the service priority of each storage server according to the matching degree of the storage demand information and the storage service information of each storage server; selecting a storage server with the highest service priority as a first storage server corresponding to the target application program; acquiring a first logic storage device generated by the first storage server corresponding to the target application program; establishing an association relationship between the first logic storage device and the virtual storage device;

21. A client server comprising a processing component and a storage component; the storage component is used for storing one or more computer instructions, wherein the one or more computer instructions are used for being called by the processing component for execution;

The processing assembly is configured to:

Acquiring a first logical storage address through a first logical storage device associated with the virtual storage device; the first logic storage device is generated by a first storage server corresponding to the target application program; the virtual storage device establishes an association with the first logical storage device as follows: acquiring storage demand information of a target application program; acquiring storage service information of each of a plurality of storage servers; determining the service priority of each storage server according to the matching degree of the storage demand information and the storage service information of each storage server; selecting a storage server with the highest service priority as a first storage server corresponding to the target application program; acquiring a first logic storage device generated by the first storage server corresponding to the target application program; establishing an association relationship between the first logic storage device and the virtual storage device;

22. A storage server comprising a processing component and a storage component; the storage component is used for storing one or more computer instructions, wherein the one or more computer instructions are used for being called by the processing component for execution;

The processing assembly is configured to:

Receiving a data storage request sent by virtual storage equipment based on a first logic storage address; wherein the first logical storage address is obtained by the virtual storage device from the first logical storage device based on the data storage request; the data storage request is sent to the virtual storage device for a target application program; the virtual storage device establishes an association with the first logical storage device as follows: acquiring storage demand information of a target application program; acquiring storage service information of each of a plurality of storage servers; determining the service priority of each storage server according to the matching degree of the storage demand information and the storage service information of each storage server; selecting a storage server with the highest service priority as a first storage server corresponding to the target application program; acquiring a first logic storage device generated by the first storage server corresponding to the target application program; establishing an association relationship between the first logic storage device and the virtual storage device;