WO2019062572A1

WO2019062572A1 - Data processing method, device and system

Info

Publication number: WO2019062572A1
Application number: PCT/CN2018/105965
Authority: WO
Inventors: 焦辉; 董欢庆
Original assignee: 华为技术有限公司
Priority date: 2017-09-27
Filing date: 2018-09-17
Publication date: 2019-04-04
Also published as: CN107786638B; CN107786638A

Abstract

The embodiments of the present invention relate to the field of computer storage, and provided thereby is a data processing method, which may avoid to a certain extent conflict occurring between a reading object and a deleting object. The method comprises: a container server in an object storage system receiving a deletion query message sent by a proxy server, the deletion query message comprising identification information of a container to which a first target object belongs and identification information of the first target object; the container server querying location information of the first target object in an object list of the container server according to the identification information of the container to which the first target object belongs and the identification information of the first target object; if the location information of the first target object is found in the object list, the container server deleting the location information of the first target object in the object list and writing the location information of the first target object into a deletion log of the container server; and the container server sending a deletion response message to the proxy server.

Description

Data processing method, device and system

Technical field

Embodiments of the present invention relate to the field of computer storage, and in particular, to a data processing method, apparatus, and system.

Background technique

With the rapid development of computer storage technology, the object storage system quickly becomes the mainstream storage system for storing data in the computer storage field with its advantages of high-speed access characteristics, high reliability and scalability.

In an object storage system, when a user uses a terminal (such as a browser on a terminal or other application), the terminal can use the terminal to write data to the object storage system (which can be called an object, the object includes metadata and the data itself), and deletes The object stores some data in the system or reads some data from the object storage system. As shown in FIG. 1, the object storage system may include a terminal 10, a proxy server 11, a container server 12, and an object server 13. When the terminal 10 reads the data (ie, the object) in the object storage system, the proxy server 11 in the object storage system receives the object read request sent by the terminal 10, and the proxy server 11 sends the query to the container server 12 of the object storage system. The location information message, and then the container server 12 can query the location information of the object in the database in the container server 12. If the location information of the object is queried, the proxy server 11 sends the location information of the object to the object server 13, The object server 13 reads the object based on the position information of the object, and returns the read object to the terminal 10.

When the terminal 10 deletes a certain data (ie, an object) in the object storage system, the proxy server 11 receives the object deletion request sent by the terminal 10. Similarly, the proxy server 11 sends a message to the container server 12 to query the location information of the object. The container server 12 then queries the location information of the object in the database of the container server 12. If the location information of the object is queried, the location information of the object is sent to the proxy server 11, and the proxy server 11 sends the object to the object server 13. Location information, and then the object server 13 deletes the object in the object server 13 according to the location information of the object, and returns a delete response message to the proxy server 11, and the proxy server 11 sends the container server 12 to delete the container. A request message for the location information of the object in the server 12, whereby the container server 12 deletes the location information of the object in the database.

However, in the above method, when a request for reading an object (for example, the first object) and a request for deleting the object are concurrently present in the object storage system, since the object server needs to be in the process of deleting the first object, The object is marked as deleted. If the object server first receives the location information indicating the deletion of the first object of the first object, and then receives the location information indicating the first object that reads the first object, because the object server is according to the first The location information of the object may have been marked for deletion of the first object. As a result, the object server may not be able to read the first object according to the location information of the first target (ie, the object fails to be read), thus causing There is a conflict between reading the first object and deleting the first object.

Summary of the invention

The present application provides a data processing method, apparatus, and system, which can avoid conflicts between a read object and a deleted object to a certain extent.

To achieve the above objectives, the present application adopts the following technical solutions:

The first aspect provides a data processing method, where the method includes: the container server receives a delete query message sent by the proxy server, where the delete query message includes the identifier information of the container to which the first target object belongs and the identifier of the first target object. The container server then queries the location information of the first target object in the object list of the container server according to the identifier information of the container to which the first target object belongs and the identification information of the first target object; if the first target is queried in the object list The location information of the object, the container server deletes the location information of the first target object in the object list, and writes the location information of the first target object into the deletion log of the container server; and the container server sends a deletion response message to the proxy server. The object list includes location information of the first type of object, the first type of object is an object stored in the object server that is not marked and deleted, and the location information of the first type of object is used to indicate that the object is in the object server. The location, the deletion log contains location information of the second type of object, and the second type of object is the location information of the object stored in the object server that is marked and deleted.

The data processing method provided by the present application, in deleting an object in the object storage system (for example, the first target object), the container server may delete the location information of the first target object from the object list, and The location of the first target object is newly written into the deletion log of the container server. It can be seen that the container server does not need to send the location information of the first target object to the proxy server, and the proxy server does not need to further locate the location information of the first target object. Is sent to the object server, so that the object server does not need to delete the first target object in the object server, so when deleting the first target object in the object storage system and the first in the read object storage system When the target object occurs concurrently, the first target object in the object server is not marked as deleted during the process of deleting the first target object, so the first target object can be read in the object server, and Compared with the prior art, it can be avoided that the first target object in the object server is Record for deletion, reading problems caused by the failure of the first target object, that is, avoid reading the first target object and delete the conflict first target object.

Further, since the container server does not need to send the location information of the first target object to the proxy server, the proxy server does not need to send the location information of the first target object to the object server, so that the object server does not need to be in the object. The first target object in the server is marked for deletion, so that the process of deleting the first target object can be simplified, and the communication resources between the servers can be saved.

In a first optional implementation manner of the first aspect, the location information of the first target object may include a file identifier of the first target object, an offset of the first target object, and a size of the first target object. The file identifier indicates that at least two objects are included in the file; or the location information of the first target object may include an ID of the first target object.

In the present application, according to the first target object written in the object storage system, the stored method (which can be understood as the method of writing) is different, and the content information of the first target object may be different, and for different storage methods, The above data processing method can implement the operation of deleting the first target object, and can simplify the process of deleting the first target object, and can save communication resources between servers.

In a second optional implementation manner of the first aspect, the data processing method provided by the present application may further include: the container server receives a read query message sent by the proxy server, where the read query message includes the second target object The identifier information of the container and the identifier information of the second target object; the container server then queries the object list for the location information of the second target object according to the identifier information of the container to which the second target object belongs and the identifier information of the second target object; If the location information of the second target object is queried in the object list, the container server sends the location information of the second target object to the proxy server, to instruct the proxy server to read the second target from the object server according to the location information of the second target object. The object, the location information of the second target object includes a file identifier of the second target object, an offset of the second target object, and a size of the second target object, and the file indicated by the file identifier includes at least two objects.

In the present application, objects in the object storage system can be smoothly read by the above method.

In a third optional implementation manner of the first aspect, if the read query message received by the container server includes the identifier information of the container to which the first target object belongs and the identifier information of the first target object, deleting the object in the container server Before the location information of the first target object in the list, and the location information of the first target object is written in the deletion log of the container server, the data processing method provided by the application may further include: the container server reads the number in the object list. The location information of a target object.

In the present application, in the object storage system, when an object is read and the object is deleted concurrently (that is, it can be understood that the object storage system processes the event of reading an object, and also needs to process the event of deleting the object) Taking the first target object as an example, the container server reads the location information of the first target object from the object list after the container server writes the location information of the first target object to the delete log of the container server. Taking the location information of the first target object in the object list, in the process of reading the first target object and deleting the first target object, even if the location information of the first target object in the container server has been deleted, And the location information of the first target object has been written into the deletion log, and the first target object can also be read this time.

Compared with the prior art, the location information of the first target object in the container server has been deleted, and the location information of the first target object has been written into the deletion log, and the first target object can also be read this time. The reason is: in the process of deleting the first target object, the location information of the first target object may be written in the deletion log to indicate that the first target object is marked for deletion, and no need to be in the object server. The first target object is marked as deleted. Therefore, the first target object still exists in the object server and has not been reclaimed. Thus, the object server may obtain the first target object according to the read query request message according to the container server. Location information, the first target object is read in the object server. It can be seen that the data processing method provided by the embodiment of the present invention can avoid conflicts between the read object and the deleted object to a certain extent (refers to when the read object and the deleted object occur concurrently, the operation of deleting the object firstly causes the read object to be read. The operation could not read the object)

In a fourth optional implementation manner of the first aspect, the data processing method provided by the application may further include: the container server sending a garbage collection request message to the object server, where the garbage collection request message includes deleting the target object in the log Location information; and the container server receives the garbage collection response message sent by the object server, and the container server deletes the location information of the target object in the deletion log, wherein the garbage collection response message indicates that the target server has deleted the target object.

In the present application, after the container server deletes the target object (ie, the location information of the target object is deleted from the object list, and the location information of the target object is written to the deletion log), the container server may also delete the target object according to the target object. The location information requests the object server to delete (ie, recycle) the content of the target object in the object server, so that the container server deletes the location information of the target object in the deletion log, thereby completely deleting the target object in the object storage system.

In a fifth optional implementation manner of the first aspect, the data processing method provided by the application may further include: the container server receives the location update message sent by the proxy server; and the container server updates the third target according to the location update message. Location information of the object, the location update message is used to update the location information of the third target object, where the location update message includes the identifier information of the container to which the third target object belongs, the identifier information of the third target object, and the third target object. First location information.

In a sixth optional implementation manner of the first aspect, the method for updating the location information of the third target object by the container server according to the location update message may include: the container server according to the identifier information of the container to which the third target object belongs, The identification information of the third target object is used to query the location information of the third target object in the object list of the container server; if the location information of the third target object is not queried in the object list, the container server will be the third in the location update message. The first location information of the target object is written to the object list; if the location information of the third target object is queried in the object list, the location information of the third target object is the second location information, and the second target object is the second location information. The location information is different from the first location information of the third target object in the location update message, the container server replaces the second location information of the third target object with the first location information of the third target object, and the third target object The second location information is written to the delete log of the container server.

In the present application, in the object storage system, the container server may update location information of an object (eg, the third target object described above) in the object storage system (eg, update the location of the third target object after writing the third target object) Information), in this way, when the third target object is subsequently read or deleted, the location information of the accurate third target object can be queried from the container server to correctly read the third target object, or delete the third object. target.

The second aspect provides a data processing method, the method may include: the object server receives a request message of the upload target object sent by the proxy server, where the request message of the upload target object includes a third target object; The third target object is written in the object server; and the object server sends the location information of the third target object to the proxy server, where the location information of the third target object includes the file identifier of the third target object and the offset of the third target object And the size of the third target object, the file identifier indicates that at least two objects are included in the file.

The data processing method provided by the present application, in the process of uploading an object (for example, may be referred to as a third target object) to the object storage system, because the object server writes the third target object into the file system file of the object server, The file may include at least two objects. It can be known that when multiple small objects are written in the object storage system, multiple small objects can be written into one file, thereby saving the storage resources of the file system of the object server. Improve the performance of the object server.

In a first optional implementation manner of the second aspect, the data processing method provided by the present application may further include: the object server receives location information of the second target object sent by the proxy server; and the object server is configured according to the second target object. The location information reads the second target object; and the object server sends a second target object to the proxy server.

In a second optional implementation manner of the second aspect, the data processing method provided by the application may further include: receiving, by the object server, a garbage collection request message sent by the container server, where the garbage collection request message includes a deletion log of the container server Position information of the first target object; and the object server deletes the first target object according to the location information of the first target object; and the object server sends a garbage collection response message to the container server, the garbage collection response message indicating that the first target has been deleted target.

For a description of the technical effects of the second optional implementation of the second aspect, refer to the description of the technical effects of the fourth optional implementation of the foregoing first aspect, and details are not described herein again.

In a third aspect, the present application provides a data processing method, the method may include: the proxy server sends a request message for uploading a target object to the object server, where the request message of the upload target object includes a target object; and then the proxy server receives the object server and sends the First location information of the target object, the first location information of the target object includes a file identifier of the target object, an offset of the target object, and a size of the target object, where at least two objects are stored in the file indicated by the file identifier; And the proxy server sends a location update message to the container server, to instruct the container server to update the location information of the target object in the object list of the container server according to the location update message, where the location update message includes the identifier information and the target of the container to which the target object belongs. The identification information of the object and the first location information of the target object.

In a first optional implementation manner of the third aspect, the data processing method provided by the present application may further include: receiving, by the proxy server, a delete response message sent by the container server, where the delete response message is a target of the container service deletion target list The location information of the object, and the location information of the target target object is sent after being written to the delete log of the container server.

For the technical effects of the third aspect and its optional implementation, reference may be made to the first aspect and various optional implementation manners thereof, and related descriptions of the technical effects of the second aspect and various optional implementation manners thereof, Let me repeat.

In a fourth aspect, a container server is provided, and the container server can include: a receiving module, a query module, an update module, and a sending module. The receiving module may be configured to receive the delete query message sent by the proxy server, where the delete query message includes the identifier information of the container to which the first target object belongs and the identifier information of the first target object; the query module may be configured to receive according to the receiving module. The identification information of the container to which the first target object belongs and the identification information of the first target object, and querying the location information of the first target object in the object list of the container server, where the object list includes location information of the first type of object, The first type of object is an object stored in the object server that is not marked for deletion, and the location information of the first type object is used to indicate the location of the object in the object server; the update module can be used to query the object list in the query module. Go to the location information of the first target object, delete the location information of the first target object in the object list, and write the location information of the first target object into the deletion log of the container server, where the deletion log includes the second type of object Location information, the second type of object is marked, deleted, object service Position information stored in the object; transmitting module may be configured to send a message deletion response to the proxy server.

In a first optional implementation manner of the fourth aspect, the location information of the first target object includes a file identifier of the first target object, an offset of the first target object, and a size of the first target object, where The file identifier indicates that at least two objects are included in the file; or the location information of the first target object includes the ID of the first target object.

In a second optional implementation manner of the fourth aspect, the receiving module may be further configured to receive a read query message sent by the proxy server, where the read query message includes the identifier information of the container to which the second target object belongs and the first The identifier information of the second target object may be used to query the location information of the second target object in the object list according to the identifier information of the container to which the second target object belongs and the identifier information of the second target object received by the receiving module. The location information of the second target object includes a file identifier of the second target object, an offset of the second target object, and a size of the second target object, where the file indicated by the file identifier includes at least two objects; the sending module The container server may further send the location information of the second target object to the proxy server, to indicate the location information of the second target object according to the location information of the second target object from the object server. Read the second target object in .

In a third optional implementation manner of the fourth aspect, the container server provided by the present application may further include a reading module. The reading module may be configured to: if the read query message received by the receiving module includes the identifier information of the container to which the first target object belongs and the identifier information of the first target object, the first target object in the object list is deleted in the update module The location information, and the location information of the first target object in the object list is read before the location information of the first target object is written in the deletion log of the container server.

In a fourth optional implementation manner of the fourth aspect, the sending module may be further configured to send a garbage collection request message to the object server, where the garbage collection request message includes deleting location information of the target object in the log; The module may be further configured to receive a garbage collection response message sent by the object server, where the garbage collection response message indicates that the target server has deleted the target object; and the update module may further be configured to delete the location information of the target object in the deletion log.

In a fifth optional implementation manner of the fourth aspect, the receiving module is further configured to receive a location update message sent by the proxy server, where the location update message is used to update location information of the third target object, where the location update message is The identifier information of the container to which the third target object belongs, the identification information of the third target object, and the first location information of the third target object are included; the update module may be further configured to update the third according to the location update message received by the receiving module. The location information of the target object.

In a sixth optional implementation manner of the fourth aspect, the query module may be further configured to query the object list of the container server according to the identifier information of the container to which the third target object belongs and the identifier information of the third target object. Position information of the third target object; the update module is specifically configured to: when the query module does not query the location information of the third target object in the object list, write the first location information of the third target object in the location update message to the object a list, if the query module queries the location information of the third target object in the object list, the location information of the third target object is the second location information, and the second location information of the third target object and the third location information in the location update message The first location information of the target object is different, the second location information of the third target object is replaced with the first location information of the third target object, and the second location information of the third target object is written into the deletion log of the container server. .

In a fifth aspect, an object server is provided, which may include a receiving module, a write data module, and a sending module. The receiving module may be configured to receive a request message of the upload target object sent by the proxy server, where the request message of the upload target object includes a third target object, and the write data module may be configured to write the third target object received by the receiving module. In the object server, the sending module may be configured to send, to the proxy server, location information of the third target object, where the location information of the third target object includes a file identifier of the third target object, an offset of the third target object, and a third The size of the target object, the file identifier indicates that at least two objects are included in the file.

In a first optional implementation manner of the fifth aspect, the object server provided by the application may further include a reading module. The receiving module may be further configured to receive location information of the second target object sent by the proxy server; the reading module may be configured to read the second target object according to the location information of the second target object received by the receiving module; Used to send a second target object to the proxy server.

In a second optional implementation manner of the fifth aspect, the object server provided by the application may further include a deletion module. The receiving module may be further configured to receive a garbage collection request message sent by the container server, where the garbage collection request message includes location information of the first target object in the deletion log of the container server; the deletion module may be configured to receive according to the receiving module. The location information of the first target object deletes the first target object; the sending module may be further configured to send a garbage collection response message to the container server, where the garbage collection response message indicates that the first target object has been deleted.

In a sixth aspect, a proxy server is provided, which can include a sending module and a receiving module. The sending module may be configured to send a request message for uploading the target object to the object server, where the request message of the upload target object includes a target object, and the receiving module may be configured to receive first location information of the target object sent by the object server, where the target The first location information of the object includes a file identifier of the target object, an offset of the target object, and a size of the target object, where at least two objects are stored in the file indicated by the file identifier; the sending module may be further configured to send to the container server. a location update message to instruct the container server to update the location information of the target object in the object list of the container server according to the location update message, where the location update message includes the identification information of the container to which the target object belongs, the identification information of the target object, and the target object. First location information.

In a first optional implementation manner of the sixth aspect, the receiving module is further configured to receive a deletion response message sent by the container server, where the deletion response message is a location information of the target object in the container service deletion object list, and The location information of the target target object is sent after being written to the delete log of the container server.

In a seventh aspect, a container server is provided, the container server can include a processor and a memory coupled to the processor. This memory can be used to store computer instructions. When the container server is running, the processor executes the computer instructions stored by the memory to cause the container server to perform the data processing method of any of the above first aspects and various alternative implementations thereof.

In an eighth aspect, a computer readable storage medium is provided, the computer readable storage medium comprising computer instructions. When the computer instruction is run on the container server, the container server is caused to perform the data processing method of any of the above first aspects and various alternative implementations thereof.

In a ninth aspect, a computer program product comprising computer instructions for causing a container server to perform any of the first aspect described above and various alternative implementations thereof when the computer program product is run on a container server The data processing method described.

For the description of the technical effects of the seventh aspect to the ninth aspect, reference may be made to the related description of the technical effects of the first aspect, and details are not described herein again.

In a tenth aspect, an object server is provided, the object server can include a processor and a memory coupled to the processor. This memory can be used to store computer instructions. When the object server is running, the processor executes the computer instructions stored by the memory to cause the object server to perform the data processing method of any of the above second aspects and various alternative implementations thereof.

In an eleventh aspect, a computer readable storage medium is provided, the computer readable storage medium comprising computer instructions. When the computer instruction is run on the object server, the object server is caused to perform the data processing method of any of the above second aspects and various alternative implementations thereof.

According to a twelfth aspect, a computer program product comprising computer instructions, when the computer program product is run on an object server, causes the object server to perform any one of the second aspect described above and various alternative implementations thereof The data processing method described.

For a description of the technical effects of the tenth to twelfth aspects, reference may be made to the related description of the technical effects of the second aspect, and details are not described herein again.

In a thirteenth aspect, a proxy server is provided, the proxy server can include a processor and a memory coupled to the processor. This memory can be used to store computer instructions. When the proxy server is running, the processor executes the computer instructions stored by the memory to cause the proxy server to perform the data processing method of any of the above third aspects and various alternative implementations thereof.

In a fourteenth aspect, a computer readable storage medium is provided, the computer readable storage medium comprising computer instructions. When the computer instruction is run on a proxy server, the proxy server is caused to perform the data processing method of any of the above third aspects and various alternative implementations thereof.

According to a fifteenth aspect, a computer program product comprising computer instructions, when executed on a proxy server, causes the proxy server to perform any one of the above third aspects and various alternative implementations thereof The data processing method described.

For a description of the technical effects of the thirteenth aspect to the fifteenth aspect, reference may be made to the related description of the technical effects of the third aspect, and details are not described herein again.

According to a sixteenth aspect, an object storage system is provided, which may include the container server according to any one of the foregoing fourth aspects and various alternative implementations thereof, the fifth aspect and various The object server of any one of the selected implementations, and the proxy server of any of the sixth aspect and various alternative implementations thereof.

Alternatively, the object storage system system may include the container server in the seventh aspect, the object server in the tenth aspect, and the proxy server in the thirteenth aspect.

For a description of the related content and technical effects of the sixteenth aspect, reference may be made to the foregoing first aspect or any possible implementation manner, the second aspect or any possible implementation manner, and the third aspect or any possible implementation. The related content of the method and the related description of the technical effect will not be described here.

DRAWINGS

1 is a schematic structural diagram of an object storage system according to an embodiment of the present invention;

2 is a schematic diagram of hardware of various servers according to an embodiment of the present invention;

3 is a schematic diagram 1 of a data processing method according to an embodiment of the present invention;

4 is a second schematic diagram of a data processing method according to an embodiment of the present invention;

FIG. 5 is a schematic diagram 3 of a data processing method according to an embodiment of the present disclosure;

FIG. 6 is a schematic diagram 4 of a data processing method according to an embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram 1 of a container server according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram 2 of a container server according to an embodiment of the present disclosure;

FIG. 9 is a schematic structural diagram 1 of an object server according to an embodiment of the present disclosure;

FIG. 10 is a schematic structural diagram 2 of an object server according to an embodiment of the present disclosure;

FIG. 11 is a schematic structural diagram 1 of a proxy server according to an embodiment of the present invention;

FIG. 12 is a schematic structural diagram 2 of a proxy server according to an embodiment of the present invention.

Detailed ways

The term "and/or" in this context is merely an association describing the associated object, indicating that there may be three relationships, for example, A and/or B, which may indicate that A exists separately, and both A and B exist, respectively. B these three situations.

The terms "first" and "second" and the like in the specification and claims of the embodiments of the present invention are used to distinguish different objects, and are not intended to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, rather than to describe a specific order of the target objects.

In the embodiments of the present invention, the words "exemplary" or "such as" are used to mean an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the invention should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the words "exemplary" or "such as" is intended to present the concepts in a particular manner.

In the description of the embodiments of the present invention, the meaning of "a plurality" means two or more unless otherwise stated. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

First, some concepts involved in the data processing method, apparatus, and system provided by the embodiments of the present invention are explained.

Container: In the field of object storage, it can also be called a bucket, in which information about an object is stored (for example, it can be called a list of objects, including location information of multiple objects, such as the ID of the object, and the offset of the object in a file) And the size of the object, etc.), the container is located in the container server.

Object: In the object storage domain, the object is the storage unit, the object is stored in the object server in the object storage system, the object includes the object metadata and the object data, and the object metadata is used to describe the information of the object's attributes, such as the size of the object. Or the type of the object (such as pictures, audio, video, etc.), etc., the object data is the object itself.

The data processing method provided by the embodiment of the present invention can be applied to an object storage system. For a schematic diagram of the architecture of the object storage system, refer to the schematic diagram of the object storage system shown in FIG. 1 , where the proxy server 11 is responsible for the object storage system. Inter-communication between the remaining components, for example, the proxy server 11 can receive hypertext transfer requests (e.g., read requests, delete requests, upload requests, etc.) of the client, and can forward requests from clients (e.g., terminal 10 in Figure 1). The message is to the container server 12 or the object server 13 in the object storage system. The information of the container and the information of the object (for example, the location information of the object) are maintained (considered to be stored) in the database of the container server 12, and the metadata and data of the plurality of objects are stored in the file system of the object server 13. In the object storage system provided by the embodiment of the present invention, an operation of uploading (ie, writing) an object, reading an object, and deleting an object may be implemented through interaction between the proxy server 11, the container server 12, and the object server 13.

It can be understood that in the object storage system shown in FIG. 1, the terminal 10 can be a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (personal). Digital assistant (PDA), smart car, sensing device, internet of thing (IOT) equipment, customer premise equipment (CPE), etc.

The proxy server 11, the container server 12, and the object server 13 shown in FIG. 1 may be a hardware server or a software server. In the embodiment of the present invention, the proxy server 11, the container server 12, and the object server 13 are hardware. Server, introduces the hardware structure and software function modules of each server.

It should be noted that, in the embodiment of the present invention, in the object storage system shown in FIG. 1 , the proxy server 11 , the container server 12 , and the object server 13 may be servers of the same structure or servers of different configurations. The components of the proxy server, the container server, and the object server are respectively described below in conjunction with the object storage system shown in FIG.

FIG. 2 is a schematic diagram of hardware of each server according to an embodiment of the present invention. The proxy server 100 shown in FIG. 2 may include: a processor 101, a memory 102, and a communication interface 103.

The processor 101 is a core component of the proxy server and is used to run the server's operating system and applications on the server (including system applications and third-party applications).

In the embodiment of the present invention, the processor 101 may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), or an application-specific integrated circuit (ASIC). a field programmable gate array (FPGA) or other programmable logic device, transistor logic device, hardware component, or any combination thereof, which may implement or perform various aspects described in the disclosure of the embodiments of the present invention. Exemplary logical blocks, modules and circuits; the processor may also be a combination of computing functions, for example, including one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like.

Memory 102: program code and data for storing a proxy server.

In the embodiment of the present invention, the memory 102 may specifically include a volatile memory, such as a random-access memory (RAM); the memory may also include a non-volatile memory. For example, a read-only memory (ROM), a flash memory, a hard disk (HDD), or a solid-state disk (SSD); the memory may further include the above-described type of memory. The combination.

The communication interface 103 is an interface circuit for the proxy server to communicate with other devices. The communication interface may be a transceiver, a transceiver circuit, or the like having a transceiving function, and the communication interface includes a serial communication interface and a parallel communication interface. For example, in an embodiment of the present invention, the proxy server 100 can communicate with a client (e.g., terminal 10 in FIG. 2) via the communication interface 103, with the container server 200, or with the object server 300.

The container server 200 shown in FIG. 2 may include a processor 201, a memory 202, and a communication interface 203. For a description of the processor 201, refer to the description of the processor 101 of the proxy server 100, and details are not described herein again. In the embodiment of the present invention, in the object storage system, after the container server 200 receives the read query message sent by the proxy server 100 or deletes the query message, the processor 101 of the container server may process the read query message or delete the query message.

For a description of the memory 202, reference may be made to the description of the memory 102 of the proxy server 100. In the embodiment of the present invention, as shown in FIG. 2, the memory 202 of the container server 200 may include at least one container (for example, M containers shown). M≥1)), each container may include a list of objects and a delete log, where the object list stores location information of a plurality of objects (which may be referred to as first type objects), and the first type of objects are unmarked and deleted. The object stored in the object server deletes the location information of a plurality of objects (which may be referred to as a second type of object), and the second type of object is the location information of the object stored in the object server that is marked and deleted.

For a description of the communication interface 203, reference may be made to the description of the communication structure 103 of the proxy server 100, in which the container server may communicate with the proxy server 100 via the communication interface 203 or with the object server 300.

The object server 300 shown in FIG. 2 may include a processor 301, a memory 302, and a communication interface 303. For a description of the processor 301, reference may be made to the description of the processor 101 of the proxy server 100, and details are not described herein again. In the embodiment of the present invention, in the object storage system, after the object server 200 receives the request message of the upload object sent by the proxy server 100 or receives the garbage collection request message sent by the container server 200, the processor 301 of the object server 300 can process Process the request message of the upload object or process the garbage collection request message.

For a description of the memory 302, reference may be made to the description of the memory 102 of the proxy server 100. In the embodiment of the present invention, as shown in FIG. 2, the file 302 of the object server 300 may include a file system, and the file system includes at least one file (for example, The file system shown in FIG. 2 includes N files (N≥1), and each of the at least one file can store a plurality of objects (for example, each of the files shown in FIG. 2 can include P objects). (P ≥ 2)).

For a description of the communication interface 303, reference may be made to the description of the communication structure 103 of the proxy server 100, in which the object server 300 may communicate with the proxy server 100 via the communication interface 303 or with the container server 200.

In the prior art, in the process of deleting an object, the object server needs to delete the object according to the location information of the object, and when the object storage system has both read an object (for example, the first object) Request, there is also a request to delete the object, because the object in the object server may have been marked as deleted when the object is read, so the failure to read the object may occur, which leads to the first reading The object conflicts with deleting the first object. The embodiment of the invention provides a data processing method, device and system, which can avoid conflicts between a read object and a deleted object to a certain extent.

In combination with the object storage system shown in FIG. 1 above, the user equipment may delete an object stored in the object storage system, or may read an object stored in the object storage system, and may also upload the object to the object storage system, and firstly through the object storage. The interaction process between the various servers in the system describes three different application scenarios of deleting objects, reading objects, and uploading objects.

In the embodiment of the present invention, the scene of the deleted object is recorded as the scene 1, the scene of the read object is recorded as the scene 2, and the scene of the uploaded object is recorded as the scene 3.

scene one

With reference to FIG. 1 and FIG. 2, as shown in FIG. 3, in the scenario of deleting an object in the object storage system, the data processing method provided by the embodiment of the present invention may include S101-S105:

S101. The proxy server sends a delete query message to the container server, where the delete query message includes the identifier information of the container to which the first target object belongs and the identifier information of the first target object.

In the embodiment of the present invention, when the terminal needs to delete an object in the object storage system (hereinafter may be referred to as a first target object), the terminal may send a request message for deleting the first target object to the proxy server, and the proxy server receives the terminal. After the sent request message for deleting the first target object is sent, and the request message for deleting the first target object is parsed, and the identifier information of the container to which the first target object belongs and the identification information of the first target object are obtained, the proxy server may The identifier information of the container to which the target object belongs and the identifier information of the first target object are carried in the delete query message and sent to the container server, where the location information of the object is stored in the container, and the location information is used to indicate the target server. The location of the object, the identifier information of the container to which the first target object belongs may identify in which container the location of the first target object to be deleted is stored, and the identifier information of the first target object may indicate that the target object to be deleted is in the container Which target object.

Optionally, in the embodiment of the present invention, the identifier information of the first target object may be a name of the first target object or other information that may be specified by the user, which may be a predetermined target, which is not specifically limited.

For the same reason, the identifier information of the container to which the first target object belongs may also be the name of the container or other information defined by the user, which may be instructed by the user.

S102. The container server receives a delete query message sent by the proxy server.

S103. The container server queries the location information of the first target object in the object list of the container server according to the identifier information of the container to which the first target object belongs and the identifier information of the first target object in the delete query message.

The object list of the container server includes location information of the first type of object, and the first type of object is an object stored in the object server that is not marked and deleted, and the location information of each first type of object is used to indicate each The location of the object in the object server.

In the embodiment of the present invention, according to the manner of storing the object, the location information of the object is also different. For example, if an object is stored in a file, it can be understood that only one object is stored in each file, then the object is The location information is the ID of the object; if at least two objects are stored in one file, it can be understood that at least two objects are stored in each file, and the location information of the object is the file identifier of the object, the offset of the object, and The size of the object, the offset of the object is the position of the object in its stored file, and the size of the object identifies the size of the object, such as the number of bits occupied by the object.

It can be seen that the location information of the first target object may include an ID of the first target object, or the location information of the first target object may include a file identifier of the first target object, an offset of the first target object, and the first The size of the target object, which includes at least two objects in the file indicated by the file identifier.

Optionally, in the embodiment of the present invention, in an actual implementation process, if only one object is stored for each file, the location information of the object may be jointly represented by the ID of the container and the ID of the object, and the object server When there are many objects in the object, the range of objects that can be represented by the ID of the object may not satisfy the number of objects, that is, the actual number of objects exceeds the number of objects that can be represented by the ID of the object. Therefore, the ID of the container and the ID of the object can be used. Two fields identify the location information of the object to indicate the object.

Optionally, in the embodiment of the present invention, the object information of the container server may store location information of multiple objects, that is, multiple pieces of location information may be stored, and each piece of location information corresponds to location information of one object, and each piece of location information may correspond to a location index, and the information tuple composed of the identification information of the container to which the object belongs and the identification information of the object also corresponds to a location index, and the container server according to the identification information of the container to which the first target object belongs and the identifier of the first target object The method for querying the location information of the first target object in the object list of the container server is specifically: the container server determines the location index of the first target according to the identifier information of the container to which the first target object belongs and the identification information of the first target object, The location information of the first target object can then be determined based on the index of the location of the first target object.

Illustratively, taking the location information of the five objects in the container server as an example, as shown in Table 1, the information binary consisting of the identification information of the container to which the object in the container server belongs and the identification information of the object (Table 1) An example of the correspondence between the positional index of the object and the positional index of the object and the positional index of the object.

Table 1

对象的信息二元组Object information binary	位置索引Location index	位置信息location information
(container1，object1)(container1, object1)	11	位置信息1Location information 1
(container2，object3)(container2, object3)	22	位置信息2Location information 2
(container1，object4)(container1, object4)	33	位置信息3Location information 3
(container3，object1)(container3, object1)	44	位置信息4Location information 4
(container4，object2)(container4, object2)	55	位置信息5Location information 5

In Table 1, object1 represents the identifier of the object 1, and container1 represents the identifier of the container to which the object 1 belongs.

S104. If the location information of the first target object is queried in the object list, the container server deletes the location information of the first target object in the object list, and writes the location information of the first target object to the deletion of the container server. In the log.

The deletion log includes location information of the second type of object, and the second type of object is location information of the object stored in the object server that is marked and deleted.

In the embodiment of the present invention, after the container server deletes the location information of the first target object from the object list, and after the location information of the first target object is written into the deletion log, the container server subsequently receives other information about the first target object. When querying a message (for example, reading a query message), the container server will not find the location information of the first target object in the object list, and thus cannot perform corresponding operations on the first target object.

It should be noted that, in the embodiment of the present invention, after the location information of the first target object is deleted from the object list, and the location information of the first target object is written into the deletion log, the first target object is considered to be deleted successfully. In fact, the content of the first target object, that is, the metadata of the first target object and the data itself have not been deleted (ie, not recovered), and are also stored in the object server.

In the embodiment of the present invention, the location information of the object in the deletion log of the container server indicates the object to be reclaimed in the object server, and the location information of the first target object is written into the deletion log, which can be understood as the first target to be deleted. The object is first marked, and at the appropriate time, the content of the first target object is recovered from the object server based on the location information of the first target object in the deletion log.

The method of reclaiming objects in the object server will be described in detail in the following embodiments and will not be described here.

S105. The container server sends a delete response message to the proxy server.

In the embodiment of the present invention, the deletion response message may include indication information indicating that the deletion of the first target object is successful, to indicate that the container server successfully deletes the first target object.

It can be understood that, in the embodiment of the present invention, if the first target object does not exist in the object storage system, that is, the location information of the first target object does not exist in the container server, the first target object does not exist in the object server. In this case, if the container server does not query the location information of the first target object in the object list, the container server may send, to the proxy server, indication information including location information indicating that the first target object is not queried, to indicate The container server does not query the location information of the first target object, that is, the deletion operation ends.

In the data processing method provided by the embodiment of the present invention, in the process of deleting an object (for example, a first target object) in the object storage system, after the container server receives the delete query message, the container server may query the object list of the container server. The location information of the first target object, and in the case where the location information of the first target object is queried in the object list, the location information of the first target object is deleted from the object list, and the location of the first target object is Information is written to the delete log of the container server. Compared with the prior art, in the embodiment of the present invention, when the container server queries the location information of the first target object in the object list, the location information of the first target object does not need to be sent to the proxy server, and the proxy server does not need to be Sending the location information of the first target object to the object server, so that the object server does not need to delete the first target object in the object server, so when deleting the first target object and reading in the object storage system When the first target object in the object storage system is concurrently generated, since the first target object in the object server is not marked as deleted during the process of deleting the first target object, the object can be read in the object server. Obtaining the first target object, compared with the prior art, avoiding the problem that the first target object in the object server is marked as deleted, causing the first target object to fail to be read, thereby avoiding reading A target object conflicts with deleting the first target object.

Scene two

With reference to FIG. 1 and FIG. 2, as shown in FIG. 4, in a scenario of reading an object in an object storage system, the data processing method provided by the embodiment of the present invention may include S201-S209:

S201. The proxy server sends a read query message to the container server, where the read query message includes the identifier information of the container to which the second target object belongs and the identifier information of the second target object.

In the embodiment of the present invention, when the terminal needs to read an object in the object storage system (which may be referred to as a second target object), the terminal may send a request message for reading the second target object to the proxy server, and the proxy server receives the request message. After the request message of the second target object is read by the terminal, and the request message for reading the first target object is parsed, and the identifier information of the container to which the second target object belongs and the identification information of the second target object are obtained, the proxy server may Carrying the identification information of the container to which the second target object belongs and the identification information of the second target object in the read query message, and sending the information to the container server to obtain the location information of the second target object, thereby reading the second target object. .

S202. The container service receives a read query message sent by the proxy server.

S203. The container server queries the location information of the second target object in the object list of the container server according to the identifier information of the container to which the second target object belongs and the identifier information of the second target object.

In the embodiment of the present invention, the location information of the second target object includes a file identifier of the second target object, an offset of the second target object, and a size of the second target object, where the file identifier includes at least two files. Object.

For other descriptions of S202, refer to the embodiment of scenario 1 above. For the related description of S103, details are not described herein again.

S204. If the location information of the second target object is queried in the object list, the container server sends the location information of the second target object to the proxy server.

In the embodiment of the present invention, the container server sends the location information of the second target object to the proxy server, and may instruct the proxy server to read the second target object from the object server according to the location information of the second target object.

S205. The proxy server receives location information of the second target object sent by the container server.

S206. The proxy server sends the location information of the second target object to the object server.

S207. The object server receives location information of the second target object sent by the container server.

S208. The object server reads the second target object according to the location information of the second target object.

S209. The object server sends a second target object to the proxy server.

In the embodiment of the present invention, the location information of the second target object that can be received by the proxy server is sent to the object server, so that the object server can read the second target object in the corresponding location in the object server according to the location information of the second target object. And sending the second target object to the proxy server, and then the proxy server sends the second target object to the terminal, so that the terminal can smoothly read the second target object.

Scene three

With reference to FIG. 1 and FIG. 2, as shown in FIG. 5, in the scenario of uploading an object to the object server, the data processing method provided by the embodiment of the present invention may further include S301-S308:

S301. The proxy server sends a request message for uploading a third target object to the object server, where the request message for uploading the third target object includes a third target object.

In the embodiment of the present invention, when the terminal needs to upload an object to the object storage system (for example, it may be referred to as a third target object), the terminal may send an upload request message to the proxy server, where the upload request message includes the third target object. The identification information of the container and the identification information of the third target object, the identification information of the container to which the third target object belongs is used to indicate in which container the third target object is updated after the third target object is written to the object server Location information; the proxy server may send the third target to the object server in the request message for uploading the third target object.

S302. The object server receives a request message that is sent by the container server and uploads the third target object.

S303. The object server writes the third target object into a file of the object server.

In the embodiment of the present invention, the file written to the third target object can support writing multiple objects, and the multiple objects are small objects, and the small object can be understood as the space occupied by the object is relatively small, that is, the size of the object is relatively small ( For example, an image, a small text, etc. can be called a small object. Compared with the prior art, each file contains only one object, and multiple objects (here, will refer to multiple small objects). Writing to a large file can save the storage resources of the file system of the object server, thereby improving the performance of the object server.

Further, in the prior art, each file contains only one object. When the object server reclaims multiple small objects, multiple space fragments are generated in the server (ie, small files after each small object are reclaimed). In the embodiment of the present invention, multiple small objects are written into one large file, so that when small objects in the large file are recovered, space fragmentation in the server can be reduced, thereby writing a large object (occupied) When the storage space is relatively large, it is not necessary to divide the large object into multiple small files, and the entire contents of the large object can be written into a large file, thereby improving the performance of the object storage system.

S304. The object server sends the first location information of the third target object to the proxy server.

The first location information of the third target object includes a file identifier of the third target object, an offset of the third target object, and a size of the third target object, where the file identifier of the third target object is in the file indicated by the file identifier Save at least two objects.

In the embodiment of the present invention, the object server writes the third target object to a certain location of the object server, and if the location is called the first location, the object server may generate the location information of the third target object, because the object server is in the current time. Before the third target object is written, the third target object may already be stored in the object server (possibly the third target object to be written this time is a modified version of the stored third target object), so in order to distinguish the present The positional position information of the third target object that has been written in the second time and the positional information of the third target object that has been previously stored are recorded as the first positional information of the third target object.

S305. The proxy server receives first location information of the third target object sent by the object server.

S306. The proxy server sends the location update message to the container server.

In the embodiment of the present invention, the location update message is used to update location information of the target object in the object list of the container server. After the proxy server receives the first location information of the third target object sent by the object server, the proxy server may carry the identifier information of the container to which the target object belongs, the identifier information of the target object, and the first location information of the target object in the location update message. Is sent to the container server to instruct the container server to update the location information of the target object in the object list of the container server according to the location update message.

S307. The container server receives a location update message sent by the proxy server.

S308. The container server updates the location information of the third target object in the object list of the container server according to the location update message.

Optionally, in the embodiment of the present invention, the foregoing S308 may be specifically implemented by using S308a-S308b:

S308a. The container server queries the location information of the third target object in the object list of the container server according to the identifier information of the container to which the third target object belongs and the identifier information of the third target object.

In the embodiment of the present invention, the container server queries the object list for the presence of the third target object in the object list according to the identification information of the container to which the third target object belongs and the identification information of the third target object in the location update message received by the container server. Location information.

For a detailed description of the S308a, refer to the embodiment of the foregoing scenario 1. For the related description of S103, details are not described herein again.

S308b: The container server updates the location information of the third target object according to the query result.

In the embodiment of the present invention, the query result of the container server querying the location information of the third target object in the object list may include: the container server does not query the location information of the third target object in the object list, or the container server is in the object list. Query the location information of the third target object. Depending on the result of the query, the above S307b may include the following 308b1 or S208b2:

S308b1: If the location information of the third target object is not queried in the object list, the container server writes the first location information of the third target object in the location update message received by the container server to the object list.

In the embodiment of the present invention, after the third target object is written in the object server in the object storage system, if the location information of the third target object does not exist in the container server, the first location information of the third target object is obtained. The object list is written, so that in the subsequent process of reading or deleting the third target object, the third target object can be read or deleted by querying the location information of the third target object in the object list.

S308b2, if the location information of the third target object is queried in the object list, the location information of the third target object is the second location information, and the second location information of the third target object and the third target in the location update message The first location information of the object is different, the container server replaces the second location information of the third target object with the first location information of the third target object, and writes the second location information of the third target object to the deletion log of the container server in.

In the embodiment of the present invention, if the location information of the third target object is queried in the object list, the third target object is already stored in the object server when the third target object is written to the object server. The third target object to be written is a modified version of the stored third target object. Therefore, the second location information of the third target object may be replaced with the first location information of the third target object, specifically, The container server deletes the second location information of the third target object, and writes the first location information of the third target object into the object list, such that in subsequent operations (eg, reading the third target object or deleting the third During the target object, the information of the accurate third target object can be queried in the container server to correctly read the third target object or delete the third target object. And the second location information of the third target object may be written into the deletion log of the container server, and the object server recovers the second location information of the third target in the object server according to the second location information of the third target object in the deletion log. The third target object indicated.

In the data processing method provided by the embodiment of the present invention, in the process of uploading an object (for example, may be referred to as a third target object) to the object storage system, the proxy server may send a request message for uploading the third target object to the object server, and the object Receiving, by the server, the request message for uploading the third target object, and writing the third target object in the request message to the object server, and sending the first location information of the third target object to the proxy server, the third target The first location information of the object includes a file identifier of the third target object, an offset of the third target object, and a size of the third target object, where the file identifier indicated by the third target object stores at least two files. Object. Compared with the prior art, in the embodiment of the present invention, since the object server writes the third target object into the file system file of the object server, the file may include at least two objects, and it is known that when the object storage system writes When multiple small objects are entered, multiple small objects can be written to one file. This saves the storage resources of the file system of the object server, thereby improving the performance of the object server.

Optionally, in the embodiment of the present invention, the first target object in the scenario 1 , the second target object in the scenario 2, and the third target object in the scenario 3 may be the same target object, or may be different targets. Object. If the first target object, the second target object, and the third target object are the same target object, for example, each is referred to as the first target object, the first target object may be written by the description of the three scenarios in the above embodiment. The object storage system is such that the first target object can be read from the object storage system or the first target object can be deleted.

Optionally, in the embodiment of the present invention, according to the related description of the foregoing embodiment, when the container server deletes the location information of an object from the object list, and writes the location information of the object to the deletion log, the container server may The interaction of the object server, which recycles the object in the object server. The following takes the first target object as an example, and exemplifies the method of recycling the object.

With reference to FIG. 1 and FIG. 2, as shown in FIG. 6, after the container server deletes the location information of the first target object from the object list, and the location information of the first target object is written into the deletion log, the embodiment of the present invention provides The data processing method may also include S401-S406:

S401. The container server sends a garbage collection request message to the object server.

The garbage collection request message includes location information of the first target object in the deletion log. The spam request message is used to request the object server to reclaim the content of the first target object in the object server (ie, including the metadata and data of the first target object).

Optionally, in the embodiment of the present invention, after the container server deletes the location information of the first target object from the object list, and the location information of the first target object is written into the deletion log, the container server is in a preset time interval. The object server sends a garbage collection request message, or the container server can send a garbage collection request to the object server to reclaim the object in the object server (also known as garbage) when the remaining storage space of the object server is relatively small.

S402. The object server receives a garbage collection request message sent by the container server.

S403. The object server deletes the first target object according to the location information of the first target object in the garbage collection request message.

In the embodiment of the present invention, the object server may reclaim the garbage object in the object server, where the garbage object is an object indicated by the location information of the object in the deletion log of the container server, and the method for the object server to collect the first target object is: The location information of the first target object in the garbage collection request message deletes the first target object, thereby releasing the storage space of the object server.

S404. The object server sends a garbage collection response message to the container server, where the garbage collection response message indicates that the first target object has been deleted.

S405. The container server receives the garbage collection response message sent by the object server.

S406. The container server deletes the location information of the first target object in the deletion log of the container server.

In the embodiment of the present invention, after the container server receives the garbage collection response message sent by the object server, the container server may delete the location information of the first target object in the deletion log, thereby completing the recovery of the first target object, and the object storage The first target object no longer exists in the system.

Optionally, in the embodiment of the present invention, in the object storage system, when an object is read and the object is deleted concurrently (that is, it can be understood that the object storage system processes the event of reading an object, and also needs to be processed. In the embodiment of the present invention, the location information of the first target object is received from the object in the container server according to the method of reading the object and deleting the object in the foregoing embodiment. After the list is deleted, and the location information of the first target object is written in the delete log of the container server, the container server has read the location information of the first target object in the object list, and then the first target object is read this time. In the process of deleting and deleting the first target object, even if the location information of the first target object in the container server has been deleted, and the location information of the first target object has been written into the deletion log, this time can be read. Go to the first target object.

Compared with the prior art, the location information of the first target object in the container server has been deleted, and the location information of the first target object has been written into the deletion log, and the first target object can also be read this time. The reason is: in the process of deleting the first target object, the location information of the first target object may be written in the deletion log to indicate that the first target object is marked for deletion, and no need to be in the object server. The first target object is marked as deleted. Therefore, the first target object still exists in the object server and has not been reclaimed. Thus, the object server may obtain the first target object according to the read query request message according to the container server. Location information, the first target object is read in the object server. It can be seen that the data processing method provided by the embodiment of the present invention can avoid conflicts between the read object and the deleted object to a certain extent (refer to when the read object and the deleted object occur concurrently, the operation of deleting the object firstly causes the read object to be read. The operation could not read the object).

In summary, in the data processing method provided by the embodiment of the present invention, the process of deleting the object, reading the object, uploading the object, and reclaiming the object may be completed in the user state, which can reduce the difficulty in developing and maintaining the object storage system.

The solution provided by the embodiment of the present invention is mainly introduced from the perspective of interaction between the network elements. It can be understood that various network elements, such as a container server, an object server, and a proxy server, etc., in order to implement the above functions, include hardware structures and/or software modules corresponding to each function. Those skilled in the art will readily appreciate that the embodiments of the present invention can be implemented in a combination of hardware or hardware and computer software in combination with the elements and algorithm steps of the various examples described in the embodiments disclosed herein. Whether a function is implemented in hardware or computer software to drive hardware depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods to implement the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present application.

The embodiments of the present invention may divide the function modules of the container server, the object server, the proxy server, and the like according to the foregoing method examples. For example, each function module may be divided according to each function, or two or more functions may be integrated into one. Processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of the module in the embodiment of the present invention is schematic, and is only a logical function division, and the actual implementation may have another division manner.

FIG. 7 is a schematic diagram of a possible structure of the container server involved in the foregoing embodiment. As shown in FIG. 7, the container server may include: a receiving module 30, The query module 31, the update module 32, and the sending module 33. The receiving module 30 can be used to support the container server to execute S102, S202, S307, and S405 in the foregoing method embodiment; the query module 31 can be used to support the container server to execute S103, S203 in the foregoing method embodiment; the update module 32 can S104, S308 (including S308a-S308b) and S406 are used to support the container server in the foregoing method embodiment; the sending module 33 can be used to support the container server to execute S105, S204 and S401 in the above method embodiment. Optionally, as shown in FIG. 7, the container server may further include a reading module 34. The reading module 34 can be used to support the container server to perform the above-mentioned embodiment: "If the read query message received by the receiving module 30 includes the identification information of the container to which the first target object belongs and the identification information of the first target object, the update is performed. The module deletes the location information of the first target object in the object list, and reads the location information of the first target object in the object list before writing the location information of the first target object to the deletion log of the container server. All the related content of the steps involved in the foregoing method embodiments may be referred to the functional descriptions of the corresponding functional modules, and details are not described herein again.

In the case of employing an integrated unit, FIG. 8 shows a possible structural diagram of the container server involved in the above embodiment. As shown in FIG. 8, the container server may include a processing module 40 and a communication module 41. The processing module 40 can be used to control and manage the actions of the container server. For example, the processing module 40 can be used to support the container server to execute S103, S104, S203, S308 (including S308a-S308b) and S406 in the foregoing method embodiment, and / or other processes for the techniques described herein. The communication module 41 can be used to support communication between the container server and other network entities. For example, the communication module 41 can be used to support the container server to execute S102, S105, S202, S204, S307, S401, and S405 in the foregoing method embodiments. Optionally, as shown in FIG. 8, the container server may further include a storage module 42 for storing program code and data of the container server.

The processing module 40 may be a processor or a controller (for example, the processor 201 shown in FIG. 2 above), and may be, for example, a central processing unit (CPU), a general-purpose processor, and a digital signal processor. (digital signal processor, DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, transistor logic device, hardware component or any combination. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the embodiments of the invention. The above processors may also be a combination of computing functions, such as one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like. The communication module 41 may be a transceiver, a transceiver circuit, or a communication interface or the like (for example, may be the communication interface 203 shown in FIG. 2 described above). The storage module 42 can be a memory (which can be, for example, the memory 202 shown above in FIG. 2).

When the processing module 40 is a processor, the communication module 41 is a transceiver, and the storage module 42 is a memory, the processor, the transceiver, and the memory can be connected by a bus. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

FIG. 9 is a schematic diagram of a possible structure of the object server involved in the foregoing embodiment. As shown in FIG. 9, the object server may include: a receiving module 50, The data module 51 and the transmitting module 52 are written. The receiving module 50 can be used to support the object server to execute S207, S302, and S402 in the foregoing method embodiment; the write data module 51 can be used to support the object server to execute S303 in the foregoing method embodiment; and the sending module 52 can be used to support the object server. S209, S304, and S404 in the above method embodiment are executed. Optionally, as shown in FIG. 9, the object server may further include a reading module 53 and a deleting module 54. The reading module 53 can be used to support the object server to execute S208 in the above method embodiment; the deleting module 54 can be used to support the object server to execute S403 in the above method embodiment. All the related content of the steps involved in the foregoing method embodiments may be referred to the functional descriptions of the corresponding functional modules, and details are not described herein again.

In the case of employing an integrated unit, FIG. 10 shows a possible structural diagram of the object server involved in the above embodiment. As shown in FIG. 10, the object server may include a processing module 60 and a communication module 61. The processing module 60 can be used to control and manage the actions of the object server. For example, the processing module 60 can be used to support the object server to execute S208, S303, and S403 in the foregoing method embodiments, and/or for the techniques described herein. Other processes. The communication module 61 can be used to support communication between the object server and other network entities. For example, the communication module 61 can be used to support the object server to execute S207, S209, S302, S304, S402, and S404 in the foregoing method embodiments. Optionally, as shown in FIG. 10, the object server may further include a storage module 62 for storing program code and data of the object server.

The processing module 60 may be a processor or a controller (for example, the processor 301 shown in FIG. 2 above), and may be, for example, a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a transistor. Logic device, hardware component, or any combination thereof. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the embodiments of the invention. The above processors may also be a combination of computing functions, such as one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like. The communication module 61 may be a transceiver, a transceiver circuit or a communication interface or the like (for example, may be the communication interface 303 shown in FIG. 2 described above). The storage module 62 can be a memory (which can be, for example, the memory 302 shown above in FIG. 2).

When the processing module 60 is a processor, the communication module 61 is a transceiver, and the storage module 62 is a memory, the processor, the transceiver, and the memory can be connected by a bus. The bus can be a PCI bus or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

FIG. 11 is a schematic diagram showing a possible structure of the proxy server involved in the foregoing embodiment. As shown in FIG. 11, the proxy server may include: a sending module 70 and Receive module 71. The sending module 70 can be used to support the proxy server to execute S101, S201, S206, S301, and S306 in the foregoing method embodiment; the receiving module 50 can be used to support the proxy server to execute S205 and S305 in the foregoing method embodiment. All the related content of the steps involved in the foregoing method embodiments may be referred to the functional descriptions of the corresponding functional modules, and details are not described herein again.

In the case of employing an integrated unit, FIG. 12 shows a possible structural diagram of the proxy server involved in the above embodiment. As shown in FIG. 12, the proxy server may include a processing module 80 and a communication module 81. The processing module 80 can be used to control and manage the actions of the proxy server. The communication module 81 can be used to support communication between the proxy server and other network entities. For example, the communication module 81 can be used to support the proxy server to execute S101, S201, S206, S205, S301, S305, and S306 in the foregoing method embodiments. Optionally, as shown in FIG. 12, the proxy server may further include a storage module 82 for storing program code and data of the proxy server.

The processing module 80 may be a processor or a controller (for example, the processor 101 shown in FIG. 2 above), and may be, for example, a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a transistor. Logic device, hardware component, or any combination thereof. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the embodiments of the invention. The above processors may also be a combination of computing functions, such as one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like. The communication module 81 may be a transceiver, a transceiver circuit or a communication interface or the like (for example, may be the communication interface 103 as shown in FIG. 2 described above). The storage module 82 can be a memory (which can be, for example, the memory 102 shown above in FIG. 2).

When the processing module 80 is a processor, the communication module 81 is a transceiver, and the storage module 82 is a memory, the processor, the transceiver, and the memory can be connected by a bus. The bus can be a PCI bus or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions in accordance with embodiments of the present invention are generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be wired from a website site, computer, server or data center (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) to another website, computer, server or data center. The computer readable storage medium can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that includes one or more available media. The usable medium may be a magnetic medium (for example, a floppy disk, a magnetic disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (such as a solid state drives (SSD)). .

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above functional modules is illustrated. In practical applications, the above functions can be allocated according to needs. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working process of the system, the device and the unit described above, reference may be made to the corresponding process in the foregoing method embodiments, and details are not described herein again.

In the several embodiments provided by the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be used. Combinations can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application, in essence or the contribution to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium. A number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to perform all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a flash memory, a mobile hard disk, a read only memory, a random access memory, a magnetic disk, or an optical disk, and the like, which can store program codes.

The foregoing is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present application should be covered by the scope of the present application. . Therefore, the scope of protection of the present application should be determined by the scope of the claims.

Claims

A data processing method, comprising:

The container server receives the delete query message sent by the proxy server, where the delete query message includes the identifier information of the container to which the first target object belongs and the identifier information of the first target object;

The container server queries the location information of the first target object in the object list of the container server according to the identifier information of the container to which the first target object belongs and the identifier information of the first target object, The object list includes location information of the first type of object, the first type of object is an object stored in the object server that is not marked and deleted, and the location information of the first type of object is used to indicate that the object is in the object. The location in the server;

If the location information of the first target object is queried in the object list, the container server deletes location information of the first target object in the object list, and positions the first target object The information is written in the deletion log of the container server, where the deletion log includes location information of the second type of object, and the second type of object is location information of the object stored in the object server that is marked and deleted;

The container server sends a delete response message to the proxy server.
The method of claim 1 wherein

The location information of the first target object includes a file identifier of the first target object, an offset of the first target object, and a size of the first target object, where the file identifier indicates that the file includes At least two objects; or,

The location information of the first target object includes an ID of the first target object.
The method according to claim 1 or 2, wherein the method further comprises:

The container server receives the read query message sent by the proxy server, where the read query message includes the identifier information of the container to which the second target object belongs and the identifier information of the second target object;

The container server queries location information of the second target object in the object list according to the identifier information of the container to which the second target object belongs and the identifier information of the second target object, and the second target The location information of the object includes a file identifier of the second target object, an offset of the second target object, and a size of the second target object, where the file identifier indicates that at least two objects are included in the file;

If the location information of the second target object is queried in the object list, the container server sends location information of the second target object to the proxy server to indicate that the proxy server is according to the second The location information of the target object reads the second target object from the object server.
The method according to any one of claims 1 to 3, wherein the read query message received by the container server includes identification information of the container to which the first target object belongs and the first target object. Identification information, where the container server deletes location information of the first target object in the object list, and writes location information of the first target object into a deletion log of the container server, The method also includes:

The container server reads location information of the first target object in the object list.
The method according to any one of claims 1 to 4, further comprising:

The container server sends a garbage collection request message to the object server, where the garbage collection request message includes location information of the target object in the deletion log;

Receiving, by the container server, a garbage collection response message sent by the object server, where the garbage collection response message indicates that the target server has deleted the target object;

The container server deletes location information of the target object in the deletion log.
The method according to any one of claims 1 to 5, further comprising:

Receiving, by the container server, a location update message sent by the proxy server, where the location update message is used to update location information of a third target object, where the location update message includes identifier information of a container to which the third target object belongs The identification information of the third target object and the first location information of the third target object;

The container server updates the location information of the third target object according to the location update message.
The method according to claim 6, wherein the container server updates the location information of the third target object according to the location update message, including:

The container server queries the location information of the third target object in the object list of the container server according to the identifier information of the container to which the third target object belongs and the identifier information of the third target object;

If the location information of the third target object is not queried in the object list, the container server writes the first location information of the third target object in the location update message into the object list;

If the location information of the third target object is queried in the object list, the location information of the third target object is second location information, and the second location information of the third target object is related to the location The first location information of the third target object in the update message is different, the container server replaces the second location information of the third target object with the first location information of the third target object, and The second location information of the third target object is written in the delete log of the container server.
A data processing method, comprising:

The object server receives a request message of the upload target object sent by the proxy server, where the request message of the upload target object includes a third target object;

Writing, by the object server, the third target object into the object server;

The object server sends the location information of the third target object to the proxy server, where the location information of the third target object includes a file identifier of the third target object, and an offset of the third target object. And the size of the third target object, wherein the file identifier indicates that at least two objects are included in the file.
The method of claim 8 further comprising:

Receiving, by the object server, location information of the second target object sent by the proxy server;

The object server reads the second target object according to location information of the second target object;

The object server sends the second target object to the proxy server.
The method according to claim 8 or 9, wherein the method further comprises:

Receiving, by the object server, a garbage collection request message sent by the container server, where the garbage collection request message includes location information of a first target object in a deletion log of the container server;

The object server deletes the first target object according to location information of the first target object;

The object server sends a garbage collection response message to the container server, the garbage collection response message indicating that the first target object has been deleted.
A data processing method, comprising:

The proxy server sends a request message for uploading the target object to the object server, where the request message of the upload target object includes the target object;

Receiving, by the proxy server, first location information of the target object sent by the object server, where the first location information of the target object includes a file identifier of the target object, an offset of the target object, and a location a size of the target object, where the file identifier indicates that at least two objects are stored in the file;

Sending, by the proxy server, the location update message to the container server, to instruct the container server to update location information of the target object in the object list of the container server according to the location update message, the location update message The identification information of the container to which the target object belongs, the identification information of the target object, and the first location information of the target object are included.
The method of claim 11 wherein the method further comprises:

Receiving, by the proxy server, a delete response message sent by the container server, where the delete response message is used by the container service to delete location information of the target object in the object list, and the target target object is The location information is sent after being written to the delete log of the container server.
A container server, comprising: a receiving module, a query module, an update module, and a sending module;

The receiving module is configured to receive a delete query message sent by the proxy server, where the delete query message includes the identifier information of the container to which the first target object belongs and the identifier information of the first target object;

The querying module is configured to query the object list in the container server according to the identifier information of the container to which the first target object belongs and the identifier information of the first target object received by the receiving module Position information of a target object, the object list includes location information of the first type of object, the first type of object is an object stored in the object server that is not marked and deleted, and the location information of the first type of object is used. Instructing the location of the object in the object server;

The update module is configured to: if the query module queries location information of the first target object in the object list, delete location information of the first target object in the object list, and The location information of the first target object is written in a deletion log of the container server, where the deletion log includes location information of a second type of object, and the second type of object is deleted by the object server stored in the object server Location information of the object;

The sending module is configured to send a delete response message to the proxy server.
A container server according to claim 13 wherein:

The location information of the first target object includes a file identifier of the first target object, an offset of the first target object, and a size of the first target object, where the file identifier indicates that the file includes At least two objects; or,

The location information of the first target object includes an ID of the first target object.
A container server according to claim 13 or 14, wherein

The receiving module is further configured to receive a read query message sent by the proxy server, where the read query message includes identifier information of a container to which the second target object belongs and identifier information of the second target object;

The querying module is further configured to query the second target in the object list according to the identifier information of the container to which the second target object belongs and the identifier information of the second target object received by the receiving module Location information of the object, the location information of the second target object includes a file identifier of the second target object, an offset of the second target object, and a size of the second target object, the file identifier The indicated file includes at least two objects;

The sending module is further configured to: if the query module queries location information of the second target object in the object list, the container server sends location information of the second target object to the proxy server Instructing the proxy server to read the second target object from the object server according to location information of the second target object.
A container server according to any one of claims 13 to 15, wherein the container server further comprises a reading module;

The reading module is configured to: if the read query message received by the receiving module includes the identifier information of the container to which the first target object belongs and the identifier information of the first target object, the update module deletes Reading the location information of the first target object in the object list, and reading the location information of the first target object into the deletion log of the container server, reading the number in the object list The location information of a target object.
A container server according to any one of claims 13 to 16, wherein

The sending module is further configured to send a garbage collection request message to the object server, where the garbage collection request message includes location information of the target object in the deletion log;

The receiving module is further configured to receive a garbage collection response message sent by the object server, where the garbage collection response message indicates that the target server has deleted the target object;

The update module is further configured to delete location information of the target object in the deletion log.
A container server according to any one of claims 13 to 17, wherein

The receiving module is further configured to receive a location update message sent by the proxy server, where the location update message is used to update location information of a third target object, where the location update message includes a third target object that belongs to Identification information of the container, identification information of the third target object, and first location information of the third target object;

The update module is further configured to update location information of the third target object according to the location update message received by the receiving module.
A container server according to claim 18, wherein

The querying module is further configured to query, according to the identifier information of the container to which the third target object belongs, and the identifier information of the third target object, the location of the third target object in the object list of the container server information;

The update module is specifically configured to: if the query module does not query the location information of the third target object in the object list, write the first location information of the third target object in the location update message Entering the object list, if the query module queries the location information of the third target object in the object list, the location information of the third target object is second location information, and the third target The second location information of the object is different from the first location information of the third target object in the location update message, and the second location information of the third target object is replaced with the first location of the third target object. Location information, and the second location information of the third target object is written into the deletion log of the container server.
An object server, comprising: a receiving module, a write data module, and a sending module;

The receiving module is configured to receive a request message of an upload target object sent by the proxy server, where the request message of the upload target object includes a third target object;

The write data module is configured to write the third target object received by the receiving module into the object server;

The sending module is configured to send location information of the third target object to the proxy server, where the location information of the third target object includes a file identifier of the third target object, and the third target object And an offset of the third target object, wherein the file identifier indicates that at least two objects are included in the file.
The object server according to claim 20, wherein the object server further comprises a reading module;

The receiving module is further configured to receive location information of the second target object sent by the proxy server;

The reading module is configured to read the second target object according to location information of the second target object received by the receiving module;

The sending module is further configured to send, send, to the proxy server, the second target object.
The object server according to claim 20 or 21, wherein the object server further comprises a deletion module;

The receiving module is further configured to receive a garbage collection request message sent by the container server, where the garbage collection request message includes location information of a first target object in a deletion log of the container server;

The deleting module is configured to delete the first target object according to location information of the first target object received by the receiving module;

The sending module is further configured to send a garbage collection response message to the container server, where the garbage collection response message indicates that the first target object has been deleted.
A proxy server, comprising: a sending module and a receiving module;

The sending module is configured to send a request message for uploading a target object to the object server, where the request message of the upload target object includes the target object;

The receiving module is configured to receive first location information of the target object that is sent by the object server, where the first location information of the target object includes a file identifier of the target object, and an offset of the target object And a size of the target object, where at least two objects are stored in the file indicated by the file identifier;

The sending module is further configured to send a location update message to the container server, to instruct the container server to update location information of the target object in the object list of the container server according to the location update message, where the location The update message includes identification information of the container to which the target object belongs, identification information of the target object, and first location information of the target object.
A proxy server according to claim 23, wherein

The receiving module is further configured to receive a deletion response message sent by the container server, where the deletion response message is used by the container service to delete location information of the target object in the object list, and the target location The location information of the target object is sent after being written to the delete log of the container server.
A container server, comprising: a processor and a memory coupled to the processor;

The memory is for storing computer instructions that, when the container server is running, execute the computer instructions stored by the memory to cause the container server to perform the method of any one of claims 1 to Data processing method.
An object server, comprising: a processor and a memory coupled to the processor;

The memory is configured to store computer instructions that, when the object server is running, execute the computer instructions stored by the memory to cause the object server to perform the method of any one of claims 8 to 10. Data processing method.
A proxy server, comprising: a processor and a memory coupled to the processor;

The memory is configured to store computer instructions that, when the proxy server is running, execute the computer instructions stored by the memory to cause the proxy server to perform data processing as claimed in claim 11 or method.
A computer readable storage medium, comprising computer instructions that, when executed on a container server, cause the container server to perform the data processing method of any one of claims 1 to 7.
A computer readable storage medium comprising computer instructions that, when executed on an object server, cause the object server to perform the data processing method of any one of claims 8 to 10.
A computer readable storage medium, comprising computer instructions that, when executed on a proxy server, cause the proxy server to perform the data processing method of claim 11 or 12.
An object storage system, comprising: the container server according to any one of claims 13 to 19 or claim 25, the object server according to any one of claims 20 to 22 or claim 26, A proxy server as claimed in claim 23 or 24 or claim 27.