WO2020019943A1 - 发送数据的方法和装置，以及接收数据的方法和装置 - Google Patents

发送数据的方法和装置，以及接收数据的方法和装置 Download PDF

Info

Publication number
WO2020019943A1
WO2020019943A1 PCT/CN2019/094112 CN2019094112W WO2020019943A1 WO 2020019943 A1 WO2020019943 A1 WO 2020019943A1 CN 2019094112 W CN2019094112 W CN 2019094112W WO 2020019943 A1 WO2020019943 A1 WO 2020019943A1
Authority
WO
WIPO (PCT)
Prior art keywords
small objects
large object
data
small
server
Prior art date
Application number
PCT/CN2019/094112
Other languages
English (en)
French (fr)
Inventor
丁敬文
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP19841345.2A priority Critical patent/EP3820107A4/en
Publication of WO2020019943A1 publication Critical patent/WO2020019943A1/zh
Priority to US17/158,815 priority patent/US11405458B2/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
    • H04L67/1078—Resource delivery mechanisms
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/22—Parsing or analysis of headers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004—Server selection for load balancing

Definitions

  • the present application relates to the storage field, and in particular, to a method and device for sending data, and a method and device for receiving data.
  • Simple storage service is an online storage service system provided by Amazon, which solves the problems of high construction and maintenance costs and low storage reliability when users build their own storage systems. Users can use the service interface provided by Amazon to Store any type of file on the S3 server temporarily or permanently.
  • the S3 system involves three basic concepts: bucket, object, and key. Buckets act like folders and are used to store objects.
  • the object is the basic storage unit of the S3 system, including data and metadata.
  • metadata is a special type of data used to describe the data contained in the above objects.
  • Metadata stores additional description information of the data, which can be the system metadata (system metadata) defined by the system by default, or the user metadata (user metadata) automatically.
  • the metadata is usually associated with the data Coexist, not separate.
  • a key is an identifier (ID) of an object. Each object has only one key.
  • the bucket, key, and version ID can uniquely identify an object.
  • the client establishes a connection with the S3 server, that is, the S3 connection;
  • the client reads the object stored locally
  • HTTP Hypertext Transfer Protocol
  • connection between the client and the S3 server can only process files serially, the connection is in a spatial state most of the time. How to improve the efficiency of uploading data in the S3 system is a problem that needs to be solved at present.
  • the present application provides a method and device for sending data.
  • a plurality of objects with a small amount of data to be uploaded are packaged to generate an object with a large amount of data, that is, a large object, and then through a client and an S3 server. The connection between them sends the large object to the S3 server. Since multiple small objects are uploaded as a whole (ie, large objects), the process of uploading the multiple small objects never needs to repeatedly send S3 requests, thereby improving Efficiency of uploading data in S3 system.
  • a method for sending data is provided, which is applied to an object storage system and includes: obtaining a plurality of small objects to be uploaded, where the small objects are objects whose data amount is less than or equal to the first data amount threshold ; Perform encapsulation processing on multiple small objects to generate large objects, large objects include multiple small objects; send S3 requests to the server, S3 requests are used to request upload of large objects; receive response messages from the server for S3 requests, and the response messages are used to indicate Start uploading large objects; send the large objects to the server.
  • the above embodiment can be executed by the client. Since multiple small objects are uploaded as a whole (ie, large objects), the client never needs to repeatedly send S3 requests when uploading the multiple small objects, thereby improving S3. Efficiency of uploading data in the system. In addition, because the client aggregates multiple small objects together through encapsulation processing, and does not further process the data in the small objects (for example, extract data from multiple small objects for fusion processing), the above implementation Examples further improve upload efficiency.
  • performing the encapsulation processing on a plurality of small objects to generate a large object includes: when the generation time of the plurality of small objects belongs to a preset time interval, performing the encapsulation processing on a plurality of small objects to generate the large object.
  • the preset time interval is, for example, 24 hours corresponding to a natural day. Multiple small objects generated in the preset time interval are packaged and sent to the server, which is beneficial for the server to manage the multiple small objects and improve the server. s efficiency.
  • performing a packaging process on a plurality of small objects to generate a large object includes: performing a packaging process on a plurality of small objects to generate a large object when a total amount of data encapsulated by the plurality of small objects is less than or equal to a second data amount threshold.
  • the above embodiment can avoid a situation in which uploading fails due to the large amount of data of a large object.
  • the large object further includes an object header
  • the object header includes an object name, a generation time, a value of the message digest algorithm 5, a length, and an offset in the large object of each of the multiple small objects. At least one message.
  • the above information is helpful for the server to classify and manage multiple small objects and improve the efficiency of the server.
  • the method before sending the S3 request to the server, the method further includes: establishing an HTTP connection with the server, where the HTTP connection is used to transfer a large object, and the HTTP connection is a long connection.
  • a long connection is a connection that will not be disconnected immediately after the object is uploaded. After the above large object upload is completed, if there are subsequent objects to be uploaded, the client can directly send an S3 request without re-establishing an HTTP connection, thereby reducing uploads. The overhead of data, and improve the efficiency of uploading data.
  • a method for receiving data is provided, which is applied to an object storage system, including: receiving an S3 request from a client, the S3 request is used to request upload of a large object, and the large object encapsulates multiple small objects.
  • the small objects are An object whose data amount is less than or equal to the data amount threshold; sending a response message to the client to the S3 request, the response message is used to instruct the start of uploading a large object; receiving a large object from the client.
  • the above embodiment can be executed by the server. Since multiple small objects are uploaded as a whole (ie, large objects), the server does not need to repeatedly send the S3 request response message during the process of receiving the multiple small objects, thereby improving Efficiency of uploading data in S3 system.
  • the server since the client aggregates multiple small objects together through encapsulation processing, and does not further process the data in the small objects (for example, extract data from multiple small objects for fusion processing), the above implementation Examples further improve upload efficiency.
  • the method further includes: storing a plurality of small objects to a persistence layer of the object storage system, and the storage space of the plurality of small objects is a continuous storage space; corresponding to each of the plurality of small objects Metadata is stored to the index layer of the object storage system.
  • the server After receiving multiple small objects, the server stores the multiple small objects in a continuous storage space, which is beneficial to managing the multiple small objects.
  • storing a plurality of small objects in the persistence layer of the object storage system includes: storing a plurality of small objects whose generation time or upload time belongs to a preset time interval to a continuous storage space in the persistence layer.
  • the preset time interval is, for example, 24 hours corresponding to a natural day.
  • the multiple small objects generated in the preset time interval are stored in a continuous storage space, which is beneficial for the server to manage the multiple small objects and improve the server. s efficiency.
  • the metadata is used to indicate the generation time or upload time of the small object
  • the method further includes: scanning metadata of multiple small objects to determine the generation time or upload time of the multiple small objects; When the generation time or upload time of the first small object in belongs to the expiration time interval, the first small object is deleted.
  • the first small object is an expired small object among any of the multiple small objects.
  • the server can determine the generation time or upload time of the small object by scanning the metadata of the small object. When some of the small objects belong to the expired object, the expired object is deleted. Object, which can improve the utilization of the server's storage space.
  • deleting the first small object includes: when the space utilization rate of the persistence layer is less than or equal to a space utilization threshold, In addition, when the generation time of the first small object among the multiple small objects belongs to an expiration time interval, the first small object is deleted.
  • the server can tag expired small objects first, and delete the expired small objects when the storage space utilization of the persistence layer is less than or equal to the space utilization threshold, which can improve the utilization of the server's storage space while Users provide better services.
  • the large object further includes an object header
  • the object header includes an object name, a generation time, a value of the message digest algorithm 5, a length, and an offset in the large object of each of the multiple small objects. At least one message.
  • the server can classify and manage multiple small objects according to the above information, improving the efficiency of the server.
  • the method before receiving the S3 request from the client, the method further includes: establishing an HTTP connection with the server, where the HTTP connection is used to transmit the large object, and the HTTP connection is a long connection.
  • a long connection is a connection that will not be disconnected immediately after the object is uploaded. After the above large object upload is completed, if there are subsequent objects to be uploaded, the client can directly send an S3 request without re-establishing an HTTP connection, thereby reducing uploads. The overhead of data, and improve the efficiency of uploading data.
  • a device for sending data can implement functions corresponding to each step in the method according to the first aspect, and the functions can be implemented by hardware or by executing corresponding software by hardware.
  • the hardware or software includes one or more units or modules corresponding to the functions described above.
  • the apparatus includes a processor configured to support the apparatus to perform a corresponding function in the method according to the first aspect.
  • the device may also include a memory for coupling to the processor, which stores program instructions and data necessary for the device.
  • the device further includes a communication interface for supporting communication between the device and other network elements.
  • a device for receiving data may implement functions corresponding to each step in the method according to the second aspect, and the functions may be implemented by hardware or corresponding software by hardware execution .
  • the hardware or software includes one or more units or modules corresponding to the functions described above.
  • the device includes a processor configured to support the device to perform a corresponding function in the method according to the second aspect.
  • the device may also include a memory for coupling to the processor, which stores program instructions and data necessary for the device.
  • the device further includes a communication interface for supporting communication between the device and other network elements.
  • a computer-readable storage medium stores computer program code.
  • the apparatus for sending data executes the functions of the first aspect. The method described.
  • a computer-readable storage medium stores computer program code.
  • the computer program code is executed by a processing unit or a processor, a device that receives data executes the second aspect. The method described.
  • a chip in which instructions are stored that, when run on a device that sends data, cause the chip to execute the method of the first aspect.
  • a chip in which instructions are stored that, when run on a device that receives data, cause the chip to execute the method of the second aspect.
  • a computer program product includes computer program code that, when the computer program code is run by a processing unit or processor of a device that uploads data, causes the device that sends data to execute the above first Methods.
  • a computer program product includes computer program code that, when the computer program code is run by a processing unit or a processor of a device that uploads data, causes the device that receives the data to execute the second method described above. Methods.
  • FIG. 1 is a schematic diagram of an object storage system suitable for the present application
  • FIG. 2 is a schematic flowchart of a method for uploading data based on an object storage system provided by the present application
  • FIG. 3 is a schematic diagram of a large object structure provided by the present application.
  • FIG. 4 is a schematic diagram of a metadata structure provided by the present application.
  • FIG. 5 is a schematic flowchart of a method for downloading data based on an object storage system provided by the present application
  • FIG. 6 is a schematic structural diagram of a data sending apparatus provided by the present application.
  • FIG. 7 is a schematic structural diagram of another apparatus for sending data according to the present application.
  • FIG. 8 is a schematic structural diagram of a data receiving apparatus provided by the present application.
  • FIG. 9 is a schematic structural diagram of another apparatus for receiving data provided by the present application.
  • FIG. 1 shows a schematic diagram of an object storage system suitable for the present application.
  • the object storage system includes a client 110 and a server.
  • the server includes a load balancing layer 120, an object service layer 130, a persistence layer 140, and an index layer. (index layer) 150.
  • Each of the above layers is composed of one or more servers.
  • the load balancing layer 120 includes a load balancing server 121
  • the object service layer includes an object service server 131, an object service server 132, and an object service. Server 133.
  • the persistence layer 140 is a cluster storage pool composed of multiple servers, and each server is configured with multiple hard disks to provide data persistence and redundancy protection capabilities. Hard disks on different servers in the persistence layer 140 are combined into a logically continuous resource pool (plog), and both the object service layer 130 and the index layer 150 can read and write this plog.
  • plog logically continuous resource pool
  • the index layer 150 is used to store metadata and provide indexing capabilities.
  • the client 110 When the client 110 needs to upload data, the client 110 sends an S3 request to the load balancing server 121 through the internal network or the public network, and the S3 request indicates that the client 110 needs to write an object (that is, an object needs to be uploaded), or the S3 The request indicates that the client 110 needs to read the object.
  • the load balancing server 121 may allocate the S3 request to one or more servers according to the load of each server in the object service layer 130. For example, the current load of the object server 131 is small, and the load balancing server 121 allocates the S3 request to the object server. 131,
  • the object server 131 stores the data part of the object to the persistence layer 140, and then stores the metadata of the object to the index layer 150.
  • the object server 131 If the S3 request indicates that the client 110 needs to read the object, the object server 131 first searches for the metadata indicated by the S3 request from the index layer 150, and then from the persistence layer 140 according to the data layout recorded in the metadata Read the object data corresponding to this metadata.
  • the process performed by the client 110 is as follows:
  • C1 Establish an HTTP connection with the server shown in FIG. 1, the HTTP connection is used to upload objects.
  • the HTTP connection is a long connection, that is, when an object upload is completed After that, the client 110 does not disconnect the HTTP connection, and continues to use the HTTP connection to send an S3 request to request to upload another object.
  • a small object refers to an object whose data amount is less than or equal to a first data amount threshold.
  • the first data amount threshold is, for example, 128KB. If the object obtained by the client 110 is less than 128KB, it is determined that the object is a small object. Step C4 is executed, and multiple small objects are encapsulated into large objects, and then an S3 request is sent; if the object obtained by the client 110 is larger than 128KB, an S3 request may be directly sent to request upload of the object.
  • C3 Calculate the value of the message digest algorithm (MD) 5 of each small object.
  • MD 5 value is used by the server to verify whether the information contained in the small object is complete.
  • C4 Encapsulate multiple small objects as large objects in memory.
  • the large object includes an object head, which records the object name, generation time, and MD5 value of each small object, and the offset and length of each small object in the large object ( length).
  • the client 110 first calculates the data amount of the encapsulated large object before performing the encapsulation processing on N (N is a positive integer greater than 1) small objects. If the second data amount threshold is exceeded, the client 110 may decrease the value of N. If the data amount of the encapsulated large object does not exceed the second data amount threshold, the client 110 may continue to perform encapsulation on the N small objects. deal with.
  • the second data amount threshold may be a value set according to the connection carrying capacity.
  • FIG. 3 is a schematic structural diagram of a large object generated by a client provided in the present application.
  • the client 110 obtains three small objects, obj1, obj2, and obj3, and encapsulates the three small objects into a large object.
  • the packaging format of the large object meets the transmission requirements of the HTTP connection.
  • the large object contains obj1,
  • the object header also contains the description information of each small object, that is, obj1.desc, obj2.desc, and obj3.desc.
  • the description information of obj1 contains the path name of obj1 (obj1 .name), the MD5 value of obj1 (obj1.md5), the generation date of obj1 (obj1.date), the offset of obj1 from the object head in the large object (obj1.offset), and the length of obj1 (obj1.length ).
  • the structure diagram of the above large object is just an example.
  • the packaging format of the large object may also be another format that meets the transmission requirements of the connection between the client 110 and the server.
  • the object header may also contain more types of information or less types of information. .
  • the client 110 does not change the data structure of multiple small objects by performing encapsulation processing on multiple small objects in memory, nor does it perform encryption processing or other processing on multiple small objects.
  • the small object is encapsulated as a data packet that meets the transmission needs of the connection between the client 110 and the server, which not only improves the upload efficiency of the client 110, but also facilitates the server to obtain multiple small objects from the large object.
  • C5 Send an S3 request to the server to request upload of a large object.
  • C7 Send a large object to the server based on the response message.
  • step C8 After the large object upload is completed, if the client 110 still has objects to be uploaded, the client 110 continues to perform step C2; if the client 110 does not have objects to be uploaded, execute step C9.
  • C9 Disconnect the HTTP connection and end.
  • the client 110 aggregates multiple small objects into one large object, and then sends the large object to the server, and only needs to send an S3 request to upload multiple small objects to the server.
  • the client needs to send an S3 request every time a small object is sent.
  • the above process provided in this application reduces the information overhead and improves the efficiency of uploading data in the object storage system.
  • the process performed by the server is as follows:
  • S1 Establish an HTTP connection with the client 110.
  • the HTTP connection is used to upload an object.
  • the HTTP connection is a long connection. That is, when an object is uploaded, the server does not disconnect the HTTP connection and continues to receive S3 using the HTTP connection. Request, and receive client 110 to upload another object.
  • S2 Receive an S3 request, which is used to request upload of a large object.
  • S3 Send a response message to the client 110, and the response message instructs the client 110 to start uploading a large object.
  • S5 Check the data integrity of the small object. For example, the position of each small object in the large object is determined based on the description information in the object header, and the MD5 value of each small object is calculated. If the MD5 value passes the verification, the data of the small object is complete. If the check fails, the data of the small object is incomplete.
  • S6 Write multiple small objects to the persistence layer.
  • write the multiple small objects to the plog of the persistence layer 140 that is, write multiple small data into continuous storage space.
  • Multiple small objects that belong to a large object usually have a certain correlation. For example, having the same generation time, writing multiple small objects that belong to a large object to continuous storage space is beneficial to server-side management.
  • the server reads the metadata of each small object, determines the generation time or upload time of each small object, and stores the small objects whose generation time or upload time belongs to the same time interval to a continuous storage space.
  • the time interval may be a preset interval.
  • the server can also classify multiple small objects received according to other methods, and store small objects of the same type in a continuous storage space.
  • S7 Parse the metadata of the small object from the object header of the large object, and store the metadata to the index layer 150.
  • FIG. 4 is a schematic structural diagram of metadata stored on a server side provided in the present application.
  • obj1 Take obj1 as an example.
  • the server parses the metadata of obj1 from the received large object, it writes the layout information of obj1 on the server (obj1.layout) into the metadata and deletes some useless information, such as obj1. offset and obj1.len.
  • layout includes the identifier (plogid) of the plog that stores obj1, the offset (offset) of obj1 from the starting position of the plog, and the data length (length) of obj1.
  • the data length of obj1 is used to indicate that obj1 is occupied. The number of bits.
  • the structure of the metadata shown in FIG. 4 is only an example, and the server can also add more descriptive information to the metadata of the small object.
  • step S8 After the upload of the large object is completed, if the client 110 has objects to be uploaded, the server continues to perform step S2; if the client 110 does not have objects to be uploaded, execute step S9.
  • the server only needs to receive an S3 request once to receive multiple small objects. Compared with the prior art, the server needs to receive each small object once.
  • the above process provided in this application reduces the information overhead and improves the efficiency of uploading data in the object storage system.
  • the server needs to perform alignment processing on the data when storing the data.
  • the minimum storage unit on the server is 128KB. If the server receives only a small 80K Object, the small object will occupy 128KB of storage space, causing a waste of storage resources.
  • the server often needs to make multiple backups of one data and store them on different disks. For example, if you make three backups of a small 80K object, the waste of storage resources will be magnified. 3 times.
  • the large object received by the server is a collection of multiple small objects.
  • the server receives three 80KB small objects
  • the three 80KB small objects need only two 128KB storages.
  • the existing technology requires three 128KB storage spaces to be able to store three 80KB small objects. Therefore, the method for uploading data provided in this application can improve the utilization of storage resources on the server side.
  • the server can scan the metadata of the small objects periodically or irregularly to obtain the generation time or upload time of the small objects. If the generation time or upload time of the small objects belongs to the expired time interval, the server side You can directly delete expired small objects to improve the space utilization of plogs. You can also add expired tags to the metadata of expired small objects. When the space utilization of plogs is lower than the preset utilization threshold, restart the space.
  • the recycling process that is, deleting the small objects corresponding to the metadata carrying the expired tags, thereby providing users with better services while improving space utilization.
  • the server stores small objects whose generation time or upload time belong to the same time interval according to the method shown in FIG. 2 in a continuous storage space, when a time interval becomes an expired time interval, the server can delete the corresponding time interval. All small objects and obtain continuous storage space, thereby avoiding the generation of space fragmentation, improving the utilization of storage resources on the server side, and reducing the life cycle management overhead of small objects.
  • This application also provides a method for downloading data. As shown in Figure 5, the method includes:
  • the client 110 sends an S3 request to the server.
  • the S3 request is used to request a small object, such as obj1.
  • the server After receiving the S3 request, the server reads the metadata of obj1 according to the information carried in the S3 request (for example, the name of the bucket, the name of obj1, and the path information of obj1) to obtain the layout information of obj1 (obj1. layout) to obtain the storage location of obj1.
  • the information carried in the S3 request for example, the name of the bucket, the name of obj1, and the path information of obj1
  • layout information of obj1 obj1. layout
  • the server reads obj1 from the plog corresponding to obj1 according to the storage location indicated by obj1.layout.
  • S505 The server sends obj1 to the client 110, thereby completing the process of downloading data.
  • the device for sending data and the device for receiving data include a hardware structure and / or a software module corresponding to each function.
  • this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is performed by hardware or computer software-driven hardware depends on the specific application of the technical solution and design constraints. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
  • the functional units may be divided according to the above method examples.
  • each function may be divided into various functional units, or two or more functions may be integrated into one process.
  • the above integrated unit may be implemented in the form of hardware or in the form of software functional unit. It should be noted that the division of the units in this application is schematic, and it is only a logical function division. In actual implementation, there may be another division manner.
  • FIG. 6 shows a possible structural diagram of a device for sending data provided by the present application.
  • the device 600 includes a processing unit 601, a sending unit 602, and a receiving unit 603.
  • the processing unit 601 is configured to control the apparatus 600 to execute the steps of the client shown in FIG. 2.
  • the processing unit 601 may also be used to perform other processes for the techniques described herein.
  • the device 600 may further include a storage unit 604 for storing program code and data of the device 600.
  • the processing unit 601 is configured to execute: obtaining a plurality of small objects to be uploaded, where the small objects are objects whose data amount is less than or equal to a first data amount threshold; performing a packaging process on the plurality of small objects to generate a large object, where The large object includes the plurality of small objects.
  • the sending unit 602 is configured to send an S3 request to the server, where the S3 request is used to request to upload the large object;
  • the receiving unit 603 is configured to receive a response message requested by the S3 from the server, where the response message is used to instruct to start uploading the large object;
  • the sending unit 602 is further configured to send the large object to the server.
  • the processing unit 601 may be a processor or a controller, for example, it may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), and an application-specific integrated circuit (application-specific integrated circuit). , ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logical blocks, modules, and circuits described in connection with the present disclosure.
  • the processor may also be a combination that implements computing functions, such as a combination containing one or more microprocessors, a combination of a DSP and a microprocessor, and so on.
  • the sending unit 602 and the receiving unit 603 may be communication interfaces.
  • the storage unit 604 may be a memory.
  • the processing unit 601 is a processor
  • the sending unit 602 and the receiving unit 603 are communication interfaces
  • the storage unit 604 is a memory
  • the device for sending data involved in this application may be the device shown in FIG. 7.
  • the apparatus 700 includes: a processor 701, a communication interface 702, and a memory 703.
  • the processor 701, the communication interface 702, and the memory 703 can communicate with each other through an internal connection path, and transfer control and / or data signals.
  • the device for sending data provided in this application first encapsulates a plurality of objects with a small amount of data to be uploaded to generate an object with a large amount of data, that is, a large object, and then passes the data between the client and the S3 server.
  • the connection sends the large object to the S3 server. Since multiple small objects are uploaded as a whole (that is, the large object), the process of uploading the multiple small objects never needs to repeatedly send S3 requests, thereby improving the S3 system. Efficiency of uploading data in.
  • FIG. 8 shows a possible structural diagram of a device for receiving data provided by the present application.
  • the device 800 includes a processing unit 801, a sending unit 802, and a receiving unit 803.
  • the processing unit 801 is configured to control the device 800 to execute the server-side steps shown in FIG. 2.
  • the processing unit 801 may also be used to perform other processes for the techniques described herein.
  • the device 800 may further include a storage unit 804 for storing program code and data of the device 800.
  • the processing unit 801 is used to control the receiving unit 803 to execute: receiving an S3 request from a client, the S3 request is used to request upload of a large object, the large object includes multiple small objects, and the small object has a data amount smaller than or Object equal to data volume threshold.
  • the processing unit 801 is configured to control the sending unit 802 to execute: sending a response message of the S3 request to the client, where the response message is used to instruct to start uploading the large object.
  • the processing unit 801 is further configured to control the receiving unit 803 to execute: receiving the large object from the client.
  • the processing unit 801 may be a processor or a controller, for example, it may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logical blocks, modules, and circuits described in connection with the present disclosure.
  • the processor may also be a combination that implements computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, and so on.
  • the sending unit 802 and the receiving unit 803 may be communication interfaces.
  • the storage unit 804 may be a memory.
  • the processing unit 801 is a processor
  • the sending unit 802 and the receiving unit 803 are communication interfaces
  • the storage unit 804 is a memory
  • the device for receiving data involved in this application may be the device shown in FIG. 9.
  • the apparatus 900 includes: a processor 901, a communication interface 902, and a memory 903. Among them, the processor 901, the communication interface 902, and the memory 903 may communicate with each other through an internal connection path, and transfer control and / or data signals.
  • the device for receiving data provided by this application can receive multiple small objects by receiving only one S3 request by receiving a large object containing multiple small objects. Compared with the prior art, each small object received by the server requires In the method for receiving an S3 request once, the above process provided in this application reduces the information overhead and improves the efficiency of uploading data in the object storage system.
  • the device embodiment and the method embodiment correspond exactly, and the corresponding module executes the corresponding steps, for example, the sending unit performs the sending step in the method embodiment, and the receiving unit performs the receiving step in the method embodiment, except for the sending step and receiving step.
  • the other steps can be performed by a processing unit or a processor.
  • a processing unit or a processor For the function of the specific unit, reference may be made to the corresponding method embodiment, which will not be described in detail.
  • the size of the sequence number of each process does not mean the order of execution.
  • the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.
  • the steps of the method or algorithm described in combination with the disclosure of this application may be implemented in a hardware manner, or may be implemented in a manner in which a processor executes software instructions.
  • Software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), and erasable programmable read-only memory (erasable (programmable ROM, EPROM), electrically erasable programmable read-only memory (EPROM), registers, hard disks, mobile hard disks, read-only optical disks (CD-ROMs), or any other form of storage medium known in the art.
  • An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium.
  • the storage medium may also be an integral part of the processor.
  • the processor and the storage medium may reside in an ASIC.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Transfer Between Computers (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请提供一种发送数据的方法和装置，以及一种接收数据的方法和装置。首先将待上传的多个数据量较小的对象进行封装处理，生成一个数据量较大的对象，即，大对象，随后通过客户端与S3服务器之间的连接向S3服务器发送该大对象，由于多个小对象作为一个整体(即，大对象)上传，因此，在上传该多个小对象的过程从无需再重复发送S3请求，从而提高了S3统系中上传数据的效率。

Description

发送数据的方法和装置，以及接收数据的方法和装置 技术领域
本申请涉及存储领域，尤其涉及一种发送数据的方法和装置，以及一种接收数据的方法和装置。
背景技术
简单存储服务(simple storage service，S3)是亚马逊提供的在线存储服务系统，解决了用户自己建存储系统面临的建设和维护成本高、存储可靠性低等问题，用户通过亚马逊提供的服务接口就可以将任意类型的文件临时或永久地保存在S3服务器上。
S3系统涉及三个基本概念：桶(bucket)、对象(object)和键(key)。桶的作用类似于文件夹，用于存储对象。对象是S3系统的基本存储单元，包含数据和元数据(metadata)，其中，元数据是一种特殊的数据，用来描述上述对象中包含的数据，为了便于理解，以下，若无特别声明，本申请所述的“数据”均不包含对象中的元数据。元数据存储的是数据的附加描述信息，这些信息可以是系统默认定义的系统元数据(system metadata)，也可以是自动以的用户元数据(user metadata)，元数据通常与它所关联的数据同时存在，不会单独存在。键是对象的标识(identifier，ID)，每个对象仅有一个键，桶、键和版本ID能够唯一确定一个对象。
目前，用户将数据上传至S3服务器的流程如下：
客户端与S3服务器建立连接，即S3连接；
客户端读取本地存储的对象；
构建S3请求，通过超文本传输协议(hyper text transfer protocol，HTTP)将对象传输至S3服务器。
由于客户端与S3服务器之间的连接只能串行处理文件，因此，该连接大多数时间都处于空间状态，如何提高S3系统中上传数据的效率是当前亟需解决的问题。
发明内容
本申请提供一种发送数据的方法和装置，首先将待上传的多个数据量较小的对象进行封装处理，生成一个数据量较大的对象，即，大对象，随后通过客户端与S3服务器之间的连接向S3服务器发送该大对象，由于多个小对象作为一个整体(即，大对象)上传，因此，在上传该多个小对象的过程从无需再重复发送S3请求，从而提高了S3统系中上传数据的效率。
第一方面，提供了一种发送数据的方法，其特征在于，应用于对象存储系统，包括：获取待上传的多个小对象，该小对象为数据量小于或等于第一数据量阈值的对象；对多个小对象执行封装处理生成大对象，大对象包括多个小对象；向服务器发送S3请求，S3请求用于请求上传大对象；从服务器接收S3请求的响应消息，响应消息用于指示开始上传大对象；向所述服务器发送所述大对象。
上述实施例可以由客户端执行，由于多个小对象作为一个整体(即，大对象)上传， 因此，客户端在上传该多个小对象的过程从无需再重复发送S3请求，从而提高了S3统系中上传数据的效率。此外，由于客户端通过封装处理将多个小对象聚合在一起，并未对小对象中的数据做进一步处理(例如，将多个小对象中的数据提取出来做融合处理)，因此，上述实施例进一步提高了上传效率。
可选地，对多个小对象执行封装处理生成大对象，包括：当多个小对象的生成时间属于预设的时间区间时，对多个小对象执行封装处理生成所述大对象。
预设的时间区间例如是一个自然日对应的24小时，把在预设的时间区间内生成的多个小对象封装在一起发送至服务器，有利于服务器对该多个小对象进行管理，提高服务器的效率。
可选地，对多个小对象执行封装处理生成大对象，包括：当多个小对象封装后的数据总量小于或等于第二数据量阈值时，对多个小对象执行封装处理生成大对象。
上述实施例可以避免因大对象的数据量过大导致上传失败的情况。
可选地，所述大对象还包括对象头，对象头包括多个小对象中每个小对象的对象名、生成时间、消息摘要算法5的值、长度以及在大对象中的偏移量中的至少一个信息。
上述信息有利于服务器对多个小对象进行分类管理，提高服务器的效率。
可选地，向服务器发送S3请求之前，所述方法还包括：建立与服务器之间的HTTP连接，该HTTP连接用于传输大对象，该HTTP连接为长连接。
长连接即对象上传之后不会立即断开的连接，当上述大对象上传完成后，若后续还有待上传的对象，则客户端可以直接发送S3请求，无需再重新建立HTTP连接，从而减少了上传数据的开销，并且，提高了上传数据的效率。
第二方面，提供了一种接收数据的方法，应用于对象存储系统，包括：从客户端接收S3请求，S3请求用于请求上传大对象，该大对象封装有多个小对象，小对象为数据量小于或等于数据量阈值的对象；向客户端发送S3请求的响应消息，响应消息用于指示开始上传大对象；从客户端接收大对象。
上述实施例可以由服务器执行，由于多个小对象作为一个整体(即，大对象)上传，因此，服务器在接收该多个小对象的过程从无需再重复发送S3请求的响应消息，从而提高了S3统系中上传数据的效率。此外，由于客户端通过封装处理将多个小对象聚合在一起，并未对小对象中的数据做进一步处理(例如，将多个小对象中的数据提取出来做融合处理)，因此，上述实施例进一步提高了上传效率。
可选地，所述方法还包括：将多个小对象存储至对象存储系统的持久层，并且，多个小对象的存储空间为连续的存储空间；将多个小对象中每个小对象对应的元数据存储至对象存储系统的索引层。
服务器接收到多个小对象后，将该多个小对象存储至连续的存储空间，有利于管理该多个小对象。
可选地，将多个小对象存储至对象存储系统的持久层，包括：将多个小对象中生成时间或上传时间属于预设的时间区间的小对象存储至持久层中连续的存储空间。
预设的时间区间例如是一个自然日对应的24小时，把在预设的时间区间内生成的多个小对象存储在连续的存储空间，有利于服务器对该多个小对象进行管理，提高服务器的效 率。
可选地，元数据用于指示小对象的生成时间或上传时间，所述方法还包括：扫描多个小对象的元数据，确定多个小对象的生成时间或上传时间；当多个小对象中的第一小对象的生成时间或上传时间属于过期时间区间时，删除第一小对象。
第一小对象为上述多个小对象中任意一个过期的小对象，服务器可以通过扫描小对象的元数据确定小对象的生成时间或上传时间，当其中的一些小对象属于过期对象时，删除过期对象，从而可以提高服务器的存储空间的利用率。
可选地，所述当多个小对象中的第一小对象的生成时间属于过期时间区间时，删除第一小对象，包括：当持久层的空间利用率小于或等于空间利用率阈值时，并且，当多个小对象中的第一小对象的生成时间属于过期时间区间时，删除第一小对象。
服务器可以先将过期的的小对象打上过期标签，当持久层的存储空间利用率小于或等于空间利用率阈值时，再删除过期的小对象，可以在提高服务器的存储空间的利用率的同时为用户提供更好的服务。
可选地，所述大对象还包括对象头，对象头包括多个小对象中每个小对象的对象名、生成时间、消息摘要算法5的值、长度以及在大对象中的偏移量中的至少一个信息。
服务器可以根据上述信息对多个小对象进行分类管理，提高服务器的效率。
可选地，从客户端接收S3请求之前，所述方法还包括：建立与服务器之间的HTTP连接，该HTTP连接用于传输所述大对象，该HTTP连接为长连接。
长连接即对象上传之后不会立即断开的连接，当上述大对象上传完成后，若后续还有待上传的对象，则客户端可以直接发送S3请求，无需再重新建立HTTP连接，从而减少了上传数据的开销，并且，提高了上传数据的效率。
第三方面，提供了一种发送数据的装置，该装置可以实现上述第一方面所涉及的方法中各个步骤所对应的功能，所述功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个与上述功能相对应的单元或模块。
在一种可能的设计中，该装置包括处理器，该处理器被配置为支持该装置执行上述第一方面所涉及的方法中相应的功能。该装置还可以包括存储器，该存储器用于与处理器耦合，其保存该装置必要的程序指令和数据。该装置还包括通信接口，该通信接口用于支持该装置与其它网元之间的通信。
第四方面，提供了一种接收数据的装置，该装置可以实现上述第二方面所涉及的方法中各个步骤所对应的功能，所述功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个与上述功能相对应的单元或模块。
在一种可能的设计中，该装置包括处理器，该处理器被配置为支持该装置执行上述第二方面所涉及的方法中相应的功能。该装置还可以包括存储器，该存储器用于与处理器耦合，其保存该装置必要的程序指令和数据。该装置还包括通信接口，该通信接口用于支持该装置与其它网元之间的通信。
第五方面，提供了一种计算机可读存储介质，该计算机可读存储介质中存储了计算机程序代码，该计算机程序代码被处理单元或处理器执行时，使得发送数据的装置执行第一 方面所述的方法。
第六方面，提供了一种计算机可读存储介质，该计算机可读存储介质中存储了计算机程序代码，该计算机程序代码被处理单元或处理器执行时，使得接收数据的装置执行第二方面所述的方法。
第七方面，提供了一种芯片，其中存储有指令，当其在发送数据的装置上运行时，使得该芯片执行上述第一方面的方法。
第八方面，提供了一种芯片，其中存储有指令，当其在接收数据的装置上运行时，使得该芯片执行上述第二方面的方法。
第九方面，提供了一种计算机程序产品，该计算机程序产品包括：计算机程序代码，当该计算机程序代码被上传数据的装置的处理单元或处理器运行时，使得发送数据的装置执行上述第一方面的方法。
第十方面，提供了一种计算机程序产品，该计算机程序产品包括：计算机程序代码，当该计算机程序代码被上传数据的装置的处理单元或处理器运行时，使得接收数据的装置执行上述第二方面的方法。
附图说明
图1是一种适用于本申请的对象存储系统的示意图；
图2是本申请提供的一种基于对象存储系统的上传数据的方法的示意性流程图；
图3是本申请提供的一种大对象结构的示意图；
图4是本申请提供的一种元数据结构的示意图；
图5是本申请提供的一种基于对象存储系统的下载数据的方法的示意性流程图；
图6是本申请提供的一种发送数据的装置的示意性结构图；
图7是本申请提供的另一种发送数据的装置的示意性结构图；
图8是本申请提供的一种接收数据的装置的示意性结构图；
图9是本申请提供的另一种接收数据的装置的示意性结构图。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。
图1示出了适用于本申请的对象存储系统(object storage system)的示意图。
该对象存储系统包括客户端(client)110以及服务器端，其中，服务器端包括负载均衡层(load balance layer)120、对象服务层(object service layer)130、持久层(persistence layer)140和索引层(index layer)150。上述每个层由一个或多个服务器组成，例如，负载均衡层120包括负载均衡服务器(load balance server)121，对象服务层包括对象服务服务器(object service server)131、对象服务服务器132和对象服务服务器133。
其中，持久层140是由多个服务器组成的集群存储池，每个服务器配置多块硬盘，提供数据持久化和冗余保护能力。持久层140中不同服务器上的硬盘被组合为一个逻辑连续的资源池(plog)，对象服务层130和索引层150均可以读写这个plog。
索引层150用于存储元数据，提供索引能力。
当客户端110需要上传数据时，客户端110通过内网或者公网向负载均衡服务器121发送S3请求，该S3请求指示客户端110需要写入对象(即，需要上传对象)，或者，该S3请求指示客户端110需要读取对象。
负载均衡服务器121可以根据对象服务层130中各服务器的负载情况将该S3请求分配给一个或多个服务器，例如，当前对象服务器131负载较小，负载均衡服务器121将该S3请求分配给对象服务器131，
若该S3请求指示客户端110需要写入对象，则对象服务器131收到该对象后，将该对象的数据部分存储至持久层140，随后将该对象的元数据存储至索引层150。
若该S3请求指示客户端110需要读取对象，则对象服务器131首先根据从索引层150中查找该S3请求所指示的元数据，随后根据该元数据中记录的数据布局，从持久层140中读取该元数据对应的对象数据。
下面，将详细描述本申请提供的基于上述对象存储系统的发送数据的方法和接收数据的方法。
如图2所示，客户端110所执行的流程如下：
C1：建立与图1所示的服务器端之间的HTTP连接，该HTTP连接用于上传对象，可选地，为了减少建立连接的开销，该HTTP连接为长连接，即，当一个对象上传完成后，客户端110不断开该HTTP连接，继续利用该HTTP连接发送S3请求，请求上传另一个对象。
C2：读取多个小对象。
在本申请中，小对象指的是数据量小于或等于第一数据量阈值的对象，第一数据量阈值例如是128KB，若客户端110获取的对象小于128KB，则确定该对象为小对象，执行步骤C4，将多个小对象封装为大对象之后发送S3请求；将若客户端110获得的对象大于128KB，则可以直接发送S3请求，请求上传该对象。
C3：计算每个小对象的消息摘要算法(message digest algorithm，MD)5的值，MD5值用于服务器端校验接收到的小对象所包含的信息是否完整。
C4：在内存中将上述多个小对象封装为大对象。可选地，该大对象包括对象头(head)，对象头中记录了每个小对象的对象名、产生时间、MD5值，以及每个小对象在大对象中的位移(offset)和长度(length)。
作为一个可选的示例，客户端110在对N(N为大于1的正整数)个小对象执行封装处理前，首先计算封装后的大对象的数据量，若封装后的大对象的数据量超过了第二数据量阈值，则客户端110可以减小N的值，若封装后的大对象的数据量未超过第二数据量阈值，则客户端110可以继续对该N个小对象执行封装处理。第二数据量阈值可以是根据连接承载能力设定的值。
图3示出了本申请提供的一种客户端生成的大对象的结构示意图。
客户端110获取三个小对象，分别为obj1、obj2和obj3，将该三个小对象封装为一个大对象，该大对象的封装格式符合HTTP连接的传输要求，其中，大对象除了包含obj1、obj2和obj3外，还包含对象头，对象头中包含每个小对象的描述信息，即，obj1.desc、obj2.desc和obj3.desc，例如，obj1的描述信息中包含obj1的路径名称(obj1.name)、obj1的MD5 值(obj1.md5)、obj1的生成日期(obj1.date)、obj1在大对象中相对于对象头的偏移量(obj1.offset)以及obj1的长度(obj1.length)。
上述大对象的结构示意图仅是举例说明，大对象的封装格式还可以是其它满足客户端110与服务器端的连接的传输需求的格式，对象头还可以包含更多类型的信息或者更少类型的信息。
在本申请中，客户端110通过在内存中对多个小对象执行封装处理，不改变多个小对象的数据结构，也不会对多个小对象进行加密处理或者其它处理，仅仅将多个小对象封装为满足客户端110与服务器端的连接的传输需求的数据包，这样不仅提高了客户端110的上传效率，还便于服务器端从大对象中获取多个小对象。
C5：向服务器端发送S3请求，请求上传大对象。
C6：从服务器端接收该S3请求的响应消息，该响应消息指示客户端110开始上传大对象。
C7：根据响应消息向服务器端发送大对象。
C8：该大对象上传完成后，若客户端110还有待上传的对象，则客户端110继续执行步骤C2；若客户端110没有需要上传的对象，则执行步骤C9。
C9：断开HTTP连接，结束。
由上述流程可知，客户端110通过将多个小对象聚合为一个大对象，再将该大对象发送至服务器端，仅需发送一次S3请求即可将多个小对象上传至服务器端，相比于现有技术中客户端每发送一个小对象均需发送一次S3请求的方法，本申请提供的上述流程减小了信息开销，提高了对象存储系统中上传数据的效率。
服务器端所执行的流程如下：
S1：建立与客户端110之间的HTTP连接。该HTTP连接用于上传对象，可选地，为了减少建立连接的开销，该HTTP连接为长连接，即，当一个对象上传完成后，服务器端不断开该HTTP连接，继续利用该HTTP连接接收S3请求，以及接收客户端110上传另一个对象。
S2：接收S3请求，该S3请求用于请求上传大对象。
S3：向客户端110发送响应消息，该响应消息指示客户端110开始上传大对象。
S4：从客户端110接收大对象。
S5：校验小对象的数据完整性。例如，根据对象头中的描述信息确定每个小对象在大对象中的位置，计算每个小对象的MD5值，若MD5值校验通过，则说明小对象的数据是完整的，若MD5至校验未通过，则说明小对象的数据是不完整的。
S6：将多个小对象写入持久层，可选地，将该多个小对象写入持久层140的plog，即，将多个小数据写入连续的存储空间。
属于一个大对象的多个小对象通常具有一定的关联性，例如，具有相同的生成时间，将属于一个大对象的多个小对象写入连续的存储空间有利于服务器端的管理工作。
可选地，服务器端读取每个小对象的元数据，确定每个小对象的生成时间或上传时间，将生成时间或上传时间属于同一个时间区间的小对象存储至连续的存储空间，该时间区间可以是预先设定的区间。
上述示例仅是举例说明，服务器端还可以依据其它方式将接收到的多个小对象进行分类，并将同类型的小对象存储至连续的存储空间。
S7：从大对象的对象头解析出小对象的元数据，并将该元数据存储至索引层150。
图4示出了本申请提供的一种服务器端存储的元数据的结构示意图。
以obj1为例，服务器端从接收到的大对象中解析出obj1的元数据后，将obj1在服务器端的布局信息(obj1.layout)写入元数据，并删除一些无用的信息，例如，obj1.offset和obj1.len。
obj1.layout包括存储obj1的plog的标识(plogid)、obj1相对于该plog的起始位置的偏移量(offset)以及obj1的数据长度(length)，obj1的数据长度用于指示obj1所占据的比特数。
图4所示的元数据的结构示意图仅是举例说明，服务器端还可以在小对象的元数据中添加更多的描述信息。
S8：上述大对象上传完成后，若客户端110还有待上传的对象，则服务器端继续执行步骤S2；若客户端110没有需要上传的对象，则执行步骤S9。
S9：断开HTTP连接，结束。
由上述流程可知，服务器端通过接收包含多个小对象的大对象，仅需接收一次S3请求即可接收多个小对象，相比于现有技术中服务器端每接收一个小对象均需接收一次S3请求的方法，本申请提供的上述流程减小了信息开销，提高了对象存储系统中上传数据的效率。
基于提高输入输出(input/output，I/O)性能的需求，服务器端在存储数据时需要对数据执行对齐处理，例如，服务器端的最小存储单元为128KB，若服务器端仅接收到一个80K的小对象，则该小对象将占据128KB的存储空间，从而造成了存储资源的浪费。此外，基于提高可靠性的需求，服务器端往往需要对一个数据做多个备份存储至不同的磁盘上，例如，对一个80K的小对象做3个备份，则上述存储资源的浪费情况将被放大3倍。
根据图2所示的方法，服务器端接收到的大对象为多个小对象的集合，例如，服务器接收到3个80KB的小对象，则该3个80KB的小对象仅需两个128KB的存储空间，而现有技术需要3个128KB的存储空间才能够存储3个80KB的小对象，因此，本申请提供的上传数据的方法能够提高服务器端的存储资源的利用率。
服务器端在存储小对象的过程中，可以定期或者不定期地扫描小对象的元数据，获取小对象的生成时间或者上传时间，若小对象的生成时间或者上传时间属于过期时间区间，则服务器端可以直接删除过期的小对象，以提高plog的空间利用率，也可以在过期的小对象的元数据中添加过期标签，当plog的空间利用率低于预设的利用率阈值时，再启动空间回收流程，即，删除携带过期标签的元数据对应的小对象，从而可以在提高空间利用率的同时为用户提供更优质的服务。
由于服务器端根据图2所示的方法将生成时间或上传时间属于相同时间区间的小对象存储在连续的存储空间，因此，当一个时间区间成为过期时间区间时，服务器端可以删除该时间区间对应的全部小对象，并获得连续的存储空间，从而避免了空间碎片的产生，提高了服务器端的存储资源的利用率，降低了小对象的生命周期的管理开销。
本申请还提供了一种下载数据的方法。如图5所示，该方法包括：
S501，当客户端110需要下载小对象时，建立与服务器端的HTTP连接。
S502，客户端110向服务器端发送S3请求，该S3请求用于请求获取小对象，该小对象例如是obj1。
S503，服务器端接收到该S3请求后，根据该S3请求所携带的信息(例如，桶的名字、obj1的名字和obj1的路径信息)读取obj1的元数据，获取obj1的布局信息(obj1.layout)，从而获得obj1的存储位置。
S504，服务器端根据obj1.layout指示的存储位置从obj1对应的plog中读取obj1。
S505，服务器端向客户端110发送obj1，从而完成了下载数据的流程。
上文详细描述了本申请提供的发送数据的方法的示例和接收数据的方法示例。可以理解的是，发送数据的装置和接收数据的装置为了实现上述功能，其包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请可以根据上述方法示例对发送数据的装置和接收数据的装置进行功能单元的划分，例如，可以将各个功能划分为各个功能单元，也可以将两个或两个以上的功能集成在一个处理单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。需要说明的是，本申请中对单元的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
在采用集成的单元的情况下，图6示出了本申请提供的发送数据的装置的一种可能的结构示意图。装置600包括：处理单元601、发送单元602和接收单元603。处理单元601用于控制装置600执行图2所示的客户端的步骤。处理单元601还可以用于执行本文所描述的技术的其它过程。装置600还可以包括存储单元604，用于存储装置600的程序代码和数据。
例如，处理单元601用于执行：获取待上传的多个小对象，所述小对象为数据量小于或等于第一数据量阈值的对象；对多个小对象执行封装处理生成大对象，所述大对象包括所述多个小对象。
发送单元602用于：向服务器发送S3请求，所述S3请求用于请求上传所述大对象；
接收单元603用于：从所述服务器接收S3请求的响应消息，所述响应消息用于指示开始上传所述大对象；
发送单元602还用于：向所述服务器发送所述大对象。
处理单元601可以是处理器或控制器，例如可以是中央处理器(central processing unit，CPU)，通用处理器，数字信号处理器(digital signal processor，DSP)，专用集成电路(application-specific integrated circuit，ASIC)，现场可编程门阵列(field programmable gate array，FPGA)或者其它可编程逻辑器件、晶体管逻辑器件、硬件部件或者其任意组合。其可以实现或执行结合本申请公开内容所描述的各种示例性的逻辑方框，模块和电路。所述处理器也可以是实现计算功能的组合，例如包含一个或多个微处理器组合，DSP和微处理 器的组合等等。发送单元602和接收单元603可以是通信接口。存储单元604可以是存储器。
当处理单元601为处理器，发送单元602和接收单元603为通信接口，存储单元604为存储器时，本申请所涉及的发送数据的装置可以为图7所示的装置。
参阅图7所示，该装置700包括：处理器701、通信接口702和存储器703。其中，处理器701、通信接口702和存储器703可以通过内部连接通路相互通信，传递控制和/或数据信号。
本领域的技术人员可以清楚地了解到，为了描述的方便和简洁，上述描述的装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
本申请提供的发送数据的装置，首先将待上传的多个数据量较小的对象进行封装处理，生成一个数据量较大的对象，即，大对象，随后通过客户端与S3服务器之间的连接向S3服务器发送该大对象，由于多个小对象作为一个整体(即，大对象)上传，因此，在上传该多个小对象的过程从无需再重复发送S3请求，从而提高了S3统系中上传数据的效率。
在采用集成的单元的情况下，图8示出了本申请提供的接收数据的装置的一种可能的结构示意图。装置800包括：处理单元801、发送单元802和接收单元803。处理单元801用于控制装置800执行图2所示的服务器端的步骤。处理单元801还可以用于执行本文所描述的技术的其它过程。装置800还可以包括存储单元804，用于存储装置800的程序代码和数据。
例如，处理单元801用于控制接收单元803执行：从客户端接收S3请求，所述S3请求用于请求上传大对象，所述大对象包括多个小对象，所述小对象为数据量小于或等于数据量阈值的对象。
处理单元801用于控制发送单元802执行：向所述客户端发送所述S3请求的响应消息，所述响应消息用于指示开始上传所述大对象。
处理单元801还用于控制接收单元803执行：从所述客户端接收所述大对象。
处理单元801可以是处理器或控制器，例如可以是CPU，通用处理器，DSP，ASIC，FPGA或者其它可编程逻辑器件、晶体管逻辑器件、硬件部件或者其任意组合。其可以实现或执行结合本申请公开内容所描述的各种示例性的逻辑方框，模块和电路。所述处理器也可以是实现计算功能的组合，例如包含一个或多个微处理器组合，DSP和微处理器的组合等等。发送单元802和接收单元803可以是通信接口。存储单元804可以是存储器。
当处理单元801为处理器，发送单元802和接收单元803为通信接口，存储单元804为存储器时，本申请所涉及的接收数据的装置可以为图9所示的装置。
参阅图9所示，该装置900包括：处理器901、通信接口902和存储器903。其中，处理器901、通信接口902和存储器903可以通过内部连接通路相互通信，传递控制和/或数据信号。
本领域的技术人员可以清楚地了解到，为了描述的方便和简洁，上述描述的装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
本申请提供的接收数据的装置，通过接收包含多个小对象的大对象，仅需接收一次S3请求即可接收多个小对象，相比于现有技术中服务器端每接收一个小对象均需接收一次S3 请求的方法，本申请提供的上述流程减小了信息开销，提高了对象存储系统中上传数据的效率。
装置实施例和方法实施例中完全对应，由相应的模块执行相应的步骤，例如发送单元执行方法实施例中的发送步骤，接收单元执行方法实施例中的接收步骤，除发送步骤和接收步骤以外的其它步骤可以由处理单元或处理器执行。具体单元的功能可以参考相应的方法实施例，不再详述。
在本申请各个实施例中，各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请的实施过程构成任何限定。
另外，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
结合本申请公开内容所描述的方法或者算法的步骤可以硬件的方式来实现，也可以是由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read only memory，ROM)、可擦除可编程只读存储器(erasable programmable ROM，EPROM)、电可擦可编程只读存储器(electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、只读光盘(CD-ROM)或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。
以上所述的具体实施方式，对本申请的目的、技术方案和有益效果进行了进一步详细说明，所应理解的是，以上所述仅为本申请的具体实施方式而已，并不用于限定本申请的保护范围，凡在本申请的技术方案的基础之上，所做的任何修改、等同替换、改进等，均应包括在本申请的保护范围之内。

Claims (28)

  1. 一种发送数据的方法，其特征在于，应用于对象存储系统，包括：
    获取待上传的多个小对象，所述小对象为数据量小于或等于第一数据量阈值的对象；
    对所述多个小对象执行封装处理生成大对象，所述大对象包括所述多个小对象；
    向所述服务器发送简单存储服务S3请求，所述S3请求用于请求上传所述大对象；
    从所述服务器接收所述S3请求的响应消息，所述响应消息用于指示开始上传所述大对象；
    向所述服务器发送所述大对象。
  2. 根据权利要求1所述的方法，其特征在于，所述对所述多个小对象执行封装处理生成大对象，包括：
    当所述多个小对象的生成时间属于预设的时间区间时，对所述多个小对象执行封装处理生成所述大对象。
  3. 根据权利要求1或2所述的方法，其特征在于，所述对所述多个小对象执行封装处理生成大对象，包括：
    当所述多个小对象封装后的数据总量小于或等于第二数据量阈值时，对所述多个小对象执行封装处理生成所述大对象。
  4. 根据权利要求1至3中任一项所述的方法，其特征在于，所述大对象还包括对象头，所述对象头包括所述多个小对象中每个小对象的对象名、生成时间、消息摘要算法5的值、长度以及在所述大对象中的偏移量中的至少一个信息。
  5. 根据权利要求1至4中任一项所述的方法，其特征在于，所述向所述服务器发送简单存储服务S3请求之前，所述方法还包括：
    建立与所述服务器之间的超文本传输协议HTTP连接，所述HTTP连接用于传输所述大对象，所述HTTP连接为长连接。
  6. 一种接收数据的方法，其特征在于，应用于对象存储系统，包括：
    从客户端接收简单存储服务S3请求，所述S3请求用于请求上传大对象，所述大对象包括多个小对象，所述小对象为数据量小于或等于数据量阈值的对象；
    向所述客户端发送所述S3请求的响应消息，所述响应消息用于指示开始上传所述大对象；
    从所述客户端接收所述大对象。
  7. 根据权利要求6所述的方法，其特征在于，所述方法还包括：
    将所述多个小对象存储至对象存储系统的持久层，并且，所述多个小对象的存储空间为连续的存储空间；
    将所述多个小对象中每个小对象对应的元数据存储至所述对象存储系统的索引层。
  8. 根据权利要求7所述的方法，其特征在于，所述将所述多个小对象存储至对象存储系统的持久层，包括：
    将所述多个小对象中生成时间或上传时间属于预设的时间区间的小对象存储至所述持久层中连续的存储空间。
  9. 根据权利要求7或8所述的方法，其特征在于，所述元数据用于指示所述小对象的生成时间或上传时间，
    所述方法还包括：
    扫描所述多个小对象的元数据，确定所述多个小对象的生成时间或上传时间；
    当所述多个小对象中的第一小对象的生成时间或上传时间属于过期时间区间时，删除所述第一小对象。
  10. 根据权利要求9所述的方法，其特征在于，所述当所述多个小对象中的第一小对象的生成时间或上传时间属于过期时间区间时，删除所述第一小对象，包括：
    当所述持久层的空间利用率小于或等于空间利用率阈值时，并且，当所述多个小对象中的第一小对象的生成时间或上传时间属于过期时间区间时，删除所述第一小对象。
  11. 根据权利要求6至10中任一项所述的方法，其特征在于，所述大对象还包括对象头，所述对象头包括所述多个小对象中每个小对象的对象名、生成时间、消息摘要算法5的值、长度以及在所述大对象中的偏移量中的至少一个信息。
  12. 根据权利要求6至11中任一项所述的方法，其特征在于，所述从客户端接收简单存储服务S3请求之前，所述方法还包括：
    建立与所述服务器之间的超文本传输协议HTTP连接，所述HTTP连接用于传输所述大对象，所述HTTP连接为长连接。
  13. 一种发送数据的装置，其特征在于，应用于对象存储系统，包括处理单元、发送单元和接收单元，
    所述处理单元用于：获取待上传的多个小对象，所述小对象为数据量小于或等于第一数据量阈值的对象；
    所述处理单元还用于：对所述多个小对象执行封装处理生成大对象，所述大对象包括所述多个小对象；
    所述发送单元用于：向所述服务器发送简单存储服务S3请求，所述S3请求用于请求上传所述大对象；
    所述接收单元用于：从所述服务器接收所述S3请求的响应消息，所述响应消息用于指示开始上传所述大对象；
    所述发送单元还用于：向所述服务器发送所述大对象。
  14. 根据权利要求13所述的装置，其特征在于，所述处理单元具体用于：
    当所述多个小对象的生成时间属于预设的时间区间时，对所述多个小对象执行封装处理生成所述大对象。
  15. 根据权利要求13或14所述的装置，其特征在于，所述处理单元具体用于：
    当所述多个小对象封装后的数据总量小于或等于第二数据量阈值时，对所述多个小对象执行封装处理生成所述大对象。
  16. 根据权利要求13至15中任一项所述的装置，其特征在于，所述大对象还包括对象头，所述对象头包括所述多个小对象中每个小对象的对象名、生成时间、消息摘要算法5的值、长度以及在所述大对象中的偏移量中的至少一个信息。
  17. 根据权利要求13至16中任一项所述的装置，其特征在于，所述处理单元还用于：
    建立与所述服务器之间的超文本传输协议HTTP连接，所述HTTP连接用于传输所述大对象，所述HTTP连接为长连接。
  18. 一种接收数据的装置，其特征在于，应用于对象存储系统，包括接收单元和发送单元，
    所述接收单元用于：从客户端接收简单存储服务S3请求，所述S3请求用于请求上传大对象，所述大对象包括多个小对象，所述小对象为数据量小于或等于数据量阈值的对象；
    所述发送单元用于：向所述客户端发送所述S3请求的响应消息，所述响应消息用于指示开始上传所述大对象；
    所述接收单元还用于：从所述客户端接收所述大对象。
  19. 根据权利要求18所述的装置，其特征在于，所述装置还包括处理单元，用于：
    将所述多个小对象存储至对象存储系统的持久层，并且，所述多个小对象的存储空间为连续的存储空间；
    将所述多个小对象中每个小对象对应的元数据存储至所述对象存储系统的索引层。
  20. 根据权利要求19所述的装置，其特征在于，所述处理单元具体用于：
    将所述多个小对象中生成时间或上传时间属于预设的时间区间的小对象存储至所述持久层中连续的存储空间。
  21. 根据权利要求19或20所述的装置，其特征在于，所述元数据用于指示所述小对象的生成时间或上传时间，
    所述处理单元还用于：
    扫描所述多个小对象的元数据，确定所述多个小对象的生成时间或上传时间；
    当所述多个小对象中的第一小对象的生成时间或上传时间属于过期时间区间时，删除所述第一小对象。
  22. 根据权利要求21所述的装置，其特征在于，所述处理单元具体用于：
    当所述持久层的空间利用率小于或等于空间利用率阈值时，并且，当所述多个小对象中的第一小对象的生成时间或上传时间属于过期时间区间时，删除所述第一小对象。
  23. 根据权利要求18至22中任一项所述的装置，其特征在于，所述大对象还包括对象头，所述对象头包括所述多个小对象中每个小对象的对象名、生成时间、消息摘要算法5的值、长度以及在所述大对象中的偏移量中的至少一个信息。
  24. 根据权利要求18至23中任一项所述的装置，其特征在于，所述处理单元还用于：
    建立与所述服务器之间的超文本传输协议HTTP连接，所述HTTP连接用于传输所述大对象，所述HTTP连接为长连接。
  25. 一种发送数据的装置，其特征在于，包括：
    存储器，用于存储指令，
    通信接口，用于接收或者发送数据，
    处理器，与所述存储器和所述通信接口耦合，用于调用所述存储器存储的指令执行权利要求1至权利要求5中任一项所述的方法的步骤，以及，用于调用所述存储器存储的指令控制所述通信接口执行权利要求1至权利要求5中任一项所述的方法的步骤。
  26. 一种发送数据的装置，其特征在于，包括：
    存储器，用于存储指令，
    通信接口，用于接收或者发送数据，
    处理器，与所述存储器和所述通信接口耦合，用于调用所述存储器存储的指令执行权利要求6至权利要求12中任一项所述的方法的步骤，以及，用于调用所述存储器存储的指令控制所述通信接口执行权利要求6至权利要求12中任一项所述的方法的步骤。
  27. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储了计算机程序代码，当所述计算机程序代码被处理单元或处理器执行时，发送数据的装置执行权利要求1至权利要求5中任一项所述的方法的步骤。
  28. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储了计算机程序代码，当所述计算机程序代码被处理单元或处理器执行时，接收数据的装置执行权利要求6至权利要求12中任一项所述的方法的步骤。
PCT/CN2019/094112 2018-07-27 2019-07-01 发送数据的方法和装置，以及接收数据的方法和装置 WO2020019943A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19841345.2A EP3820107A4 (en) 2018-07-27 2019-07-01 DATA TRANSMISSION DEVICE AND METHOD, AND DATA RECEPTION METHOD AND APPARATUS
US17/158,815 US11405458B2 (en) 2018-07-27 2021-01-26 Sending and receiving data using multiple objects with a single storage request

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810844624.XA CN109067733B (zh) 2018-07-27 2018-07-27 发送数据的方法和装置，以及接收数据的方法和装置
CN201810844624.X 2018-07-27

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/158,815 Continuation US11405458B2 (en) 2018-07-27 2021-01-26 Sending and receiving data using multiple objects with a single storage request

Publications (1)

Publication Number Publication Date
WO2020019943A1 true WO2020019943A1 (zh) 2020-01-30

Family

ID=64836776

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/094112 WO2020019943A1 (zh) 2018-07-27 2019-07-01 发送数据的方法和装置，以及接收数据的方法和装置

Country Status (4)

Country Link
US (1) US11405458B2 (zh)
EP (1) EP3820107A4 (zh)
CN (1) CN109067733B (zh)
WO (1) WO2020019943A1 (zh)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109067733B (zh) 2018-07-27 2021-01-05 成都华为技术有限公司 发送数据的方法和装置，以及接收数据的方法和装置
CN109828828A (zh) * 2019-01-23 2019-05-31 杭州宏杉科技股份有限公司 对象上传方法、装置、电子设备及机器可读存储介质
CN111240855A (zh) * 2019-12-31 2020-06-05 中国建设银行股份有限公司 Api调试系统、方法、存储介质和计算机设备
CN111367865A (zh) * 2020-03-10 2020-07-03 山东汇贸电子口岸有限公司 一种基于aws s3的文件分片上传的实现方法
CN111581015B (zh) * 2020-04-14 2021-06-29 上海爱数信息技术股份有限公司 一种现代应用的持续数据保护系统及方法
CN111581016B (zh) * 2020-04-14 2021-05-18 上海爱数信息技术股份有限公司 一种现代应用的副本数据管理系统及方法
CN112579006A (zh) * 2020-12-25 2021-03-30 华录光存储研究院（大连）有限公司 一种数据存储生命周期管理方法及系统
CN114697376A (zh) * 2022-03-16 2022-07-01 浪潮云信息技术股份公司 一种超大消息传输方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106686148A (zh) * 2017-03-20 2017-05-17 郑州云海信息技术有限公司 一种用于提高对象存储系统中对象上传速度的方法和系统
WO2017161050A2 (en) * 2016-03-15 2017-09-21 Cloud Crowding Corp. Distributed storage system data management and security
CN107566463A (zh) * 2017-08-21 2018-01-09 北京航空航天大学 一种提高存储可用性的多云存储管理系统
CN107911461A (zh) * 2017-11-24 2018-04-13 网宿科技股份有限公司 云存储系统中的对象处理方法、存储服务器及云存储系统
CN109067733A (zh) * 2018-07-27 2018-12-21 成都华为技术有限公司 发送数据的方法和装置，以及接收数据的方法和装置

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8914330B2 (en) * 2004-09-17 2014-12-16 International Business Machines Corporation Bulk deletion through segmented files
CN101106550A (zh) * 2007-08-31 2008-01-16 金蝶软件（中国）有限公司 大尺寸消息的发送方法、接收方法及传输系统
CN102262539A (zh) * 2011-07-29 2011-11-30 深圳市科陆电子科技股份有限公司 一种基于struts2技术自定义的分页dataTable控件的实现方法
US9058338B2 (en) * 2011-10-26 2015-06-16 International Business Machines Corporation Storing a small file with a reduced storage and memory footprint
US9117027B2 (en) * 2012-09-03 2015-08-25 Tata Consultancy Services Limited Method and system for compliance testing in a cloud storage environment
CN104247376B (zh) * 2013-04-02 2018-06-26 华为技术有限公司 云存储的文件上传方法、客户端、应用服务器及云存储系统
US9104940B2 (en) * 2013-08-30 2015-08-11 Konica Minolta Laboratory U.S.A., Inc. Line segmentation method applicable to document images containing handwriting and printed text characters or skewed text lines
US10261725B2 (en) * 2015-04-10 2019-04-16 Toshiba Memory Corporation Storage system capable of invalidating data stored in a storage device thereof
US9912752B1 (en) * 2015-06-29 2018-03-06 Amazon Technologies, Inc. Retention-based data management in a network-based data store
CN105653209A (zh) * 2015-12-31 2016-06-08 浪潮(北京)电子信息产业有限公司 一种对象存储数据传输方法及装置
CN107040474A (zh) * 2016-02-03 2017-08-11 中兴通讯股份有限公司 数据包发送方法、接收方法、发送装置及接收装置
US10810157B1 (en) * 2016-09-28 2020-10-20 Amazon Technologies, Inc. Command aggregation for data storage operations

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017161050A2 (en) * 2016-03-15 2017-09-21 Cloud Crowding Corp. Distributed storage system data management and security
CN106686148A (zh) * 2017-03-20 2017-05-17 郑州云海信息技术有限公司 一种用于提高对象存储系统中对象上传速度的方法和系统
CN107566463A (zh) * 2017-08-21 2018-01-09 北京航空航天大学 一种提高存储可用性的多云存储管理系统
CN107911461A (zh) * 2017-11-24 2018-04-13 网宿科技股份有限公司 云存储系统中的对象处理方法、存储服务器及云存储系统
CN109067733A (zh) * 2018-07-27 2018-12-21 成都华为技术有限公司 发送数据的方法和装置，以及接收数据的方法和装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3820107A4 *

Also Published As

Publication number Publication date
US20210160319A1 (en) 2021-05-27
US11405458B2 (en) 2022-08-02
EP3820107A4 (en) 2021-07-07
EP3820107A1 (en) 2021-05-12
CN109067733A (zh) 2018-12-21
CN109067733B (zh) 2021-01-05

Similar Documents

Publication Publication Date Title
WO2020019943A1 (zh) 发送数据的方法和装置，以及接收数据的方法和装置
US11928029B2 (en) Backup of partitioned database tables
US11036591B2 (en) Restoring partitioned database tables from backup
US11789925B2 (en) System and method for conditionally updating an item with attribute granularity
US11327949B2 (en) Verification of database table partitions during backup
US20230400990A1 (en) System and method for performing live partitioning in a data store
US20210103604A1 (en) System and method for implementing a scalable data storage service
US9372911B2 (en) System and method for performing replica copying using a physical copy mechanism
US8819027B1 (en) System and method for partitioning and indexing table data using a composite primary key
US11609697B2 (en) System and method for providing a committed throughput level in a data store
CN104573068A (zh) 一种基于大数据的信息处理方法
CN111262726B (zh) 一种配置信息更新方法、装置及计算机可读存储介质
WO2023011022A1 (zh) 基于区块链的数据处理方法、设备及计算机可读存储介质
WO2023020242A1 (zh) 基于区块链的数据处理方法、装置、计算机设备、计算机可读存储介质及计算机程序产品
WO2023098050A1 (zh) 远程数据访问方法及装置
WO2023142605A1 (zh) 一种基于区块链的数据处理方法和相关装置
CN110417860A (zh) 文件传输管理方法、装置、设备及存储介质
PH12015500675B1 (en) Method and system for transmission management of full configuration synchronization between eml-nml.
JP2022530953A (ja) 管理デバイスによってリモートデバイスを管理する方法
CN114172945B (zh) 一种模拟实现全双工即时通信方法与设备
CN116055450A (zh) 通讯录数据的处理方法、装置和存储介质及电子设备
CN117332009A (zh) 一种数据处理方法及相关设备
CN117255130A (zh) 一种基于区块链的数据处理方法、装置、设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19841345

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE