CN114546273B - Method, system, device and storage medium for aggregation property compatible multi-site synchronization - Google Patents

Method, system, device and storage medium for aggregation property compatible multi-site synchronization Download PDF

Info

Publication number
CN114546273B
CN114546273B CN202210161548.9A CN202210161548A CN114546273B CN 114546273 B CN114546273 B CN 114546273B CN 202210161548 A CN202210161548 A CN 202210161548A CN 114546273 B CN114546273 B CN 114546273B
Authority
CN
China
Prior art keywords
aggregation
objects
attribute
head
pool
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210161548.9A
Other languages
Chinese (zh)
Other versions
CN114546273A (en
Inventor
何文龙
武模仁
李欢欢
陶桐桐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN202210161548.9A priority Critical patent/CN114546273B/en
Publication of CN114546273A publication Critical patent/CN114546273A/en
Application granted granted Critical
Publication of CN114546273B publication Critical patent/CN114546273B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604—Improving or facilitating administration, e.g. storage management
    • G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0656—Data buffering arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention provides a method, a system, equipment and a storage medium for synchronizing aggregation characteristic compatible multi-site, wherein the method comprises the following steps: the synchronous gateway pulls the objects of other sites according to the log, and judges whether the objects are aggregation objects or not; responding to the object being an aggregation object, deleting the aggregation attribute of the object and writing the object into a solid state disk pool; traversing each hot pool object, and aggregating objects smaller than a preset value to form a first aggregate object which is stored in a cache; and adding an aggregation extension attribute in the head object of the object smaller than the preset value and deleting the data in the head object. The invention improves the compatibility of the characteristics in the object storage, reduces the number of the objects of each site, improves the reconstruction speed and the performance of the system, and improves the utilization rate of the storage space.

Description

Method, system, device and storage medium for aggregation property compatible multi-site synchronization
Technical Field
The present invention relates to the field of distributed object storage systems, and more particularly, to a method, system, device, and storage medium for aggregation property compatible multi-site synchronization.
Background
In the current distributed object storage system, a data synchronization function between a plurality of sites is supported, objects uploaded at one site are automatically synchronized to other sites, objects can be downloaded at other sites, and the like, and cross-domain cooperation is performed. In the current distributed object storage system, the function of small object aggregation is realized aiming at common massive small objects, a plurality of small objects are aggregated into one large object, the number of objects of Rados (distributed object storage) is reduced, the reconstruction speed of clusters can be greatly increased, and the performance of the system is improved. However, the current small object aggregation function is not compatible with multi-site synchronization, and has the following problems: after synchronizing the aggregated object data to other sites, synchronizing the attribute of the object aggregated metadata in one block, but not synchronously aggregating large objects to other sites, so that the object data cannot be found when the other sites download the objects, namely the data is lost; and a second problem: objects synchronized to other sites do not write to ssd (solid state drive, solid state disk) pools, but rather to hd (hard disk drive) pools, resulting in other sites not aggregating synchronized objects.
Disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a method, a system, a computer device and a computer readable storage medium for synchronizing multiple sites with aggregated characteristics, which improve the compatibility of characteristics in object storage, reduce the number of objects in each site, improve the reconstruction speed and the performance of the system, improve the utilization rate of storage space, bring good user experience to clients, enrich the functions of object storage, and improve the market competitiveness of distributed object products.
Based on the above objects, an aspect of the embodiments of the present invention provides a method for synchronizing multiple stations compatible with aggregation characteristics, including the following steps: the synchronous gateway pulls the objects of other sites according to the log, and judges whether the objects are aggregation objects or not; responding to the object being an aggregation object, deleting the aggregation attribute of the object and writing the object into a solid state disk pool; traversing each hot pool object, and aggregating objects smaller than a preset value to form a first aggregate object which is stored in a cache; and adding an aggregation extension attribute in the head object of the object smaller than the preset value and deleting the data in the head object.
In some embodiments, the method further comprises: and in response to reaching a brushing condition, reading the first aggregation object from the cache and writing the first aggregation object into an object storage device disk.
In some embodiments, the method further comprises: responding to a request of a server for receiving a downloaded object sent by a client, and reading an aggregation attribute from a head object of the object; and acquiring a first aggregation object of the object according to the aggregation attribute, acquiring data corresponding to the object from the first aggregation object, and returning the data to the client.
In some embodiments, the method further comprises: and adding a storage switch in the synchronous gateway, and responding to the fact that the object pulled by the synchronous gateway is an aggregation object, and starting the storage switch to write the pulled object into a solid state disk pool.
In another aspect of the embodiments of the present invention, there is provided a system for aggregation-characteristic-compatible multi-site synchronization, including: the judging module is configured to pull the objects of other sites according to the log by the synchronous gateway and judge whether the objects are aggregation objects or not; the deleting module is configured to respond to the fact that the object is an aggregation object, delete the aggregation attribute of the object and write the object into the solid state disk pool; the traversing module is configured to traverse each hot pool object and aggregate objects smaller than a preset value to form a first aggregate object and store the first aggregate object in the cache; and an execution module configured to add an aggregate extension attribute to a header object of the object smaller than a preset value and delete data in the header object.
In some embodiments, the system further comprises a reading module configured to: and in response to reaching a brushing condition, reading the first aggregation object from the cache and writing the first aggregation object into an object storage device disk.
In some embodiments, the system further comprises a download module configured to: responding to a request of a server for receiving a downloaded object sent by a client, and reading an aggregation attribute from a head object of the object; and acquiring a first aggregation object of the object according to the aggregation attribute, acquiring data corresponding to the object from the first aggregation object, and returning the data to the client.
In some embodiments, the system further comprises a switch module configured to: and adding a storage switch in the synchronous gateway, and responding to the fact that the object pulled by the synchronous gateway is an aggregation object, and starting the storage switch to write the pulled object into a solid state disk pool.
In yet another aspect of the embodiment of the present invention, there is also provided a computer apparatus, including: at least one processor; and a memory storing computer instructions executable on the processor, which when executed by the processor, perform the steps of the method as above.
In yet another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the method steps as described above.
The invention has the following beneficial technical effects: the method improves the compatibility of characteristics in the object storage, reduces the number of objects of each site, improves the reconstruction speed and the performance of the system, improves the storage space utilization rate, brings good user experience to clients, enriches the functions of the object storage, and improves the market competitiveness of distributed object products.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other embodiments may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of an embodiment of a method for aggregation property compatible multi-site synchronization provided by the present invention;
FIG. 2 is a schematic diagram of an embodiment of a system for aggregation-characteristics compatible multi-site synchronization provided by the present invention;
FIG. 3 is a schematic hardware architecture diagram of an embodiment of an aggregate property compatible multi-site synchronized computer device provided by the present invention;
FIG. 4 is a schematic diagram of an embodiment of an aggregate property compatible multi-site synchronized computer storage medium provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention will be described in further detail with reference to the accompanying drawings.
It should be noted that, in the embodiments of the present invention, all the expressions "first" and "second" are used to distinguish two entities with the same name but different entities or different parameters, and it is noted that the "first" and "second" are only used for convenience of expression, and should not be construed as limiting the embodiments of the present invention, and the following embodiments are not described one by one.
In a first aspect of the embodiment of the present invention, an embodiment of a method for aggregation property-compatible multi-site synchronization is provided. Fig. 1 is a schematic diagram of an embodiment of a method for aggregation property compatible multi-site synchronization provided by the present invention. As shown in fig. 1, the embodiment of the present invention includes the following steps:
s1, the synchronous gateway pulls objects of other sites according to the log, and judges whether the objects are aggregation objects or not;
s2, deleting the aggregation attribute of the object and writing the object into a solid state disk pool in response to the object being the aggregation object;
s3, traversing each hot pool object, and aggregating objects smaller than a preset value to form a first aggregation object and storing the first aggregation object in a cache; and
s4, adding the aggregation extension attribute in the head object of the object smaller than the preset value and deleting the data in the head object.
In the distributed storage system, the current aggregation function needs to be performed on the basis of opening and grading, when small objects are written into the system through front-end business, the small objects are written into a rapid pool first, then when data are transferred from a hot pool to a cold pool, whether the small objects are aggregated into large objects is judged according to agg (aggregation) configuration information of a bucket, and then the aggregated large objects are brushed down into a rados storage pool. Under the conditions of normal uploading and aggregation starting, the objects smaller than 512KB are converged into a large object of 4M, the data in the head object is deleted, and meanwhile aggregation information is added in the small object extension attribute to index to the actual storage position of the data, so that the data is read.
The embodiment of the invention aims to solve the compatibility of aggregation characteristics and multi-station synchronization characteristics. Firstly, the business gateway aggregates the small objects after uploading the objects. The synchronization gateway of another site then pulls the object.
And the synchronous gateway pulls the objects of other sites according to the log and judges whether the objects are aggregation objects or not. And deleting the aggregation attribute of the object and writing the object into the solid state disk pool in response to the object being the aggregation object. Firstly, whether the object is an aggregation object is judged, if the object is an aggregation object, the aggregation attribute of the object needs to be deleted, and then the pulled object is saved so as to ensure that the object is downloaded correctly.
In some embodiments, the method further comprises: and adding a storage switch in the synchronous gateway, and responding to the fact that the object pulled by the synchronous gateway is an aggregation object, and starting the storage switch to write the pulled object into a solid state disk pool. And adding a switch in the synchronous gateway to enable the pulled object to be stored in the ssd pool, so that the pulled object can be aggregated at the site.
Traversing each hot pool object, and aggregating objects smaller than a preset value to form a first aggregate object to be stored in a cache. And adding the aggregation extension attribute in the head object of the object smaller than the preset value and deleting the data in the head object.
In some embodiments, the method further comprises: and in response to reaching a brushing condition, reading the first aggregation object from the cache and writing the first aggregation object into an object storage device disk.
In some embodiments, the method further comprises: responding to a request of a server for receiving a downloaded object sent by a client, and reading an aggregation attribute from a head object of the object; and acquiring a first aggregation object of the object according to the aggregation attribute, acquiring data corresponding to the object from the first aggregation object, and returning the data to the client. When downloading the aggregation small object, each site client firstly judges whether the object is an aggregation object according to whether aggregation attributes exist or not, if not, the normal downloading flow is carried out; if the object is aggregated, acquiring an aggregated large object stored by the small object data according to the aggregated information in the aggregated attribute, further acquiring the data corresponding to the aggregated large object, and returning the data to the client.
The polymerization procedure is described as follows:
(1) The aggregation thread traverses each hot pool object, and aggregates small objects smaller than 512k into a large object to be stored in a cache;
(2) After the condition of brushing the disk is reached, reading the aggregated large Object from the cache and writing the aggregated large Object into an osd (Object-based Storage Device, object storage device) disk;
(3) And adding the aggregation extension attribute in the original small object head object, and deleting the data in the original small object head object.
The following is described for the synchronization flow:
(1) And the synchronous gateway pulls the objects of other sites according to the log.
(2) Determining whether the pulled object is an aggregate object
a) If the object is an aggregation object, deleting the aggregation attribute of the object;
b) If the object is not the aggregation object, the normal synchronous flow is carried out;
(3) Judging whether a newly added switch (judging whether to write the pulling object into ssd) in the synchronous gateway is started:
a) If the switch is opened, writing the pulled object into the site ssd pool;
b) If the switch is closed, writing the pulled object into the site hdd pool;
(4) The objects are aggregated at the site, and the aggregation process is as above.
The following is described for the download flow:
(1) After receiving a request of downloading an object sent by a client, a server side firstly reads an aggregation attribute from a head object of a small object, if the attribute does not exist, the original downloading flow is carried out, if the attribute exists, an aggregation large object stored by the object is required to be acquired according to aggregation information, and data corresponding to the small object is acquired from the aggregation large object;
(2) And returning the read data to the client.
The invention provides a scheme for synchronizing aggregation compatible multiple sites in a distributed object storage system, which solves the problem of synchronizing the aggregation incompatible multiple sites in the current aggregation scheme, improves the compatibility of characteristics in object storage, reduces the number of objects of each site, improves the reconstruction speed and the performance of the system, improves the utilization rate of storage space, brings good user experience to clients, enriches the functions of object storage, and improves the market competitiveness of distributed object products.
It should be noted that, the steps in the embodiments of the method for synchronizing multiple stations with the aggregation characteristic can be intersected, replaced, added and deleted, so that the method for synchronizing multiple stations with the aggregation characteristic with the reasonable permutation and combination transformation shall also belong to the protection scope of the present invention, and shall not limit the protection scope of the present invention to the embodiments.
Based on the above object, a second aspect of the embodiments of the present invention proposes a system with aggregation characteristics compatible with multi-site synchronization. As shown in fig. 2, the system 200 includes the following modules: the judging module is configured to pull the objects of other sites according to the log by the synchronous gateway and judge whether the objects are aggregation objects or not; the deleting module is configured to respond to the fact that the object is an aggregation object, delete the aggregation attribute of the object and write the object into the solid state disk pool; the traversing module is configured to traverse each hot pool object and aggregate objects smaller than a preset value to form a first aggregate object and store the first aggregate object in the cache; and an execution module configured to add an aggregate extension attribute to a header object of the object smaller than a preset value and delete data in the header object.
In some embodiments, the system further comprises a reading module configured to: and in response to reaching a brushing condition, reading the first aggregation object from the cache and writing the first aggregation object into an object storage device disk.
In some embodiments, the system further comprises a download module configured to: responding to a request of a server for receiving a downloaded object sent by a client, and reading an aggregation attribute from a head object of the object; and acquiring a first aggregation object of the object according to the aggregation attribute, acquiring data corresponding to the object from the first aggregation object, and returning the data to the client.
In some embodiments, the system further comprises a switch module configured to: and adding a storage switch in the synchronous gateway, and responding to the fact that the object pulled by the synchronous gateway is an aggregation object, and starting the storage switch to write the pulled object into a solid state disk pool.
In view of the above object, a third aspect of the embodiments of the present invention provides a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions being executable by the processor to perform the steps of: s1, the synchronous gateway pulls objects of other sites according to the log, and judges whether the objects are aggregation objects or not; s2, deleting the aggregation attribute of the object and writing the object into a solid state disk pool in response to the object being the aggregation object; s3, traversing each hot pool object, and aggregating objects smaller than a preset value to form a first aggregation object and storing the first aggregation object in a cache; and S4, adding the aggregation extension attribute in the head object of the object smaller than the preset value and deleting the data in the head object.
In some embodiments, the steps further comprise: and in response to reaching a brushing condition, reading the first aggregation object from the cache and writing the first aggregation object into an object storage device disk.
In some embodiments, the steps further comprise: responding to a request of a server for receiving a downloaded object sent by a client, and reading an aggregation attribute from a head object of the object; and acquiring a first aggregation object of the object according to the aggregation attribute, acquiring data corresponding to the object from the first aggregation object, and returning the data to the client.
In some embodiments, the steps further comprise: and adding a storage switch in the synchronous gateway, and responding to the fact that the object pulled by the synchronous gateway is an aggregation object, and starting the storage switch to write the pulled object into a solid state disk pool.
As shown in fig. 3, a schematic hardware structure of an embodiment of the computer device with aggregation characteristics compatible with multi-site synchronization according to the present invention is shown.
Taking the example of the device shown in fig. 3, a processor 301 and a memory 302 are included in the device.
The processor 301 and the memory 302 may be connected by a bus or otherwise, for example in fig. 3.
The memory 302 is used as a non-volatile computer readable storage medium, and can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as program instructions/modules corresponding to the aggregation characteristic compatible multi-site synchronization method in the embodiments of the present application. The processor 301 executes various functional applications of the server and data processing, i.e., a method of achieving aggregation property compatible multi-site synchronization, by running non-volatile software programs, instructions, and modules stored in the memory 302.
Memory 302 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created according to the use of aggregation characteristics compatible with the multi-site synchronization method, and the like. In addition, memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 302 may optionally include memory located remotely from processor 301, which may be connected to the local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Computer instructions 303 corresponding to one or more methods of aggregate property compatible multi-site synchronization are stored in memory 302 that, when executed by processor 301, perform the methods of aggregate property compatible multi-site synchronization in any of the method embodiments described above.
Any one embodiment of a computer device that performs the above method for aggregate property-compatible multi-site synchronization may achieve the same or similar effects as any of the foregoing method embodiments corresponding thereto.
The invention also provides a computer readable storage medium storing a computer program which when executed by a processor performs a method of aggregation property compatible multi-site synchronization.
FIG. 4 is a schematic diagram of an embodiment of the aggregation characteristics compatible with multi-site synchronization according to the present invention. Taking a computer storage medium as shown in fig. 4 as an example, the computer readable storage medium 401 stores a computer program 402 that when executed by a processor performs the above method.
Finally, it should be noted that, as will be appreciated by those skilled in the art, implementing all or part of the above-described methods in the embodiments may be implemented by a computer program to instruct related hardware, and the program for aggregating the methods of the feature-compatible multi-site synchronization may be stored in a computer readable storage medium, where the program when executed may include the steps of the embodiments of the methods described above. The storage medium of the program may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (RAM), or the like. The computer program embodiments described above may achieve the same or similar effects as any of the method embodiments described above.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The foregoing embodiment of the present invention has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, and the program may be stored in a computer readable storage medium, where the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will appreciate that: the above discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the invention, including the claims, is limited to such examples; combinations of features of the above embodiments or in different embodiments are also possible within the idea of an embodiment of the invention, and many other variations of the different aspects of the embodiments of the invention as described above exist, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present invention.

Claims (10)

1. A method for aggregation-characteristic-compatible multi-site synchronization, comprising the steps of:
the synchronous gateway pulls the objects of other sites according to the log, and judges whether the objects are aggregation objects or not;
responding to the object being an aggregation object, deleting the aggregation attribute of the object and writing the object into a solid state disk pool;
traversing each hot pool object, and aggregating objects smaller than a preset value to form a first aggregate object which is stored in a cache; and
and adding the aggregation extension attribute in the head object of the object smaller than the preset value and deleting the data in the head object.
2. The method according to claim 1, wherein the method further comprises:
and in response to reaching a brushing condition, reading the first aggregation object from the cache and writing the first aggregation object into an object storage device disk.
3. The method according to claim 1, wherein the method further comprises:
responding to a request of a server for receiving a downloaded object sent by a client, and reading an aggregation attribute from a head object of the object;
and acquiring a first aggregation object of the object according to the aggregation attribute, acquiring data corresponding to the object from the first aggregation object, and returning the data to the client.
4. The method according to claim 1, wherein the method further comprises:
and adding a storage switch in the synchronous gateway, and responding to the fact that the object pulled by the synchronous gateway is an aggregation object, and starting the storage switch to write the pulled object into a solid state disk pool.
5. A system for aggregation-characteristic-compatible multi-site synchronization, comprising:
the judging module is configured to pull the objects of other sites according to the log by the synchronous gateway and judge whether the objects are aggregation objects or not;
the deleting module is configured to respond to the fact that the object is an aggregation object, delete the aggregation attribute of the object and write the object into the solid state disk pool;
the traversing module is configured to traverse each hot pool object and aggregate objects smaller than a preset value to form a first aggregate object and store the first aggregate object in the cache; and
and the execution module is configured to add the aggregation extension attribute in the head object of the object smaller than the preset value and delete the data in the head object.
6. The system of claim 5, further comprising a reading module configured to:
and in response to reaching a brushing condition, reading the first aggregation object from the cache and writing the first aggregation object into an object storage device disk.
7. The system of claim 5, further comprising a download module configured to:
responding to a request of a server for receiving a downloaded object sent by a client, and reading an aggregation attribute from a head object of the object;
and acquiring a first aggregation object of the object according to the aggregation attribute, acquiring data corresponding to the object from the first aggregation object, and returning the data to the client.
8. The system of claim 5, further comprising a switch module configured to:
and adding a storage switch in the synchronous gateway, and responding to the fact that the object pulled by the synchronous gateway is an aggregation object, and starting the storage switch to write the pulled object into a solid state disk pool.
9. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, which when executed by the processor, perform the steps of the method of any one of claims 1-4.
10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of any of claims 1-4.
CN202210161548.9A 2022-02-22 2022-02-22 Method, system, device and storage medium for aggregation property compatible multi-site synchronization Active CN114546273B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210161548.9A CN114546273B (en) 2022-02-22 2022-02-22 Method, system, device and storage medium for aggregation property compatible multi-site synchronization

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210161548.9A CN114546273B (en) 2022-02-22 2022-02-22 Method, system, device and storage medium for aggregation property compatible multi-site synchronization

Publications (2)

Publication Number Publication Date
CN114546273A CN114546273A (en) 2022-05-27
CN114546273B true CN114546273B (en) 2023-08-04

Family

ID=81677922

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210161548.9A Active CN114546273B (en) 2022-02-22 2022-02-22 Method, system, device and storage medium for aggregation property compatible multi-site synchronization

Country Status (1)

Country Link
CN (1) CN114546273B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101568919A (en) * 2005-07-12 2009-10-28 微软公司 Single view of data in a networked computer system with distributed storage
CN111581157A (en) * 2020-05-07 2020-08-25 广州市百果园信息技术有限公司 Object storage platform, object operation method and device and server
CN112988899A (en) * 2021-04-02 2021-06-18 山东英信计算机技术有限公司 Additional data synchronization method, system, storage medium and equipment
CN113553325A (en) * 2021-06-18 2021-10-26 济南浪潮数据技术有限公司 Synchronization method and system for aggregation objects in object storage system
CN113687783A (en) * 2021-07-31 2021-11-23 济南浪潮数据技术有限公司 Object aggregation method, system, device and computer storage medium
CN113821166A (en) * 2021-08-20 2021-12-21 济南浪潮数据技术有限公司 Method, device and equipment for aggregating multi-version small objects

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050172072A1 (en) * 2004-01-30 2005-08-04 Cochran Robert A. Multiple site data replication

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101568919A (en) * 2005-07-12 2009-10-28 微软公司 Single view of data in a networked computer system with distributed storage
CN111581157A (en) * 2020-05-07 2020-08-25 广州市百果园信息技术有限公司 Object storage platform, object operation method and device and server
CN112988899A (en) * 2021-04-02 2021-06-18 山东英信计算机技术有限公司 Additional data synchronization method, system, storage medium and equipment
CN113553325A (en) * 2021-06-18 2021-10-26 济南浪潮数据技术有限公司 Synchronization method and system for aggregation objects in object storage system
CN113687783A (en) * 2021-07-31 2021-11-23 济南浪潮数据技术有限公司 Object aggregation method, system, device and computer storage medium
CN113821166A (en) * 2021-08-20 2021-12-21 济南浪潮数据技术有限公司 Method, device and equipment for aggregating multi-version small objects

Also Published As

Publication number Publication date
CN114546273A (en) 2022-05-27

Similar Documents

Publication Publication Date Title
CN107861686A (en) File memory method, service end and computer-readable recording medium
CN105407117B (en) The methods, devices and systems of distributed backup data
CN104737135B (en) The information processing terminal and synchronisation control means
CN105721883B (en) Video sharing method and system based on information of tracing to the source in a kind of cloud storage system
US20180189315A1 (en) Method and device for calling a distributed file system
CN109063192B (en) Working method of high-performance mass file storage system
CN103634406A (en) File synchronization method and terminal thereof
CN111177159B (en) Data processing system and method and data updating equipment
CN109408536A (en) A kind of cloud platform data managing method
WO2017174013A1 (en) Data storage management method and apparatus, and data storage system
CN111966632B (en) Streaming data storage method, reading method, equipment and readable storage medium
CN105162622B (en) A kind of storage method and system
CN114546273B (en) Method, system, device and storage medium for aggregation property compatible multi-site synchronization
CN111737201A (en) Method for closing opened file, computer equipment and storage medium
CN113590027B (en) Data storage method, data acquisition method, system, device and medium
CN106384292A (en) System and method for collecting relation among social network users
CN107341203A (en) The access control and optimiged index method and apparatus of a kind of distributed search engine
CN107665224B (en) Method, system and device for scanning HDFS cold data
CN108566421A (en) A kind of network type location mode and system based on Network Attached Storage
US10498683B2 (en) Compressed message sets for storage efficiency
CN108717420A (en) A kind of method that small documents reading efficiency is improved in distributed file system
CN112003944B (en) Method, system, equipment and storage medium for uploading object file
CN104902542B (en) A kind of information sharing method and mobile communication terminal
CN116820323A (en) Data storage method, device, electronic equipment and computer readable storage medium
CN107483637A (en) A kind of client link management method and device based on NFS

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant