CN110968259A - Step-by-step object storage system, object storage method and storage medium - Google Patents
Step-by-step object storage system, object storage method and storage medium Download PDFInfo
- Publication number
- CN110968259A CN110968259A CN201811161869.9A CN201811161869A CN110968259A CN 110968259 A CN110968259 A CN 110968259A CN 201811161869 A CN201811161869 A CN 201811161869A CN 110968259 A CN110968259 A CN 110968259A
- Authority
- CN
- China
- Prior art keywords
- data service
- service node
- nodes
- interface
- objects
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 230000015654 memory Effects 0.000 claims abstract description 37
- 238000004590 computer program Methods 0.000 claims description 5
- 238000013500 data storage Methods 0.000 abstract description 7
- 230000004083 survival effect Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000001902 propagating effect Effects 0.000 description 1
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The invention discloses a step-by-step object storage system, a storage method and a storage medium, which relate to the field of data storage, wherein the system comprises an interface service layer and a data service layer, wherein the interface service layer comprises at least one interface service node, and the interface service node is used for receiving an object to be stored; the data service nodes comprise at least one data service node, each data service node comprises a memory used for storing objects, and the data service nodes are used for storing the objects received by the interface service nodes. The data service node periodically sends heartbeat information to an interface service layer in a UDP (user Datagram protocol) broadcasting mode; a queue used for caching heartbeat information sent by a data service node is established in the interface service layer; the invention adopts a framework with separated interfaces and data storage, does not need to use an external message queue, and is easy to develop and deploy.
Description
Technical Field
The invention relates to the field of data storage, in particular to a step-by-step object storage system, an object storage method and a storage medium.
Background
The traditional Network Storage mainly comprises two types of NAS (Network Attached Storage) and SAN (Storage Area Network), wherein the NAS provides a Storage and file system Network for a client to use, and is managed by complete files, and uploaded and downloaded by a calling party; a SAN is a service that provides block storage, and users can freely format and create file systems. Both of the above two network storage modes provide a target address and are managed in a file mode, resulting in poor system expansibility.
The current object storage systems such as Kafka, Redis and the like, wherein Kafka is an open source stream processing platform and is written by Scala and Java, and Redis is an open source log-type and Key-Value database which is written by using ANSI C language, supports network, can be based on memory and can also be persisted, and provides APIs (application programming interfaces) of multiple languages. The two storage systems perform summary recording on information to be stored through object metadata information, and users can store files which are not simple any more, but can also be arrays, classes and the like.
Disclosure of Invention
In view of the defects in the prior art, the present invention provides a step-by-step object storage system, an object storage method and a storage medium, which adopt a framework with separate interfaces and data storage, do not need to use an external message queue, and are easy to develop and deploy.
A first aspect of the present invention provides a hierarchical object storage system, including:
the interface service layer comprises at least one interface service node, and the interface service node is used for receiving an object to be stored;
the data service nodes comprise at least one data service node, each data service node comprises a memory for storing objects, and the data service nodes are used for storing the objects received by the interface service nodes.
With reference to the first aspect, in a first possible implementation manner,
the data service node periodically sends heartbeat information to an interface service layer in a UDP (user Datagram protocol) broadcasting mode;
a queue used for caching heartbeat information sent by a data service node is established in the interface service layer;
the queue created in the interface service layer is further used for recording the surviving data service nodes based on the cached heartbeat information, and the objects received by the interface service nodes are stored in the surviving data service nodes.
With reference to the first possible implementation manner of the first aspect, in a second possible implementation manner,
the queue is also used for recording the time when the surviving data service node sends heartbeat information to the interface service layer last time, and removing the failed data service node in the queue according to the refreshing rule;
the refresh rule is as follows: and for the surviving data service nodes recorded in the queue, based on the time of the data service node sending the heartbeat information last time, if the set timing time is reached and the interface service layer does not receive the heartbeat information sent by the data service node any more, marking the data service node as invalid and removing the data service node from the queue.
A second aspect of the present invention provides an object storage method, including the steps of:
randomly selecting n data service nodes, wherein n is a positive integer not less than 2;
sending the object to be stored to all the selected data service nodes;
the data service node serializes the received objects;
the serialized objects are stored in the memory of the current data service node, and the names of the objects are recorded in the index file of the current data service node.
With reference to the second aspect, in a first possible implementation manner, the method for storing the serialized objects in the memory of the current data service node includes:
s401: judging the size of the residual capacity of the current data service node memory, if the residual capacity can store the serialized objects, turning to S402, otherwise, turning to S403;
s402: storing the serialized objects in a memory of the current data service node, and ending;
s403: and sequentially selecting another unselected data service node of the data service layer, then judging the residual capacity of the memory of the selected data service node, if the residual capacity can store the serialized objects, turning to S402, otherwise, repeating the step S403.
With reference to the first possible implementation manner of the second aspect, in a second possible implementation manner, the method is characterized in that: and when the residual capacity of the memories of all the data service nodes in the data service layer can not store the serialized objects, adding the data service nodes in the data service layer.
With reference to the second aspect, in a third possible implementation manner, the method is characterized in that: when the object stored in the data service node is acquired, the method specifically comprises the following steps:
a: the interface service node broadcasts the name of the object to be acquired to all the data service nodes;
b: the data service nodes traverse the respective index files, and if the names of the objects to be acquired exist in the index files of the data service nodes, the current data service nodes send the IP information of the data service nodes to the interface service nodes;
c: and the interface service node is connected with the data service node corresponding to the received first IP information to acquire the object to be acquired.
With reference to the third possible implementation manner of the second aspect, in a fourth possible implementation manner, the method is characterized in that: the interface service node is connected with a data service node corresponding to the received first IP information to acquire an object to be acquired, and the specific steps are as follows:
c1: the interface service node waits for receiving the IP information sent by the data service node, if the IP information is received within the set time, the step is switched to C2, otherwise, the step is switched to A;
c2: and the interface service node is connected with the data service node corresponding to the received first IP information to acquire the object to be acquired.
With reference to the second aspect, in a fifth possible implementation manner, the method is characterized in that: the method for randomly selecting n data service nodes comprises the following specific steps:
s101: traversing the index files of all the data service nodes, and turning to S102;
s102: judging whether the name of the object to be stored exists in the index file of the data service node, if so, ending, otherwise, turning to S103;
s103: and randomly selecting n data service nodes.
A third aspect of the invention provides a storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of:
randomly selecting n data service nodes, wherein n is a positive integer not less than 2;
sending the object to be stored to all the selected data service nodes;
the data service node serializes the received objects;
the serialized objects are stored in the memory of the current data service node, and the names of the objects are recorded in the index file of the current data service node.
With reference to the third aspect, in a first possible implementation manner, the step of storing the serialized object in a memory of a current data service node includes:
s401: judging the size of the residual capacity of the current data service node memory, if the residual capacity can store the serialized objects, turning to S402, otherwise, turning to S403;
s402: storing the serialized objects in a memory of the current data service node, and ending;
s403: and sequentially selecting another unselected data service node of the data service layer, then judging the residual capacity of the memory of the selected data service node, if the residual capacity can store the serialized objects, turning to S402, otherwise, repeating the step S403.
Compared with the prior art, the invention has the advantages that: the interface service layer and the data service layer are separately arranged, a plurality of data service nodes are randomly selected, objects to be stored are sent to all selected data service nodes, then the objects are stored in a memory of the current data service nodes, an architecture with separated interfaces and data storage is adopted, an external message queue is not needed, development and deployment are easy, system expansion and customization can be carried out according to actual requirements, for example, the number of the data service nodes and the number of the interface service nodes are dynamically increased, the purposes of high performance and economy of space can be achieved, and the use requirements of micro services are effectively met.
Drawings
FIG. 1 is a block diagram of a step-by-step object storage system according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating an object storage method according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a step-by-step object storage system, which adopts an architecture with an interface and a data storage separated structure and is a micro-service-oriented small distributed object storage system. The embodiment of the invention also provides a corresponding object storage method and a storage medium.
The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, an embodiment of a hierarchical object storage system according to an embodiment of the present invention includes:
the interface service layer comprises at least one interface service node, and the interface service node is used for receiving an object to be stored;
and the data service nodes comprise at least one data service node, each data service node comprises a memory for storing objects, and the data service nodes are used for storing the objects received by the interface service nodes.
In the embodiment of the invention, the interface service layer and the data service layer are independent from each other, and are associated with each other when the object storage is carried out, which is different from the object storage model of a single machine.
In the embodiment of the invention, the memory is hardware for storing data, such as a solid state disk, a mechanical hard disk and the like. The interface service layer is equivalent to provide an external REST (Representational State Transfer) interface.
Optionally, on the basis of the embodiment corresponding to fig. 1, in a first optional embodiment of the step-by-step object storage system provided in the embodiment of the present invention, to ensure that the interface service node can know the survival state of each data service node, so as to store the object, the data service node periodically sends heartbeat information to the interface service layer in a UDP (User data program Protocol) broadcast manner, a queue for caching the heartbeat information sent by the data service node is created in the interface service layer, the queue created in the interface service layer is further used for recording the surviving data service node based on the cached heartbeat information, and the object received by the interface service node is stored in the surviving data service node.
In the embodiment of the invention, when the interface service layer receives the heartbeat information sent by the data service node, the data service node is in a survival state, and the object can be stored.
Optionally, on the basis of the first optional embodiment corresponding to fig. 1, in a second optional embodiment of the step-by-step object storage system provided in the embodiment of the present invention, the queue is further configured to record a time when the surviving data service node last sends the heartbeat information to the interface service layer, and remove the failed data service node in the queue according to the refresh rule; the refresh rule is: and for the surviving data service nodes recorded in the queue, based on the time of the data service node sending the heartbeat information last time, if the set timing time is reached and the interface service layer does not receive the heartbeat information sent by the data service node any more, marking the data service node as invalid and removing the data service node from the queue.
In the embodiment of the present invention, the set timing time may be 500ms, when the interface service layer receives the heartbeat information sent by a certain data service node, the interface service layer starts to time for the data service node, and if the interface service layer does not receive the heartbeat information sent by the data service node again within 500ms, the interface service layer marks the data service node as invalid and removes the data service node from the queue. The heartbeat information is mainly used for judging the survival state of the data service nodes, and only the survival data service nodes can be used for receiving and storing data.
Referring to fig. 2, on the basis of all embodiments corresponding to fig. 1, an embodiment of an object storage method provided in an embodiment of the present invention includes:
s1: randomly selecting n data service nodes, wherein n is a positive integer not less than 2;
in the embodiment of the invention, a plurality of data service nodes are selected, the objects to be stored are stored in the selected data service nodes when the objects are stored subsequently, a multi-node storage mode is adopted, which is equivalent to backup of the objects, the data in a certain data service node is lost subsequently, the same data is stored in other data service nodes, and the problem of data loss is effectively avoided. Meanwhile, the selected data service node is the data service node in the survival state.
S2: sending the object to be stored to all the selected data service nodes;
s3: the data service node serializes the received objects;
s4: the serialized objects are stored in a memory of the current data service node, meanwhile, the names of the objects are recorded in an index file of the current data service node, and the index file stores the names of the objects, so that the retrieval in the subsequent object acquisition process is facilitated. In Microsoft Windows operating system, the index file is a file created by Internet Explorer and resource manager.
According to the object storage method provided by the embodiment of the invention, the interface service layer and the data service layer are separately arranged, a plurality of data service nodes are randomly selected, the object to be stored is sent to all the selected data service nodes, then the object is stored in the memory of the current data service node, an architecture with separated interfaces and data storage is adopted, external message queues such as redis and RabbitMQ are not used, development and deployment are easy, system expansion and customization can be carried out according to actual requirements, for example, the number of the data service nodes and the number of the interface service nodes are dynamically increased, the purposes of high performance and economical space can be achieved, and the use requirement of micro-service is effectively met.
In the embodiment of the present invention, in order to ensure that the interface service node can know the survival state of each data service node, so as to facilitate storage of the object, the data service node periodically sends heartbeat information to the interface service layer in a UDP (User Datagram Protocol) broadcast manner, a queue for caching the heartbeat information sent by the data service node is created in the interface service layer, the queue created in the interface service layer is further used for recording the surviving data service nodes based on the cached heartbeat information, and the object received by the interface service node is stored in the surviving data service nodes.
The queue is also used for recording the time when the surviving data service node sends heartbeat information to the interface service layer last time, and removing the failed data service node in the queue according to the refreshing rule; the refresh rule is: and for the surviving data service nodes recorded in the queue, based on the time of the data service node sending the heartbeat information last time, if the set timing time is reached and the interface service layer does not receive the heartbeat information sent by the data service node any more, marking the data service node as invalid and removing the data service node from the queue.
Optionally, on the basis of the embodiment corresponding to fig. 2, in a first optional embodiment of the object storage method according to the embodiment of the present invention, the serialized object is stored in a memory of the current data service node, and the specific steps are as follows:
s401: judging the size of the residual capacity of the current data service node memory, if the residual capacity can store the serialized objects, turning to S402, otherwise, turning to S403;
s402: storing the serialized objects in a memory of the current data service node, and ending;
s403: and sequentially selecting another unselected data service node of the data service layer, then judging the size of the remaining capacity of the memory of the selected data service node, if the remaining capacity can store the serialized objects, turning to S402, otherwise, repeating the step S403, namely searching the next data service node capable of storing the objects, wherein the selected data service node is the data service node in a survival state.
In the embodiment of the invention, when the residual capacity of the memories of all the data service nodes in the data service layer can not store the serialized objects, which indicates that all the current data service nodes do not have enough storage space, an alarm can be given to prompt space abnormity, and then the data service nodes in the data service layer are added.
Optionally, on the basis of the embodiment corresponding to fig. 2, in a second optional embodiment of the object storage method provided in the embodiment of the present invention, when acquiring an object stored in a data service node, the method specifically includes the following steps:
a: the interface service node broadcasts the name of the object to be acquired to all the data service nodes, wherein the name is equivalent to the unique identifier of the object;
in the embodiment of the invention, when the object is acquired, the acquiescence is only carried out through one interface service node.
B: the data service nodes traverse the respective index files, and if the names of the objects to be acquired exist in the index files of the data service nodes, the current data service nodes send the IP information of the data service nodes to the interface service nodes;
c: and the interface service node is connected with the data service node corresponding to the received first IP information to acquire the object to be acquired. Because the object is backed up and stored in the plurality of data service nodes, when traversing the index file and judging whether the name of the object to be acquired exists, the name of the object exists in the index files of the plurality of data service nodes, so that the data service nodes receive the IP information sent by the plurality of data service nodes, and at the moment, the data service node corresponding to the received first IP information is selected to acquire the object to be acquired.
Optionally, on the basis of the second optional embodiment corresponding to fig. 2, in a third optional embodiment of the object storage method provided in the embodiment of the present invention, the interface service node connects the data service node corresponding to the received first IP information, and acquires the object to be acquired, where the specific steps are as follows:
c1: the interface service node waits for receiving the IP information sent by the data service node, if the IP information is received within the set time, the step is switched to C2, otherwise, the step is switched to A;
c2: and the interface service node is connected with the data service node corresponding to the received first IP information to acquire the object to be acquired.
In the embodiment of the invention, the set time is 2 seconds, if the IP information sent by the data service node is not received within the set time, the network jitter exists or the object to be acquired does not exist in the data service layer, if the IP information is the network jitter, the step A is carried out, and retry is carried out; and if the object to be acquired does not exist, ending the operation.
Optionally, on the basis of the embodiment corresponding to fig. 2, in a fourth optional embodiment of the object storage method provided in the embodiment of the present invention, n data service nodes are randomly selected, and the specific steps are as follows:
s101: traversing the index files of all the data service nodes, and turning to S102;
s102: judging whether the name of the object to be stored exists in the index file of the data service node, if so, ending, otherwise, turning to S103;
s103: and randomly selecting n data service nodes.
In the embodiment of the invention, because the data service nodes are randomly selected in the process of storing the object, when the number of times of storing the same object is more, each data service node is likely to store the object, thereby causing great waste of space, judging whether the object is stored before storing the object, and avoiding storing the object again.
An embodiment of a storage medium provided in an embodiment of the present invention includes: the storage medium has stored thereon a computer program which, when executed by a processor, performs the steps of:
randomly selecting n data service nodes, wherein n is a positive integer not less than 2;
sending the object to be stored to all the selected data service nodes;
the data service node serializes the received objects;
the serialized objects are stored in the memory of the current data service node, and the names of the objects are recorded in the index file of the current data service node.
Optionally, on the basis of the above storage medium embodiment, in a first optional embodiment of the storage medium provided by the embodiment of the present invention, the storage medium may adopt any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer-readable storage medium may be, for example but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
Alternatively, on the basis of the above embodiment of the storage medium and the first alternative embodiment, in a second alternative embodiment of the storage medium provided by the embodiment of the present invention, the computer-readable signal medium may include a data signal propagating in a baseband or as a part of a carrier wave, in which a computer-readable program code is carried. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Optionally, on the basis of the above embodiment of the storage medium and the first and second alternative embodiments, in a third alternative embodiment of the storage medium provided by the embodiment of the present invention, the computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, the programming languages including an object oriented programming language, such as Java, Smalltalk, C + +, and further including a conventional procedural programming language, such as "C" programming language or a similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The present invention is not limited to the above-described embodiments, and it will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and such modifications and improvements are also considered to be within the scope of the present invention. Those not described in detail in this specification are within the skill of the art.
Claims (10)
1. A step-wise object storage system, comprising:
the interface service layer comprises at least one interface service node, and the interface service node is used for receiving an object to be stored;
the data service nodes comprise at least one data service node, each data service node comprises a memory for storing objects, and the data service nodes are used for storing the objects received by the interface service nodes.
2. The hierarchical object storage system of claim 1, wherein:
the data service node periodically sends heartbeat information to an interface service layer in a UDP (user Datagram protocol) broadcasting mode;
a queue used for caching heartbeat information sent by a data service node is established in the interface service layer;
the queue created in the interface service layer is further used for recording the surviving data service nodes based on the cached heartbeat information, and the objects received by the interface service nodes are stored in the surviving data service nodes.
3. A hierarchical object storage system in accordance with claim 2, wherein:
the queue is also used for recording the time when the surviving data service node sends heartbeat information to the interface service layer last time, and removing the failed data service node in the queue according to the refreshing rule;
the refresh rule is as follows: and for the surviving data service nodes recorded in the queue, based on the time of the data service node sending the heartbeat information last time, if the set timing time is reached and the interface service layer does not receive the heartbeat information sent by the data service node any more, marking the data service node as invalid and removing the data service node from the queue.
4. An object storage method for storing objects by using the hierarchical object storage system of claim 1, comprising the steps of:
randomly selecting n data service nodes, wherein n is a positive integer not less than 2;
sending the object to be stored to all the selected data service nodes;
the data service node serializes the received objects;
the serialized objects are stored in the memory of the current data service node, and the names of the objects are recorded in the index file of the current data service node.
5. An object storage method according to claim 4, wherein: the serialized objects are stored in a memory of the current data service node, and the concrete steps are as follows:
s401: judging the size of the residual capacity of the current data service node memory, if the residual capacity can store the serialized objects, turning to S402, otherwise, turning to S403;
s402: storing the serialized objects in a memory of the current data service node, and ending;
s403: and sequentially selecting another unselected data service node of the data service layer, then judging the residual capacity of the memory of the selected data service node, if the residual capacity can store the serialized objects, turning to S402, otherwise, repeating the step S403.
6. An object storage method according to claim 5, wherein: and when the residual capacity of the memories of all the data service nodes in the data service layer can not store the serialized objects, adding the data service nodes in the data service layer.
7. An object storage method according to claim 4, wherein: when the object stored in the data service node is acquired, the method specifically comprises the following steps:
a: the interface service node broadcasts the name of the object to be acquired to all the data service nodes;
b: the data service nodes traverse the respective index files, and if the names of the objects to be acquired exist in the index files of the data service nodes, the current data service nodes send the IP information of the data service nodes to the interface service nodes;
c: and the interface service node is connected with the data service node corresponding to the received first IP information to acquire the object to be acquired.
8. An object storage method according to claim 7, wherein: the interface service node is connected with a data service node corresponding to the received first IP information to acquire an object to be acquired, and the specific steps are as follows:
c1: the interface service node waits for receiving the IP information sent by the data service node, if the IP information is received within the set time, the step is switched to C2, otherwise, the step is switched to A;
c2: and the interface service node is connected with the data service node corresponding to the received first IP information to acquire the object to be acquired.
9. An object storage method according to claim 4, wherein: the method for randomly selecting n data service nodes comprises the following specific steps:
s101: traversing the index files of all the data service nodes, and turning to S102;
s102: judging whether the name of the object to be stored exists in the index file of the data service node, if so, ending, otherwise, turning to S103;
s103: and randomly selecting n data service nodes.
10. A storage medium having a computer program stored thereon, characterized in that: the computer program when executed by a processor implementing the steps of:
randomly selecting n data service nodes, wherein n is a positive integer not less than 2;
sending the object to be stored to all the selected data service nodes;
the data service node serializes the received objects;
the serialized objects are stored in the memory of the current data service node, and the names of the objects are recorded in the index file of the current data service node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811161869.9A CN110968259A (en) | 2018-09-30 | 2018-09-30 | Step-by-step object storage system, object storage method and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811161869.9A CN110968259A (en) | 2018-09-30 | 2018-09-30 | Step-by-step object storage system, object storage method and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110968259A true CN110968259A (en) | 2020-04-07 |
Family
ID=70029278
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811161869.9A Pending CN110968259A (en) | 2018-09-30 | 2018-09-30 | Step-by-step object storage system, object storage method and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110968259A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113485978A (en) * | 2021-06-23 | 2021-10-08 | 华泰证券股份有限公司 | Method and system for improving read-write throughput of file storage NAS and storage |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103763155A (en) * | 2014-01-24 | 2014-04-30 | 国家电网公司 | Multi-service heartbeat monitoring method for distributed type cloud storage system |
| CN104391930A (en) * | 2014-11-21 | 2015-03-04 | 用友软件股份有限公司 | Distributed file storage device and method |
| CN106027647A (en) * | 2016-05-20 | 2016-10-12 | 云南云电同方科技有限公司 | LXPFS (Linux XProgram File System) cluster distributed file storage system |
| CN106663030A (en) * | 2014-08-13 | 2017-05-10 | 微软技术许可有限责任公司 | Scalable fault resilient communications within distributed clusters |
| CN107844542A (en) * | 2017-10-26 | 2018-03-27 | 山东浪潮通软信息科技有限公司 | A kind of distributed document storage method and device |
| CN107948248A (en) * | 2017-11-01 | 2018-04-20 | 平安科技（深圳）有限公司 | Distributed storage method, control server and computer-readable recording medium |
-
2018
- 2018-09-30 CN CN201811161869.9A patent/CN110968259A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103763155A (en) * | 2014-01-24 | 2014-04-30 | 国家电网公司 | Multi-service heartbeat monitoring method for distributed type cloud storage system |
| CN106663030A (en) * | 2014-08-13 | 2017-05-10 | 微软技术许可有限责任公司 | Scalable fault resilient communications within distributed clusters |
| CN104391930A (en) * | 2014-11-21 | 2015-03-04 | 用友软件股份有限公司 | Distributed file storage device and method |
| CN106027647A (en) * | 2016-05-20 | 2016-10-12 | 云南云电同方科技有限公司 | LXPFS (Linux XProgram File System) cluster distributed file storage system |
| CN107844542A (en) * | 2017-10-26 | 2018-03-27 | 山东浪潮通软信息科技有限公司 | A kind of distributed document storage method and device |
| CN107948248A (en) * | 2017-11-01 | 2018-04-20 | 平安科技（深圳）有限公司 | Distributed storage method, control server and computer-readable recording medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113485978A (en) * | 2021-06-23 | 2021-10-08 | 华泰证券股份有限公司 | Method and system for improving read-write throughput of file storage NAS and storage |
| CN113485978B (en) * | 2021-06-23 | 2023-07-21 | 华泰证券股份有限公司 | Method, system and memory for improving read-write throughput capacity of file storage NAS |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107861686B (en) | File storage method, server and computer readable storage medium | |
| CN103765406B (en) | For the method and apparatus remotely updating executive process | |
| US10146848B2 (en) | Systems and methods for autonomous, scalable, and distributed database management | |
| CN104301373B (en) | Via the synchronous sending out notice of file-sharing service | |
| US20120150930A1 (en) | Cloud storage and method for managing the same | |
| CN111078504A (en) | Distributed call chain tracking method and device, computer equipment and storage medium | |
| JP2017531250A (en) | Granular / semi-synchronous architecture | |
| US20180101558A1 (en) | Log-shipping data replication with early log record fetching | |
| WO2016029744A1 (en) | Metadata recovery method and relevant device | |
| JP2016513306A (en) | Data storage method, data storage device, and storage device | |
| WO2016173441A1 (en) | Server cache processing method, apparatus and system | |
| EP3449352B1 (en) | Self-organizing storage system for asynchronous storage service | |
| US20150056961A1 (en) | Providing dynamically-translated public address system announcements to mobile devices | |
| US9858152B2 (en) | Collaborative information source recovery | |
| CN111338893A (en) | Process log processing method and device, computer equipment and storage medium | |
| CN111708738A (en) | Method and system for realizing data inter-access between hdfs of hadoop file system and s3 of object storage | |
| CN105786539B (en) | File downloading method and device | |
| WO2017032152A1 (en) | Method for writing data into storage device and storage device | |
| CN113411363A (en) | Uploading method of image file, related equipment and computer storage medium | |
| CN107516048B (en) | Method and equipment for controlling file access in distributed file system | |
| JP2010113617A (en) | Cluster system-control program, cluster system, and cluster system-control method | |
| US10545667B1 (en) | Dynamic data partitioning for stateless request routing | |
| US10169441B2 (en) | Synchronous data replication in a content management system | |
| EP3163469B1 (en) | Method and device for realizing ip disk file storage | |
| CN110968259A (en) | Step-by-step object storage system, object storage method and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200407 |
|
| RJ01 | Rejection of invention patent application after publication |