CN110908590A - Distributed storage method and system for transformer substation data - Google Patents
Distributed storage method and system for transformer substation data Download PDFInfo
- Publication number
- CN110908590A CN110908590A CN201811079823.2A CN201811079823A CN110908590A CN 110908590 A CN110908590 A CN 110908590A CN 201811079823 A CN201811079823 A CN 201811079823A CN 110908590 A CN110908590 A CN 110908590A
- Authority
- CN
- China
- Prior art keywords
- storage node
- virtual storage
- storage
- virtual
- storage object
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/06—Electricity, gas or water supply
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The invention relates to a distributed storage method and a distributed storage system for transformer substation data, wherein the method comprises the following steps: the mapping relation between the storage object and the virtual storage node is determined by using the identification of the storage object, and then the storage object is stored on the physical storage node corresponding to the virtual storage node corresponding to the storage object according to the mapping relation between the storage object and the virtual storage node.
Description
Technical Field
The invention relates to the technical field of power station data storage, in particular to a distributed storage method and system for substation data.
Background
Along with the construction of an intelligent power grid, the scale of the power grid is larger and larger, research and application of the intelligent power grid, an intelligent substation and the like are deepened continuously, a system faces more and more acquisition points, the acquisition amount of a middle-scale area can reach 2-10 ten thousand at present, a large-scale area can face a data acquisition scale of 50-100 ten thousand in the future, and data in one year is preliminarily estimated to be converted from the current gigabyte level to the terabyte level; in addition, with the continuous improvement of the dispatching automation level, the requirement of real-time operation data is provided, and the higher requirement of continuous storage according to the actual operation time sequence instead of periodic sampling storage is adopted to meet more application requirements, which will lead the transformer substation to be used as a data acquisition source, the data storage scale is increased by tens of times, with the expansion of the data scale, the storage scale of the storage system needs to be continuously and dynamically expanded, and the storage system must be capable of supporting the continuous addition of new storage nodes, so as to ensure the uniform distribution of data in each storage node, and therefore, the requirement on the data storage technology is also improved.
The collected data are intensively deployed on a single-node storage device in the conventional power grid, so that resources (such as internal memory and disk I/O) of a single host cannot meet the requirement of massive data and subsequent capacity expansion is needed along with the expansion of the data scale, but the capacity expansion cost is very high, and in addition, when a storage node of a server is added or deleted or crashed, the data stored in a large amount are relocated, so that the efficiency is low; the existing distributed storage technology usually adopts a traditional hash algorithm or a consistent hash algorithm for fragmentation of mass data, for the traditional hash algorithm, the mapping positions of all data objects of an adding and deleting storage node need to be recalculated, so that the problem of large data relocation is caused, and for the consistent hash algorithm, if the number of the storage nodes is less, the data objects cannot be uniformly mapped onto the storage nodes, so that the problem of unbalanced data distribution is caused.
Disclosure of Invention
The invention provides a distributed storage method and a distributed storage system for transformer substation data, and aims to determine the mapping relation between a storage object and a virtual storage node based on the identification of the storage object, and further store the storage object on a physical storage node corresponding to the virtual storage node corresponding to the storage object according to the mapping relation between the storage object and the virtual storage node.
The purpose of the invention is realized by adopting the following technical scheme:
in a method of distributed storage of substation data, the improvement comprising:
determining the mapping relation between the storage object and the virtual storage node by using the identification of the storage object;
and storing the storage object on the physical storage node corresponding to the virtual storage node corresponding to the storage object according to the mapping relation between the storage object and the virtual storage node.
Preferably, the storage object includes: SCADA data and unstructured data;
the unstructured data comprises: dynamic data files, fault recording files and graphic files.
Further, if the storage object is SCADA data, the storage object is identified as follows:
occur_time:alias
wherein occur _ time is the current time, and alias is the description alias of the SCADA data object;
and if the storage object is unstructured data, identifying and selecting the file name of the unstructured data.
Preferably, the determining the mapping relationship between the storage object and the virtual storage node by using the identification of the storage object includes:
respectively acquiring a hash value of a storage object identifier and a hash value of a virtual storage node identifier;
mapping the storage object and the virtual storage node in a data range of 0-2 according to the hash value of the storage object identifier and the hash value of the virtual storage node identifier32-1 hash ring space;
and searching a virtual storage node corresponding to the storage object on the Hash ring space.
Further, the obtaining the hash value of the storage object identifier and the hash value of the virtual storage node identifier includes:
respectively acquiring an md5 digest of a storage object identifier and an md5 digest of a virtual storage node identifier;
respectively converting the first 4 bytes of the md5 digest of the storage object identifier or the md5 digest of the virtual storage node identifier into Long integers, leftwards shifting the Long integer corresponding to the 4 th byte in the first 4 bytes by 24 bits, leftwards shifting the Long integer corresponding to the 3 rd byte in the first 4 bytes by 16 bits, leftwards shifting the Long integer corresponding to the 2 nd byte in the first 4 bytes by 8 bits, and acquiring the hash value of the storage object identifier or the virtual storage node identifier.
Further, the virtual storage nodes are identified as follows:
IP#Num
the number of the virtual storage nodes corresponding to the virtual storage nodes is Num, wherein Num belongs to [0, (numReps-1) ], IP is an IP address of the physical storage node corresponding to the virtual storage node, Num is a digital suffix of the virtual storage node identifier, and numReps is the number of the virtual storage nodes corresponding to the physical storage node.
Further, the finding a virtual storage node corresponding to a storage object on the hash ring space includes:
the first virtual storage node which is found clockwise along the Hash ring space by taking the storage object i as a starting point is the virtual storage node corresponding to the storage object i; if it exceeds 232If the virtual storage node corresponding to the storage object i can not be found at the corresponding position of the Hash ring space, the first virtual storage node which is found clockwise by taking 0 at the corresponding position of the Hash ring space as a starting point is taken as the corresponding virtual storage node, wherein i belongs to [1, n ]]And n is the number of memory objects.
In a distributed storage system for substation data, the improvement comprising:
the determining module is used for determining the mapping relation between the storage object and the virtual storage node by using the identification of the storage object;
and the storage module is used for storing the storage object on the physical storage node corresponding to the virtual storage node corresponding to the storage object according to the mapping relation between the storage object and the virtual storage node.
Preferably, the storage object includes: SCADA data and unstructured data;
the unstructured data comprises: dynamic data files, fault recording files and graphic files.
Further, if the storage object is SCADA data, the storage object is identified as follows:
occur_time:alias
wherein occur _ time is the current time, and alias is the description alias of the SCADA data object;
and if the storage object is unstructured data, identifying and selecting the file name of the unstructured data.
Preferably, the determining module comprises:
the acquiring unit is used for respectively acquiring the hash value of the storage object identifier and the hash value of the virtual storage node identifier;
a mapping unit for storing the hash value of the storage object identifier and the hash value of the virtual storage node identifierThe storage object and the virtual storage node are deployed in a data range of 0-232-1 hash ring space;
and the searching unit is used for searching the virtual storage node corresponding to the storage object on the Hash ring space.
Further, the obtaining unit is configured to:
respectively acquiring an md5 digest of a storage object identifier and an md5 digest of a virtual storage node identifier;
respectively converting the first 4 bytes of the md5 digest of the storage object identifier or the md5 digest of the virtual storage node identifier into Long integers, leftwards shifting the Long integer corresponding to the 4 th byte in the first 4 bytes by 24 bits, leftwards shifting the Long integer corresponding to the 3 rd byte in the first 4 bytes by 16 bits, leftwards shifting the Long integer corresponding to the 2 nd byte in the first 4 bytes by 8 bits, and acquiring the hash value of the storage object identifier or the virtual storage node identifier.
Further, the virtual storage nodes are identified as follows:
IP#Num
the number of the virtual storage nodes corresponding to the virtual storage nodes is Num, wherein Num belongs to [0, (numReps-1) ], IP is an IP address of the physical storage node corresponding to the virtual storage node, Num is a digital suffix of the virtual storage node identifier, and numReps is the number of the virtual storage nodes corresponding to the physical storage node.
Further, the mapping unit is configured to:
the first virtual storage node which is found clockwise along the Hash ring space by taking the storage object i as a starting point is the virtual storage node corresponding to the storage object i; if it exceeds 232If the virtual storage node corresponding to the storage object i can not be found at the corresponding position of the Hash ring space, the first virtual storage node which is found clockwise by taking 0 at the corresponding position of the Hash ring space as a starting point is taken as the corresponding virtual storage node, wherein i belongs to [1, n ]]And n is the number of memory objects.
Compared with the closest prior art, the invention also has the following beneficial effects:
by adopting the technical scheme of the invention, the mapping relation between the storage object and the virtual storage node is determined by utilizing the identification of the storage object, and the unique identification is used as the basis for storing data, so that the storage and the identification are convenient, and the later-stage management is facilitated; the storage objects are stored on the physical storage nodes corresponding to the virtual storage nodes corresponding to the storage objects according to the mapping relation between the storage objects and the virtual storage nodes, the storage objects can be efficiently and uniformly distributed and stored on different physical storage nodes based on the technical scheme, the problem of large-scale data relocation under the condition that the storage nodes of the server are added or deleted or the server is down is solved, meanwhile, the pressure of a centralized storage node is reduced, and the cost of storage space expansion is saved.
Drawings
Fig. 1 is a flowchart of a distributed storage method of substation data according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating an implementation of a distributed storage method for substation data according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a hash ring space of a distributed storage method for substation data according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a distributed storage system for substation data according to an embodiment of the present invention.
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings.
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The scale of a power grid is continuously enlarged, the intelligent substation and even a new generation of intelligent substations have higher requirements on the intelligent level of application functions, and the integrated monitoring system of the substation has more and more data volume to be accessed, and particularly, the popularization of a PMU acquisition device and the development of a wide area state monitoring system WAMS bring about a more prominent problem of mass power information data storage.
Aiming at the problems existing in the centralized storage, the invention provides a consistent Hash fragmentation algorithm introducing virtual storage nodes for distributed storage of transformer substation data, each virtual storage node corresponds to one physical storage node, each physical storage node corresponds to a plurality of virtual storage nodes, the mapping relationship is that the data is stored in the virtual storage nodes and then in the physical storage nodes, and the Hash value of the identification of each virtual storage node and each storage object is deployed to 0-232The method comprises the steps of-1, searching clockwise from the deployed position of each storage object, mapping the storage object to a corresponding virtual storage node, mapping the virtual storage node to a physical storage node, and finally finding out the physical storage node to which each storage object is mapped, namely the storage object is stored in the mapped physical storage node, so that the requirement of distributed storage of large-scale mass historical data of a transformer substation is met, meanwhile, data distribution balance is achieved, the effect of load balance is achieved, and the relocation of mass data cannot be caused by adding and deleting nodes.
The invention provides a distributed storage method and a distributed storage system for substation data, which are explained below.
Fig. 1 shows a flowchart of a distributed storage method for substation data in an embodiment of the present invention, and as shown in fig. 1, the method may include:
101. determining the mapping relation between the storage object and the virtual storage node by using the identification of the storage object;
102. and storing the storage object on the physical storage node corresponding to the virtual storage node corresponding to the storage object according to the mapping relation between the storage object and the virtual storage node.
Specifically, the storage object may include: SCADA data and unstructured data;
the unstructured data may include: dynamic data files, fault recording files and graphic files;
the SCADA historical data can comprise an accident information table, a channel working condition log-in table, a hard disk space alarm table, an equipment working condition log-in table, a node working condition log-in table, a manual logic equipment locking table, a manual plate placing table, a manual remote measurement locking table, a manual remote control table, a manual remote adjustment table, a manual remote communication locking table, other application alarm tables, a process start-stop record table, a protection self-diagnosis log-in table, a protection event measurement value table, a protection fault report table, a protection operation record table, a protection operation value table, a user event log-in table, a waveform log-in table, an out-of-limit log-in table, a remote communication log-in table and a remote measurement historical sampling data table;
the graphic files can comprise all pictures and graphic element files of the monitoring host, wherein the picture files can comprise a system diagram, a wiring diagram, a curve diagram, a bar diagram, a pie diagram, a table, a geographical diagram and other picture files pic, and the graphic element files can comprise a switch, a disconnecting link, a two-turn, a three-turn, a generator, a power flow direction, a capacitor, a voltage transformer, a current transformer, a lightning arrester, a sign board, a working condition, a background icon and other graphic element files icon;
the fault wave recording files are waveform files recorded by all protection devices and fault wave recorders, and comprise hdr files, cfg files and dat files;
the dynamic data files are dynamic data files collected by all PMU devices, including dyn files.
Specifically, if the storage object is SCADA data, the storage object is identified as follows:
occur_time:alias
wherein occur _ time is the current time, alias is the description alias of the SCADA data object, and the naming mode adopts 'plant station name, voltage level name, interval name, equipment name and measuring point name'.
And if the storage object is unstructured data, identifying and selecting the file name of the unstructured data.
Fig. 2 is a schematic diagram illustrating an execution principle of a distributed storage method for substation data according to an embodiment of the present invention, and as shown in fig. 2, the determining a mapping relationship between a storage object and a virtual storage node by using an identifier of the storage object may include:
respectively acquiring a hash value of a storage object identifier and a hash value of a virtual storage node identifier;
fig. 3 shows a hash ring space diagram of a distributed storage method for substation data according to an embodiment of the present invention, as shown in fig. 3,
mapping the storage object and the virtual storage node in a data range of 0-2 according to the hash value of the storage object identifier and the hash value of the virtual storage node identifier32-1 hash ring space;
and searching a virtual storage node corresponding to the storage object on the Hash ring space.
The obtaining the hash value of the storage object identifier and the hash value of the virtual storage node identifier may include:
respectively acquiring an md5 digest of a storage object identifier and an md5 digest of a virtual storage node identifier;
respectively converting the first 4 bytes of the md5 digest of the storage object identifier or the md5 digest of the virtual storage node identifier into Long integers, leftwards shifting the Long integer corresponding to the 4 th byte in the first 4 bytes by 24 bits, leftwards shifting the Long integer corresponding to the 3 rd byte in the first 4 bytes by 16 bits, leftwards shifting the Long integer corresponding to the 2 nd byte in the first 4 bytes by 8 bits, and acquiring the hash value of the storage object identifier or the virtual storage node identifier.
Identifying each virtual storage node by adopting a mode of adding a digital suffix to an IP address of the physical storage node, and combining a hash value of the virtual storage node identification by calculating a digest value and shifting or calculating; the virtual storage nodes are identified as follows:
IP#Num
the number of the virtual storage nodes corresponding to the virtual storage nodes is Num, wherein Num belongs to [0, (numReps-1) ], IP is an IP address of the physical storage node corresponding to the virtual storage node, Num is a digital suffix of the virtual storage node identifier, and numReps is the number of the virtual storage nodes corresponding to the physical storage node.
The finding of the virtual storage node corresponding to the storage object in the hash ring space may include:
the first virtual storage node which is found clockwise along the Hash ring space by taking the storage object i as a starting point is the virtual storage node corresponding to the storage object i; if it exceeds 232If the virtual storage node corresponding to the storage object i can not be found at the corresponding position of the Hash ring space, the first virtual storage node which is found clockwise by taking 0 at the corresponding position of the Hash ring space as a starting point is taken as the corresponding virtual storage node, wherein i belongs to [1, n ]]And n is the number of memory objects.
Fig. 4 shows a schematic structural diagram of the distributed storage system for substation data of the present invention, as shown in title 4, the system may include:
the determining module is used for determining the mapping relation between the storage object and the virtual storage node by using the identification of the storage object;
and the storage module is used for storing the storage object on the physical storage node corresponding to the virtual storage node corresponding to the storage object according to the mapping relation between the storage object and the virtual storage node.
Specifically, the storage object may include: SCADA data and unstructured data;
the unstructured data may include: dynamic data files, fault recording files and graphic files.
If the storage object is SCADA data, the storage object is identified as follows:
occur_time:alias
wherein occur _ time is the current time, and alias is the description alias of the SCADA data object;
and if the storage object is unstructured data, identifying and selecting the file name of the unstructured data.
A determination module, which may include: the acquiring unit is used for respectively acquiring the hash value of the storage object identifier and the hash value of the virtual storage node identifier;
a mapping unit, configured to deploy the storage object and the virtual storage node in a data range of 0-2 according to the hash value of the storage object identifier and the hash value of the virtual storage node identifier32-1 hash ring space;
and the searching unit is used for searching the virtual storage node corresponding to the storage object on the Hash ring space.
The acquisition unit is configured to:
respectively acquiring an md5 digest of a storage object identifier and an md5 digest of a virtual storage node identifier;
respectively converting the first 4 bytes of the md5 digest of the storage object identifier or the md5 digest of the virtual storage node identifier into Long integers, leftwards shifting the Long integer corresponding to the 4 th byte in the first 4 bytes by 24 bits, leftwards shifting the Long integer corresponding to the 3 rd byte in the first 4 bytes by 16 bits, leftwards shifting the Long integer corresponding to the 2 nd byte in the first 4 bytes by 8 bits, and acquiring the hash value of the storage object identifier or the virtual storage node identifier.
Specifically, the virtual storage nodes are identified as follows:
IP#Num
the number of the virtual storage nodes corresponding to the virtual storage nodes is Num, wherein Num belongs to [0, (numReps-1) ], IP is an IP address of the physical storage node corresponding to the virtual storage node, Num is a digital suffix of the virtual storage node identifier, and numReps is the number of the virtual storage nodes corresponding to the physical storage node.
The mapping unit is configured to: the first virtual storage node which is found clockwise along the Hash ring space by taking the storage object i as a starting point is the virtual storage node corresponding to the storage object i; if it exceeds 232If the virtual storage node corresponding to the storage object i can not be found at the corresponding position of the Hash ring space, the first virtual storage node which is found clockwise by taking 0 at the corresponding position of the Hash ring space as a starting point is taken as the corresponding virtual storage node, wherein i belongs to [1, n ]]And n is the number of memory objects.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting the same, and although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
Claims (14)
1. A distributed storage method of substation data is characterized by comprising the following steps:
determining the mapping relation between the storage object and the virtual storage node by using the identification of the storage object;
and storing the storage object on the physical storage node corresponding to the virtual storage node corresponding to the storage object according to the mapping relation between the storage object and the virtual storage node.
2. The method of claim 1, wherein storing the object comprises: SCADA data and unstructured data;
the unstructured data comprises: dynamic data files, fault recording files and graphic files.
3. The method of claim 2, wherein if the storage object is SCADA data, it is identified as follows:
occur_time:alias
wherein occur _ time is the current time, and alias is the description alias of the SCADA data object;
and if the storage object is unstructured data, identifying and selecting the file name of the unstructured data.
4. The method of claim 1, wherein determining a mapping relationship of a storage object to a virtual storage node using an identification of the storage object comprises:
respectively acquiring a hash value of a storage object identifier and a hash value of a virtual storage node identifier;
mapping the storage object and the virtual storage node in a data range of 0-2 according to the hash value of the storage object identifier and the hash value of the virtual storage node identifier32-1 hash ring space;
and searching a virtual storage node corresponding to the storage object on the Hash ring space.
5. The method of claim 4, wherein obtaining the hash value of the storage object identification and the hash value of the virtual storage node identification comprises:
respectively acquiring an md5 digest of a storage object identifier and an md5 digest of a virtual storage node identifier;
respectively converting the first 4 bytes of the md5 digest of the storage object identifier or the md5 digest of the virtual storage node identifier into Long integers, leftwards shifting the Long integer corresponding to the 4 th byte in the first 4 bytes by 24 bits, leftwards shifting the Long integer corresponding to the 3 rd byte in the first 4 bytes by 16 bits, leftwards shifting the Long integer corresponding to the 2 nd byte in the first 4 bytes by 8 bits, and acquiring the hash value of the storage object identifier or the virtual storage node identifier.
6. The method of claim 5, wherein the virtual storage nodes are identified as follows:
IP#Num
the number of the virtual storage nodes corresponding to the virtual storage nodes is Num, wherein Num belongs to [0, (numReps-1) ], IP is an IP address of the physical storage node corresponding to the virtual storage node, Num is a digital suffix of the virtual storage node identifier, and numReps is the number of the virtual storage nodes corresponding to the physical storage node.
7. The method of claim 4, wherein finding the virtual storage node corresponding to the storage object on the hash ring space comprises:
clockwise along the hash ring space starting from storage object iThe first virtual storage node found out is the virtual storage node corresponding to the storage object i; if it exceeds 232If the virtual storage node corresponding to the storage object i can not be found at the corresponding position of the Hash ring space, the first virtual storage node which is found clockwise by taking 0 at the corresponding position of the Hash ring space as a starting point is taken as the corresponding virtual storage node, wherein i belongs to [1, n ]]And n is the number of memory objects.
8. A distributed storage system for substation data, the system comprising:
the determining module is used for determining the mapping relation between the storage object and the virtual storage node by using the identification of the storage object;
and the storage module is used for storing the storage object on the physical storage node corresponding to the virtual storage node corresponding to the storage object according to the mapping relation between the storage object and the virtual storage node.
9. The system of claim 8, wherein the storage object comprises: SCADA data and unstructured data;
the unstructured data comprises: dynamic data files, fault recording files and graphic files.
10. The system of claim 9, wherein if the storage object is SCADA data, it is identified as follows:
occur_time:alias
wherein occur _ time is the current time, and alias is the description alias of the SCADA data object;
and if the storage object is unstructured data, identifying and selecting the file name of the unstructured data.
11. The system of claim 8, wherein the determination module comprises:
the acquiring unit is used for respectively acquiring the hash value of the storage object identifier and the hash value of the virtual storage node identifier;
a mapping unit, configured to deploy the storage object and the virtual storage node in a data range of 0-2 according to the hash value of the storage object identifier and the hash value of the virtual storage node identifier32-1 hash ring space;
and the searching unit is used for searching the virtual storage node corresponding to the storage object on the Hash ring space.
12. The system of claim 11, wherein the acquisition unit is to:
respectively acquiring an md5 digest of a storage object identifier and an md5 digest of a virtual storage node identifier;
respectively converting the first 4 bytes of the md5 digest of the storage object identifier or the md5 digest of the virtual storage node identifier into Long integers, leftwards shifting the Long integer corresponding to the 4 th byte in the first 4 bytes by 24 bits, leftwards shifting the Long integer corresponding to the 3 rd byte in the first 4 bytes by 16 bits, leftwards shifting the Long integer corresponding to the 2 nd byte in the first 4 bytes by 8 bits, and acquiring the hash value of the storage object identifier or the virtual storage node identifier.
13. The system of claim 11, wherein the virtual storage nodes are identified as follows:
IP#Num
the number of the virtual storage nodes corresponding to the virtual storage nodes is Num, wherein Num belongs to [0, (numReps-1) ], IP is an IP address of the physical storage node corresponding to the virtual storage node, Num is a digital suffix of the virtual storage node identifier, and numReps is the number of the virtual storage nodes corresponding to the physical storage node.
14. The system of claim 11, wherein the mapping unit is to:
the first virtual storage node which is found clockwise along the Hash ring space by taking the storage object i as a starting point is the virtual storage node corresponding to the storage object i; if it exceeds 232If the virtual storage node corresponding to the storage object i cannot be found at the corresponding position of the hash ring space, the virtual storage node is 0The first virtual storage node which is found clockwise by taking the corresponding position of the Hash ring space as a starting point is taken as the corresponding virtual storage node, wherein i belongs to [1, n ]]And n is the number of memory objects.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811079823.2A CN110908590A (en) | 2018-09-17 | 2018-09-17 | Distributed storage method and system for transformer substation data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811079823.2A CN110908590A (en) | 2018-09-17 | 2018-09-17 | Distributed storage method and system for transformer substation data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110908590A true CN110908590A (en) | 2020-03-24 |
Family
ID=69813254
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811079823.2A Pending CN110908590A (en) | 2018-09-17 | 2018-09-17 | Distributed storage method and system for transformer substation data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110908590A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111781890A (en) * | 2020-07-14 | 2020-10-16 | 厦门海为科技有限公司 | Multi-equipment engineering communication method and system in configuration engineering |
| CN112100185A (en) * | 2020-11-03 | 2020-12-18 | 江西云本数字科技有限公司 | Indexing system and method for block chain data balance load |
| CN112162707A (en) * | 2020-10-15 | 2021-01-01 | 科大讯飞股份有限公司 | Storage method, electronic device and storage medium for distributed storage system |
| CN112198354A (en) * | 2020-09-27 | 2021-01-08 | 广东电网有限责任公司梅州供电局 | Clamp meter |
| CN112261097A (en) * | 2020-10-15 | 2021-01-22 | 科大讯飞股份有限公司 | Object positioning method for distributed storage system and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101986276A (en) * | 2010-10-21 | 2011-03-16 | 成都市华为赛门铁克科技有限公司 | Methods and systems for storing and recovering files and server |
| CN103188345A (en) * | 2013-03-01 | 2013-07-03 | 北京邮电大学 | Distributive dynamic load management system and distributive dynamic load management method |
| CN104050270A (en) * | 2014-06-23 | 2014-09-17 | 成都康赛信息技术有限公司 | Distributed storage method based on consistent Hash algorithm |
| CN108124012A (en) * | 2017-12-21 | 2018-06-05 | 中通服公众信息产业股份有限公司 | A kind of distributed caching computational methods based on hash algorithm |
-
2018
- 2018-09-17 CN CN201811079823.2A patent/CN110908590A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101986276A (en) * | 2010-10-21 | 2011-03-16 | 成都市华为赛门铁克科技有限公司 | Methods and systems for storing and recovering files and server |
| CN103188345A (en) * | 2013-03-01 | 2013-07-03 | 北京邮电大学 | Distributive dynamic load management system and distributive dynamic load management method |
| CN104050270A (en) * | 2014-06-23 | 2014-09-17 | 成都康赛信息技术有限公司 | Distributed storage method based on consistent Hash algorithm |
| CN108124012A (en) * | 2017-12-21 | 2018-06-05 | 中通服公众信息产业股份有限公司 | A kind of distributed caching computational methods based on hash algorithm |
Non-Patent Citations (3)
| Title |
|---|
| 宋亚奇等: "云平台下输变电设备状态监测大数据存储优化与并行处理", 《中国电机工程学报》 * |
| 葛磊蛟等: "智能配用电大数据存储架构设计", 《电力自动化设备》 * |
| 龚霄等: "一种输变电设备多源数据处理系统的功能设计", 《机电工程》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111781890A (en) * | 2020-07-14 | 2020-10-16 | 厦门海为科技有限公司 | Multi-equipment engineering communication method and system in configuration engineering |
| CN112198354A (en) * | 2020-09-27 | 2021-01-08 | 广东电网有限责任公司梅州供电局 | Clamp meter |
| CN112162707A (en) * | 2020-10-15 | 2021-01-01 | 科大讯飞股份有限公司 | Storage method, electronic device and storage medium for distributed storage system |
| CN112261097A (en) * | 2020-10-15 | 2021-01-22 | 科大讯飞股份有限公司 | Object positioning method for distributed storage system and electronic equipment |
| CN112261097B (en) * | 2020-10-15 | 2023-11-24 | 科大讯飞股份有限公司 | Object positioning method for distributed storage system and electronic equipment |
| CN112100185A (en) * | 2020-11-03 | 2020-12-18 | 江西云本数字科技有限公司 | Indexing system and method for block chain data balance load |
| CN112100185B (en) * | 2020-11-03 | 2021-04-30 | 深圳市穗彩科技开发有限公司 | Indexing system and method for block chain data balance load |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110908590A (en) | Distributed storage method and system for transformer substation data | |
| CN106682077B (en) | Mass time sequence data storage implementation method based on Hadoop technology | |
| CN104636084B (en) | A kind of apparatus and method of the large data of electric power being carried out to distributed store | |
| CN104317800A (en) | Hybrid storage system and method for mass intelligent power utilization data | |
| CN105426482B (en) | A kind of railway 10 kV power distribution net magnanimity monitoring information HBase dump methods of picture library one | |
| CN104539750A (en) | IP locating method and device | |
| CN104035956A (en) | Time-series data storage method based on distributive column storage | |
| CN106649652A (en) | Construction method for on-line transformer substation main equipment monitoring system based on Hadoop | |
| CN111404270A (en) | Low-voltage distribution area topology level identification method, device and equipment based on synchronous counting method | |
| CN111523004B (en) | Storage method and system for edge computing gateway data | |
| CN103530335B (en) | In-stockroom operation method and device of electric power measurement acquisition system | |
| CN106557550B (en) | Method and device for realizing rapid storage, retrieval and completion of self-description fixed-point records of power distribution terminal | |
| CN101917282A (en) | Method, device and system for processing alarm shielding rules | |
| CN111124313A (en) | Data reading and writing method and device for power acquisition terminal and electronic equipment | |
| CN114996535A (en) | Data retrieval method and system for power equipment in power grid and electronic equipment | |
| CN107463540B (en) | Electric energy quality data processing method and electric energy quality monitoring device | |
| CN105138278A (en) | Method for storing network messages of intelligent substation in bare disk | |
| CN112286884B (en) | Method, system and medium for automatically generating topology of ring main unit of power distribution network | |
| CN102447589B (en) | Method and device for aggregating records | |
| CN111553040B (en) | Power grid topology analysis high-performance computing method and device based on GPU acceleration | |
| CN105868441B (en) | Multi-source data-based WAMS application system data integration test platform | |
| CN107967301B (en) | Method and device for storing and inquiring monitoring data of power cable tunnel | |
| CN112014669A (en) | Online detection method and system for angle difference and ratio difference of merging unit | |
| CN109992625B (en) | Remote integrated power grid data storage system and method | |
| CN117056133B (en) | Data backup method, device and medium based on distributed Internet of things architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |