CN107609143B - Fragment information storage method of distributed real-time memory database - Google Patents
Fragment information storage method of distributed real-time memory database Download PDFInfo
- Publication number
- CN107609143B CN107609143B CN201710858824.6A CN201710858824A CN107609143B CN 107609143 B CN107609143 B CN 107609143B CN 201710858824 A CN201710858824 A CN 201710858824A CN 107609143 B CN107609143 B CN 107609143B
- Authority
- CN
- China
- Prior art keywords
- real
- time data
- keyword
- fragment
- record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000012634 fragment Substances 0.000 title claims abstract description 37
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000013467 fragmentation Methods 0.000 claims abstract description 13
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 13
- 238000013507 mapping Methods 0.000 claims abstract description 5
- 238000010586 diagram Methods 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a fragment information storage method of a distributed real-time memory database, which comprises the following steps of (1) acquiring real-time data of different applications to form a real-time database; (2) using long and integer data as a keyword of a real-time data table in a real-time database, wherein the keyword can be divided into a table identifier and a record serial number; each record in the real-time data table has an application number corresponding to the application to which the record belongs; (3) independently fragmenting real-time data of different applications in a real-time data table, storing the real-time data into a corresponding storage file, and generating a fragment serial number of a fragment belonging to each record; (4) and generating a storage file number and an internal offset according to the key words and the application numbers, and recording the mapping relation between the key words and the fragment serial numbers by using the fragment information index. The invention realizes that the distributed real-time data can be subjected to multi-application independent fragmentation, and the fragmentation information is independently stored and independently retrieved; and real-time data fragmentation information can be retrieved with constant level performance.
Description
Technical Field
The invention belongs to the technical field of automatic power grid model management of a power system, and particularly relates to a fragment information storage method of a distributed real-time memory database.
Background
With the increasingly complex operation mode of the power grid, the dispatching operation management of the power grid faces new challenges, and higher requirements are put forward on a power grid model and parameters serving as a basis for safe and stable operation of the power grid. A single node and a centrally stored real-time database cannot cope with the increasing real-time data scale, fragment and partition management needs to be performed on the real-time data, and meanwhile, in order to guarantee the data direction performance, the mapping relationship between the data of the data and the fragments to which the data belong, namely partition information, needs to be recorded efficiently.
Disclosure of Invention
Aiming at the problems, the invention provides a fragment information storage method of a distributed real-time memory database, which realizes that distributed real-time data can be subjected to multi-application independent fragmentation, and fragment information is independently stored and independently retrieved; and real-time data fragmentation information can be retrieved with constant level (O (1)) performance.
The technical purpose is achieved, the technical effect is achieved, and the invention is realized through the following technical scheme:
a fragment information storage method of a distributed real-time memory database comprises the following steps:
(1) acquiring real-time data of different applications to form a real-time database;
(2) using long and integer data as a keyword of a real-time data table in a real-time database, wherein the keyword can be divided into a table identifier and a record serial number; each record in the real-time data table has an application number corresponding to the application to which the record belongs;
(3) independently fragmenting real-time data of different applications in a real-time data table, storing the real-time data into a corresponding storage file, and generating a fragment serial number of a fragment belonging to each record;
(4) and generating a storage file number and an internal offset according to the key words and the application numbers, and recording the mapping relation between the key words and the fragment serial numbers by using the fragment information index.
Furthermore, the table identifier of the table where the data corresponding to the keyword is located is recorded by using 2 bytes of data in the keyword, and the record sequence number corresponding to the keyword is recorded by using 4 bytes of data.
Further, the step (4) of calculating the storage location of the fragmentation information according to the application number and the keyword specifically includes:
splitting a record serial number in the keyword into high 2 bytes and low 2 bytes, and combining the high 2 bytes, the table identifier in the keyword and the corresponding application number into a storage file number; and the low 2 bytes are used as the offset in the storage file and used for storing the fragment sequence number corresponding to the record in the corresponding offset position of the storage file corresponding to the keyword.
Further, the application number is encoded by 2 bytes.
Further, the same data table in the real-time database can belong to different applications and be managed independently.
Further, the sizes of the storage files are the same and are block files.
The invention has the beneficial effects that:
(1) the distributed real-time data can be subjected to multi-application independent fragmentation, and fragmentation information is independently stored and independently retrieved; and real-time data fragmentation information can be retrieved with constant level performance.
(2) The data synchronization mode of the multi-application real-time database is realized by combining the self-developed real-time database working on an application layer, does not depend on a specific operating system, and is suitable for various operating systems.
(3) The multi-application real-time database data synchronization mode related by the invention ensures the portability, and can be safely and stably operated on operation system platforms such as IBM AIX, Sun Solaris, Alpha Tru64, HPUX, Linux, Windows and the like along with a real-time database system; the system programming language adopts ANSI C/C + +, and conforms to the IEEE POSIX.2 standard.
Drawings
FIG. 1 is a schematic diagram illustrating a process of creating a tile information index according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a retrieval process of a fragment information index according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The following detailed description of the principles of the invention is provided in connection with the accompanying drawings.
Example 1
A fragment information storage method of a distributed real-time memory database comprises the following steps:
(1) acquiring real-time data of different applications to form a real-time database; the same data table in the real-time database can belong to different applications and is managed independently;
(2) using long and integer data as a keyword of a real-time data table in a real-time database, wherein 2 bytes of data are used in the keyword to record a table identifier of a table where the data corresponding to the keyword are located, and 4 bytes of data are used to record a record serial number corresponding to the keyword; each record in the real-time data table has an application number corresponding to the application to which the record belongs; the application number is preferably encoded by 2 bytes;
(3) independently fragmenting real-time data of different applications in a real-time data table, storing the real-time data into a corresponding storage file, and generating a fragment serial number of a fragment belonging to each record; the sizes of all the storage files are the same and are block files.
(4) Generating a storage file number and an internal offset according to the keywords and the application numbers, and recording the mapping relation between the keywords and the affiliated fragment serial numbers by using the fragment information index, which specifically comprises the following steps:
splitting a record serial number in the keyword into high 2 bytes and low 2 bytes, and combining the high 2 bytes, the table identifier in the keyword and the corresponding application number into a storage file number; and the low 2 bytes are used as the offset in the storage file and used for storing the fragment sequence number corresponding to the record in the corresponding offset position of the storage file corresponding to the keyword.
As shown in fig. 2, when performing real-time data retrieval, in combination with the fragment information index, the storage location of the fragment data can be calculated by the keyword, and the fragment number is queried.
Example 2
In the embodiment of the invention, the data in the real-time database all have the applications, and are numbered by 2 bytes, such as application 1, application 2 and the like;
in the embodiment of the invention, the data in the real-time database all use long integer numbers as key words and are coded according to the following modes:
table identification of high 2 bytes recording data and low 4 bytes recording sequence number;
if the table is marked as 405 and the record serial number is 80332, the key is 113997365567896012, and the 16-system is represented as 0x 1950000000139 CC. The upper 2 bytes of data are 405(0x195) and the lower 4 bytes are 80332(0x000139 CC).
In the embodiment of the present invention, real-time data is firstly fragmented, and a fragment serial number of each record is generated under different applications, for example, the fragment serial number of each record corresponding to the keyword is 25. Then, a storage file number and an internal offset are generated according to the key and the application number, in the case of the above record, the record sequence number is split into 1 and 14796 according to the upper 2 bytes and the lower 2 bytes, in the application 1, the upper 2 bytes (1) of the record sequence number, the table identifier (405) and the application number (1) are merged into a file number 000101950001, 14796 is used as the internal offset, and the key and the fragment sequence number are merged and stored in a 14796 data unit of the file 000101950001.
Similarly, for other applications, the storage file number and the storage location should be generated according to the above rules.
In summary, the following steps:
(1) the distributed real-time data can be subjected to multi-application independent fragmentation, and fragmentation information is independently stored and independently retrieved; and real-time data fragmentation information can be retrieved with constant level performance.
(2) The data synchronization mode of the multi-application real-time database is realized by combining the self-developed real-time database working on an application layer, does not depend on a specific operating system, and is suitable for various operating systems.
(3) The multi-application real-time database data synchronization mode related by the invention ensures the portability, and can be safely and stably operated on operation system platforms such as IBM AIX, Sun Solaris, Alpha Tru64, HPUX, Linux, Windows and the like along with a real-time database system; the system programming language adopts ANSI C/C + +, and conforms to the IEEE POSIX.2 standard.
The foregoing shows and describes the general principles and broad features of the present invention and advantages thereof. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (6)
1. A fragment information storage method of a distributed real-time memory database is characterized by comprising the following steps:
(1) acquiring real-time data of different applications to form a real-time database;
(2) using long and integer data as a keyword of a real-time data table in a real-time database, wherein the keyword can be divided into a table identifier and a record serial number; each record in the real-time data table has an application number corresponding to the application to which the record belongs;
(3) independently fragmenting real-time data of different applications in a real-time data table, storing the real-time data into a corresponding storage file, and generating a fragment serial number of a fragment belonging to each record;
(4) and generating a storage file number and an internal offset according to the key words and the application numbers, and recording the mapping relation between the key words and the fragment serial numbers by using the fragment information index.
2. The method according to claim 1, wherein the method comprises the following steps: and recording the table identifier of the table where the data corresponding to the keyword is located by using 2 bytes of data in the keyword, and recording the recording sequence number corresponding to the keyword by using 4 bytes of data.
3. The method according to claim 1, wherein the method comprises the following steps: the step (4) of calculating the storage location of the fragmentation information according to the application number and the keywords specifically includes:
splitting a record serial number in the keyword into high 2 bytes and low 2 bytes, and combining the high 2 bytes, the table identifier in the keyword and the corresponding application number into a storage file number; and the low 2 bytes are used as the offset in the storage file and used for storing the fragment sequence number corresponding to the record in the corresponding offset position of the storage file corresponding to the keyword.
4. The method according to claim 1, wherein the method comprises the following steps: the application number is encoded by 2 bytes.
5. The method according to claim 1, wherein the method comprises the following steps: the same data table in the real-time database can belong to different applications and is managed independently.
6. The method according to claim 1, wherein the method comprises the following steps: the sizes of all the storage files are the same and are block files.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710858824.6A CN107609143B (en) | 2017-09-21 | 2017-09-21 | Fragment information storage method of distributed real-time memory database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710858824.6A CN107609143B (en) | 2017-09-21 | 2017-09-21 | Fragment information storage method of distributed real-time memory database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107609143A CN107609143A (en) | 2018-01-19 |
| CN107609143B true CN107609143B (en) | 2020-06-05 |
Family
ID=61060543
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710858824.6A Active CN107609143B (en) | 2017-09-21 | 2017-09-21 | Fragment information storage method of distributed real-time memory database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107609143B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108664660A (en) * | 2018-05-21 | 2018-10-16 | 北京五八信息技术有限公司 | Distributed implementation method, apparatus, equipment and the storage medium of time series database |
| CN110008289B (en) * | 2019-03-01 | 2022-08-26 | 国电南瑞科技股份有限公司 | Relational database and power grid model data storage and retrieval method |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120078874A1 (en) * | 2010-09-27 | 2012-03-29 | International Business Machine Corporation | Search Engine Indexing |
| CN102024057B (en) * | 2010-12-24 | 2015-07-01 | 中兴通讯股份有限公司 | Method and device for building index of mass data record |
| CN102541968B (en) * | 2010-12-31 | 2014-04-09 | 百度在线网络技术（北京）有限公司 | Indexing method |
| CN102541623B (en) * | 2011-12-20 | 2015-02-11 | 北京控制工程研究所 | Memory space simulation method for embedded processor |
| CN103294710B (en) * | 2012-02-28 | 2016-08-31 | 北京新媒传信科技有限公司 | A kind of data access method and device |
| CN104504137A (en) * | 2014-12-31 | 2015-04-08 | 深圳市科漫达智能管理科技有限公司 | Data storage method and system |
| CN106648446B (en) * | 2015-10-30 | 2020-07-07 | 阿里巴巴集团控股有限公司 | Storage method and device for time sequence data and electronic equipment |
| CN105912601A (en) * | 2016-04-05 | 2016-08-31 | 国电南瑞科技股份有限公司 | Partition storage method for distributed real-time memory database of energy management system |
| CN106682077B (en) * | 2016-11-18 | 2020-06-09 | 山东鲁能软件技术有限公司 | Mass time sequence data storage implementation method based on Hadoop technology |
-
2017
- 2017-09-21 CN CN201710858824.6A patent/CN107609143B/en active Active
Non-Patent Citations (2)
| Title |
|---|
| Chenggang Zhen，等.Memory management research based on real-time database.《2009 International Conference on Test and Measurement》.2009, * |
| 喻宜，等.分布式海量时序数据管理平台研究.《电力系挽保护与控制》.2016,第44卷(第17期), * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107609143A (en) | 2018-01-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5996088B2 (en) | Cryptographic hash database | |
| US9953107B2 (en) | Memory system including key-value store | |
| US9727573B1 (en) | Out-of core similarity matching | |
| US20180307428A1 (en) | Data storage method, electronic device, and computer non-volatile storage medium | |
| CN106874348B (en) | File storage and index method and device and file reading method | |
| CN105468642A (en) | Data storage method and apparatus | |
| US20110040761A1 (en) | Estimation of postings list length in a search system using an approximation table | |
| CN103294710A (en) | Data access method and device | |
| EP2821924A1 (en) | Method, device and system for querying data index | |
| CN102591864B (en) | Data updating method and device in comparison system | |
| CN110888837B (en) | Object storage small file merging method and device | |
| CN104539750A (en) | IP locating method and device | |
| WO2023143096A1 (en) | Data query method and apparatus, and device and storage medium | |
| CN107609143B (en) | Fragment information storage method of distributed real-time memory database | |
| CN104714957A (en) | List structure changing method and device | |
| CN104778252A (en) | Index storage method and index storage device | |
| CN112416879B (en) | NTFS file system-based block-level data deduplication method | |
| JP2007287147A (en) | Fast file attribute search | |
| CN108304384B (en) | Word splitting method and device | |
| CN111026736B (en) | Data blood margin management method and device and data blood margin analysis method and device | |
| KR20190123819A (en) | Method for managing of memory address mapping table for data storage device | |
| CN105740374A (en) | Distributed memory based three-dimensional platform data fuzzy query method | |
| CN116501695A (en) | Data file processing method, apparatus, device, medium and program product | |
| CN112394876A (en) | Large file storage/reading method, storage/reading device and computer equipment | |
| CN112380174B (en) | XFS file system analysis method containing deleted files, terminal device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |