WO2017101478A1 - Postgresql block storage device read-write module - Google Patents

Postgresql block storage device read-write module Download PDF

Info

Publication number
WO2017101478A1
WO2017101478A1 PCT/CN2016/095367 CN2016095367W WO2017101478A1 WO 2017101478 A1 WO2017101478 A1 WO 2017101478A1 CN 2016095367 W CN2016095367 W CN 2016095367W WO 2017101478 A1 WO2017101478 A1 WO 2017101478A1
Authority
WO
WIPO (PCT)
Prior art keywords
postgresql
postgresql block
block
data
storage device
Prior art date
Application number
PCT/CN2016/095367
Other languages
French (fr)
Chinese (zh)
Inventor
朱晓伟
Original Assignee
深圳市华讯方舟软件技术有限公司
华讯方舟科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市华讯方舟软件技术有限公司, 华讯方舟科技有限公司 filed Critical 深圳市华讯方舟软件技术有限公司
Publication of WO2017101478A1 publication Critical patent/WO2017101478A1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/214Database migration support

Definitions

  • the invention relates to a PostgreSQL database storage device, in particular to a PostgreSQL block storage device read/write module.
  • the original PostgreSQL database system's underlying storage architecture is on the file system. From the table to the disk, the four layers of the tablespace, file system, logical volume, and disk are finally written to the physical disk.
  • Fsm file It stores information about the free space in the data table file.
  • Vm file A tuple that marks which file blocks in the data table file are not invalid.
  • Table data file This file is mainly used to store data, but the file size will be limited. Normally, 2G will be saved. Files larger than 2G will be saved in files.
  • the general operating system has a limit on the number of files that can be opened.
  • the PostgreSQL database system internally uses its own virtual file management to manage file handles to ensure that many tables can be opened at the same time. For the database, there may be many tables at the same time, and the PostgreSQL database system uses one table and multiple files to save data.
  • you open a file you will first find in the file handle management whether there is a file handle that has been opened. Set the offset according to the file handle.
  • centralized storage disk array, etc.
  • This method only hardly improves the I/O throughput of the back end, and does not improve the utilization of the underlying I/O in principle. And in terms of security, this method does not protect the data at any level, and still exposes the data. It can only achieve the speedup of the database system to a certain extent. But for the high security and high performance requirements, we can't just rely on this pure hardware acceleration and ignore the optimization of the software itself.
  • the above is the original PostgreSQL database system storage architecture.
  • the above structure has the following disadvantages:
  • the database data files are directly exposed in the operating system, the data has security risks, and the data security is not guaranteed. It is a great security vulnerability for some environments with high security requirements.
  • 64-bit space is used to record the number of blocks and the number of i-nodes.
  • 64-bit space is used to record the number of blocks and the number of i-nodes.
  • database systems there may be a large number of tables, and each table may have at least 3 files. , resulting in a lot of files in a folder.
  • the file system When the file system performs block allocation, it is basically allocated in a 4K block mode. At the same time, our database application is applied in 8K units. This means that the blocks allocated by the system are always smaller than the blocks of our table.
  • the direct problem with this is that the database's business blocks are cluttered to disk. The direct problem is that the disk seek time is long and the read and write speed is slowed down.
  • the technical problem to be solved by the present invention is to provide a PostgreSQL block storage device read/write module, through which the read and write modules of the PostgreSQL block storage device read and write, the database data file does not Direct exposure to the operating system eliminates the security risks of data, and data security is guaranteed. It eliminates a huge security hole for some environments with high security requirements.
  • the PostgreSQL block storage device read/write module of the present invention is a PostgreSQL block storage device read/write module for managing a PostgreSQL block in a PostgreSQL block storage device, and the PostgreSQL block storage device read/write module is architected in a PostgreSQL database. On the system.
  • the PostgreSQL block storage device read/write module manages PostgreSQL blocks in the PostgreSQL block storage device through a mapping relationship table between PostgreSQL block-data tables and an idle PostgreSQL block table.
  • the mapping relationship table between the PostgreSQL block and the data table includes fields Relfilenode, Reltablespace, Forknum, Blockid, Blockno, and the free PostgreSQL block table includes fields Blockid, Isfree, Dev.
  • the PostgreSQL block storage device read and write module has the following submodules:
  • a submodule that writes data in a PostgreSQL block is a submodule that writes data in a PostgreSQL block.
  • the sub-module that allocates the PostgreSQL block allocates a PostgreSQL block to the PostgreSQL data table by using a nearby allocation policy or a hot-hot data hierarchical allocation policy, and the nearby allocation strategy is a PostgreSQL block nearest allocation policy or an idle PostgreSQL block table recording nearest allocation policy.
  • the hot-cold data table hierarchical allocation strategy is a common data table allocation strategy or a recently used data table allocation strategy.
  • the PostgreSQL block nearest allocation strategy is to find an idle PostgreSQL block to be assigned to the data table before and after the last allocated PostgreSQL block of the data table.
  • the free PostgreSQL block table record near allocation strategy is to find the first free PostgreSQL block from the idle PostgreSQL block table record, the commonly used data table allocation strategy is to preferentially allocate the data table to the faster PostgreSQL block device.
  • the recently used data table allocation strategy is to prioritize the recently used data table to a faster PostgreSQL block device for a PostgreSQL block device having two or more different read and write speeds.
  • the submodule of the Recycled PostgreSQL block is used to reclaim the PostgreSQL block that is no longer used by the data table, and the corresponding PostgreSQL block record is deleted from the mapping relationship table between the PostgreSQL block and the data table, and the record of the corresponding PostgreSQL block is sent to the free table.
  • the sub-module for locating the PostgreSQL block is used to locate a page in the data table to a location specified by the PostgreSQL block device;
  • the submodule for reading data in the PostgreSQL block is used to read data of a specified size in a specified position in the specified block;
  • the sub-module that writes the data in the PostgreSQL block is used to write data of a specified size within a specified location within the specified block.
  • the submodule that writes the data in the PostgreSQL block writes the data to the PostgreSQL block in an encrypted manner; accordingly,
  • the sub-module that reads the data in the PostgreSQL block reads the data back to the database business layer in a decrypted manner.
  • the encryption method is a method of performing storage encryption on the user password and data XOR;
  • the decryption method is a method of encrypting the user password with the data read from the PostgreSQL block and then returning it to the database service layer.
  • the Dev field in the idle PostgreSQL block table identifies different storage devices to implement capacity expansion.
  • the PostgreSQL block is a minimum unit allocated to a data table in a PostgreSQL database system, the storage capacity of the PostgreSQL block is greater than 4 KB, and the PostgreSQL block storage device read/write module is configured by a data read/write method based on a PostgreSQL block storage device. PostgreSQL blocks are read and written.
  • the storage capacity of the PostgreSQL block is a positive integer multiple of 8 KB.
  • the storage capacity of the PostgreSQL block is a natural multiple of 8 KB of 2 KB.
  • the storage capacity of the PostgreSQL block is 1 MB, 2 MB, 4 MB, 8 MB, 16 MB, 32 MB, 64 MB, 128 MB, 256 MB, 512 MB or 1024 MB.
  • a data read/write method based on a PostgreSQL block storage device which reads and writes a PostgreSQL block as described above by using a PostgreSQL block storage device read/write module as described above.
  • the driver of the PostgreSQL block storage device read/write module completes the operation of the block file, and its main tasks are:
  • the table space is located to a specified position of the specified block device, and a certain offset of the block device is located by the pageid of the table.
  • Each table in the PostgreSQL database system is assigned at least one of the PostgreSQL blocks, and the PostgreSQL block can be dynamically allocated incrementally;
  • Each PostgreSQL block can write to 512 8-KB PostgreSQL block database system pages.
  • a block device can contain an infinite number of tables.
  • a table can contain an unlimited number of PostgreSQL blocks. These block file information is stored in PostgreSQL block-data. The mapping relationship between the tables;
  • the information of all the PostgreSQL blocks of a data table can be obtained by one oid, and the PostgreSQL block information will sequentially form the entire PostgreSQL block-data table file.
  • the pageid*8k of this table is the internal bias of the table. Move, and at the same time be able to determine the first block of this block file, and thus get the position of the page we want to get in the block device.
  • the PostgreSQL block device read/write storage module of the present invention has the following advantageous effects compared with the prior art.
  • the data in the database will be more secure.
  • the data in the database will be saved to the block device according to the rules we have established, which is more secure in terms of security.
  • the data will be saved in pieces and saved in multiple copies to different data nodes, which is more than 100 times more secure than a single disk.
  • Database business and data separation As long as it is the same version of the database, it can be freely migrated in different operating systems. As long as you can pass the verification code set at the head of the data block, Can achieve direct mount of the database. It is free to migrate from different versions of Linux, and can also migrate with the Windows operating system platform.
  • PostgreSQL block the PostgreSQL block device read and write storage module, and the data read and write method based on the PostgreSQL block storage device of the present invention are further described in detail below with reference to the accompanying drawings and specific embodiments.
  • FIG. 1 is a schematic diagram of a file system read and write hierarchical structure of an operating system in the prior art.
  • FIG. 2 is a schematic diagram of a file system storage space allocation structure of an operating system in the prior art.
  • FIG. 3 is a schematic diagram of the principle of read and write of a file system head of an operating system in the prior art.
  • FIG. 4 is a schematic diagram showing the structure and principle of a PostgreSQL block, a PostgreSQL block device read/write memory module, and a data read/write method based on a PostgreSQL block storage device.
  • Embodiment 1 is a diagrammatic representation of Embodiment 1:
  • the PostgreSQL block of the present invention is the smallest unit allocated to the data table in the PostgreSQL database system, and the storage capacity of the PostgreSQL block is equal to 4 MB.
  • the storage capacity of the PostgreSQL block is greater than 4 KB.
  • the storage capacity of the PostgreSQL block is a positive integer multiple of 8 KB.
  • the storage capacity of the PostgreSQL block is a natural multiple of 8 KB of 2 KB.
  • the storage capacity of the PostgreSQL block is 1 MB, 2 MB, 8 MB, 16 MB, 32 MB, 64 MB, 128 MB, 256 MB, 512 MB or 1024 MB.
  • Embodiment 2 is a diagrammatic representation of Embodiment 1:
  • the PostgreSQL block storage device read/write module of the present invention is a PostgreSQL block storage device read/write module for managing a PostgreSQL block as described above in a PostgreSQL block storage device.
  • the PostgreSQL block storage device read/write module can be architected on a PostgreSQL database system.
  • the PostgreSQL block storage device read and write module passes the PostgreSQL block-data table mapping table and the idle PostgreSQL block table to the PostgreSQL block storage device. PostgreSQL blocks are managed.
  • the mapping relationship table between the PostgreSQL block and the data table includes fields Relfilenode, Reltablespace, Forknum, Blockid, Blockno, and the free PostgreSQL block table includes fields Blockid, Isfree, Dev.
  • the PostgreSQL block storage device read and write module has the following submodules:
  • a submodule that writes data in a PostgreSQL block is a submodule that writes data in a PostgreSQL block.
  • the sub-module that allocates the PostgreSQL block allocates a PostgreSQL block to the PostgreSQL data table by using a nearby allocation policy or a hot-hot data hierarchical allocation policy, and the nearby allocation strategy is a PostgreSQL block nearest allocation policy or an idle PostgreSQL block table recording nearest allocation policy.
  • the hot-cold data table hierarchical allocation strategy is a common data table allocation strategy or a recently used data table allocation strategy.
  • the PostgreSQL block nearest allocation strategy is to find an idle PostgreSQL block to be assigned to the data table before and after the last allocated PostgreSQL block of the data table.
  • the free PostgreSQL block table record near allocation strategy is to find the first free PostgreSQL block from the idle PostgreSQL block table record, the commonly used data table allocation strategy is to preferentially allocate the data table to the faster PostgreSQL block device.
  • the recently used data table allocation strategy is to prioritize the recently used data table to a faster PostgreSQL block device for a PostgreSQL block device having two or more different read and write speeds. on;
  • the submodule of the Recycled PostgreSQL block is used to reclaim the PostgreSQL block that is no longer used by the data table, and the corresponding PostgreSQL block record is deleted from the mapping relationship table between the PostgreSQL block and the data table, and the record of the corresponding PostgreSQL block is sent to the free table.
  • the sub-module for locating the PostgreSQL block is used to locate a page in the data table to a location specified by the PostgreSQL block device;
  • the submodule for reading data in the PostgreSQL block is used to read data of a specified size in a specified position in the specified block;
  • the sub-module that writes the data in the PostgreSQL block is used to write data of a specified size within a specified location within the specified block.
  • the submodule that writes the data in the PostgreSQL block writes the data to the PostgreSQL block in an encrypted manner; accordingly,
  • the sub-module that reads the data in the PostgreSQL block reads the data back to the database business layer in a decrypted manner.
  • the encryption method is a method of performing storage encryption on the user password and data XOR;
  • the decryption method is a method of deciphering the user password and the data read from the PostgreSQL block and then deciphering it to the database service layer.
  • the Dev field in the idle PostgreSQL block table identifies different storage devices to implement capacity expansion.
  • Embodiment 3 is a diagrammatic representation of Embodiment 3
  • the present invention is based on a data read/write method of a PostgreSQL block storage device, and reads and writes a PostgreSQL block as described above by using a PostgreSQL block storage device read/write module as described above.
  • the driver of the PostgreSQL block storage device read/write module completes the operation of the block file, and its main tasks are:
  • the table space is located to a specified position of the specified block device, and a certain offset of the block device is located by the pageid of the table.
  • Each table in the PostgreSQL database system is assigned at least one of the PostgreSQL blocks, and the PostgreSQL block can be dynamically allocated incrementally;
  • Each PostgreSQL block can write to 512 8-KB PostgreSQL block database system pages.
  • a block device can contain an infinite number of tables.
  • a table can contain an unlimited number of PostgreSQL blocks. These block file information is stored in PostgreSQL block-data. Mapping between tables Within the relationship table;
  • the information of all the PostgreSQL blocks of a data table can be obtained by one oid, and the PostgreSQL block information will sequentially form the entire PostgreSQL block-data table file.
  • the pageid*8k of this table is the internal bias of the table. Move, and at the same time be able to determine the first block of this block file, and thus get the position of the page we want to get in the block device.
  • the PostgreSQL block storage device In general, we split the PostgreSQL block storage device by 4MB. We only need to calculate the total number of PostgreSQL block storage devices by dividing the total capacity of the PostgreSQL block storage device by the capacity of a PostgreSQL block. This does not require scanning the PostgreSQL block storage device. This allocation time is extremely short. For example, the 2T PostgreSQL block storage device only needs to be divided into 524288 PostgreSQL blocks. We only need to add 524288 records to the free PostgreSQL block table, and the blockid can be incremented from 0--524287. Also mark isfree as true.
  • the storage space of the data table is allocated in units of 4 MB.
  • the actual occupied space of a table is allocated on demand.
  • the original SMgrRelation reln, ForkNumber forknum two parameters are located to the file system's table file modified to locate the offset of the block device.
  • PostgreSQL block-data table exists in the mapping table, we can directly locate the PostgreSQL block storage device to pageno*4*1024KB+(reln.pageid%512)*8KB. Read this 8K data.
  • the database business layer is separated from the data layer. Mount the block device directly to the new node. This upper level system error does not affect the underlying data.
  • PostgreSQL block storage device selection rules can be customized.
  • step B we are the choice of order.
  • PostgreSQL database system background storage is stored on a bare disk basis.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

A PostgreSQL block storage device read-write module, the PostgreSQL block storage device read-write module being a PostgreSQL block storage device read-write module that manages PostgreSQL blocks in a PostgreSQL block storage device, and the PostgreSQL block storage device read-write module being constructed on a PostgreSQL database system. The PostgreSQL block storage device read-write module manages, by means of a idle PostgreSQL block table and a table of mapping relationships between PostgreSQL blocks and data tables, the PostgreSQL blocks in the PostgreSQL block storage device. The PostgreSQL block storage device read-write module will perform more secure data reading and writing operations to the database. The data in the database will be saved into the block device according to an established rule, such that the data is more secure in terms of security. Moreover, due to the use of distributed storage, the data can be saved in slices and multiple copies thereof are saved in different data nodes, and thus the security is more than a hundred times higher than that of a single disk.

Description

一种PostgreSQL块存储设备读写模块A PostgreSQL block storage device read/write module 技术领域Technical field
本发明涉及一种PostgreSQL数据库存储设备,尤其涉及一种PostgreSQL块存储设备读写模块。The invention relates to a PostgreSQL database storage device, in particular to a PostgreSQL block storage device read/write module.
背景技术Background technique
随着互联网、移动互联网和物联网的发展,我们迎来了一个海量数据的时代,而数据库内保存的数据也越来越多,而我们需要的查询时间反而要越来越小。现在的众多应用场景都需要后台的存储具有高并发,高容量,高响应。高速的入库需求迫使我们不得不放弃实时索引,而大数据量的数据扫描又被存储端的I/O所限制。现在的PostgreSQL数据库系统中,高并发的多数据库扫描需要碰到基本都是随机读,而与此同时,普通磁盘的吞吐量已经不能满足需求。With the development of the Internet, the mobile Internet and the Internet of Things, we are ushered in an era of massive data, and the data stored in the database is increasing, and the query time we need is getting smaller and smaller. Many application scenarios today require background storage with high concurrency, high capacity, and high response. High-speed inbound demand forces us to abandon real-time indexing, and large data volumes are limited by I/O on the storage side. In today's PostgreSQL database system, high-concurrency multi-database scanning needs to be basically random read, while at the same time, the throughput of ordinary disks can no longer meet the demand.
为了适应大数据量的应用场景,官方推出的PostgreSQL-XC、PostgreSQL-XL这两个MPP数据库还没有成熟,而且还存在很多安全性问题。而传统的提速方案仅仅是利用集中式存储的高I/O(输入/输出)来在一定程度上提高整个系统的。但这种提升依旧存在很大的资源浪费。In order to adapt to the application scenario of large data volume, the officially launched PostgreSQL-XC and PostgreSQL-XL two MPP databases are not mature yet, and there are still many security problems. The traditional speed-up solution only uses the high I/O (input/output) of centralized storage to improve the whole system to some extent. But this kind of improvement still has a lot of waste of resources.
如图1所示,原有的PostgreSQL数据库系统底层存储架构在文件系统上,从表到磁盘需要经过:表空间,文件系统,逻辑卷、磁盘这四层最终才会写入到物理磁盘内。As shown in Figure 1, the original PostgreSQL database system's underlying storage architecture is on the file system. From the table to the disk, the four layers of the tablespace, file system, logical volume, and disk are finally written to the physical disk.
这样的架构首先会使得磁盘的I/O形成衰减。最终的磁盘I/O利用率只能达到80%左右,甚至更低。其次,当数据库表存在大量小表的情况下,势必会增加文件系统的压力。而且对于数据库操作而言,表的写入读出都较为随机,极容易造成磁盘碎片,而大量的随机读写,也必将会使得整个数据的读写性能下降。对于小数据量的数据库而言,现有的缺点也许不明显。但现在更多的使用场景是大数据量、高并发、高I/O。这些使用场景下,这些缺点将会大大影响数据库性能。此外在不同的操作系统迁移中也需要极复杂的数据库迁移操作。 Such an architecture will first cause the I/O of the disk to form attenuate. The final disk I/O utilization can only reach about 80% or even lower. Secondly, when there are a large number of small tables in the database table, it will inevitably increase the pressure on the file system. Moreover, for database operations, the write and read of the table are relatively random, which is very likely to cause disk fragmentation, and a large number of random reads and writes will also make the read and write performance of the entire data degrade. For small data volumes, the existing shortcomings may not be obvious. But now more usage scenarios are big data volumes, high concurrency, and high I/O. These shortcomings will greatly affect database performance in these usage scenarios. In addition, very complex database migration operations are required in different operating system migrations.
普通磁盘的I/O速度较低,传统的解决方法是使用集中式存储(即磁盘阵列)。这种方式只是简单的提高磁盘的I/O速度,而且成本极高,具有一定的局限性。Ordinary disks have low I/O speeds, and the traditional solution is to use centralized storage (that is, disk arrays). This method is simply to increase the I/O speed of the disk, and the cost is extremely high, which has certain limitations.
如图2和图3所示,PostgreSQL数据库系统现有的后台存储架构:As shown in Figure 2 and Figure 3, the existing backend storage architecture of the PostgreSQL database system:
数据库的数据操作每条记录都有几个主要标识。表ID、记录所在black id、文件类型、行记录对应的标识。现有的PostgreSQL数据库系统读写表操作需要:The data manipulation of the database has several primary identifiers for each record. Table ID, the black id of the record, the file type, and the identifier corresponding to the row record. Existing PostgreSQL database system read and write table operations need to:
a根据表ID查找到这个表的表空间,然后定位到文件系统中表空间的位置。a Find the tablespace of this table based on the table ID, and then locate the location of the tablespace in the file system.
b根据文件类型要获取到指定的表文件。一般一个表会包含3种类型的文件。bAcquire the specified table file according to the file type. Usually a table will contain 3 types of files.
fsm文件:其中存放了数据表文件中空闲空间的信息。Fsm file: It stores information about the free space in the data table file.
vm文件:标记了数据表文件中哪些文件块没有失效的元组。Vm file: A tuple that marks which file blocks in the data table file are not invalid.
表数据文件:此文件主要用于存储数据,但是此文件大小会有一定的限制,普通情况下会保存2G,超过2G的文件会分文件保存。Table data file: This file is mainly used to store data, but the file size will be limited. Normally, 2G will be saved. Files larger than 2G will be saved in files.
c根据记录号以及fsm文件,定位page。c Position the page based on the record number and the fsm file.
而一般的操作系统对于打开的文件数会有限制。PostgreSQL数据库系统内部使用自己的虚拟文件管理来管理文件句柄,来保证能够同时打开很多的表。对于数据库而言可能同时存在很多的表,而PostgreSQL数据库系统采用1个表多个文件方式来保存数据。当打开一个文件的时候,会首先到文件句柄管理中找一下是不是存在已经打开的文件句柄。根据文件句柄设定一下offset。The general operating system has a limit on the number of files that can be opened. The PostgreSQL database system internally uses its own virtual file management to manage file handles to ensure that many tables can be opened at the same time. For the database, there may be many tables at the same time, and the PostgreSQL database system uses one table and multiple files to save data. When you open a file, you will first find in the file handle management whether there is a file handle that has been opened. Set the offset according to the file handle.
d从文件中取到对应的page,并取到对应的行。d Take the corresponding page from the file and take the corresponding row.
根据记录的blockid定位到文件,读取指定大小的块。而这些记录就保存在page内。Locate the file according to the recorded blockid and read the block of the specified size. These records are saved in the page.
现有技术中,采用集中式存储作为数据库后端。In the prior art, centralized storage is employed as the database backend.
1、采用集中式存储(磁盘阵列等)做为存储后端。把集中式存储提供的磁盘挂载到PostgreSQL数据库系统所安装的机器。并根据不同的操作系统格式化成不同的文件系统。1. Use centralized storage (disk array, etc.) as the storage backend. Mount the disk provided by the centralized storage to the machine installed on the PostgreSQL database system. And formatted into different file systems according to different operating systems.
2、把数据库中的数据指定到集中式存储提供的磁盘。 2. Assign the data in the database to the disk provided by the centralized storage.
此种方法仅仅硬性提高后端的I/O吞吐量,并没有从原理上提高底层I/O的利用率。而且就安全性而言,这种方法并没有对数据做任何层面的保护,依旧让数据暴露在外。只能够在一定程度上实现数据库系统的提速。但是对于高安全高性能的需求来说,我们不能仅仅依靠这种纯硬件的加速而忽略软件本身的优化。This method only hardly improves the I/O throughput of the back end, and does not improve the utilization of the underlying I/O in principle. And in terms of security, this method does not protect the data at any level, and still exposes the data. It can only achieve the speedup of the database system to a certain extent. But for the high security and high performance requirements, we can't just rely on this pure hardware acceleration and ignore the optimization of the software itself.
以上是原有的PostgreSQL数据库系统存储架构。以上结构存在以下几个缺点:The above is the original PostgreSQL database system storage architecture. The above structure has the following disadvantages:
第一、数据库数据文件直接暴露在操作系统中,数据存在安全隐患,数据安全也就得不到保证,对于一些对安全级别要求较高的使用环境而言是个极大的安全漏洞。First, the database data files are directly exposed in the operating system, the data has security risks, and the data security is not guaranteed. It is a great security vulnerability for some environments with high security requirements.
第二、对磁盘的使用必经过文件系统层。Second, the use of the disk must pass through the file system layer.
就linux系统下较为流行的文件系统而言,采用64位空间来记录块数量和i-节点数量,对于数据库系统,可能存在及大量的表,每个表都将可能至少存在3个以上的文件,导致一个文件夹内存在极多的文件。For the more popular file system under Linux system, 64-bit space is used to record the number of blocks and the number of i-nodes. For database systems, there may be a large number of tables, and each table may have at least 3 files. , resulting in a lot of files in a folder.
文件系统进行块分配时,基本上都是按照4K一个块的模式进行分配。同时我们数据库申请的时候都是以8K为单位进行申请。也就是说系统分配的块总是比我们的表的块要小。这样导致的直接问题就是数据库的业务块杂乱的分配到磁盘上。导致的直接问题就是磁盘寻道时间长,读写速度减慢。When the file system performs block allocation, it is basically allocated in a 4K block mode. At the same time, our database application is applied in 8K units. This means that the blocks allocated by the system are always smaller than the blocks of our table. The direct problem with this is that the database's business blocks are cluttered to disk. The direct problem is that the disk seek time is long and the read and write speed is slowed down.
假如我们的数据库中存在一个极大的表,文件系统在处理的时候效率极其低。例如,在ext3文件系统中100MB的文件就需要近25600个数据块。而对于PostgreSQL数据库系统而言,达到或超过GB级别的表处理会很常见,而且是随机的读写。If there is a huge table in our database, the file system is extremely inefficient at processing. For example, a 100MB file in an ext3 file system would require nearly 25,600 data blocks. For the PostgreSQL database system, table processing that meets or exceeds the GB level is common and is random read and write.
第三、文件系统的扩容也非常麻烦。除了已知的极个别商业文件系统外,其他的文件系统扩容多会要求关闭数据库。Third, the expansion of the file system is also very troublesome. In addition to the very few known commercial file systems, other file system expansions will require a shutdown of the database.
第四、数据库在不同的系统之间进行迁移及其复杂,对操作人员专业知识要求较高。Fourth, the database is migrated between different systems and its complexity, and the professional knowledge of the operators is relatively high.
发明内容Summary of the invention
本发明要解决的技术问题是提供一种PostgreSQL块存储设备读写模块,通过该PostgreSQL块存储设备读写模块的读写,数据库数据文件不会 直接暴露在操作系统中,消除了数据存在的安全隐患,数据安全也就得到了保证,对于一些对安全级别要求较高的使用环境而言消除了一个极大的安全漏洞。The technical problem to be solved by the present invention is to provide a PostgreSQL block storage device read/write module, through which the read and write modules of the PostgreSQL block storage device read and write, the database data file does not Direct exposure to the operating system eliminates the security risks of data, and data security is guaranteed. It eliminates a huge security hole for some environments with high security requirements.
为了解决上述技术问题,本发明的PostgreSQL块存储设备读写模块是对PostgreSQL块存储设备中的PostgreSQL块进行管理的PostgreSQL块存储设备读写模块,所述PostgreSQL块存储设备读写模块架构在PostgreSQL数据库系统上。In order to solve the above technical problem, the PostgreSQL block storage device read/write module of the present invention is a PostgreSQL block storage device read/write module for managing a PostgreSQL block in a PostgreSQL block storage device, and the PostgreSQL block storage device read/write module is architected in a PostgreSQL database. On the system.
所述PostgreSQL块存储设备读写模块通过PostgreSQL块-数据表之间的映射关系表和空闲PostgreSQL块表对PostgreSQL块存储设备中的PostgreSQL块进行管理。The PostgreSQL block storage device read/write module manages PostgreSQL blocks in the PostgreSQL block storage device through a mapping relationship table between PostgreSQL block-data tables and an idle PostgreSQL block table.
所述PostgreSQL块-数据表之间的映射关系表包括字段Relfilenode、Reltablespace、Forknum、Blockid、Blockno,所述空闲PostgreSQL块表包括字段Blockid、Isfree、Dev。The mapping relationship table between the PostgreSQL block and the data table includes fields Relfilenode, Reltablespace, Forknum, Blockid, Blockno, and the free PostgreSQL block table includes fields Blockid, Isfree, Dev.
所述PostgreSQL块存储设备读写模块,具有以下子模块:The PostgreSQL block storage device read and write module has the following submodules:
分配PostgreSQL块的子模块,Allocating submodules of the PostgreSQL block,
回收PostgreSQL块的子模块,Reclaim the submodules of the PostgreSQL block,
定位PostgreSQL块的子模块,Locate the submodule of the PostgreSQL block,
读出PostgreSQL块中数据的子模块,Read the submodule of the data in the PostgreSQL block,
写入PostgreSQL块中数据的子模块。A submodule that writes data in a PostgreSQL block.
所述分配PostgreSQL块的子模块采用就近分配策略或冷热数据分层分配策略给PostgreSQL数据表分配PostgreSQL块,所述就近分配策略是PostgreSQL块就近分配策略或空闲PostgreSQL块表记录就近分配策略,所述冷热数据表分层分配策略是常用数据表分配策略或近期使用数据表分配策略,所述PostgreSQL块就近分配策略是在数据表上次分配的PostgreSQL块前后就近寻找空闲PostgreSQL块分配给数据表,所述空闲PostgreSQL块表记录就近分配策略是从空闲PostgreSQL块表记录中寻找第一个空闲PostgreSQL块,所述常用数据表分配策略是经常使用的数据表优先分配到较快的PostgreSQL块设备上,所述近期使用数据表分配策略是将近期使用数据表优先分配到较快的PostgreSQL块设备上,所述冷热数据表分层分配策略用于具有两块以上不同读写速度的PostgreSQL块设备上; The sub-module that allocates the PostgreSQL block allocates a PostgreSQL block to the PostgreSQL data table by using a nearby allocation policy or a hot-hot data hierarchical allocation policy, and the nearby allocation strategy is a PostgreSQL block nearest allocation policy or an idle PostgreSQL block table recording nearest allocation policy. The hot-cold data table hierarchical allocation strategy is a common data table allocation strategy or a recently used data table allocation strategy. The PostgreSQL block nearest allocation strategy is to find an idle PostgreSQL block to be assigned to the data table before and after the last allocated PostgreSQL block of the data table. The free PostgreSQL block table record near allocation strategy is to find the first free PostgreSQL block from the idle PostgreSQL block table record, the commonly used data table allocation strategy is to preferentially allocate the data table to the faster PostgreSQL block device. The recently used data table allocation strategy is to prioritize the recently used data table to a faster PostgreSQL block device for a PostgreSQL block device having two or more different read and write speeds. Upper
所述回收PostgreSQL块的子模块用于回收数据表不再使用的PostgreSQL块,从PostgreSQL块-数据表之间的映射关系表中删除对应的PostgreSQL块记录,到空闲表中将相应PostgreSQL块的记录设置为空闲;The submodule of the Recycled PostgreSQL block is used to reclaim the PostgreSQL block that is no longer used by the data table, and the corresponding PostgreSQL block record is deleted from the mapping relationship table between the PostgreSQL block and the data table, and the record of the corresponding PostgreSQL block is sent to the free table. Set to idle;
所述定位PostgreSQL块的子模块用于将数据表中的页定位到PostgreSQL块设备指定的位置上;The sub-module for locating the PostgreSQL block is used to locate a page in the data table to a location specified by the PostgreSQL block device;
所述读出PostgreSQL块中数据的子模块用于读取指定块内指定位置指定大小的数据;The submodule for reading data in the PostgreSQL block is used to read data of a specified size in a specified position in the specified block;
所述写入PostgreSQL块中数据的子模块用于写入指定块内指定位置指定大小的数据。The sub-module that writes the data in the PostgreSQL block is used to write data of a specified size within a specified location within the specified block.
所述写入PostgreSQL块中数据的子模块以加密方式将数据写入PostgreSQL块中;相应地,The submodule that writes the data in the PostgreSQL block writes the data to the PostgreSQL block in an encrypted manner; accordingly,
所述读出PostgreSQL块中数据的子模块以解密的方式将数据读出返回给数据库业务层。The sub-module that reads the data in the PostgreSQL block reads the data back to the database business layer in a decrypted manner.
所述加密的方式是采用将用户密码与数据异或计算后进行存储加密的方式;The encryption method is a method of performing storage encryption on the user password and data XOR;
所述解密的方式是采用将用户密码与从PostgreSQL块中读出的数据异或计算后反回给数据库业务层加密的方式。The decryption method is a method of encrypting the user password with the data read from the PostgreSQL block and then returning it to the database service layer.
所述空闲PostgreSQL块表中的Dev字段对不同的存储设备进行标识实现扩容。The Dev field in the idle PostgreSQL block table identifies different storage devices to implement capacity expansion.
将所述PostgreSQL块-数据表之间的映射关系表和所述空闲PostgreSQL块表从原库复制到新库,将PostgreSQL块存储设备从原库所在的计算机迁移到新库所在的计算机。Copying the mapping relationship table between the PostgreSQL block-data table and the free PostgreSQL block table from the original library to the new library, and migrating the PostgreSQL block storage device from the computer where the original library is located to the computer where the new library is located.
所述PostgreSQL块是分配给PostgreSQL数据库系统中数据表的最小单位,所述PostgreSQL块的存储容量大于4KB,所述PostgreSQL块存储设备读写模块通过基于PostgreSQL块存储设备的数据读写方法对所述PostgreSQL块进行读写操作。The PostgreSQL block is a minimum unit allocated to a data table in a PostgreSQL database system, the storage capacity of the PostgreSQL block is greater than 4 KB, and the PostgreSQL block storage device read/write module is configured by a data read/write method based on a PostgreSQL block storage device. PostgreSQL blocks are read and written.
所述PostgreSQL块的存储容量是8KB的正整数倍。The storage capacity of the PostgreSQL block is a positive integer multiple of 8 KB.
所述PostgreSQL块的存储容量是8KB之2的自然数次方倍。The storage capacity of the PostgreSQL block is a natural multiple of 8 KB of 2 KB.
所述PostgreSQL块的存储容量是1MB、2MB、4MB、8MB、16MB、32MB、64MB、128MB、256MB、512MB或1024MB。 The storage capacity of the PostgreSQL block is 1 MB, 2 MB, 4 MB, 8 MB, 16 MB, 32 MB, 64 MB, 128 MB, 256 MB, 512 MB or 1024 MB.
一种基于PostgreSQL块存储设备的数据读写方法,利用如前面所述的PostgreSQL块存储设备读写模块对如前面所述的PostgreSQL块进行读写。A data read/write method based on a PostgreSQL block storage device, which reads and writes a PostgreSQL block as described above by using a PostgreSQL block storage device read/write module as described above.
A、在数据库后台驱动层中,PostgreSQL块存储设备读写模块的驱动完成对块文件的操作,其主要任务为:A. In the database background driver layer, the driver of the PostgreSQL block storage device read/write module completes the operation of the block file, and its main tasks are:
a、通过文件句柄管理器管理上层业务对底层设备的读写接口,实现提供高并发的读写接口;a, through the file handle manager to manage the upper and lower services read and write interfaces of the underlying device, to achieve a high concurrent read and write interface;
b、实现对PostgreSQL块存储设备的覆盖写,直接定位到PostgreSQL块存储设备的指定位置,并写入指定长度的数据;b. Implementing overwrite writing to the PostgreSQL block storage device, directly positioning to a specified location of the PostgreSQL block storage device, and writing data of a specified length;
B、根据所述PostgreSQL块-数据表之间的映射关系表oid、表空间定位到指定块设备的指定位置,通过表的pageid定位到块设备的某个偏移。B. According to the mapping relationship table oid between the PostgreSQL block and the data table, the table space is located to a specified position of the specified block device, and a certain offset of the block device is located by the pageid of the table.
PostgreSQL数据库系统中的每一个表分配有至少一个所述PostgreSQL块,而且可动态地增量分配PostgreSQL块;Each table in the PostgreSQL database system is assigned at least one of the PostgreSQL blocks, and the PostgreSQL block can be dynamically allocated incrementally;
每个PostgreSQL块可写入512个8KB的PostgreSQL块数据库系统的页,一个块设备可以包含无数多个表,一个表将可以包含无限多个PostgreSQL块,这些块文件信息被保存在PostgreSQL块-数据表之间的映射关系表内;Each PostgreSQL block can write to 512 8-KB PostgreSQL block database system pages. A block device can contain an infinite number of tables. A table can contain an unlimited number of PostgreSQL blocks. These block file information is stored in PostgreSQL block-data. The mapping relationship between the tables;
通过1个oid能获取到一个数据表的所有PostgreSQL块的信息,而这些PostgreSQL块信息会顺序地组成整个的所述PostgreSQL块-数据表的文件,这个表的pageid*8k就是这个表内部的偏移,同时也就能够确定是这个块文件的第几个块,至此获得我们想要获取的页在块设备内位置。The information of all the PostgreSQL blocks of a data table can be obtained by one oid, and the PostgreSQL block information will sequentially form the entire PostgreSQL block-data table file. The pageid*8k of this table is the internal bias of the table. Move, and at the same time be able to determine the first block of this block file, and thus get the position of the page we want to get in the block device.
本发明的PostgreSQL块设备读写存储模块与现有技术相比具有以下有益效果。The PostgreSQL block device read/write storage module of the present invention has the following advantageous effects compared with the prior art.
1)数据库的数据将会更为安全。数据库的数据将按照我们制定的规则保存到块设备内,从安全上来说更为安全。而且,由于使用分布式存储,将会使得数据将会分片保存,并保存多份到不同的数据节点内,就安全度而言是单磁盘百倍以上。1) The data in the database will be more secure. The data in the database will be saved to the block device according to the rules we have established, which is more secure in terms of security. Moreover, due to the use of distributed storage, the data will be saved in pieces and saved in multiple copies to different data nodes, which is more than 100 times more secure than a single disk.
2)查询速度、读写速度能够有极大提升,在相同的后端存储情况下能够达到10%~20%的速度提升。这个提升对于数据库而言是十分可观的。2) The query speed and read/write speed can be greatly improved, and the speed can be increased by 10% to 20% under the same back-end storage. This improvement is very impressive for the database.
3)数据库业务和数据分离。只要是同一个版本的数据库,就能够在不同的操作系统中自由迁移。只要能够通过设定在数据块头部的验证码,就 能够实现数据库的直接挂载。能够自由的从linux不同版本之间进行迁移,也能够和windows操作系统平台进行相互迁移。3) Database business and data separation. As long as it is the same version of the database, it can be freely migrated in different operating systems. As long as you can pass the verification code set at the head of the data block, Can achieve direct mount of the database. It is free to migrate from different versions of Linux, and can also migrate with the Windows operating system platform.
4)数据库在不同的系统之间进行迁移及其简单,对操作人员专业知识要求不高。4) The database is migrated between different systems and is simple, and the requirements of the operator's professional knowledge are not high.
附图说明DRAWINGS
下面结合附图和具体实施方式对本发明的PostgreSQL块、PostgreSQL块设备读写存储模块及基于PostgreSQL块存储设备的数据读写方法作进一步的详细描述。The PostgreSQL block, the PostgreSQL block device read and write storage module, and the data read and write method based on the PostgreSQL block storage device of the present invention are further described in detail below with reference to the accompanying drawings and specific embodiments.
图1是现有技术中操作系统的文件系统读写分层结构示意图。FIG. 1 is a schematic diagram of a file system read and write hierarchical structure of an operating system in the prior art.
图2是现有技术中操作系统的文件系统存储空间分配结构示意图。2 is a schematic diagram of a file system storage space allocation structure of an operating system in the prior art.
图3是现有技术中操作系统的文件系统磁头读写原理示意图。FIG. 3 is a schematic diagram of the principle of read and write of a file system head of an operating system in the prior art.
图4是本发明的PostgreSQL块、PostgreSQL块设备读写存储模块及基于PostgreSQL块存储设备的数据读写方法的结构、原理示意图。4 is a schematic diagram showing the structure and principle of a PostgreSQL block, a PostgreSQL block device read/write memory module, and a data read/write method based on a PostgreSQL block storage device.
具体实施方式detailed description
实施例一:Embodiment 1:
如图4所示,本发明的PostgreSQL块是分配给PostgreSQL数据库系统中数据表的最小单位,所述PostgreSQL块的存储容量等于4MB。As shown in FIG. 4, the PostgreSQL block of the present invention is the smallest unit allocated to the data table in the PostgreSQL database system, and the storage capacity of the PostgreSQL block is equal to 4 MB.
当然,也可以是,所述PostgreSQL块的存储容量大于4KB。Of course, it is also possible that the storage capacity of the PostgreSQL block is greater than 4 KB.
还可以是,所述PostgreSQL块的存储容量是8KB的正整数倍。It is also possible that the storage capacity of the PostgreSQL block is a positive integer multiple of 8 KB.
又可以是,所述PostgreSQL块的存储容量是8KB之2的自然数次方倍。Alternatively, the storage capacity of the PostgreSQL block is a natural multiple of 8 KB of 2 KB.
更可以是,所述PostgreSQL块的存储容量是1MB、2MB、8MB、16MB、32MB、64MB、128MB、256MB、512MB或1024MB。More preferably, the storage capacity of the PostgreSQL block is 1 MB, 2 MB, 8 MB, 16 MB, 32 MB, 64 MB, 128 MB, 256 MB, 512 MB or 1024 MB.
实施例二:Embodiment 2:
如图4所示,本发明的PostgreSQL块存储设备读写模块是对PostgreSQL块存储设备中如前面所述的PostgreSQL块进行管理的PostgreSQL块存储设备读写模块。As shown in FIG. 4, the PostgreSQL block storage device read/write module of the present invention is a PostgreSQL block storage device read/write module for managing a PostgreSQL block as described above in a PostgreSQL block storage device.
所述PostgreSQL块存储设备读写模块可以架构在PostgreSQL数据库系统上。The PostgreSQL block storage device read/write module can be architected on a PostgreSQL database system.
所述PostgreSQL块存储设备读写模块通过PostgreSQL块-数据表之间的映射关系表和空闲PostgreSQL块表对PostgreSQL块存储设备中的 PostgreSQL块进行管理。The PostgreSQL block storage device read and write module passes the PostgreSQL block-data table mapping table and the idle PostgreSQL block table to the PostgreSQL block storage device. PostgreSQL blocks are managed.
所述PostgreSQL块-数据表之间的映射关系表包括字段Relfilenode、Reltablespace、Forknum、Blockid、Blockno,所述空闲PostgreSQL块表包括字段Blockid、Isfree、Dev。The mapping relationship table between the PostgreSQL block and the data table includes fields Relfilenode, Reltablespace, Forknum, Blockid, Blockno, and the free PostgreSQL block table includes fields Blockid, Isfree, Dev.
所述PostgreSQL块存储设备读写模块,具有以下子模块:The PostgreSQL block storage device read and write module has the following submodules:
分配PostgreSQL块的子模块,Allocating submodules of the PostgreSQL block,
回收PostgreSQL块的子模块,Reclaim the submodules of the PostgreSQL block,
定位PostgreSQL块的子模块,Locate the submodule of the PostgreSQL block,
读出PostgreSQL块中数据的子模块,Read the submodule of the data in the PostgreSQL block,
写入PostgreSQL块中数据的子模块。A submodule that writes data in a PostgreSQL block.
所述分配PostgreSQL块的子模块采用就近分配策略或冷热数据分层分配策略给PostgreSQL数据表分配PostgreSQL块,所述就近分配策略是PostgreSQL块就近分配策略或空闲PostgreSQL块表记录就近分配策略,所述冷热数据表分层分配策略是常用数据表分配策略或近期使用数据表分配策略,所述PostgreSQL块就近分配策略是在数据表上次分配的PostgreSQL块前后就近寻找空闲PostgreSQL块分配给数据表,所述空闲PostgreSQL块表记录就近分配策略是从空闲PostgreSQL块表记录中寻找第一个空闲PostgreSQL块,所述常用数据表分配策略是经常使用的数据表优先分配到较快的PostgreSQL块设备上,所述近期使用数据表分配策略是将近期使用数据表优先分配到较快的PostgreSQL块设备上,所述冷热数据表分层分配策略用于具有两块以上不同读写速度的PostgreSQL块设备上;The sub-module that allocates the PostgreSQL block allocates a PostgreSQL block to the PostgreSQL data table by using a nearby allocation policy or a hot-hot data hierarchical allocation policy, and the nearby allocation strategy is a PostgreSQL block nearest allocation policy or an idle PostgreSQL block table recording nearest allocation policy. The hot-cold data table hierarchical allocation strategy is a common data table allocation strategy or a recently used data table allocation strategy. The PostgreSQL block nearest allocation strategy is to find an idle PostgreSQL block to be assigned to the data table before and after the last allocated PostgreSQL block of the data table. The free PostgreSQL block table record near allocation strategy is to find the first free PostgreSQL block from the idle PostgreSQL block table record, the commonly used data table allocation strategy is to preferentially allocate the data table to the faster PostgreSQL block device. The recently used data table allocation strategy is to prioritize the recently used data table to a faster PostgreSQL block device for a PostgreSQL block device having two or more different read and write speeds. on;
所述回收PostgreSQL块的子模块用于回收数据表不再使用的PostgreSQL块,从PostgreSQL块-数据表之间的映射关系表中删除对应的PostgreSQL块记录,到空闲表中将相应PostgreSQL块的记录设置为空闲;The submodule of the Recycled PostgreSQL block is used to reclaim the PostgreSQL block that is no longer used by the data table, and the corresponding PostgreSQL block record is deleted from the mapping relationship table between the PostgreSQL block and the data table, and the record of the corresponding PostgreSQL block is sent to the free table. Set to idle;
所述定位PostgreSQL块的子模块用于将数据表中的页定位到PostgreSQL块设备指定的位置上;The sub-module for locating the PostgreSQL block is used to locate a page in the data table to a location specified by the PostgreSQL block device;
所述读出PostgreSQL块中数据的子模块用于读取指定块内指定位置指定大小的数据;The submodule for reading data in the PostgreSQL block is used to read data of a specified size in a specified position in the specified block;
所述写入PostgreSQL块中数据的子模块用于写入指定块内指定位置指定大小的数据。 The sub-module that writes the data in the PostgreSQL block is used to write data of a specified size within a specified location within the specified block.
所述写入PostgreSQL块中数据的子模块以加密方式将数据写入PostgreSQL块中;相应地,The submodule that writes the data in the PostgreSQL block writes the data to the PostgreSQL block in an encrypted manner; accordingly,
所述读出PostgreSQL块中数据的子模块以解密的方式将数据读出返回给数据库业务层。The sub-module that reads the data in the PostgreSQL block reads the data back to the database business layer in a decrypted manner.
所述加密的方式是采用将用户密码与数据异或计算后进行存储加密的方式;The encryption method is a method of performing storage encryption on the user password and data XOR;
所述解密的方式是采用将用户密码与从PostgreSQL块中读出的数据异或计算后反回给数据库业务层解密的方式。The decryption method is a method of deciphering the user password and the data read from the PostgreSQL block and then deciphering it to the database service layer.
所述空闲PostgreSQL块表中的Dev字段对不同的存储设备进行标识实现扩容。The Dev field in the idle PostgreSQL block table identifies different storage devices to implement capacity expansion.
将所述PostgreSQL块-数据表之间的映射关系表和所述空闲PostgreSQL块表从原库复制到新库,将PostgreSQL块存储设备从原库所在的计算机迁移到新库所在的计算机。Copying the mapping relationship table between the PostgreSQL block-data table and the free PostgreSQL block table from the original library to the new library, and migrating the PostgreSQL block storage device from the computer where the original library is located to the computer where the new library is located.
实施例三:Embodiment 3:
如图4所示,本发明基于PostgreSQL块存储设备的数据读写方法,利用如前面所述的PostgreSQL块存储设备读写模块对如前面所述的PostgreSQL块进行读写。As shown in FIG. 4, the present invention is based on a data read/write method of a PostgreSQL block storage device, and reads and writes a PostgreSQL block as described above by using a PostgreSQL block storage device read/write module as described above.
A、在数据库后台驱动层中,PostgreSQL块存储设备读写模块的驱动完成对块文件的操作,其主要任务为:A. In the database background driver layer, the driver of the PostgreSQL block storage device read/write module completes the operation of the block file, and its main tasks are:
a、通过文件句柄管理器管理上层业务对底层设备的读写接口,实现提供高并发的读写接口;a, through the file handle manager to manage the upper and lower services read and write interfaces of the underlying device, to achieve a high concurrent read and write interface;
b、实现对PostgreSQL块存储设备的覆盖写,直接定位到PostgreSQL块存储设备的指定位置,并写入指定长度的数据;b. Implementing overwrite writing to the PostgreSQL block storage device, directly positioning to a specified location of the PostgreSQL block storage device, and writing data of a specified length;
B、根据所述PostgreSQL块-数据表之间的映射关系表oid、表空间定位到指定块设备的指定位置,通过表的pageid定位到块设备的某个偏移。B. According to the mapping relationship table oid between the PostgreSQL block and the data table, the table space is located to a specified position of the specified block device, and a certain offset of the block device is located by the pageid of the table.
PostgreSQL数据库系统中的每一个表分配有至少一个所述PostgreSQL块,而且可动态地增量分配PostgreSQL块;Each table in the PostgreSQL database system is assigned at least one of the PostgreSQL blocks, and the PostgreSQL block can be dynamically allocated incrementally;
每个PostgreSQL块可写入512个8KB的PostgreSQL块数据库系统的页,一个块设备可以包含无数多个表,一个表将可以包含无限多个PostgreSQL块,这些块文件信息被保存在PostgreSQL块-数据表之间的映射 关系表内;Each PostgreSQL block can write to 512 8-KB PostgreSQL block database system pages. A block device can contain an infinite number of tables. A table can contain an unlimited number of PostgreSQL blocks. These block file information is stored in PostgreSQL block-data. Mapping between tables Within the relationship table;
通过1个oid能获取到一个数据表的所有PostgreSQL块的信息,而这些PostgreSQL块信息会顺序地组成整个的所述PostgreSQL块-数据表的文件,这个表的pageid*8k就是这个表内部的偏移,同时也就能够确定是这个块文件的第几个块,至此获得我们想要获取的页在块设备内位置。The information of all the PostgreSQL blocks of a data table can be obtained by one oid, and the PostgreSQL block information will sequentially form the entire PostgreSQL block-data table file. The pageid*8k of this table is the internal bias of the table. Move, and at the same time be able to determine the first block of this block file, and thus get the position of the page we want to get in the block device.
如图4所示,详细设计如下:As shown in Figure 4, the detailed design is as follows:
创建PostgreSQL块-数据表之间的映射关系表和空闲PostgreSQL块表。Create a PostgreSQL block-to-data table mapping table and a free PostgreSQL block table.
表一:PostgreSQL块-数据表之间映射关系表Table 1: PostgreSQL block-data table mapping table
  字段Field 类型Types of 描述 description
11 Relfilenode(表编号)Relfilenode (table number) oid Oid 表IDTable ID
22 Reltablespace(表空间编号)Reltablespace (table space number) oidOid 表空间ID Table space ID
33 Forknum(文件类型)Forknum (file type) Int2Int2 表fork类型 Table fork type
44 Blockid(块编号)Blockid (block number) oidOid 表所占用的块IDThe block ID occupied by the table
55 Blockno(块位置)Blockno (block location) oidOid 被占用的块所在数据表中的位置The location in the data table where the occupied block is located
表二:空闲PostgreSQL块表Table 2: Idle PostgreSQL Block Table
  字段Field 类型Types of 描述 description
11 Blockid(块编号)Blockid (block number) oid Oid 块IDBlock ID
22 Isfree(占用标识)Isfree (occupation mark) boolBool 是否被使用Is it used?
33 Dev(设备标识)Dev (device identification) stringString 块设备标识Block device identification
A首先我们把PostgreSQL块存储设备在逻辑上以4MB为单位分割为N个PostgreSQL块。A First, we logically partition the PostgreSQL block storage device into N PostgreSQL blocks in units of 4MB.
一般情况下我们把PostgreSQL块存储设备按照4MB为单位进行分割。我们只需要通过PostgreSQL块存储设备的总容量除以一个PostgreSQL块的容量计算出PostgreSQL块存储设备的总块数即可,不需要对PostgreSQL块存储设备进行扫描,这个分配时间极短。比如2T的PostgreSQL块存储设备只需要分成524288个PostgreSQL块,我们只需要在空闲PostgreSQL块表中新增524288条记录,blockid从0--524287递增即可。同时标注isfree为true。 In general, we split the PostgreSQL block storage device by 4MB. We only need to calculate the total number of PostgreSQL block storage devices by dividing the total capacity of the PostgreSQL block storage device by the capacity of a PostgreSQL block. This does not require scanning the PostgreSQL block storage device. This allocation time is extremely short. For example, the 2T PostgreSQL block storage device only needs to be divided into 524288 PostgreSQL blocks. We only need to add 524288 records to the free PostgreSQL block table, and the blockid can be incremented from 0--524287. Also mark isfree as true.
B、数据表的存储空间以4MB为单位进行分配。B. The storage space of the data table is allocated in units of 4 MB.
按照PostgreSQL数据库系统的流程而言,一个表实际占用空间是按需分配的。我们修改PostgreSQL数据库系统后台驱动文件storage/smgr/smgr.c类型typedef struct f_smgr中需要的方法进行重写。According to the process of the PostgreSQL database system, the actual occupied space of a table is allocated on demand. We modify the PostgreSQL database system background driver file storage / smgr / smgr.c type typedef struct f_smgr required method to rewrite.
包括{include{
mdinit,Mdinit,
mdclose,Mdclose,
mdcreate,Mdcreate,
mdexists,Mdexists,
mdunlink,Mdunlink,
mdextend,Mdextend,
mdprefetch,Mdprefetch,
mdread,Mdread,
mdwrite,Mdwrite,
mdnblocks,Mdnblocks,
mdtruncate,Mdtruncate,
mdimmedsync,Mdimmedsync,
mdpreckpt,Mdpreckpt,
mdsync,Mdsync,
mdpostckptMdpostckpt
}}
这些方法的具体实现在md.c文件中。(此文件中的其他方法也要进行修改)The specific implementation of these methods is in the md.c file. (Other methods in this file should also be modified)
把各个方法中涉及到的MdfdVec进行修改。Modify the MdfdVec involved in each method.
修改的主要思想为:The main ideas of the revision are:
原本SMgrRelation reln,ForkNumber forknum两个参数定位到文件系统的表文件修改为定位到块设备的offset。我们能根据reln获取到relfilenode,reltablespace,forknum我们从PostgreSQL块-数据表之间的映射关系表中能够定位到我们的表所占用的数据。保证(pageno+1)*4*1024KB>reln.pageid*8KB>pageno*4*1024KB。 The original SMgrRelation reln, ForkNumber forknum two parameters are located to the file system's table file modified to locate the offset of the block device. We can get relfilenode, reltablespace, forknum according to reln. We can locate the data occupied by our table from the mapping table between PostgreSQL block and data tables. Guarantee (pageno+1)*4*1024KB>reln.pageid*8KB>pageno*4*1024KB.
假如PostgreSQL块-数据表之间的映射关系表中不存在,说明我们需要新增一个PostgreSQL块给数据表。这时我们从空闲PostgreSQL块表中直接搜到一个没被使用的块设备就可以直接使用。直接避免文件系统在处理大文件时候的繁杂分配过程。If the mapping between the PostgreSQL block and the data table does not exist, we need to add a PostgreSQL block to the data table. At this point, we can directly use a block device that is not used from the free PostgreSQL block table. Directly avoid the complicated allocation process of the file system when processing large files.
假如PostgreSQL块-数据表之间的映射关系表中存在,我们就能把PostgreSQL块存储设备直接定位到pageno*4*1024KB+(reln.pageid%512)*8KB。读出这个8K数据即可。If the PostgreSQL block-data table exists in the mapping table, we can directly locate the PostgreSQL block storage device to pageno*4*1024KB+(reln.pageid%512)*8KB. Read this 8K data.
PostgreSQL数据库业务到磁盘的中间过程及其简单。磁盘利用率极高。The intermediate process of PostgreSQL database business to disk and its simplicity. Disk utilization is extremely high.
C、PostgreSQL块存储设备的扩容。C. Expansion of the PostgreSQL block storage device.
当我们的PostgreSQL块存储设备容量不够的时候,我们的分布式存储能够极容易给其扩容。只需要在空闲PostgreSQL块表新增新条目即可。扩容的方式有两种:一种方式是原有的块设备扩容;另一种方式是新增块设备。对于前者我们更新PostgreSQL块-数据表之间的映射关系表,比如从2TB新增到4TB,我们只需要新插入524288条记录,524288-1048575递增即可。对于后者,比如新增2T块设备。我们新增524287条记录,blockid从0-524287递增,dev能够定位即可。When our PostgreSQL block storage device is not large enough, our distributed storage can be easily expanded. Just add a new entry to the free PostgreSQL block table. There are two ways to expand capacity: one is to expand the original block device; the other is to add a new block device. For the former we update the mapping table between PostgreSQL block and data tables, for example, from 2TB to 4TB, we only need to insert 524288 new records, 524288-1048575 increments. For the latter, for example, adding a 2T block device. We added 524,287 records, the blockid is incremented from 0-524287, and dev can be located.
D、数据库迁移。D, database migration.
数据库业务层跟数据层分割。把块设备直接挂载到新的节点上。这样上层的系统错误不会影响到底层数据。The database business layer is separated from the data layer. Mount the block device directly to the new node. This upper level system error does not affect the underlying data.
E、数据安全。E, data security.
PostgreSQL块存储设备选择规则可自定义,在B步骤中,我们是顺序的选择,我们也可以在我们的选择方法中添加一些规则(比如我们可以在其中插入一致性hash算法),打乱我们数据库的存储规则,保证数据安全。PostgreSQL block storage device selection rules can be customized. In step B, we are the choice of order. We can also add some rules to our selection method (for example, we can insert a consistent hash algorithm in it) to disrupt our database. Storage rules to ensure data security.
本发明的关键点如下。The key points of the present invention are as follows.
1)PostgreSQL数据库系统后台存储以裸盘为基础进行保存。1) PostgreSQL database system background storage is stored on a bare disk basis.
2)PostgreSQL数据库系统迁移的跨操作系统直接迁移,以及验证。2) Cross-OS direct migration of PostgreSQL database system migration, and verification.
3)PostgreSQL数据库系统对分布式文件系统(ceph)块设备的直接支持。3) Direct support for the distributed file system (ceph) block device by the PostgreSQL database system.
与现有技术相比,本发明的有益效果如下。The advantageous effects of the present invention are as follows in comparison with the prior art.
1)插入和查询速度更快。 1) Insert and query faster.
2)迁移更灵活。2) Migration is more flexible.
3)数据更为安全。3) The data is more secure.
4)PostgreSQL数据库系统业务和数据存储相分离。4) PostgreSQL database system business and data storage are separated.
需要说明的是,以上参照附图所描述的各个实施例仅用以说明本发明而非限制本发明的范围,本领域的普通技术人员应当理解,在不脱离本发明的精神和范围的前提下对本发明进行的修改或者等同替换,均应涵盖在本发明的范围之内。此外,除上下文另有所指外,以单数形式出现的词包括复数形式,反之亦然。另外,除非特别说明,那么任何实施例的全部或一部分可结合任何其它实施例的全部或一部分来使用。 It should be noted that the various embodiments described above with reference to the accompanying drawings are only to illustrate the invention and not to limit the scope of the invention, and those of ordinary skill in the art should understand that without departing from the spirit and scope of the invention Modifications or equivalents to the invention are intended to be included within the scope of the invention. In addition, unless the context indicates otherwise, words in the singular include plural and vice versa. In addition, all or a portion of any embodiment can be used in combination with all or a portion of any other embodiment, unless otherwise stated.

Claims (10)

  1. 一种PostgreSQL块存储设备读写模块,其特征在于:所述PostgreSQL块存储设备读写模块是对PostgreSQL块存储设备中的PostgreSQL块进行管理的PostgreSQL块存储设备读写模块,所述PostgreSQL块存储设备读写模块架构在PostgreSQL数据库系统上。A PostgreSQL block storage device read/write module, characterized in that: the PostgreSQL block storage device read/write module is a PostgreSQL block storage device read/write module that manages a PostgreSQL block in a PostgreSQL block storage device, and the PostgreSQL block storage device The read/write module architecture is on the PostgreSQL database system.
  2. 根据权利要求1所述的PostgreSQL块存储设备读写模块,其特征在于:所述PostgreSQL块存储设备读写模块通过PostgreSQL块-数据表之间的映射关系表和空闲PostgreSQL块表对PostgreSQL块存储设备中的PostgreSQL块进行管理。The PostgreSQL block storage device read/write module according to claim 1, wherein: said PostgreSQL block storage device read/write module passes a PostgreSQL block-data table mapping table and an idle PostgreSQL block table to a PostgreSQL block storage device. The PostgreSQL block is managed.
  3. 根据权利要求2所述的PostgreSQL块存储设备读写模块,其特征在于:所述PostgreSQL块-数据表之间的映射关系表包括字段Relfilenode、Reltablespace、Forknum、Blockid、Blockno,所述空闲PostgreSQL块表包括字段Blockid、Isfree、Dev。The PostgreSQL block storage device read/write module according to claim 2, wherein the mapping relationship table between the PostgreSQL block and the data table includes fields Relfilenode, Reltablespace, Forknum, Blockid, Blockno, and the free PostgreSQL block table. Includes fields Blockid, Isfree, Dev.
  4. 根据权利要求3所述的PostgreSQL块存储设备读写模块,其特征在于:所述PostgreSQL块存储设备读写模块,具有以下子模块:The PostgreSQL block storage device read/write module according to claim 3, wherein the PostgreSQL block storage device read/write module has the following submodules:
    分配PostgreSQL块的子模块,Allocating submodules of the PostgreSQL block,
    回收PostgreSQL块的子模块,Reclaim the submodules of the PostgreSQL block,
    定位PostgreSQL块的子模块,Locate the submodule of the PostgreSQL block,
    读出PostgreSQL块中数据的子模块,Read the submodule of the data in the PostgreSQL block,
    写入PostgreSQL块中数据的子模块。A submodule that writes data in a PostgreSQL block.
  5. 根据权利要求4所述的PostgreSQL块存储设备读写模块,其特征在于:The PostgreSQL block storage device read/write module according to claim 4, wherein:
    所述分配PostgreSQL块的子模块采用就近分配策略或冷热数据分层分配策略给PostgreSQL数据表分配PostgreSQL块,所述就近分配策略是PostgreSQL块就近分配策略或空闲PostgreSQL块表记录就近分配策略,所述冷热数据表分层分配策略是常用数据表分配策略或近期使用数据表分配策略,所述PostgreSQL块就近分配策略是在数据表上次分配的PostgreSQL块前后就近寻找空闲PostgreSQL块分配给数据表,所述空闲PostgreSQL块表记录就近分配策略是从空闲PostgreSQL块表记录中寻找第一个空闲 PostgreSQL块,所述常用数据表分配策略是经常使用的数据表优先分配到较快的PostgreSQL块设备上,所述近期使用数据表分配策略是将近期使用数据表优先分配到较快的PostgreSQL块设备上,所述冷热数据表分层分配策略用于具有两块以上不同读写速度的PostgreSQL块设备上;The sub-module that allocates the PostgreSQL block allocates a PostgreSQL block to the PostgreSQL data table by using a nearby allocation policy or a hot-hot data hierarchical allocation policy, and the nearby allocation strategy is a PostgreSQL block nearest allocation policy or an idle PostgreSQL block table recording nearest allocation policy. The hot-cold data table hierarchical allocation strategy is a common data table allocation strategy or a recently used data table allocation strategy. The PostgreSQL block nearest allocation strategy is to find an idle PostgreSQL block to be assigned to the data table before and after the last allocated PostgreSQL block of the data table. The free PostgreSQL block table record near allocation strategy is to find the first idle from the idle PostgreSQL block table record PostgreSQL block, the common data table allocation strategy is that the frequently used data table is preferentially allocated to the faster PostgreSQL block device, and the recently used data table allocation strategy is to prioritize the recently used data table to the faster PostgreSQL block device. The hot-cold data table hierarchical allocation strategy is used on a PostgreSQL block device having two or more different read/write speeds;
    所述回收PostgreSQL块的子模块用于回收数据表不再使用的PostgreSQL块,从PostgreSQL块-数据表之间的映射关系表中删除对应的PostgreSQL块记录,到空闲表中将相应PostgreSQL块的记录设置为空闲;The submodule of the Recycled PostgreSQL block is used to reclaim the PostgreSQL block that is no longer used by the data table, and the corresponding PostgreSQL block record is deleted from the mapping relationship table between the PostgreSQL block and the data table, and the record of the corresponding PostgreSQL block is sent to the free table. Set to idle;
    所述定位PostgreSQL块的子模块用于将数据表中的页定位到PostgreSQL块设备指定的位置上;The sub-module for locating the PostgreSQL block is used to locate a page in the data table to a location specified by the PostgreSQL block device;
    所述读出PostgreSQL块中数据的子模块用于读取指定块内指定位置指定大小的数据;The submodule for reading data in the PostgreSQL block is used to read data of a specified size in a specified position in the specified block;
    所述写入PostgreSQL块中数据的子模块用于写入指定块内指定位置指定大小的数据。The sub-module that writes the data in the PostgreSQL block is used to write data of a specified size within a specified location within the specified block.
  6. 根据权利要求5所述的PostgreSQL块存储设备读写模块,其特征在于:The PostgreSQL block storage device read/write module according to claim 5, wherein:
    所述写入PostgreSQL块中数据的子模块以加密方式将数据写入PostgreSQL块中;相应地,The submodule that writes the data in the PostgreSQL block writes the data to the PostgreSQL block in an encrypted manner; accordingly,
    所述读出PostgreSQL块中数据的子模块以解密的方式将数据读出返回给数据库业务层。The sub-module that reads the data in the PostgreSQL block reads the data back to the database business layer in a decrypted manner.
  7. 根据权利要求6所述的PostgreSQL块存储设备读写模块,其特征在于:The PostgreSQL block storage device read/write module according to claim 6, wherein:
    所述加密的方式是采用将用户密码与数据异或计算后进行存储加密的方式;The encryption method is a method of performing storage encryption on the user password and data XOR;
    所述解密的方式是采用将用户密码与从PostgreSQL块中读出的数据异或计算后反回给数据库业务层加密的方式。The decryption method is a method of encrypting the user password with the data read from the PostgreSQL block and then returning it to the database service layer.
  8. 根据权利要求7所述的PostgreSQL块存储设备读写模块,其特征在于:所述空闲PostgreSQL块表中的Dev字段对不同的存储设备进行标识实现扩容。The PostgreSQL block storage device read/write module according to claim 7, wherein the Dev field in the free PostgreSQL block table identifies different storage devices to implement capacity expansion.
  9. 根据权利要求7所述的PostgreSQL块存储设备读写模块,其特征在于:将所述PostgreSQL块-数据表之间的映射关系表和所述空闲PostgreSQL 块表从原库复制到新库,将PostgreSQL块存储设备从原库所在的计算机迁移到新库所在的计算机。The PostgreSQL block storage device read/write module according to claim 7, wherein the mapping relationship table between the PostgreSQL block and the data table and the idle PostgreSQL The block table is copied from the original library to the new library, and the PostgreSQL block storage device is migrated from the computer where the original library is located to the computer where the new library is located.
  10. 根据权利要求1至9之一所述的PostgreSQL块存储设备读写模块,其特征在于:所述PostgreSQL块是分配给PostgreSQL数据库系统中数据表的最小单位,其特征在于:所述PostgreSQL块的存储容量大于4KB,所述PostgreSQL块存储设备读写模块通过基于PostgreSQL块存储设备的数据读写方法对所述PostgreSQL块进行读写操作。 The PostgreSQL block storage device read/write module according to any one of claims 1 to 9, wherein the PostgreSQL block is a minimum unit allocated to a data table in a PostgreSQL database system, and is characterized in that: the PostgreSQL block is stored. The capacity is greater than 4 KB, and the PostgreSQL block storage device read/write module reads and writes the PostgreSQL block through a data read/write method based on the PostgreSQL block storage device.
PCT/CN2016/095367 2015-12-17 2016-08-15 Postgresql block storage device read-write module WO2017101478A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510955631.3 2015-12-17
CN201510955631.3A CN105630879B (en) 2015-12-17 2015-12-17 A kind of PostgreSQL block storage equipment module for reading and writing

Publications (1)

Publication Number Publication Date
WO2017101478A1 true WO2017101478A1 (en) 2017-06-22

Family

ID=56045812

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/095367 WO2017101478A1 (en) 2015-12-17 2016-08-15 Postgresql block storage device read-write module

Country Status (2)

Country Link
CN (1) CN105630879B (en)
WO (1) WO2017101478A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116910310A (en) * 2023-06-16 2023-10-20 广东电网有限责任公司佛山供电局 Unstructured data storage method and device based on distributed database

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105630879B (en) * 2015-12-17 2019-03-26 深圳市华讯方舟软件技术有限公司 A kind of PostgreSQL block storage equipment module for reading and writing
CN105824879B (en) * 2015-12-17 2019-06-28 深圳市华讯方舟软件技术有限公司 A kind of moving method based on PostgreSQL block storage equipment
CN105573678B (en) * 2015-12-17 2018-11-09 深圳市华讯方舟软件技术有限公司 A kind of PostgreSQL blocks
CN109460331B (en) * 2018-10-24 2021-10-29 郑州云海信息技术有限公司 Clone characteristic testing method, device, equipment and storage medium
CN112380217B (en) * 2020-11-17 2024-04-12 安徽鸿程光电有限公司 Data processing method, device, equipment and medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101009645A (en) * 2006-12-22 2007-08-01 清华大学 Stream queue-based extensible device for CAM-based broadband network service stream
WO2010005447A1 (en) * 2008-07-07 2010-01-14 Kickfire, Inc. Methods and systems for generating query plans that are compatible for execution in hardware
CN102243629A (en) * 2010-05-12 2011-11-16 北京安华金和科技有限公司 Transparent encryption and decryption method for database based on multi-level view and trigger
CN102768672A (en) * 2012-06-12 2012-11-07 上海方正数字出版技术有限公司 Disk space management method and device
CN105573678A (en) * 2015-12-17 2016-05-11 深圳市华讯方舟软件技术有限公司 PostgreSQL block
CN105630879A (en) * 2015-12-17 2016-06-01 深圳市华讯方舟软件技术有限公司 Read-write module of PostgreSQL block storage device
CN105824879A (en) * 2015-12-17 2016-08-03 深圳市华讯方舟软件技术有限公司 Migration method based on PostgreSQL block storage equipment

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100703753B1 (en) * 2005-04-14 2007-04-06 삼성전자주식회사 Apparatus and method for managing file system
CN1936864B (en) * 2005-09-22 2010-09-08 康佳集团股份有限公司 Non-fixed-length record data recognition method
US7546307B2 (en) * 2006-09-28 2009-06-09 Nvidia Corporation Virtual block storage to filesystem translator
CN104462147A (en) * 2013-09-25 2015-03-25 天津书生投资有限公司 Storage method for document

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101009645A (en) * 2006-12-22 2007-08-01 清华大学 Stream queue-based extensible device for CAM-based broadband network service stream
WO2010005447A1 (en) * 2008-07-07 2010-01-14 Kickfire, Inc. Methods and systems for generating query plans that are compatible for execution in hardware
CN102243629A (en) * 2010-05-12 2011-11-16 北京安华金和科技有限公司 Transparent encryption and decryption method for database based on multi-level view and trigger
CN102768672A (en) * 2012-06-12 2012-11-07 上海方正数字出版技术有限公司 Disk space management method and device
CN105573678A (en) * 2015-12-17 2016-05-11 深圳市华讯方舟软件技术有限公司 PostgreSQL block
CN105630879A (en) * 2015-12-17 2016-06-01 深圳市华讯方舟软件技术有限公司 Read-write module of PostgreSQL block storage device
CN105824879A (en) * 2015-12-17 2016-08-03 深圳市华讯方舟软件技术有限公司 Migration method based on PostgreSQL block storage equipment

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116910310A (en) * 2023-06-16 2023-10-20 广东电网有限责任公司佛山供电局 Unstructured data storage method and device based on distributed database
CN116910310B (en) * 2023-06-16 2024-02-13 广东电网有限责任公司佛山供电局 Unstructured data storage method and device based on distributed database

Also Published As

Publication number Publication date
CN105630879A (en) 2016-06-01
CN105630879B (en) 2019-03-26

Similar Documents

Publication Publication Date Title
WO2017101505A1 (en) Migration method based on postgresql block storage device
WO2017101478A1 (en) Postgresql block storage device read-write module
Ramakrishnan et al. Azure data lake store: a hyperscale distributed file service for big data analytics
KR102147905B1 (en) Address based multi-stream storage device access
US11287994B2 (en) Native key-value storage enabled distributed storage system
US9047301B2 (en) Method for optimizing the memory usage and performance of data deduplication storage systems
JP5876592B2 (en) File storage system and file cloning method
KR20190095089A (en) System and method for leveraging key-value storage to efficiently store data and metadata in a distributed file system
CN106663047A (en) Systems and methods for oprtimized signature comparisons and data replication
US11301421B2 (en) Scalable multi-tier storage structures and techniques for accessing entries therein
CN103597440A (en) Method for creating clone file, and file system adopting the same
WO2017101477A1 (en) Postgresql block
US11775476B2 (en) Techniques for snapshotting scalable multitier storage structures
CN102012852A (en) Method for implementing incremental snapshots-on-write
CN111324305B (en) Data writing/reading method in distributed storage system
CN113535670B (en) Virtual resource mirror image storage system and implementation method thereof
US8140886B2 (en) Apparatus, system, and method for virtual storage access method volume data set recovery
US20150058554A1 (en) Systems, Methods, and Computer Program Products Implementing Hybrid File Structures for Data Storage
US7549029B2 (en) Methods for creating hierarchical copies
JP2024525170A (en) Data compression method and device
US8918621B1 (en) Block address isolation for file systems
WO2023235040A1 (en) File system improvements for zoned storage device operations
US20230334011A1 (en) Snapshot mapping structure characterizing data dependencies between a storage volume and its snapshots
US6910214B1 (en) Method, system, and program for converting an input parameter list into an output parameter list
US20140344538A1 (en) Systems, methods, and computer program products for determining block characteristics in a computer data storage system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16874538

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16874538

Country of ref document: EP

Kind code of ref document: A1