WO2022037015A1 - 一种基于持久性内存的列式存储方法、装置及设备 - Google Patents

一种基于持久性内存的列式存储方法、装置及设备 Download PDF

Info

Publication number
WO2022037015A1
WO2022037015A1 PCT/CN2021/073472 CN2021073472W WO2022037015A1 WO 2022037015 A1 WO2022037015 A1 WO 2022037015A1 CN 2021073472 W CN2021073472 W CN 2021073472W WO 2022037015 A1 WO2022037015 A1 WO 2022037015A1
Authority
WO
WIPO (PCT)
Prior art keywords
persistent memory
column block
data
column
structure data
Prior art date
Application number
PCT/CN2021/073472
Other languages
English (en)
French (fr)
Inventor
秦朝阳
Original Assignee
苏州浪潮智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 苏州浪潮智能科技有限公司 filed Critical 苏州浪潮智能科技有限公司
Publication of WO2022037015A1 publication Critical patent/WO2022037015A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2228—Indexing structures
    • G06F16/2237—Vectors, bitmaps or matrices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices
    • G06F16/137—Hash-based
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/172—Caching, prefetching or hoarding of files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2228—Indexing structures
    • G06F16/2255—Hash tables
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof

Definitions

  • the present application relates to the field of storage technologies, and in particular, to a persistent memory-based columnar storage method, apparatus, device, and readable storage medium.
  • OLTP online transaction processing
  • OLAP online analytical processing
  • the traditional relational database is mainly oriented to OLTP, which is used to perform some basic and real-time transaction processing, such as the addition, deletion, modification, and query of database records; while the distributed data processing system is mainly oriented to OLAP, which requires more real-time performance.
  • Column-based storage is relative to row-based storage.
  • HBase, ORC, Parquet, etc. all use column-based storage.
  • data is stored in a column-based logical storage unit, and the data in a column exists in a continuous storage form in a storage medium.
  • columnar storage is designed based on the common disk file system. Reading and writing data needs to go through the file system I/O stack. Therefore, the data access speed is slow and cannot meet the increasingly high timeliness requirements. Moreover, columnar storage is not suitable for scenarios that require frequent data updates, and is not suitable for real-time operations such as deletion and update.
  • the purpose of this application is to provide a persistent memory-based columnar storage method, device, device, and readable storage medium to solve the problem that the current columnar storage solution has slow data access and is not suitable for frequent deletion, update, etc.
  • the problem of real-time operation is as follows:
  • the present application provides a persistent memory-based columnar storage method, including:
  • the metadata of the formatted file is generated according to the file division, wherein the metadata includes identification information of each line group in the formatted file, position information of each line group in the formatted file, and also includes all line groups in the formatted file. Identification information of each column block in the row group, position information of each column block in the row group;
  • the bitmap index data of the column block is generated according to whether each element in the actual data of the column block is empty.
  • the converting the column block into hash structure data, and writing the hash structure data into the persistent memory includes:
  • the writing of the hash structure data into the persistent memory includes:
  • the value in the hash structure data is encoded and compressed, and the encoded and compressed hash structure data is written into the persistent memory.
  • the writing of the hash structure data into the persistent memory includes:
  • the hash structure data is written into the persistent memory by using the operation function in the persistent memory development kit.
  • the converting the column block into hash structure data, and writing the hash structure data into the persistent memory includes:
  • generating the metadata of the formatted file according to the file division includes:
  • the metadata of the formatted file is generated according to the file division, wherein the metadata further includes a null value flag bit of the column block, and the null value flag bit is used to indicate that the values of the column block are all null.
  • the method further includes:
  • the metadata is stored in a two-dimensional-like array.
  • the present application provides a persistent memory-based columnar storage device, including:
  • Rowgroup keying module including:
  • File division unit used to divide the formatted file to be stored, and each preset number of lines is divided into a line group; the line group is divided, and each column is used as a column block;
  • Hash unit used to convert the column block into hash structure data, and write the hash structure data into persistent memory
  • Metadata module used to generate the metadata of the formatted file according to the file division situation, wherein the metadata includes the identification information of each line group in the formatted file, and the identification information of each line group in the formatted file.
  • Position information also includes identification information of each column block in the row group, and position information of each column block in the row group;
  • Bitmap index module used to generate bitmap index data of the column block according to whether each element in the actual data of the column block is empty.
  • the hash unit is specifically used for:
  • the hash unit is specifically used for:
  • the value in the hash structure data is encoded and compressed, and the encoded and compressed hash structure data is written into the persistent memory.
  • the hash unit is specifically used for:
  • the hash structure data is written into the persistent memory by using the operation function in the persistent memory development kit.
  • the hash unit is specifically used for:
  • the metadata module is specifically used for:
  • the metadata of the formatted file is generated according to the file division, wherein the metadata further includes a null value flag bit of the column block, and the null value flag bit is used to indicate that the values of the column block are all null.
  • it also includes:
  • Metadata storage module used to store the metadata in a class two-dimensional array.
  • the present application provides a persistent memory-based columnar storage device, including:
  • Memory used to store computer programs
  • Processor steps for executing the computer program to implement the above persistent memory-based columnar storage method.
  • the present application provides a readable storage medium, where a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, is used to implement the above persistent memory-based columnar format Store the steps of the method.
  • a persistent memory-based columnar storage method includes: dividing a formatted file to be stored, and dividing each preset number of rows into a row group; dividing the row group, and using each column as a row group Column block; convert the column block into hash structure data, and write the hash structure data into persistent memory; generate the metadata of the formatted file according to the file division, wherein the metadata includes the identification of each row group in the formatted file information, the position information of each row group in the format file, and also includes the identification information of each column block in the row group, and the position information of each column block in the row group; according to whether each element in the actual data of the column block is empty, Generates bitmap index data for column blocks.
  • this method converts the file into hash structure data and stores it in the persistent memory. Since the persistent memory is byte-addressable and non-volatile, it can avoid going through the file system stack during the data reading and writing process, greatly reducing the size of the data. Compared with traditional columnar storage, this storage method is easier to realize real-time data operations such as modification and deletion, which expands the application scenarios of columnar storage; in addition, this method adopts a simple logic and takes up space.
  • the extremely small bitmap index data describes the data distribution on the column block, avoids the complexity of record shredding and assembly algorithms, simplifies the storage logic while retaining the advantages of columnar storage, and further improves the data read and write performance of columnar storage.
  • the present application also provides a persistent memory-based columnar storage device, device, and readable storage medium, the technical effects of which correspond to those of the above method, and are not repeated here.
  • FIG. 1 is an implementation flowchart of Embodiment 1 of a persistent memory-based columnar storage method provided by the present application
  • FIG. 2 is an implementation flowchart of Embodiment 2 of a persistent memory-based columnar storage method provided by the present application
  • FIG. 3 is a schematic diagram of a process in practical application of an embodiment of a persistent memory-based columnar storage method provided by the present application;
  • FIG. 4 is a schematic diagram of a row group keying process in Embodiment 2 of a persistent memory-based columnar storage method provided by the present application;
  • FIG. 5 is a schematic diagram of a process of generating bitmap index data in Embodiment 2 of a persistent memory-based columnar storage method provided by the present application;
  • FIG. 6 is a functional block diagram of an embodiment of a persistent memory-based columnar storage device provided by the present application.
  • FIG. 7 is a schematic structural diagram of an embodiment of a persistent memory-based columnar storage device provided by the present application.
  • the core of this application is to provide a persistent memory-based columnar storage method, device, device, and readable storage medium, which can divide files into row groups, convert row groups into hash structure data, and write persistent memory.
  • memory it can avoid going through the file system stack during data reading and writing, greatly improving the speed of data query, and making it easier to implement real-time data operations such as modification and deletion.
  • Bitmap index data is used to describe the data distribution on column blocks. The logic is simple and takes up very little space, which simplifies the storage logic and further improves the data read and write performance of the columnar storage.
  • Embodiment 1 of a persistent memory-based columnar storage method provided by the present application will be introduced below. Referring to FIG. 1 , Embodiment 1 includes:
  • the above formatted file refers to a file in which each line represents a piece of data, each piece of data includes multiple attributes, and the attributes of each column of data are the same.
  • the above-mentioned preset number can be set by itself according to actual needs, which is not limited in this embodiment.
  • HashMap is the most frequently used data type for mapping (key-value pair) processing in Java, and has a fast access speed.
  • S104 Generate bitmap index data of the column block according to whether each element in the actual data of the column block is empty.
  • Persistent Memory is a type of non-volatile memory, which has characteristics similar to memory and characteristics similar to storage. It can be addressed by bytes like memory, and its stored content is non-volatile. of. Data located on persistent memory can be accessed directly from user space, with no kernel code, filesystem page caches, or interrupts on the access data path.
  • this embodiment Based on the physical characteristics of persistent memory devices, this embodiment provides a columnar storage solution with a key-value structure, which can greatly improve the data query speed, meet the needs of instant query, and better support data modification and deletion. Expanded the applicable scenarios of columnar storage.
  • This embodiment mainly includes a row group keying stage, a metadata generation stage and a bitmap index generation stage.
  • the row group keying stage this embodiment divides the file into row groups, and converts the column blocks in the row group into the key-value hash structure of the persistent memory medium, so as to make full use of the efficient read and write advantages of the persistent memory ;
  • the metadata generation stage generate metadata for recording which row groups the file is divided into, which column blocks each row group includes, the position information of the row group in the file, and the position information of the column block in the row group;
  • the bitmap index generation stage generates bitmap index data for recording whether each element of the actual data of the column block is empty.
  • this embodiment divides the formatted file, and each preset number of rows is used as a row group; each row group contains data of multiple columns, and the row group is further divided, and each column is used as a row group.
  • a column block is converted into hash-structured data, and the hash-structured data is written into persistent memory.
  • the column name and row group serial number of the column block can be used as the key, and the actual data of the column block can be used as the value, so as to convert the column block into hash structure data, and write the hash structure data into the persistence in memory. It can be understood that before writing the hash structure data into the persistent memory, the values in the hash structure data need to be encoded. In order to further reduce the space occupation, the values in the hash structure data can be encoded after encoding. Compression, which encoding method or compression method is specifically adopted, is not limited in this embodiment.
  • the Concurrent Hashmap unique to persistent memory can be selected.
  • the column block into hash structure data specifically convert the column block into concurrent hash structure data, and write the concurrent hash structure data into the persistent memory.
  • a function closer to the underlying layer can be used to implement the writing operation.
  • an operation function in a persistent memory development kit such as libpmemkv (a sub-function library of PMDK)
  • libpmemkv a sub-function library of PMDK
  • the metadata generation stage it is mainly used to generate the metadata of the formatted file, wherein the metadata includes but is not limited to which row groups are included in the formatted file, the position or order of each row group in the formatted file, and which columns are included in the row group Block, the position or order of each column block in the row group; in addition, it can also include the encoding method of the value of the column block, the value compression method of the column block, and some data statistics, such as the maximum and minimum values in the column block value etc.
  • the metadata may further include a null value flag bit used to indicate whether the values of the column block are all null. Therefore, in the process of data read and write, whether the column to be read or written is empty is directly determined according to the null value flag bit, and the data read and write performance is improved.
  • the metadata of the column block in this embodiment needs to store both the element name and the position of the element, in practical applications, the metadata of the column block can be stored by using two-dimensional data.
  • this embodiment generates corresponding bitmap index data for each column block, and the bitmap index data mainly indicates the presence or absence of actual data in the corresponding column block. Preferably, if a certain column block has no actual data, the value of the column may not be stored during column storage, which will save more space than row storage.
  • a bitmap is a data structure that stores specific data through an array of bits, and each bit can independently contain information.
  • each bit in the bitmap index data is used to indicate whether the corresponding element in the column block is empty. Since a bit is the smallest storage unit of data, a large amount of space can be saved.
  • This embodiment provides a persistent memory-based columnar storage method, which converts files into hash structure data and stores them in the persistent memory. Since the persistent memory is byte-addressable and non-volatile, it can In the process of data reading and writing, it avoids going through the file system stack, which greatly improves the speed of data query. Compared with traditional columnar storage, this storage method is easier to implement real-time data operations such as modification and deletion, which expands the application scenarios of columnar storage. , this method uses a logically simple bitmap index data to describe the data distribution on column blocks, avoids the complexity of record shredding and assembly algorithms, and simplifies storage while retaining the advantages of columnar storage. logic to further improve the data read and write performance of columnar storage.
  • the second embodiment of a persistent memory-based columnar storage method provided by the present application will be described in detail below.
  • the second embodiment is implemented based on the foregoing first embodiment, and is expanded to a certain extent on the basis of the first embodiment.
  • this embodiment limits the key and value of the keyed column block, adopts a concurrent hash structure to further improve data read and write performance, and encodes and compresses the value of the column block to save storage space.
  • Use the underlying operation functions to perform write operations to improve write efficiency. Metadata is stored in two-dimensional arrays, which is more efficient.
  • the second embodiment specifically includes:
  • the metadata includes the name of each row group in the formatted file, the position information of each row group in the formatted file, the name of each column block in the row group, the position of each column block in the row.
  • the position information in the group also includes the null value flag bit of the column block, the null value flag bit is used to indicate that the value of the column block is all empty; also includes the compression method and encoding method of the value of the column block; It also includes the maximum value and the minimum value among the values of the column block, and the like.
  • This embodiment provides a columnar storage method based on persistent memory. Based on the byte addressing characteristics of persistent memory, a columnar storage solution with a key-value structure is provided, which mainly includes the row group keying stage, Metadata generation stage and bitmap index generation stage.
  • the row group keying phase converts each column block of the row group into a key-value hash structure of the persistent memory medium to take full advantage of the efficient read and write advantages of persistent memory;
  • the metadata generation phase records what format files are split into The key value, the attributes, location and other information of each key value;
  • the bitmap index generation stage generates a bitmap index structure used to record whether the consecutive values in the column block are empty, and the column data is stored according to the index when accessing a row of data. Organized into row-style data.
  • Table 1 Suppose, there is currently a format file to be stored as shown in Table 1, and a row in Table 1 represents a piece of data:
  • the format file is horizontally divided into row groups of a certain length. If the specific length is 1024, each 1024 rows of data is a row group, each row group contains several columns, and each column contains a maximum of 1024 elements. As shown in Figure 4, it includes the following sub-steps:
  • Sub-step 1 In the data shown in Table 1, the first row group is the data from ID1 to ID1024, the second row group is the data from ID 1025 to ID 2048, and so on. If there are N rows of data in total, there are N/1024 row groups (rounded up).
  • Sub-step 2 In the data shown in Table 1, the first row group is divided into 8 column blocks, and the column name is used as the key.
  • the key of the key-value storage can be designed as The column name is added with the row group serial number, and the value is designed to be the actual data in the column block.
  • the key of the first column block of the first row group is "ID_1", and its value is [1,2,3,4,5,6,7,8]; the second row of the first row group
  • the key of the column block is "name_1”, and its value is [Xiaoming, Xiaoli, Xiaohong, Xiaohua, Xiaozhang, Xiaohe, Xiaohuang, Xiaosong]; the 3rd column block of the 1st row group
  • the key is "gender_1", and its value is [male, female, female, female, male];
  • the key of the 4th column block of the 1st row group is "birth_1", and its value is empty, followed by By analogy, all the values in the row group are converted. It can be seen that the values of each column block are elements of the same type, so efficient data compression can be performed.
  • Sub-step 3 Use the operation functions in the persistent memory development kit to read and write data, and write the above key-valued data into the persistent memory through the storage method in libpmemkv (a sub-function library of the persistent memory development kit). .
  • the recorded metadata information mainly includes which row groups are included in the format file, which column blocks are included in each row group, what is the key of the column block, and the value of each column block What method is used to compress and encode, and the statistics of column block values, such as maximum and minimum, etc.
  • Metadata information can be stored in a matrix using a two-dimensional array.
  • the metadata of the format file shown in Table 1 can be expressed as follows:
  • “formatted file” records the data shown in Table 1, which is divided according to the length of 1024, file attributes, etc.;
  • row group 1 represents 1024 rows of data in the first row group, including “ID_1", “Name_1” and other 8 column name blocks;
  • ID_1", “ID_2” to “ID_n” represent all the data in the ID column, which can be filtered by column without scanning irrelevant columns; such as the row group of example data 1
  • the column block values of "birth” of 1 are all empty, then "birth_1" should be marked with a null value. When scanning and reading the data of the "birth” column, row group 1 is skipped, and so on, to reduce the query purpose of time.
  • bitmap index file is created for each column block value.
  • the number of bits occupied by the bitmap index is equal to the division length of the row group, such as 1024 bits.
  • 8 rows of data in Table 1 are used for illustration, that is, it is assumed that a bitmap index has only 8 bits.
  • the bitmap index is: 11111111; another example is the "gender_1" column block value, it has several empty elements, and its bitmap index is: 10100111; another example is the column block value of "occupation_1", the first 4 elements of which exist, the bitmap index is: 11110000; if a column block value is completely empty, such as "birth_1", no matter what the length of the row group is , whose bitmap indices are all 8-bit 0:00000000.
  • bitmap index occupies very little space.
  • a 1M bitmap index file can represent more than 8 million rows of data. If the column block value is empty, the space occupied will be less.
  • the locations where the index file can be stored include but are not limited to:
  • Position 1 is directly bound to the key, that is, the key is a valid combination of column name, row group serial number, and bitmap index;
  • the second position, stored in the metadata, is stored with the metadata of each column block.
  • the following describes a persistent memory-based columnar storage device provided by the embodiments of the present application, the persistent memory-based columnar storage device described below and the persistent memory-based columnar storage described above
  • the methods can refer to each other correspondingly.
  • the persistent memory-based columnar storage device of this embodiment includes:
  • the row group keying module 601 includes:
  • File division unit used to divide the formatted file to be stored, and each preset number of lines is divided into a line group; the line group is divided, and each column is used as a column block;
  • Hash unit used to convert the column block into hash structure data, and write the hash structure data into persistent memory
  • Metadata module 602 used to generate metadata of the formatted file according to file division, wherein the metadata includes identification information of each line group in the formatted file, and each line group in the formatted file
  • the position information also includes the identification information of each column block in the row group, the position information of each column block in the row group;
  • Bitmap index module 603 configured to generate bitmap index data of the column block according to whether each element in the actual data of the column block is empty.
  • the hash unit is specifically used for:
  • the hash unit is specifically used for:
  • the value in the hash structure data is encoded and compressed, and the encoded and compressed hash structure data is written into the persistent memory.
  • the hash unit is specifically used for:
  • the hash structure data is written into the persistent memory by using the operation function in the persistent memory development kit.
  • the hash unit is specifically used for:
  • the metadata module is specifically used for:
  • the metadata of the formatted file is generated according to the file division, wherein the metadata further includes a null value flag bit of the column block, and the null value flag bit is used to indicate that the values of the column block are all null.
  • it also includes:
  • Metadata storage module used to store the metadata in a class two-dimensional array.
  • the persistent memory-based columnar storage device in this embodiment is used to implement the aforementioned persistent memory-based columnar storage method. Therefore, the specific implementation of the device can be seen in the foregoing persistent memory-based columnar storage method. Examples section. Therefore, reference may be made to the descriptions of the corresponding partial embodiments for specific implementations thereof, which will not be described herein again.
  • this embodiment provides an efficient columnar storage device based on persistent memory, which mainly includes a row group keying module, a metadata module and a bitmap indexing module.
  • the row group keying module converts the column blocks of the row group into the key-value hash structure management of the persistent memory medium, so as to make full use of the efficient read and write advantages of the persistent memory;
  • the metadata module records the key value of the file split, Attribute, location and other information of each key value;
  • the bitmap index module generates a bitmap index structure for recording whether the consecutive values in the column block are empty, and organizes the column data into row data according to the index when accessing a row of data .
  • the present application also provides a persistent memory-based columnar storage device, as shown in FIG. 7 , including:
  • Memory 100 used to store computer programs
  • Processor 200 for executing the computer program to implement the steps of the persistent memory-based columnar storage method as described above.
  • the memory includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory, magnetic memory, magnetic disk, optical disk, and the like.
  • the memory can be used not only to store application software and various types of data, but also to temporarily store data that has been output or will be output.
  • the memory may be an internal storage unit in some embodiments, or an external storage device.
  • the processor may in some embodiments be a central processing unit, controller, microcontroller, microprocessor, etc., for executing program code stored in memory.
  • a bus is provided between the memory and the processor, and the bus may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like.
  • PCI peripheral component interconnect
  • EISA extended industry standard architecture
  • the bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 7, but it does not mean that there is only one bus or one type of bus.
  • the present application provides a readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, is used to implement the above-mentioned persistent memory-based columnar storage method A step of.
  • a software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.
  • RAM random access memory
  • ROM read only memory
  • electrically programmable ROM electrically erasable programmable ROM
  • registers hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种基于持久性内存的列式存储方法，该方法将文件转换为哈希结构数据并存储在持久性内存上，由于持久性内存具备字节寻址及非易失性，因此能够在数据读写过程中避免经过文件系统栈，大幅地提升数据查询速度；相比传统列式存储，该存储方式更易于实现修改、删除等实时数据操作，扩大了列式存储的应用场景；此外，该方法了采用一种逻辑简单的且占用空间极小的位图索引数据描述列块上的数据分布情况，避免了记录粉碎和组装算法的复杂性，在保留列式存储优势的同时简化存储逻辑，进一步提升列式存储的数据读写性能。此外，本申请还公开了一种基于持久性内存的列式存储装置、设备及可读存储介质，其技术效果与上述方法的技术效果相对应。

Description

一种基于持久性内存的列式存储方法、装置及设备
本申请要求于2020年08月21日提交至中国专利局、申请号为202010850276.4、发明名称为“一种基于持久性内存的列式存储方法、装置及设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及存储技术领域，特别涉及一种基于持久性内存的列式存储方法、装置、设备及可读存储介质。
背景技术
当今的数据处理大致可以分为两类，即联机事务处理(OLTP)和联机分析处理(OLAP)。传统关系型数据库的主要面向OLTP，用来执行一些基本的、即时的事务处理，如数据库记录的增、删、改、查等；而分布式数据处理系统主要面向OLAP，它对实时性要求较低，但处理的数据量大，通常用于复杂的数据分析与批量报表上。
传统的关系型数据库如Oracle、MySQL等采用行式存储法(Row-based)，在基于行式存储的数据库中，数据是按照行数据为基础逻辑存储单元进行存储的，一行中的数据在存储介质中以连续存储形式存在。
列式存储(Column-based)是相对于行式存储来说的，HBase、ORC、Parquet等均采用列式存储。在基于列式存储的数据库或文件存储中，数据是按照列为基础逻辑存储单元进行存储的，一列中的数据在存储介质中以连续存储形式存在。
目前，列式存储都是基于普通的磁盘文件系统设计的，读写数据都需要经过文件系统I/O栈，因此数据访问速度慢，无法满足越来越高的时效性需求。而且列式存储不适合需要频繁更新数据的场景，不适合做删除、更新等实时操作。
发明内容
本申请的目的是提供一种基于持久性内存的列式存储方法、装置、设备及可读存储介质，用以解决当前的列式存储方案数据访问速度慢，且不适于频繁进行删除、更新等实时操作的问题。其具体方案如下：
第一方面，本申请提供了一种基于持久性内存的列式存储方法，包括：
对待存储的格式化文件进行划分，每预设数量的行划分为一个行组；对所述行组进行划分，每一列作为一个列块；
将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中；
根据文件划分情况生成所述格式化文件的元数据，其中所述元数据包括所述格式化文件中各个行组的标识信息、各个行组在所述格式化文件中的位置信息，还包括所述行组中各个列块的标识信息、各个列块在所述行组中的位置信息；
根据列块的实际数据中各个元素是否为空，生成所述列块的位图索引数据。
优选的，所述将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中，包括：
以所述列块的列名和行组序号为键，以所述列块的实际数据为值，将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中。
优选的，所述将所述哈希结构数据写入持久性内存中，包括：
对所述哈希结构数据中的值进行编码压缩，将编码压缩后的哈希结构数据写入持久性内存中。
优选的，所述将所述哈希结构数据写入持久性内存中，包括：
利用持久性内存开发套件中的操作函数，将所述哈希结构数据写入持久性内存中。
优选的，所述将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中，包括：
将所述列块转换为并发哈希结构数据，将所述并发哈希结构数据写入持久性内存中。
优选的，所述根据文件划分情况生成所述格式化文件的元数据，包括：
根据文件划分情况生成所述格式化文件的元数据，其中所述元数据还包括所述列块的空值标志位，所述空值标志位用于表示列块的值全部为空。
优选的，在所述根据文件划分情况生成所述格式化文件的元数据之后， 还包括：
将所述元数据存储至类二维数组中。
第二方面，本申请提供了一种基于持久性内存的列式存储装置，包括：
行组键值化模块，包括：
文件划分单元：用于对待存储的格式化文件进行划分，每预设数量的行划分为一个行组；对所述行组进行划分，每一列作为一个列块；
哈希单元：用于将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中；
元数据模块：用于根据文件划分情况生成所述格式化文件的元数据，其中所述元数据包括所述格式化文件中各个行组的标识信息、各个行组在所述格式化文件中的位置信息，还包括所述行组中各个列块的标识信息、各个列块在所述行组中的位置信息；
位图索引模块：用于根据列块的实际数据中各个元素是否为空，生成所述列块的位图索引数据。
优选的，所述哈希单元具体用于：
以所述列块的列名和行组序号为键，以所述列块的实际数据为值，将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中。
优选的，所述哈希单元具体用于：
对所述哈希结构数据中的值进行编码压缩，将编码压缩后的哈希结构数据写入持久性内存中。
优选的，所述哈希单元具体用于：
利用持久性内存开发套件中的操作函数，将所述哈希结构数据写入持久性内存中。
优选的，所述哈希单元具体用于：
将所述列块转换为并发哈希结构数据，将所述并发哈希结构数据写入持久性内存中。
优选的，所述元数据模块具体用于：
根据文件划分情况生成所述格式化文件的元数据，其中所述元数据还包括所述列块的空值标志位，所述空值标志位用于表示列块的值全部为空。
优选的，还包括：
元数据存储模块：用于将所述元数据存储至类二维数组中。
第三方面，本申请提供了一种基于持久性内存的列式存储设备，包括：
存储器：用于存储计算机程序；
处理器：用于执行所述计算机程序，以实现如上所述的基于持久性内存的列式存储方法的步骤。
第四方面，本申请提供了一种可读存储介质，所述可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时用于实现如上所述的基于持久性内存的列式存储方法的步骤。
本申请所提供的一种基于持久性内存的列式存储方法，包括：对待存储的格式化文件进行划分，每预设数量的行划分为一个行组；对行组进行划分，每一列作为一个列块；将列块转换为哈希结构数据，将哈希结构数据写入持久性内存中；根据文件划分情况生成格式化文件的元数据，其中元数据包括格式化文件中各个行组的标识信息、各个行组在格式化文件中的位置信息，还包括行组中各个列块的标识信息、各个列块在行组中的位置信息；根据列块的实际数据中各个元素是否为空，生成列块的位图索引数据。
可见，该方法将文件转换为哈希结构数据并存储在持久性内存上，由于持久性内存具备字节寻址及非易失性，因此能够在数据读写过程中避免经过文件系统栈，大幅地提升数据查询速度；相比传统列式存储，该存储方式更易于实现修改、删除等实时数据操作，扩大了列式存储的应用场景；此外，该方法了采用一种逻辑简单的且占用空间极小的位图索引数据描述列块上的数据分布情况，避免了记录粉碎和组装算法的复杂性，在保留列式存储优势的同时简化存储逻辑，进一步提升列式存储的数据读写性能。
此外，本申请还提供了一种基于持久性内存的列式存储装置、设备及可读存储介质，其技术效果与上述方法的技术效果相对应，这里不再赘述。
附图说明
为了更清楚的说明本申请实施例或现有技术的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单的介绍，显而易见地，下 面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请所提供的一种基于持久性内存的列式存储方法实施例一的实现流程图；
图2为本申请所提供的一种基于持久性内存的列式存储方法实施例二的实现流程图；
图3为本申请所提供的一种基于持久性内存的列式存储方法实施例在实际应用中的过程示意图；
图4为本申请所提供的一种基于持久性内存的列式存储方法实施例二中行组键值化过程的示意图；
图5为本申请所提供的一种基于持久性内存的列式存储方法实施例二中位图索引数据生成过程的示意图；
图6为本申请所提供的一种基于持久性内存的列式存储装置实施例的功能框图；
图7为本申请所提供的一种基于持久性内存的列式存储设备实施例的结构示意图。
具体实施方式
本申请的核心是提供一种基于持久性内存的列式存储方法、装置、设备及可读存储介质，能够将文件划分为行组，将行组转换为哈希结构数据，并写入持久性内存中，能够在数据读写过程中避免经过文件系统栈，大幅地提升数据查询速度，且更易于实现修改、删除等实时数据操作；且采用位图索引数据描述列块上的数据分布情况，逻辑简单占用空间极小，简化了存储逻辑，进一步提升列式存储的数据读写性能。
为了使本技术领域的人员更好地理解本申请方案，下面结合附图和具体实施方式对本申请作进一步的详细说明。显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施 例，都属于本申请保护的范围。
下面对本申请提供的一种基于持久性内存的列式存储方法实施例一进行介绍，参见图1，实施例一包括：
S101、对待存储的格式化文件进行划分，每预设数量的行划分为一个行组；对所述行组进行划分，每一列作为一个列块；
上述格式化文件是指每一行表示一条数据，每条数据包括多个属性，每一列的数据的属性相同的文件。
上述预设数量可以根据实际需求自行设定，本实施例不做限定。
S102、将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中；
哈希结构(HashMap)是Java中使用频率最高的用于映射(键值对)处理的数据类型，具有很快的访问速度。
S103、根据文件划分情况生成所述格式化文件的元数据，其中所述元数据包括所述格式化文件中各个行组的标识信息、各个行组在所述格式化文件中的位置信息，还包括所述行组中各个列块的标识信息、各个列块在所述行组中的位置信息；
S104、根据列块的实际数据中各个元素是否为空，生成所述列块的位图索引数据。
持久性内存(Persistent Memory，PM)是非易失性内存的一种，具有类似于内存的特性和类似于存储的特性，可像内存一样按字节寻址，且其存储的内容是非易失性的。位于持久性内存上的数据可以从用户空间直接访问，访问数据路径上没有内核代码、文件系统页缓存或中断。
本实施例基于持久性内存设备的物理特性，提供了一种键值结构的列式存储方案，能够大幅提高数据查询速度，满足即时查询的需求，同时可以更好地支持数据的修改与删除，拓展了列式存储的适用场景。
本实施例主要包括行组键值化阶段、元数据生成阶段和位图索引生成阶段。在行组键值化阶段，本实施例将文件划分为行组，并将行组中的列块转换为持久性内存介质的键值哈希结构，以充分利用持久性内存的高效 读写优势；在元数据生成阶段，生成用于记录文件划分为哪些行组、每个行组包括哪些列块、行组在文件中的位置信息、列块在行组中的位置信息的元数据；在位图索引生成阶段，生成用于记录列块的实际数据的各个元素是否为空的位图索引数据。下面分别对这三个阶段进行介绍：
在行组键值化阶段，本实施例对格式化文件进行划分，每预设数量的行作为一个行组；每个行组含有多个列的数据，对行组做进一步划分，每一列作为一个列块。然后，为提升数据读写性能，将列块转换为哈希结构数据，并将哈希结构数据写入持久性内存中。
作为一种具体的实现方式，可以以列块的列名和行组序号为键，以列块的实际数据为值，从而将列块转换为哈希结构数据，将哈希结构数据写入持久性内存中。可以理解的是，在将哈希结构数据写入持久性内存之前，需要对哈希结构数据中的值进行编码，为了进一步减少空间占用，可以在编码之后，对哈希结构数据中的值进行压缩，具体采用何种编码方法或何种压缩方式，本实施例不做限定。
为了进一步提升数据读写性能，可以选用持久性内存中特有的并发哈希结构(Concurrent Hashmap)。在将列块转换为哈希结构数据的时候，具体将列块转换为并发哈希结构数据，并将并发哈希结构数据写入持久性内存中。
在将哈希结构数据写入持久性内存的时候，为了提升写入效率，可以采用更接近底层的函数实现该写入操作。具体的，可以利用持久性内存开发套件(PMDK)中的操作函数，例如libpmemkv(PMDK的一个子函数库)，将所述哈希结构数据写入持久性内存中。通过这种写入方式，写操作不需要预先在内存中分配缓存，读取时应用也可以直接访问PM中的数据，速度远高于传统磁盘存储。
在元数据生成阶段，主要用于生成格式化文件的元数据，其中元数据包括但不限于格式化文件包括哪些行组，各个行组在格式化文件中的位置或顺序，行组包括哪些列块，各个列块在行组中的位置或顺序；此外，还可以包括列块的值的编码方法、列块的值压缩方法以及一些数据统计信息，例如列块的值中的最大值和最小值等。
作为一种优选的实施方式，元数据中还可以包括用于表示列块的值是否全部为空的空值标志位。从而在数据读写过程中，直接根据该空值标志位确定待读写的列是否为空，提升数据读写性能。
由于本实施例中列块的元数据既需要存储元素名称，也需要记录元素的位置，因此，在实际应用中，可以利用类二维数据存储列块的元数据。
在位图索引生成阶段，本实施例针对每一个列块生成对应的位图索引数据，位图索引数据主要表示对应列块中实际数据的有无。优选的，如果某一列块没有实际数据，那在列存储时，就可以不存储该列的值，这将比行式存储更节省空间。位图是通过一个比特数组来存储特定数据的一种数据结构，每一个比特位都能独立包含信息。本实施例中，位图索引数据中的每个比特用于表示列块中相应的元素是否为空，由于比特是数据的最小存储单元，因此能大量节省空间。
在数据查询时如果指定目标列，可以通过行组键值化阶段的键快速获取到目标列的值，基于持久性内存的直接访问特性可以绕过文件系统栈，使这个过程非常快速；如果需要查询目标行的数据，则通过位图索引数据，根据该索引将列式数据组织成行式数据，直接将索引相等的值还原为一行数据返回，而不用实现记录粉碎和组装算法(striping and assembly algorithms，前述Parquet采用的算法)。
本实施例所提供一种基于持久性内存的列式存储方法，将文件转换为哈希结构数据并存储在持久性内存上，由于持久性内存具备字节寻址及非易失性，因此能够在数据读写过程中避免经过文件系统栈，大幅地提升数据查询速度；相比传统列式存储，该存储方式更易于实现修改、删除等实时数据操作，扩大了列式存储的应用场景；此外，该方法了采用一种逻辑简单的且占用空间极小的位图索引数据描述列块上的数据分布情况，避免了记录粉碎和组装算法的复杂性，在保留列式存储优势的同时简化存储逻辑，进一步提升列式存储的数据读写性能。
下面开始详细介绍本申请提供的一种基于持久性内存的列式存储方法实施例二，实施例二基于前述实施例一实现，并在实施例一的基础上进行 了一定程度上的拓展。
具体的，本实施例对键值化的列块的键和值进行了限定，采用了并发哈希结构进一步提升数据读写性能，对列块的值进行了编码压缩以节省存储空间。利用底层的操作函数执行写入操作，提升写入效率。利用类二维数组存储元数据，效率较高。
参见图2，实施例二具体包括：
S201、对待存储的格式化文件进行划分，每预设数量的行划分为一个行组；对所述行组进行划分，每一列作为一个列块；
S202、以所述列块的列名和行组序号为键，以所述列块的实际数据为值，将所述列块转换为并发哈希结构数据；
S203、对所述并发哈希结构数据中的值进行编码压缩；
S204、利用持久性内存开发套件中的操作函数，将所述哈希结构数据写入持久性内存中；
S205、根据文件划分情况生成所述格式化文件的元数据，将所述元数据存储至类二维数组中；
其中所述元数据包括所述格式化文件中各个行组的名称、各个行组在所述格式化文件中的位置信息、所述行组中各个列块的名称、各个列块在所述行组中的位置信息；还包括所述列块的空值标志位，所述空值标志位用于表示列块的值全部为空；还包括所述列块的值的压缩方法、编码方法；还包括所述列块的值中的最大值和最小值等。
S206、根据列块的实际数据中各个元素是否为空，生成所述列块的位图索引数据。
本实施例所提供一种基于持久性内存的列式存储方法，基于持久性内存的字节寻址特性，提供了一种键值结构的列式存储方案，主要包括行组键值化阶段、元数据生成阶段和位图索引生成阶段。行组键值化阶段将行组的各个列块转换为持久性内存介质的键值哈希结构，以充分利用持久性内存的高效读写优势；元数据生成阶段记录格式化文件拆分为何种键值，每种键值的属性、位置等信息；位图索引生成阶段生成用于记录列块中的连续数值是否为空的位图索引结构，在访问一行数据时根据该索引将列式 数据组织成行式数据。
下面以实际应用为例，对本申请的基于持久性内存的列式存储过程进行说明，整体的实现过程如图3所示。
假设，目前有一个待存储的格式化文件如表1所示，表1中一行表示一条数据：
表1
Figure PCTCN2021073472-appb-000001
在行组键值化阶段，将格式化文件按特定长度水平划分成行组。如特定长度为1024，则每1024行数据为一个行组，每个行组含有若干列，则每一列最多含有1024个元素。如图4所示，包括以下子步骤：
子步骤1：在如表1所示的数据中，第1个行组为ID1到ID1024的数据，第2个行组为ID 1025到ID 2048的数据，依次类推。如果一共N行数据，则共有N/1024个行组(向上取整数)。
子步骤2：在如表1所示的数据中，第1个行组划分为8个列块，将列名作为键，为区别不同行组的键，则键值化存储的键可设计为列名加行组序号，值设计为列块中的实际数据。比如，第1个行组的第1个列块的键为“ID_1”，其值为[1,2,3,4,5,6,7,8]；第1个行组的第2个列块的键为“姓名_1”，其值为[小明，小李,小红,小华，小张，小黑，小黄，小宋]；第1个行组的第3个列块的键为“性别_1”，其值为[男，女，女，女，男]；第 1个行组的第4个列块的键为“出生_1”，其值为空，依次类推，将行组中的所有值转换完毕。可以看出，每个列块的值都是相同类型的元素，因此可以进行高效的数据压缩。
子步骤3：使用持久性内存开发套件中的操作函数进行数据读写，将上述键值化数据通过libpmemkv(持久性内存开发套件的一个子函数库)中的存储方法写入到持久性内存中。
在元数据生成阶段，过程如图5所示，记录的元数据信息主要有格式化文件包括哪些行组，每个行组包括哪些列块，列块的键是什么，每个列块的值采用了什么方法压缩编码，以及列块值的统计信息，如最大最小值等。元数据信息可以使用类二维数组进行矩阵化存储，表1所示的格式化文件的元数据可这样表示：
[格式化文件 (表1)]
Figure PCTCN2021073472-appb-000002
其中，“格式化文件”记录如表1所示的数据按照1024的长度进行切分，文件属性等；“行组1”表示第1个行组的1024行数据，包含了“ID_1”,“姓名_1”等8个列名块；“ID_1”,“ID_2”到“ID_n”表示ID这一列的全部数据，可以按照列进行过滤，不需要扫描无关的列；如示例数据1的行组1的“出生”的列块值全部为空，则“出生_1”应添加空值标记，在扫描读取“出生”这一列的数据时，跳过行组1，依次类推，达到减少查询时间的目的。
在位图索引生成阶段，为每一个列块的值创建一个位图索引文件。一般地，该位图索引所占的比特数等于行组的划分长度，如1024位。为了便于说明问题，此处使用表1中的8行数据进行阐述，即假设一个位图索引仅8位。
比如“姓名_1”列块值，它的8个元素都存在，则位图索引为：11111111； 再比如“性别_1”列块值，它有几个空元素，其位图索引为：10100111；又比如“职业_1”列块值，它前4个元素都存在，则位图索引为：11110000；如果一个列块值全空，如“出生_1”，无论行组长度为多少，其位图索引都是8位0：00000000。
位图索引占用的空间极小，1M的位图索引文件可以表示超过800万行数据，如果遇到列块值为空，则占用的空间会更少。
在实际应用中，该索引文件可以存放的位置包括但不限于：
位置一，直接与键绑定，即键为列名、行组序号、位图索引的有效组合；
位置二，存放在元数据中，与每个列块的元数据一起存放。
下面对本申请实施例提供的一种基于持久性内存的列式存储装置进行介绍，下文描述的一种基于持久性内存的列式存储装置与上文描述的一种基于持久性内存的列式存储方法可相互对应参照。
如图6所示，本实施例的基于持久性内存的列式存储装置，包括：
行组键值化模块601，包括：
文件划分单元：用于对待存储的格式化文件进行划分，每预设数量的行划分为一个行组；对所述行组进行划分，每一列作为一个列块；
哈希单元：用于将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中；
元数据模块602：用于根据文件划分情况生成所述格式化文件的元数据，其中所述元数据包括所述格式化文件中各个行组的标识信息、各个行组在所述格式化文件中的位置信息，还包括所述行组中各个列块的标识信息、各个列块在所述行组中的位置信息；
位图索引模块603：用于根据列块的实际数据中各个元素是否为空，生成所述列块的位图索引数据。
在一些具体的实施例中，所述哈希单元具体用于：
以所述列块的列名和行组序号为键，以所述列块的实际数据为值，将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中。
在一些具体的实施例中，所述哈希单元具体用于：
对所述哈希结构数据中的值进行编码压缩，将编码压缩后的哈希结构数据写入持久性内存中。
在一些具体的实施例中，所述哈希单元具体用于：
利用持久性内存开发套件中的操作函数，将所述哈希结构数据写入持久性内存中。
在一些具体的实施例中，所述哈希单元具体用于：
将所述列块转换为并发哈希结构数据，将所述并发哈希结构数据写入持久性内存中。
在一些具体的实施例中，所述元数据模块具体用于：
根据文件划分情况生成所述格式化文件的元数据，其中所述元数据还包括所述列块的空值标志位，所述空值标志位用于表示列块的值全部为空。
在一些具体的实施例中，还包括：
元数据存储模块：用于将所述元数据存储至类二维数组中。
本实施例的基于持久性内存的列式存储装置用于实现前述的基于持久性内存的列式存储方法，因此该装置中的具体实施方式可见前文中的基于持久性内存的列式存储方法的实施例部分。所以，其具体实施方式可以参照相应的各个部分实施例的描述，在此不再展开介绍。
本实施例基于持久性内存的字节寻址特性，提供了基于持久性内存的高效列式存储装置，主要包括行组键值化模块、元数据模块和位图索引模块。行组键值化模块将行组的列块转换为持久性内存介质的键值哈希结构管理，以充分利用持久性内存的高效读写优势；元数据模块记录文件拆分为何种键值，每种键值的属性、位置等信息；位图索引模块生成用于记录列块中的连续数值是否为空的位图索引结构，在访问一行数据时根据该索引将列式数据组织成行式数据。
此外，本申请还提供了一种基于持久性内存的列式存储设备，如图7所示，包括：
存储器100：用于存储计算机程序；
处理器200：用于执行所述计算机程序，以实现如上文所述的基于持久性内存的列式存储方法的步骤。
其中，存储器至少包括一种类型的可读存储介质，例如闪存、硬盘、多媒体卡、卡型存储器、磁性存储器、磁盘、光盘等。存储器不仅可以用于存储应用软件及各类数据，还可以用于暂时地存储已经输出或者将要输出的数据。存储器在一些实施例中可以是内部存储单元，也可以是外部存储设备。
处理器在一些实施例中可以是一中央处理器、控制器、微控制器、微处理器等，用于运行存储器中存储的程序代码。
存储器和处理器之间设置有总线，该总线可以是外设部件互连标准(peripheral component interconnect，简称PCI)总线、扩展工业标准结构(extended industry standard architecture，简称EISA)总线等。该总线可以分为地址总线、数据总线、控制总线等。为便于表示，图7中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
最后，本申请提供了一种可读存储介质，所述可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时用于实现如上所述的基于持久性内存的列式存储方法的步骤。
本说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其它实施例的不同之处，各个实施例之间相同或相似部分互相参见即可。对于实施例公开的装置而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。
结合本文中所公开的实施例描述的方法或算法的步骤可以直接用硬件、处理器执行的软件模块，或者二者的结合来实施。软件模块可以置于随机存储器(RAM)、内存、只读存储器(ROM)、电可编程ROM、电可擦除可编程ROM、寄存器、硬盘、可移动磁盘、CD-ROM、或技术领域内所公知的任意其它形式的存储介质中。
以上对本申请所提供的方案进行了详细介绍，本文中应用了具体个例 对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想；同时，对于本领域的一般技术人员，依据本申请的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本申请的限制。

Claims (16)

  1. 一种基于持久性内存的列式存储方法，其特征在于，包括：
    对待存储的格式化文件进行划分，每预设数量的行划分为一个行组；对所述行组进行划分，每一列作为一个列块；
    将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中；
    根据文件划分情况生成所述格式化文件的元数据，其中所述元数据包括所述格式化文件中各个行组的标识信息、各个行组在所述格式化文件中的位置信息，还包括所述行组中各个列块的标识信息、各个列块在所述行组中的位置信息；
    根据列块的实际数据中各个元素是否为空，生成所述列块的位图索引数据。
  2. 如权利要求1所述的基于持久性内存的列式存储方法，其特征在于，所述将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中，包括：
    以所述列块的列名和行组序号为键，以所述列块的实际数据为值，将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中。
  3. 如权利要求2所述的基于持久性内存的列式存储方法，其特征在于，所述将所述哈希结构数据写入持久性内存中，包括：
    对所述哈希结构数据中的值进行编码压缩，将编码压缩后的哈希结构数据写入持久性内存中。
  4. 如权利要求2所述的基于持久性内存的列式存储方法，其特征在于，所述将所述哈希结构数据写入持久性内存中，包括：
    利用持久性内存开发套件中的操作函数，将所述哈希结构数据写入持久性内存中。
  5. 如权利要求2所述的基于持久性内存的列式存储方法，其特征在于，所述将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中，包括：
    将所述列块转换为并发哈希结构数据，将所述并发哈希结构数据写入 持久性内存中。
  6. 如权利要求1-5任意一项所述的基于持久性内存的列式存储方法，其特征在于，所述根据文件划分情况生成所述格式化文件的元数据，包括：
    根据文件划分情况生成所述格式化文件的元数据，其中所述元数据还包括所述列块的空值标志位，所述空值标志位用于表示列块的值全部为空。
  7. 如权利要求6所述的基于持久性内存的列式存储方法，其特征在于，在所述根据文件划分情况生成所述格式化文件的元数据之后，还包括：
    将所述元数据存储至类二维数组中。
  8. 一种基于持久性内存的列式存储装置，其特征在于，包括：
    行组键值化模块，包括：
    文件划分单元：用于对待存储的格式化文件进行划分，每预设数量的行划分为一个行组；对所述行组进行划分，每一列作为一个列块；
    哈希单元：用于将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中；
    元数据模块：用于根据文件划分情况生成所述格式化文件的元数据，其中所述元数据包括所述格式化文件中各个行组的标识信息、各个行组在所述格式化文件中的位置信息，还包括所述行组中各个列块的标识信息、各个列块在所述行组中的位置信息；
    位图索引模块：用于根据列块的实际数据中各个元素是否为空，生成所述列块的位图索引数据。
  9. 如权利要求8所述的基于持久性内存的列式存储装置，其特征在于，所述哈希单元具体用于：
    以所述列块的列名和行组序号为键，以所述列块的实际数据为值，将所述列块转换为哈希结构数据，将所述哈希结构数据写入持久性内存中。
  10. 如权利要求9所述的基于持久性内存的列式存储装置，其特征在于，所述哈希单元具体用于：
    对所述哈希结构数据中的值进行编码压缩，将编码压缩后的哈希结构数据写入持久性内存中。
  11. 如权利要求9所述的基于持久性内存的列式存储装置，其特征在 于，所述哈希单元具体用于：
    利用持久性内存开发套件中的操作函数，将所述哈希结构数据写入持久性内存中。
  12. 如权利要求9所述的基于持久性内存的列式存储装置，其特征在于，所述哈希单元具体用于：
    将所述列块转换为并发哈希结构数据，将所述并发哈希结构数据写入持久性内存中。
  13. 如权利要求8-12任意一项所述的基于持久性内存的列式存储装置，其特征在于，所述元数据模块具体用于：
    根据文件划分情况生成所述格式化文件的元数据，其中所述元数据还包括所述列块的空值标志位，所述空值标志位用于表示列块的值全部为空。
  14. 如权利要求13所述的基于持久性内存的列式存储装置，其特征在于，还包括：
    元数据存储模块：用于将所述元数据存储至类二维数组中。
  15. 一种基于持久性内存的列式存储设备，其特征在于，包括：
    存储器：用于存储计算机程序；
    处理器：用于执行所述计算机程序，以实现如权利要求1-7任意一项所述的基于持久性内存的列式存储方法的步骤。
  16. 一种可读存储介质，其特征在于，所述可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时用于实现如权利要求1-7任意一项所述的基于持久性内存的列式存储方法的步骤。
PCT/CN2021/073472 2020-08-21 2021-01-25 一种基于持久性内存的列式存储方法、装置及设备 WO2022037015A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010850276.4A CN111984651A (zh) 2020-08-21 2020-08-21 一种基于持久性内存的列式存储方法、装置及设备
CN202010850276.4 2020-08-21

Publications (1)

Publication Number Publication Date
WO2022037015A1 true WO2022037015A1 (zh) 2022-02-24

Family

ID=73443652

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/073472 WO2022037015A1 (zh) 2020-08-21 2021-01-25 一种基于持久性内存的列式存储方法、装置及设备

Country Status (2)

Country Link
CN (1) CN111984651A (zh)
WO (1) WO2022037015A1 (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111984651A (zh) * 2020-08-21 2020-11-24 苏州浪潮智能科技有限公司 一种基于持久性内存的列式存储方法、装置及设备
CN113553327B (zh) * 2021-07-06 2024-08-23 杭州网易云音乐科技有限公司 数据处理方法和装置、介质和计算设备
CN113722623B (zh) * 2021-09-03 2024-07-05 锐掣(杭州)科技有限公司 数据处理方法、装置、电子设备及存储介质
CN114443703A (zh) * 2021-12-15 2022-05-06 北京达佳互联信息技术有限公司 数据处理方法、装置、电子设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110153650A1 (en) * 2009-12-18 2011-06-23 Electronics And Telecommunications Research Institute Column-based data managing method and apparatus, and column-based data searching method
CN102214360A (zh) * 2010-04-06 2011-10-12 腾讯科技（深圳）有限公司 一种位图构建方法及装置、求交的方法及装置
CN102402617A (zh) * 2011-12-23 2012-04-04 天津神舟通用数据技术有限公司 一种利用分片及稀疏位图的易压缩数据库索引存储系统及其相应的构建、调度和查询处理方法
CN107967124A (zh) * 2017-12-14 2018-04-27 南京云创大数据科技股份有限公司 一种分布式持久性内存存储系统及方法
CN111190895A (zh) * 2018-11-15 2020-05-22 百度在线网络技术（北京）有限公司 列式存储数据的组织方法、装置及存储介质
CN111984651A (zh) * 2020-08-21 2020-11-24 苏州浪潮智能科技有限公司 一种基于持久性内存的列式存储方法、装置及设备

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104866497B (zh) * 2014-02-24 2018-06-15 华为技术有限公司 分布式文件系统列式存储的元数据更新方法、装置、主机
CN107545021B (zh) * 2017-05-10 2020-12-11 新华三信息安全技术有限公司 一种数据存储方法及装置
CN110166221B (zh) * 2019-03-05 2022-02-22 西安电子科技大学 一种密文数据压缩存储结构rcpe及数据动态读写方法
CN110968585B (zh) * 2019-12-20 2023-11-03 深圳前海微众银行股份有限公司 面向列的存储方法、装置、设备及计算机可读存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110153650A1 (en) * 2009-12-18 2011-06-23 Electronics And Telecommunications Research Institute Column-based data managing method and apparatus, and column-based data searching method
CN102214360A (zh) * 2010-04-06 2011-10-12 腾讯科技（深圳）有限公司 一种位图构建方法及装置、求交的方法及装置
CN102402617A (zh) * 2011-12-23 2012-04-04 天津神舟通用数据技术有限公司 一种利用分片及稀疏位图的易压缩数据库索引存储系统及其相应的构建、调度和查询处理方法
CN107967124A (zh) * 2017-12-14 2018-04-27 南京云创大数据科技股份有限公司 一种分布式持久性内存存储系统及方法
CN111190895A (zh) * 2018-11-15 2020-05-22 百度在线网络技术（北京）有限公司 列式存储数据的组织方法、装置及存储介质
CN111984651A (zh) * 2020-08-21 2020-11-24 苏州浪潮智能科技有限公司 一种基于持久性内存的列式存储方法、装置及设备

Also Published As

Publication number Publication date
CN111984651A (zh) 2020-11-24

Similar Documents

Publication Publication Date Title
WO2022037015A1 (zh) 一种基于持久性内存的列式存储方法、装置及设备
US10824673B2 (en) Column store main fragments in non-volatile RAM and the column store main fragments are merged with delta fragments, wherein the column store main fragments are not allocated to volatile random access memory and initialized from disk
US10719450B2 (en) Storage of run-length encoded database column data in non-volatile memory
US11886401B2 (en) Database key compression
US10061693B2 (en) Method of generating secondary index and apparatus for storing secondary index
US11200159B2 (en) System and method for facilitating efficient utilization of NAND flash memory
CN111475507B (zh) 一种工作负载自适应单层lsmt的键值数据索引方法
US11308054B2 (en) Efficient large column values storage in columnar databases
US9104726B2 (en) Columnar databases
US11176033B2 (en) Data storage devices and data processing methods
CN107423425B (zh) 一种对k/v格式的数据快速存储和查询方法
JP2024525170A (ja) データ圧縮方法及び装置
CN110780806A (zh) 促进元数据和数据捆绑存储的原子性保证的方法和系统
US20240220470A1 (en) Data storage device and storage control method based on log-structured merge tree
CN115774699B (zh) 数据库共享字典压缩方法、装置、电子设备及存储介质
CN115576868B (zh) 一种多级映射框架、数据操作请求处理方法及系统
CN114415966B (zh) 一种kv ssd存储引擎的构建方法
CN114003169B (zh) 一种用于ssd的数据压缩方法
KR102251811B1 (ko) 하드웨어 필터를 포함하는 데이터 저장 장치 및 상기 데이터 저장 장치를 포함하는 데이터 처리 시스템
WO2016194159A1 (ja) 計算機、データベース管理方法、データベース管理システム
CN111694847B (zh) 一种特大lob数据高并发低延迟的更新访问方法
CN112433673B (zh) 一种固态硬盘存储数据的方法和设备
US11829341B2 (en) Space-efficient persistent hash table data structure
US11055231B2 (en) Data storage devices and data processing methods of skipping editing of fields in H2F table when consecutive addresses are present in F2H table
CN118363955A (zh) 一种数据处理方法、设备及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21857121

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21857121

Country of ref document: EP

Kind code of ref document: A1