CN113157692B

CN113157692B - Relational memory database system

Info

Publication number: CN113157692B
Application number: CN202110175982.8A
Authority: CN
Inventors: 朱悬宁; 张锐; 单兰山
Original assignee: Shanghai Wandehonghui Information Technology Co ltd
Current assignee: Shanghai Wandehonghui Information Technology Co ltd
Priority date: 2021-02-07
Filing date: 2021-02-07
Publication date: 2023-10-27
Anticipated expiration: 2041-02-07
Also published as: CN113157692A

Abstract

The application provides a relational memory database system, which can extend the relational database operation of the traditional database, and the memory database is directly accessed through the shared memory of the system, thereby greatly improving the read-write speed and being particularly suitable for computationally intensive application. The application constructs the basic relational database function in the shared memory through the data line management module and the table definition management module. The application automatically accelerates various column attribute queries through the index management module, and does not need to manually construct various key values to accelerate access like Redis. According to the application, all data are stored in the shared memory through the function of the memory block management module for reading and writing of each application program, so that the performance of the application, especially the computationally intensive application, is accelerated.

Description

Relational memory database system

Technical Field

The application relates to a relational-based memory database system, which is suitable for the computer software and information technology service industry and belongs to the technical field of data caching.

Background

Software programs typically involve read and write operations of data. Many systems use magnetic disks as media for data operations, as shown in fig. 1, resulting in slower system operation speeds. As memory capacity increases and prices decrease, many programs store data in memory for read and write operations to speed up system operation, as shown in fig. 2. This creates a need for an in-memory database system.

As shown in fig. 3, redis (Remote Dictionary Serve, remote dictionary service) is a very popular in-memory database that finds application in many software systems. The basic principle is to provide a service accessible through a network so that a client can store key-value pairs and also read data according to key values. The method is characterized in that:

1. the remote service may be provided through a network, i.e., the Redis service and the program using the service may be located on different machines as long as they are connected through the network.

2. The remote service may store key-value pairs. By key-value pair is meant a pair of data comprising a key and a value. The key may be a string and the value may be a string or a more complex structure. Such as: { "user 001", "Ming" } is a key value pair, "user 001" is a key, and "Ming" is a corresponding value.

When the client program stores such a key-value pair in Redis, it can retransmit the key "user 001" to read its corresponding value: "Xiaoming". This key may also be modified.

Redis's data takes a key-value pattern, but in practice conventional databases tend to be relational databases. A relational database refers to a database that employs a relational model to organize data, which stores data in rows and columns for ease of user understanding, the series of rows and columns of the relational database being referred to as tables, a set of tables comprising the database. The user retrieves the data in the database by querying, which is an executable code that defines certain areas in the database. A relational model can be understood simply as a two-dimensional tabular model, and a relational database is a data organization consisting of two-dimensional tables and relationships between them. The most obvious point of relational databases is that data is stored using tables instead of key patterns. Such as the following is a table:

the data of each table in the relational data is composed of more basic units: data lines. In the above table, each row is a data row except for the header. A data line may have a plurality of attributes, some of which may be used as indexes. For example, the table above can query the eligible rows of data by the attributes of [ client number ], [ name ], or [ phone number ], and the attributes used for the query are referred to as indexes.

Redis has some undesirable places:

redis is an in-memory database based on key-value patterns, whereas conventional software development uses a file database system that is typically a relational database. This gives the developer great flexibility and is not easy to correspond to the data of the file database so as to perform the data persistence operation.

2. Databases based on key-value patterns are also far less powerful when used than relational databases. In particular, the same value often needs to be retrieved by multiple attributes, and a key-value schema based database needs to be manually constructed with additional key values to support the index of the relational database itself.

Redis, although an in-memory database, is accessed through network requests, and for computationally intensive applications, one operation may involve many times of data reading and writing, each time the Redis data needs to be read/written through the network, which may cause serious degradation in operation performance.

Disclosure of Invention

The purpose of the application is that: the memory database can extend the relational database operation of the traditional database, is directly accessed through the shared memory of the system, greatly improves the reading and writing speed, and is particularly suitable for computationally intensive application.

In order to achieve the above object, the present application provides a relational memory database system, which is characterized in that all data in the relational memory database system is stored in a shared memory, and an application program accesses the data in the shared memory through an external access interface provided by the relational memory database system;

the relational memory database system further comprises a data row management module, an index management module, a table definition management module and a memory block management module; the external access interface provides functions by calling a data line management module, an index management module and a table definition management module, and the data line management module, the index management module and the table definition management module call a memory block management module to realize access to a certain unified shared memory address or realize application of a new memory block address, wherein:

the memory block management module is used for managing the shared memory in the operating system, unifying all the shared memory blocks into a unified and continuous memory address to apply for access, defining the unified and continuous memory address as a unified shared memory address, wherein the unified shared memory address comprises a shared memory block number and a shared memory block internal address, and each shared memory block number corresponds to each shared memory block one by one;

when the data line management module, the index management module or the table definition management module accesses a certain unified shared memory address, the memory block management module performs the following processing:

step 101: checking the shared memory block number in the current unified shared memory address, judging whether the shared memory block corresponding to the shared memory block number is loaded into the local process address, if so, entering step 103, and if not, entering step 102;

step 102: loading the shared memory block corresponding to the current shared memory block number into a local process address, and entering step 103;

step 103: acquiring a real memory address for access according to the internal address of the shared memory block in the current unified shared memory address;

the memory block management module maintains information of all shared memory blocks and memory allocation conditions; when the data line management module, the index management module or the table definition management module applies for the memory from the memory block management module, the memory block management module feeds back a new unified shared memory address, and the method comprises the following steps:

step 201: judging whether the current shared memory block has residual space for allocation according to the memory allocation condition of the current shared memory block, if so, entering step 202, and if not, entering step 203;

step 202: taking the number of the current shared memory block as the shared memory block number, taking the memory allocation position of the current shared memory block as the shared memory block internal address, and feeding back the unified shared memory address comprising the shared memory block number and the shared memory block internal address to the data line management module, the index management module or the table definition management module;

step 203: the memory block management module creates a new shared memory block, takes the number of the newly created shared memory block as the shared memory block number, takes the memory allocation position of the newly created shared memory block as the internal address of the shared memory block, and feeds back the unified shared memory address comprising the shared memory block number and the internal address of the shared memory block to the data line management module, the index management module or the table definition management module;

the data line management module is used for managing all data lines in the relational memory database system; the data line management module applies for a memory for storing data and for storing management information by using the memory block management module, initializes the applied memory to obtain a data line, and uniformly manages all the data lines belonging to the same data table based on the management information; one data line is represented by a unified shared memory address applied in a memory block management module; the data line management module performs read-write operation on the applied unified shared memory address so as to complete the read-write operation of the data line;

the index management module is used for managing the inquiry of the data rows, wherein each data row is provided with a plurality of attribute columns, when inquiring, all the data rows of a certain data table inquire the data rows meeting the attribute according to the attribute of the attribute columns of K columns, and K is more than or equal to 1, and the index management module adopts the following steps to manage the inquiry of the data rows:

step 301: when the data table is created, M indexes are created for the column attribute column of the current data table through an external access interface, M is more than or equal to 1, and information of the data table and information of the indexes are stored in the table definition management module;

applying for an index information handle array in the shared memory for each index, wherein elements in each index information handle array store a unified shared memory address and point to a shared memory block of data of a hash bucket which is created by a memory block management module and used for storing;

for the data table, applying a table information handle array in the shared memory, wherein elements in each table information handle array store a unified shared memory address and point to a shared memory block which is created by a memory block management module and used for storing information of the data table;

step 302: when data is inserted, a data line management module is used for applying new line data, wherein the line data comprises M index link bits, and M indexes respectively use one index link bit;

step 303: after the data lines are initialized, linking all the data lines in the current data table together by the management information;

step 304: for the mth index in the current data table, m=1, …, M: performing hash operation on key value data of an attribute column corresponding to the m index, then linking data lines at corresponding hanging positions in a hash bucket according to hash operation results, linking data lines with the same hash operation results at the same hanging positions in the hash bucket, and linking data lines with different hash operation results at different hanging positions in the hash bucket;

when searching corresponding data rows in the current data table according to certain attribute columns, the method comprises the following steps:

step 401: judging whether the searched attribute column contains the attribute column with the index created in the step 301, if not, carrying out row-by-row searching, and if so, entering the step 402;

step 402: for the attribute columns of the created index, an integer hash value is calculated by using the same hash method during insertion in step 304, and after the corresponding hooking position is positioned in the hash bucket according to the integer hash value, the attribute columns of the searched attribute columns except for the attribute columns of the created index are used for searching based on all the data rows of the current hooking position.

The table definition management module stores table information, data column information of the table and index information of the table in the relational memory database system in the shared memory and manages the table information, and the external access interface provides an interface for the outside by calling the function of the table definition management module, so that the application software creates the data table and creates M indexes for M column attribute columns of the current data table; the table definition management module applies memory to the memory block management module and is used for storing a table information handle array and an index information handle array.

Preferably, if the current shared memory is insufficient, the memory block management module applies for the shared memory to the operating system, and the memory block management module applies for the shared memory with a fixed size only to the operating system at a time.

Preferably, the set formed by different data rows is defined as a data row set, all the data rows of the same data table can form a data row set, the data rows belonging to the same data table can form a plurality of data row sets, when the data row management module applies for the memory of the current data row, if the current data row belongs to N different data row sets, N is greater than or equal to 1, the management information comprises N data table linking bits, the different data rows are mutually linked into N different data row sets through the data table linking bits, and the different data rows are linked together through the N data table linking bits to form the N data row set, n=1, … and N.

Compared with the prior art, the application has the following advantages:

1. and constructing a basic relational database function in the shared memory through the data line management module and the table definition management module. The developer is more adapted to this approach than to the in-memory database of key-value patterns. And can be in one-to-one correspondence with the data of the traditional relational file database.

2. Various column attribute queries are automatically accelerated by the index management module without the need to manually construct various key values to accelerate access, as with Redis.

3. All data are stored in the shared memory through the function of the memory block management module and are read and written by each application program, so that the performance of the application, especially the computationally intensive application, is accelerated.

Drawings

FIG. 1 is a schematic diagram of a conventional system for performing data operations using a disk as a medium;

FIG. 2 is a schematic diagram of a conventional system for storing data in a memory for read/write operations;

FIG. 3 is a schematic diagram of a Redis system;

FIG. 4 is a schematic diagram of an application architecture of the present application;

FIG. 5 is a system block diagram of the present application;

FIG. 6 is an example of linking non-passing data rows together by a linking bit;

fig. 7 is a construction index example.

Detailed Description

The application will be further illustrated with reference to specific examples. It is to be understood that these examples are illustrative of the present application and are not intended to limit the scope of the present application. Furthermore, it should be understood that various changes and modifications can be made by one skilled in the art after reading the teachings of the present application, and such equivalents are intended to fall within the scope of the application as defined in the appended claims.

The terms related to the present application are explained as follows:

(1) Shared memory: shared memory is a memory local to a program, as opposed to local memory used in a program, which is typically used by programs, and which is private and accessible only by owned programs. The shared memory is not a specific program, but is managed by the system and can be accessed by a plurality of programs at the same time.

(2) Hash algorithm, chinese name Hash algorithm. For converting information of arbitrary length into an integer. Different information may get the same integer, but better hash algorithms avoid this to the greatest extent possible. When the information is stored, if the information is directly stored in a certain position according to the converted integer, the stored position can be quickly positioned according to the hash value of the information during searching. Even though the same hash value may correspond to multiple pieces of information, the speed of searching is greatly increased.

Based on the above noun definition, the relational memory database system provided by the application specifically comprises the following contents:

the relational memory database system provided by the application provides an external access interface 001 for an application program to access data in a shared memory. Since the data is stored in the shared memory, different applications can access the same data through the external access interface 001. The external access interface 001 provides functions by calling the data line management module 002, the index management module 003, and the table definition management module 004, all of which need to call the functions of the memory block management module 005. The modules of the present application are described below in order from the bottom layer to the top layer.

Memory block management module 005: the module is used for managing the shared memory in the operating system. All data in the relational memory database system are stored in the shared memory, so that the management of the shared memory is the basis of all other modules.

Since it is not known in advance how much memory the application will use to store data, the memory block management module 005 applies for only 32M of shared memory to the operating system at a time. When it is insufficient, another block of shared memory 32M is applied. The function of the memory block management module 005 is to provide an interface for the upper module, so that the upper module does not need to care about how many blocks of shared memory of the operating system are used, and as in the case of operating the local memory, the shared memory blocks are unified into a unified and continuous memory address to apply for access. The following is shown: operating system shared memory block→unified shared memory address.

The unified shared memory address contains 2 parts: shared memory block numbers and shared memory block internal addresses. In this way, the upper module may use a unified shared memory address to reference a location of a shared memory block of an operating system.

When the upper module accesses a certain unified shared memory address, the memory block management module 005 firstly checks the shared memory block number in the unified shared memory address, and if the shared memory block corresponding to the shared memory block number is not loaded with the local process address, the shared memory block is loaded first. And then acquiring a real memory address for access according to the internal address of the shared memory block in the unified shared memory address.

When the upper module applies for a new memory block address, the memory block management module 005 maintains information about a current shared memory block and its memory allocation. If the current shared memory block has residual space available for allocation, the method returns directly according to the number and the memory allocation position of the current shared memory block. If the remaining space is not sufficient to allocate a new memory block, the memory block management module 005 creates a new shared memory block for use.

Data line management module 002: the module is used to manage all data rows in the relational memory database system. The basic data components of a relational database are rows of data. The hierarchical relationship is that: database instance- > table- > data line, i.e.: each database instance contains multiple tables, each table containing multiple rows of data, and each row of data containing multiple columns of attributes. The data row is the basic unit of database management data.

The data line management module 002 applies for the memory by using the memory block management module 005, and initializes the applied memory to obtain a data line. One data line is represented by the unified shared memory address applied in the memory block management module 005. The data line management module 002 performs read-write operation on the applied unified shared memory address, so as to complete the read-write operation of the data line.

The rows of data are associated, for example, all rows of a data table form a set of rows of data. It is sometimes necessary to traverse this set. When the data line management module 002 applies for the memory of the data line, not only applies for the memory of the data line, but also applies for some additional space to place management information. There are link bits used to link different rows of data together as shown in fig. 6.

In fig. 6, there are 3 rows of data { data row 1, data row 2, data row 3}.

Each data line is divided into three parts of a management data field 010, a link bit field 011, and a data line data field 012. The management data field 010 stores therein some basic management information such as a table number to which the data line belongs, a data line size, the number of link bits, and the like. The actual data is stored in a row data field 012. Different data lines are linked to each other into different sets by a link bit field 011.

In fig. 6, the data line sets { data line 1, data line 2, data line 3} are linked by the link bit 1, and the other data line sets { data line 1, data line 3} are linked by the link bit 2.

Index management module 003: for managing queries of data rows. Each data row has a plurality of columns of attributes, and the data rows meeting the attributes are sometimes required to be queried according to a plurality of columns of attributes in all the data rows of a certain table. The columns of attributes of these queries together are referred to as the query key. For example: each row of the user table describes information of one user, and each row of the data has a plurality of columns to describe information of the user, such as: user number, user name, phone number, identification number, mailing address, etc. The user may be queried through various columns, such as { user number }, { phone number + identification card number }, each set of such query columns being referred to as a query key.

Typically, the data is queried by a row-by-row comparison method. However, when there is a lot of data, this approach is too slow, and the present application accelerates queries by indexing rows of table data. The specific method comprises the following steps:

1. when creating a table, indexes may be created for some columns through the external access interface 001, and information of the table and indexes is stored in the table definition management module 004. For each index, a chunk array is applied in the shared memory for storing the data of the hash bucket 020.

2. When inserting data, the data row management module 002 applies for new row data, where the row data includes n+1 link bits 011 (N is the number of indexes of the data table), each index uses one link bit 011, and all table data are concatenated using 1 link bit 011, for enumerating all data of the table.

3. After the data lines are initialized, the 0 th link bit is used to link all the data lines in the current data table together.

4. For each index i in the table: first, hash operation is performed on the key value data of the data column corresponding to the index i (i.e., a series of values are converted into an integer by some algorithm, such as CRC 32), and then the data line is linked at a certain position in the hash bucket 020 according to the result of the hash operation.

Thus, an index is constructed as shown in fig. 7.

When looking up a data row in a table according to some columns, the following steps are followed:

1. checking whether the searched column contains a component column of a certain index, and if not, searching row by using a common method.

2. If the searched column contains the component columns of a certain index, an integer hash value is calculated for the component columns of the index by using the same hash method as the insertion.

3. According to the hash value, a certain hooking position is located in the hash bucket 020, and as one hash bucket 020 often has a plurality of hooking positions, after the hash bucket 020 is located at a certain hooking position, data to be compared are very less, so that the searching speed is greatly increased.

Table definition management module 004: in the above description, it has been observed that the information of the table and its index also needs to be kept in shared memory. The table definition management module 004 stores and manages table information, table data column information, and table index information in the shared memory in the relational memory database system. The external access interface 001 creates a table and its data column for the application software by calling the function of the table definition management module 004 to provide an interface to the outside.

The table definition management module 004 applies for shared memory blocks using the memory block management module 005 when managing table definition information. All information starts with a table information handle array and an index information handle array. The elements in each table information handle array store a unified shared memory address pointing to the shared memory block created by the memory block management module 005, and store definition information of a table therein. The same mechanism is used for the index information handle array.

Claims

1. A relational memory database system, wherein all data in the relational memory database system is stored in a shared memory, and an application program accesses the data in the shared memory through an external access interface provided by the relational memory database system;

step 301: when the data table is created, M indexes are created for the attribute columns of the current data table through an external access interface, M is more than or equal to 1, and information of the data table and information of the indexes are stored in a table definition management module;

step 402: for the attribute columns of the created index, calculating an integer hash value by using the same hash method in the inserting in the step 304, and searching by using other attribute columns except for the attribute columns of the created index in the step 401 based on all data rows of the current link position after the corresponding link position is positioned in the hash bucket according to the integer hash value;

2. The relational memory database system of claim 1 wherein the memory block management module applies for shared memory to the operating system if current shared memory is insufficient, and the memory block management module applies for only shared memory of a fixed size to the operating system at a time.

3. The relational memory database system according to claim 1, wherein the set of different data rows is defined as a data row set, all data rows of a same data table can form one data row set, and data rows belonging to a same data table can form a plurality of data row sets, and when the data row management module applies for the memory of the current data row, if the current data row belongs to N different data row sets, N is greater than or equal to 1, the management information includes N data table link bits, the different data rows are linked to each other into N different data row sets through the data table link bits, and the different data rows are linked together through the N data table link bits to form an N data row set, n=1, …, N.