CN113157692A

CN113157692A - Relational memory database system

Info

Publication number: CN113157692A
Application number: CN202110175982.8A
Authority: CN
Inventors: 朱悬宁; 张锐; 单兰山
Original assignee: Shanghai Wandehonghui Information Technology Co ltd
Current assignee: Shanghai Wandehonghui Information Technology Co ltd
Priority date: 2021-02-07
Filing date: 2021-02-07
Publication date: 2021-07-23
Anticipated expiration: 2041-02-07
Also published as: CN113157692B

Abstract

The invention provides a relational memory database system, which can continue the memory database of the relational database operation of the traditional database, and the memory database is directly accessed through the shared memory of the system, thereby greatly improving the reading and writing speed and being particularly suitable for the application with intensive calculation. The invention constructs the basic relational database function in the shared memory through the data row management module and the table definition management module. According to the method and the device, various column attribute queries are automatically accelerated through the index management module, and various key values do not need to be manually constructed to accelerate access like Redis. According to the invention, all data are stored in the shared memory through the function of the memory block management module for being read and written by each application program, so that the application performance is accelerated, especially the calculation-intensive application.

Description

Relational memory database system

Technical Field

The invention relates to a relational memory database system, which is suitable for the computer software and information technology service industry and belongs to the technical field of data caching.

Background

Software programs typically involve read and write operations of data. Many systems operate on disks as media, as shown in fig. 1, resulting in slower operating speeds of the systems. As the memory capacity increases and the price decreases, many programs store data in the memory for read/write operations, as shown in fig. 2, to accelerate the system operation speed. This creates a need for in-memory database systems.

As shown in fig. 3, a Remote Dictionary service (Remote Dictionary service) is a very popular in-memory database, and is used in many software systems. The basic principle is to provide a network accessible service that allows clients to store key-value pairs and to read data based on key values. The method is characterized in that:

1. the remote service, i.e., the Redis service and the program using the service may be located in different machines through a network as long as they can communicate through the network.

2. The remote service may store key-value pairs. A key-value pair is a pair of data comprising a key and a value. A key may be a string and a value may be a string or more complex structure. Such as: { "user 001", "Xiaoming" } is a key-value pair, "user 001" is a key, and "Xiaoming" is the corresponding value.

When the client stores such a key-value pair in Redis, the key "user 001" can be sent again to read its corresponding value: "Xiaoming". This key value may also be modified.

Redis's data assumes a key-value schema, but in practice traditional databases tend to be relational databases. A relational database refers to a database that uses a relational model to organize data, and stores data in rows and columns for a user to understand, and a series of rows and columns of the relational database are called tables, and a set of tables constitutes the database. A user retrieves data in a database by a query, which is an executable code that defines certain areas in the database. The relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and relations between them. The most obvious point of the relational database is that the data is stored not by the key-value mode but by the table. For example, the following is a table:

the data of each table in the relational data is composed of more basic units: and (4) forming data lines. In the above table, each row is a data row except the header. A data line may have multiple attributes, some of which may be used as an index. For example, in the above table, the eligible data lines may be queried according to the attributes [ customer number ], [ name ], or [ mobile phone number ], and these attributes for querying are referred to as indexes.

Redis exists in several unsatisfactory places:

redis is a key-value schema based in-memory database, whereas file database systems used in traditional software development are typically all relational databases. This brings great inadaptation to developers, and is not easy to correspond to data of the file database so as to perform data persistence operation.

2. Databases based on key-value schemas are also far less powerful than relational databases when used. Especially the same value, often needs to be searched by multiple attributes, and the database based on the key-value mode needs to manually construct an index supported by the relational database itself through an additional key value.

Redis is a memory database, access is performed through network requests, and for compute-intensive applications, one operation may include many times of data reading and writing, and data reading/writing of Redis through the network is required each time, which may cause serious reduction in operation performance.

Disclosure of Invention

The purpose of the invention is: the memory database can continue the operation of the relational database of the traditional database, is directly accessed through the shared memory of the system, greatly improves the reading and writing speed, and is particularly suitable for intensive calculation application.

In order to achieve the above object, the technical solution of the present invention is to provide a relational memory database system, which is characterized in that all data in the relational memory database system are stored in a shared memory, and an application program accesses the data stored in the shared memory through an external access interface provided by the relational memory database system;

the relational memory database system further comprises a data line management module, an index management module, a table definition management module and a memory block management module; the external access interface provides functions by calling a data line management module, an index management module and a table definition management module, and the data line management module, the index management module and the table definition management module call a memory block management module to realize access to a certain unified shared memory address or application of a new memory block address, wherein:

the memory block management module is used for managing shared memories in an operating system, unifying all the shared memory blocks into a unified and continuous memory address to apply for access, defining the unified and continuous memory address as a unified shared memory address, wherein the unified shared memory address comprises a shared memory block number and a shared memory block internal address, and the shared memory block numbers correspond to the shared memory blocks one to one;

when the data row management module, the index management module or the table definition management module accesses a certain unified shared memory address, the memory block management module performs the following processing:

step 101: checking the number of the shared memory block in the current unified shared memory address, judging whether the shared memory block corresponding to the number of the shared memory block is loaded into the local process address, if so, entering step 103, otherwise, entering step 102;

step 102: loading the shared memory block corresponding to the current shared memory block number into the local process address, and entering step 103;

step 103: acquiring a real memory address for accessing according to an internal address of a shared memory block in the current unified shared memory address;

the memory block management module maintains information of all shared memory blocks and memory allocation conditions of the shared memory blocks; when the data row management module, the index management module or the table definition management module applies for the memory from the memory block management module, the memory block management module feeds back a new unified shared memory address, which comprises the following steps:

step 201: judging whether the current shared memory block has a remaining space for allocation according to the memory allocation condition of the current shared memory block, if so, entering step 202, and if not, entering step 203;

step 202: using the number of the current shared memory block as the number of the shared memory block, using the memory allocation position of the current shared memory block as the internal address of the shared memory block, and feeding back the unified shared memory address comprising the number of the shared memory block and the internal address of the shared memory block to the data line management module, the index management module or the table definition management module;

step 203: the memory block management module creates a new shared memory block, takes the number of the newly created shared memory block as the number of the shared memory block, takes the memory allocation position of the newly created shared memory block as the internal address of the shared memory block, and feeds back the unified shared memory address comprising the number of the shared memory block and the internal address of the shared memory block to the data line management module, the index management module or the table definition management module;

the data line management module is used for managing all data lines in the relational memory database system; the data line management module utilizes the memory block management module to apply for memories for storing data and management information, initializes the applied memories to obtain a data line, and uniformly manages all the data lines belonging to the same data table based on the management information; one data line is represented by a uniform shared memory address applied in a memory block management module; the data line management module performs read-write operation on the applied unified shared memory address so as to complete the read-write operation of the data line;

the index management module is used for managing the query of data rows, wherein each data row has a plurality of attribute columns, when in query, all data rows of a certain data table query the row data meeting the attributes according to the attributes of the K-column attribute columns, and K is more than or equal to 1, the index management module adopts the following steps to manage the query of the data rows:

step 301: when a data table is created, M indexes are created for a column attribute column of the current data table through an external access interface, wherein M is more than or equal to 1, and information of the data table and information of the indexes are stored in a table definition management module;

for each index, applying for an index information handle array in the shared memory, wherein elements in each index information handle array store a uniform shared memory address pointing to a shared memory block of the data of the hash bucket created by the memory block management module for storage;

for a data table, applying for a table information handle array in a shared memory, wherein elements in each table information handle array store a uniform shared memory address and point to a shared memory block which is created by a memory block management module and used for storing information of the data table;

step 302: when data is inserted, a data line management module is used for applying for new line data, wherein the line data comprises M index link bits, and M indexes respectively use one index link bit;

step 303: after the data rows are initialized, all the data rows in the current data table are linked together by the management information;

step 304: for the mth index in the current data table, M is 1, …, M: performing hash operation on the key value data of the attribute column corresponding to the mth index, and then linking data rows to corresponding hanging positions in a hash bucket according to hash operation results, wherein the data rows with the same hash operation results are linked to the same hanging positions in the hash bucket, and the data rows with different hash operation results are linked to different hanging positions in the hash bucket;

when looking up the corresponding data row in the current data table according to some attribute columns, the method comprises the following steps:

step 401: judging whether the searched attribute column contains the attribute column of the index created in the step 301, if not, searching line by line, and if so, entering the step 402;

step 402: for the attribute column with the index created, an integer hash value is calculated by using the same hash method during insertion in step 304, and after the corresponding hook position is located in the hash bucket according to the integer hash value, based on all data rows linking the current hook position, the other attribute columns except the attribute column with the index created in step 401 are used for searching.

The table definition management module stores and manages table information, table data column information and table index information in the relational memory database system in a shared memory, and an external access interface provides an interface for external use by calling the function of the table definition management module, and provides for software to create a data table and M indexes for an M-column attribute column of the current data table; the table definition management module applies for the memory from the memory block management module and is used for storing the table information handle array and the index information handle array.

Preferably, if the current shared memory is not enough, the memory block management module applies for the shared memory to the operating system, and the memory block management module only applies for the shared memory of a fixed size to the operating system each time.

Preferably, a set composed of different data rows is defined as a data row set, all data rows of the same data table may form one data row set, and data rows belonging to the same data table may form a plurality of data row sets, so that when the data row management module applies for the memory of the current data row, if the current data row belongs to N different data row sets, N is greater than or equal to 1, the management information includes N data table linking bits, different data rows are linked with each other into N different data row sets through the data table linking bits, different data rows are linked together through the nth data table linking bit to form an nth data row set, N is 1, …, N.

Compared with the prior art, the invention has the following advantages:

1. the basic relational database function is constructed in the shared memory through the data row management module and the table definition management module. Developers are more adaptable to this approach than to in-memory databases of key-value patterns. And can correspond to the data of the traditional relational file database one to one.

2. Various column attribute queries are automatically accelerated by the index management module without manually constructing various key values to accelerate access as in Redis.

3. All data are stored in the shared memory for being read and written by each application program through the function of the memory block management module, so that the performance of the application is accelerated, and particularly the calculation-intensive application is accelerated.

Drawings

FIG. 1 is a schematic diagram of a conventional system for performing data operations on a magnetic disk;

FIG. 2 is a diagram illustrating a conventional system for storing data in a memory for read/write operations;

FIG. 3 is a schematic diagram of a Redis system;

FIG. 4 is a diagram illustrating an application architecture of the present invention;

FIG. 5 is a block diagram of the system of the present invention;

FIG. 6 is an example of linking rows of blind data together by a link bit;

FIG. 7 is a construction index example.

Detailed Description

The invention will be further illustrated with reference to the following specific examples. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the present invention. Further, it should be understood that various changes or modifications of the present invention may be made by those skilled in the art after reading the teaching of the present invention, and such equivalents may fall within the scope of the present invention as defined in the appended claims.

The terms referred to in the present invention are explained below:

(1) sharing the memory: shared memory is relative to local memory used in programs, which typically use memory local to the program and are private and accessible only to the owning program. The shared memory is not associated with any specific program, but is associated with system management, and can be accessed by a plurality of programs simultaneously.

(2) Hash algorithm, Chinese name Hash algorithm. For converting information of arbitrary length into an integer. Different messages may get the same integer, but a better hash algorithm avoids this as much as possible. When the information is stored, if the converted integer is directly stored in a certain position of the storage, the stored position can be quickly positioned according to the hash value of the information during searching. Even if the same hash value may correspond to a plurality of pieces of information, the searching speed is greatly increased.

Based on the above noun definitions, the relational memory database system provided by the present invention specifically includes the following contents:

the relational memory database system provided by the invention provides an external access interface 001 for an application program to access data stored in a shared memory. Since the data is stored in the shared memory, different applications can access the same piece of data through the external access interface 001. External access interface 001 provides functionality by calling data line management module 002, index management module 003, table definition management module 004, all of which require the calling of the functionality of memory block management module 005. The modules of the present invention are described below in order from the bottom layer to the top layer.

Memory block management module 005: the module is used to manage shared memory in an operating system. All data in the relational memory database system are stored in the shared memory, so that management of the shared memory is the basis of all other modules.

Since it is not known in advance how much memory the application will use to store data, the memory block management module 005 only applies for 32M of shared memory to the operating system each time. When the memory is not enough, another 32M shared memory is applied. The memory block management module 005 is used for providing an interface to the upper module, so that the upper module does not need to care how many blocks of shared memory of the operating system are used, but unifies the shared memory blocks into a uniform and continuous memory address to apply for access, like operating a local memory. As follows: operating system shared memory block → unified shared memory address.

The unified shared memory address contains 2 parts: shared memory block number, shared memory block internal address. In this way, the upper module can use a unified shared memory address to refer to a location of a shared memory block of an operating system.

When the upper module accesses a certain unified shared memory address, the memory block management module 005 first checks the number of the shared memory block in the unified shared memory address, and if the shared memory block corresponding to the shared memory block number is not loaded into the local process address, the shared memory block is loaded first. And then acquiring a real memory address for accessing according to the internal address of the shared memory block in the unified shared memory address.

When the upper module applies for a new memory block address, the memory block management module 005 maintains information of a current shared memory block and its memory allocation status. And if the current shared memory block has residual space for allocation, returning the residual space directly according to the number of the current shared memory block and the memory allocation position. If the remaining space is not enough to allocate a new memory block, the memory block management module 005 creates a new shared memory block for use.

The data line management module 002: the module is used to manage all data rows in the relational memory database system. The basic data component of a relational database is a row of data. The hierarchical relationship is as follows: database instance- > table- > data row, i.e.: each database instance contains multiple tables, each table containing multiple rows of data, and each row of data containing multiple columns of attributes. The data row is the basic unit of database management data.

The data line management module 002 uses the memory block management module 005 to apply for the memory, and initializes the memory to obtain a data line. One data line is represented by the unified shared memory address applied in the memory block management module 005. The data line management module 002 performs read-write operation on the applied unified shared memory address, so as to complete the read-write operation of the data line.

There is a relationship between the data rows, for example, all rows of a data table form a set of data rows. Sometimes it is necessary to traverse this set. When the data line management module 002 applies for the memory of the data line, it not only applies for the memory of the data, but also applies for some extra space to place the management information. There are link bits for linking different rows of data together, as shown in fig. 6.

In fig. 6, 3 lines of data { data line 1, data line 2, data line 3} are in total.

Each data line is divided into three parts, a management data field 010, a link bit field 011 and a row data field 012. The management data field 010 stores therein some basic management information, such as a table number to which the data line belongs, a size of the data line, a number of link bits, and the like. The actual data is stored in the row data field 012. Different rows of data are linked to each other into different sets by link bit field 011.

In fig. 6, a set of data lines { data line 1, data line 2, data line 3} is linked by a link bit 1, and another set of data lines { data line 1, data line 3} is linked by a link bit 2.

The index management module 003: for managing queries of rows of data. Each data row has multiple columns of attributes, and sometimes, the data rows satisfying the attributes need to be queried according to the attributes of some columns in all the data rows of a certain table. The attribute columns of these queries are grouped together and referred to as query keys. For example: each row of data of the user table describes information of one user, and each row of data has a plurality of columns to describe information of the user, such as: user number, user name, telephone number, identification number, mailing address, and the like. The user may be queried through various columns, such as { user number }, { telephone number + identification number }, each set of such query columns being referred to as a query key.

Generally, when data is queried, the data is compared line by line. However, when there is a lot of data, this approach is too slow and the present invention speeds up the lookup by indexing rows of table data. The specific method comprises the following steps:

1. when creating a table, indexes can be created for some columns through the external access interface 001, and information of the table and the indexes is stored in the table definition management module 004. For each index, an array is applied in the shared memory for storing the data of the hash bucket 020.

2. When inserting data, the data line management module 002 applies for a new line of data, where the line of data will contain N +1 link bits 011(N is the number of indexes of the data table), each index uses one link bit 011, and all table data are concatenated using 1 link bit 011 for enumerating all table data.

3. After the data row is initialized, the 0 th linking bit is used to link all the data rows in the current data table together.

4. For each index i in the table: the key-value data of the data column corresponding to the index i is first hashed (i.e. a series of values are converted into an integer by an algorithm such as CRC 32), and then the data row is linked to a position in the hash bucket 020 according to the hash result.

This constructs the index as shown in FIG. 7.

When looking up rows of data in a table according to some columns, look up is done according to the following steps:

1. and checking whether the searched column contains a composition column of a certain index, and if not, searching line by using a common method.

2. If the column being looked up contains the constituent columns of an index, an integer hash value is calculated for the constituent columns of the index using the same hash method as for the insertion.

3. According to the hash value, a certain hanging position is located in the hash bucket 020, and due to the fact that one hash bucket 020 often has a plurality of hanging positions, after the certain hanging position is located, data to be compared are few, and therefore the searching speed is greatly accelerated.

The table definition management module 004: in the above description, it has been seen that the information of the table and its index also needs to be stored in the shared memory. The table definition management module 004 stores and manages the table information, the data column information of the table, and the index information of the table in the relational memory database system in the shared memory. The external access interface 001 provides an interface to the outside by calling the function of the table definition management module 004, creates a table and its data column for application software, and creates an index column for the table.

The table definition management module 004 also applies for the shared memory block using the memory block management module 005 when managing the table definition information. All information begins with the table information handle array and the index information handle array. Each element in the table information handle array stores a uniform shared memory address, which points to the shared memory block created by the memory block management module 005, and stores therein table definition information. The same mechanism is used for index information handle arrays.

Claims

1. A relational memory database system is characterized in that all data in the relational memory database system are stored in a shared memory, and an application program accesses the data stored in the shared memory through an external access interface provided by the relational memory database system;

step 301: when a data table is created, M indexes are created for the attribute column of the current data table through an external access interface, wherein M is more than or equal to 1, and the information of the data table and the information of the indexes are stored in a table definition management module;

2. The relational memory database system according to claim 1, wherein if the current shared memory is not enough, the memory block management module applies for a shared memory to the operating system, and the memory block management module only applies for a fixed-size shared memory to the operating system each time.

3. A relational memory database system according to claim 1, wherein a set composed of different data rows is defined as a data row set, all data rows of the same data table may constitute one data row set, data rows belonging to the same data table may constitute a plurality of data row sets, the data row management module, when applying for the memory of the current data row, if the current data row belongs to N different data row sets, N ≧ 1, the management information includes N data table linking bits, different data rows are linked to each other by the data table linking bits into N different data row sets, different data rows are linked together by the nth data table linking bit to form an nth data row set, N ═ 1, …, N.