CN115269609A

CN115269609A - Non-relational database generation and query method and device based on skip list data

Info

Publication number: CN115269609A
Application number: CN202210992575.0A
Authority: CN
Inventors: 钟悦; 林镇熙; 尹强; 黄熠珂
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2022-08-18
Filing date: 2022-08-18
Publication date: 2022-11-01

Abstract

The invention can be used in the technical field of application of big data technology in finance, and provides a method and a device for generating and querying a non-relational database based on skip list data, wherein the method for generating the non-relational database based on skip list data comprises the following steps: generating a query index array according to a plurality of historical query expressions of the relational source database; determining data corresponding to the query index array in the relational source database; and generating a non-relational database according to the data corresponding to the query index array and the skip list model. The method and the device for generating and querying the non-relational database based on the skip list data fully utilize the characteristics of the non-relational database, and utilize a specific interface to finish operations such as quick retrieval, classification, range query and the like on a local dump data set on the premise of ensuring high-efficiency query and only sacrificing a small part of local storage space, thereby efficiently finishing tasks such as data analysis and the like.

Description

Method and device for generating and querying non-relational database based on skip list data

Technical Field

The application belongs to the technical field of big data, and particularly relates to a method and a device for generating and querying a non-relational database based on skip list data.

Background

In the prior art, computer storage technologies are diverse, and for general application development, most of the traditional relational databases are the first choice for development because of the advantages of easy understanding, convenient use and the like. But as time goes on and computer technology has developed dramatically, the user's scene of using data has changed from day to day. Particularly, with the advent of the era of big data, cloud computing and artificial intelligence, the storage capacity of user and business data is multiplied, and the problem of the traditional relational database is gradually highlighted in certain application scenes. For example, the performance of reading and writing mass data is not advantageous, and especially in a local stand-alone scenario, it is difficult for users to meet the requirements in terms of time efficiency by using a traditional database to store and query mass data.

The advent of non-relational databases (NoSQL) has allowed this situation to improve because it naturally has well-organized structured data structures. Generally speaking, a non-relational database has excellent read-write performance without being particularly optimized, so that a single computer can exert maximum efficiency and realize storage and query of a large amount of data, and data analysis and machine learning can be completed with less resources. Typical NoSQL implementations are further divided into key value storage, document storage column family storage, graph storage, and other storage.

However, in some scenarios, such as when data with time and contract ID as elements needs to be queried, the data source to be analyzed is generally from a traditional relational database, whereas for relational data, the data is generally represented as a two-dimensional table. And for the data sets in the data tables, one or more indexes are established when the tables are initially created for quickly searching one or more data with similar attributes in the table sets. In a relational database, data within a limited range can be conveniently found through structured query statements. If the data is dumped into the NoSQL database, due to the data structure characteristics of the levelDB, if the data is not processed and modified, the data of a single designated key value or a data set larger than the designated key value can be obtained only through a set interface. Data in a limited range cannot be searched through an index condition, so that the potential value of the data cannot be fully mined and exerted through a technical means.

Disclosure of Invention

The invention can be used in the technical field of large data technology application in finance, and can also be used in any field except the finance field.

In order to solve the technical problems in the background art of the application, the invention provides the following technical scheme:

in a first aspect, the present invention provides a method for generating a non-relational database based on skip list data, where the method includes:

generating a query index array according to a plurality of historical query expressions of the relational source database;

determining data corresponding to the query index array in the relational source database;

and generating a non-relational database according to the data corresponding to the query index array and the skip list model.

In an embodiment, the generating a query index array according to a plurality of historical query expressions of a relational source database includes:

parsing the plurality of historical query expressions to determine a plurality of query terms;

sequencing the plurality of query keywords according to the query sequence of the plurality of historical query expressions, and generating a keyword array according to the sequenced plurality of query keywords;

and generating the query index array according to a plurality of historical query expressions and the keyword array.

In an embodiment, the determining, in the relational source database, the data corresponding to the query index array includes:

and acquiring data of columns or rows where a plurality of query keywords are located in the relational source database.

In an embodiment, the generating a non-relational database according to the data corresponding to the query index array and the skip list model includes:

and inserting the data corresponding to the query index array into a skip list model according to a preset sequence to generate a non-relational database.

In a second aspect, the present invention provides a method for querying a non-relational database based on skip list data in the non-relational database, the method comprising:

in response to the received query expression, sorting the plurality of query keywords according to the priority of the operational characters to generate a query index array;

converting the name of the target data according to the query expression and the query index array;

inquiring the target data in a non-relational database according to the name of the converted target data; wherein the query keyword, target data name, and operator are generated by parsing the query expression.

In an embodiment, the converting the name of the target data according to the query expression and the query index tuple includes:

and splicing the elements in the query index array according to the priority order of the operators to generate the name of the converted target data.

In an embodiment, the querying in the non-relational database according to the name of the converted target data includes:

analyzing the query expression to determine a value array corresponding to the query expression;

determining an index prefix corresponding to the query expression according to the elements in the value array;

determining a data iterator corresponding to the query expression according to the index prefix;

and querying the target data in the non-relational database according to the name of the converted target data by using the data iterator.

In a third aspect, the present invention provides a device for generating a non-relational database based on skip list data, the device comprising:

the index array generating module is used for generating a query index array according to a plurality of historical query expressions of the relational source database;

the data determining module is used for determining data corresponding to the query index array in the relational source database;

and the non-relational database generation module is used for generating a non-relational database according to the data corresponding to the query index array and the skip list model.

In one embodiment, the index array generation module comprises:

a keyword determination unit for parsing the plurality of historical query expressions to determine a plurality of query keywords;

the keyword array generating unit is used for sequencing the plurality of query keywords according to the query sequence of the plurality of historical query expressions and generating a keyword array according to the sequenced plurality of query keywords;

and the index array generating unit is used for generating the query index array according to a plurality of historical query expressions and the keyword array.

In one embodiment, the data determination module comprises:

and the data determining unit is used for acquiring data of columns or rows where the plurality of query keywords are located in the relational source database.

In one embodiment, the non-relational database generation module comprises:

and the non-relational database generating unit is used for inserting the data corresponding to the query index array into the skip list model according to a preset sequence so as to generate a non-relational database.

In a fourth aspect, the present invention provides a device for querying a non-relational database based on skip list data, the device comprising:

the index array generating module is used for responding to the received query expression and sequencing the query keywords according to the priority of the operational characters so as to generate a query index array;

the name conversion module is used for converting the name of the target data according to the query expression and the query index array;

the data query module is used for querying the target data in the non-relational database according to the name of the converted target data; wherein the query keyword, target data name, and operator are generated by parsing the query expression.

In one embodiment, the name conversion module includes:

and the name generation unit is used for splicing the elements in the query index array according to the priority order of the operational characters so as to generate the name of the converted target data.

In one embodiment, the data query module comprises:

a prefix determining unit, configured to determine, according to the element, an index prefix corresponding to the query expression;

an iterator determining unit, configured to determine, according to the index prefix, a data iterator corresponding to the query expression;

and the data query unit is used for querying the target data in the non-relational database according to the name of the converted target data by using the data iterator.

In a fifth aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the skip list data-based non-relational database generation and query method when executing the program.

In a sixth aspect, the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of a method for non-relational database generation, querying based on skip list data.

As can be seen from the above description, an embodiment of the present invention provides a method and an apparatus for generating and querying a non-relational database based on skip list data, where the corresponding method for generating a non-relational database based on skip list data includes: firstly, generating a query index array according to a plurality of historical query expressions of a relational source database; then, determining data corresponding to the query index array in a relational source database; and finally, generating a non-relational database according to the data corresponding to the query index array and the skip list model.

The non-relational database query method based on skip list data comprises the following steps: firstly, in response to a received query expression, sequencing a plurality of query keywords according to the priority of an operational character to generate a query index array; then, converting the name of the target data according to the query expression and the query index array; finally, inquiring the target data in a non-relational database according to the name of the converted target data; wherein the query keyword, the target data name, and the operator are generated by parsing the query expression.

The invention fully utilizes the characteristics of the non-relational database, and completes the operations of quick retrieval, classification, range query and the like on the local dump data set by utilizing a specific interface on the premise of ensuring high-efficiency query and only sacrificing a small part of local storage space, thereby efficiently completing tasks such as data analysis and the like.

Drawings

In order to more clearly illustrate the embodiments or technical solutions of the present invention, the drawings used in the embodiments or technical solutions in the prior art are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.

FIG. 1 is a schematic flow chart illustrating a method for generating a non-relational database based on skip list data according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a step 100 of a method for generating a non-relational database based on skip list data according to an embodiment of the present invention;

FIG. 3 is a flowchart illustrating a step 200 of a method for generating a non-relational database based on skip list data according to an embodiment of the present invention;

FIG. 4 is a flowchart illustrating a step 300 of a method for generating a non-relational database based on skip list data according to an embodiment of the present invention;

FIG. 5 is a first flowchart illustrating a method for querying a non-relational database based on skip list data according to an embodiment of the present invention;

FIG. 6 is a flowchart illustrating step B of a non-relational database query method based on skip list data according to an embodiment of the present invention;

FIG. 7 is a flowchart illustrating step C of a non-relational database query method based on skip list data according to an embodiment of the present invention;

FIG. 8 is a flowchart illustrating a second non-relational database query method based on skip list data according to an embodiment of the present invention;

FIG. 9 is a flowchart illustrating a method for generating a non-relational database based on skip list data according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of a two-dimensional array according to an embodiment of the present invention;

FIG. 11 is a flowchart illustrating a method for querying a non-relational database based on skip list data according to an embodiment of the present invention;

FIG. 12 is a schematic structural diagram of a JSON array for data dump in an embodiment of the present invention;

FIG. 13 is a block diagram of an apparatus for generating a non-relational database based on skip list data according to an embodiment of the present invention;

FIG. 14 is a block diagram of the index array generation module 10 according to an embodiment of the present invention;

FIG. 15 is a block diagram of the data determination module 20 according to an embodiment of the present invention;

FIG. 16 is a block diagram of the non-relational database generation module 30 in an embodiment of the invention;

FIG. 17 is a block diagram of a non-relational database lookup apparatus based on skip list data according to an embodiment of the present invention;

FIG. 18 is a block diagram of a name translation module B according to an embodiment of the present invention;

FIG. 19 is a block diagram of a data query module C according to an embodiment of the present invention;

fig. 20 is a schematic structural diagram of an electronic device in an embodiment of the invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

It should be noted that the terms "comprises" and "comprising," and any variations thereof, in the description and claims of this application and the above-described drawings, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. The embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

According to the technical scheme, the data acquisition, storage, use, processing and the like meet relevant regulations of national laws and regulations.

Based on the technical pain point in the prior art, an embodiment of the present invention provides a specific implementation of a method for generating a non-relational database based on skip list data, and referring to fig. 1, the method specifically includes the following steps:

step 100: generating a query index array according to a plurality of historical query expressions of the relational source database;

it is understood that a relational database refers to a database that uses a relational model to organize data, and stores data in rows and columns for a user to understand, and the rows and columns of the relational database are called tables, and a group of tables constitutes the database. A user retrieves data in a database by a query, which is an executable code that defines certain areas in the database. The relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and relations between the two-dimensional tables.

Preferably, the relational source database comprises: mySQL, oracle, SQLServer, or csv. In addition, the query index array in step 100 can be understood as the query condition of the relational source database.

Step 200: and determining data corresponding to the query index array in the relational source database.

Taking the column name of the data source table as an index (row name), explaining step 200, taking the element in the query index data as a keyword, locating in the column header, and obtaining the row data in a plurality of columns, where the row data is the data corresponding to the query index array in step 200.

Step 300: and generating a non-relational database according to the data corresponding to the query index array and the skip list model.

Skip list models (skip lists), also known as skip lists, are a typical implementation of NoSQL key value storage. The average time complexity of the sequential lookup and insert operations containing n elements is O (logn), which is better than the O (n) time complexity of the array. Essentially, the skip list model is a randomized data structure, and is an ordered linked list capable of performing binary search. The skip list is added with a multi-level index on the original ordered linked list, and the fast search is realized through the index. The skip list can not only improve the search performance, but also improve the performance of the insert and delete operations.

Skip list as a practical storage structure, its code implementation is also frequently present in various types of applications, SDKs, such as: IMAP server, QMap of QT, ordered collections of Redis, etc. The database storage structure is realized by the news DB, the skiDB, the levelDB and the like in sequence.

In a specific scene, in data processing application, the application dumps data originally in a relational database into a levelDB, transfers the data to the levelDB as the applied data or transfers the data to a local cache, a memory database, a file database and the like, completes operations such as storage, query, analysis and the like of the data locally according to indexes, and completes a certain task by utilizing the performance advantages of NoSQL.

Specifically, the data determined in step 200 is inserted into the skip list model in a certain order (initially, the content of the model is empty), i.e., a non-relational database is formed.

As can be seen from the above description, the method for generating a non-relational database based on skip list data according to the embodiment of the present invention includes: firstly, generating a query index array according to a plurality of historical query expressions of a relational source database; then, determining data corresponding to the query index array in a relational source database; and finally, generating a non-relational database according to the data corresponding to the query index array and the skip list model.

In a plurality of scenes (for example, data processing application is used for locally performing machine learning and mathematical modeling), massive relational data can be dumped to a plurality of local cache files in a single machine environment with limited resources according to requirements, single or a plurality of index combinations are specified in a limited range for searching and comparing through customizing query statements, and data in a jump table (level DB) database is effectively retrieved.

In some embodiments, referring to fig. 2, step 100 comprises:

step 101: parsing the plurality of historical query expressions to determine a plurality of query terms;

analyzing N different query expressions or analyzing the query expressions, screening the list names of the data source as indexes, and selecting M list names in the N queries as query keywords (index list names) respectively.

Step 102: sequencing the plurality of query keywords according to the query sequence of the plurality of historical query expressions, and generating a keyword array according to the sequenced plurality of query keywords;

specifically, the M index column names are arranged according to the occurrence order of the query conditions of the query expression, and after arrangement, an index column group with a length of M, that is, a keyword array, is generated.

And constitute a two-dimensional array with the number of queries N and the number of index column names max (M), i.e., called a query-index column array

Step 103: and generating the query index array according to a plurality of historical query expressions and the keyword array.

On the basis of step 102, the plurality of historical query expressions and the keyword array are combined into a two-dimensional array with the query number N and the index column name number max (M), that is, a query index array.

In some embodiments, referring to fig. 3, step 200 comprises:

step 201: and acquiring data of columns or rows where a plurality of query keywords are located in the relational source database.

In some embodiments, referring to fig. 4, step 300 comprises:

step 301: and inserting the data corresponding to the query index array into a skip list model according to a preset sequence to generate a non-relational database.

It is understood that non-relational databases (NoSQL: not only SQL), unlike relational databases, do Not guarantee the ACID nature of relational data. The database provides a mechanism for storing and retrieving data that is modeled in a manner other than in a tabular relationship used in relational databases.

NoSQL has the following advantages: the expansion is easy, and the NoSQL database has a plurality of varieties, but one common characteristic is that the relational characteristic of the relational database is removed. There is no relation between the data, so it is very easy to expand. Intangibles also bring scalability at the architectural level. The NoSQL database has very high read-write performance, and especially has excellent performance under the condition of large data volume. This benefits from its independence, the simple structure of the database.

When the step 301 is implemented, the method specifically comprises the following steps: when inserting the data corresponding to the query index array into the skip list model, an ordered set (non-relational database) is finally obtained according to the 'Alpha-beta' value sorting of the key values, preferably, in order to prevent the concatenated characters from being included in the skip list key sorting as 'Alpha-beta' characters, ASCII characters 'to' which are different from ordinary numbers and letters are used as the concatenated characters, and the characters have the largest sequence in the visible characters.

On the other hand, an embodiment of the present invention provides a specific implementation of the non-relational database query method based on skip list data, which is applied to the above non-relational database, and referring to fig. 5, the method specifically includes the following steps:

step A: in response to the received query expression, sorting the plurality of query keywords according to the priority of the operational characters to generate a query index array;

specifically, the operators are sorted according to their priorities, and the corresponding query keywords are sorted according to the order, so as to form an array, i.e., a query index array.

And B, step B: converting the name of the target data according to the query expression and the query index array;

and connecting the elements in the value array by using a preset symbol according to the priority of the operator, wherein the connected result is the name of the converted target data.

And C: inquiring the target data in a non-relational database according to the name of the converted target data; wherein the query keyword, target data name, and operator are generated by parsing the query expression.

And B, inquiring target data in the non-relational database by using a predetermined data iterator according to the converted name in the step B.

As can be seen from the above description, an embodiment of the present invention provides a method and an apparatus for querying a non-relational database based on skip list data, where the corresponding method for querying a non-relational database based on skip list data includes: firstly, in response to a received query expression, sequencing a plurality of query keywords according to the priority of an operational character to generate a query index array; then, converting the name of the target data according to the query expression and the query index array; finally, inquiring the target data in a non-relational database according to the name of the converted target data; wherein the query keyword, the target data name, and the operator are generated by parsing the query expression.

In some embodiments, referring to fig. 6, step B comprises:

step B1: and splicing the elements in the query index array according to the priority sequence of the operators to generate the name of the converted target data.

Specifically, elements in the analyzed query index array are spliced by using symbols 'to' according to the order of the priority of the operators so as to determine the name of the converted target data.

In some embodiments, referring to fig. 7, step C comprises:

step C1: analyzing the query expression to determine a value array corresponding to the query expression;

and step C2: determining an index prefix corresponding to the query expression according to the elements in the value array;

specifically, elements in the value array are spliced with the symbols "-" in the order of the priority of the operator to obtain the index prefix.

And C3: determining a data iterator corresponding to the query expression according to the index prefix;

it is to be understood that a data iterator is a data structure, which is an object that can traverse a collection, provides a common operation interface for various containers, and isolates traversal operations and underlying implementation of the containers, thereby decoupling them. Essentially, the data iterator is a tool used to iterate values, and the iteration is an activity of repeating the feedback process, and the purpose of the iteration is generally to approach a desired target or result, each repetition of the process is referred to as an "iteration", and the result obtained by each iteration is used as an initial value of the next iteration, and a simple iteration is not an iteration, and preferably, the data iterator in step 300 may use an existing tool in the levelDB database.

And C4: and querying the target data in the non-relational database according to the name of the converted target data by using the data iterator.

In some embodiments, referring to fig. 8, the method for querying a non-relational database based on skip list data further comprises, before step a:

step D: and determining a query keyword, a target data name and an operator according to a query expression received in advance.

Specifically, a query expression sent by a user is received first, and then the expression is parsed to determine a query keyword, a target data name, and an operator.

In a specific embodiment, the present invention further provides a specific embodiment of a method for generating and querying a non-relational database based on skip list data, taking a Key-Value storage database as an example, which specifically includes the following contents.

It is understood that Key-Value storage databases are one type of non-relational databases that use a hash table having a particular Key and a pointer to particular data.

The invention provides a non-relational database generation method based on skip list data, which mainly explains the ways of dumping a relational data source into non-relational data files and arranging and storing data keys, all dumped files are stored in a pre-established data list file directory, and the directory can be regarded as a logical list for data analysis application. Referring to fig. 9, the method includes the steps of:

brief description of terms:

software Development Kit (SDK) refers to a collection of Development tools used by Software engineers to create application Software for a particular Software package, software framework, hardware platform, operating system, and the like.

Database indexing: a data structure requiring additional maintenance increases the speed of data retrieval operations on database tables at the expense of additional write and storage space.

Key-value (KV) stores use associative arrays (also called mappings or dictionaries) as their underlying data model.

T01: and connecting the data source to obtain the name of the data source table of the relational database.

The data source may be a relational database or a data set file, such as: mySQL, oracle, SQLServer or csv files

T02: the data source table name is processed.

And splicing the data source table names or generating a table name character string according to the data source names, and using the character string as a folder to be dumped.

T03: and (4) judging whether the table name character string in the step T02 judges that a dump folder exists, if so, turning to the step T05, otherwise, turning to the step T04.

T04: a logical table is created.

Creating a folder as the dumped logical table name using the step T02 table name string

T05: and analyzing the query expression.

Specifically, N different queries are analyzed, query expressions are analyzed, data source list names are screened as indexes, M columns in the N queries are selected as index column names respectively

T06: a query-index array is generated.

The M index column names selected in step T05 are arranged in the order of occurrence of the query conditions of the expression, respectively, and after arrangement, an index column group with a length of M is generated, and a two-dimensional array (see fig. 10) with a query number of N and an index column name number of max (M) is formed, which is called a query-index column group.

T07: and traversing the array and judging whether the operation is finished.

Starting to traverse the query-index column group generated by the T06, judging whether the query-index column group is finished or not, turning to the step T0E if the query-index column group is finished, and turning to the step T08 if the query-index column group is not finished

T08: the connection index.

Specifically, an index column group is obtained, and characters 'to' are connected in the order of the index column group to be used as the name of a query dump file

T09: and opening the write jump table.

Creating a local file which is inquired about the dump file name by the step T08 and opens a jump table (levelDB) in a read-write mode

T10: the data source is read.

Open T01 arbitrary data sources and read the full amount of data

T11: and judging whether the reading of the data source is finished or not.

Judging whether the reading data source is finished or not, and jumping to the step T1E when the reading data source is finished

T12: and reading the full data to the memory line by line.

T13: and splicing the index prefixes.

And acquiring the row data of the corresponding column according to the index column group obtained in the step T08, and splicing the row data into a skip list (levelDB) index prefix by using the symbols '-'. When data is inserted into the skip list, the skip list is sorted according to the 'Alpha-beta' value of the key to finally obtain an ordered set, and in order to prevent the concatenated characters from being taken as 'Alpha-beta' characters to be added into the skip list key sorting, ASCII characters which are different from common numbers and letters are used as connection characters, and the characters have the largest sequence in visible characters.

T14: the data key value is stored.

And (3) connecting the 3-index prefix obtained in the step (T1) with a sign-' connecting self-growing number as a skip list (levelDB) index key, resolving the full data into a JSON character string as a value, storing the JSON character string into the skip list (levelDB) pointed in the step (T09) by using a skip list (levelDB) interface, and returning to the step (T11).

T1E: the reading of the data source is ended.

And closes the jump table (levelDB) pointed to by T09, returning to step T07.

T0E: and (6) ending.

This data dump step ends

Then, taking level db as an example, the specific embodiment of the present invention provides a method for querying a non-relational database based on skip list data, where the level db provides two interfaces mainly used for querying:

(1) Get the data key starting with the specified prefix and return to the iterator, which can only traverse backwards.

(2) And searching for the specified data key, returning to the iterator if the specified data key is successfully searched, and returning to the null state if the specified data key is failed.

The non-relational database query method based on skip list data provided by the specific embodiment of the invention provides a specific JSON query format for expressing query conditions and converting and utilizing a levelDB interface for query on the basis of the above, and the format is as follows:

{

table name,

'Filters' [ 'conditional expression 1', 'conditional expression 2', 'conditional expression 3', ] or

}

By using the format, the data which is specifically arranged is inquired by a comparison operator in the conditional expression.

The comparison operator in the solution of the invention is defined as: equal to or greater than a sign (=), equal to or greater than a sign (> =), greater than a sign (>), equal to or less than a sign (<=), and less than a sign (<).

The comparison operator priority here is: equal to (=) > greater than or equal to (> =) > greater than (>) > less than or equal to (=) > less than (<)

Referring to fig. 11, a method for querying a non-relational database based on skip list data according to an embodiment of the present invention includes the following steps:

s01: and analyzing the query expression in the JSON format.

It should be noted that, using JSON as the query language interface is only an embodiment of the present invention, and the query may be performed by other or even self-created custom intermediate languages, or the standard structured query statement is combined with EBNF to implement the structured query, which is not limited by the present invention.

S02: and acquiring the name of the database table.

Obtaining the database table name parsed from step S01

S03: and acquiring a query expression.

Obtaining a query expression in Filter

S04: and judging whether the table name exists.

And (5) searching a folder with the same table name as the table name in the step (S02) in the local file system, judging whether the folder path exists, if so, turning to the step (S05), otherwise, turning to the step (S1E).

S05: and resolving the index column.

And analyzing the index column name according to the query expression acquired in the step S03.

S06: and analyzing the operator.

And analyzing the operational characters according to the query expression acquired in the step S03.

S07: an array of values is determined.

And analyzing the value array according to the query expression acquired in the step S03.

S08: and arranging the operator.

And D, according to the operator array generated in the step S06, prioritizing and forming the array.

S09: and arranging index column groups.

According to the sequence of the operator array in the step S08, the index column names in the step S05 are combined into an array according to the sequence.

S10: the index column groups are concatenated.

And splicing the index column group generated in the step S09 into the name of the dump file by character 'to' connection.

S11: and judging whether the file exists.

And according to the dump file name in the step S10, synchronizing the database table names in the step S02, splicing and searching for corresponding files, judging whether the files exist, if so, turning to the step S12, and if not, turning to the step S2E.

S12: and opening the skip list.

And opening a jump table (levelDB) local file in a reading mode.

S13: and acquiring an index prefix.

And according to the sequence of the operator array in the step S08, splicing the value array contents analyzed in the step S07 by using symbols 'to' according to the sequence to obtain the index prefix.

S14: a determination is made as to whether there is a known iterator.

And (6) judging whether a known iterator exists or not according to the step (S13), if so, acquiring a data iterator according to the known iterator and the index prefix of the step (S13), and turning to the step (S16), otherwise, turning to the step (S15).

S15: and opening the skip list file to obtain the iterator.

According to S13, the skip list (levelDB) in the step S12 is opened, and the data iterator is obtained.

S16: and judging whether the index prefix is finished or not.

And (5) judging whether the index prefix acquired in the step (S13) is finished or not, if so, turning to the step (S17), otherwise, turning to the step (S13).

S17: and traversing the iterator.

And traversing the data iterators obtained in the step S15 and the step S14.

S18: data is acquired.

And analyzing the acquired data into JSON, and combining the JSON into a JSON array.

S19: and judging whether the traversal is finished or not.

And judging whether the iterator is finished, if so, turning to the step S0E, and otherwise, turning to the step S17.

S2E: returning JSON, 'unsupported queries'.

S1E: returning to JSON, 'table does not exist'.

S0E: and returning JSON containing the query result.

Next, the two methods are exemplified in the embodiments of the present invention:

two common query approaches: the user has data stored in relational data tables, specifically, referring to tables 1 to 3, three data tables are defined as follows, and each table stores data:

TABLE 1

ID	Code	Date	Price	Volume
						1	000001.XSHE	2022-03-01	100	10
2	000002.XSHE	2022-03-01	70	20
					3	000003.XSHE	2022-03-01	80	15

TABLE 2

TABLE 3

ID	Code	Date	Price	Volume
						1	000001.XSHE	2022-03-03	106	9
2	000002.XSHE	2022-03-03	71	18
					3	000003.XSHE	2022-03-03	73	12

Firstly, merging class data in a specific mode, converting the merged class data into a skip list and storing the skip list in a file mode (using a levelDB), and providing a method which is equivalent to the following query functions of the following structured sentences for making relational query:

1.SELECT Code,Date,Price,Volume FROM TableA,TableB,TableC WHERE Code＝‘000001.XSHE’

query all data with ' Code ' equal to '000001.XSHE

2.SELECT Code,Date,Price,Volume FROM TableA,TableB,TableC WHERE Code＝’000002.XSHE’AND Date>＝’2022-03-02’AND Date<＝’2022-03-03’

Query all data for ' Code ' equal to '000002.XSHE ' and ' Date ' from '2022-03-02' to '2022-03-03

Data dump: referring to FIG. 12, for purposes of this disclosure, an 'ABC' directory is considered a logical table. In this scenario, the query is to retrieve all the fields, so the data value store is not specifically processed.

The data values are JSON arrays [ 'Code', 'Date', 'Price', 'Volume' ], which are formed by combining the data of each table by using the data of the line unit

(1) The first type:

in this case, the main search field is 'Code', which is first used as the main index.

At this time, the key 'Code to ID' combined by 'Code' field and 'ID' field is obtained, and for multi-table merging, the field 'ID' may have repeated appearance, so the ID may be the generated ordered character during storage, and the content is spliced and stored as a data key, for example: 000001. XSHE. About.1 'and 000001. XSHE. About.2'.

Data may be saved as 'CODE to ID to abc.db' in a file headed by the key 'CODE to ID'

(2) The second type:

in this case, the main search fields are ' Code ' and ' Date ', the ' Code ' is used as a main index, the ' Date ' is used as a secondary index and spliced after the main index, and finally the ' ID ' is spliced to generate a self-increment field, and characters ' to connecting characters are used.

At this time, keys ' Code to Date to ID ' are obtained in combination of three fields of ' Code ' field, ' Date ' field and ' ID ' field '

Data may be stored as 'CODE to DATE to ID to ABC.db' in files headed by the keys 'CODE to DATE to ID'.

Data query:

the first type:

{

‘Table’:’ABC’,

‘Filters’:[‘Code＝＝000001.XSHE’]

}

1. analyzing JSON and obtaining 'Table' and 'Filters';

2. sequentially analyzing an expression 1 in 'Filters';

3. obtain prefix 'Code', operator '= =', and data key '000001.Xshe';

4. the prefix of the spliced data file is 'CODE-ID';

5. finding out a data file with a path of ABC/CODE-ID-ABC.db';

6. obtaining an iterator with '000001. XSHE-' as a prefix;

7. traversing the iterator;

8. and (6) combining the results and outputting.

The second type:

{

‘Table’:’ABC’,

‘Filters’:[‘Code＝＝000002.XSHE’,‘Date bt[2022-03-02,2022-03-03]’]

}

1. analyzing JSON and obtaining 'Table' and 'Filters';

2. sequentially analyzing an expression 1 in 'Filters';

3. obtain prefix 'Code', operator '= =', and data key '000002.Xshe';

4. analyzing an expression 2 in 'Filters' in sequence;

5. acquiring a prefix ' Date, an operator ' bt ', and a data key '2022-03-02,2022-03-03';

6. the prefix of the spliced data file is 'CODE-DATE-ID';

7. finding out data files with the paths of ABC/CODE-DATE-ID-ABC.db';

8. obtaining an iterator with '000002. XSHE-' as a prefix;

9. checking the data key, wherein Date is small in front;

10. searching an iterator with a prefix of '000002. XSHE-2022-03-02-' in the iterator according to the iterator obtained by the step 8;

11. searching an iterator with a prefix of '000002. XSHE-2022-03-' in the iterator according to the iterator obtained by the step 8;

12. iterating according to the iterator obtained by 10 until the iterator is equal to the iterator obtained by 11;

13. and (6) combining the results and outputting.

It should be noted that the fields 'Code', 'Date', 'Price', 'Volume', 'ID', etc. used for searching in the present invention can be replaced by any other data fields that need to be searched, and are not limited to the ones shown herein. The method can expand the processing of the universal table data, and the storage and the query of the universal data can be realized only by performing proper mapping on various table fields. The data result of the invention can be filtered by a specific mode to achieve the purpose of selecting a specific field.

The invention uses JSON as the query language interface only in one mode, and can also query in other or even self-created self-defined intermediate language modes, or combine the standard structured query statement with EBNF to realize structured query.

In addition, the levelDB provided by the invention is only one way of realizing the jump table in a key value storage mode, and can be equivalently replaced by other jump tables or NoSQL databases in other key value storage modes realized in any other modes.

Based on the same inventive concept, the embodiment of the present application further provides a device for generating and querying a non-relational database based on skip list data, which can be used to implement the method described in the above embodiment, such as the following embodiments. Because the principle of generating and querying the non-relational database based on the skip list data is similar to the method of generating and querying the non-relational database based on the skip list data, the implementation of the non-relational database based on the skip list data and the query device can refer to the implementation of the non-relational database based on the skip list data and the query method, and repeated parts are not repeated. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. While the system described in the embodiments below is preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

The embodiment of the present invention first provides a specific implementation manner of a non-relational database generation apparatus based on skip list data, which can implement a non-relational database generation method based on skip list data, and referring to fig. 13, the non-relational database generation apparatus based on skip list data specifically includes the following contents:

an index array generating module 10, configured to generate a query index array according to a plurality of historical query expressions of a relational source database;

a data determining module 20, configured to determine, in the relational source database, data corresponding to the query index array;

and a non-relational database generation module 30, configured to generate a non-relational database according to the data corresponding to the query index array and the skip list model.

In one embodiment, referring to fig. 14, the index array generation module 10 includes:

a keyword determination unit 101 configured to parse a plurality of historical query expressions to determine a plurality of query keywords;

a keyword array generating unit 102, configured to sort the multiple query keywords according to the query order of the multiple historical query expressions, and generate a keyword array according to the sorted multiple query keywords;

an index array generating unit 103, configured to generate the query index array according to a plurality of historical query expressions and the keyword array.

In one embodiment, referring to fig. 15, the data determination module 20 includes:

a data determining unit 201, configured to obtain data of columns or rows where multiple query keywords are located in the relational source database.

In one embodiment, referring to FIG. 16, the non-relational database generation module 30 comprises:

the non-relational database generating unit 301 is configured to insert the data corresponding to the query index array into a skip list model according to a preset sequence to generate a non-relational database.

Secondly, an embodiment of the present invention further provides a specific implementation manner of a non-relational database query apparatus based on skip list data, which is capable of implementing a non-relational database query method based on skip list data, and referring to fig. 17, the non-relational database query apparatus based on skip list data specifically includes the following contents:

the index array generating module A is used for responding to the received query expression and sequencing a plurality of query keywords according to the priority of the operational characters so as to generate a query index array;

the name conversion module B is used for converting the name of the target data according to the query expression and the query index array;

the data query module C is used for querying the target data in the non-relational database according to the name of the converted target data; wherein the query keyword, target data name, and operator are generated by parsing the query expression.

In one embodiment, referring to fig. 18, the name conversion module B includes:

and a name generating unit B1, configured to splice elements in the query index array according to the order of priorities of the operators, so as to generate the name of the converted target data.

In one embodiment, referring to fig. 19, the data query module C includes:

a prefix determining unit C1, configured to determine, according to the element, an index prefix corresponding to the query expression;

an iterator determining unit C2, configured to determine, according to the index prefix, a data iterator corresponding to the query expression;

and the data query unit C3 is used for querying the target data in the non-relational database according to the name of the converted target data by using the data iterator.

As can be seen from the foregoing description, an embodiment of the present invention provides a device for generating and querying a non-relational database based on skip list data, where the corresponding device for generating a non-relational database based on skip list data includes: firstly, generating a query index array according to a plurality of historical query expressions of a relational source database; then, determining data corresponding to the query index array in a relational source database; and finally, generating a non-relational database according to the data corresponding to the query index array and the skip list model.

The non-relational database query device based on the skip list data comprises: firstly, in response to a received query expression, sequencing a plurality of query keywords according to the priority of an operational character to generate a query index array; then, converting the name of the target data according to the query expression and the query index array; finally, inquiring the target data in a non-relational database according to the name of the converted target data; wherein the query key, the target data name, and the operator are generated by parsing the query expression.

An embodiment of the present application further provides a specific implementation manner of an electronic device, which is capable of implementing all steps in the method for generating and querying a non-relational database based on skip list data in the foregoing embodiment, and referring to fig. 20, the electronic device specifically includes the following contents:

a processor (processor) 1201, a memory (memory) 1202, a communication Interface 1203, and a bus 1204;

the processor 1201, the memory 1202, and the communication interface 1203 complete communication with each other through the bus 1204; the communication interface 1203 is used for implementing information transmission between related devices such as a server-side device and a client-side device;

the processor 1201 is configured to call the computer program in the memory 1202, and the processor executes the computer program to implement all the steps in the method for generating and querying the non-relational database based on the skip list data in the foregoing embodiments, for example, the processor executes the computer program to implement the following steps:

step 200: determining data corresponding to the query index array in the relational source database;

step 300: and generating a non-relational database according to the data corresponding to the query index array and the skip list model. And

and B: converting the name of the target data according to the query expression and the query index array;

An embodiment of the present application further provides a computer-readable storage medium capable of implementing all steps in the skip list data-based non-relational database generation and query method in the foregoing embodiment, where the computer-readable storage medium stores a computer program, and the computer program implements all steps of the skip list data-based non-relational database generation and query method in the foregoing embodiment when being executed by a processor, for example, the processor implements the following steps when executing the computer program:

step C: inquiring the target data in a non-relational database according to the name of the converted target data; wherein the query keyword, target data name, and operator are generated by parsing the query expression.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the hardware + program class embodiment, since it is substantially similar to the method embodiment, the description is simple, and the relevant points can be referred to the partial description of the method embodiment.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

Although the present application provides method steps as in an embodiment or a flowchart, more or fewer steps may be included based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an actual apparatus or client product executes, it may execute sequentially or in parallel (e.g., in the context of parallel processors or multi-threaded processing) according to the embodiments or methods shown in the figures.

For convenience of description, the above devices are described as being divided into various modules by functions, which are described separately. Of course, in implementing the embodiments of the present description, the functions of each module may be implemented in one or more software and/or hardware, or a module implementing the same function may be implemented by a combination of multiple sub-modules or sub-units, and the like. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may therefore be considered as a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

The embodiments of this specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The described embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment. In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of an embodiment of the specification. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

The above description is only an example of the embodiments of the present disclosure, and is not intended to limit the embodiments of the present disclosure. Various modifications and variations to the embodiments described herein will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present specification should be included in the scope of the claims of the embodiments of the present specification.

Claims

1. A method for generating a non-relational database based on skip list data is characterized by comprising the following steps:

2. The non-relational database generation method according to claim 1, wherein generating the query index array from the plurality of historical query expressions for the relational source database comprises:

3. The method for generating a non-relational database according to claim 2, wherein the determining the data corresponding to the query index array in the relational source database comprises:

4. The method for generating a non-relational database according to claim 1, wherein the generating a non-relational database according to the data corresponding to the query index array and the skip list model comprises:

5. A non-relational database query method based on skip list data, applied to the non-relational database according to any one of claims 1 to 4, comprising:

6. The method of claim 5, wherein said converting the name of the target data according to the query expression and the query index tuple comprises:

7. The method for querying a non-relational database based on skip list data according to claim 6, wherein the querying the target data in the non-relational database according to the name of the converted target data comprises:

8. A non-relational database generation apparatus based on skip list data, comprising:

9. A non-relational database query device based on skip list data, comprising:

10. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the skip list data based non-relational database generation and query method according to any one of claims 1 to 7.

11. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when being executed by a processor, implements the steps of the skip list data based non-relational database generation and query method according to any one of claims 1 to 7.