CN110134681B - Data storage and query method and device, computer equipment and storage medium - Google Patents
Data storage and query method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110134681B CN110134681B CN201910299922.XA CN201910299922A CN110134681B CN 110134681 B CN110134681 B CN 110134681B CN 201910299922 A CN201910299922 A CN 201910299922A CN 110134681 B CN110134681 B CN 110134681B
- Authority
- CN
- China
- Prior art keywords
- data
- tree structure
- information
- node
- main key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a data storage method, which comprises the following steps: responding to a data storage instruction, generating an abstract tree structure representing index relations among a plurality of data tables, wherein the abstract tree structure comprises a plurality of nodes, each node comprises category information and mapping information, the category information represents the category of data to be stored on the node, and the mapping information is used for representing other nodes with mapping relations with the current node; obtaining data to be stored, and storing the data to be stored on corresponding nodes of the abstract tree according to the type of the data to be stored to form a full tree structure; extracting the main key information and the mapping information of each node from the full tree structure to form a main key tree structure, wherein the main key information corresponds to the category information; and storing the full tree structure into a database, and storing the main key tree structure as an inverted index.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and apparatus for querying a high-speed data storage independent of database performance, a computer device, and a storage medium.
Background
With the wide popularization of database technology, various data information has a mass high-speed increasing trend. At present, most of application scenes of computer software are relational databases, and a great deal of database resources including storage space and calling time are consumed when the databases are queried and stored under the condition of complex relations, so that under the current industry large data background, the bottleneck of the databases can be easily reached along with the increase of the data quantity, and the data storage and query efficiency are seriously affected.
Disclosure of Invention
The invention aims to provide a high-speed data storage query method, a system, computer equipment and a storage medium which are independent of database performance, and the method, the system, the computer equipment and the storage medium are used for solving the problems in the prior art.
In order to achieve the above object, the present invention provides a high-speed data storage method, comprising the steps of:
responding to a data storage instruction, generating an abstract tree structure representing index relations among a plurality of data tables, wherein the abstract tree structure comprises a plurality of nodes, each node comprises category information and mapping information, the category information represents the category of data to be stored on the node, and the mapping information is used for representing other nodes with mapping relations with the current node;
obtaining data to be stored, and storing the data to be stored on corresponding nodes of the abstract tree according to the type of the data to be stored to form a full tree structure;
extracting the main key information and the mapping information of each node from the full tree structure to form a main key tree structure, wherein the main key information corresponds to the category information;
and storing the full tree structure into a database, and storing the main key tree structure as an inverted index.
Further, the step of obtaining the data to be stored, storing the data to be stored on the corresponding node of the abstract tree according to the type of the data to be stored, and forming the full tree structure includes:
and disassembling each node in the abstract tree structure to obtain a data table to be updated, and adding data into each data table to be updated according to the type of the data to be stored.
Further, the main key tree structure also stores a non-main key index for representing the correspondence between the non-main keys and the main keys in the plurality of data tables.
In order to achieve the above object, the present invention provides a high-speed data query method, comprising the steps of:
responding to a data query instruction, acquiring a main key tree structure stored in an inverted index mode, wherein the main key tree structure comprises a plurality of nodes, and each node corresponds to one or more data tables with the same main key;
inputting a field to be queried into the main key tree structure, and acquiring a node corresponding to the field;
querying target data from a data table corresponding to the node;
and outputting the target data according to an output mode.
Further, the step of inputting the field to be queried into the primary key tree structure and acquiring the node corresponding to the field comprises any one of the following steps:
the field is a main key, and nodes are inquired according to the main key;
the field is a non-primary key, a primary key corresponding to the non-primary key is queried from an index table, and nodes are queried from a primary key tree according to the primary key.
Further, the outputting data according to the output mode includes outputting data according to a specified mode or outputting data according to a default mode.
To achieve the above object, the present invention provides a high-speed data storage system comprising:
the abstract tree generation module is suitable for generating an abstract tree structure representing index relations among a plurality of data tables, the abstract tree structure comprises a plurality of nodes, each node comprises category information and mapping information, the category information represents the category of data to be stored on the node, and the mapping information is used for representing other nodes with mapping relations with the current node;
the full tree generation module is suitable for acquiring data to be stored, and storing the data to be stored onto corresponding nodes of the abstract tree according to the type of the data to be stored to form a full tree structure;
the main key tree generation module is suitable for extracting main key information and mapping information of each node from the full tree structure to form a main key tree structure, wherein the main key information corresponds to the category information;
the data storage module is suitable for storing the full tree structure into an oracle database and storing the main key tree structure as an inverted index.
To achieve the above object, the present invention provides a data query system, including:
the main key tree acquisition module is suitable for responding to a data query instruction and acquiring a main key tree structure stored in an inverted index mode, wherein the main key tree structure comprises a plurality of nodes, and each node corresponds to one or more data tables with the same main key;
the node acquisition module is suitable for inputting the field to be queried into the main key tree structure to acquire the node corresponding to the field;
the data query module is suitable for querying target data from a data table corresponding to the node;
and the data output module is suitable for outputting the target data according to an output mode.
To achieve the above object, the present invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the computer program.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above method.
The high-speed data storage and query method, the device, the computer equipment and the storage medium provided by the invention are suitable for the technical field of data processing, and provide an automatic implementation scheme independent of database performance for storing and querying mass data. The database in the invention does not need to maintain a large amount of relation data and indexes any more, but introduces a high-speed inverted index frame ES (elasticsearch) with a large data background to store the primary key information of the business relation data, and only the details of the data and the backup of the relation are reserved in the database. When a query request exists, the invention firstly searches the inverted index frame ES according to the condition, queries the primary key information, and then executes the query to the database by the primary key, thereby shielding all non-primary key database requests and greatly improving the search efficiency.
Drawings
FIG. 1 is a flow chart of a first embodiment of a data storage method of the present invention;
FIG. 2 is a schematic diagram of a program module of a first embodiment of a data storage device according to the present invention;
FIG. 3 is a schematic diagram of a hardware structure of a first embodiment of a data storage device according to the present invention;
FIG. 4 is a flowchart of a data query method according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a program module of a data query device according to a first embodiment of the invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The high-speed data storage and query method, the device, the computer equipment and the storage medium provided by the invention are suitable for the technical field of data processing, and provide an automatic implementation scheme independent of database performance for storing and querying mass data. The invention mainly comprises two parts of data storage and data query. In the data storage process, the method firstly acquires the logic relationship and the primary key thereof between each data table, and generates an abstract tree containing a plurality of nodes according to the logic relationship and the primary key to represent the logic relationship, wherein each node is provided with a unique primary key and corresponds to one or more data tables correspondingly. When data storage is carried out, the data information in the data to be updated is stored in the data table under each corresponding node according to the structure of the abstract tree, meanwhile, the abstract tree (namely the main key tree) only containing the main key information is stored in the form of an inverted index (ES), and other data information except the main key is stored in an oracle database. When data query is carried out, the invention firstly calls the primary key tree from the inverted index to definitely carry out the node where the query data is located, and then queries one or more data tables corresponding to the node from the oracle database according to the node information. The scheme of the invention greatly simplifies the data query process, and does not need to traverse all data tables to query, thereby effectively improving the efficiency of data storage and query.
Example 1
Referring to fig. 1, in a high-speed data storage method of the present embodiment, the method includes the following steps:
s1: in response to a data storage instruction, an abstract tree structure representing index relationships between a plurality of data tables is generated, the abstract tree structure comprising a plurality of nodes, each node comprising category information representing a category of data to be stored on the node and mapping information for representing other nodes having a mapping relationship with a current node.
The step is to first build a frame for data storage, which already arranges the storage locations of different data tables and the mapping relations between the data tables, so as to store specific data in corresponding locations subsequently. The framework for data storage is called an abstract tree structure, and the abstract tree structure comprises a plurality of nodes, wherein the nodes represent the storage positions of data tables, and each node correspondingly stores one or more storage tables. The nodes in the invention are divided according to the types of data to be stored, and specific types can be client information, account information, bank card information and other contents. The nodes have mapping relation, for example, account information is generated on the basis of client information, and the basic condition information of clients is recorded first and then the account information of each client is recorded. In this case, the node where the client information is located is referred to as a parent node a, and the node where the account information is located is referred to as a child node B. The mapping relation between nodes or the mapping information of the nodes in the invention can be determined by adding father node information or child node information which is connected with the father node or child node information on each node. It should be noted that, when the mapping information is represented by way of a parent node, since the node at the uppermost layer no longer has the parent node, the parent node information of the node at the uppermost layer is set to be empty; when the mapping information is represented by way of child nodes, child node information of the lowest level node is nulled since the lowest level node no longer has child nodes.
Taking a bank background database system as an example, a plurality of contents such as customer basic information, customer application account information, customer bank card information and the like are generally stored, and each content comprises one or more data sheets. It is assumed that the data related to the customer is divided into the following data tables in the database of a certain bank background: a customer information table (A table), a customer information supplementary table (A1 table), a customer application record table (C table), a customer account information table (B table), an account limit control table (B1 table) and an account bank card table (D table). The client information table (A table) and the client information supplementing table (A1 table) have the same main key custNo, the client account information table (B table) and the account limit control table (B1 table) have the same main key accountNo, the client application record table (C table) has the main key applyNo, and the account bank card table (D table) has the main key cardNo.
The invention divides the data tables again according to the logical index relation and displays the data tables through a tree structure. First, the customer information table (table a) contains the most basic information such as the number, name, identification number, address, etc. of the customer, and therefore, it is regarded as a parent layer. In addition, a client information supplementary table (A1 table) having the same main key as the client information table (A table) is provided as a further supplementary to the client information in the client information table, such as the income status of the client. In order to simplify the structure, the invention classifies the tables with the same main key into the same node, which is marked as node A and is used as a father layer. The tables having an index relationship with the client information table include a client application record table (C table) and a client account information table (B table), and thus the two tables are taken as the first sub-layer. Since the client application record table (C table) and the client account information table (B table) do not have an association relationship with each other, the client application record table (C table) is denoted as node C, and the client account information table (B table) is denoted as node B. In addition, the account limit control table (B1 table) and the client information table (a table) have a direct index relationship, and the same primary key accountNo is provided between the account limit control table (B1 table) and the client account information table (B table), so the account limit control table (B1 table) is also included in the node B for the sake of simplicity. Next, the table having an index relationship with the customer account information table (B table) is an account bank card table (D table), and thus the account bank card table (D table) is classified as a second sub-layer and labeled as node D. As shown in fig. 2, the tree structure including a plurality of layers and a plurality of nodes is called an abstract tree of the present invention, and can intuitively display the index relationship between the data tables.
The hierarchical relationship among the parent layer, the first sub-layer and the second sub-layer can be represented by sub-node information, for example, sub-node information having a mapping relationship with the node is recorded in each node. In the above, the child node information of the node a includes the node B and the node C, and the child node information of the node B includes the node D. And if the sub-nodes with the mapping relation exist after the node C and the node D, the sub-node information in the node C and the node D can be emptied.
When in implementation, the generated abstract tree structure is stored in a configuration file and named as dataSchema.
And S2, acquiring data to be stored, and storing the data to be stored on corresponding nodes of the abstract tree according to the type of the data to be stored to form a full tree structure.
The step adds data information to be stored on the basis of the abstract tree to make the data information into a full tree structure. The abstract book comprises a plurality of nodes, each node corresponds to one kind of information, and each kind of information contains one or more data tables. Filling the data records into the corresponding data tables of the corresponding nodes according to the types, so that the conversion from the abstract tree structure to the full tree structure is realized. Preferably, the category information in the present invention may be determined according to the primary key of the data table.
For example, the following data entries need to be stored to the database:
(1) Customer information table
custNo00123; customer ID:210123190101019987;
(2) Account information table
Account number: acc01;
(3) Account bank card meter
Card number: a card662266; card details: abc
(4) Application record form
Application number: apl0012333552, application country: a CN;
(5) Application record form
Application number: apl0012334663, cell phone: 136XXXXXXXX, applied city: shanghai.
When the method is implemented, abstract tree dataSchema is obtained from the configuration file, the dataSchema is disassembled to obtain each node and a corresponding data table under the node, and data items to be stored are stored into different data tables according to field contents.
The data is stored in a corresponding data table under each node on the basis of an abstract tree dataSchema, and a tree structure which corresponds to the abstract tree and stores complete data is called a full tree structure.
And S3, extracting the main key information and the mapping information of each node from the full tree structure to form a main key tree structure, wherein the main key information corresponds to the category information.
The step extracts the primary key on the basis of the full tree structure, thereby creating data of the primary key tree structure describing the index relationship between the plurality of data tables. Taking the data entry stored in step S2 as an example, on the basis of extracting only primary key data, the data entry information stored in ES is as follows:
(1) Customer information table
Customer number: custNo00123;
(2) Account information table
Account number: acc01;
(3) Account bank card meter
Card number: a card662266;
(4) Application record form
Application number: apl0012333552;
(5) Application record form
Application number: apl0012334663.
Compared with the full tree, the main key tree structure generated in the step keeps the index information of necessary main key data on the basis of removing a large amount of data content, and a person skilled in the art can obtain the storage position of the data only through the index information of the main key data, so that the storage and query efficiency of the data can be effectively improved.
And S4, storing the full tree structure into an oracle database, and storing the main key tree structure as an inverted index.
The full tree structure is the complete data record stored according to the index relation, and the complete data record is stored in the oracle database in the step, so that the storage and query operations are convenient for the person skilled in the art. Unlike available technology, the oracle database of the present invention is not directly connected to common application program, i.e. when the application program needs to store or query the data in the database, the method does not directly operate the oracle database, but first extracts the main key tree structure representing the index relation of the data table from the ES index library, obtains the relation between the data tables and the position information of the data tables from the main key tree structure, and then calls the corresponding data table from the oracle database to operate according to the relation and the position information of the data table. According to the invention, all data tables do not need to be traversed from the oracle database, so that the data storage and query efficiency can be greatly improved. Meanwhile, because the index relation of the data table is stored separately, the oracle database is not required to maintain a large amount of relation data and indexes, so that the limit of data storage is not dependent on the performance of the oracle database.
In addition, the invention saves the extracted main key tree structure in a reverse index (elastic search) mode. The inverted index refers to the position and occurrence frequency of the storage keywords in the articles, so that the efficiency of inquiring the keywords is improved.
With continued reference to FIG. 2, a high-speed data storage device is illustrated, in which the data storage device 10 may include or be partitioned into one or more program modules that are stored in a storage medium and executed by one or more processors to implement the present invention and achieve the above-described automatic update method. Program modules in the present invention are understood to mean a series of computer program instruction segments capable of performing a specified function, more suitable than the program itself for describing the execution of the data storage device 10 in a storage medium. The following description will specifically describe functions of each program module of the present embodiment:
the abstract tree generating module 11 is adapted to generate an abstract tree structure representing index relationships between a plurality of data tables, the abstract tree structure comprises a plurality of nodes, each node comprises category information and mapping information, the category information represents the category of data to be stored on the node, and the mapping information is used for representing other nodes having mapping relationships with the current node;
the full tree generation module 12 is adapted to acquire data to be stored, store the data to be stored on corresponding nodes of the abstract tree according to the type of the data to be stored, and form a full tree structure;
a primary key tree generating module 13, adapted to extract primary key information and mapping information of each node from the full tree structure, to form a primary key tree structure, wherein the primary key information corresponds to the category information;
the data storage module 14 is adapted to store the full tree structure in an oracle database and store the primary key tree structure as an inverted index.
The present embodiment also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a rack-mounted server (including an independent server or a server cluster formed by a plurality of servers) that can execute a program. The computer device 20 of the present embodiment includes at least, but is not limited to: a memory 21, a processor 22, which may be communicatively coupled to each other via a system bus, as shown in fig. 3. It should be noted that fig. 3 only shows a computer device 20 having components 21-22, but it should be understood that not all of the illustrated components are required to be implemented, and that more or fewer components may be implemented instead.
In the present embodiment, the memory 21 (i.e., readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 21 may be an internal storage unit of the computer device 20, such as a hard disk or memory of the computer device 20. In other embodiments, the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the computer device 20. Of course, the memory 21 may also include both internal storage units of the computer device 20 and external storage devices. In this embodiment, the memory 21 is typically used to store an operating system and various types of application software installed on the computer device 20, such as program codes of the high-speed data storage apparatus 10 of the first embodiment. Further, the memory 21 may be used to temporarily store various types of data that have been output or are to be output.
The processor 22 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 22 is generally used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, execute the high-speed data storage device 10, to implement the automatic updating method of the first embodiment.
The present embodiment also provides a computer-readable storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor, performs the corresponding functions. The computer readable storage medium of the present embodiment is used for storing the automatic updating apparatus 10, and when executed by a processor, implements the automatic updating method of the first embodiment.
Example two
Referring to fig. 4, the high-speed data query method of the present embodiment is based on embodiment one, and includes the following steps:
and S1, responding to a data query instruction, and acquiring a main key tree structure which is saved in an inverted index square test, wherein the main key tree structure comprises a plurality of nodes, and each node corresponds to one or more data tables with the same main key.
The present embodiment is directed to a data query operation process in a database. When a certain piece of data is required to be queried from the database, the invention does not directly access the oracle database, but firstly searches the tree structure representing the index relation between the data tables in the ES index database, for example, a main key tree structure (called a main key tree for short) containing main key information. The tree structure is divided into a plurality of layers according to index relation among the data tables, each layer comprises a plurality of nodes, and each node corresponds to one or a plurality of data tables.
It is assumed that the data related to the customer is divided into the following data tables in the database of a certain bank background: a customer information table (A table), a customer information supplementary table (A1 table), a customer application record table (C table), a customer account information table (B table), an account limit control table (B1 table) and an account bank card table (D table). The client information table (A table) and the client information supplementing table (A1 table) have the same main key custNo, the client account information table (B table) and the account limit control table (B1 table) have the same main key accountNo, the client application record table (C table) has the main key applyNo, and the account bank card table (D table) has the main key cardNo.
The invention divides the data tables again according to the logical index relation and displays the data tables through a tree structure. First, the customer information table (table a) contains the most basic information such as the number, name, identification number, address, etc. of the customer, and therefore, it is regarded as a parent layer. In addition, a client information supplementary table (A1 table) having the same main key as the client information table (A table) is provided as a further supplementary to the client information in the client information table, such as the income status of the client. In order to simplify the structure, the invention classifies the tables with the same main key into the same node, and is marked as node A. The tables having an index relationship with the client information table include a client application record table (C table) and a client account information table (B table), and thus the two tables are taken as the first sub-layer. Since the client application record table (C table) and the client account information table (B table) do not have an association relationship with each other, the client application record table (C table) is denoted as node C, and the client account information table (B table) is denoted as node B. In addition, the account limit control table (B1 table) and the client information table (a table) have a direct index relationship, and the same primary key accountNo is provided between the account limit control table (B1 table) and the client account information table (B table), so the account limit control table (B1 table) is also included in the node B for the sake of simplicity. Next, the table having an index relationship with the customer account information table (B table) is an account bank card table (D table), and thus the account bank card table (D table) is classified as a second sub-layer and labeled as node D.
S2, inputting the field to be queried into the main key tree structure, and obtaining the node corresponding to the field.
On the basis of obtaining the main key tree structure, the position of the data table where the data to be searched is located is further determined. The primary key tree contains the primary key information of each data table, and ideally, when the data field to be queried is a primary key, node information corresponding to the primary key can be directly obtained from the primary key tree, and then only needed data is queried from the data table under the node, and all the data tables are not required to be traversed for query.
When the data field to be queried is a non-primary key, the primary key information corresponding to the non-primary key field is required to be searched from the index table, and then the corresponding node is searched from the primary key tree according to the searched primary key information.
And S3, inquiring target data from a data table corresponding to the node.
Each node of the invention comprises one or more data tables which have parallel index relation and have the same main key. On the basis of determining the node, the invention only needs to inquire the target data from the data tables belonging to the node, and all the data tables in the database are not needed to be convenient, so that the data inquiry efficiency can be obviously improved.
And S4, outputting the target data according to an output mode.
The data targeted to a certain field may contain a plurality of contents, for example, looking up the data with the client number 001, and may query the information of the name, account number, account name, application number, loan code, etc. of the client at the same time, so that the output format needs to be limited. In this embodiment, the content to be displayed may be set manually, for example, the data of the client number 001 is queried, and several items of information such as the client name, account name, loan code and the like are specified to be displayed. A default format may also be selected, for example, when no specific content to be displayed is set, all the queried data information is displayed by default.
With continued reference to fig. 5, the high-speed data query device 30 of the present embodiment is based on the first embodiment, and is configured to implement the high-speed data query method of the second embodiment, and includes the functions of each program module:
the primary key tree obtaining module 31 is adapted to respond to a data query instruction and obtain a primary key tree structure stored in an inverted index mode, where the primary key tree structure includes a plurality of nodes, and each node corresponds to one or more data tables with the same primary key;
the node obtaining module 32 is adapted to input the field to be queried into the primary key tree structure and obtain the node corresponding to the field;
a data query module 33 adapted to query target data from a data table corresponding to the node;
the data output module 34 is adapted to output the target data in an output mode.
In summary, the high-speed data storage and query method, the device, the computer equipment and the storage medium provided by the invention have high-speed and high-efficiency data storage and query functions, and the high-speed inverted index frame ES assembly is utilized, so that the performance of data storage and retrieval is greatly improved, the access pressure of a database is reduced, and the potential hazard influence possibly brought to the data performance when a developer writes a program is reduced.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
It will be appreciated by those of ordinary skill in the art that all or part of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, and the program may be stored in a computer readable medium, where the program when executed includes one or a combination of the steps of the method embodiment.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (9)
1. A data storage method comprising the steps of:
generating an abstract tree structure representing index relationships among a plurality of data tables in response to a data storage instruction, wherein the abstract tree structure comprises a plurality of nodes, each node comprises category information, one or more data tables corresponding to the category information, and mapping information, the category information represents the category of data to be stored on the node, and the mapping information is used for representing other nodes with mapping relationships with the current node;
obtaining data to be stored, and storing the data to be stored on a corresponding data table of a corresponding node of the abstract tree structure according to the type of the data to be stored to form a full tree structure, wherein the full tree structure is specifically as follows: disassembling each node in the abstract tree structure to obtain a data table to be updated, and adding data into each data table to be updated according to the type of the data to be stored;
extracting main key information and mapping information of each node from the full tree structure to form a main key tree structure, wherein the main key information corresponds to the type information, and the storage position of a data table can be obtained through index information of the main key information;
and storing the full tree structure into a database, and storing the main key tree structure as an inverted index.
2. The data storage method according to claim 1, wherein the primary key tree structure further stores a non-primary key index for characterizing correspondence between non-primary keys and primary keys in the plurality of data tables.
3. A data query method, characterized in that it is based on the data storage method according to any one of claims 1-2, comprising the steps of:
responding to a data query instruction, acquiring a main key tree structure stored in an inverted index mode, wherein the main key tree structure comprises main key information of each data table, and the storage position of the data table can be obtained through the main key information; the data tables are stored on nodes of a full tree structure in the database, and each node has unique primary key information and corresponds to one or more data tables;
inputting a field to be queried into the primary key tree structure, acquiring a target node corresponding to the field, and querying one or more data tables corresponding to the target node from the database according to the information of the target node;
querying target data from a data table corresponding to the target node;
and outputting the target data according to an output mode.
4. A data query method as claimed in claim 3, wherein said step of entering a field to be queried into said primary key tree structure, obtaining a target node corresponding to said field comprises any one of the following steps:
the field is a main key, and the target node is inquired according to the main key;
the field is a non-primary key, a primary key corresponding to the non-primary key is queried from an index table, and a target node is queried from a primary key tree according to the primary key.
5. The data query method of claim 3, wherein outputting data in an output mode comprises outputting data in a specified mode or outputting data in a default mode.
6. A data storage system, comprising:
the abstract tree generation module is suitable for generating an abstract tree structure representing index relations among a plurality of data tables, the abstract tree structure comprises a plurality of nodes, each node comprises category information, one or more data tables corresponding to the category information and mapping information, the category information represents the category of data to be stored on the node, and the mapping information is used for representing other nodes with mapping relations with the current node;
the full tree generating module is suitable for acquiring data to be stored, storing the data to be stored onto a corresponding data table of a corresponding node of the abstract tree structure according to the type of the data to be stored, and forming a full tree structure, specifically: disassembling each node in the abstract tree structure to obtain a data table to be updated, and adding data into each data table to be updated according to the type of the data to be stored;
the main key tree generation module is suitable for extracting main key information and mapping information of each node from the full tree structure to form a main key tree structure, wherein the main key information corresponds to the type information, and the storage position of a data table can be obtained through index information of the main key information;
the data storage module is suitable for storing the full tree structure into an oracle database and storing the main key tree structure as an inverted index.
7. A data query system, based on the data storage system of claim 6, comprising:
the main key tree acquisition module is suitable for responding to the data query instruction and acquiring a main key tree structure stored in an inverted index mode, wherein the main key tree structure comprises main key information of each data table, and the storage position of the data table can be obtained through the index information of the main key information; the data tables are stored on nodes of a full tree structure in the database, and each node has unique primary key information and corresponds to one or more data tables;
the node acquisition module is suitable for inputting a field to be queried into the primary key tree structure, acquiring a target node corresponding to the field, and querying one or more data tables corresponding to the target node from the database according to the information of the target node;
the data query module is suitable for querying target data from a data table corresponding to the target node;
and the data output module is suitable for outputting the target data according to an output mode.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 5 when the computer program is executed by the processor.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910299922.XA CN110134681B (en) | 2019-04-15 | 2019-04-15 | Data storage and query method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910299922.XA CN110134681B (en) | 2019-04-15 | 2019-04-15 | Data storage and query method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110134681A CN110134681A (en) | 2019-08-16 |
| CN110134681B true CN110134681B (en) | 2023-10-13 |
Family
ID=67569712
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910299922.XA Active CN110134681B (en) | 2019-04-15 | 2019-04-15 | Data storage and query method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110134681B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112583761B (en) * | 2019-09-27 | 2023-03-28 | 奇安信科技集团股份有限公司 | Management method and device of security entity, computer equipment and storage medium |
| CN110990403A (en) * | 2019-11-26 | 2020-04-10 | 深圳壹账通智能科技有限公司 | Business data storage method, system, computer equipment and storage medium |
| CN111475511A (en) * | 2020-04-03 | 2020-07-31 | 弦子科技(北京)有限公司 | Data storage method, data access method, data storage device, data access device and data access equipment based on tree structure |
| CN114373173A (en) * | 2022-01-12 | 2022-04-19 | 中国平安人寿保险股份有限公司 | Data processing method and device, terminal equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6266660B1 (en) * | 1998-07-31 | 2001-07-24 | Unisys Corporation | Secondary index search |
| CN102646111A (en) * | 2012-02-16 | 2012-08-22 | 中国测绘科学研究院 | Knowledge base-based fast construction method of common correlation information query tree |
| CN103412897A (en) * | 2013-07-25 | 2013-11-27 | 中国科学院软件研究所 | Parallel data processing method based on distributed structure |
| CN109388636A (en) * | 2017-08-11 | 2019-02-26 | 中国移动通信集团重庆有限公司 | Business datum is inserted into database method, apparatus, computer equipment and storage medium |
| CN109558128A (en) * | 2018-10-25 | 2019-04-02 | 平安科技（深圳）有限公司 | Json data analysis method, device and computer readable storage medium |
-
2019
- 2019-04-15 CN CN201910299922.XA patent/CN110134681B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6266660B1 (en) * | 1998-07-31 | 2001-07-24 | Unisys Corporation | Secondary index search |
| CN102646111A (en) * | 2012-02-16 | 2012-08-22 | 中国测绘科学研究院 | Knowledge base-based fast construction method of common correlation information query tree |
| CN103412897A (en) * | 2013-07-25 | 2013-11-27 | 中国科学院软件研究所 | Parallel data processing method based on distributed structure |
| CN109388636A (en) * | 2017-08-11 | 2019-02-26 | 中国移动通信集团重庆有限公司 | Business datum is inserted into database method, apparatus, computer equipment and storage medium |
| CN109558128A (en) * | 2018-10-25 | 2019-04-02 | 平安科技（深圳）有限公司 | Json data analysis method, device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110134681A (en) | 2019-08-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110134681B (en) | Data storage and query method and device, computer equipment and storage medium | |
| CN107391653B (en) | Distributed NewSQL database system and picture data storage method | |
| CN108228817B (en) | Data processing method, device and system | |
| CN106202207B (en) | HBase-ORM-based indexing and retrieval system | |
| US9146955B2 (en) | In-memory, columnar database multidimensional analytical view integration | |
| US8751437B2 (en) | Single persistence implementation of business objects | |
| US20140046928A1 (en) | Query plans with parameter markers in place of object identifiers | |
| CN110795455A (en) | Dependency relationship analysis method, electronic device, computer device and readable storage medium | |
| CN106407360B (en) | Data processing method and device | |
| US20110252049A1 (en) | Function execution using sql | |
| CN111241108B (en) | Key value based indexing method and device for KV system, electronic equipment and medium | |
| KR20130139724A (en) | A computing system, a method for controlling thereof, and a computer-readable recording medium having a computer program for controlling thereof | |
| CN113407565B (en) | Cross-database data query method, device and equipment | |
| CN100527131C (en) | Interdynamic access method and tool of IMS data base | |
| CN112835638A (en) | Configuration information management method and device based on embedded application program | |
| US20230153455A1 (en) | Query-based database redaction | |
| US8005844B2 (en) | On-line organization of data sets | |
| CN111125216A (en) | Method and device for importing data into Phoenix | |
| CN111897837A (en) | Data query method, device, equipment and medium | |
| CN116010345A (en) | Method, device and equipment for realizing table service scheme of flow batch integrated data lake | |
| CN115062023A (en) | Wide table optimization method and device, electronic equipment and computer readable storage medium | |
| CN114564621A (en) | Method, device and equipment for associating data and readable storage medium | |
| CN113434585A (en) | Resource saving method and equipment | |
| CN113779120A (en) | Data query method, device, equipment and storage medium | |
| CN113761102A (en) | Data processing method, device, server, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |