CN112395294A - Database data management method and system and database - Google Patents
Database data management method and system and database Download PDFInfo
- Publication number
- CN112395294A CN112395294A CN202011364309.0A CN202011364309A CN112395294A CN 112395294 A CN112395294 A CN 112395294A CN 202011364309 A CN202011364309 A CN 202011364309A CN 112395294 A CN112395294 A CN 112395294A
- Authority
- CN
- China
- Prior art keywords
- data
- storage area
- database
- write data
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a database data management method, a database data management system and a database. The data are directly stored in the first storage area and asynchronously transmitted to the second storage area to be stored in the form of the data table, so that the relational storage and the non-relational storage of the data can be simultaneously supported, and technical preparation and support are provided for a database to support richer data processing scenes.
Description
Technical Field
The invention relates to the technical field of databases, in particular to a database data management method and a database data management system. The invention also relates to a database.
Background
The database management system has various classification methods, can be divided into a distributed type and a centralized type from the aspect of system architecture, can be divided into a relational type and a non-relational type according to a data storage structure, and can be divided into an SQL database and an NoSQL database from the aspect of supporting capability of the data storage structure. Sql (structured Query language) refers to a relational database, and has the advantages of uniformity and usability, but the disadvantage is that the performance of the database is rapidly reduced along with the increase of the database when a large amount of data is faced. NoSQL (not Only sql) generally refers to a non-relational database at the expense of relaxing the ACID principle, which is adopted by NoSQL rather than strictly adhering to the ACID principle as a relational database, meaning that if there is no update of a particular data item within a certain period of time, all accesses thereto will eventually return the last updated value, representing basic availability, soft state and final consistency. In practical application, the relational database and the non-relational database are suitable for different application scenarios based on respective characteristics, and each database cannot be suitable for all application scenarios, so that the application range of the databases is limited.
Disclosure of Invention
The invention aims to provide a database data management method and a database data management system, which provide technical preparation and support for supporting richer data processing scenes by a database. The invention also provides a database.
In order to achieve the purpose, the invention provides the following technical scheme:
a database data management method, comprising:
when receiving write data, creating a data table according to the write data and storing mode information of the data table into a second storage area;
storing the write data to a first storage area;
and transmitting the write data stored in the first storage area to the second storage area so as to store the write data in the form of a data table in the second storage area according to the mode information of the data table.
Preferably, the storing the write data in the first storage area includes: and generating an identifier according to the identifier information of the written data, generating the data pointed by the identifier from the written data, and storing the written data in the first storage area.
Preferably, the first storage area includes a plurality of nodes, the plurality of nodes includes a management node and a member node, the management node is configured to acquire the write data transmitted by the client, generate a copy of the write data from the write data, and transmit the copy of the write data to the member node, and the member node is configured to store the copy of the write data according to an instruction of the management node.
Preferably, when the created mode information of the data table is changed, the changed mode information is stored in the second storage area to update the mode information stored in the second storage area.
Preferably, storing the write data in the second storage area in a data table form includes: the data table includes a plurality of columns, each column storing data of a different field of the write data, respectively.
Preferably, when a data reading request is received, if the data reading request indicates to read data from the second storage area, whether the data stored in the second storage area is consistent with the data stored in the first storage area is determined;
and if the data reading request is not received, the data reading request is responded after the storage data of the first storage area is transmitted to the second storage area and storage is finished.
A database data management system for performing the database data management method described above.
A database comprising the database data management system described above.
According to the technical scheme, when the write data are received, the data table is created according to the write data, the mode information of the data table is stored in the second storage area, the write data are stored in the first storage area, then the write data stored in the first storage area are transmitted to the second storage area, and the write data are stored in the second storage area in a data table mode according to the mode information of the data table.
According to the database management method and system, data are directly stored in the first storage area and asynchronously transmitted to the second storage area to be stored in the form of the data table, so that relational storage and non-relational storage of the data can be simultaneously supported, and technical preparation and support are provided for supporting richer data processing scenes by the database.
The database provided by the invention can achieve the beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a database data management method according to an embodiment of the present invention;
FIG. 2(a) is a schematic diagram illustrating transferring storage data of a first storage area to a second storage area for storage according to an embodiment of the present invention;
FIG. 2(b) is a diagram illustrating reading data from the second storage area according to an embodiment of the present invention;
fig. 3 is a flowchart of a method for responding to a data read request in a database data management method according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart of a database data management method according to an embodiment of the present invention, and it can be seen that the database data management method includes the following steps:
s10: when receiving the write data, creating a data table according to the write data and storing the mode information of the data table into a second storage area.
The data table defines a format for storing the write data, and the mode information of the data table describes the relation between rows and columns of the data table and the format for storing the write data according to the data table.
And when receiving the write data provided by the client, creating a data table according to the write data, and storing the mode information of the created data table into the second storage area. Alternatively, the second storage area may be any one of the distributed storage nodes.
S11: and storing the write data into a first storage area.
The write data is stored directly to the first storage area upon receipt of the write data. Optionally, an identifier may be generated according to the identifier information of the write data, and the write data is generated into data pointed by the identifier, so that the write data is stored in the first storage area. The written data can be stored in the first storage area in a line memory mode, an initial key in the written data is coded to form a key, and other data are correspondingly stored as data pointed by the key.
Alternatively, the first storage area may adopt an embeddable key-value storage system supporting persistence, in which all data is stored in sequence, and common operations include get (key), put (key), delete (key), and scan (key). The storage system has three basic structures: RocksDB memtable, sstfile, and logfile. memtable is a memory data structure that will insert new data into the memtable and log file (optional). The log file is written sequentially and is located on the disk. When the memtable is full, the data will be flushed to the sstfile on the disk, and the corresponding log file can be safely deleted. The data in the sstfile is sorted for the purpose of expediting key lookups.
Further preferably, the first storage area may include a plurality of nodes, where the plurality of nodes include a management node and a member node, the management node is configured to obtain the write data transmitted by the client, generate a copy of the write data from the write data, and transmit the copy of the write data to the member node, and the member node is configured to store the copy of the write data according to an instruction of the management node. In the storage mode, a management node is selected from a node cluster, and the management node is responsible for receiving a transaction request from a client and notifying other nodes in the cluster to ensure that the other nodes and the log of the other nodes are synchronous. And when the management node is down, other nodes in the cluster initiate elections to re-elect the management node.
S12: and transmitting the write data stored in the first storage area to the second storage area so as to store the write data in the form of a data table in the second storage area according to the mode information of the data table.
And pushing and transmitting the write data stored in the first storage area to a second storage area, and further storing the write data in the second storage area in a data table form according to the mode information of the created data table. Optionally, the created data table includes a plurality of columns, and each column stores data written in different fields of the data. Each field may be assigned a data type defining their data length and other attributes, and may contain various characters, numbers, and even graphics. A row of the data table is called a record and the intersection of the row and column indicates a certain attribute value. For example, referring to fig. 2(a), fig. 2(a) is a schematic diagram illustrating that the storage data of the first storage area is transferred to the second storage area for storage in an embodiment. It can be seen that the data in the first storage area is stored in a key-value form, and after the data in the first storage area is transmitted to the second storage area, the data is split in a column, and is reorganized and stored according to a format specified by the data table.
In the method, after the write-in data is completely stored in the first storage area, the execution result can be returned, and the write-in data in the second storage area does not need to be waited for to be completely written.
Preferably, in the method of this embodiment, since the mode information that does not include data in the process of transferring the data to be written from the first storage area to the second storage area, a mode information update triggering mechanism is established for the method of this embodiment, and when the mode information of the created data table is changed, the changed mode information is stored in the second storage area to update the mode information stored in the second storage area, thereby ensuring that the mode information stored in the second storage area is kept updated.
According to the database data management method, data are directly stored in the first storage area and asynchronously transmitted to the second storage area to be stored in a data table form, so that relational storage and non-relational storage of the data can be simultaneously supported, and technical preparation and support are provided for supporting richer data processing scenes by the database.
Referring to fig. 3, fig. 3 is a flowchart illustrating a method for responding to a data read request in a database data management method according to another embodiment, which shows that the database data management method according to the present embodiment further includes the following steps:
s20: a data read request is received, the data read request indicating that data is to be read from the second storage area.
The write data is stored in the first storage area and the second storage area, respectively, where the data may be stored in different forms. In an actual application scenario, an application end may require to read and obtain data in a form that data is stored in a first storage area, and at this time, data needs to be read from the first storage area; or may require reading data in the form of data stored in the second storage area, which is needed at this time.
When a data read request is received, data is read from the first storage area or data is read from the second storage area according to the requirements of the data read request. In the method of the embodiment, the written data is directly stored in the first storage area, so that when the data reading request indicates to read data from the first storage area, the data can be directly read from the first storage area according to the data reading request and returned to the application end.
If the data reading request indicates to read data from the second storage area, since the data written in the first storage area is transferred to the second storage area for storage when the data is written in, and the data is transferred to the second storage area for storage when the data reading request is received, the data is not necessarily successfully written in or not yet synchronously completed, so that in order to obtain the latest data by reading, it is necessary to ensure that the data stored in the second storage area when the data is read from the second storage area is consistent with the data stored in the first storage area.
S21: and judging whether the data stored in the second storage area is consistent with the data stored in the first storage area. If so, the process proceeds to step S22, and if not, the process proceeds to step S23.
S22: responding to the data reading request.
If the data stored in the second storage area is consistent with the data stored in the first storage area, the data reading request can be directly responded, and the data can be read from the second storage area according to the data reading request.
S23: waiting for the storage data of the first storage area to be transferred to the second storage area and finishing storage. And proceeds to step S22 after the storage data of the first storage area is transferred to the second storage area and the storage is completed. For example, referring to fig. 2(b), fig. 2(b) is a schematic diagram of reading data from the second storage area in an embodiment.
Correspondingly, the embodiment of the invention also provides a database data management system, which is used for executing the database data management method.
The database data management system of the present embodiment, when receiving the write data, creates a data table according to the write data and stores mode information of the data table in the second storage area, stores the write data in the first storage area, and then transfers the write data stored in the first storage area to the second storage area to store the write data in the form of a data table in the second storage area according to the mode information of the data table. The database data management system of the embodiment directly stores the data in the first storage area, and asynchronously transmits the data to the second storage area to be stored in a data table form, so that the relational storage and the non-relational storage of the data can be simultaneously supported, and technical preparation and support are provided for supporting richer data processing scenes by the database.
Correspondingly, the embodiment of the invention also provides a database, which comprises the database data management system.
The database of the present embodiment creates a data table based on write data and stores mode information of the data table to the second storage area when receiving the write data, stores the write data to the first storage area, and then transfers the write data stored to the first storage area to the second storage area to store the write data to the second storage area in a data table form based on the mode information of the data table. The database of the embodiment stores data in the first storage area directly, and asynchronously transmits the data to the second storage area to be stored in the form of a data table, so that the relational storage and the non-relational storage of the data can be simultaneously supported, and technical preparation and support are provided for the database to support richer data processing scenes.
The above detailed description is provided for the database data management method, system and database provided by the present invention. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (8)
1. A database data management method, comprising:
when receiving write data, creating a data table according to the write data and storing mode information of the data table into a second storage area;
storing the write data to a first storage area;
and transmitting the write data stored in the first storage area to the second storage area so as to store the write data in the form of a data table in the second storage area according to the mode information of the data table.
2. The database data management method according to claim 1, wherein storing the write data in a first storage area comprises: and generating an identifier according to the identifier information of the written data, generating the data pointed by the identifier from the written data, and storing the written data in the first storage area.
3. The database data management method according to claim 1, wherein the first storage area includes a plurality of nodes, the plurality of nodes include a management node and a member node, the management node is configured to obtain the write data transmitted by the client, generate a copy of the write data from the write data and transmit the copy of the write data to the member node, and the member node is configured to store the copy of the write data according to an instruction of the management node.
4. The database data management method according to claim 1, wherein when the mode information of the created data table is changed, the changed mode information is stored to the second storage area to update the mode information stored in the second storage area.
5. The database data management method according to claim 1, wherein storing the write data in the form of a data table to the second storage area comprises: the data table includes a plurality of columns, each column storing data of a different field of the write data, respectively.
6. The database data management method according to any one of claims 1 to 5, wherein when a data read request is received, if the data read request indicates to read data from the second storage area, it is determined whether the data stored in the second storage area is consistent with the data stored in the first storage area;
and if the data reading request is not received, the data reading request is responded after the storage data of the first storage area is transmitted to the second storage area and storage is finished.
7. A database data management system for performing the database data management method of any one of claims 1 to 6.
8. A database comprising the database data management system of claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011364309.0A CN112395294B (en) | 2020-11-27 | 2020-11-27 | Database data management method and system and database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011364309.0A CN112395294B (en) | 2020-11-27 | 2020-11-27 | Database data management method and system and database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112395294A true CN112395294A (en) | 2021-02-23 |
| CN112395294B CN112395294B (en) | 2023-07-18 |
Family
ID=74605461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011364309.0A Active CN112395294B (en) | 2020-11-27 | 2020-11-27 | Database data management method and system and database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112395294B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113535729A (en) * | 2021-07-21 | 2021-10-22 | 浪潮云信息技术股份公司 | Method for realizing row and column mixed storage based on RocksDB |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106294421A (en) * | 2015-05-25 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of data write, read method and device |
| CN110032604A (en) * | 2019-02-02 | 2019-07-19 | 阿里巴巴集团控股有限公司 | Data storage device, transfer device and data bank access method |
| CN111209304A (en) * | 2019-12-30 | 2020-05-29 | 华为技术有限公司 | Data processing method, device and system |
| CN111797279A (en) * | 2020-07-17 | 2020-10-20 | 西安数据如金信息科技有限公司 | Data storage method and device |
| CN111858097A (en) * | 2020-07-22 | 2020-10-30 | 安徽华典大数据科技有限公司 | Distributed database system and database access method |
| CN111984696A (en) * | 2020-07-23 | 2020-11-24 | 深圳市赢时胜信息技术股份有限公司 | Novel database and method |
-
2020
- 2020-11-27 CN CN202011364309.0A patent/CN112395294B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106294421A (en) * | 2015-05-25 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of data write, read method and device |
| CN110032604A (en) * | 2019-02-02 | 2019-07-19 | 阿里巴巴集团控股有限公司 | Data storage device, transfer device and data bank access method |
| CN111209304A (en) * | 2019-12-30 | 2020-05-29 | 华为技术有限公司 | Data processing method, device and system |
| CN111797279A (en) * | 2020-07-17 | 2020-10-20 | 西安数据如金信息科技有限公司 | Data storage method and device |
| CN111858097A (en) * | 2020-07-22 | 2020-10-30 | 安徽华典大数据科技有限公司 | Distributed database system and database access method |
| CN111984696A (en) * | 2020-07-23 | 2020-11-24 | 深圳市赢时胜信息技术股份有限公司 | Novel database and method |
Non-Patent Citations (2)
| Title |
|---|
| 赵江: "基于LevelDB的分布式数据库的研究与实现", 《中国优秀博硕士学位论文全文数据库（硕士）信息科技辑（月刊）》 * |
| 赵江: "基于LevelDB的分布式数据库的研究与实现", 《中国优秀博硕士学位论文全文数据库（硕士）信息科技辑（月刊）》, no. 01, 15 January 2020 (2020-01-15) * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113535729A (en) * | 2021-07-21 | 2021-10-22 | 浪潮云信息技术股份公司 | Method for realizing row and column mixed storage based on RocksDB |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112395294B (en) | 2023-07-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10303691B2 (en) | Column-oriented database processing method and processing device | |
| US6125360A (en) | Incremental maintenance of materialized views containing one-to-N lossless joins | |
| CN103198159B (en) | A kind of many copy consistency maintaining methods of isomeric group reformed based on affairs | |
| US20230161758A1 (en) | Distributed Database System and Data Processing Method | |
| US6134543A (en) | Incremental maintenance of materialized views containing one-to-one lossless joins | |
| CN111190935B (en) | Data reading method and device, computer equipment and storage medium | |
| US20120221534A1 (en) | Database index management | |
| WO2021184761A1 (en) | Data access method and apparatus, and data storage method and device | |
| EP4170509A1 (en) | Method for playing back log on data node, data node, and system | |
| CN104317944B (en) | A kind of timestamp dynamic adjustment concurrency control method based on formula | |
| CN112286941A (en) | Big data synchronization method and device based on Binlog + HBase + Hive | |
| CN116108057B (en) | Distributed database access method, device, equipment and storage medium | |
| CN113495872A (en) | Transaction processing method and system in distributed database | |
| CN103810219A (en) | Line storage database-based data processing method and device | |
| CN115617571A (en) | Data backup method, device, system, equipment and storage medium | |
| CN112395294B (en) | Database data management method and system and database | |
| CN112000649A (en) | Incremental data synchronization method and device based on map reduce | |
| JPH04219844A (en) | High-speed medium preferential release type exclusive system | |
| CN112783927A (en) | Database query method and system | |
| CN113448964A (en) | Hybrid storage method and device based on graph-KV | |
| CN115408383A (en) | High-concurrency data storage method and device, electronic equipment and storage medium | |
| JPH09146804A (en) | Data matching device | |
| US8706769B1 (en) | Processing insert with normalize statements | |
| JP2933486B2 (en) | How to search all databases simultaneously | |
| CN115905402B (en) | Method and device for processing transaction log |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |