CN112395294B - Database data management method and system and database - Google Patents

Database data management method and system and database Download PDF

Info

Publication number
CN112395294B
CN112395294B CN202011364309.0A CN202011364309A CN112395294B CN 112395294 B CN112395294 B CN 112395294B CN 202011364309 A CN202011364309 A CN 202011364309A CN 112395294 B CN112395294 B CN 112395294B
Authority
CN
China
Prior art keywords
data
storage area
database
stored
write
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011364309.0A
Other languages
Chinese (zh)
Other versions
CN112395294A (en
Inventor
马静伟
周恒�
于伟
高传集
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Cloud Information Technology Co Ltd
Original Assignee
Inspur Cloud Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Cloud Information Technology Co Ltd filed Critical Inspur Cloud Information Technology Co Ltd
Priority to CN202011364309.0A priority Critical patent/CN112395294B/en
Publication of CN112395294A publication Critical patent/CN112395294A/en
Application granted granted Critical
Publication of CN112395294B publication Critical patent/CN112395294B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a database data management method, a system and a database, when writing data are received, a data table is created according to the writing data, mode information of the data table is stored in a second storage area, the writing data are stored in a first storage area, then the writing data stored in the first storage area are transmitted to the second storage area, and the writing data are stored in the second storage area in a data table mode according to the mode information of the data table. The invention directly stores the data into the first storage area, asynchronously transmits the data to the second storage area and stores the data in the form of the data table, and can realize the simultaneous support of relational storage and non-relational storage of the data, thereby providing technical preparation and support for supporting richer data processing scenes for the database.

Description

Database data management method and system and database
Technical Field
The present invention relates to the field of database technologies, and in particular, to a database data management method and system. The invention also relates to a database.
Background
Database management systems have various classification methods, which can be classified into a distributed type and a centralized type from the aspect of system architecture, can be classified into a relational type and a non-relational type according to a data storage structure, can be classified into an SQL database and a NoSQL database from the aspect of supporting capability of the data storage structure. SQL (Structured Query Language) refers to relational databases, which have the advantage of uniformity and ease of use, with the disadvantage that their performance drops dramatically as databases grow in face of large amounts of data. NoSQL (Not Only SQL) refers broadly to a non-relational database at the expense of relaxing the ACID principle, noSQL resorts to the final consistency principle, rather than adhering exactly to the ACID principle as in a relational database, meaning that if there is no update of a particular data item within a particular time period, all accesses thereto will eventually return a last updated value, exhibiting substantial availability, soft state and final consistency. In practical application, the relational database and the non-relational database are applicable to different application scenes based on the characteristics of the relational database and the non-relational database, and each database cannot be applicable to all application scenes, so that the application range of the databases is limited.
Disclosure of Invention
The invention aims to provide a database data management method and system, which provide technical preparation and support for supporting richer data processing scenes for a database. The invention also provides a database.
In order to achieve the above purpose, the present invention provides the following technical solutions:
a database data management method, comprising:
when write-in data is received, creating a data table according to the write-in data and storing mode information of the data table into a second storage area;
storing the write data to a first storage area;
the write data stored to the first storage area is transferred to the second storage area to store the write data in the form of a data table to the second storage area according to the pattern information of the data table.
Preferably, storing the write data in the first storage area includes: generating an identifier according to the identifier information of the writing data, generating the data pointed by the identifier from the writing data, and storing the writing data into the first storage area.
Preferably, the first storage area includes a plurality of nodes, the plurality of nodes include a management node and a member node, the management node is configured to obtain the write data transmitted by the client, generate a write data copy from the write data and transmit the write data copy to the member node, and the member node is configured to store the write data copy according to an instruction of the management node.
Preferably, when the created pattern information of the data table is changed, the changed pattern information is stored in the second storage area to update the pattern information stored in the second storage area.
Preferably, storing the write data in the second storage area in the form of a data table includes: the data table includes a plurality of columns, each column storing data of a different field of the write data, respectively.
Preferably, when a data reading request is received, if the data reading request indicates that data is read from the second storage area, whether the data stored in the second storage area is consistent with the data stored in the first storage area is judged;
if yes, responding to the data reading request, and if not, responding to the data reading request after the stored data of the first storage area is transmitted to the second storage area and the storage is completed.
A database data management system for performing the database data management method described above.
A database comprising the database data management system described above.
According to the technical scheme, when the writing data is received, the data table is created according to the writing data, the mode information of the data table is stored in the second storage area, the writing data is stored in the first storage area, then the writing data stored in the first storage area is transmitted to the second storage area, and the writing data is stored in the second storage area in a data table mode according to the mode information of the data table.
The database management method and the system of the invention directly store the data in the first storage area, asynchronously transmit the data to the second storage area and store the data in the form of a data table, and can realize the simultaneous support of relational storage and non-relational storage of the data, thereby providing technical preparation and support for the database to support richer data processing scenes.
The database provided by the invention can achieve the beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a database data management method according to an embodiment of the present invention;
FIG. 2 (a) is a schematic diagram illustrating a first storage area transferring storage data to a second storage area for storage according to an embodiment of the present invention;
FIG. 2 (b) is a schematic diagram illustrating reading data from the second storage area according to an embodiment of the present invention;
fig. 3 is a flowchart of a method for responding to a data reading request in a database data management method according to an embodiment of the present invention.
Detailed Description
In order to make the technical solution of the present invention better understood by those skilled in the art, the technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart of a database data management method according to an embodiment of the present invention, and as can be seen from the figure, the database data management method includes the following steps:
s10: when the write data is received, a data table is created from the write data and the mode information of the data table is stored in the second storage area.
The data table defines a format in which write data is stored, and pattern information of the data table describes a relationship of rows and columns of the data table and a format in which write data is stored according to the data table.
Upon receiving the write data provided by the client, creating a data table from the write data, and storing pattern information of the created data table to the second storage area. Alternatively, the second storage area may be any one of the distributed storage nodes.
S11: the write data is stored to a first storage area.
The write data is stored directly to the first storage area upon receipt of the write data. Alternatively, an identifier may be generated according to the identifier information of the write data, and the write data may be generated to the data to which the identifier points, so that the write data is stored in the first storage area. The first storage area can store the written data in a line memory mode, an initial key code in the written data is formed into a key, and other data are correspondingly stored as data pointed by the key.
Alternatively, the first storage area may employ an embeddable, persistent-enabled key-value storage system in which all data is stored in sequence, common operations including Get (key), put (key), delete (key), and Scan (key). The storage system has three basic structures: rocksDB memtable, sstfile, and logfile. memtable is a memory data structure that inserts new data into the memtable and log file (optional). The log files are written sequentially and located on disk. When the femto is full, the data will be refreshed to the sstfile file on the disk, and the corresponding log file can be deleted safely. The data in sstfile is ordered in order to speed up key lookups.
Further preferably, the first storage area may include a plurality of nodes, the plurality of nodes including a management node and a member node, the management node being configured to obtain the write data transmitted by the client, generate a copy of the write data from the write data, and transmit the copy of the write data to the member node, and the member node being configured to store the copy of the write data according to an instruction of the management node. In the storage mode, a management node is selected from the node cluster, and the management node is responsible for receiving the transaction request from the client and notifying other nodes in the cluster to ensure that the other nodes are synchronous with own log. When the management node is down, other nodes in the cluster can initiate election to reselect the management node.
S12: the write data stored to the first storage area is transferred to the second storage area to store the write data in the form of a data table to the second storage area according to the pattern information of the data table.
And pushing and transmitting the writing data stored in the first storage area to the second storage area, and further storing the writing data in the form of a data table in the second storage area according to the mode information of the created data table. Optionally, the created data table includes a plurality of columns, each column storing data written to a different field of the data. Each field may be assigned a data type defining their data length and other attributes, and may contain various characters, numbers, or even graphics. A row of the data table is called a record and the intersection of a row and a column indicates a certain attribute value. For example, please refer to fig. 2 (a), fig. 2 (a) is a schematic diagram illustrating the transmission of the storage data of the first storage area to the second storage area for storage in an embodiment. It can be seen that the data is stored in key-value form in the first memory area, and after the data of the first memory area is transferred to the second memory area, the data is subjected to determinant splitting, and is recombined and stored according to the format specified by the data table.
In the method of the embodiment, after the writing data is stored in the first storage area, the execution result can be returned without waiting for the completion of writing the data in the second storage area.
Preferably, in the method of this embodiment, since the mode information that does not include data in the process of transferring the data to be written from the first storage area to the second storage area does not include data, a mode information update trigger mechanism is established for the method of this embodiment, and when the mode information of the created data table is changed, the changed mode information is stored in the second storage area to update the mode information stored in the second storage area, so that it is ensured that the mode information stored in the second storage area remains updated.
According to the database data management method, data is directly stored in the first storage area, and is asynchronously transmitted to the second storage area to be stored in the form of a data table, so that the relational storage and the non-relational storage of the data can be simultaneously supported, and technical preparation and support are provided for the database to support richer data processing scenes.
Referring to fig. 3, fig. 3 is a flowchart of a method for responding to a data reading request in a database data management method according to another embodiment, and the database data management method according to this embodiment further includes the following steps:
s20: a data read request is received, the data read request indicating reading of data from the second storage area.
Write data is stored in a first storage area and a second storage area, respectively, where the data may be stored in different forms. In an actual application scenario, an application end may require reading the obtained data in the form of data stored in the first storage area, and then the data needs to be read from the first storage area; or may require that the data be read in the form of the second storage area where it is desired to read the data from the second storage area.
When a data read request is received, data is read from the first storage area or from the second storage area according to a requirement of the data read request. In the method of this embodiment, the write data is directly stored in the first storage area, so when the data reading request indicates to read the data from the first storage area, the data can be directly read from the first storage area according to the data reading request, and returned to the application end.
If the data read request indicates that the data is read from the second storage area, since the data written in the first storage area is transferred to the second storage area for storage when the written data is stored, the data transfer to the second storage area for storage when the data read request is received is not necessarily successful or is not completed synchronously, and therefore, in order to obtain the latest data, it is necessary to ensure that the data stored in the second storage area is consistent with the data stored in the first storage area when the data is read from the second storage area.
S21: and judging whether the data stored in the second storage area is consistent with the data stored in the first storage area. If yes, the process proceeds to step S22, and if no, the process proceeds to step S23.
S22: responding to the data reading request.
If the data stored in the second storage area is consistent with the data stored in the first storage area, the data can be directly responded to the data reading request, and the data can be read from the second storage area according to the data reading request.
S23: waiting for the storage data of the first storage area to be transferred to the second storage area and completing the storage. And proceeds to step S22 after the stored data of the first storage area is transferred to the second storage area and the storage is completed. For example, please refer to fig. 2 (b), fig. 2 (b) is a schematic diagram illustrating reading data from the second storage area in an embodiment.
Correspondingly, the embodiment of the invention also provides a database data management system for executing the database data management method.
The database data management system of the present embodiment, when receiving write data, creates a data table from the write data and stores pattern information of the data table to the second storage area, stores the write data to the first storage area, and then transfers the write data stored to the first storage area to the second storage area to store the write data in the form of a data table to the second storage area according to the pattern information of the data table. The database data management system of the embodiment directly stores data in the first storage area, asynchronously transmits the data to the second storage area to be stored in the form of a data table, and can simultaneously support relational storage and non-relational storage of the data, thereby providing technical preparation and support for supporting richer data processing scenes for the database.
Correspondingly, the embodiment of the invention also provides a database, which comprises the database data management system.
The database of the present embodiment creates a data table from the write data and stores pattern information of the data table to the second storage area when the write data is received, stores the write data to the first storage area, and then transfers the write data stored to the first storage area to the second storage area to store the write data in the form of a data table to the second storage area according to the pattern information of the data table. The database of the embodiment directly stores the data in the first storage area, asynchronously transmits the data to the second storage area and stores the data in the form of a data table, and can simultaneously support relational storage and non-relational storage of the data, thereby providing technical preparation and support for the database to support richer data processing scenes.
The database data management method, the database data management system and the database provided by the invention are described in detail. The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to facilitate an understanding of the method of the present invention and its core ideas. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the invention can be made without departing from the principles of the invention and these modifications and adaptations are intended to be within the scope of the invention as defined in the following claims.

Claims (6)

1. A database data management method, comprising:
when write data is received, creating a data table according to the write data and storing mode information of the data table into a second storage area, wherein the data table defines a format for storing the write data, and the mode information of the data table describes a relation between rows and columns of the data table and a format for storing the write data according to the data table;
storing the write data to a first storage area;
transferring the write data stored to the first storage area to the second storage area to store the write data in the form of a data table to the second storage area according to pattern information of the data table;
when the created pattern information of the data table is changed, the changed pattern information is stored in the second storage area to update the pattern information stored in the second storage area.
2. The database data management method of claim 1, wherein storing the write data to the first storage area comprises: generating an identifier according to the identifier information of the written data, generating the data pointed by the identifier by the written data, and storing the identifier and the data pointed by the identifier into the first storage area.
3. The database data management method according to claim 1, wherein the first storage area includes a plurality of nodes including a management node for acquiring the write data transmitted by the client and generating a write data copy from the write data for transmission to the member node, and the member node for storing the write data copy according to an instruction of the management node.
4. The database data management method according to claim 1, wherein storing the write data in the form of a data table to the second storage area includes: the data table includes a plurality of columns, each column storing data of a different field of the write data, respectively.
5. The database data management method according to any one of claims 1 to 4, wherein when a data read request is received, if the data read request indicates reading of data from the second storage area, it is judged whether or not the data stored in the second storage area is identical to the data stored in the first storage area;
if yes, responding to the data reading request, and if not, responding to the data reading request after the stored data of the first storage area is transmitted to the second storage area and the storage is completed.
6. A database data management system for performing the database data management method of any of claims 1-5.
CN202011364309.0A 2020-11-27 2020-11-27 Database data management method and system and database Active CN112395294B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011364309.0A CN112395294B (en) 2020-11-27 2020-11-27 Database data management method and system and database

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011364309.0A CN112395294B (en) 2020-11-27 2020-11-27 Database data management method and system and database

Publications (2)

Publication Number Publication Date
CN112395294A CN112395294A (en) 2021-02-23
CN112395294B true CN112395294B (en) 2023-07-18

Family

ID=74605461

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011364309.0A Active CN112395294B (en) 2020-11-27 2020-11-27 Database data management method and system and database

Country Status (1)

Country Link
CN (1) CN112395294B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113535729A (en) * 2021-07-21 2021-10-22 浪潮云信息技术股份公司 Method for realizing row and column mixed storage based on RocksDB

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111209304A (en) * 2019-12-30 2020-05-29 华为技术有限公司 Data processing method, device and system
CN111797279A (en) * 2020-07-17 2020-10-20 西安数据如金信息科技有限公司 Data storage method and device

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106294421B (en) * 2015-05-25 2020-02-04 阿里巴巴集团控股有限公司 Data writing and reading method and device
CN110032604B (en) * 2019-02-02 2021-12-07 北京奥星贝斯科技有限公司 Data storage device, translation device and database access method
CN111858097A (en) * 2020-07-22 2020-10-30 安徽华典大数据科技有限公司 Distributed database system and database access method
CN111984696B (en) * 2020-07-23 2023-11-10 深圳市赢时胜信息技术股份有限公司 Novel database and method

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111209304A (en) * 2019-12-30 2020-05-29 华为技术有限公司 Data processing method, device and system
CN111797279A (en) * 2020-07-17 2020-10-20 西安数据如金信息科技有限公司 Data storage method and device

Also Published As

Publication number Publication date
CN112395294A (en) 2021-02-23

Similar Documents

Publication Publication Date Title
US8924365B2 (en) System and method for range search over distributive storage systems
CN102395962B (en) Composite hash and list partitioning of database tables
CN101809558B (en) System and method for remote asynchronous data replication
CN109189608B (en) A kind of method guaranteeing duplication transaction consistency and corresponding reproducing unit
US20230161758A1 (en) Distributed Database System and Data Processing Method
US20080059469A1 (en) Replication Token Based Synchronization
EP4170509A1 (en) Method for playing back log on data node, data node, and system
WO2012108015A1 (en) Data synchronization method, data synchronization program, and data synchronization control device
CN112395294B (en) Database data management method and system and database
US12111817B2 (en) Log execution method and apparatus, computer device and storage medium
CN104317944A (en) Formula-based concurrency control method by timestamp dynamic adjustment
CN116108057B (en) Distributed database access method, device, equipment and storage medium
WO2023083237A1 (en) Graph data management
CN113495872A (en) Transaction processing method and system in distributed database
CN114138907A (en) Data processing method, computer device, storage medium, and computer program product
CN113704267A (en) Data query method, system, equipment and storage medium based on elastic search
JPH04219844A (en) High-speed medium preferential release type exclusive system
US12067278B2 (en) Live data migration in document stores
CN107885824A (en) A kind of data read-write method and system
CN111061759A (en) Data query method and device
EP3345107B1 (en) Apparatus and method for managing storage of primary database and replica database
Sethia et al. MRDMS-mobile replicated database management synchronization
CN112256202B (en) Distributed storage system and method for deleting volumes in distributed storage system
CN114443654A (en) Method and system for modifying length of spatial data block of database table on line
CN114153857A (en) Data synchronization method, data synchronization apparatus, and computer-readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant