CN103365987B - Clustered database system and data processing method based on shared-disk framework - Google Patents
Clustered database system and data processing method based on shared-disk framework Download PDFInfo
- Publication number
- CN103365987B CN103365987B CN201310282114.5A CN201310282114A CN103365987B CN 103365987 B CN103365987 B CN 103365987B CN 201310282114 A CN201310282114 A CN 201310282114A CN 103365987 B CN103365987 B CN 103365987B
- Authority
- CN
- China
- Prior art keywords
- data
- main controlled
- static
- controlled node
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The invention relates to a clustered database system and data processing method based on a shared-disk framework. The system comprises main control nodes and transaction nodes, wherein the transaction nodes are used for processing transactions and querying static data in a clustered database; the main control nodes are used for carrying out update operation on data in the clustered database, generating dynamic data, saving and querying the dynamic data. According to a novel database management system framework and the data processing method under the cluster environment of shared-disk framework, advantages (such as large internal storage, SSD (Solid State Disk) and other latest hardware technologies) of latest hardware technologies are adopted, so that the performance of the database is improved remarkably, the cooperative work of multiple computes is utilized fully, limitation of the database on the aspect of expansibility is broken through, and the system performance and throughput of transaction processing are improved.
Description
Technical field
The present invention relates to database technical field, more particularly to a kind of clustered database system based on shared disk framework
And data processing method.
Background technology
Database is for preserving the final result of calculating, so being the most important components of whole information system.
Meeting ever-increasing Transaction Processing application aspect, current database technology also exists in fact much urgently needs solution
Technical problem certainly.
For all of database, in addition to recording correct result, they all suffer from four aspects
Challenge：How processing speed, availability of data, Information Security and data set extensibility are improved, that is to say, that how to make to work as
Front database has the scalability of this four aspect, enables client while obtaining higher processing speed, higher data can
With property, higher Information Security and bigger data set.Multiple databases are linked togather composition data storehouse cluster to reach
Above-mentioned target is a natural idea.Clustering is to use specific connected mode, and price is relatively low
Hardware device combine, there is provided the suitable task disposal ability of high-performance, at the same ensure issued transaction correctness.
Data-base cluster mainly has following three kinds of structures：
1）Shared drive（Main storage）Structure（Shared_Memory, SM）
2）Shared-disk architecture（Shared_Disk, SD）
3）Shared nothing structure（Shared_Nothing, SN）
It is at present Oracle Real Application Cluster than more typical system under shared-disk architecture
（Oracle RAC）, this system is the parallel cluster of Oracle, and the Oracle examples positioned at different server system are visited simultaneously
Same oracle database is asked, is communicated by private network between node, all of control file, online daily record sum
Be stored on shared equipment according to file, can by cluster in all nodes read while write.Each node in RAC is equity
Relation, the request of each node receive user, and give user's returning result.RAC employs Cache Fusion (caches
Merge) technology, the data buffer zone of each node carries out the transmission of data block by the internal network of high speed, low latency.The party
The subject matter that case is present is that inter-node traffic is big, and node buffering area simultaneous techniques is complicated.
The content of the invention
The purpose of the present invention be for the problems referred to above, propose a kind of clustered database system based on shared disk framework and
Data processing method, to overcome the traffic between prior art interior joint big, the complicated problem of node buffering area simultaneous techniques.
For achieving the above object, the present invention proposes a kind of clustered database system based on shared disk framework, the system
Including：Main controlled node and transaction nodes；
The transaction nodes, for processing Cluster Database in issued transaction and static data inquiry；
The main controlled node, for carrying out data to Cluster Database operation is updated, and produces dynamic data, is preserved and is inquired about
The dynamic data.
Optionally, in an embodiment of the present invention, the main controlled node is further used for inquiring about the dynamic data, will look into
Ask result and be back to the transaction nodes；
The static data acquisition static state that the transaction nodes are further used for being inquired about on disk according to data processing request is looked into
Result is ask, the Query Result of the main controlled node is merged with the Static Inquiry result according to mark is obtained result set.
Optionally, in an embodiment of the present invention, the main controlled node is used to be preserved the data of insertion.
Optionally, in an embodiment of the present invention, the main controlled node be used for record need modification data mark and
Data after correspondence renewal；
Optionally, in an embodiment of the present invention, the main controlled node is used for the mark of the data that record needs to delete.
Optionally, in an embodiment of the present invention, the transaction nodes adopt relational database management system engine.
Optionally, in an embodiment of the present invention, the main controlled node adopts main storage data base engine.
Optionally, in an embodiment of the present invention, the main controlled node is further used for periodically or according to core buffer
Service condition by the daynamic transformation be disk on static data.
Optionally, in an embodiment of the present invention, the main controlled node is further used for periodically or according to core buffer
Service condition the dynamic data is forwarded to into the transaction nodes；
The transaction nodes are further used for that the data that the main controlled node is forwarded are converted to the static state on disk
Data.
For achieving the above object, the present invention also proposes a kind of data processing method, and the method includes：
The transaction nodes process the inquiry operation of the static data in Cluster Database；
The main controlled node carries out data to Cluster Database and updates operation, produces dynamic data, preserves and inquires about and be described
Dynamic data.
Optionally, in an embodiment of the present invention, the method also includes：
The main controlled node inquires about the dynamic data and obtains the first Query Result, and is forwarded to the transaction nodes；
The transaction nodes obtain Static Inquiry result according to the static data that data processing request is inquired about on disk, according to
Identify to merge first Query Result with the Static Inquiry result and obtain result set.
Optionally, in an embodiment of the present invention, the main controlled node is preserved the data of insertion.
Optionally, in an embodiment of the present invention, main controlled node record needs the mark of data to be modified and right
Data after should updating；
Optionally, in an embodiment of the present invention, the main controlled node record needs the mark of the data deleted.
Above-mentioned technical proposal has the advantages that：Under a kind of cluster environment of new shared disk framework, data
Base management system framework and its data processing method, using the advantage of newest hardware technology（Such as big internal memory, the newest hardware of SSD
Technology）, the performance of database is significantly improved, and multiple stage computers collaborative work is made full use of, database is broken through in extension
Property in terms of limitation, improve system performance and issued transaction handling capacity.
Description of the drawings
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
The accompanying drawing to be used needed for having technology description is briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is a kind of clustered database system block diagram based on shared disk framework proposed by the present invention；
Fig. 2 is a kind of one of data processing method flow chart proposed by the present invention；
Fig. 3 is the two of a kind of data processing method flow chart proposed by the present invention；
Fig. 4 is clustered database system Organization Chart of the embodiment of the present invention based on shared disk framework；
Fig. 5 is data processing method flow chart in the embodiment of the present invention；
Fig. 6 is data processing method flow chart of the embodiment of the present invention based on the system architecture of Fig. 4；
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described.Obviously, described embodiment is only a part of embodiment of the invention, rather than the embodiment of whole.It is based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under the premise of creative work is not made
Embodiment, belongs to the scope of protection of the invention.
As shown in figure 1, for a kind of clustered database system block diagram based on shared disk framework proposed by the present invention.This is
System includes：Main controlled node 101 and transaction nodes 102；
The transaction nodes 102, for processing Cluster Database in issued transaction and static data inquiry operation；
The main controlled node 101, for Cluster Database is carried out data update operation, produce dynamic data, preserve and
Inquire about the dynamic data.
Optionally, in an embodiment of the present invention, the main controlled node 101 is further used for inquiring about the dynamic data,
Query Result is back to into the transaction nodes；
The static data that the transaction nodes 102 are further used for being inquired about on disk according to data processing request obtains static
Query Result, the Query Result of the main controlled node is merged obtain result set according to mark with the Static Inquiry result.
Optionally, in an embodiment of the present invention, the main controlled node 101 is used to be preserved the data of insertion.
Optionally, in an embodiment of the present invention, the main controlled node 101 be used for the mark of data that record changed and
Data after correspondence renewal；
Optionally, in an embodiment of the present invention, the main controlled node 101 is used for the mark of the data that record needs to delete
Know.
Optionally, in an embodiment of the present invention, the transaction nodes 102 adopt relational database management system engine.
Optionally, in an embodiment of the present invention, the main controlled node 101 adopts main storage data base engine.
Optionally, in an embodiment of the present invention, the main controlled node 101 is further used for the dynamic data periodically
Be converted to the static data on disk.
Optionally, in an embodiment of the present invention, the main controlled node 101 is further used for the dynamic data periodically
It is forwarded to the transaction nodes；
The transaction nodes 102 are further used for the data that the main controlled node 101 is forwarded being converted on disk
Static data.
As shown in Fig. 2 being a kind of one of data processing method flow chart proposed by the present invention.The method includes：
Step 201）：The transaction nodes process the inquiry operation of the static data in Cluster Database；
Step 202）：The main controlled node Cluster Database is carried out data update operation, produce dynamic data, preserve and
Inquire about the dynamic data.
As shown in figure 3, being the two of a kind of data processing method flow chart proposed by the present invention.The method also includes：
Step 203）：The main controlled node inquires about the dynamic data and obtains the first Query Result, and is forwarded to the thing
Business node；
Step 204）：The transaction nodes obtain Static Inquiry according to the static data that data processing request is inquired about on disk
As a result, first Query Result is merged with the Static Inquiry result according to mark and obtains result set.
Embodiment：
The present embodiment provides database management system architecture and its number under a kind of cluster environment of new shared disk framework
According to processing method, using the advantage of newest hardware technology（Such as big internal memory, the newest hardware technologies of SSD）, significantly improve data
The performance in storehouse, and multiple stage computers collaborative work is made full use of, limitation of the database in terms of autgmentability is broken through, improve system
Performance and issued transaction handling capacity.
As shown in figure 4, for a kind of clustered database system Organization Chart based on shared disk framework of the embodiment of the present invention.
The computer node of composition data storehouse cluster has two types：Transaction nodes and main controlled node；Transaction nodes adopt relation data
Base management system（RDB）Engine, main controlled node adopts main storage data base（MMDB）Engine.Transaction nodes receive at the data of user
Reason request.Static data on transaction nodes inquiry disk, all increasings to data-base cluster, deletes, changes all on main controlled node
Carry out, update result and be stored in the internal memory of main controlled node.When the data buffer zone of main controlled node is full of or system is idle, it is
System is then merged on disk batch is updated the data, and forms new static data.
Increasing in request, delete, change operation by main controlled node process；For inquiry operation in the request of user, then it is divided into two
Part is carried out：Static data on transaction nodes inquiry disk, master control node inquiry dynamic data, this two parts is merged into just
True result set, returns to user.
For the insertion operation of database（INSERT sentences）, by main controlled node process, it is stored in internal storage data buffering area
In；
For the modification of database is operated（UPDATE statement）, inquire needs the old tuple of modification first, generates and updates
New tuple afterwards；New tuple by main controlled node process, in the data buffer zone of main controlled node the mark of record modification tuple and
Corresponding new tuple；
For the deletion action of database（DELETE sentences）, the old tuple that first inquiry needs are changed, if it is present
The tuple identity of record deletion in the data buffer zone of main controlled node；
For the inquiry operation of database（SELECT statement）, it is divided into two parts：Static data inquire about and dynamic number it is investigated that
Ask, static data inquiry is carried out on transaction nodes, and dynamic data retrieval is carried out on main controlled node, according to tuple identity this
Two parts merge to form final correct result and return to user；
Under normal circumstances, a typical shared disk framework data-base cluster is by a main controlled node and multiple affairs sections
Point is constituted.Under High Availabitity configuring condition, can also increase a server, be that main controlled node is matched somebody with somebody as the standby host of main controlled node
Put special two-shipper and do memory mirror, so as to ensure the real uninterrupted of Transaction Service, while standby host is also subjected to read request.
Furthermore it is also possible on the same server while operational relation database（RDB）Engine and main storage data base（M
MDB）Engine, i.e. main controlled node and transaction nodes are integrated on same server, in the manner described above while processing data storehouse
User's request.
As shown in figure 5, being data processing method flow chart in the embodiment of the present invention.The method includes：
Step 501）：Client to transaction nodes send SQL query instruction；
Step 502）：Transaction nodes carry out parsing and generate inquiry plan to SQL query instruction；
Step 503）：When inquiry plan inserts a data, the transaction nodes are by the data is activation inserted to main controlled node；
When inquiry plan deletes a data, the transaction nodes and the main controlled node perform inquiry operation to obtain needs
The mark of the data of deletion, and the mark for needing the data deleted is sent to main controlled node；
When inquiry plan changes a data, the transaction nodes inquire the mark of the data changed, by what is changed
Data is activation after the mark of data and correspondence renewal is to the main controlled node；
Step 504）：When inquiry plan inquires about a data, the static data on the transaction nodes inquiry disk obtains quiet
State Query Result；
Step 505）：The main controlled node performs the operation of inquiry plan, and returns a result set to the transaction nodes；
Step 506）：Result set and the Static Inquiry that the transaction nodes return the main controlled node according to mark
As a result merge and obtain final result collection, and the final result collection is back to into the client.
Preferably, also include：The transaction nodes carry out parsing and generate appearance after inquiry plan to SQL query instruction
Abnormality processing, then the transaction nodes are to client transmission mistake instruction.
As shown in fig. 6, the data processing method flow chart for the embodiment of the present invention based on the system architecture of Fig. 4.Make below
Further illustrated with reference to Fig. 6 with a specific database manipulation example：
Assume that data-base cluster is made up of three nodes N1, N2 and N3, wherein N1, N2 are transaction nodes, and N3 is master control section
Point；
Assume there is Table A in database, there are three tuples in A（1,1,1）,（2,2,2）,（3,3,3）, it is assumed that first row is table
The unique mark of middle tuple；
When node N1 receives user inquires about the request of A tables, three tuples of Table A read the data of node N1 from disk
In buffering area, 3 tuples are inquired from node N1, then query node N3, the tuple without A tables then returns to user three
Tuple；
New tuple is inserted when node N1 receives user to A tables（4,4,4）During request, node N1 constructs new tuple, sends
Node N3 is given, the data buffer zone of node N3 is inserted into；
When node N1 receives user the tuple in A tables（1,1,1）It is modified as（1,2,2）Request when, then node N1
Construct new tuple（1,2,2）, node N3 is sent to, new tuple is inserted into the data buffer zone of node N3；
When node N1 receives the tuple in user's deletion A tables（2,2,2）During request, then node N1 acknowledgment of your inquiry first is deposited
In tuple（2,2,2）, node N3 is then notified, a special tuple is inserted in the data buffer zone of node N3（2, del）；
When node N1 is connected to when user inquires about again A tables, three tuples are inquired from node N1（1,1,1）,（2,2,
2）,（3,3,3）, three tuples are inquired from node N3（4,4,4）,（1,2,2）,（2, del）, node N1 is two result sets
Merge, result set can be obtained（1,2,2）,（3,3,3）,（4,4,4）, return to user.
Above-described specific embodiment, has been carried out further to the purpose of the present invention, technical scheme and beneficial effect
Describe in detail, should be understood that the specific embodiment that the foregoing is only the present invention, be not intended to limit the present invention
Protection domain, all any modification, equivalent substitution and improvements within the spirit and principles in the present invention, done etc. all should include
Within protection scope of the present invention.
Claims (6)
1. a kind of clustered database system based on shared disk framework, it is characterised in that the system includes：Main controlled node and thing
Business node；
The transaction nodes, for receiving the data processing request of user；And inquire about quiet on disk according to data processing request
State data acquisition Static Inquiry result, merges the Query Result of the main controlled node with the Static Inquiry result according to mark
Obtain result set；Wherein, union operation includes：According to the content recorded in inquiry dynamic data, the data addition of new insertion
Corresponding initial data in former Static Inquiry result is replaced in Static Inquiry result, the data for updating, according to the number deleted
The corresponding data of Static Inquiry result is deleted according to mark；
The main controlled node, for data to be carried out to Cluster Database according to data processing request operation is updated, and produces dynamic number
According to；And for inquiring about the dynamic data, Query Result is back to into the transaction nodes；
Wherein, the main controlled node Cluster Database is carried out data update operation specifically include:The data of insertion are protected
Deposit, record need modification data mark and correspondence update after data, record need delete data mark, the master
Control node carries out the data of data renewal operation generation and is referred to as dynamic data.
2. the system as claimed in claim 1, it is characterised in that the transaction nodes are drawn using relational database management system
Hold up.
3. the system as claimed in claim 1, it is characterised in that the main controlled node adopts main storage data base engine.
4. the system as claimed in claim 1, it is characterised in that the main controlled node is further used for periodically or slow according to internal memory
The daynamic transformation is the static data on disk by the service condition for rushing area.
5. the system as claimed in claim 1, it is characterised in that the main controlled node is further used for periodically or slow according to internal memory
The dynamic data is forwarded to the transaction nodes by the service condition for rushing area；
The transaction nodes are further used for that the data that the main controlled node is forwarded are converted to the static data on disk.
6. a kind of data processing method, it is characterised in that the method includes：
Transaction nodes receive the data processing request of user；And the static data inquired about on disk according to data processing request is obtained
Static Inquiry result, the Query Result of main controlled node is merged obtain result set according to mark with the Static Inquiry result；Its
In, union operation includes：According to the content recorded in inquiry dynamic data, the data of new insertion are added to Static Inquiry result
In, the data for updating are replaced corresponding initial data in former Static Inquiry result, delete static according to the Data Identification deleted
The corresponding data of Query Result；
The main controlled node carries out data to Cluster Database and updates operation according to data processing request, produces dynamic data, and
For inquiring about the dynamic data, Query Result is back to into the transaction nodes；
Wherein, the main controlled node Cluster Database is carried out data update operation specifically include:The data of insertion are protected
Deposit, record need modification data mark and correspondence update after data, record need delete data mark, the master
Control node carries out the data of data renewal operation generation and is referred to as dynamic data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310282114.5A CN103365987B (en) | 2013-07-05 | 2013-07-05 | Clustered database system and data processing method based on shared-disk framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310282114.5A CN103365987B (en) | 2013-07-05 | 2013-07-05 | Clustered database system and data processing method based on shared-disk framework |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103365987A CN103365987A (en) | 2013-10-23 |
| CN103365987B true CN103365987B (en) | 2017-04-12 |
Family
ID=49367328
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310282114.5A Active CN103365987B (en) | 2013-07-05 | 2013-07-05 | Clustered database system and data processing method based on shared-disk framework |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103365987B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984768B (en) | 2014-05-30 | 2017-09-29 | 华为技术有限公司 | A kind of data-base cluster manages method, node and the system of data |
| CN104408097B (en) * | 2014-11-17 | 2018-07-20 | 深圳市比一比网络科技有限公司 | One kind is based on the newer hybrid index method and system of character field heat |
| CN104504147B (en) | 2015-01-04 | 2018-04-10 | 华为技术有限公司 | A kind of resource coordination method of data-base cluster, apparatus and system |
| CN105354046B (en) * | 2015-09-15 | 2019-03-26 | 深信服科技股份有限公司 | Database update processing method and system based on shared disk |
| CN106372162A (en) * | 2016-08-31 | 2017-02-01 | 天津南大通用数据技术股份有限公司 | Extension method and device of database cluster application |
| CN112162832B (en) * | 2020-09-08 | 2024-02-09 | 北京人大金仓信息技术股份有限公司 | Method and device for realizing audit data storage under multi-version concurrency control |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7484007B2 (en) * | 2002-02-01 | 2009-01-27 | Codekko Inc. | System and method for partial data compression and data transfer |
| CN1182465C (en) * | 2002-12-04 | 2004-12-29 | 联想(北京)有限公司 | Method for dynamic transferring data and its storing system |
| CN101068237B (en) * | 2006-08-28 | 2012-01-25 | 腾讯科技（深圳）有限公司 | Data access system and data access method |
| CN102375884A (en) * | 2011-10-21 | 2012-03-14 | 北京百度网讯科技有限公司 | Method and device for data compression for page access object |
-
2013
- 2013-07-05 CN CN201310282114.5A patent/CN103365987B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN103365987A (en) | 2013-10-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11816126B2 (en) | Large scale unstructured database systems | |
| CN103365987B (en) | Clustered database system and data processing method based on shared-disk framework | |
| Baker et al. | Megastore: Providing scalable, highly available storage for interactive services. | |
| Bronson et al. | {TAO}:{Facebook’s} distributed data store for the social graph | |
| US9146934B2 (en) | Reduced disk space standby | |
| Padhy et al. | RDBMS to NoSQL: reviewing some next-generation non-relational database’s | |
| US9922075B2 (en) | Scalable distributed transaction processing system | |
| Qader et al. | A comparative study of secondary indexing techniques in LSM-based NoSQL databases | |
| CN102663117B (en) | OLAP (On Line Analytical Processing) inquiry processing method facing database and Hadoop mixing platform | |
| Tatarowicz et al. | Lookup tables: Fine-grained partitioning for distributed databases | |
| Gajendran | A survey on nosql databases | |
| CN103605698A (en) | Cloud database system used for distributed heterogeneous data resource integration | |
| CN109656958B (en) | Data query method and system | |
| US10853193B2 (en) | Database system recovery using non-volatile system memory | |
| CN104850572A (en) | HBase non-primary key index building and inquiring method and system | |
| CN109299056B (en) | A kind of method of data synchronization and device based on distributed file system | |
| US20140012867A1 (en) | Method And Process For Enabling Distributing Cache Data Sources For Query Processing And Distributed Disk Caching Of Large Data And Analysis Requests | |
| Xiong et al. | Data vitalization: a new paradigm for large-scale dataset analysis | |
| CN110287150A (en) | A kind of large-scale storage systems meta-data distribution formula management method and system | |
| US11250019B1 (en) | Eventually consistent replication in a time-series database | |
| US11803572B2 (en) | Schema-based spatial partitioning in a time-series database | |
| CN107766355B (en) | Hierarchical data management method, hierarchical data management system and instant messaging system | |
| CN108769166A (en) | A kind of CDN cache contents managing devices based on metadata | |
| Zhang et al. | Blockchain storage middleware based on external database | |
| CN114385577A (en) | Distributed file system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |