CN105912678A - Data storage method, data reading method and corresponding system - Google Patents

Data storage method, data reading method and corresponding system Download PDF

Info

Publication number
CN105912678A
CN105912678A CN201610228439.9A CN201610228439A CN105912678A CN 105912678 A CN105912678 A CN 105912678A CN 201610228439 A CN201610228439 A CN 201610228439A CN 105912678 A CN105912678 A CN 105912678A
Authority
CN
China
Prior art keywords
data
mdl
data base
ddl
rule
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610228439.9A
Other languages
Chinese (zh)
Inventor
张少峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Si Tech Information Technology Co Ltd
Original Assignee
Beijing Si Tech Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Si Tech Information Technology Co Ltd filed Critical Beijing Si Tech Information Technology Co Ltd
Priority to CN201610228439.9A priority Critical patent/CN105912678A/en
Publication of CN105912678A publication Critical patent/CN105912678A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/275Synchronous replication

Abstract

The invention discloses a data storage method, a data reading method and a corresponding system, wherein the data storage method comprises the steps that library division and list division rules are defined on a DDL frame layer; library division and list division are carried out to source data on the DDL frame layer according to the defined library division and list division rules, and sub-data blocks processed by the library division and the list division are stored in corresponding main databases; and each main database synchronizes the corresponding sub-data blocks to multiple subsidiary databases corresponding to the main database, wherein all the main databases are taken as writing databases, and all the subsidiary databases are taken as reading databases. According to the invention, the library division and the list division can be carried out to a large batch of data according to certain rules, so that pressure borne by the database is reduced, and separate reading and writing functions can be implemented to the data.

Description

A kind of system of date storage method, method for reading data and correspondence thereof
Technical field
The present invention relates to technical field of data processing, be specifically related to a kind of date storage method, digital independent Method and the system of correspondence thereof.
Background technology
J2EE application and development at present much uses Spring+Mybatis framework to develop, but Spring+Mybatis itself cannot realize point storehouse and divide the function of table and read and write abruption, also cannot realize moving The switch data source of state.When application system has large-scale data, need according to certain regular logarithm Divide table according to a point storehouse, alleviate data base's pressure.It is also required to be simultaneously connected with multiple heterogeneous database simultaneously, Realize certain operation flow.
Summary of the invention
The technical problem to be solved be to provide a kind of date storage method, method for reading data and The system of its correspondence, it is possible to solve the deficiencies in the prior art.
The technical scheme is that
On the one hand, the invention provides a kind of date storage method, including:
S1a, divide table regular in a DDL ccf layer definition point storehouse;
S2a, divide according to a point storehouse for definition at DDL ccf layer that table is regular to carry out a point storehouse by source data and divide table, Form multiple subdata fast, and each sub-block is stored to corresponding MDL;
The sub-block of self is synchronized to corresponding multiple of this MDL by S3a, each MDL From data base;
Wherein, all of MDL as writing data base, all of from data base as read from database.
On the other hand, the invention provides a kind of method for reading data, including:
S1b, define multiple data queries from data base corresponding to each MDL at DDL ccf layer Routing rule;
S2b, when receiving data read request, according in data read request request data identify, Search corresponding MDL;
S3b, according to multiple data query routing rules from data base corresponding to this MDL, from from Data base searches the data corresponding with data read request.
Another further aspect, the invention provides a kind of data-storage system, including:
First rule definition unit, for dividing table rule in a DDL ccf layer definition point storehouse;
Divide library storage unit, for dividing table rule by source data at DDL ccf layer according to a point storehouse for definition Carry out a point storehouse and divide table, form multiple sub-block, and each sub-block is stored to corresponding main number According in storehouse;
Data synchronisation unit, for being synchronized to the multiple of correspondence by the sub-block in each MDL From data base;
Wherein, all of MDL as writing data base, all of from data base as read from database.
Another aspect, the invention provides a kind of data reading system, including:
Second Rule definition unit, for defining corresponding multiple of each MDL at DDL ccf layer From the data query routing rule of data base;
First searches unit, for when receiving data read request, according in data read request Request data identifies, and searches corresponding MDL;
Second searches unit, for the multiple data query roads from data base corresponding according to this MDL By rule, from searching the data corresponding with data read request from data base.
The system of a kind of date storage method, method for reading data and correspondence thereof that the present invention provides, to greatly The data of batch divide table rule to carry out a point storehouse according to certain point storehouse to form multiple sub-block, and by subnumber Store to corresponding MDL according to block, and each MDL is multiple from data base, often to having Data syn-chronization in one MDL is to multiple from data base, using MDL as writing data base, Using multiple from data base as read from database, when realizing digital independent, first navigate to MDL, Regular from the data searching correspondence from data base according to certain query routing, it is achieved read and write abruption Function.
Accompanying drawing explanation
Fig. 1 is a kind of date storage method flow chart of the embodiment of the present invention one;
Fig. 2 is at data storage rule schematic diagram defined in DDL framework;
Fig. 3 is data Stored Procedure figures;
Fig. 4 is a kind of method for reading data flow chart of the embodiment of the present invention two;
Fig. 5 is a kind of data-storage system schematic diagram of the embodiment of the present invention three;
Fig. 6 is a kind of data reading system schematic diagram of the embodiment of the present invention four.
Detailed description of the invention
Being described principle and the feature of the present invention below in conjunction with accompanying drawing, example is served only for explaining this Invention, is not intended to limit the scope of the present invention.
Embodiment one, a kind of date storage method.Number the present embodiment provided below in conjunction with Fig. 1-Fig. 3 Illustrate according to storage method.
Seeing Fig. 1, the date storage method that the present embodiment provides includes: S1a, fixed at DDL ccf layer Table rule is divided in a justice point storehouse.
S2a, divide according to a point storehouse for definition at DDL ccf layer that table is regular to carry out a point storehouse by source data and divide table, Form multiple sub-block, and each sub-block is stored to corresponding MDL.
The sub-block of self is synchronized to the multiple from data base of correspondence by S3a, each MDL;
Wherein, all of MDL as writing data base, all of from data base as read from database.
Concrete, when there being mass data to need storage, application program is at DDL (Distributed Data Layer, distributed data layer) point storehouses of ccf layer definition data divides table rule, concrete mainly defines The data division rule of source data and each the sub-block correspondence formed after source data being divided are deposited The type of database of storage, such as, which sub-block is stored in oracle database, which subnumber According to block depositor in Mysql data base.
At DDL ccf layer, according to the division rule of definition, source data is divided into multiple sub-block, And multiple sub-blocks are deposited by the type of database of each the sub-block correspondence storage according to definition Store up to the MDL of corresponding types data base.Wherein, can be found in the Web of Fig. 2, J2EE application layer Source data is sent to service layer by layer, and source data is sent to again the distributed number of DDL ccf layer by service layer According to layer (Distributed Data Layer), a point storehouse for distributed data layer definition source data divides table to advise Then, each sub-block after dividing table by point storehouse stores to corresponding MDL, each main number Multiple from data base to having according to storehouse, each MDL will be stored in the sub-block of self and synchronizes to deposit Store up to corresponding multiple from data base of this MDL, i.e. each data from data base and main number According to the data consistent in storehouse.See Fig. 3, J2EE layer and source data is sent to DDL ccf layer, at DDL Point storehouse of ccf layer definition source data divides table regular and that each MDL is corresponding is multiple from data The data query routing rule in storehouse, divides table rule that source data is divided into multiple sub-block according to a point storehouse, Being stored by each sub-block to corresponding MDL, all of MDL is as writing data Storehouse, all correspondences from data base as read from database, therefore, by the data syn-chronization in MDL extremely After data base, all of attribute from data base is set to read-only (read-only) attribute. Such as, now need to be stored in data base the ID card information of each provinces and cities, can be according to identity card institute Belong to provinces and cities to divide, and the ID card information of each provinces and cities after dividing stores the master to corresponding types In data base, such as, the ID card information of Beijing area is stored in the MDL of Orale data, The ID card information of District of Shanghai is stored in the MDL of Mysql data base.Each MDL Corresponding multiple from data base, by the data syn-chronization in each MDL to corresponding multiple from data base In, MDL as writing data base, multiple from data base as read from database.
It addition, data point storehouse divides table rule can dynamically adjust change, the present embodiment can use note The mode dynamic of ShardingRule changes a point storehouse and divides table rule.
Embodiment two, a kind of method for reading data.The data provided the present embodiment below in conjunction with Fig. 4 are read Access method is described.
Seeing Fig. 4, the method for reading data that the present embodiment provides includes: S1b, fixed at DDL ccf layer Multiple data query routing rules from data base that each MDL of justice is corresponding.
S2b, when receiving data read request, according in data read request request data identify, Search corresponding MDL.
S3b, according to multiple data query routing rules from data base corresponding to this MDL, from from Data base searches the data corresponding with data read request.
Concrete, large batch of data point storehouse is divided table to store to corresponding data base by above-described embodiment one After, application program can be corresponding multiple from data base's of MDL each defined in DDL framework Data query routing rule, i.e. under same MDL, inquires about each order from data base, It is specifically as follows poll inquiry, it is also possible to for each, priority, first Query priority are set from data base High from data base, then Query priority low from data base.
When receiving data read request, first identify according to the request data in data read request, Search corresponding MDL, such as, when in data read request, the ID card information in Shanghai is read in request, First search the MDL of correspondence storage Shanghai ID card information, then corresponding many according to this MDL The individual data query routing rule from data base, from corresponding the searching and number of this MDL from data base According to the data that read requests is corresponding.
Embodiment three, a kind of data-storage system.The data provided the present embodiment below in conjunction with Fig. 5 are deposited Storage system illustrates.
Seeing Fig. 5, the data-storage system that the present embodiment provides includes the first rule definition unit 51, divides Library storage unit 52, data synchronisation unit 53 and attribute arrange unit 54.
Wherein, the first rule definition unit 51, it is mainly used in dividing table to advise in a DDL ccf layer definition point storehouse Then.
Divide library storage unit 52, be mainly used at DDL ccf layer according to the first rule definition unit 51 A point storehouse for definition is divided table rule that source data is carried out a point storehouse and is divided table, forms multiple sub-block, and by each Individual sub-block stores to corresponding MDL.
Data synchronisation unit 53, is mainly used in the sub-block in each MDL is synchronized to correspondence Multiple from data base, wherein, all of MDL is as writing data base, all of from data base As read from database.
Wherein, first rule definition unit 51 specifically for:
Each sub-block after point storehouse of DDL ccf layer definition source data is divided table rule and divided The type of database of corresponding storage;
Point library storage unit 52 specifically for:
Divide table rule at DDL ccf layer according to a point storehouse for the data source of definition, source data is divided into many Individual sub-block, and each sub-block is stored to the MDL of corresponding types data base.
The data-storage system that the present embodiment provides also includes that attribute arranges unit 54, is mainly used in all The attribute from data base be set to read-only attribute.
Embodiment four, a kind of data reading system.The data provided the present embodiment below in conjunction with Fig. 6 are read The system of taking is described.
See Fig. 6, the data reading system that the present embodiment provides include Second Rule definition unit 61, the One searches unit 62 and second searches unit 63.
Wherein, Second Rule definition unit 61, it is mainly used in for defining each master at DDL ccf layer Multiple data query routing rules from data base that data base is corresponding.
First searches unit 62, is mainly used in when receiving data read request, please according to digital independent Request data mark in asking, searches corresponding MDL.
Second searches unit 63, is mainly used in the multiple data from data base corresponding according to this MDL Query routing rule, from searching the data corresponding with data read request from data base.
The system of a kind of date storage method, method for reading data and correspondence thereof that the present invention provides, to greatly The data of batch divide table rule to carry out a point storehouse according to certain point storehouse to form multiple sub-block, and by subnumber Store to corresponding MDL according to block, and each MDL is multiple from data base, often to having Data syn-chronization in one MDL is to multiple from data base, using MDL as writing data base, Using multiple from data base as read from database, when realizing digital independent, first navigate to MDL, Regular from the data searching correspondence from data base according to certain query routing, it is achieved read and write abruption Function;It addition, multiple MDLs may belong to heterogeneous database, it is only necessary at DDL ccf layer Define the data after point storehouse divides the routing rule of table can divide table by point storehouse and store the master to corresponding types In data base, it is achieved that use while between heterogeneous database.
In the description of this specification, reference term " embodiment one ", " example ", " concrete example ", Or the description of " some examples " etc. means to combine this embodiment or example describes concrete grammar, device or Person's feature is contained at least one embodiment or the example of the present invention.In this manual, to above-mentioned art The schematic representation of language is necessarily directed to identical embodiment or example.And, the concrete spy of description Levy, method, device or feature can with in one or more embodiments in office or example in an appropriate manner In conjunction with.Additionally, in the case of the most conflicting, those skilled in the art can be by this specification The feature of the different embodiments described or example and different embodiment or example is combined and combines.
The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all in the present invention Spirit and principle within, any modification, equivalent substitution and improvement etc. made, should be included in this Within bright protection domain.

Claims (10)

1. a date storage method, it is characterised in that including:
S1a, divide table regular in a DDL ccf layer definition point storehouse;
S2a, divide according to a point storehouse for definition at DDL ccf layer that table is regular to carry out a point storehouse by source data and divide table, Form multiple subdata fast, and each sub-block is stored to corresponding MDL;
The sub-block of self is synchronized to corresponding multiple of this MDL by S3a, each MDL From data base;
Wherein, all of MDL as writing data base, all of from data base as read from database.
2. a kind of date storage method as claimed in claim 1, it is characterised in that described step S1a Specifically include:
Each subdata after point storehouse of DDL ccf layer definition source data is divided table rule and divided The type of database of block correspondence storage;
Described step S2a specifically includes:
Divide table rule at DDL ccf layer according to a point storehouse for the data source of definition, source data is divided into many Individual sub-block, and each sub-block is stored to the MDL of corresponding types data base.
3. date storage method as claimed in claim 1 a kind of, it is characterised in that by all of from The attribute of data base is set to read-only read-only attribute.
4. a kind of date storage method as claimed in claim 2, it is characterised in that use and explain The mode of ShardingRule dynamically defines a point storehouse and divides table rule.
5. a method for reading data, it is characterised in that including:
S1b, define multiple data queries from data base corresponding to each MDL at DDL ccf layer Routing rule;
S2b, when receiving data read request, according in data read request request data identify, Search corresponding MDL;
S3b, according to multiple data query routing rules from data base corresponding to this MDL, from from Data base searches the data corresponding with data read request.
6. a kind of method for reading data as claimed in claim 5, it is characterised in that use and explain Multiple data from data base that the mode of ShardingRule dynamically defines each MDL corresponding are looked into Ask routing rule.
7. a data-storage system, it is characterised in that including:
First rule definition unit, for dividing table rule in a DDL ccf layer definition point storehouse;
Divide library storage unit, for dividing table rule by source data at DDL ccf layer according to a point storehouse for definition Carry out a point storehouse and divide table, form multiple sub-block, and each sub-block is stored to corresponding main number According in storehouse;
Data synchronisation unit, for being synchronized to the multiple of correspondence by the sub-block in each MDL From data base;Wherein, all of MDL as writing data base, all of from data base as reading Data base.
8. a kind of data-storage system as claimed in claim 7, it is characterised in that described first rule Then definition unit specifically for:
Each subdata after point storehouse of DDL ccf layer definition source data is divided table rule and divided The type of database of block correspondence storage;
Described point of library storage unit specifically for:
Divide table rule at DDL ccf layer according to a point storehouse for the data source of definition, source data is divided into many Individual sub-block, and each sub-block is stored to the MDL of corresponding types data base.
9. a kind of data-storage system as claimed in claim 7, it is characterised in that also include:
Attribute arranges unit, belongs to for all of attribute from data base is set to read-only read-only Property.
10. a data reading system, it is characterised in that including:
Second Rule definition unit, for defining corresponding multiple of each MDL at DDL ccf layer From the data query routing rule of data base;
First searches unit, for when receiving data read request, according in data read request Request data identifies, and searches corresponding MDL;
Second searches unit, for the multiple data query roads from data base corresponding according to this MDL By rule, from searching the data corresponding with data read request from data base.
CN201610228439.9A 2016-04-13 2016-04-13 Data storage method, data reading method and corresponding system Pending CN105912678A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610228439.9A CN105912678A (en) 2016-04-13 2016-04-13 Data storage method, data reading method and corresponding system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610228439.9A CN105912678A (en) 2016-04-13 2016-04-13 Data storage method, data reading method and corresponding system

Publications (1)

Publication Number Publication Date
CN105912678A true CN105912678A (en) 2016-08-31

Family

ID=56746929

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610228439.9A Pending CN105912678A (en) 2016-04-13 2016-04-13 Data storage method, data reading method and corresponding system

Country Status (1)

Country Link
CN (1) CN105912678A (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107590257A (en) * 2017-09-20 2018-01-16 郑州云海信息技术有限公司 A kind of data base management method and device
CN107844490A (en) * 2016-09-19 2018-03-27 华为技术有限公司 A kind of database divides storehouse method and device
CN108268517A (en) * 2016-12-30 2018-07-10 希姆通信息技术(上海)有限公司 The management method and system of label in database
CN108509450A (en) * 2017-02-27 2018-09-07 北京京东尚科信息技术有限公司 For handling the newer method and apparatus of database high concurrent
CN109062546A (en) * 2018-07-01 2018-12-21 甘肃万维信息技术有限责任公司 A kind of multi-source data interaction platform based on spring and mybatis frame
CN111737228A (en) * 2020-06-23 2020-10-02 平安医疗健康管理股份有限公司 Database and table dividing method and device
CN111782634A (en) * 2020-06-29 2020-10-16 京东数字科技控股有限公司 Data distributed storage method and device, electronic equipment and storage medium
CN112256793A (en) * 2020-09-28 2021-01-22 建信金融科技有限责任公司 Distributed TPS real-time calculation method and facility for implementing same
CN112327777A (en) * 2020-11-13 2021-02-05 上海能誉科技股份有限公司 Data acquisition system and method
CN112784136A (en) * 2019-11-05 2021-05-11 阿里巴巴集团控股有限公司 Data query method and device and electronic equipment

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102567399A (en) * 2010-12-31 2012-07-11 北京新媒传信科技有限公司 Method and device for accessing database
CN104714945A (en) * 2013-12-11 2015-06-17 世纪禾光科技发展(北京)有限公司 Commodity information system establishing method and system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102567399A (en) * 2010-12-31 2012-07-11 北京新媒传信科技有限公司 Method and device for accessing database
CN104714945A (en) * 2013-12-11 2015-06-17 世纪禾光科技发展(北京)有限公司 Commodity information system establishing method and system

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
FREESABER: "数据库水平切分的实现原理解析——分离,分表,主从,集群,负载均衡器", 《博客园》 *
GAO-XIANGLONG: "剖析淘宝TDDL(TAOBAO DISTRIBUTE DATA LAYER)", 《推酷》 *
李洪心: "《电子商务安全》", 30 April 2012 *
梁晓涛: "《网络视频》", 1 July 2013 *

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107844490A (en) * 2016-09-19 2018-03-27 华为技术有限公司 A kind of database divides storehouse method and device
CN108268517A (en) * 2016-12-30 2018-07-10 希姆通信息技术(上海)有限公司 The management method and system of label in database
CN108509450A (en) * 2017-02-27 2018-09-07 北京京东尚科信息技术有限公司 For handling the newer method and apparatus of database high concurrent
CN108509450B (en) * 2017-02-27 2022-04-12 北京京东尚科信息技术有限公司 Method and device for processing high-concurrency update of database
CN107590257A (en) * 2017-09-20 2018-01-16 郑州云海信息技术有限公司 A kind of data base management method and device
CN109062546A (en) * 2018-07-01 2018-12-21 甘肃万维信息技术有限责任公司 A kind of multi-source data interaction platform based on spring and mybatis frame
CN112784136A (en) * 2019-11-05 2021-05-11 阿里巴巴集团控股有限公司 Data query method and device and electronic equipment
CN111737228A (en) * 2020-06-23 2020-10-02 平安医疗健康管理股份有限公司 Database and table dividing method and device
CN111737228B (en) * 2020-06-23 2022-11-15 深圳平安医疗健康科技服务有限公司 Database and table dividing method and device
CN111782634A (en) * 2020-06-29 2020-10-16 京东数字科技控股有限公司 Data distributed storage method and device, electronic equipment and storage medium
CN112256793A (en) * 2020-09-28 2021-01-22 建信金融科技有限责任公司 Distributed TPS real-time calculation method and facility for implementing same
CN112327777A (en) * 2020-11-13 2021-02-05 上海能誉科技股份有限公司 Data acquisition system and method

Similar Documents

Publication Publication Date Title
CN105912678A (en) Data storage method, data reading method and corresponding system
CN102968503B (en) The data processing method of Database Systems and Database Systems
US11334544B2 (en) Method, apparatus, device and medium for storing and querying data
US20180025024A1 (en) Pluggable storage system for parallel query engines
US10614127B2 (en) Two-phase construction of data graphs from disparate inputs
EP3186728B1 (en) File system with data block sharing
US20130218837A1 (en) Cloud data synchronization with local data
US11914585B2 (en) Servicing queries of a hybrid event index
CN102752372A (en) File based database synchronization method
WO2015103407A3 (en) Geographic information system for researching, identifying and comparing locations for economic development
US10296497B2 (en) Storing a key value to a deleted row based on key range density
CN103699638A (en) Method for realizing cross-database type synchronous data based on configuration parameters
CN106682042B (en) A kind of relation data caching and querying method and device
CN104778266A (en) Multi-data source searching method and device
CN105677915A (en) Distributed service data access method based on engine
US11288287B2 (en) Methods and apparatus to partition a database
CN105843933A (en) Index building method for distributed memory columnar database
WO2020041950A1 (en) Data update method, device, and storage device employing b+ tree indexing
CN108509636A (en) It is a kind of to realize that the big data of read and write abruption manages disaster recovery method based on partition table technology
CN109716321A (en) The hash connection of the road N
CN102567399A (en) Method and device for accessing database
WO2016086695A1 (en) Multipath device naming method, device, host and communication system
US20180239776A1 (en) Pluggable storage system for parallel query engines across non-native file systems
CN110263057A (en) A kind of storage/the querying method and device of ROWID mapping table
JP2024502119A (en) Spatial 2-tuple determination method, device, computer device, and storage medium

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20160831