CN102385628A - Java data base connectivity (JDBC)-based data distributed processing method - Google Patents
Java data base connectivity (JDBC)-based data distributed processing method Download PDFInfo
- Publication number
- CN102385628A CN102385628A CN2011103594351A CN201110359435A CN102385628A CN 102385628 A CN102385628 A CN 102385628A CN 2011103594351 A CN2011103594351 A CN 2011103594351A CN 201110359435 A CN201110359435 A CN 201110359435A CN 102385628 A CN102385628 A CN 102385628A
- Authority
- CN
- China
- Prior art keywords
- data
- jdbc
- sql
- virtual
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a Java data base connectivity (JDBC)-based data distributed processing method, which comprises the following steps of: 1) setting a virtual database JDBC driver packet file in a client or a Web application program, establishing a virtual database connection, and sending a structured query language (SQL) calling request to a virtual database according to the JDBC specifications; 2) setting parameters in the virtual database, establishing a connection between a JDBC driver layer and each entity database, and caching and/or storing the data in a plurality of databases of a distributed system; and 3) receiving the data SQL calling request by a data distributed processor of the virtual database, and performing statement analysis and distribution, so that distributed processing of the data is realized. According to the method, the data processing capacity of the original program can be improved only by adding the virtual database driver packet into a class loading path of the original application program and simply configuring the virtual database; and the method is strong in expansibility, the data storage capacity can be improved in a vertical expansion mode, and the concurrent processing capacity can also be improved in a transverse expansion mode.
Description
Technical field
The present invention relates to data processing, particularly a kind of DATA DISTRIBUTION formula disposal route based on JDBC belongs to the data storage and retrieval field.
Background technology
When the data scale of application system processing constantly increased, the storage and the processing requirements that satisfy these mass datas generally had dual mode, and a kind of is to adopt higher minicomputer, the large scale computer equipment of performance, and another kind is the cluster that adopts common machines to form.Because cluster has outstanding cost performance, replaced large scale computer at present to a great extent, the application on Web server and the application server is exactly well to embody.JDBC (Java Data Base Connectivity, the java database connects) is a kind of Java API that is used to carry out SQL statement, can unified visit be provided for multiple relational database, and it is made up of one group of class and interface of writing with Java language.JDBC provides a kind of benchmark, can make up more senior instrument and interface in view of the above, connects with database, the statement and the result of transmit operation database.JVM is the abbreviation of (Java Virtual Machine, Java Virtual Machine), and JVM is a kind of standard that is used for computing equipment, and it is a computing machine of fabricating out, realizes through the various computer functions of analogue simulation on the actual calculation machine.Java Virtual Machine comprises a cover byte code instruction collection, one group of register, a stack, a garbage reclamation heap and a storage means territory.JVM has shielded the information relevant with the specific operating system platform, and the object code (bytecode) that java applet only need be created on move on the Java Virtual Machine just can not add on kinds of platform and revises the ground operation.
Data storage with handle, though cluster provides high-performance and strong fault tolerance ability, the instrument of this respect is with regard to much less, and mainly is the solution towards large enterprises.Business solution such as Oracle Real Application Clusters have taken to use and have shared storage system; Use network in the DB2 Integrated Cluster environment of IBM and shared storage; And aspect the data-base cluster in camp of increasing income, the MySQL reproduction technology is used principal and subordinate's mechanism and is realized.These technological obvious defects are in order to realize the characteristic of cluster; Necessary extending database engine, so application program will be used some extra API, API (Application Programming Interface; API) is some predefined functions; Purpose provide application program and developer based on certain software or hardware visiting the ability of one group of routine, and need not to visit source code, or understand the details of internal work mechanism.And be difficult to cooperation each other between the different implementations of these different technologies.
Summary of the invention
The objective of the invention is to propose a kind of DATA DISTRIBUTION formula disposal route based on JDBC; It can convert the entity data bak of series of different to a unified virtual data base; The list structure of each entity data bak is identical; Client or web application drive through loading virtual data base JDBC, can send the SQL request to virtual data base according to the JDBC standard, realize the operations such as data retrieval, insertion, renewal and deletion to each entity data bak.
In order to solve the problems of the technologies described above, technical scheme of the present invention comprises:
A kind of DATA DISTRIBUTION formula disposal route based on JDBC, concrete steps are following:
1) in the class load path of client of developing based on the Java standard or web application, virtual data base JDBC is set and drives the APMB package position; Make said application program can load this driving bag; Connect with virtual data base, and send the SQL call request to virtual data base according to the JDBC standard;
2) virtual data base is a kind of server-side application based on the exploitation of Java standard; Use configuration management tool to carry out the parameter setting; So that the driving that provides through manufacturer in the JDBC Drive Layer connects with each entity data bak respectively, with metadata cache and/or be stored in a plurality of databases of distributed system;
3) the DATA DISTRIBUTION formula processor of said virtual data base reception data SQL call request, the line statement analysis of going forward side by side, distribution is carried out then, the data in inquiry and/or the renewal entity data bak, the distributed treatment of realization data.
Data volume as using is big, has surpassed the single node processing power, then disperses data to be stored in the multinode cluster; Data volume as using is little, then the data multiple duplication in a plurality of node clusters.
Said DATA DISTRIBUTION formula processor comprises: modules such as SQL calling interface, SQL statement analyzer, SQL statement actuator, scheduling distributor, metadata cache, connection pool, data access plugin manager, Oracle data access, MySQL data access, user security, daily record maintenance, configuration management.
Said JDBC Drive Layer comprises: the JDBC standard that meets that Oracle driving, MYsql Drive Layer and other manufacturers provide drives.
Said database comprises oracle database, MySQL database and other manufacturer data storehouses.
Said virtual data base JDBC drives bag: be a driver that meets the JDBC standard; Can in JVM, move; Make the user carry out communication with the virtual data base management system, SQL statement of user is sent in the virtual data base, and its result will be sent back to the user.
The present invention is mainly used under specific environment, and is big like the data volume of using, and surpassed the single node processing power, can adopt this method to press the extends perpendicular mode, disperses data to be stored in the multinode cluster, satisfies the demands through distributed treatment; Data volume as using is little, but user's concurrent request is many, can adopt this method to press mode extending transversely, in a plurality of node clusters, improves concurrent responding ability to the data multiple duplication through load balancing is carried out in request.
Beneficial effect of the present invention:
1. application is simple, only need in the class load path of original application program, add virtual data base driving bag, and virtual data base is carried out easy configuration, can improve the data-handling capacity of original program, adapts to the data processing of different pieces of information amount.
2. extendability is strong, can improve data storage capacities through the extends perpendicular mode, also can improve the concurrent processing ability through mode extending transversely.
Description of drawings
Fig. 1 is that DATA DISTRIBUTION formula disposal route of the present invention is implemented synoptic diagram;
Fig. 2 is DATA DISTRIBUTION formula process flow figure of the present invention;
Embodiment
Below in conjunction with accompanying drawing, preferred embodiment of the present invention is explained further details.
As shown in Figure 1; In the class load path of client of developing based on the Java standard or web application, virtual data base JDBC is set and drives the APMB package position; Make said application program can load this driving bag; Connect with virtual data base, virtual data base is given in the operation calls requests such as SQL retrieval, insertion, renewal and deletion of sending according to the JDBC standard; Virtual data base is a kind of server-side application based on the exploitation of Java standard; Use configuration management tool to carry out the parameter setting; So that the driving that provides through manufacturer in the JDBC Drive Layer connects with each entity data bak respectively, with metadata cache and/or be stored in a plurality of databases of distributed system; After virtual data base receives the SQL call request of application program; Carry out the statement analysis through DATA DISTRIBUTION formula processor; Distribution is carried out then; To realize the distributed treatment of data, DATA DISTRIBUTION formula processor comprises modules such as SQL calling interface, SQL statement analyzer, SQL statement actuator, scheduling distributor, metadata cache, connection pool, data access plugin manager, Oracle data access, MySQL data access, user security, daily record maintenance, configuration management.
Entity data bak can be oracle database, MySQL database, SQL Server database, also can be other manufacturer data storehouses, as long as the corresponding data access plug-in unit of exploitation is deployed in the DATA DISTRIBUTION formula processor then.
As shown in Figure 2, the present invention proposes a kind of DATA DISTRIBUTION formula disposal route based on JDBC, is example with the web application, and its treatment scheme is following:
(1) in web application, according to operation calls requests such as JDBC standard encapsulation SQL retrieval, insertion, renewal and deletions;
(2) drive bag through virtual data base JDBC, web application and virtual data base connect, and through this connection SQL call request packaged in the step 1) are sent to virtual data base.
(3) virtual data base set up through connection pool module management and web application all be connected, and through user security and daily record maintenance module the SQL of web application is operated and to authorize audit.
(4),, convert the SQL call statement that virtual data base is supported to through the SQL calling interface for the SQL call request that receives.
(5) the SQL statement analyzer is according to the parameter configuration of virtual data base; Judge which table is this SQL call request relate to; Need which data access plug-in unit to participate in; And the processing mode of result set, and be assembled into the fill order that each data access plug-in unit can be discerned, send to the SQL statement actuator to fill order then.According to the concrete condition of service application, the parameter configuration of virtual data base is different: for the table that has mass data, parameter configuration can be disperseed data to be stored in the database of a plurality of reality; For the table of small data quantity, can carry out redundant storage, and then increase the reliability and the concurrent performance of system through multiple duplication.The SQL statement analyzer can be provided with the result set processing mode according to the different service applicable cases, for disperseing memory module, adopts many storehouses result set merger processing mode; For the multiple duplication pattern, adopt load balancing, select the data optimal storehouse to carry out and return results.
(6) after the SQL statement actuator receives fill order; At first judge whether to use buffer memory and data cached effective, then directly call data cache module in this way the result is returned to web application, carry out otherwise give the scheduling distributor with fill order; Result set processing mode according to the fill order setting; Result set to the scheduling distributor returns is handled, and then the result is returned to web application, and calls data cache module and upgrade buffer memory.
(7) fill order of scheduling distributor to receiving; From the data access plugin manager, obtain corresponding data access plug-in unit according to the order classification; Call data access plug-in unit the corresponding interface fill order then, and the result is returned to the SQL statement actuator handle.If relate to a plurality of entity data baks, each execution that all can be scheduled.
(8) data access plug-in unit package implementation access of database interface, the mode and the entity data bak that generally drive through JDBC connect, and the SQL call request is sent to the solid data base management system carry out and collect return results.The database of different vendor has data access plug-in unit separately; Like Oracle data access plug-in unit, MySQL data access plug-in unit, SQL Server data access plug-in unit etc., all data access plug-in units all need be registered in the data access plugin manager through the virtual data base configuration management tool.
(9) DATA DISTRIBUTION formula user security that processor provides, daily record maintenance, Configuration Manager belong to basic module, for other modules provide configuration, authorize auditing service, to increase security, the reliability of system.
Claims (7)
1. DATA DISTRIBUTION formula disposal route based on JDBC, its step comprises:
1) virtual data base JDBC is set and drives APMB package in client or web application, set up virtual data base and connect, and send the SQL call request to virtual data base according to the JDBC standard;
2) virtual data base carries out the parameter setting, connects through JDBC Drive Layer and each entity data bak, with metadata cache and/or be stored in a plurality of databases of distributed system;
3) the DATA DISTRIBUTION formula processor of said virtual data base receives data SQL call request, the line statement analysis of going forward side by side, and distribution is carried out then, realizes the distributed treatment of data.
2. disposal route as claimed in claim 1 is characterized in that, said DATA DISTRIBUTION formula processor comprises: parameter configuration is configured the unified virtual data base of formation to the multinode entity data bak; The SQL statement analyzer carries out the statement analysis to the SQL call request that receives; The SQL statement actuator is carried out sql command; The scheduling distributor realizes that the SQL of multinode entity data bak calls the distribution execution; Data buffer, the Data Update of buffer memory JDBC.
3. disposal route as claimed in claim 1 is characterized in that, and is big like the data volume of using in the said virtual data base, surpassed the single node processing power, then disperses data to be stored in the actual database in the multinode cluster; Data volume as using is little, then the data multiple duplication in actual database in a plurality of node clusters.
4. disposal route as claimed in claim 1 is characterized in that, said JDBC Drive Layer comprises: Oracle driving, MySQL Drive Layer, SQL Server Drive Layer and other manufacturers meet the driving of JDBC standard.
5. disposal route as claimed in claim 1 is characterized in that, said entity data bak comprises oracle database, MySQL database, SQL Server database.
6. disposal route as claimed in claim 1 is characterized in that, said DATA DISTRIBUTION formula processor also comprises the data access plugin manager.
7. disposal route as claimed in claim 1 is characterized in that, said DATA DISTRIBUTION formula processor also comprises user security, daily record maintenance and Configuration Manager.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110359435.1A CN102385628B (en) | 2011-11-14 | 2011-11-14 | Java data base connectivity (JDBC)-based data distributed processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110359435.1A CN102385628B (en) | 2011-11-14 | 2011-11-14 | Java data base connectivity (JDBC)-based data distributed processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102385628A true CN102385628A (en) | 2012-03-21 |
| CN102385628B CN102385628B (en) | 2015-05-13 |
Family
ID=45825044
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110359435.1A Active CN102385628B (en) | 2011-11-14 | 2011-11-14 | Java data base connectivity (JDBC)-based data distributed processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102385628B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103336782A (en) * | 2013-05-30 | 2013-10-02 | 莱诺斯科技(北京)有限公司 | Relational distributed database system |
| CN103678354A (en) * | 2012-09-11 | 2014-03-26 | 中国移动通信集团公司 | Local relation type database node scheduling method and device based on cloud computing platform |
| CN103885995A (en) * | 2012-12-21 | 2014-06-25 | 中国移动通信集团河北有限公司 | List-based database monitoring method and list-based database monitoring device |
| CN104363165A (en) * | 2014-11-14 | 2015-02-18 | 华东电网有限公司 | Information interactive system under internal and external network isolation environment and data integrating method |
| CN106066890A (en) * | 2016-06-16 | 2016-11-02 | 上海天玑科技股份有限公司 | A kind of distributed high-performance data storehouse integrated machine system |
| CN106874109A (en) * | 2016-12-29 | 2017-06-20 | 朗新科技股份有限公司 | A kind of distributed job distribution processing method and system |
| CN107402941A (en) * | 2016-07-22 | 2017-11-28 | 延边众生云计算科技有限公司 | Conventional data Fabric Interface and its implementation |
| CN107463511A (en) * | 2017-01-23 | 2017-12-12 | 北京思特奇信息技术股份有限公司 | A kind of data internationalization implementation method and device based on multi-level buffer |
| CN108762112A (en) * | 2018-06-12 | 2018-11-06 | 哈尔滨理工大学 | A kind of industrial robot emulation and real-time control system based on virtual reality |
| CN109828983A (en) * | 2018-12-15 | 2019-05-31 | 平安科技（深圳）有限公司 | PG data base processing method, device, electronic equipment and storage medium |
| CN110147508A (en) * | 2017-10-26 | 2019-08-20 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of system current limliting |
| CN111367983A (en) * | 2020-03-10 | 2020-07-03 | 中国联合网络通信集团有限公司 | Database access method, system, device and storage medium |
| CN112688976A (en) * | 2019-10-17 | 2021-04-20 | 广州迈安信息科技有限公司 | Data processing transmission service system adopting JDBC/HTTP standard |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101093502A (en) * | 2007-06-19 | 2007-12-26 | 深圳市迈科龙电子有限公司 | Security structure of database, and method of use |
| CN101187937A (en) * | 2007-10-30 | 2008-05-28 | 北京航空航天大学 | Mode multiplexing isomerous database access and integration method under gridding environment |
| CN101833620A (en) * | 2010-04-28 | 2010-09-15 | 国网电力科学研究院 | Custom security JDBC driver-based database protective method |
| CN101840352A (en) * | 2010-04-29 | 2010-09-22 | 中兴通讯股份有限公司 | Method and device for monitoring database connection pool |
| CN101853274A (en) * | 2010-05-10 | 2010-10-06 | 浪潮电子信息产业股份有限公司 | Method for realizing interconnection of heterogeneous databases |
| US20110093435A1 (en) * | 2009-10-21 | 2011-04-21 | Delphix Corp. | Virtual Database System |
| CN102065097A (en) * | 2010-12-27 | 2011-05-18 | 北京像素软件科技股份有限公司 | Synchronous operation method of clients and servers |
-
2011
- 2011-11-14 CN CN201110359435.1A patent/CN102385628B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101093502A (en) * | 2007-06-19 | 2007-12-26 | 深圳市迈科龙电子有限公司 | Security structure of database, and method of use |
| CN101187937A (en) * | 2007-10-30 | 2008-05-28 | 北京航空航天大学 | Mode multiplexing isomerous database access and integration method under gridding environment |
| US20110093435A1 (en) * | 2009-10-21 | 2011-04-21 | Delphix Corp. | Virtual Database System |
| CN101833620A (en) * | 2010-04-28 | 2010-09-15 | 国网电力科学研究院 | Custom security JDBC driver-based database protective method |
| CN101840352A (en) * | 2010-04-29 | 2010-09-22 | 中兴通讯股份有限公司 | Method and device for monitoring database connection pool |
| CN101853274A (en) * | 2010-05-10 | 2010-10-06 | 浪潮电子信息产业股份有限公司 | Method for realizing interconnection of heterogeneous databases |
| CN102065097A (en) * | 2010-12-27 | 2011-05-18 | 北京像素软件科技股份有限公司 | Synchronous operation method of clients and servers |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678354A (en) * | 2012-09-11 | 2014-03-26 | 中国移动通信集团公司 | Local relation type database node scheduling method and device based on cloud computing platform |
| CN103885995A (en) * | 2012-12-21 | 2014-06-25 | 中国移动通信集团河北有限公司 | List-based database monitoring method and list-based database monitoring device |
| CN103885995B (en) * | 2012-12-21 | 2017-05-03 | 中国移动通信集团河北有限公司 | List-based database monitoring method and list-based database monitoring device |
| CN103336782A (en) * | 2013-05-30 | 2013-10-02 | 莱诺斯科技(北京)有限公司 | Relational distributed database system |
| CN103336782B (en) * | 2013-05-30 | 2016-09-21 | 莱诺斯科技（北京）股份有限公司 | A kind of relationship type distributed data base system |
| CN104363165A (en) * | 2014-11-14 | 2015-02-18 | 华东电网有限公司 | Information interactive system under internal and external network isolation environment and data integrating method |
| CN106066890A (en) * | 2016-06-16 | 2016-11-02 | 上海天玑科技股份有限公司 | A kind of distributed high-performance data storehouse integrated machine system |
| CN106066890B (en) * | 2016-06-16 | 2020-02-18 | 上海天玑科技股份有限公司 | Distributed high-performance database all-in-one machine system |
| CN107402941A (en) * | 2016-07-22 | 2017-11-28 | 延边众生云计算科技有限公司 | Conventional data Fabric Interface and its implementation |
| CN106874109A (en) * | 2016-12-29 | 2017-06-20 | 朗新科技股份有限公司 | A kind of distributed job distribution processing method and system |
| CN107463511A (en) * | 2017-01-23 | 2017-12-12 | 北京思特奇信息技术股份有限公司 | A kind of data internationalization implementation method and device based on multi-level buffer |
| CN107463511B (en) * | 2017-01-23 | 2020-06-26 | 北京思特奇信息技术股份有限公司 | Data internationalization realization method and device based on multi-level cache |
| CN110147508A (en) * | 2017-10-26 | 2019-08-20 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of system current limliting |
| CN108762112A (en) * | 2018-06-12 | 2018-11-06 | 哈尔滨理工大学 | A kind of industrial robot emulation and real-time control system based on virtual reality |
| CN109828983A (en) * | 2018-12-15 | 2019-05-31 | 平安科技（深圳）有限公司 | PG data base processing method, device, electronic equipment and storage medium |
| CN109828983B (en) * | 2018-12-15 | 2024-05-07 | 平安科技（深圳）有限公司 | PG database processing method, device, electronic equipment and storage medium |
| CN112688976A (en) * | 2019-10-17 | 2021-04-20 | 广州迈安信息科技有限公司 | Data processing transmission service system adopting JDBC/HTTP standard |
| CN111367983A (en) * | 2020-03-10 | 2020-07-03 | 中国联合网络通信集团有限公司 | Database access method, system, device and storage medium |
| CN111367983B (en) * | 2020-03-10 | 2023-08-15 | 中国联合网络通信集团有限公司 | Database access method, system, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102385628B (en) | 2015-05-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102385628B (en) | Java data base connectivity (JDBC)-based data distributed processing method | |
| US20220171759A1 (en) | Detecting schema incompatibilities for generating views at target data stores | |
| CN100565510C (en) | Data access layer class generator | |
| CN103377290B (en) | Delete the record in multistage storage architecture | |
| CN103930875B (en) | Software virtual machine for acceleration of transactional data processing | |
| CN103377289B (en) | Unified table query processing | |
| CN103425722B (en) | The method and system of atomic data movement | |
| US10210221B2 (en) | System and method for distributed database query engines | |
| CN102591946B (en) | It is divided using index and coordinates to carry out data deduplication | |
| US20220253433A1 (en) | Maintaining data stream history for generating materialized views | |
| CN103377148B (en) | The method and system that executable portion merges in a kind of unified dial framework structure | |
| CN101853287B (en) | Data compression quick retrieval file system and method thereof | |
| CN103593422B (en) | Virtual access management method of heterogeneous database | |
| CN112534396A (en) | Diary watch in database system | |
| CN104484472B (en) | A kind of data-base cluster and implementation method of a variety of heterogeneous data sources of mixing | |
| CN103399942B (en) | A kind of data engine system supporting the many tenants of SaaS and method of work thereof | |
| US11341317B2 (en) | Supporting piecewise update of JSON document efficiently | |
| CN105243155A (en) | Big data extracting and exchanging system | |
| CN103562910A (en) | Managing data queries | |
| CN102096684A (en) | Grid real-time data integrating and sharing platform | |
| JP2005196602A (en) | System configuration changing method in unshared type database management system | |
| CN105164674A (en) | Queries involving multiple databases and execution engines | |
| US9930113B2 (en) | Data retrieval via a telecommunication network | |
| US11899659B2 (en) | Dynamically adjusting performance of materialized view maintenance | |
| US11544286B2 (en) | Replicating materialized views across heterogeneous target systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |