CN107480251A - A kind of system for managing data access - Google Patents
A kind of system for managing data access Download PDFInfo
- Publication number
- CN107480251A CN107480251A CN201710691745.0A CN201710691745A CN107480251A CN 107480251 A CN107480251 A CN 107480251A CN 201710691745 A CN201710691745 A CN 201710691745A CN 107480251 A CN107480251 A CN 107480251A
- Authority
- CN
- China
- Prior art keywords
- data
- client
- service end
- access
- affairs
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of system for managing data access, client passes through load balancing module access service end, behind data access service end, the availability of the service end accessed is ensured by High Availabitity support module, request of the flow-control module to client counts, and the request to not being inconsistent interflow control configuration is refused, RPC communication module is used to realize client and service end data transmission, and affairs support module is used for the type for judging current data operation；And actual data access operation is completed by data manipulation execution module, carry out record statistics to accessing vestige eventually through data abatement modules.By the above-mentioned means, the achievable data of the present invention decouple with application, so that the deployment change of data does not interfere with application, and the access process of management and control data.
Description
Technical field
The present invention relates to database technical field, more particularly, to a kind of system for managing data access.
Background technology
In large business system, over time with the development of business, the table in database can be more and more, hot spot service
Data volume in table also can be increasing, correspondingly, data manipulation, as additions and deletions change the expense looked into also can be increasing, and be used for
The disposal ability (disk/CPU/ internal memories/bandwidth) for carrying the host server of traditional database is limited, in business development
Bottleneck is easily reached in journey.With the expansion of application cluster, horizontal split of database will turn into normality with vertical fractionation.
Vertically (longitudinal direction) splits and refers to split by functional module, for example is divided into order store, commodity storehouse and user library etc., this
Table structure between the multiple databases of mode is different.
It is that the data of same table are carried out into piecemeal to be saved in different databases that horizontal (transverse direction), which is split, these data
Table structure in storehouse is identical.
In general operation system is moved using the implementation by Data Access Integration in application service, database
Moving change can bring the association of service code to change, and the cost of the change of each database will become expensive and give business system
Regiment commander carrys out great risk.
Original implementation by Data Access Integration in application service has following features and problem：
1st, Data expansion is carried out based on separate unit database, needed if certain mass data table needs the horizontal more storehouses of progress to split
Adjust mass data and access logic.
2nd, traditional data access process is distributed in the code of application service, and complexity is very high, it is difficult to visual
Model is managed collectively and administered.
3rd, it is heavy in hand for the change and adjustment complexity of type of database, it is more difficult to use the geo-database integration of polymorphic type
With the advantageous characteristic using all types of databases in itself.
The content of the invention
The technical problems to be solved by the invention are：The problem of existing for prior art, there is provided one kind management data are visited
The system asked, data can be achieved and decoupled with application, so that the deployment change of data does not interfere with application, and the visit of management and control data
Ask process.
In order to solve the above-mentioned technical problem, the technical solution adopted by the present invention is：There is provided a kind of data access that manages is
System, including client and service end, wherein also including：Load balancing module, High Availabitity support module, RPC communication module, flow
Control module, affairs support module, data abatement modules and data manipulation execution module；Wherein
Client passes through load balancing module access service end, behind data access service end, passes through High Availabitity support module
The availability of the service end accessed is ensured, request of the flow-control module to client counts, and matches somebody with somebody to not being inconsistent interflow control
The request put is refused, and RPC communication module is used to realize client and service end data transmission, and affairs support module is used to sentence
The type of settled preceding data manipulation, and actual data access operation is completed by data manipulation execution module, with mathematical logic-number
Implementation procedure is controlled and recorded according to the mode of table-data source, and is recorded eventually through data abatement modules to accessing vestige
Statistics.
Wherein, the load balancing module is integrated in client, to obtain currently available service using ZooKeeper
Address list, load balancing is realized in client.
Wherein, the High Availabitity support module is by the real-time monitoring service address lists of ZooKeeper, and rejects data clothes
The address of service of business node failure；And enable asynchronous thread and heartbeat monitor is carried out to the back end in service address list, with
Ensure that exception and the network of application client end node and data service node occur for the network of ZooKeeper and data serving node
Data serving node is not rejected when without exception.
Wherein, the RPC communication module uses THRIFT frameworks, and secondary encapsulation is carried out to communication process；It includes：
IDL definition units, for defining unified idl file to client and service end, to enter line interface identification, and it is raw
Into corresponding serializing code and unserializing code；
Protocol encapsulation unit, for being embedded in encapsulation mode in serializing, and data parsing is carried out in unserializing；
Data format unit, for dividing data type；
Asynchronous communication unit, for the NIO asynchronous communications of client and service end, line is handled using epoll technologies and I/O
Cheng Chi, lift the concurrency and handling capacity of data, services.
Wherein, the flow-control module provides distributed traffic control, supports multi-stage data granularity, and control accesses frequency
It is secondary；It includes：
Flowmeter counter unit, for collecting the access information of each data serving node, and it is sent to
ZooKeeper, and limit being accessed beyond the unit interval, then think the abnormal information of client feedback flow control；
Control logic injection unit, the visitation frequency of sentence, tables of data and data source is accessed for controlling, and realizes dynamic
Injection control.
Wherein, the affairs support module includes：
Single-point transaction units, for controlling conversation affair on single application node by data, services and being saved by the application
The application, submission and rollback that point initiates affairs are called；
Distributed transaction unit, for cooperation with service end affairs and client transaction.
Wherein, the service end provides affairs application interface, affairs submit interface and transaction rollback interface to be adjusted for client
With；Wherein
After client call affairs application interface, by transaction context object local cache in service end, and will be upper
Hereafter mark returns to client, and the transaction context object cached is automatically positioned in follow-up client call every time；
After client is initiated to ask to service end, service end is led to by the database connection list of context object association
Crossing affairs submits interface that affairs are submitted into database one by one, and discharges transaction context；
After client is initiated to ask to service end, service end is led to by the database connection list of context object association
Transaction rollback interface is crossed by affairs one by one from rollback database, and discharges transaction context；
Service end carries out overtime management to affairs, overtime session is cleared up automatically, wherein automatic cleaning operation is by asynchronous thread
Automatic triggering；
Client call affairs application interface, when being passed to Context identifier when data, services are dispatched, client end AP I roots
Forward requests to data serving node during affairs application automatically according to Context identifier, after the completion of data manipulation calling, adjust
Interface or transaction rollback interface are submitted with affairs；
Client call affairs application interface, when the not incoming Context identifier when data, services are dispatched, then it represents that make
With stateless data, services, service end does not cache transaction context information after calling.
Wherein, the data abatement modules include：Data access monitoring unit, data access registering unit and data target
Output unit；Wherein
After service end enables asynchronous thread, data access monitoring unit is automatically by the status information and effect of data serving node
Rate statistical information is synchronized to ZooKeeper；
The IP address of client itself and process number are registered to ZooKeeper by client automatically；
The information of the data access monitoring unit inquiry service end and client, service end or client to access exception
End carries out shielding isolation；
The data access registering unit carries out unified static note to all data access logics, data object and data source
Volume, by the docking with external management platform, data object, executive plan and sensitive data are accessed be managed collectively and examined
Meter；
The access index output unit generates automatically or on demand indication information, and visit by data according to a variety of dimensions
Ask indication information, to operation when data manipulation perform Management strategy；Wherein indication information includes visit capacity, performance, abnormal amount
And access source；The Management strategy includes access safety audit, data storage scheme and access strategy and the Fault Isolation Strategy adjustment.
Wherein, the data manipulation execution module includes standby unit and data routing units；Wherein
Standby unit is used for service end and carries out automated availability detection to back-end data source, redundant be present detecting
After data source, then in current data source exception, the redundant data source is enabled automatically, realize that automatic fault during operation is cut
Change；
Data routing units are used for service end parsing data manipulation, and according to input condition and the context mark of current accessed
Know the data source that data access request is routed to other configurations automatically, realize and a point storehouse for data source is operated.
The beneficial effects of the present invention are：Prior art is different from, the present invention is managed collectively by Distributed database service
The mode of data access process, data and application are truly realized decoupling, to possess data Governance Ability, taken in distributed data
Classification modeling is carried out using " mathematical logic-tables of data-data source " to all data accesses in business, reaches the access of data distribution formula
With the target for concentrating some management, after data and application decoupling, the deployment change of data does not interfere with application, and management and control number
According to access process.
Brief description of the drawings
Fig. 1 is the functional module framework schematic diagram of present invention management data access system；
Fig. 2 is the configuration diagram of High Availabitity data source model of the present invention；
Fig. 3 is the process schematic that the present invention obtains the connection of True Data storehouse on data serving node.
Embodiment
To describe the technology contents of the present invention, the objects and the effects in detail, below in conjunction with embodiment and coordinate attached
Figure is explained.
Fig. 1~Fig. 3 is refer to, Fig. 1 is the functional frame composition of Distributed database service of the present invention, and the present invention provides a kind of pipe
The system 100 of data access is managed, includes load balancing module 110, High Availabitity support module 120, RPC communication module 130, flow
Control module 140, affairs support module 150, data abatement modules 160 and data manipulation execution module 170.
The client of the present invention is integrated with load balancing module 110, and data access request is to be at this with such path
Data access operation is carried out in system 100：Data access service end first, ensure what is accessed by High Availabitity support module 120
The availability of service end, counted in request of the flow-control module 140 to client and asked to not being inconsistent interflow control configuration
Ask and refused, client and service end data transmission are realized based on RPC communication module 130, sentenced by affairs support module 150
Settled preceding data manipulation is to add global distributed transaction or voluntarily establish single-point affairs, in data manipulation execution module 170
Actual data access operation is completed, record statistics is carried out to accessing vestige eventually through data abatement modules 160.
Above-mentioned stream control configuration is that support is per minute by the Task-size Controlling of IP address/database statement/data source/tables of data
The permission visitation frequency of (or per hour), therefore when client request number exceedes the calling amount threshold value that the unit interval allows, then
The request is represented not to be inconsistent interflow control configuration；RPC (Remote Procedure Call Protocol) refers to remote process
Invocation protocol, it is that one kind asks service by network from remote computer program, without understanding underlying network technology
Agreement.
It is illustrated below for the above-mentioned module and implementation strategy of the present invention, the distribution of data is built by correlation module
Formula accesses, High Availabitity, high-performance, across language, the critical capability that can be monitored.
First, load balancing module 110：
The data, services of Distributed Data Visits layer are provided with trunking mode, are integrated with data, services client customizable
The load balancing module 110 of extension.
ZooKeeper is introduced, currently available service address list is obtained using ZooKeeper, is realized in client end AP I
Load balancing.Load balancing supports poll, random, local first and the preferential scheduling algorithm of degree of load.Various load balancing plans
It is slightly each advantageous, for example preferentially the machine request is routed on the data serving node of the machine using local first dispatching algorithm,
Network overhead can be minimized, by the rank of communication efficiency lifting to 10us.
ZooKeeper is one distributed, the distributed application program coordination service of open source code, be Hadoop and
Hbase significant components.It is the software that Consistency service is provided for Distributed Application, there is provided function include：Configuration
Maintenance, domain name service, distributed synchronization, group service etc..ZooKeeper target is exactly the error-prone crucial clothes of packaged complexity
Business, user is supplied to by the system of interface and performance efficiency easy to use, function-stable.
2nd, High Availabitity support module 120：
The High Availabitity protection of system is realized by client end AP I, specifically, address of service is arranged using ZooKeeper
Table is monitored in real time, automatically rejects the invalid service address of client when some data serving node fails.Meanwhile open
Heartbeat monitor is carried out to the back end main frame in service list with asynchronous thread, to ensure ZooKeeper and data, services section
When the network of point occurs abnormal, but when the network of application client end node and data service node is without exception, not to data, services
Node is rejected.This is that High Availabitity support module 120 ensures available two links in address of service, first, from ZooKeeper
Obtain ZooKeeper and think available service list, second, the result detection based on first link, only in ZooKeeper
The normal service end of heartbeat that normal registration and client are sent is appeared in the service address list of client.
3rd, RPC communication module 130：
Using the THRIFT frameworks of enhancing, secondary encapsulation is carried out to communication process, realizes efficiently across language RPC scheduling frame
Frame.It includes IDL definition units 131, protocol encapsulation unit 132, data format unit 133 and asynchronous communication unit 134, specifically
Ground：
In IDL definition units 131, client and server end is by defining unified IDL (interface
Definition language) file identifies entering line interface and serializing and unserializing code corresponding to generating.
Protocol encapsulation unit 132 supports embedded a variety of pack modes in serialization process, such as encrypts, compresses, corresponding
Receiving terminal receive the data after packaging then the corresponding unserializing process of Automatic dispatching carry out initial data parsing.
Data format unit 133 supports numerous types of data, the data that process operation data needs is divided into several big basic
Type, support numerical value, text, structure, simple receptacle etc..
Asynchronous communication unit 134 supports the NIO asynchronous communications of client and service end, while supports epoll to realize, greatly
Lift the concurrency and handling capacity of data, services.
4th, flow-control module 140：
By the module, system can carry out distributed flow control, support multi-stage data granularity, including IP address/number
According to the granularity of storehouse sentence/data source/tables of data, to facilitate control visitation frequency.It further comprises following unit：
Flowmeter counter unit 141, there is provided distributed counter, the access information of each data serving node is received in real time
Collect ZooKeeper, if limitation is accessed beyond the unit interval of configuration, to the abnormal information of client return flow control.
Control logic injection unit 142, support to be controlled the visitation frequency of access sentence, tables of data, data source, can
Realize Dynamic injection control.
5th, affairs support module 150 includes single-point transaction units 151 and distributed transaction unit 152, specifically：
Single-point transaction units 151 are supported to be controlled the conversation affair on single application node by data, services, by
Application node actively initiates the application of affairs and the submission of affairs or rollback is called.
Distributed transaction typically ensures high uniformity by XA modes in the prior art, but this often brings and higher is
System load.Under highly reliable network communications environment, using simplified distributed transaction unit 152, it is not sacrificing the present invention
In the case of transaction consistency, there is provided server end thing is completed in the transaction management ability of more efficient and low system consumption, collaboration
Business and client transaction.
In service end affairs, service end provides stateful data, services, there is provided 3 class transaction interfaces are adjusted for client
With：Affairs application interface, affairs submit interface and transaction rollback interface.
It is after each affairs application calling that transaction context object is locally slow after client call affairs application interface
On presence service end, and client context mark is returned to, be automatically positioned and delayed in follow-up client call every time
The transaction context object identity deposited.
After client is initiated to ask to service end, the database connection list that service end is associated by context object passes through
Affairs submit interface that affairs are submitted into database one by one, and discharge transaction context；Specifically, client is in a trade fair
After words completion or processing terminate, service end calls affairs to submit interface that affairs are submitted into database one by one.
After client is initiated to ask to service end, the database connection list that service end is associated by context object passes through
Affairs one by one from rollback database, and are discharged transaction context by transaction rollback interface；Specifically, client is in a trade fair
After words failure or processing exception, service end calls transaction rollback interface by affairs one by one from rollback database；Either in client
End is that service end triggers calling automatically after handling transaction session in exceeding schedule time.
Service end provide not only above-mentioned 3 class interface, also realizes the time-out management of affairs, clears up overtime session automatically, its
Middle cleaning is triggered automatically by asynchronous thread.
In client transaction, when client wants keep global transaction, ensure first to call affairs application interface, and after
Continuous each data, services scheduling all takes context id information, and based on context identification information automatically please by client end AP I
Seek data serving node when being forwarded to affairs application.After the completion of data manipulation calling, affairs are called to submit interface or affairs
Rollback interface；
Client incoming Context identifier when data, services are dispatched then is given tacit consent to using stateless data, services, now
Service end does not cache transaction context information after each call.
6th, data abatement modules 160：
The system provides data abatement modules 160, passes through the Unified Model management of data access, the whole process to data access
Monitoring and index outgoing management, make system possess data inactivity and Dynamic control ability, data assets and access feelings to system
Condition is accomplished clear visual；It further comprises data access monitoring unit 161, data access registering unit 162 and accesses index
Output unit 163, wherein
After the service end of data, services enables asynchronous thread, data access monitoring unit 161 is automatically by the shape of this node
The data such as state information and Efficiency Statistics information (access times, average behavior and abnormal amount etc.) is synchronized to ZooKeeper.
Data, services client automatically arrives the Details Registrations such as the IP address of this node request end and process number
ZooKeeper。
The information of data services end and client can be inquired about by keeper, to the service end or client of access exception
Carry out shielding the management such as isolation operation.
Data access registering unit 162 provides data access registering functional.When all data access logics, data object and
Data source, which has all carried out unified static registration, to be used, and this can lift the transparency of data access, reduce data and administer hardly possible
Degree, by the docking with other management platforms, data object, executive plan, sensitive data, which accesses, can be managed collectively audit, such as
Outside security system, after interfaced, the management of the access sentence of sensitive table can be achieved.
Access index output unit 163 and access index output function is provided.It can be generated automatically or on demand according to a variety of dimensions
The indication informations such as visit capacity, performance, abnormal amount and access source, data when can be run by the data access index
Improvement includes access safety audit and data storage scheme and access strategy, the adjustment of the Fault Isolation Strategy etc..
The abatement modules 160 and data operation executing module 170 of data are all according to " mathematical logic-tables of data-data source "
Mode carries out classification modeling management.Data abatement modules 160 carry out information gathering by basic management and control key element of this 3 dimensions.
7th, data manipulation execution module 170：
Data manipulation execution module 170 is by data manipulation with three key elements of " mathematical logic-tables of data-data source "
Decompose and the process of control data operation execution, including data failure redundant (failover) unit 171 and data are route
Unit 172；Wherein
The service end that data failure backup unit 171 provides data, services carries out automated availability detection to back-end data source
Function, if standby failover data sources be present in data source, then in current data source during exception, will enable automatically
Failover data sources, realize that operationally automatic fault switches.
Data routing units 172 are used for the " mathematical logic-tables of data-number of the service end parsing data manipulation of data, services
According to source " key element and data access request is routed to the number of multiple configurations automatically according to input condition and the context of current accessed
According to source.When can realize that table handling is divided in data source point storehouse by data routing units, application system is not influenceed.
The process that the connection of True Data storehouse is obtained on data serving node is shown in Figure 3.
The core of data manipulation execution module 170 is High Availabitity data source model, and model is shown in Figure 2.
The model is the secondary encapsulation based on database connection pool, so as to support the access strategy control between multiple databases
System.Support accesses data source in a manner of read and write abruption reduces read request to the pressure of database, supports with load balancing mode
Access multiple data sources of equal value to share the pressure of database, support that Fault Isolation in data source exception and automatic calamity are standby
Route requests in normal data source.
The encapsulation and modeling of data access process can be realized by the model, realizes management and control and improvement to data, is shielded
Back-end data base changes the influence to front end data fetcher code, there is provided a kind of unified stable data access patterns.
The system of management data access of the present invention has following features：
1st, using Distributed Services, realize that the height of data, services can by technologies such as clustering management, introducing ZooKeeper
The property used.
2nd, it can facilitate and the tables of data that certain data volume reaches the limit of is carried out more storehouses minimumly and divides table by cost, simultaneously
The logic of application service does not have to modification.
3rd, the database interactive access process of comprehensive adapter system, possesses visual data Governance Ability.
4th, data access process is supported the feature such as read and write abruption, intelligent failover, optimize the performance of data access simultaneously
Improve data access reliability.
5th, physical database supports the frequently-used data storehouses such as Oracle, MySql, possesses the expansible advantage used with.
Embodiments of the invention are the foregoing is only, are not intended to limit the scope of the invention, it is every to utilize this hair
The equivalents that bright specification and accompanying drawing content are made, or the technical field of correlation is directly or indirectly used in, similarly include
In the scope of patent protection of the present invention.
Claims (9)
1. a kind of system for managing data access, including client and service end, it is characterised in that also include：Load balancing mould
Block, High Availabitity support module, RPC communication module, flow-control module, affairs support module, data abatement modules and data behaviour
Make execution module；Wherein
Client passes through load balancing module access service end, behind data access service end, is ensured by High Availabitity support module
The availability of the service end of access, request of the flow-control module to client count, and to not being inconsistent interflow control configuration
Request is refused, and RPC communication module is used to realize client and service end data transmission, and affairs support module is used to judge to work as
The type of preceding data manipulation；And actual data access operation is completed by data manipulation execution module, with mathematical logic-data
The mode of table-data source controls and recorded implementation procedure, and carries out record system to accessing vestige eventually through data abatement modules
Meter.
2. the system of management data access according to claim 1, it is characterised in that the load balancing module is integrated in
In client, to obtain currently available service address list using ZooKeeper, load balancing is realized in client.
3. the system of management data access according to claim 2, it is characterised in that the High Availabitity support module passes through
The real-time monitoring service address lists of ZooKeeper, and reject the address of service of data serving node failure；And enable asynchronous thread
Heartbeat monitor is carried out to the back end in service address list, to ensure that ZooKeeper and data serving node network are sent out
Data serving node is not picked when raw exception and the network of application client end node and data service node without exception
Remove.
4. the system of management data access according to claim 1, it is characterised in that the RPC communication module uses
THRIFT frameworks, secondary encapsulation is carried out to communication process；It includes：
IDL definition units, for defining unified idl file to client and service end, to enter line interface identification, and generation pair
The serializing code and unserializing code answered；
Protocol encapsulation unit, for being embedded in encapsulation mode in serializing, and data parsing is carried out in unserializing；
Data format unit, for dividing data type；
Asynchronous communication unit, for the NIO asynchronous communications of client and service end, thread is handled using epoll technologies and I/O
Pond, lift the concurrency and handling capacity of data, services.
5. the system of management data access according to claim 1, it is characterised in that the flow-control module, which provides, to be divided
Cloth flow controls, and supports multi-stage data granularity, controls visitation frequency；It includes：
Flowmeter counter unit, for collecting the access information of each data serving node, and ZooKeeper is sent to, and
Limit being accessed beyond the unit interval, then think the abnormal information of client feedback flow control；
Control logic injection unit, the visitation frequency of sentence, tables of data and data source is accessed for controlling, and realizes Dynamic injection
Control.
6. the system of management data access according to claim 1, it is characterised in that the affairs support module includes：
Single-point transaction units, for controlling conversation affair on single application node by data, services and being sent out by the application node
The application, submission and rollback for playing affairs are called；
Distributed transaction unit, for cooperation with service end affairs and client transaction.
7. the system of management data access according to claim 6, it is characterised in that the service end provides affairs application
Interface, affairs submit interface and transaction rollback interface to supply client call；Wherein
After client call affairs application interface, by transaction context object local cache in service end, and by context
Mark returns to client, and the transaction context object cached is automatically positioned in follow-up client call every time；
After client is initiated to ask to service end, the database connection list that service end is associated by context object passes through affairs
Submit interface that affairs are submitted into database one by one, and discharge transaction context；
After client is initiated to ask to service end, the database connection list that service end is associated by context object passes through affairs
Affairs one by one from rollback database, and are discharged transaction context by rollback interface；
Service end carries out overtime management to affairs, overtime session is cleared up automatically, wherein automatic cleaning operation is automatic by asynchronous thread
Triggering；
Client call affairs application interface, when being passed to Context identifier when data, services are dispatched, client end AP I is according to upper
Data serving node when forwarding requests to affairs application automatically is hereafter identified, after the completion of data manipulation calling, calls thing
Interface or transaction rollback interface are submitted in business；
Client call affairs application interface, when the not incoming Context identifier when data, services are dispatched, then it represents that use nothing
Status data service, service end do not cache transaction context information after calling.
8. the system of management data access according to claim 1, it is characterised in that the data abatement modules include：
Data access monitoring unit, data access registering unit and data target output unit；Wherein
After service end enables asynchronous thread, the status information of data serving node and Efficiency Statistics synchronizing information are arrived automatically
ZooKeeper；
The IP address of client and process number are registered to ZooKeeper by client automatically；
The information of the data access monitoring unit inquiry service end and client, enters to the service end or client of access exception
Row shielding isolation；
The data access registering unit carries out unified static registration to all data access logics, data object and data source,
By the docking with external management platform, data object, executive plan and sensitive data are accessed and carry out unified management audit；
The access index output unit generates automatically or on demand indication information, and refer to by data access according to a variety of dimensions
Mark information, to operation when data manipulation perform Management strategy；Wherein indication information includes visit capacity, performance, abnormal amount and visit
Ask source；The Management strategy includes access safety audit, data storage scheme and access strategy and the Fault Isolation Strategy.
9. the system of management data access according to claim 1, it is characterised in that the data manipulation execution module bag
Include standby unit and data routing units；Wherein
Standby unit is used for service end and carries out automated availability detection to back-end data source, redundant data be present detecting
Behind source, then in current data source exception, the redundant data source is enabled automatically, realize automatic fault switching during operation；
Data routing units are used for service end in parsing data manipulation, and according to input condition and the Context identifier of current accessed
Data access request is routed to the data source of other configurations automatically, realizes and a point storehouse for data source is operated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710691745.0A CN107480251A (en) | 2017-08-14 | 2017-08-14 | A kind of system for managing data access |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710691745.0A CN107480251A (en) | 2017-08-14 | 2017-08-14 | A kind of system for managing data access |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107480251A true CN107480251A (en) | 2017-12-15 |
Family
ID=60599356
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710691745.0A Pending CN107480251A (en) | 2017-08-14 | 2017-08-14 | A kind of system for managing data access |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107480251A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509282A (en) * | 2018-02-08 | 2018-09-07 | 厦门快商通信息技术有限公司 | Across the language call service administering methods of rpc based on golang reflection technologies |
| CN109525570A (en) * | 2018-11-06 | 2019-03-26 | 东南大学 | A kind of data hierarchy safety access control method of Cargo Oriented on Group client |
| CN111176988A (en) * | 2019-12-20 | 2020-05-19 | 珠海金山网络游戏科技有限公司 | Python client based on GoCD |
| CN111245822A (en) * | 2020-01-08 | 2020-06-05 | 北京松果电子有限公司 | Remote procedure call processing method and device and computer storage medium |
| CN111245887A (en) * | 2019-12-17 | 2020-06-05 | 航天信息股份有限公司 | Hbase connection dynamic holding method, Hbase connection dynamic holding equipment, Hbase connection dynamic storage medium and Hbase connection dynamic storage system |
| CN111538605A (en) * | 2020-04-24 | 2020-08-14 | 北京思特奇信息技术股份有限公司 | Distributed data access layer middleware and command execution method and device |
| CN111552675A (en) * | 2020-04-24 | 2020-08-18 | 北京达佳互联信息技术有限公司 | Information query method and device, computer equipment and storage medium |
| CN112732820A (en) * | 2021-01-26 | 2021-04-30 | 中国人寿保险股份有限公司上海数据中心 | Database session management system and method thereof |
| CN113051042A (en) * | 2021-01-25 | 2021-06-29 | 北京思特奇信息技术股份有限公司 | Zookeeper-based transaction implementation method and system |
| CN113760465A (en) * | 2020-11-16 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Transaction execution method, device, server, equipment and storage medium |
| CN114428822A (en) * | 2022-01-27 | 2022-05-03 | 云启智慧科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114780251A (en) * | 2022-06-10 | 2022-07-22 | 深圳联友科技有限公司 | Method and system for improving computing performance by using distributed database architecture |
| CN115705031A (en) * | 2021-08-10 | 2023-02-17 | 唐山新海纳自动化有限公司 | Industrial equipment life-prolonging deep maintenance intelligent association expert system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102110016A (en) * | 2009-12-25 | 2011-06-29 | 华为技术有限公司 | Method, device and system for obtaining database connection in a database cluster system |
| CN102945264A (en) * | 2012-10-24 | 2013-02-27 | 浪潮集团山东通用软件有限公司 | Method for intelligently starting distributed transaction |
| CN103473318A (en) * | 2013-09-12 | 2013-12-25 | 中国科学院软件研究所 | Distributed transaction security method for memory data grid |
| CN103902560A (en) * | 2012-12-25 | 2014-07-02 | 中国移动通信集团福建有限公司 | Distributed transaction processing method and system |
| US20150109901A1 (en) * | 2012-06-30 | 2015-04-23 | Huawei Technologies Co., Ltd. | Method for managing forwarding plane tunnel resource under control and forwarding decoupled architecture |
| CN105045929A (en) * | 2015-08-31 | 2015-11-11 | 国家电网公司 | MPP architecture based distributed relational database |
| CN106657287A (en) * | 2016-11-30 | 2017-05-10 | 广州市千钧网络科技有限公司 | Data access method and system |
| CN107025270A (en) * | 2017-03-09 | 2017-08-08 | 珠海昊星自动化系统有限公司 | A kind of distributed high-performance high concurrent big data system |
-
2017
- 2017-08-14 CN CN201710691745.0A patent/CN107480251A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102110016A (en) * | 2009-12-25 | 2011-06-29 | 华为技术有限公司 | Method, device and system for obtaining database connection in a database cluster system |
| US20150109901A1 (en) * | 2012-06-30 | 2015-04-23 | Huawei Technologies Co., Ltd. | Method for managing forwarding plane tunnel resource under control and forwarding decoupled architecture |
| CN102945264A (en) * | 2012-10-24 | 2013-02-27 | 浪潮集团山东通用软件有限公司 | Method for intelligently starting distributed transaction |
| CN103902560A (en) * | 2012-12-25 | 2014-07-02 | 中国移动通信集团福建有限公司 | Distributed transaction processing method and system |
| CN103473318A (en) * | 2013-09-12 | 2013-12-25 | 中国科学院软件研究所 | Distributed transaction security method for memory data grid |
| CN105045929A (en) * | 2015-08-31 | 2015-11-11 | 国家电网公司 | MPP architecture based distributed relational database |
| CN106657287A (en) * | 2016-11-30 | 2017-05-10 | 广州市千钧网络科技有限公司 | Data access method and system |
| CN107025270A (en) * | 2017-03-09 | 2017-08-08 | 珠海昊星自动化系统有限公司 | A kind of distributed high-performance high concurrent big data system |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509282A (en) * | 2018-02-08 | 2018-09-07 | 厦门快商通信息技术有限公司 | Across the language call service administering methods of rpc based on golang reflection technologies |
| CN109525570A (en) * | 2018-11-06 | 2019-03-26 | 东南大学 | A kind of data hierarchy safety access control method of Cargo Oriented on Group client |
| CN109525570B (en) * | 2018-11-06 | 2021-01-12 | 东南大学 | Group client-oriented data layered security access control method |
| CN111245887A (en) * | 2019-12-17 | 2020-06-05 | 航天信息股份有限公司 | Hbase connection dynamic holding method, Hbase connection dynamic holding equipment, Hbase connection dynamic storage medium and Hbase connection dynamic storage system |
| CN111245887B (en) * | 2019-12-17 | 2022-03-25 | 航天信息股份有限公司 | Hbase connection dynamic holding method, Hbase connection dynamic holding equipment, Hbase connection dynamic storage medium and Hbase connection dynamic storage system |
| CN111176988A (en) * | 2019-12-20 | 2020-05-19 | 珠海金山网络游戏科技有限公司 | Python client based on GoCD |
| CN111176988B (en) * | 2019-12-20 | 2023-07-04 | 珠海金山数字网络科技有限公司 | GoCD-based Python client system |
| CN111245822A (en) * | 2020-01-08 | 2020-06-05 | 北京松果电子有限公司 | Remote procedure call processing method and device and computer storage medium |
| CN111552675A (en) * | 2020-04-24 | 2020-08-18 | 北京达佳互联信息技术有限公司 | Information query method and device, computer equipment and storage medium |
| CN111538605A (en) * | 2020-04-24 | 2020-08-14 | 北京思特奇信息技术股份有限公司 | Distributed data access layer middleware and command execution method and device |
| CN113760465A (en) * | 2020-11-16 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Transaction execution method, device, server, equipment and storage medium |
| CN113051042A (en) * | 2021-01-25 | 2021-06-29 | 北京思特奇信息技术股份有限公司 | Zookeeper-based transaction implementation method and system |
| CN113051042B (en) * | 2021-01-25 | 2024-04-19 | 北京思特奇信息技术股份有限公司 | Transaction realization method and system based on zookeeper |
| CN112732820A (en) * | 2021-01-26 | 2021-04-30 | 中国人寿保险股份有限公司上海数据中心 | Database session management system and method thereof |
| CN112732820B (en) * | 2021-01-26 | 2024-05-17 | 中国人寿保险股份有限公司上海数据中心 | Database session management system and method thereof |
| CN115705031A (en) * | 2021-08-10 | 2023-02-17 | 唐山新海纳自动化有限公司 | Industrial equipment life-prolonging deep maintenance intelligent association expert system |
| CN114428822B (en) * | 2022-01-27 | 2022-07-29 | 云启智慧科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114428822A (en) * | 2022-01-27 | 2022-05-03 | 云启智慧科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114780251A (en) * | 2022-06-10 | 2022-07-22 | 深圳联友科技有限公司 | Method and system for improving computing performance by using distributed database architecture |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107480251A (en) | A kind of system for managing data access | |
| CN106651633B (en) | Power utilization information acquisition system based on big data technology and acquisition method thereof | |
| CN104270417B (en) | A kind of integrated service based on cloud computing provides system and method | |
| CN104754045B (en) | Data center services and application interface management platform | |
| CN109840253A (en) | Enterprise-level big data platform framework | |
| CN107862572A (en) | A kind of power marketing business application system | |
| CN106780027A (en) | A kind of data handling system and method | |
| CN107315776A (en) | A kind of data management system based on cloud computing | |
| CN103458033A (en) | System for providing services of event-driven service-oriented internet of things and working method thereof | |
| CN107193546A (en) | A kind of micro services business application system | |
| CN107864222A (en) | A kind of industrial big data computing architecture based on PaaS platform | |
| CN104463492A (en) | Operation management method of electric power system cloud simulation platform | |
| CN106559488A (en) | A kind of method of the electrical network geographical information space service for setting up tenant's driving | |
| CN109445802A (en) | The method of privatization Paas platform and its publication application based on container | |
| CN110188132B (en) | Data exchange method and system | |
| CN104104738A (en) | FTP-based (file transfer protocol-based) data exchange system | |
| CN109677465B (en) | Distributed real-time system architecture for rail transit integrated monitoring system | |
| CN111724046B (en) | Electricity purchase management system | |
| CN109542583A (en) | A kind of virtual device management method based on dual bus | |
| CN107357791A (en) | A kind of data sharing method and device | |
| CN109302308A (en) | A kind of data center IDC resource management system and management method | |
| CN102207978A (en) | Database access method and system | |
| CN104240070A (en) | Data release service system and method | |
| CN106446268A (en) | Database lateral extension system and method | |
| CN109409685A (en) | Information O&M operation area management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20171215 |