CN105516344A - Data management system and service-processing method - Google Patents
Data management system and service-processing method Download PDFInfo
- Publication number
- CN105516344A CN105516344A CN201511027392.1A CN201511027392A CN105516344A CN 105516344 A CN105516344 A CN 105516344A CN 201511027392 A CN201511027392 A CN 201511027392A CN 105516344 A CN105516344 A CN 105516344A
- Authority
- CN
- China
- Prior art keywords
- data
- result
- management system
- client
- buffer memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data management system and a service-processing method. The service-processing method comprises steps of: receiving updated service information sent by a client, wherein the updated service information at least includes updated content; writing the updated content into a data persistence layer and a cache according to the updated content and acquiring a writing result; and generating a updated service processing result according to the writing result and returning the updated service processing result to the client. The invention also discloses a database management system. When a data management system has a mass of data, the method is capable of quickly responding to a mass of requests, timely processing the service requests, guaranteeing service-processing reliability and service data consistency.
Description
Technical field
The present invention relates to data management field, particularly relate to data management system and method for processing business.
Background technology
As shown in Figure 1, available data management system generally adopts DB (Database, database) to store user data, provides API (ApplicationProgrammingInterface, application programming interface) for client.Mainly contain three-decker, be respectively client api layer, Business Logic and Data Persistence Layer.User data register or more new technological process as shown in Figure 2, client transmit a request to GNS (GlobalNamingServices, global naming is served) upgrade service, then GNS update server is write data or is changed data to MYSQL (relevant database).As shown in Figure 3, client sends inquiry request to GNS inquiry service to user data query flow process, and then GNS querying server data query from MYSQL returns to client.Under prior art, suppose currently have 3 route needs, be respectively and carry out routed customer according to card number, account number, customer ID.Therefore the data structure of GNS will have 3 to table t_card, t_accout, t_ecif, as shown in Figure 4.Wherein t_card be card number to mapping relations, the t_accout of DCN (DataCenterNode, data center's node) number be account to mapping relations, the t_ecif of No. DCN be the mapping relations of customer ID to DCN.
Prior art adopts aforesaid way to use MYSQL as storage and inquiry, because MYSQL is as a kind of DB, when mass data, it inserts, upgrade and query performance all can be limited, our demand for magnanimity request magnanimity QPS (QueryPerSecond, query rate per second) cannot be met.The inquiry request of magnanimity can puncture MYSQL, causes the hydraulic performance decline of MYSQL.
Summary of the invention
Main purpose of the present invention is to provide a kind of data management system and method for processing business, is intended to the technical problem solving degraded performance when inserting mass data, upgrade and inquire about.
For achieving the above object, the invention provides a kind of data management system method for processing business, described data management system method for processing business comprises:
Receive the renewal business information that client sends, described renewal business information at least comprises more fresh content;
To more fresh content according to described renewal business information, be written in Data Persistence Layer and buffer memory, and obtain the result of write;
Generate according to described result and upgrade service processing result and return to client.
Preferably, describedly generate service processing result according to described result and comprise after returning to the step of client:
Receive the inquiry business information that client sends;
By presetting search algorithm, buffer memory being inquired about according to described inquiry business information, obtaining Query Result;
Return to client according to described Query Result generated query service processing result.
Preferably, described generation according to described result upgrades service processing result and also comprises after returning to the step of client:
According to the data in Data Persistence Layer, by default checking algorithm, data check is carried out to the data in buffer memory, the data in buffer memory and the data in Data Persistence Layer are consistent.
Preferably, described generation according to described result upgrades service processing result and also comprises after returning to the step of client:
The business flowing water produced when sending business information according to client, carries out reconciliation process by default reconciliation algorithm to the data in Data Persistence Layer, the data in Data Persistence Layer and business flowing water is consistent.
Preferably, described according to described renewal business information general more fresh content, be written in Data Persistence Layer and buffer memory, and the step obtaining the result of write comprises:
To more be written in Data Persistence Layer and buffer memory according to hierarchical index structure by fresh content according to described renewal business information by upgrading service, and obtain the result of write.
In addition, for achieving the above object, the present invention also provides a kind of data management system, and described data management system comprises:
Receive update module, for receiving the renewal business information that client sends, described renewal business information at least comprises more fresh content;
Data write. module, for inciting somebody to action more fresh content according to described renewal business information, is written in Data Persistence Layer and buffer memory, and obtains the result of write;
Renewal returns module, upgrades service processing result for generating according to described result and returns to client.
Preferably, described data management system also comprises:
Receive enquiry module, for receiving the inquiry business information that client sends;
Enquiry module, for inquiring about buffer memory by presetting search algorithm according to described inquiry business information, obtains Query Result;
Inquiry returns module, for returning to client according to described Query Result generated query service processing result.
Preferably, described data management system also comprises:
Data check module, for according to the data in Data Persistence Layer, carries out data check by default checking algorithm to the data in buffer memory, and the data in buffer memory and the data in Data Persistence Layer are consistent.
Preferably, described data management system also comprises:
Flowing water reconciliation module, the business flowing water produced during for sending business information according to client, carries out reconciliation process by default reconciliation algorithm to the data in Data Persistence Layer, the data in Data Persistence Layer and business flowing water is consistent.
Preferably, described Data write. module comprises:
Hierarchical index unit, for will more be written in Data Persistence Layer and buffer memory according to hierarchical index structure by fresh content according to described renewal business information by upgrading service, and obtains the result of write.
A kind of data management system that the embodiment of the present invention proposes and method for processing business, by increasing buffer memory in data management system, and keep the data consistent that stores in the data that store in buffer memory and Data Persistence Layer, achieve data management system when mass data, response can be made to magnanimity request fast.
Accompanying drawing explanation
Fig. 1 is prior art data management system hierarchy schematic diagram;
Fig. 2 is that prior art data management system upgrades business processing flow schematic diagram;
Fig. 3 is prior art data management system inquiry business handling process schematic diagram;
Fig. 4 is prior art data management system data store organisation schematic diagram;
Fig. 5 is a kind of possible data management system hierarchy schematic diagram of the present invention;
Fig. 6 is the schematic flow sheet of the first embodiment of data management system operational approach of the present invention;
Fig. 7 is a kind of possible renewal business processing flow schematic diagram of data management system of the present invention;
Fig. 8 is the schematic flow sheet of the second embodiment of data management system operational approach of the present invention;
Fig. 9 is a kind of possible inquiry business handling process schematic diagram of data management system of the present invention;
Figure 10 is the schematic flow sheet of the 3rd embodiment of data management system operational approach of the present invention;
Figure 11 is the schematic flow sheet of the 4th embodiment of data management system operational approach of the present invention;
Figure 12 is the schematic flow sheet of the 5th embodiment of data management system operational approach of the present invention;
Figure 13 is the high-level schematic functional block diagram of the first embodiment of data management system of the present invention;
Figure 14 is the high-level schematic functional block diagram of the second embodiment of data management system of the present invention;
Figure 15 is the high-level schematic functional block diagram of the 3rd embodiment of data management system of the present invention;
Figure 16 is the high-level schematic functional block diagram of the 4th embodiment of data management system of the present invention;
Figure 17 is the high-level schematic functional block diagram of the 5th embodiment of data management system of the present invention;
Figure 18 is a kind of possible hierarchical index structured data table design diagram of the present invention;
Figure 19 is the another kind of possible hierarchical index structured data table design diagram of the present invention;
The realization of the object of the invention, functional characteristics and advantage will in conjunction with the embodiments, are described further with reference to accompanying drawing.
Embodiment
Should be appreciated that specific embodiment described herein only in order to explain the present invention, be not intended to limit the present invention.
The primary solutions of the embodiment of the present invention is: receive the renewal business information that client sends, described renewal business information at least comprises more fresh content; To more fresh content according to described renewal business information, be written in Data Persistence Layer and buffer memory, and obtain the result of write; Generate according to described result and upgrade service processing result and return to client.
Because the direct usage data storehouse of prior art carries out storing and the Business Processing such as inquiry, therefore when mass data, cannot response request in time, data management system traffic handing capacity is reduced, cannot satisfy the demands.
The invention provides a solution, buffer memory is increased in data management system, and ensure the data consistent that stores in the data that store in buffer memory and Data Persistence Layer, make data management system can response service request in time when mass data, and in time described service request is processed, meet the demand of process magnanimity service request.
As shown in Figure 5, be a kind of possible application structure of data management system of the present invention and method for processing business, wherein:
Business Stream water layer: record be that GNS mapping relations produce the business operation flowing water of source ECIF (EnterpriseCustomerInformationFacility, corporate client information factory) system.Flowing water is stored in FPS (chronological file storage system);
Data Persistence Layer: Data Persistence Layer adopts MYSQL to land, GNS inner with main MYSQL for AUTHORITATIVE DATA;
Cache layer: cache layer provides high performance inquiry service, current employing CKV (CloudKey_Value, high in the clouds Key-Value store);
Business Logic: comprise GNS inquiry service that GNS server provides and GNS upgrades service.The service logic such as network processes, request process, DCN distribution of main process GNS;
Api layer: provide java, c++API to call to business.Be divided into inquiry API and upgrade API two bag.
With reference to Fig. 6, be the first embodiment of data management system method for processing business of the present invention, described data management system method for processing business comprises:
Step S100, receive the renewal business information that client sends, described renewal business information at least comprises more fresh content;
Receive the renewal business information containing more fresh content that client sends, described more new business can be that insertion to data, deletion, amendment and renewal etc. exist the business of amendment to the data in described data management system, and described more fresh content can insert the data in described data management system for needing.
Step S200, incites somebody to action more fresh content according to described renewal business information, is written in Data Persistence Layer and buffer memory, and obtains the result of write;
According to described renewal business information, by the more fresh content comprised in described renewal business information, be written in Data Persistence Layer, be written in buffer memory simultaneously, and obtain write result, in described data management system, the data consistent in Data Persistence Layer and buffer memory, a kind of optional execution mode is, according to the more fresh content comprised in described renewal business information, be synchronously written in Data Persistence Layer, asynchronous write is in buffer memory simultaneously, and obtains the write result be synchronously written in Data Persistence Layer.
Step S300, generates according to described result and upgrades service processing result and return to client;
According to said write result, generate according to preset standard and upgrade service processing result, and returned to client, so that client processes accordingly according to described renewal service processing result.
During concrete enforcement, for the concrete structure shown in Fig. 5, the concrete implementation step of the present embodiment as shown in Figure 7, comprising:
1, GNS server receives client send renewal or registration service request;
2, according to described service request will need upgrade or registration content synchronization write MYSQL in, i.e. Data Persistence Layer, simultaneously will need upgrade or registration content asynchronous write buffer memory in;
3, the write result being synchronously written MYSQL is obtained;
4, generate renewal or registration service request result according to said write result, and described service request result is returned to client.
In the present embodiment, by client more new business renewal content synchronization write Data Persistence Layer in and in asynchronous write buffer memory, guarantee data security in write Data Persistence Layer, and by preserving the data content consistent with Data Persistence Layer in the buffer, make data management system can response service request in time when magnanimity service request, and return service request result, improve described data management system Business Processing reliability.
Further, with reference to Fig. 8, be the second embodiment of data management system method for processing business of the present invention, based on the embodiment shown in above-mentioned Fig. 6, described step S300, generates according to described result and upgrades service processing result and comprise after returning to client:
Step S400, receives the inquiry business information that client sends;
Receive the inquiry business information that client sends, described inquiry business information may be inquired about for magnanimity business datum, and described inquiry business information comprises querying condition.
Step S500, inquiring about buffer memory by presetting search algorithm according to described inquiry business information, obtaining Query Result;
According to the querying condition comprised in described inquiry business information, inquire about by presetting search algorithm the data of preserving in buffer memory, obtain Query Result, described buffer memory is fast reading and writing unit.
Step S600, returns to client according to described Query Result generated query service processing result;
According to described Query Result, according to preset standard generated query service processing result, and returned to client, so that client processes accordingly according to described inquiry business result.
During concrete enforcement, for the concrete structure shown in Fig. 5, the concrete implementation step of the present embodiment comprises:
1, GNS server receives client send renewal or registration service request;
2, according to described service request will need upgrade or registration content synchronization write MYSQL in, i.e. Data Persistence Layer, simultaneously will need upgrade or registration content asynchronous write buffer memory in;
3, the write result being synchronously written MYSQL is obtained;
4, generate renewal or registration service request result according to said write result, and described service request result is returned to client;
5, the inquiry business request of GNS server receives client transmission;
6, by presetting search algorithm, CKV being inquired about according to described inquiry business request, obtaining Query Result;
7, according to described Query Result generated query business result, and described inquiry business result is returned to client;
Wherein, as shown in Figure 7, being the schematic flow sheet of above-mentioned steps 1-4, as shown in Figure 9, is the schematic flow sheet of above-mentioned steps 5-7.
In the present embodiment, based on the advantage described in a upper embodiment, when receiving the inquiry business information that client sends, by default search algorithm, buffer memory is inquired about, and obtain Query Result, when making to carry out magnanimity QPS request when mass data to described data management system, described data management system can response request returning results timely fast, improves the efficiency of described data management system Business Processing.
Further, with reference to Figure 10, be the 3rd embodiment of data management system method for processing business of the present invention, based on the embodiment shown in above-mentioned Fig. 6, described step S300, generates according to described result and upgrades service processing result and comprise after returning to client:
Step S301, according to the data in Data Persistence Layer, carries out data check by default checking algorithm to the data in buffer memory, and the data in buffer memory and the data in Data Persistence Layer are consistent;
With the data of preserving in Data Persistence Layer for benchmark, verified to the data in buffer memory by default checking algorithm, if there is the place inconsistent with the data in Data Persistence Layer in the data in buffer memory, then by presetting restore design, the data in buffer memory are repaired, ensure the data in buffer memory and the data consistent in Data Persistence Layer, describedly may to carry out for high frequency periodic the triggering that the data in buffer memory verify, ensure to repair the data in buffer memory fast, make the data in buffer memory and the data consistent in Data Persistence Layer.
During concrete enforcement, for the concrete structure shown in Fig. 5, the concrete implementation step of the present embodiment comprises:
1, GNS server receives client send renewal or registration service request;
2, according to described service request will need upgrade or registration content synchronization write MYSQL in, i.e. Data Persistence Layer, simultaneously will need upgrade or registration content asynchronous write buffer memory in;
3, the write result being synchronously written MYSQL is obtained;
4, generate renewal or registration service request result according to said write result, and described service request result is returned to client;
5, trigger the consistency of data in data and buffer memory in the data verification system verification MYSQL that GNS server provides by arranging overall poll, a kind of possible situation is 10 seconds for arranging polling interval;
6, when data are inconsistent in the data in buffer memory and Data Persistence Layer, find fast and repair this inconsistency.
In the present embodiment, based on the advantage described in a upper embodiment, by verifying the data in data in buffer memory and Data Persistence Layer, and the place that data in time in repair cache in data and Data Persistence Layer are inconsistent, ensure that the data in buffer memory and the data consistent in Data Persistence Layer.
Further, with reference to Figure 11, be the 4th embodiment of data management system method for processing business of the present invention, based on the embodiment shown in above-mentioned Fig. 6, described step S300, generates according to described result and upgrades service processing result and comprise after returning to client:
Step S302, the business flowing water produced when sending business information according to client, carries out reconciliation process by default reconciliation algorithm to the data in Data Persistence Layer, the data in Data Persistence Layer and business flowing water is consistent;
Client will produce corresponding business flowing water when sending business information, by the reconciliation algorithm preset, reconciliation process is carried out to the data in Data Persistence Layer according to described business flowing water, and according to the inconsistency of the data in business flowing water repair data persistent layer and business flowing water, one preferably execution mode is: according to client traffic visit capacity, and every day carries out reconciliation process in the moment that client traffic visit capacity is less to the data in Data Persistence Layer.
During concrete enforcement, for the concrete structure shown in Fig. 5, the concrete implementation step of the present embodiment comprises:
1, GNS server receives client send renewal or registration service request;
2, according to described service request will need upgrade or registration content synchronization write MYSQL in, i.e. Data Persistence Layer, simultaneously will need upgrade or registration content asynchronous write buffer memory in;
3, the write result being synchronously written MYSQL is obtained;
4, generate renewal or registration service request result according to said write result, and described service request result is returned to client;
5, the flowing water reconciliation system using GNS server to provide can receive ECIF system Notice of Reconciliation morning every day;
6, from FPS downloading service pipelined data data and MYSQL, data are checked, the consistency of verification MYSQL and business pipelined data;
7, when MYSQL and business flowing water inconsistency, accurate location finding also repairs this inconsistency.
In the present embodiment, based on the advantage described in a upper embodiment, by carrying out reconciliation process to the data in Data Persistence Layer and business flowing water, ensureing that the data in Data Persistence Layer are consistent with business flowing water, making data that described data management system provides accurately and reliably.
Further, with reference to Figure 12, for the 5th embodiment of data management system method for processing business of the present invention, based on the embodiment shown in above-mentioned Fig. 6, described step S200, to more fresh content according to described renewal business information, be written in Data Persistence Layer and buffer memory, and the step obtaining the result of write comprises:
Step S201, will more be written in Data Persistence Layer and buffer memory according to hierarchical index structure by fresh content according to described renewal business information by upgrading service, and obtain the result of write;
The more fresh content in described renewal business information is obtained according to described renewal business information, hierarchical index structure is generated according to described more fresh content by upgrading service, and described renewal content synchronization is written in Data Persistence Layer and asynchronous write in buffer memory, and obtain the result that is synchronously written.
During concrete enforcement, for the Data Structure Design of described hierarchical index structure route information for type+app_id+key_value to ECIF number, a kind of possible embodiment comprises:
1, as shown in figure 18, routing iinformation is divided into 100 tables t_key_0, t_key_1 to t_key_99 to the mapping of customer ID;
2, submeter routing algorithm adopts BKDRHash algorithm, is input as the key_value value in routing iinformation, exports the numerical value into 0-99, be the sequence number of table;
3, as described in Figure 19, customer ID is divided into 100 tables t_cif_0, t_cif_1 to t_cif_99 to the mapping of No. DCN;
4, submeter routing algorithm adopts BKDRHash algorithm, is input as No. ECIF, customer ID, exports the numerical value into 0-99, be the sequence number of table.
In the present embodiment, based on the advantage described in a upper embodiment, by preserving the mode of data according to hierarchical index structure, routing iinformation data balancing being distributed, adds data management system autgmentability, reduce the complexity of data additions and deletions simultaneously.
With reference to Figure 13, be the first embodiment of data management system of the present invention, described data management system comprises:
Receive update module 100, for receiving the renewal business information that client sends, described renewal business information at least comprises more fresh content;
Receive the renewal business information containing more fresh content that client sends, described more new business can be that insertion to data, deletion, amendment and renewal etc. exist the business of amendment to the data in described data management system, and described more fresh content can insert the data in described data management system for needing.
Data write. module 200, for inciting somebody to action more fresh content according to described renewal business information, is written in Data Persistence Layer and buffer memory, and obtains the result of write;
According to described renewal business information, by the more fresh content comprised in described renewal business information, be written in Data Persistence Layer, asynchronous write is in buffer memory simultaneously, and obtain write result, in described data management system, the data consistent in Data Persistence Layer and buffer memory, a kind of optional execution mode is, according to the more fresh content comprised in described renewal business information, be synchronously written in Data Persistence Layer, asynchronous write is in buffer memory simultaneously, and obtains the write result be synchronously written in Data Persistence Layer.
Renewal returns module 300, upgrades service processing result for generating according to described result and returns to client;
According to said write result, generate according to preset standard and upgrade service processing result, and returned to client, so that client processes accordingly according to described renewal service processing result.
During concrete enforcement, for the concrete structure shown in Fig. 5, the concrete implementation step of the present embodiment as shown in Figure 7, comprising:
1, GNS server receives client send renewal or registration service request;
2, according to described service request will need upgrade or registration content synchronization write MYSQL in, i.e. Data Persistence Layer, simultaneously will need upgrade or registration content asynchronous write buffer memory in;
3, the write result being synchronously written MYSQL is obtained;
4, generate renewal or registration service request result according to said write result, and described service request result is returned to client.
In the present embodiment, by client more new business renewal content synchronization write Data Persistence Layer in and in asynchronous write buffer memory, guarantee data security in write Data Persistence Layer, and by preserving the data content consistent with Data Persistence Layer in the buffer, make data management system can response service request in time when magnanimity service request, and return service request result, improve described data management system Business Processing reliability.
Further, with reference to Figure 14, be the second embodiment of data management system of the present invention, based on the embodiment shown in above-mentioned Figure 13, described data management system comprises:
Receive enquiry module 400, for receiving the inquiry business information that client sends;
Receive the inquiry business information that client sends, described inquiry business information may be inquired about for magnanimity business datum, and described inquiry business information comprises querying condition.
Enquiry module 500, for inquiring about buffer memory by presetting search algorithm according to described inquiry business information, obtains Query Result;
According to the querying condition comprised in described inquiry business information, inquire about by presetting search algorithm the data of preserving in buffer memory, obtain Query Result, described buffer memory is fast reading and writing unit.
Inquiry returns module 600, for returning to client according to described Query Result generated query service processing result;
According to described Query Result, according to preset standard generated query service processing result, and returned to client, so that client processes accordingly according to described inquiry business result.
During concrete enforcement, for the concrete structure shown in Fig. 5, the concrete implementation step of the present embodiment comprises:
1, GNS server receives client send renewal or registration service request;
2, according to described service request will need upgrade or registration content synchronization write MYSQL in, i.e. Data Persistence Layer, simultaneously will need upgrade or registration content asynchronous write buffer memory in;
3, the write result being synchronously written MYSQL is obtained;
4, generate renewal or registration service request result according to said write result, and described service request result is returned to client;
5, the inquiry business request of GNS server receives client transmission;
6, by presetting search algorithm, CKV being inquired about according to described inquiry business request, obtaining Query Result;
7, according to described Query Result generated query business result, and described inquiry business result is returned to client.
Wherein, as shown in Figure 7, being the schematic flow sheet of above-mentioned steps 1-4, as shown in Figure 9, is the schematic flow sheet of above-mentioned steps 5-7.
In the present embodiment, based on the advantage described in a upper embodiment, when receiving the inquiry business information that client sends, by default search algorithm, buffer memory is inquired about, and obtain Query Result, when making to carry out magnanimity QPS request when mass data to described data management system, described data management system can response request returning results timely fast, improves the efficiency of described data management system Business Processing.
Further, with reference to Figure 15, be the 3rd embodiment of data management system of the present invention, based on the embodiment shown in above-mentioned Figure 13, described data management system comprises:
Data check module 301, for according to the data in Data Persistence Layer, carries out data check by default checking algorithm to the data in buffer memory, and the data in buffer memory and the data in Data Persistence Layer are consistent;
With the data of preserving in Data Persistence Layer for benchmark, verified to the data in buffer memory by default checking algorithm, if there is the place inconsistent with the data in Data Persistence Layer in the data in buffer memory, then by presetting restore design, the data in buffer memory are repaired, ensure the data in buffer memory and the data consistent in Data Persistence Layer, describedly may to carry out for high frequency periodic the triggering that the data in buffer memory verify, ensure to repair the data in buffer memory fast, make the data in buffer memory and the data consistent in Data Persistence Layer.
During concrete enforcement, for the concrete structure shown in Fig. 5, the concrete implementation step of the present embodiment comprises:
1, GNS server receives client send renewal or registration service request;
2, according to described service request will need upgrade or registration content synchronization write MYSQL in, i.e. Data Persistence Layer, simultaneously will need upgrade or registration content asynchronous write buffer memory in;
3, the write result being synchronously written MYSQL is obtained;
4, generate renewal or registration service request result according to said write result, and described service request result is returned to client;
5, trigger the consistency of data in data and buffer memory in the data verification system verification MYSQL that GNS server provides by arranging overall poll, a kind of possible situation is 10 seconds for arranging polling interval;
6, when data are inconsistent in the data in buffer memory and Data Persistence Layer, find fast and repair this inconsistency.
In the present embodiment, based on the advantage described in a upper embodiment, by verifying the data in data in buffer memory and Data Persistence Layer, and the place that data in time in repair cache in data and Data Persistence Layer are inconsistent, ensure that the data in buffer memory and the data consistent in Data Persistence Layer.
Further, with reference to Figure 16, be the 4th embodiment of data management system of the present invention, based on the embodiment shown in above-mentioned Figure 13, described data management system comprises:
Flowing water reconciliation module 302, the business flowing water produced during for sending business information according to client, carries out reconciliation process by default reconciliation algorithm to the data in Data Persistence Layer, the data in Data Persistence Layer and business flowing water is consistent;
Client will produce corresponding business flowing water when sending business information, by the reconciliation algorithm preset, reconciliation process is carried out to the data in Data Persistence Layer according to described business flowing water, and according to the inconsistency of the data in business flowing water repair data persistent layer and business flowing water, one preferably execution mode is: according to client traffic visit capacity, and every day carries out reconciliation process in the moment that client traffic visit capacity is less to the data in Data Persistence Layer.
During concrete enforcement, for the concrete structure shown in Fig. 5, the concrete implementation step of the present embodiment comprises:
1, GNS server receives client send renewal or registration service request;
2, according to described service request will need upgrade or registration content synchronization write MYSQL in, i.e. Data Persistence Layer, simultaneously will need upgrade or registration content asynchronous write buffer memory in;
3, the write result being synchronously written MYSQL is obtained;
4, generate renewal or registration service request result according to said write result, and described service request result is returned to client;
5, the flowing water reconciliation system using GNS server to provide can receive ECIF system Notice of Reconciliation morning every day;
6, from FPS downloading service pipelined data data and MYSQL, data are checked, the consistency of verification MYSQL and business pipelined data;
7, when MYSQL and business flowing water inconsistency, accurate location finding also repairs this inconsistency.
In the present embodiment, based on the advantage described in a upper embodiment, by carrying out reconciliation process to the data in Data Persistence Layer and business flowing water, ensureing that the data in Data Persistence Layer are consistent with business flowing water, making data that described data management system provides accurately and reliably.
Further, with reference to Figure 17, be the 5th embodiment of data management system of the present invention, based on the embodiment shown in above-mentioned Figure 13, described Data write. module 200 comprises:
Hierarchical index unit 201, for will more be written in Data Persistence Layer and buffer memory according to hierarchical index structure by fresh content according to described renewal business information by upgrading service, and obtains the result of write;
The more fresh content in described renewal business information is obtained according to described renewal business information, hierarchical index structure is generated according to described more fresh content by upgrading service, and described renewal content synchronization is written in Data Persistence Layer and asynchronous write in buffer memory, and obtain the result that is synchronously written.
During concrete enforcement, for the Data Structure Design of described hierarchical index structure route information for type+app_id+key_value to ECIF number, a kind of possible embodiment comprises:
1, as shown in figure 18, routing iinformation is divided into 100 tables t_key_0, t_key_1 to t_key_99 to the mapping of customer ID;
2, submeter routing algorithm adopts BKDRHash algorithm, is input as the key_value value in routing iinformation, exports the numerical value into 0-99, be the sequence number of table;
3, as described in Figure 19, customer ID is divided into 100 tables t_cif_0, t_cif_1 to t_cif_99 to the mapping of No. DCN;
4, submeter routing algorithm adopts BKDRHash algorithm, is input as No. ECIF, customer ID, exports the numerical value into 0-99, be the sequence number of table.
In the present embodiment, based on the advantage described in a upper embodiment, by preserving the mode of data according to hierarchical index structure, routing iinformation data balancing being distributed, adds data management system autgmentability, reduce the complexity of data additions and deletions simultaneously.
These are only the preferred embodiments of the present invention; not thereby the scope of the claims of the present invention is limited; every utilize specification of the present invention and accompanying drawing content to do equivalent structure or equivalent flow process conversion; or be directly or indirectly used in other relevant technical fields, be all in like manner included in scope of patent protection of the present invention.
Claims (10)
1. a data management system method for processing business, is characterized in that, described data management system method for processing business comprises the following steps:
Receive the renewal business information that client sends, described renewal business information at least comprises more fresh content;
To more fresh content according to described renewal business information, be written in Data Persistence Layer and buffer memory, and obtain the result of write;
Generate according to described result and upgrade service processing result and return to client.
2. the method for claim 1, is characterized in that, describedly generates service processing result according to described result and comprises after returning to the step of client:
Receive the inquiry business information that client sends;
By presetting search algorithm, buffer memory being inquired about according to described inquiry business information, obtaining Query Result;
Return to client according to described Query Result generated query service processing result.
3. method as claimed in claim 1 or 2, is characterized in that, described generation according to described result upgrades service processing result and also comprise after returning to the step of client:
According to the data in Data Persistence Layer, by default checking algorithm, data check is carried out to the data in buffer memory, the data in buffer memory and the data in Data Persistence Layer are consistent.
4. method as claimed in claim 3, is characterized in that, described generation according to described result upgrades service processing result and also comprise after returning to the step of client:
The business flowing water produced when sending business information according to client, carries out reconciliation process by default reconciliation algorithm to the data in Data Persistence Layer, the data in Data Persistence Layer and business flowing water is consistent.
5. method as claimed in claim 4, is characterized in that, described according to described renewal business information general more fresh content, be written in Data Persistence Layer and buffer memory, and the step obtaining the result of write comprises:
To more be written in Data Persistence Layer and buffer memory according to hierarchical index structure by fresh content according to described renewal business information by upgrading service, and obtain the result of write.
6. a data management system, is characterized in that, described data management system comprises:
Receive update module, for receiving the renewal business information that client sends, described renewal business information at least comprises more fresh content;
Data write. module, for inciting somebody to action more fresh content according to described renewal business information, is written in Data Persistence Layer and buffer memory, and obtains the result of write;
Renewal returns module, upgrades service processing result for generating according to described result and returns to client.
7. data management system as claimed in claim 6, it is characterized in that, described data management system also comprises:
Receive enquiry module, for receiving the inquiry business information that client sends;
Enquiry module, for inquiring about buffer memory by presetting search algorithm according to described inquiry business information, obtains Query Result;
Inquiry returns module, for returning to client according to described Query Result generated query service processing result.
8. data management system as claimed in claims 6 or 7, it is characterized in that, described data management system also comprises:
Data check module, for according to the data in Data Persistence Layer, carries out data check by default checking algorithm to the data in buffer memory, and the data in buffer memory and the data in Data Persistence Layer are consistent.
9. data management system as claimed in claim 8, it is characterized in that, described data management system also comprises:
Flowing water reconciliation module, the business flowing water produced during for sending business information according to client, carries out reconciliation process by default reconciliation algorithm to the data in Data Persistence Layer, the data in Data Persistence Layer and business flowing water is consistent.
10. data management system as claimed in claim 9, it is characterized in that, described Data write. module comprises:
Hierarchical index unit, for will more be written in Data Persistence Layer and buffer memory according to hierarchical index structure by fresh content according to described renewal business information by upgrading service, and obtains the result of write.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511027392.1A CN105516344A (en) | 2015-12-31 | 2015-12-31 | Data management system and service-processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511027392.1A CN105516344A (en) | 2015-12-31 | 2015-12-31 | Data management system and service-processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105516344A true CN105516344A (en) | 2016-04-20 |
Family
ID=55723974
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511027392.1A Pending CN105516344A (en) | 2015-12-31 | 2015-12-31 | Data management system and service-processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105516344A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107330084A (en) * | 2017-07-03 | 2017-11-07 | 深圳市全民合伙人科技有限公司 | A kind of network selling data managing method and system |
| CN107844575A (en) * | 2017-11-06 | 2018-03-27 | 国家计算机网络与信息安全管理中心 | A kind of personal data shared system and method for internet financial platform |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140359063A1 (en) * | 2013-05-28 | 2014-12-04 | Hitachi, Ltd. | Computer system, cache control method, and server |
| CN104462194A (en) * | 2014-10-28 | 2015-03-25 | 北京国双科技有限公司 | Service data processing method, device and server |
| CN104750746A (en) * | 2013-12-30 | 2015-07-01 | 中国移动通信集团上海有限公司 | Service data processing method and device and distributed internal memory database system |
| CN104850509A (en) * | 2015-04-27 | 2015-08-19 | 交通银行股份有限公司 | Method and system for operating bank business data memory cache |
-
2015
- 2015-12-31 CN CN201511027392.1A patent/CN105516344A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140359063A1 (en) * | 2013-05-28 | 2014-12-04 | Hitachi, Ltd. | Computer system, cache control method, and server |
| CN104750746A (en) * | 2013-12-30 | 2015-07-01 | 中国移动通信集团上海有限公司 | Service data processing method and device and distributed internal memory database system |
| CN104462194A (en) * | 2014-10-28 | 2015-03-25 | 北京国双科技有限公司 | Service data processing method, device and server |
| CN104850509A (en) * | 2015-04-27 | 2015-08-19 | 交通银行股份有限公司 | Method and system for operating bank business data memory cache |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107330084A (en) * | 2017-07-03 | 2017-11-07 | 深圳市全民合伙人科技有限公司 | A kind of network selling data managing method and system |
| CN107844575A (en) * | 2017-11-06 | 2018-03-27 | 国家计算机网络与信息安全管理中心 | A kind of personal data shared system and method for internet financial platform |
| CN107844575B (en) * | 2017-11-06 | 2021-10-15 | 国家计算机网络与信息安全管理中心 | Personal data sharing system and method for Internet financial platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101090401B (en) | Data buffer store method and system at duster environment | |
| CN103067433B (en) | A kind of data migration method of distributed memory system, equipment and system | |
| CN100505762C (en) | Distributed multi-stage buffer storage system suitable for object network storage | |
| US20160364407A1 (en) | Method and Device for Responding to Request, and Distributed File System | |
| CN102880557B (en) | look-up method of multistage distribution type high-speed cache of heterogeneous data source | |
| CN103312624B (en) | A kind of Message Queuing Services system and method | |
| EP2710477B1 (en) | Distributed caching and cache analysis | |
| CN104954468A (en) | Resource allocation method and resource allocation device | |
| CN100437590C (en) | Method for prefetching object | |
| CN103150394A (en) | Distributed file system metadata management method facing to high-performance calculation | |
| CN106326499B (en) | A kind of data processing method and device | |
| CN104378452A (en) | Method, device and system for domain name resolution | |
| CN102694828A (en) | Method and apparatus for data access in distributed caching system | |
| US9380127B2 (en) | Distributed caching and cache analysis | |
| CN111177143B (en) | Key value data storage method and device, storage medium and electronic equipment | |
| US20200301944A1 (en) | Method and apparatus for storing off-chain data | |
| CN103294610A (en) | Reusable content addressable stores | |
| CN109271449A (en) | A kind of distributed storage inquiry system file-based and querying method | |
| CN105468660A (en) | Reading method for distributed file system, client device and distributed file system | |
| CN109299111A (en) | A kind of metadata query method, apparatus, equipment and computer readable storage medium | |
| CN108399175B (en) | Data storage and query method and device | |
| CN109254981A (en) | A kind of data managing method and device of distributed cache system | |
| WO2020215580A1 (en) | Distributed global data deduplication method and device | |
| CN107704475B (en) | Multilayer distributed unstructured data storage method, query method and device | |
| CN105516344A (en) | Data management system and service-processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160420 |