CN114547052B - Method for updating cache of distributed system - Google Patents
Method for updating cache of distributed system Download PDFInfo
- Publication number
- CN114547052B CN114547052B CN202210132987.7A CN202210132987A CN114547052B CN 114547052 B CN114547052 B CN 114547052B CN 202210132987 A CN202210132987 A CN 202210132987A CN 114547052 B CN114547052 B CN 114547052B
- Authority
- CN
- China
- Prior art keywords
- service
- core module
- cache
- management module
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 18
- 230000010354 integration Effects 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 230000007958 sleep Effects 0.000 description 2
- GNFTZDOKVXKIBK-UHFFFAOYSA-N 3-(2-methoxyethoxy)benzohydrazide Chemical compound COCCOC1=CC=CC(C(=O)NN)=C1 GNFTZDOKVXKIBK-UHFFFAOYSA-N 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Abstract
The invention provides a method for updating a cache of a distributed system, which comprises the following steps: creating a database table by the software system; the software system comprises a core module and a management module; starting a management module, creating a daemon thread, starting to query a database at regular time, acquiring all the core modules which are registered to be available, and removing the core modules which are unavailable due to overtime; starting a core module, creating a daemon thread, and starting a rpc server by using netty; the core module creates a second daemon thread, starts to report heartbeat to the management module at regular time in an http interface mode, and registers rpc server information to the management module; the service system finds a corresponding rpc server; requesting to operate an API processing cache built in the core module by sending rpc a request to a registered rpc server; on the premise that the distributed system reduces database IO and network IO operations and improves system concurrency by using a caching technology, the caches of all nodes in the distributed system are synchronously updated, cache consistency is ensured, and accuracy of acquired data is ensured.
Description
Technical Field
The invention relates to a method for updating a cache of a distributed system, belonging to the technical field of software system design.
Background
In modern software systems, a distributed system architecture and a caching technology are generally adopted as a countermeasure in the face of high concurrent access, more service nodes are coordinated by using the distributed system architecture and system services are provided at the same time, and IO operation efficiency of the system can be improved by using the caching technology, so that the overall performance of the software system is improved.
The distributed system generally accelerates data access through the cache, the memory database is selected as the cache in the current mainstream cache design scheme to improve disk IO and further improve system performance, however, network IO from application service to the memory database also becomes a system bottleneck to affect the overall performance of the distributed system in some cases, so that the first-level jvm cache is generally reused on the basis of using the memory database, data is cached in the local area of the distributed nodes, and network IO is reduced.
When the content in the database is changed, jvm caches of all relevant distributed nodes need to be updated on the basis of updating the memory database, so a set of cache updating mechanism must be designed to ensure cache consistency of the distributed system.
Disclosure of Invention
The invention aims to provide a method for updating cache of a distributed system, which ensures that a request for cache processing can access all relevant nodes in the distributed system, executes a cache processing command and ensures cache consistency of the distributed system. .
The invention aims to achieve the aim, and the aim is achieved by the following technical scheme:
a method for updating cache of a distributed system is characterized in that a software system consists of two independent modules, namely a core module and a management module, wherein an API for cache operation is arranged in the core module, and the cache can be managed by calling the API; the management module is internally provided with an API for management operation, and the core module can be managed by calling the API; the method specifically comprises the following steps:
1) The software system creates a database table comprising a business model table and a service instance table; the method specifically comprises the following steps:
a-1) when all the service systems integrated with the core module are started, registering own service system information into a service model table,
a-2) when all the service systems integrated with the core module are started, the rpc server information started by the service system and the relationship of the service model are registered in a service instance table,
2) Starting a management module, creating a daemon thread, starting to query a database at regular time, acquiring all the core modules which are registered to be available, and removing the core modules which are unavailable due to overtime;
3) Starting a core module, creating a daemon thread, and starting a rpc server by using netty;
4) The core module creates a second daemon thread, starts to report heartbeat to the management module at regular time in an http interface mode, and registers rpc server information to the management module;
5) The service system calls a management module interface to process the cache, the management system obtains all core modules available for registration, and a corresponding rpc server is found through registration information;
6) The built-in API process cache of the core module is operated by sending rpc a request to the registered rpc server.
Preferably, the step 2 management module starts creating a daemon thread, and the thread mainly performs the following steps:
2-1) reading a service instance registry to obtain all registered service instance data;
2-2) comparing the current time of the system with the registration time of the service instance, and if the time difference exceeds the set heartbeat interval time, judging that the service instance is not available;
2-3) removing service instances determined to be unavailable from the registry;
2-4) updating a locally maintained list of available service instances;
2-5) the thread sleeps, waiting for the next time to continue to execute the above steps.
Preferably, the step 3 core module initiates creation of a daemon thread that performs mainly the steps of:
3-1) acquiring an IP address of a server to which the core module belongs;
3-2) obtaining the service port number appointed in the configuration file, if no configuration needs to create available random port;
3-3) creating a netty server according to the IP and the port.
Preferably, the step 4 kernel module initiates creation of a daemon thread that performs mainly the following steps:
4-1) judging the current service state;
4-2) if the state is the starting state, calling a registration interface of the management module, and registering the created netty server information to the management module;
4-3) if the state is the ending state, calling a deregistration interface of the management module, and removing the self netty server information in the management module.
Preferably, after receiving the service system cache processing request, the management module in step 6 executes the following steps:
6-1) obtaining netty server information of a core module by obtaining a locally maintained service instance list of an available core module;
6-2) packaging the cache processing parameters according to specific service requirements;
6-3) sending rpc request to the netty server of the core module, and calling the cache operation API of the core module.
Preferably, the core module is integrated by a service system which needs to execute a cache operation in a sdk mode; the core module is required to be packed and then added into a lib catalog of the service system, if maven integration is used, the core module is required to be added into a maven warehouse to provide maven coordinates, and the management module is independently deployed in a web mode and can receive a cache processing request of http or rpc of the service system; and meanwhile, the management module needs to provide an http interface to receive the registration and deregistration requests of the core module.
The invention has the advantages that: the core module ensures the real-time update of the node state and the availability of executing the caching operation through a heartbeat registration mechanism. The management module and the core module avoid the unavailability of the cache processing request caused by the fact that a service system to which the core module belongs bears a large number of http concurrent requests through rpc call. Therefore, the request of the cache processing can be ensured to access all relevant nodes in the distributed system, the cache processing command is executed, and the cache consistency of the distributed system is ensured.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, serve to explain the invention.
FIG. 1 is a schematic diagram of an interaction flow of a software system according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
A method of distributed system cache updating, comprising:
a) Creating a database table by the software system;
b) The software system consists of two independent modules, namely a core module and a management module;
c) Starting a management module, creating a daemon thread, starting to query a database at regular time, acquiring all the core modules which are registered to be available, and removing the core modules which are unavailable due to overtime;
d) Starting a core module, creating a daemon thread, and starting a rpc server by using netty;
e) Starting a core module, creating a daemon thread, reporting heartbeat to a management module at the beginning of timing in an http interface mode, and registering rpc server information to the management module;
f) The service system calls a management module interface to process the cache, the management system obtains all core modules available for registration, and a corresponding rpc server is found through registration information;
g) Requesting to operate an API processing cache built in the core module by sending rpc a request to a registered rpc server;
further, the step of creating the database table by the software system in the step a) includes:
a-1) when all the service systems integrated with the core module are started, registering own service system information into a service model table, wherein the table comprises the following specific contents:
| field name | Field type | Field length | Description of the invention |
| id | Bigint | 20 | Main key |
| name | Varchar | 45 | Business service identification name |
| data_index | Bigint | 20 | Ordering of |
| data_status | Tinyint | 4 | Status of |
| create_time | Timestamps | Creation time | |
| update_time | Timestamps | Update time |
a-2) when all the service systems integrated with the core module are started, the rpc server information started by the service system and the relationship of the service model to which the service system belongs are registered in a service instance table, and the contents of the table are as follows:
further, the relationship between the two modules in step b) is:
b-1) the core module is internally provided with an API for caching operation, and the cache can be managed by calling the API;
b-2) the management module is internally provided with an API for management operation, and the core module can be managed by calling the API;
b-3) integrating the core module by a business system needing to execute caching operation in a sdk mode;
the core module needs to be packed and added into the lib catalogue of the service system, if maven integration is used, the core module needs to be added into a maven warehouse to provide maven coordinates,
b-4) the management module is independently deployed in a web mode, and can receive the cache processing request of the http or rpc of the service system; meanwhile, the management module needs to provide an http interface to receive registration and deregistration requests of the core module;
further, step c) the management module initiates creation of a daemon thread that essentially performs the steps of:
c-1) reading a service instance registry to obtain all registered service instance data;
c-2) comparing the current time of the system with the registration time of the service instance, and if the phase difference time exceeds the set heartbeat interval time, judging that the service instance is not available;
c-3) removing the service instance determined to be unavailable from the registry;
c-4) updating a locally maintained list of available service instances;
c-5) the thread sleeps and waits for the next time to continue to execute the steps.
Further, step d) the core module initiates creation of a daemon thread that essentially performs the steps of:
d-1) acquiring an IP address of a server to which the core module belongs;
d-2) obtaining the service port number appointed in the configuration file, if no configuration needs to create an available random port;
d-3) creating a netty server according to the IP and the port.
Further, step e) the core module initiates creation of a daemon thread that essentially performs the steps of:
e-1) judging the current service state;
e-2) if the state is the starting state, calling a registration interface of the management module, and registering the created netty server information to the management module;
e-3) if the state is the ending state, calling a deregistration interface of the management module, and removing the self netty server information in the management module.
Further, after receiving the service system cache processing request, the management module in step g) executes the following steps:
g-1) obtaining netty server information of a core module by obtaining a locally maintained service instance list of an available core module;
g-2) packaging the cache processing parameters according to specific service requirements;
g-3) sending rpc request to the netty server of the core module, and calling the cache operation API of the core module.
Claims (3)
1. A method for updating cache of a distributed system is characterized in that a software system consists of two independent modules, namely a core module and a management module, wherein an API for cache operation is arranged in the core module, and the cache can be managed by calling the API; the management module is internally provided with an API for management operation, and the core module can be managed by calling the API; the method specifically comprises the following steps:
1) The software system creates a database table comprising a business model table and a service instance table; the method specifically comprises the following steps:
1-1) when all the service systems integrated with the core module are started, registering own service system information into a service model table,
1-2) when all the service systems integrated with the core module are started, the rpc server information started by the service system and the relationship of the service model are registered in a service instance table,
2) Starting a management module, creating a daemon thread, starting to query a database at regular time, acquiring all the core modules which are registered to be available, and removing the core modules which are unavailable due to overtime;
3) Starting a core module, creating a daemon thread, and starting a rpc server by using netty;
4) The core module creates a second daemon thread, starts to report heartbeat to the management module at regular time in an http interface mode, and registers rpc server information to the management module;
5) The service system calls a management module interface to process the cache, the management system obtains all core modules available for registration, and a corresponding rpc server is found through registration information;
6) Requesting to operate an API processing cache built in the core module by sending rpc a request to a registered rpc server;
the step 2 management module starts to create a daemon thread, and the thread mainly executes the following steps:
2-1) reading a service instance registry to obtain all registered service instance data;
2-2) comparing the current time of the system with the registration time of the service instance, and if the time difference exceeds the set heartbeat interval time, judging that the service instance is not available;
2-3) removing service instances determined to be unavailable from the registry;
2-4) updating a locally maintained list of available service instances;
2-5) sleeping the thread, waiting for the next continuous execution of the steps;
step 3, the core module starts to create a daemon thread, and the thread mainly executes the following steps:
3-1) acquiring an IP address of a server to which the core module belongs;
3-2) obtaining the service port number appointed in the configuration file, if no configuration needs to create available random port;
3-3) creating a netty server according to the IP and the port;
step 4, the core module starts to create a daemon thread, and the thread mainly executes the following steps:
4-1) judging the current service state;
4-2) if the state is the starting state, calling a registration interface of the management module, and registering the created netty server information to the management module;
4-3) if the state is the ending state, calling a deregistration interface of the management module, and removing the self netty server information in the management module.
2. The method for updating a distributed system cache as recited in claim 1, wherein after the step 6 management module receives the service system cache processing request, the step of:
6-1) obtaining netty server information of a core module by obtaining a locally maintained service instance list of an available core module;
6-2) packaging the cache processing parameters according to specific service requirements;
6-3) sending rpc request to the netty server of the core module, and calling the cache operation API of the core module.
3. The method for cache updating of a distributed system according to claim 1, wherein the core module is integrated by a service system that needs to perform a cache operation by way of sdk; the core module is required to be packed and then added into a lib catalog of the service system, if maven integration is used, the core module is required to be added into a maven warehouse to provide maven coordinates, and the management module is independently deployed in a web mode and can receive a cache processing request of http or rpc of the service system; and meanwhile, the management module needs to provide an http interface to receive the registration and deregistration requests of the core module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210132987.7A CN114547052B (en) | 2022-02-14 | 2022-02-14 | Method for updating cache of distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210132987.7A CN114547052B (en) | 2022-02-14 | 2022-02-14 | Method for updating cache of distributed system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114547052A CN114547052A (en) | 2022-05-27 |
| CN114547052B true CN114547052B (en) | 2024-03-22 |
Family
ID=81673450
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210132987.7A Active CN114547052B (en) | 2022-02-14 | 2022-02-14 | Method for updating cache of distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114547052B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115563224A (en) * | 2022-11-23 | 2023-01-03 | 北京江融信科技有限公司 | Distributed parameter caching system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968470A (en) * | 2018-09-28 | 2020-04-07 | 江苏赛融科技股份有限公司 | Operation and maintenance monitoring and aggregation management system |
| WO2020237797A1 (en) * | 2019-05-31 | 2020-12-03 | 烽火通信科技股份有限公司 | Dynamic configuration management method and system in microservice framework |
| CN112114979A (en) * | 2020-08-31 | 2020-12-22 | 厦门安胜网络科技有限公司 | Remote procedure call method and device |
| CN112311869A (en) * | 2020-10-28 | 2021-02-02 | 苏州万店掌网络科技有限公司 | Multi-type micro-service registration center management system and method |
| WO2021203968A1 (en) * | 2020-10-28 | 2021-10-14 | 平安科技（深圳）有限公司 | Method for unified management of micro-services of multiple registration centers, and apparatus, device and medium |
-
2022
- 2022-02-14 CN CN202210132987.7A patent/CN114547052B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968470A (en) * | 2018-09-28 | 2020-04-07 | 江苏赛融科技股份有限公司 | Operation and maintenance monitoring and aggregation management system |
| WO2020237797A1 (en) * | 2019-05-31 | 2020-12-03 | 烽火通信科技股份有限公司 | Dynamic configuration management method and system in microservice framework |
| CN112114979A (en) * | 2020-08-31 | 2020-12-22 | 厦门安胜网络科技有限公司 | Remote procedure call method and device |
| CN112311869A (en) * | 2020-10-28 | 2021-02-02 | 苏州万店掌网络科技有限公司 | Multi-type micro-service registration center management system and method |
| WO2021203968A1 (en) * | 2020-10-28 | 2021-10-14 | 平安科技（深圳）有限公司 | Method for unified management of micro-services of multiple registration centers, and apparatus, device and medium |
Non-Patent Citations (2)
| Title |
|---|
| 杨谦 ; .面向分布式系统的高性能缓存管理软件设计及应用.信息技术与信息化.2020,(第05期),全文. * |
| 蒋勇 ; .基于微服务架构的基础设施设计.软件.2016,(第05期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114547052A (en) | 2022-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10296629B2 (en) | Server supporting a consistent client-side cache | |
| CN109783438B (en) | Distributed NFS system based on librados and construction method thereof | |
| CN103390041B (en) | A kind of method and system that data, services is provided based on middleware | |
| CN102904949B (en) | Replica-based dynamic metadata cluster system | |
| CN111881223B (en) | Data management method, device, system and storage medium | |
| WO2016184175A1 (en) | Database processing method and apparatus | |
| CN101877002B (en) | Memory database distributed type access method and system based on unified interface | |
| CN108040079A (en) | Distributed container cluster service discovery method based on domestic CPU and operating system | |
| CN114547052B (en) | Method for updating cache of distributed system | |
| CN102710763A (en) | Method and system for pooling, slicing and fault transfer of distributed cache | |
| CN101562606A (en) | Open type data calling system and method | |
| CN102508881A (en) | Method for clustering multiple nodes of memory database of power information system | |
| CN112650545A (en) | Configuration management system, method and storage medium | |
| US20050144299A1 (en) | System and method for supporting XA 2-phase commit protocols with a loosely coupled clustered database server | |
| CN105956041A (en) | Data model processing method based on Spring Data for MongoDB cluster | |
| US7752225B2 (en) | Replication and mapping mechanism for recreating memory durations | |
| CN113111036A (en) | Small file processing method, device, medium and electronic equipment based on HDFS | |
| CN112416414A (en) | Micro-service architecture containerized lightweight workflow system based on state machine | |
| CN109753245A (en) | A kind of multiple disks load balancing asynchronous read and write dispatching method and device | |
| CN115563224A (en) | Distributed parameter caching system | |
| CN114238518A (en) | Data processing method, device, equipment and storage medium | |
| CN112559138B (en) | Resource scheduling system and method | |
| CN114154825A (en) | Two-dimensional power grid distributed cache service system | |
| CA3127346A1 (en) | Data caching method, device, computer equipment and storage medium | |
| CN113391820A (en) | High-performance micro-service registration calling system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |