CN113590637A

CN113590637A - Method and device for updating multi-process shared database table and database system

Info

Publication number: CN113590637A
Application number: CN202110676255.XA
Authority: CN
Inventors: 刘祥涛
Original assignee: Shenzhen Fandora Information Technology Co ltd
Current assignee: Shenzhen Fandora Information Technology Co ltd
Priority date: 2021-06-21
Filing date: 2021-06-21
Publication date: 2021-11-02

Abstract

The application discloses a method and a device for updating a multi-process shared database table and a database system, which can effectively improve the updating speed of the database table and ensure the availability and consistency of shared data under the condition of multi-process shared data, and are applied to the device and the database system for updating the multi-process shared database table, wherein the method comprises the following steps: loading data of the process when the process is started; monitoring whether an abnormal event occurs in the process running process; receiving a calling instruction of data updating operation in the process through a data updating interface; performing data updating operation according to the calling instruction to obtain data to be updated; judging whether the data to be updated meets the condition of writing into the system database, and limiting the updating operation times of writing into the system database through the condition so as to enable more updating operations to occur on the process cache; if not, writing the data to be updated into a process cache, wherein the process cache is a distributed cache; and if so, writing the data to be updated into a table corresponding to the system database.

Description

Method and device for updating multi-process shared database table and database system

Technical Field

The application relates to the technical field of databases, in particular to a method and a device for updating a multi-process shared database table and a database system.

Background

A database is a "warehouse that organizes, stores, and manages data according to a data structure," which is an organized, sharable, uniformly managed collection of large amounts of data that is stored in a computer for a long period of time. The database management system can perform operations of adding, deleting, modifying and searching stored data, wherein the modification operation comprises changing a plurality of records at a time and changing one record at a time, aiming at the operation of changing a plurality of records at a time, the cache provided by the traditional database table index, storage process and database engine can realize positioning quick modification or one-time modification after caching, but for the operation of changing one record at a time of a large amount of converged single, especially for shared data (centralized modification base table) facing a multi-process high-concurrency system (for example, record modification bearing a high-concurrency cash transfer table of a Unionpay system), the distance and modification heterogeneity can bring great delay to the modification operation of the database table; in the worst case, aggregating N class 2 operations, such as the database server being too far away from the application server (e.g., >1000KM), requiring different databases to be connected to modify different fields of different tables (modification heterogeneity is too high), thereby incurring significant modification delay.

The current speed-up technology aiming at modifying a database table mainly comprises the following steps: database table index acceleration, storage process acceleration, cache acceleration provided by a database engine, and the like.

1. Database table index acceleration: an index is a structure that sorts one or more columns of values in a database table, and the index can be used to quickly access specific information in the database table. The time complexity of locating database records can be reduced from O (N) to O (log) typically by building a binary tree index over the commonly used query fields of the database tables₂N) to achieve database table modification acceleration.

2. The storage process is accelerated: a Stored Procedure (Stored Procedure) is a set of SQL statements in a large database system for performing a specific function, which is Stored in the database and is permanently valid after one compilation, and is executed by a user by specifying the name of the Stored Procedure and giving parameters (if the Stored Procedure carries parameters). Stored procedures are an important object in databases. The efficiency improvement of speed multiplication can be achieved by using the storage process under the condition of extremely large data volume. But the storage process has the following bottlenecks:

a, generally only for structured databases, such as MySQL, Oracle, SQL Server, etc.;

and 2, specific storage processes need to be written according to different database tables and service requirements, customization is strong, and universality is not achieved.

3. Based on multi-level cache acceleration: an acceleration mechanism applying Amdahl's Law is widely used in computer systems, such as system multi-level storage acceleration, database write acceleration and the like, typical memory cache systems for database write acceleration include Memcached, Redis, Etcd and the like, and generally, data needing to be modified are written into the memory cache systems firstly, and then the data are summarized and written into the memory cache systems; however, a process may crash and multiple processes may share data, in which case the multi-level caching mechanism may not maintain continuous availability and consistency of the data.

According to the CAP theorem, Consistency, Availability, and Partition tolerance are incompatible in a distributed system. Consistency (C) refers to whether all data backups in a distributed system have the same value at the same time. (equivalent to all nodes accessing the same up-to-date copy of the data); availability (a) refers to ensuring that each request responds regardless of success or failure; partition tolerance (P) means that the loss or failure of any information in the system does not affect the continued operation of the system. The essence of the CAP theorem is either AP, CP, or AC, but no CAP is present.

The multi-process system is a typical distributed system, under the condition of sharing data, if 1 unified database is used for storing the shared data of multiple processes, the condition of data inconsistency cannot occur, so that the C and P of the distributed system are ensured, but if the system has a network partition condition or is down, certain data cannot be accessed, at the moment, the availability condition cannot be met, namely, a CP system is obtained under the condition, but CAPs cannot be met simultaneously, and meanwhile, because 1 database is shared, when data updating is carried out, the table needs to be locked and unlocked, and the data updating sequence is ensured to be consistent with the service time sequence, so that the data updating speed is greatly reduced.

In summary, regarding database table updating, when the prior art is oriented to multi-process data sharing, the database table updating efficiency cannot be effectively improved.

Disclosure of Invention

The application provides a method and a device for updating a multi-process shared database table and a database system, which are used for solving the defects in the updating operation of the multi-process shared data-oriented database table, effectively improving the updating speed of the database table and ensuring the availability and consistency of shared data.

The first aspect of the present application provides a method for updating a multi-process shared database table, where the method is applied to a process subsystem, the process subsystem is in communication connection with a system database, the system database is connected with at least one process subsystem, and at least one process runs on the process subsystem, and the method includes:

loading data of the process when the process is started;

monitoring whether an abnormal event occurs in the process running process;

receiving a calling instruction of data updating operation in the process through a data updating interface;

performing data updating operation according to the calling instruction to obtain data to be updated;

judging whether the data to be updated meets the condition of writing into the system database, and limiting the updating operation times of writing into the system database through the condition so as to enable more updating operations to occur on the process cache;

if not, writing the data to be updated into a process cache, wherein the process cache is a distributed cache;

and if so, writing the data to be updated into a table corresponding to the system database.

Optionally, the determining whether the data to be updated meets the condition of writing into the system database includes:

updating a writing library threshold value at regular time according to preset time, wherein the writing library threshold value is determined by prior knowledge of data updating operation in the past preset time and a preset sliding window;

and judging whether the data to be updated is larger than the writing library threshold value.

Optionally, loading the data of the process when the process is started includes:

starting the process, and loading a program of the process into a memory of a computer;

inquiring whether corresponding cache data exist in the process cache according to the process identifier;

and if so, loading the cache data to the memory of the process.

Optionally, after monitoring whether an abnormal event occurs during the process running process, the method further includes:

and if so, writing the data of the process into the process cache within preset exit time.

Optionally, after writing the data to be updated into the table corresponding to the system database, the method further includes:

and clearing the data called by the calling instruction from the process cache.

A second aspect of the present application provides a data reading method, where the method is applied to an upper layer application of a process subsystem, and the process subsystem is in communication connection with a system database, and includes:

receiving a request for reading target data;

reading cache data corresponding to the target data from a process cache according to the identifier of the target data;

reading table data corresponding to the target data from a table corresponding to the system database according to the identification of the target data;

and adding the cache data and the table data to obtain target data.

A third aspect of the present application provides a multi-process shared database table updating apparatus, including:

the data loading unit is used for loading the data of the process when the process is started;

the abnormal event unit is used for monitoring whether an abnormal event occurs in the process running process, wherein the abnormal event comprises process abnormality and system abnormality;

the first receiving unit is used for receiving a calling instruction of data updating operation in the process through a data updating interface;

the updating unit is used for obtaining data to be updated after data updating operation is carried out according to the calling instruction;

the first judgment unit is used for judging whether the data to be updated meets the condition of writing into the system database or not, and limiting the updating operation times of writing into the system database through the condition so as to enable more updating operations to occur on the process cache;

the first execution unit is used for writing the data to be updated into a process cache when the first judgment unit determines that the data to be updated does not meet a database writing threshold value written into the system database, wherein the process cache is a distributed cache;

and the second execution unit is used for writing the data to be updated into a table corresponding to the system database when the first judgment unit determines that the data to be updated meets a database writing threshold written into the system database.

Optionally, the first determining unit further includes:

the determining module is used for updating a writing library threshold at regular time according to preset time, and the writing library threshold is determined by prior knowledge of data updating operation in the past preset time and a preset sliding window;

and the first judgment module is used for judging whether the data to be updated is larger than the writing library threshold value.

Optionally, the data loading unit includes:

the starting module is used for starting the process and loading the program of the process into the memory of the computer;

the second judgment module is used for inquiring whether corresponding cache data exist in the process cache according to the process identifier;

and the loading module is used for loading the cache data into the memory of the process when the second judging module inquires corresponding cache data from the process cache according to the identification of the process.

Optionally, the apparatus further comprises:

and the third execution unit is used for writing the data of the process into the process cache when the abnormal event sending unit monitors that the abnormal event is sent.

Optionally, the apparatus further comprises:

and the clearing unit is used for clearing the data called by the calling instruction in the process cache.

A fourth aspect of the present application provides a data reading apparatus, comprising:

a receiving unit configured to receive a request to read target data;

the first acquisition unit is used for reading cache data corresponding to the target data from a process cache according to the identifier of the target data;

the second acquisition unit is used for reading the table data corresponding to the target data from the table corresponding to the system database according to the identification of the target data;

and the execution unit is used for adding the cache data and the table data to obtain target data.

A fifth aspect of the present application provides a database system, including a system database and at least one process subsystem, the system database is in communication connection with the process subsystem, wherein the process subsystem includes a data processing unit and a process cache unit, wherein the data processing unit includes the above database table updating apparatus, including:

the system database is used for storing the data shared by the process subsystems;

the process cache unit is used for caching and storing the data operated by the process subsystem, and adopts distributed cache;

and the data processing unit is used for processing data update and abnormal events of the process running on the process subsystem.

A sixth aspect of the present application provides a device for updating a multi-process shared database table, including:

the device comprises a processor, a memory, an input and output unit and a bus;

the processor is connected with the memory, the input and output unit and the bus;

the processor specifically performs the following operations:

loading data of the process when the process is started;

monitoring whether an abnormal event occurs in the process running process;

A seventh aspect of embodiments of the present application provides a computer-readable storage medium, having a program stored thereon, where the program, when executed on a computer, performs the aforementioned database table updating method.

According to the technical scheme, the distributed computing principle is adopted in the multi-process shared database table updating method, when the process subsystem executes updating operation, most data can be written into the cache, and if the data are written into the system database, too much time is occupied; if not, writing the data into the process cache, and limiting the updating operation times written into the system database through the condition so as to enable more updating operations to occur on the process cache, thereby reducing the occupation of the updating operations on the system database and greatly improving the updating speed of the system database under the condition of sharing data by multiple processes.

Drawings

FIG. 1 is a schematic block diagram of an embodiment of a database system according to the present application;

FIG. 2 is a schematic structural diagram of an embodiment of a core component of a process subsystem in the present application;

FIG. 3 is a flowchart illustrating an embodiment of a method for updating a multi-process shared database table according to the present application;

FIG. 4 is a flow diagram illustrating an embodiment of data loading during process startup according to the present application;

FIG. 5 is a flow diagram illustrating an embodiment of data protection during a process exception according to the present application;

FIG. 6 is a flowchart illustrating a method for updating a multi-process shared database table according to another embodiment of the present application;

FIG. 7 is a flowchart illustrating an embodiment of a data reading method in an upper application of the present application;

FIG. 8 is a schematic structural diagram of another embodiment of a database system according to the present application;

FIG. 9 is a flowchart illustrating a method for updating a multi-process shared database table according to another embodiment of the present application;

FIG. 10 is a flow chart illustrating a data reading method in an upper layer application according to another embodiment of the present application;

FIG. 11 is a schematic structural diagram illustrating an embodiment of an apparatus for updating a multi-process shared database table according to the present application;

FIG. 12 is a schematic structural diagram of an embodiment of a data reading apparatus according to the present application;

FIG. 13 is a schematic structural diagram of an apparatus for updating a multi-process shared database table according to another embodiment of the present application.

Detailed Description

The application provides a database table updating method and device and a database system, which can make up for the defects in the updating operation of a database table oriented to multi-process shared data, effectively improve the updating speed of the system database and keep the usability and consistency of data.

The technical solutions in the present application will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

To facilitate understanding of the present application, a database system in the present application will be described first. Referring to fig. 1, an embodiment of a database system of the present application includes:

the system comprises a system database 1 and at least one process subsystem 2, wherein the system database 1 is in communication connection with the process subsystem 2, and the process subsystem 2 comprises a data processing unit 21 and a process cache unit 22; the system database 1 is used for storing the data shared by the process subsystems 2; the process cache unit 22 is configured to perform cache storage on the data operated by the process subsystem, and the process cache unit employs distributed cache; the data processing unit 21 is configured to process data update and exception events of a process running on the process subsystem.

In this embodiment of the application, communication between the system database and each process subsystem may be implemented by a local area network, a wide area network, or other communication protocols, which are not specifically limited herein, and a distributed computing network structure is formed between the system database and each process subsystem, so that each process subsystem can access the system database in parallel, and a specific embodiment of the process subsystem may refer to fig. 2, which is not specifically described herein. When a process running on the process subsystem needs to perform data updating operation, the data is not directly written into the system database, whether the updated part of the data meets the condition of writing into the system database is judged, if the updated part of the data meets the condition, the updated part of the data is written into the system database, otherwise, the updated part of the data is written into a process cache unit unique to the process subsystem, so that the occupation of the process on the system database is reduced, and the updating efficiency of the system database is improved.

Referring to fig. 2, fig. 2 is a schematic structural diagram of an embodiment of a core component of a process subsystem 2 in the present application, configured to implement a database table updating method provided in the present application, including:

the data processing unit 21 includes:

a cache data loading module 211, which loads the cache data of the process when the process is started, and loads the data from the process cache unit 22 when an abnormal event occurs, such as a system crash or a process error exits and then the system/process is started again;

the data updating module 212 is configured to provide a data updating interface for the upper layer data updating operation, issue the data updating operation, add the part of data to the corresponding data in the process cache to obtain data to be updated, and thus invoke and shield the bottom layer operation for the upper layer data updating;

a judging module 213, configured to judge whether the data to be updated meets a condition for updating to the system database 1;

the execution module 214 is configured to execute a data writing operation, and when the determination module determines that the data to be updated needs to be updated to the system database 1, write the data to be updated into the system database 1, and clear the updated part of the data in the process cache unit 22; otherwise, updating the data to be updated to the process cache unit 22;

the exception handling module 215 may cause a process exception (which may cause a process crash) because the cache data loading module 211, the data updating module 212, the determining module 213, and the executing module 214 may cause an exception, or even the entire process may cause an exception; or the process subsystem may cause system abnormality (possibly causing system down) due to power failure or other reasons; under the condition of system exception, the system has short exit time to protect the process running site, the exception handling module 215 writes the memory data to be updated and written into the library into the process cache unit 22 before the exception and/or downtime exit, so as to ensure that the last cache data can be loaded from the process cache unit 22 by the cache data loading module 211 after the process is restarted, thereby recovering the last running site of the process;

the process cache unit 22 is configured to buffer and store data of the process, may employ a distributed memory cache system to support high-speed reading and writing of data, and is deployed in the same machine room as much as possible as the data processing unit 21, so as to reduce network delay.

In order to more clearly illustrate the embodiments of the present application, the method performed in the database system and the process subsystem will be described below.

Referring to fig. 3, an embodiment of a method for updating a multi-process shared database table in the present application includes:

301. receiving a calling instruction of data updating operation in the process through a data updating interface;

in the running process of the process, the upper layer application issues a corresponding operation instruction to update the system database, and a data updating interface of the process receives data updating operation call of the upper layer data updating operation, so that the corresponding data updating operation is executed.

302. Performing data updating operation according to the calling instruction to obtain data to be updated;

303. judging whether the data to be updated meets the condition of writing into the system database, if not, executing a step 304, and if so, executing a step 305;

in the embodiment of the present application, the condition of writing into the system database is that the data to be updated is greater than the write library threshold, and other determination conditions may be set in practical application according to corresponding requirements, for example, whether the data to be updated is smaller than the write library threshold or is within a preset interval or not, and specific details are not limited herein.

304. Writing the data to be updated into a process cache;

and writing the data to be updated corresponding to the key to be updated into the process cache so as to reduce the occupation of the update operation of the current process on the system database.

305. Writing the data to be updated into a table corresponding to the system database;

and when the data to be updated meets the condition of writing into the system database, writing the data to be updated corresponding to the key to be updated into the system database table.

For example, a user table is stored in the system database, the user table has 1 integral field (type is floating point number, for example, value 102.31), a certain number of integral values are added for each update, a library writing threshold value (for example, value 1000) is set to limit the update operating condition of the system database, and the data update interface of the process receives an operation call of adding 50 integrals to the integral value corresponding to key three. Acquiring the accumulated integral value of the multiple caches corresponding to Zhang III from the process cache to be 960, and executing corresponding data updating operation to obtain data to be updated (960+50 equals to 1100), and executing step 305 to write the data with key Zhang III into the integral field of the user table of the system database because 1100 is greater than 1000.

306. And clearing the data called by the calling instruction from the process cache.

And clearing the data written into the system database from the process cache.

If the system database is directly updated, the searching and writing actions of the N records need to be executed on the system database, and the operation is relatively time-consuming.

Formalized deduction is carried out on the database updating speed improved by the application as follows:

the data update speed of the write system database in the prior art, i.e., the number of read/write Operations Per Second (IOPS) is assumed to be V₁After the optimization mode of writing the process cache in the application is adopted, the speed of updating the data to the process cache is V₂If the data with the proportion of P needs to be updated to the process cache, the proportion of updating to the database is (1-P), and the optimized acceleration ratio is as follows:

S_p＝[V₁*(1-P)+V₂*P]/V₁

wherein, P is 0.9 and V₁2000IOPS (traditional databases include MySQL or MongoDB), V₂100000IOPS (memory cache database including Redis, etc.), then S_p＝[2000*0.1+100000*0.9]45.1, i.e. the speed after optimization is 45.1 times the speed before optimization.

Referring to fig. 4, an embodiment of data loading when starting a process in the present application includes:

401. starting a process, and loading a program of the process into a memory of a computer;

402. inquiring whether corresponding cache data exist in the process cache according to the process identifier, if so, executing step 403, otherwise, ending the data loading process, and continuing to execute other tasks of the process;

403. and loading the cache data to the memory of the process.

In this embodiment of the application, the started process may be a process that is restarted after being interrupted and exited due to an exception, and when exiting, because the process subsystem is provided with an exception protection mechanism (see fig. 5, which is not described herein in detail), the running data of the process is stored in the cache unit, so that when starting the process, whether corresponding cache data exists in the process cache needs to be queried, and if so, the cache data is loaded to the memory of the process and then other tasks of the process are continuously executed, so that the process can directly recover the data site when exiting last time, and thus, the continuous availability of the process is ensured.

Referring to fig. 5, an embodiment of data protection for process exceptions in the present application includes:

501. monitoring whether an abnormal event occurs in the process during the running, wherein the abnormal event comprises process abnormality or system abnormality, if so, executing step 502, and if not, executing corresponding steps along with the process until the process is finished;

abnormal events may occur in the process running process, the abnormal events include process abnormality and system abnormality, the process abnormality degree is low, and abnormal exit (for example, except for 0 operation) caused by abnormal operation of the process can be generally captured by a statement block adapted to abnormal processing, so as to perform short-time processing action; if the system exception occurs (for example, power off), the system exception is generally captured by the operating system and is transmitted to the corresponding system process and application process, and the application process can also perform corresponding short-time processing action on the system exception.

502. And writing the memory data of the process into the process cache within preset exit time.

And writing the memory data of the process into a process cache within a short exit time, for example, in Golang, a field protection program capable of adapting to system abnormality can be written in a defer () anonymous function or in a catch sub statement block, and the memory data is quickly written into the process cache, so that the field is protected.

When a process fault or a system fault occurs, the memory data of the process is written into the process cache before the process exits, so that the process data is reserved, and the process data is ensured not to be lost.

Referring to fig. 6, another embodiment of a database table updating method in the present application includes:

601. starting a process, and loading a program of the process into a memory of a computer;

602. inquiring whether corresponding cache data exist in the process cache according to the process identifier, if so, executing step 603, and if not, executing step 604;

603. loading the cache data to a memory of the process;

in the embodiment of the present application, steps 601 to 603 are similar to steps 401 to 403 in the previous embodiment, and are not described herein again.

604. Receiving a calling instruction of data updating operation in the process through a data updating interface;

605. performing data updating operation according to the calling instruction to obtain data to be updated;

in the embodiment of the present application, steps 604 to 605 are similar to steps 301 to 302 of the previous embodiment, and are not described herein again.

606. Updating a writing library threshold value at regular time according to preset time, wherein the writing library threshold value is determined by prior knowledge of data updating operation in the past preset time and a preset sliding window;

in the embodiment of the present application, the determination condition for writing in the database/writing in the local cache may be determined by a priori knowledge and a Sliding Window Algorithm (Sliding Window Algorithm) in a past period of time, for example, more than 9 times of operations of updating the database to the current process cache (the part is obtained according to the history, and the current time is not limited); the average value N (for example, N9) of the fields that need to be updated in the last period of time in the past may be taken as the judgment condition for writing to the library/local cache.

For example, there is a user table, in which there are 1 point field, and if the average value of the point increases or decreases of 100 ten thousand points accumulated in the past 10 days is 100, it can be determined whether it exceeds 900 as the determination condition for writing data in the database/writing local cache, so that it can be ensured that more than 9 times of operations of updating to the database 10 times are written to the process cache; in order to ensure the adaptability of the prior change along with the time, the prior window can slide along with the current time, namely, the average value of plus and minus integrals accumulated within 10 days after the current time can be recalculated every 1 day, so that the value of the average value is continuously refreshed, and the high probability of the updating operation can be ensured to fall into the process cache.

607. Judging whether the data to be updated is larger than the library writing threshold value, if not, executing a step 608, and if so, executing a step 609;

608. if not, writing the data to be updated into a process cache;

609. if yes, writing the data to be updated into a table corresponding to the system database;

610. and clearing the data called by the calling instruction in the process cache.

In the embodiment of the present application, steps 608 to 610 are similar to steps 304 to 306 in the previous embodiment, and are not described herein again.

It should be noted that, in the running process of the processes, the embodiment of the present application still needs to monitor whether an abnormal event occurs in the system in real time, and if an abnormal event occurs, exception handling is executed to ensure data availability of the processes; the reduce is completed by using inc (increase) or dec (decrease) operations (instead of set operations, since set operations can flush previous data) which reserve the last operation on a system database through each process (no independent 1 reduce process is needed), so that consistency aggregation (namely reduce, reduction) of data of each process is completed, and the application has the effect of being suitable for data consistency of multiple processes.

The result of executing the above-described write library operation based on the prior art is: 1-core CPU/2GB memory machine writes 1 ten thousand records, directly writes a remote database, and needs 55 s; the results obtained by optimizing the embodiments of the present application are: 3ms to 2s are needed (3ms is the best case when the process cache is written all the time, 2s is the average running case), and the writing speed is improved by 27.5 to 18333.3 times.

Referring to fig. 7, an embodiment of a data reading method according to the present application is applied to a process subsystem upper layer application, and includes:

701. receiving a request for reading target data;

702. reading cache data corresponding to the target data from a process cache according to the identifier of the target data;

703. reading table data corresponding to the target data from a table corresponding to the system database according to the identification of the target data;

704. and adding the cache data and the table data to obtain target data.

In the embodiment of the application, the upper layer application sends the call of the data updating operation to the bottom layer module through the data updating interface according to the running condition of the process, and shields the upper layer application with the bottom layer operation.

For further understanding of the application of the database table updating method in the present application, the following description is given by way of example of the database system deployment and the application of the database table updating method.

Referring to fig. 8, an embodiment of a database system deployment in the present application:

the database system is composed of a plurality of process subsystems and a system database, wherein the process subsystem 801 is composed of a process server 8011 and a process cache server 8012, the system database is a system database server 802, wherein:

the process server 8011 is configured to deploy modules for cache data loading, data updating, determining and executing, exception handling, and the like, is in charge of cache data loading, data updating, determining and executing, exception handling, and the like, and is in communication with the process cache server 8012 and the system database server 802, and when it is determined that the data of the part does not need to be updated to the system database server 802, the data is directly updated to the process cache server 8012; when it is determined that the update to the system database server 802 is required, the data is accumulated and then updated to the system database server 802, and the data is removed from the process cache server 8012. It is noted that the multiple process servers 8011 support distributed cluster deployment of 1 program/multiple programs, and may support multiple process servers 8011 to do 1 event simultaneously or respectively do different events, thereby implementing flexible support for a high-concurrency and high-availability system.

The process cache server 8012 is configured to deploy a process cache module and is responsible for data caching of each process subsystem; generally, mature open source distributed memory cache database systems, such as Redis/Memcached/Etcd and the like, are adopted, and are not limited herein specifically, the distributed systems can obtain more than 10 ten thousand IOPS performances by adopting a memory acceleration technology; the system can be configured to be in a synchronous/asynchronous disk refreshing mode, so that data can be ensured not to be lost when the process cache server 8012 is powered off and other exceptions occur; multiple servers can also be used to carry process cache to eliminate single point of failure.

The system database server 802 is configured to deploy data shared by multiple process subsystems, including service data, user data, log data, and the like, and may adopt MySQL, SQLServer, Oracle, MongoDB, and the like, so as to ensure final consistency of the data.

Referring to fig. 9, based on the foregoing embodiment, a Redis cluster (3 servers) is adopted as the process cache server, a MongoDB is adopted as the system database, and three copy clusters may be adopted as the system database, where an embodiment of the database table updating method in this application includes:

901. setting a threshold value of the accumulated integral value;

setting the threshold value of the accumulated integral value through configuration parameters, assuming that Cmax is 1000 here, that is, when the integral value of a certain user + the integral value of the user cached by the process is 1000 at this time, the operation of writing the system database is triggered.

902. Acquiring an integral value of a user Ui needing to be updated;

and acquiring the integral value Cj of the user UI needing to be updated at this time through the data updating interface.

903. Acquiring a current cache integral value Ci of a user Ui from Redis;

a value of key ═ Ui is read from the Redis, set to Ci, and if a value corresponding to key ═ Ui is not obtained, Ci ═ 0.

904. Calculating the cumulative integral value of the user Ui: ct ═ Cj + Ci;

905. judging whether the accumulated integral value is larger than a threshold value, if so, executing a step 906, and if not, executing a step 908;

and judging whether Ct is larger than Cmax, if so, executing step 906 to write the accumulated integral value into the system database, and if not, executing step 908 to write the accumulated integral value into the process cache.

906. The record of the corresponding user Ui on the user table of the positioning system database MongoDB and the Inc (addition) operation is executed on the integral field;

907. clearing Value records of key ═ Ui from the process cache;

908. refreshing the value of key ═ Ui in Redis to Ct;

909. and judging whether the data needs to be updated or not, if so, executing the step 902, and if not, ending the execution flow of data updating.

Because the application data of the upper layer and the operation of the bottom layer are shielded, and the integral value updating mechanism in the embodiment of the application updates the user integral value of the system database when the integral value cached by the process reaches the threshold value, when the upper layer application needs to read the current integral value of a certain user, the integral value cached by the process may not reach the threshold value and is not updated to the system database in time. Therefore, when the upper layer application reads the current integral value of the user, the upper layer application needs to read the integral values corresponding to the user in the process cache and the system database respectively, and the integral values are the current integral value of the user after addition.

Referring to fig. 10, in the embodiment of the present application, a method for updating a database table of an upper application corresponds to a method for reading data of the database table, including:

1001. receiving an integral value reading request of a user Ui;

1002. acquiring a current cache integral value Ci of a user Ui from Redis;

in the embodiment of the present application, it is assumed that the current cache integration value Ci is acquired as 50.

1003. Acquiring an integral value Cd corresponding to a user Ui from a user table of a system database MongoDB;

suppose that the integral value Cd corresponding to the user Ui is 1688 obtained in the user table.

1004. Calculating the cumulative integral value of the user Ui: cr ═ Ci + Cd.

And calculating the current integral value of the user according to the sum of the sums of the sum of the sums of the sum of the.

It should be noted that the data reading method described in the embodiment of the present application is a case where a single process subsystem accumulates corresponding field data of a system database, and if a plurality of process subsystems have the field data cache, corresponding field data of the plurality of process subsystems need to be superimposed, which is not described herein again.

Referring to fig. 11, an embodiment of the apparatus for updating a multi-process shared database table in the present application includes:

a data loading unit 1101, configured to load data of the process when the process is started;

an exception unit 1102, configured to monitor whether an exception event occurs in the process running process, where the exception event includes a process exception and a system exception;

a first receiving unit 1103, configured to receive, through a data update interface, a call instruction of a data update operation in the process;

the updating unit 1104 is configured to perform data updating operation according to the call instruction to obtain data to be updated;

a first determining unit 1105, configured to determine whether the data to be updated meets a condition for writing into the system database, and limit, according to the condition, the number of update operations written into the system database, so that more update operations occur on the process cache;

a first executing unit 1106, configured to, when the first determining unit determines that the data to be updated does not meet a library writing threshold written in the system database, write the data to be updated into a process cache, where the process cache is a distributed cache;

a second executing unit 1107, configured to write the data to be updated into a table corresponding to the system database when the first determining unit determines that the data to be updated satisfies a library writing threshold written into the system database.

The third execution unit 1108 is configured to, when the exception event unit monitors that an exception event is sent, write the data of the process into the process cache.

A clearing unit 1109, configured to clear, in the process cache, data called by the call instruction.

In this embodiment, the first determining unit 1105 includes:

a determining module 11051, configured to update a write library threshold at regular time according to a preset time, where the write library threshold is determined by a priori knowledge of data update operation in a past preset time and a preset sliding window;

the first determining module 11052 determines whether the data to be updated is greater than the write bank threshold.

In this embodiment, the data loading unit 1101 includes:

a starting module 11011, configured to start the process, and load a program of the process into a memory of a computer;

a second determining module 11012, configured to query, according to the identifier of the process, whether there is corresponding cache data in the process cache;

a loading module 11013, configured to load, when the second determining module queries that there is corresponding cache data from the process cache according to the identifier of the process, the cache data to the memory of the process.

Referring to fig. 12, an embodiment of the database table updating apparatus for reading data by an upper application in the present application includes:

a receiving unit 1201 for receiving a request to read target data;

a first obtaining unit 1202, configured to read, according to an identifier of the target data, cache data corresponding to the target data from a process cache;

a second obtaining unit 1203, configured to read, according to the identifier of the target data, table data corresponding to the target data from a table corresponding to the system database;

an execution unit 1204, configured to add the cache data and the table data to obtain target data.

Referring to FIG. 13, another embodiment of the database table updating apparatus of the present application includes:

a processor 1301, a memory 1302, an input-output unit 1303, a bus 1304;

the processor 1301 is connected to the memory 1302, the input/output unit 1303 and the bus 1304;

the processor 1301 specifically performs the following operations:

judging whether the data to be updated meets the condition of writing into the system database;

if not, writing the data to be updated into a process cache;

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like.

Claims

1. A multi-process shared database table updating method is applied to a process subsystem, the process subsystem is in communication connection with a system database, the system database is connected with at least one process subsystem, and at least one process runs on the process subsystem, and the method comprises the following steps:

loading data of the process when the process is started;

monitoring whether an abnormal event occurs in the process running process;

2. The method for updating the multi-process shared database table according to claim 1, wherein judging whether the data to be updated meets the condition of writing into the system database comprises:

3. The method for updating the multi-process shared database table according to claim 1, wherein loading the data of the process when the process is started comprises:

and if so, loading the cache data to the memory of the process.

4. The method for updating the multi-process shared database table according to claim 1, wherein after monitoring whether an exception event occurs during the process running process, the method further comprises:

5. The multi-process shared database table updating method according to any one of claims 1 to 4, wherein after writing the data to be updated into the corresponding table of the system database, the method further comprises:

and clearing the data called by the calling instruction from the process cache.

6. A method for reading data, the method being applied to an upper layer application of a process subsystem, the process subsystem being communicatively connected to a system database, the method comprising:

receiving a request for reading target data;

and adding the cache data and the table data to obtain target data.

7. An apparatus for updating a multi-process shared database table, comprising:

8. The apparatus for updating the table of the multi-process shared database according to claim 7, wherein the first determining unit further comprises:

9. The apparatus for updating the table of a multi-process shared database according to claim 7, wherein the data loading unit comprises:

10. The apparatus for updating a multi-process shared database table according to claim 7, further comprising:

11. The apparatus for updating a multi-process shared database table according to any of claims 7 to 10, further comprising:

12. A data reading apparatus, comprising:

a receiving unit that receives a request to read target data;

the second acquisition unit is used for reading table data corresponding to the target data from a table corresponding to the system database according to the identification of the target data;

13. A database system comprising a system database and at least one process subsystem, said system database being communicatively connected to said process subsystem, wherein said process subsystem comprises a data processing unit and a process caching unit, wherein said data processing unit comprises a database table updating apparatus as claimed in any one of claims 7 to 12, wherein: