CN107608662B - MongoDB-based distributed timing system - Google Patents
MongoDB-based distributed timing system Download PDFInfo
- Publication number
- CN107608662B CN107608662B CN201710827425.3A CN201710827425A CN107608662B CN 107608662 B CN107608662 B CN 107608662B CN 201710827425 A CN201710827425 A CN 201710827425A CN 107608662 B CN107608662 B CN 107608662B
- Authority
- CN
- China
- Prior art keywords
- timing task
- timing
- task
- thread
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of system timing, and discloses a MongoDB-based distributed timing system which improves the reliability and concurrency of timing, is easy to maintain and can be configured. The system comprises a MongoDB memory database, an Executors thread pool, a timing task creating module, a timing task executing module, a timing task canceling module and a timing task initializing module; the MongoDB memory database maintains a timing task table which is used for persistently storing timing tasks and synchronizing the timing tasks with the timing tasks in the memory in real time, and when the service is down and restarted, the persistently stored timing tasks are screened and then added into the memory timing tasks again; the timing task creating module, the timing task executing module, the timing task canceling module and the timing task initializing module are respectively used for completing a timing task creating process, a timing task executing process, a timing task canceling process and a timing task initializing process. The invention is suitable for financial cloud platform products.
Description
Technical Field
The invention relates to the field of system timing, in particular to a MongoDB-based distributed timing system.
Background
MongoDB is a database based on distributed file storage. Written in the C + + language. It is intended to provide an extensible high performance data storage solution for WEB applications. The method has the characteristics of high performance, easiness in deployment, easiness in use and the like.
The main functional characteristics are:
(1) the data of the object type is easy to store by facing to the set storage;
(2) mode free;
(3) supporting dynamic inquiry;
(4) supporting full indexing, including internal objects;
(5) supporting the query;
(6) support replication and failover;
(7) using efficient binary data storage, including large objects (e.g., video, etc.);
(8) automatically processing fragments to support the expansibility of a cloud computing hierarchy;
(9) supporting various languages such as RUBY, PYTHON, JAVA, C + +, PHP and the like;
(10) the file storage format is BSON (an extension of JSON);
(11) accessible through a network;
by "collection-Oriented", it is meant that data is stored in a data set in groups, called a collection (collection). Each collection has a unique identification name in the database and may contain an unlimited number of documents. The concept of a collection is similar to a table in a Relational Database (RDBMS), except that it does not need to define any schema.
Schema-free, meaning that we do not need to know any of its structural definitions for files stored in the mongodb database. You can store files of different structures in the same database at all, if necessary.
Documents stored in the collection are stored in the form of key-value pairs. Keys are used to uniquely identify a document, and are of the string type, while values can be of various complex file types. We call this form of storage bson (binaryjson).
JDK (Java Development kit) is a product developed by Sun Microsystems for Java, which provides developers with a variety of basic capabilities. The scheduledExecutionService develops a timer function base class for Java, which provides a number of timer dependent execution methods.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: a MongoDB-based distributed timing system is provided, which improves the reliability and concurrency of timing, is easy to maintain and can be configured.
In order to solve the problems, the invention adopts the technical scheme that: the MongoDB-based distributed timing system comprises a MongoDB memory database, an Executors thread pool, a timing task creating module, a timing task executing module, a timing task canceling module and an initialization timing task module;
the MongoDB memory database maintains a timing task table which is used for persistently storing timing tasks and synchronizing the timing tasks with the timing tasks in the memory in real time, and when the service is down and restarted, the persistently stored timing tasks are screened and then added into the memory timing tasks again;
the Executors thread pool is used for managing timing threads;
the timing task creating module, the timing task executing module, the timing task canceling module and the timing task initializing module are respectively used for completing a timing task creating process, a timing task executing process, a timing task canceling process and a timing task initializing process.
Further, the timing task table includes the following fields: unique primary key, data ID, data type, application code, timeout time, creation time, valid flag.
Furthermore, the timing task execution module can accurately position a single timing task by using the codes, the data ID and the data type as a joint main key.
Further, the timing task execution module obtains timing time by comparing the current time of the system with the creation time.
Further, the creating of the timing task flow includes:
a1. inquiring timing task records from the timing task table according to the data ID and the data type, and if the timing task records do not exist, directly entering the step a 4; if the timing task record exists, judging whether the timing task is the timing task of the local computer, if so, entering a step a2, otherwise, setting the timing task as invalid, and entering a step a 4;
a2. judging whether a timing task thread exists in the memory, if so, stopping the timing task thread, clearing the timing task thread in storage, and entering step a3, otherwise, directly entering step a 3;
a3. deleting the timing task record, and entering the step a 4;
a4. and establishing a timing task thread, storing the timing task thread in a memory, and simultaneously storing a timing task record.
Further, the process of executing the timing task includes:
b1. inquiring a local timing task record from the timing task table according to the data ID and the data type, judging whether the local timing task record exists, if not, ending, and if so, entering a step b 2;
b2. judging whether the timing task record is valid, if so, performing service processing, and entering the step b3, and if not, directly entering the step b 3;
b3. acquiring a task thread from the memory, if the task thread is successfully acquired, stopping timing the task thread, clearing the timing task thread in the memory, and entering step b4, and if the task thread is not successfully acquired, directly entering step b 4;
b4. and deleting the timing task record and ending.
Further, the timing revocation task flow includes:
c1. inquiring a timing task record from the timing task table according to the data ID and the data type, judging whether the timing task record is the local timing task record, if not, setting the timing task record to be invalid, and ending, if so, entering a step c 2;
c2. judging whether a timing task thread exists in the memory, if so, stopping the timing task thread, clearing the timing task thread in storage, and entering step c3, otherwise, directly entering step c 3;
c3. and deleting the timing task record and ending.
Further, the initializing timing task process includes:
d1. inquiring all timing task records of the local machine from the timing task table according to the data ID and the data type, judging whether the timing task records of the local machine exist, if not, ending, and if so, entering the step d 2;
d2. judging whether the timing task record is valid, if so, performing service processing, and entering the step d3, and if not, deleting the timing task record and ending;
d3. and judging whether the time is out or not, if not, performing service processing, and if so, creating a timing task.
The invention has the beneficial effects that:
(1) the problem of high concurrency efficiency is solved: due to the fact that resources are easily contended or queuing and waiting are easily caused when the concurrency is high, the timing task is failed to be established or delay is serious, and the efficiency is low. The application adopts the Executors thread pool, effectively manages the timing thread through the thread pool, and greatly improves the processing capacity during concurrency, thereby achieving the purpose of improving the efficiency.
(2) The problems of repeated timing and low accuracy caused by distributed deployment are solved: according to the method, a unique identifier, namely the application code (appCode), is set for each deployed application, and a single task is accurately positioned by taking the application code (appCode), the data ID (dataId) and the data type (dataType) as a joint main key, so that the problem of repeated timing caused by distributed deployment of the application is avoided, and the timing accuracy is improved.
(3) The problems that timing tasks are lost and reliability is low due to service downtime are solved: the timing tasks are persisted into a high-speed memory database (Mongodb) and are synchronized with the timing tasks in the memory in real time, and when the service is down and restarted, the timing tasks persisted in the Mongodb are screened and then added into the memory timing tasks again, so that the loss of the timing tasks when the service is down is avoided, and the reliability is greatly improved.
(4) The problem of unconfigurable inefficiency is solved: the method and the device well solve the problem of low efficiency of the timing function by utilizing the characteristics of high concurrency and high-speed query of a high-speed memory database (Mongodb); meanwhile, the flexibility of the timing function is greatly improved by storing the creating time (createTime) of the timing task and comparing the current time with the creating time to obtain the timing time. And configuring the timeout time of different classes of service operation by using the management interface to achieve the purpose of configuration.
(5) The problem of difficult maintenance of procedure is solved: in the application, the memory timing task can be synchronized to a high-speed memory database (Mongodb) in real time, and the Mongodb has the characteristics of the database and can be visually operated, so that the maintenance difficulty of the timing program is greatly increased.
Detailed Description
The embodiment provides a MongoDB-based distributed timing system, which has the timing functions of high reliability, high concurrency, easy maintenance, configurability and under a distributed architecture and can create the core competitive advantage for financial cloud platform products. The embodiment comprises a MongoDB memory database, an Executors thread pool, a timing task creating module, a timing task executing module, a timing task canceling module and a timing task initializing module.
The MongoDB memory database maintains a timing task table (ExistFutureInfo) which is used for persistently storing timing tasks, synchronizing the timing tasks with the timing tasks in the memory in real time, and when the service is delayed and restarted, screening the persistently stored timing tasks and adding the screened persistently stored timing tasks into the memory timing tasks again; the timing task table comprises the following fields: unique primary key (id), data id (dataid), data type (dataType), application code (appCode), timeOut time (timeOut), creation time (createTime), valid flag (validFlag).
The Executors thread pool is used for managing timing threads.
The timing task creating module is used for completing a timing task creating process, and the specific process is as follows:
a1. inquiring timing task records from the timing task table according to the data ID and the data type, and if the timing task records do not exist, directly entering the step a 4; if the timing task record exists, judging whether the timing task is the timing task of the local computer, if so, entering a step a2, otherwise, setting the timing task as invalid, and entering a step a 4;
a2. judging whether a timing task thread exists in the memory, if so, stopping the timing task thread, clearing the timing task thread in storage, and entering step a3, otherwise, directly entering step a 3;
a3. deleting the timing task record, and entering the step a 4;
a4. and establishing a timing task thread, storing the timing task thread in a memory, and simultaneously storing a timing task record.
The timing task execution module is used for completing a timing task execution process, and the specific process is as follows:
b1. inquiring a local timing task record from the timing task table according to the data ID and the data type, judging whether the local timing task record exists, if not, ending, and if so, entering a step b 2;
b2. judging whether the timing task record is valid, if so, performing service processing, and entering the step b3, and if not, directly entering the step b 3;
b3. acquiring a task thread from the memory, if the task thread is successfully acquired, stopping the timing task thread, clearing the timing task thread in the memory, and entering step b4, if the task thread is not successfully acquired, indicating that the timing task thread does not exist in the memory, and directly entering step b 4;
b4. and deleting the timing task record and ending.
The timing cancellation task module is used for completing a timing cancellation task process, and the specific process is as follows:
c1. inquiring a timing task record from the timing task table according to the data ID and the data type, judging whether the timing task record is the local timing task record, if not, setting the timing task record to be invalid, and ending, if so, entering a step c 2;
c2. judging whether a timing task thread exists in the memory, if so, stopping the timing task thread, clearing the timing task thread in storage, and entering step c3, otherwise, directly entering step c 3;
c3. and deleting the timing task record and ending.
The initialization timing task module is used for completing an initialization timing task process, and the specific process is as follows:
d1. inquiring all timing task records of the local machine from the timing task table according to the data ID and the data type, judging whether the timing task records of the local machine exist, if not, ending, and if so, entering the step d 2;
d2. judging whether the timing task record is valid, if so, performing service processing, and entering the step d3, and if not, deleting the timing task record and ending;
d3. and judging whether the time is out or not, if not, performing service processing, and if so, creating a timing task.
The foregoing describes the general principles and features of the present invention and, together with the general principles of the invention, further modifications and improvements thereto, may be made without departing from the spirit and scope of the invention as set forth in the appended claims.
Claims (3)
1. The MongoDB-based distributed timing system is characterized by comprising a MongoDB memory database, Executors thread pools, a timing task creating module, a timing task executing module, a timing task canceling module and a timing task initializing module;
the MongoDB memory database maintains a timing task table which is used for persistently storing timing tasks and synchronizing the timing tasks with the timing tasks in the memory in real time, and when the service is down and restarted, the persistently stored timing tasks are screened and then added into the memory timing tasks again; the timing task table includes the following fields: a unique primary key, a data ID, a data type, an application code, a timeout time, a creation time, and a valid flag;
the Executors thread pool is used for managing timing threads;
the timing task creating module, the timing task executing module, the timing task canceling module and the timing task initializing module are respectively used for completing a timing task creating process, a timing task executing process, a timing task canceling process and a timing task initializing process;
the step of creating the timing task flow comprises the following steps:
a1. inquiring timing task records from the timing task table according to the data ID and the data type, and if the timing task records do not exist, directly entering the step a 4; if the timing task record exists, judging whether the timing task is the timing task of the local computer, if so, entering a step a2, otherwise, setting the timing task as invalid, and entering a step a 4;
a2. judging whether a timing task thread exists in the memory, if so, stopping the timing task thread, clearing the timing task thread in storage, and entering step a3, otherwise, directly entering step a 3;
a3. deleting the timing task record, and entering the step a 4;
a4. creating a timing task thread, storing the timing task thread in a memory, and simultaneously storing a timing task record;
the timing task execution process comprises the following steps:
b1. inquiring a local timing task record from the timing task table according to the data ID and the data type, judging whether the local timing task record exists, if not, ending, and if so, entering a step b 2;
b2. judging whether the timing task record is valid, if so, performing service processing, and entering the step b3, and if not, directly entering the step b 3;
b3. acquiring a task thread from the memory, if the task thread is successfully acquired, stopping timing the task thread, clearing the timing task thread in the memory, and entering step b4, and if the task thread is not successfully acquired, directly entering step b 4;
b4. deleting the timing task record and ending;
the timing task revocation process comprises the following steps:
c1. inquiring a timing task record from the timing task table according to the data ID and the data type, judging whether the timing task record is the local timing task record, if not, setting the timing task record to be invalid, and ending, if so, entering a step c 2;
c2. judging whether a timing task thread exists in the memory, if so, stopping the timing task thread, clearing the timing task thread in storage, and entering step c3, otherwise, directly entering step c 3;
c3. deleting the timing task record and ending;
the initialization timing task flow comprises the following steps:
d1. inquiring all timing task records of the local machine from the timing task table according to the data ID and the data type, judging whether the timing task records of the local machine exist, if not, ending, and if so, entering the step d 2;
d2. judging whether the timing task record is valid, if so, performing service processing, and entering the step d3, and if not, deleting the timing task record and ending;
d3. and judging whether the time is out or not, if not, performing service processing, and if so, creating a timing task.
2. The MongoDB-based distributed timing system of claim 1, wherein the executive timing task module locates individual timing tasks accurately by applying code, data ID, data type as joint primary keys.
3. The MongoDB-based distributed timekeeping system of claim 1, wherein the executive timekeeping task module derives a timekeeping time by comparing a system current time with the creation time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710827425.3A CN107608662B (en) | 2017-09-14 | 2017-09-14 | MongoDB-based distributed timing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710827425.3A CN107608662B (en) | 2017-09-14 | 2017-09-14 | MongoDB-based distributed timing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107608662A CN107608662A (en) | 2018-01-19 |
| CN107608662B true CN107608662B (en) | 2020-11-03 |
Family
ID=61063891
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710827425.3A Expired - Fee Related CN107608662B (en) | 2017-09-14 | 2017-09-14 | MongoDB-based distributed timing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107608662B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116360955A (en) * | 2023-05-31 | 2023-06-30 | 天翼云科技有限公司 | Asynchronous task processing method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106384283A (en) * | 2016-08-26 | 2017-02-08 | 山东尚捷信息科技有限公司 | Internet plus based service bus structure and service bus system |
| CN106530810A (en) * | 2016-11-18 | 2017-03-22 | 安徽超清科技股份有限公司 | Traffic guidance system based on information of parking lot |
| CN106528051A (en) * | 2016-11-15 | 2017-03-22 | 国云科技股份有限公司 | High-efficiency operation method for queuing and stacking big data based on MongoDB |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9100291B2 (en) * | 2012-01-31 | 2015-08-04 | Db Networks, Inc. | Systems and methods for extracting structured application data from a communications link |
-
2017
- 2017-09-14 CN CN201710827425.3A patent/CN107608662B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106384283A (en) * | 2016-08-26 | 2017-02-08 | 山东尚捷信息科技有限公司 | Internet plus based service bus structure and service bus system |
| CN106528051A (en) * | 2016-11-15 | 2017-03-22 | 国云科技股份有限公司 | High-efficiency operation method for queuing and stacking big data based on MongoDB |
| CN106530810A (en) * | 2016-11-18 | 2017-03-22 | 安徽超清科技股份有限公司 | Traffic guidance system based on information of parking lot |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107608662A (en) | 2018-01-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9619430B2 (en) | Active non-volatile memory post-processing | |
| CN110309161B (en) | Data synchronization method and device and server | |
| KR102072726B1 (en) | Systems and methods for supporting inline delegation of middle-tier transaction logs to database | |
| US8938421B2 (en) | Method and a system for synchronizing data | |
| WO2019223173A1 (en) | High-frequency task scheduling method and system, and computer device and storage medium | |
| CN104809202A (en) | Database synchronization method and device | |
| CN107025224B (en) | Method and equipment for monitoring task operation | |
| US20190324760A1 (en) | Transaction Processing Method and Apparatus | |
| CN109063005B (en) | Data migration method and system, storage medium and electronic device | |
| CN109871290B (en) | Call stack tracking method and device applied to Java and storage medium | |
| CN112148788A (en) | Data synchronization method and system for heterogeneous data source | |
| CN104021043A (en) | Interruption reentry method and system of batch applications | |
| WO2020232951A1 (en) | Task execution method and device | |
| CN115934855A (en) | Full-link field level blood margin analysis method, system, equipment and storage medium | |
| CN105306581A (en) | Router and cloud disk based data synchronization method | |
| US20170031662A1 (en) | Distributed compilation of statically typed languages | |
| CN109800069B (en) | Method and device for realizing data management | |
| CN108399095B (en) | Method, system, device and storage medium for supporting dynamic management of timed tasks | |
| CN107608662B (en) | MongoDB-based distributed timing system | |
| CN104750849B (en) | For safeguarding the method and system of the catalogue relation based on tree structure | |
| TW201600975A (en) | Processing tasks in a distributed system | |
| CN105447012A (en) | Write-in exclusive method and device used for database | |
| CN104317906A (en) | Processing method for large-scale storage of reported data of GPS (global positioning system) monitoring radios | |
| US8180739B2 (en) | Duplicate filtering in a data processing environment | |
| CN104102583A (en) | High-availability cluster software distributed automated testing framework |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20201103 Termination date: 20210914 |