CN113590708A - Adaptive delay consumption method, program product, and storage medium - Google Patents
Adaptive delay consumption method, program product, and storage medium Download PDFInfo
- Publication number
- CN113590708A CN113590708A CN202110672428.0A CN202110672428A CN113590708A CN 113590708 A CN113590708 A CN 113590708A CN 202110672428 A CN202110672428 A CN 202110672428A CN 113590708 A CN113590708 A CN 113590708A
- Authority
- CN
- China
- Prior art keywords
- task distribution
- delay time
- distribution speed
- consumed
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 230000003044 adaptive effect Effects 0.000 title claims description 48
- 238000012545 processing Methods 0.000 claims abstract description 14
- 238000004590 computer program Methods 0.000 claims description 12
- 230000003111 delayed effect Effects 0.000 claims description 6
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000011144 upstream manufacturing Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a self-adaptive delay consumption method, a program product and a storage medium, wherein the method comprises the following steps: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed. The self-adaptive delay consumption method, the program product and the storage medium provided by the invention can automatically detect the master-slave delay time of the database while finishing data synchronization, and self-adaptively adjust the synchronization speed according to the master-slave delay degree, thereby improving the system stability, having high timeliness and saving the manpower consumption.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to an adaptive delay consumption method, a program product, and a storage medium.
Background
In an existing data synchronization system, data is generally dropped into storage services such as mysql and the like, and meanwhile, in order to ensure the availability of the storage services, the storage services generally adopt a master-slave architecture, so that a phenomenon that the master-slave delay of a database is too high due to too high synchronization speed may occur in a synchronization process. The storage service of the master-slave architecture is realized by adopting a master database and a slave database. Master-slave latency refers to the time difference between the consumption of the same data received by the master and slave database pairs for storage. In order to reduce the master-slave delay, the synchronization speed is generally artificially reduced after receiving a database master-slave delay alarm to reduce the delay degree, and the manual operation increases the complexity of solving the problem and reduces the timeliness.
Disclosure of Invention
To solve the problems of the prior art, the present invention provides an adaptive delay consumption method, a program product and a storage medium.
The invention provides a self-adaptive delay consumption method, which comprises the following steps: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed.
According to the adaptive delay consumption method provided by the invention, the detection of the master-slave delay time of the database system specifically comprises the following steps: updating a preset database record in a main database, and storing the current updating time in a preset field; and inquiring a field value of the preset field obtained from a database, and calculating the absolute value of the difference value between the current updating time and the field value so as to obtain the master-slave delay time.
According to the adaptive delay consumption method provided by the invention, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, and the method specifically comprises the following steps: if the master-slave delay time is greater than or equal to the preset maximum delay time, reducing the task distribution speed until the task distribution speed reaches the preset minimum task distribution speed; and if the master-slave delay time is less than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed.
According to the adaptive delay consumption method provided by the invention, the initial value of the task distribution speed is set as the preset maximum task distribution speed.
According to the adaptive delay consumption method provided by the invention, the reducing of the task distribution speed specifically comprises the following steps: the task distribution speed is reduced to 1/2 as it is.
According to the adaptive delay consumption method provided by the invention, the increasing of the task distribution speed specifically comprises the following steps: and increasing the task distribution speed to 2 times of the original speed.
According to the adaptive delay consumption method provided by the invention, the acquiring of the event to be consumed specifically comprises the following steps: and acquiring the events to be consumed from the kafka message queue.
The present invention also provides an adaptive delay consuming apparatus, comprising: an event collection module to: acquiring an event to be consumed, and adding the event to be consumed into a task pool; a task distribution module to: detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; an event consumption module to: and taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed.
The present invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the adaptive delay consumption method as described in any of the above.
The present invention also provides a computer program product comprising computer programs/instructions which, when executed by a processor, implement the steps of the adaptive delay consumption method as described in any of the above.
The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the adaptive delayed consumption method as described in any one of the above.
According to the self-adaptive delay consumption method, the program product and the storage medium, the events to be consumed are obtained and added into the task pool, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, the events to be consumed in the task pool are stored into the task queue at the task distribution speed, the events to be consumed are taken out from the task queue and are consumed, data synchronization is completed, the master-slave delay time of the database can be automatically detected, the synchronization speed is adaptively adjusted according to the master-slave delay degree, the system stability is improved, the timeliness is high, and manpower consumption is saved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of an adaptive delay consumption method provided by the present invention;
FIG. 2 is a schematic diagram of an adaptive delay consumer according to the present invention;
fig. 3 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The adaptive delay consumption method, program product and storage medium of the present invention are described below in conjunction with fig. 1-3.
Fig. 1 is a flow chart of an adaptive delay consumption method provided by the present invention. As shown in fig. 1, the method includes:
In a data synchronization system, an upstream system sends a consumption message to a downstream system, and the consumption message is generally stored in a message queue. A to-be-consumed event may be obtained from the message queue, the to-be-consumed event including a message that has not been consumed. And after the event to be consumed is obtained, adding the event to be consumed into the task pool for subsequent processing.
And 102, detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed.
And detecting the master-slave delay time of the database system, for example, detecting the master-slave delay time of the database system according to a preset time period. And adjusting the task distribution speed according to the relation between the master-slave delay time and the preset maximum delay time, wherein the master-slave delay time should not exceed the preset maximum delay time, the task distribution speed can be adjusted according to the size relation between the master-slave delay time and the preset maximum delay time, and the events to be consumed in the task pool are stored into the task queue based on the task distribution speed, so that the task distribution processing is realized.
And 103, taking out the event to be consumed from the task queue, and consuming the event to be consumed.
And carrying out consumption processing on the event to be consumed, namely realizing data synchronization of upstream and downstream systems. And when the consumption is carried out, taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed. Adjusting the task distribution speed of the task queue is equivalent to adjusting the consumption speed, i.e. adjusting the data synchronization speed. And distributing the consumption events in the task pool based on the task distribution speed, consuming after the task distribution is finished, and storing the consumption events in the database. And generating master-slave delay time in the data storage process, and obtaining the task distribution speed according to the relation between the master-slave delay time and the preset maximum delay time. Therefore, a controlled closed loop can be formed, so that a data synchronization process (consumption process) and a database storage process have a restriction relationship, the consumption progress can be adjusted in a self-adaptive manner, and the master-slave delay is optimized.
According to the self-adaptive delay consumption method provided by the invention, the event to be consumed is obtained and added into the task pool, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, the event to be consumed in the task pool is stored into the task queue at the task distribution speed, the event to be consumed is taken out from the task queue and is consumed, the master-slave delay time of the database can be automatically detected while the data synchronization is finished, the synchronization speed is self-adaptively adjusted according to the master-slave delay degree, the system stability is improved, the timeliness is high, and the manpower consumption is saved.
According to the adaptive delay consumption method provided by the invention, the detection of the master-slave delay time of the database system specifically comprises the following steps: updating a preset database record in a main database, and storing the current updating time in a preset field; and inquiring a field value of the preset field obtained from a database, and calculating the absolute value of the difference value between the current updating time and the field value so as to obtain the master-slave delay time.
The synchronous storage of the master and slave databases can be realized in the prior art. The data and fields stored in the master and slave databases are the same. The master-slave delay time of the database system can be detected regularly. When the master-slave delay time of the database system is detected, the preset database record in the master database can be updated, and the current update time is stored in the preset field. If the master database performs data updating, the slave database also performs the same updating and stores the data in the same field, namely the preset field. Therefore, after the update of the master database is completed, the query obtains the field value of the preset field from the database, and the field value indicates the update time of the preset database record in the slave database. Since the latest update may not have been made available from the database at this time, the field value will be less than the current update time. If the latest update described above has been made from the database, the field value will be greater than the current update time. No matter the field value of the slave database is larger or smaller than the current updating time, the absolute value of the difference value between the current updating time and the field value of the preset field in the slave database can reflect the master-slave delay time, and therefore the absolute value of the difference value between the current updating time and the field value is calculated, and therefore the master-slave delay time is obtained.
The preset database record may be a database record pre-written in the master database and the slave database by creating a table.
The self-adaptive delay consumption method provided by the invention updates the record of the preset database in the main database and stores the current updating time in the preset field; and inquiring a field value of a preset field obtained from a database, and calculating the absolute value of the difference value between the current updating time and the field value so as to obtain the master-slave delay time and realize the quick and reliable acquisition of the master-slave delay time.
According to the adaptive delay consumption method provided by the invention, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, and the method specifically comprises the following steps: if the master-slave delay time is greater than or equal to the preset maximum delay time, reducing the task distribution speed until the task distribution speed reaches the preset minimum task distribution speed; and if the master-slave delay time is less than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed.
A preset maximum task distribution speed and a preset minimum task distribution speed may be preset to indicate the maximum limit and the minimum limit of task distribution, that is, the task distribution speed should be between the preset minimum task distribution speed and the preset maximum task distribution speed.
When the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, if the master-slave delay time is greater than or equal to the preset maximum delay time, the task distribution speed is reduced until the task distribution speed reaches the preset minimum task distribution speed, and the mode of reducing the task distribution speed can be set according to actual requirements. And if the master-slave delay time is less than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed, wherein the mode of increasing the task distribution speed can be set according to actual requirements.
According to the self-adaptive delay consumption method provided by the invention, when the master-slave delay time is greater than or equal to the preset maximum delay time, the task distribution speed is reduced until the task distribution speed reaches the preset minimum task distribution speed; and when the master-slave delay time is less than the preset maximum delay time, the task distribution speed is increased until the task distribution speed reaches the preset maximum task distribution speed, so that the task distribution speed is ensured to be within the range of the preset interval, and the task distribution speed and the master-slave delay time are balanced, thereby balancing the consumption progress and the master-slave delay.
According to the adaptive delay consumption method provided by the invention, the initial value of the task distribution speed is set as the preset maximum task distribution speed.
The initial value (starting speed in one period) of the task distribution speed is set to be the preset maximum task distribution speed, and if the master-slave delay time is smaller than the maximum delay time, the task distribution is carried out at the preset maximum task distribution speed, so that the event consumption efficiency is improved on the basis of meeting the master-slave delay requirement.
According to the self-adaptive delay consumption method provided by the invention, the initial value of the task distribution speed is set as the preset maximum task distribution speed, so that the event consumption efficiency is improved on the basis of meeting the master-slave delay requirement.
According to the adaptive delay consumption method provided by the invention, the reducing of the task distribution speed specifically comprises the following steps: the task distribution speed is reduced to 1/2 as it is.
When the master-slave delay time is greater than or equal to the preset maximum delay time and the task distribution speed is reduced, the task distribution speed can be reduced to 1/2. For example, if the task distribution speed before adjustment is ns, ns is adjusted to ns/2. And then, calculating the master-slave delay time again at the task distribution speed, if the master-slave delay time is still greater than or equal to the preset maximum delay time, adjusting ns/2 to ns/4, and so on until the task distribution speed reaches the preset minimum task distribution speed.
The invention reduces the task distribution speed to 1/2 when reducing the task distribution speed, thereby improving the convenience of task distribution speed adjustment.
According to the adaptive delay consumption method provided by the invention, the increasing of the task distribution speed specifically comprises the following steps: and increasing the task distribution speed to 2 times of the original speed.
When the master-slave delay time is less than the preset maximum delay time and the task distribution speed is increased, the task distribution speed can be increased to 2 times of the original speed. For example, if the task distribution speed before adjustment is ns, ns is adjusted to 2 ns. And then, calculating the master-slave delay time again at the task distribution speed, if the master-slave delay time is still less than the preset maximum delay time, adjusting 2ns to 4ns, and so on until the task distribution speed reaches the preset maximum task distribution speed.
The invention increases the task distribution speed to 2 times of the original speed when increasing the task distribution speed, thereby improving the convenience of the task distribution speed adjustment.
According to the adaptive delay consumption method provided by the invention, the acquiring of the event to be consumed specifically comprises the following steps: and acquiring the events to be consumed from the kafka message queue.
The kafka consumption event can be utilized, and when the to-be-consumed event is obtained, the to-be-consumed event can be obtained from a kafka message queue.
According to the self-adaptive delay consumption method, the event to be consumed is acquired from the kafka message queue, and the simplicity of event consumption is improved.
The flow of the adaptive delay consumption method provided by the present invention is further illustrated by an embodiment. The self-adaptive delay consumption method provided by the invention comprises the following steps:
1. acquiring an upstream event to be consumed, and adding the event to be consumed into a task pool, such as: and starting an event collection process, acquiring the events to be consumed from a message queue such as kafka and the like, and adding the events to be consumed into a task pool.
2. Detecting the time of master-slave delay of the database system, such as: starting a monitoring process, updating a record in the master library at regular time, storing the updating time in an mtime field, simultaneously inquiring the mtime field value T of the record from the slave library, and calculating the absolute value of the difference value between the current time and the T, namely the master-slave delay time T.
3. Putting the tasks in the task pool into a task queue, and automatically adjusting the task distribution speed according to the master-slave delay time of the database, such as: presetting a maximum distribution speed max and a minimum distribution speed min, starting a task distribution process, setting the starting speed to be ns as max, putting tasks in a task pool into a task queue according to ns, and starting circulation, wherein if master-slave delay time T > is maximum delay time mt, ns is ns/2, until the minimum distribution speed min is reached; if T < mt, ns is 2ns, until the maximum dispensing speed max is reached.
4. Obtaining the event from the task queue, and performing corresponding processing on the event, such as: and starting the consumer process, taking out the tasks from the task queue, and landing the data by the consumption tasks.
The following describes the adaptive delay consuming apparatus provided by the present invention, and the adaptive delay consuming apparatus described below and the adaptive delay consuming method described above may be referred to correspondingly.
Fig. 2 is a schematic structural diagram of an adaptive delay consuming apparatus provided in the present invention. As shown in fig. 2, the apparatus includes: an event collection module 10, a task distribution module 20, and an event consumption module 30, wherein: the event collection module 10 is configured to: acquiring an event to be consumed, and adding the event to be consumed into a task pool; the task distribution module 20 is configured to: detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; the event consumption module 30 is configured to: and taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed.
According to the self-adaptive delay consumption method provided by the invention, the event to be consumed is obtained and added into the task pool, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, the event to be consumed in the task pool is stored into the task queue at the task distribution speed, the event to be consumed is taken out from the task queue and is consumed, the master-slave delay time of the database can be automatically detected while the data synchronization is finished, the synchronization speed is self-adaptively adjusted according to the master-slave delay degree, the system stability is improved, the timeliness is high, and the manpower consumption is saved.
According to the adaptive delay consumption device provided by the present invention, when the task distribution module 20 is used for detecting a master-slave delay time of a database system, it is specifically configured to: updating a preset database record in a main database, and storing the current updating time in a preset field; and inquiring a field value of the preset field obtained from a database, and calculating the absolute value of the difference value between the current updating time and the field value so as to obtain the master-slave delay time.
The self-adaptive delay consumption method provided by the invention updates the record of the preset database in the main database and stores the current updating time in the preset field; and inquiring a field value of a preset field obtained from a database, and calculating the absolute value of the difference value between the current updating time and the field value so as to obtain the master-slave delay time and realize the quick and reliable acquisition of the master-slave delay time.
According to the adaptive delay consuming apparatus provided by the present invention, when the task distributing module 20 is configured to adjust the task distributing speed according to the relationship between the master-slave delay time and the preset maximum delay time, it is specifically configured to: if the master-slave delay time is greater than or equal to the preset maximum delay time, reducing the task distribution speed until the task distribution speed reaches the preset minimum task distribution speed; and if the master-slave delay time is less than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed.
According to the self-adaptive delay consumption method provided by the invention, when the master-slave delay time is greater than or equal to the preset maximum delay time, the task distribution speed is reduced until the task distribution speed reaches the preset minimum task distribution speed; and when the master-slave delay time is less than the preset maximum delay time, the task distribution speed is increased until the task distribution speed reaches the preset maximum task distribution speed, so that the task distribution speed is ensured to be within the range of the preset interval, and the task distribution speed and the master-slave delay time are balanced, thereby balancing the consumption progress and the master-slave delay.
According to the adaptive delay consumption device provided by the invention, the initial value of the task distribution speed is set to be the preset maximum task distribution speed.
According to the self-adaptive delay consumption method provided by the invention, the initial value of the task distribution speed is set as the preset maximum task distribution speed, so that the event consumption efficiency is improved on the basis of meeting the master-slave delay requirement.
According to the adaptive delay consumption device provided by the invention, the reducing of the task distribution speed specifically comprises: the task distribution speed is reduced to 1/2 as it is.
The invention reduces the task distribution speed to 1/2 when reducing the task distribution speed, thereby improving the convenience of task distribution speed adjustment.
According to the adaptive delay consumption device provided by the present invention, the increasing the task distribution speed specifically includes: and increasing the task distribution speed to 2 times of the original speed.
The invention increases the task distribution speed to 2 times of the original speed when increasing the task distribution speed, thereby improving the convenience of the task distribution speed adjustment.
According to the adaptive delay consumption device provided by the invention, the acquiring of the event to be consumed specifically comprises: and acquiring the events to be consumed from the kafka message queue.
According to the self-adaptive delay consumption method, the event to be consumed is acquired from the kafka message queue, and the simplicity of event consumption is improved.
Fig. 3 is a schematic structural diagram of an electronic device provided in the present invention, and as shown in fig. 3, the electronic device may include: a processor (processor)310, a communication Interface (communication Interface)320, a memory (memory)330 and a communication bus 340, wherein the processor 310, the communication Interface 320 and the memory 330 communicate with each other via the communication bus 340. The processor 310 may invoke logic instructions in the memory 330 to perform an adaptive delay consumption method comprising: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed.
In addition, the logic instructions in the memory 330 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the adaptive delay consumption method provided by the above methods, the method comprising: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the adaptive delayed consumption method provided above, the method comprising: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. An adaptive delay consumption method, comprising:
acquiring an event to be consumed, and adding the event to be consumed into a task pool;
detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed;
and taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed.
2. The adaptive delay consumption method according to claim 1, wherein the detecting master-slave delay time of the database system specifically comprises:
updating a preset database record in a main database, and storing the current updating time in a preset field; and inquiring a field value of the preset field obtained from a database, and calculating the absolute value of the difference value between the current updating time and the field value so as to obtain the master-slave delay time.
3. The adaptive delay consumption method according to claim 1, wherein the adjusting the task distribution speed according to the relationship between the master-slave delay time and the preset maximum delay time specifically comprises:
if the master-slave delay time is greater than or equal to the preset maximum delay time, reducing the task distribution speed until the task distribution speed reaches the preset minimum task distribution speed;
and if the master-slave delay time is less than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed.
4. The adaptive delayed consumption method of claim 3, wherein an initial value of the task distribution speed is set to the preset maximum task distribution speed.
5. The adaptive delay consumption method according to claim 4, wherein the reducing the task distribution speed specifically comprises:
the task distribution speed is reduced to 1/2 as it is.
6. The adaptive delay consumption method according to claim 4, wherein the increasing the task distribution speed specifically comprises:
and increasing the task distribution speed to 2 times of the original speed.
7. The adaptive delayed consumption method according to claim 1, wherein the acquiring the event to be consumed specifically comprises:
and acquiring the events to be consumed from the kafka message queue.
8. An adaptive delay consuming apparatus, comprising:
an event collection module to: acquiring an event to be consumed, and adding the event to be consumed into a task pool;
a task distribution module to: detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed;
an event consumption module to: and taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed.
9. A computer program product comprising computer program/instructions, characterized in that the computer program/instructions, when executed by a processor, implement the steps of the adaptive delayed consumption method of any of claims 1 to 7.
10. A non-transitory computer readable storage medium, having stored thereon a computer program, which, when being executed by a processor, carries out the steps of the adaptive delayed consumption method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110672428.0A CN113590708B (en) | 2021-06-17 | 2021-06-17 | Adaptive delay consuming method, program product and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110672428.0A CN113590708B (en) | 2021-06-17 | 2021-06-17 | Adaptive delay consuming method, program product and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113590708A true CN113590708A (en) | 2021-11-02 |
| CN113590708B CN113590708B (en) | 2024-02-20 |
Family
ID=78243851
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110672428.0A Active CN113590708B (en) | 2021-06-17 | 2021-06-17 | Adaptive delay consuming method, program product and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113590708B (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170118042A1 (en) * | 2015-10-21 | 2017-04-27 | Brocade Communications Systems, Inc. | High availability for distributed network services in an extended bridge |
| CN106648904A (en) * | 2017-01-09 | 2017-05-10 | 大连理工大学 | Self-adaptive rate control method for stream data processing |
| US20170329384A1 (en) * | 2009-07-21 | 2017-11-16 | The Research Foundation For The State University Of New York | Apparatus and method for efficient estimation of the energy dissipation of processor based systems |
| CN109298951A (en) * | 2018-08-21 | 2019-02-01 | 平安科技（深圳）有限公司 | The execution method and terminal device of Celery task |
| US20190138638A1 (en) * | 2016-09-26 | 2019-05-09 | Splunk Inc. | Task distribution in an execution node of a distributed execution environment |
| CN109743137A (en) * | 2019-01-10 | 2019-05-10 | 浙江小泰科技有限公司 | A kind of Distributed Delay message queue processing system for supporting to update |
| CN110012039A (en) * | 2018-01-04 | 2019-07-12 | 华北电力大学 | Task distribution and power control scheme in a kind of car networking based on ADMM |
| CN110196761A (en) * | 2019-04-15 | 2019-09-03 | 北京达佳互联信息技术有限公司 | Delay task processing method and processing device |
| CN110636130A (en) * | 2019-09-23 | 2019-12-31 | 上海钧正网络科技有限公司 | Delayed message processing method and device, computer equipment and storage medium |
| CN111796946A (en) * | 2020-06-22 | 2020-10-20 | 深圳壹账通智能科技有限公司 | Data processing method, device, equipment and storage medium |
| CN111813513A (en) * | 2020-06-24 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Real-time task scheduling method, device, equipment and medium based on distribution |
| CN112073526A (en) * | 2020-09-14 | 2020-12-11 | 苏州颂康智能科技有限公司 | Distributed IOT remote monitoring system based on 5G |
| CN112104519A (en) * | 2020-08-06 | 2020-12-18 | 北京健康之家科技有限公司 | Delayed message delivery method and device, storage medium and computer equipment |
| CN112612607A (en) * | 2020-12-16 | 2021-04-06 | 平安消费金融有限公司 | Data processing method, device, computer equipment and medium based on message queue |
-
2021
- 2021-06-17 CN CN202110672428.0A patent/CN113590708B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170329384A1 (en) * | 2009-07-21 | 2017-11-16 | The Research Foundation For The State University Of New York | Apparatus and method for efficient estimation of the energy dissipation of processor based systems |
| US20170118042A1 (en) * | 2015-10-21 | 2017-04-27 | Brocade Communications Systems, Inc. | High availability for distributed network services in an extended bridge |
| US20190138638A1 (en) * | 2016-09-26 | 2019-05-09 | Splunk Inc. | Task distribution in an execution node of a distributed execution environment |
| CN106648904A (en) * | 2017-01-09 | 2017-05-10 | 大连理工大学 | Self-adaptive rate control method for stream data processing |
| CN110012039A (en) * | 2018-01-04 | 2019-07-12 | 华北电力大学 | Task distribution and power control scheme in a kind of car networking based on ADMM |
| CN109298951A (en) * | 2018-08-21 | 2019-02-01 | 平安科技（深圳）有限公司 | The execution method and terminal device of Celery task |
| CN109743137A (en) * | 2019-01-10 | 2019-05-10 | 浙江小泰科技有限公司 | A kind of Distributed Delay message queue processing system for supporting to update |
| CN110196761A (en) * | 2019-04-15 | 2019-09-03 | 北京达佳互联信息技术有限公司 | Delay task processing method and processing device |
| CN110636130A (en) * | 2019-09-23 | 2019-12-31 | 上海钧正网络科技有限公司 | Delayed message processing method and device, computer equipment and storage medium |
| CN111796946A (en) * | 2020-06-22 | 2020-10-20 | 深圳壹账通智能科技有限公司 | Data processing method, device, equipment and storage medium |
| CN111813513A (en) * | 2020-06-24 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Real-time task scheduling method, device, equipment and medium based on distribution |
| CN112104519A (en) * | 2020-08-06 | 2020-12-18 | 北京健康之家科技有限公司 | Delayed message delivery method and device, storage medium and computer equipment |
| CN112073526A (en) * | 2020-09-14 | 2020-12-11 | 苏州颂康智能科技有限公司 | Distributed IOT remote monitoring system based on 5G |
| CN112612607A (en) * | 2020-12-16 | 2021-04-06 | 平安消费金融有限公司 | Data processing method, device, computer equipment and medium based on message queue |
Non-Patent Citations (2)
| Title |
|---|
| MIAN GUO: ""Optimal Scheduling of VMs in Queueing Cloud Computing Systems With a Heterogeneous Workload"", 《IEEE ACCESS ( VOLUME: 6)》, pages 15178 - 15191 * |
| 李祎 等: ""互联网延迟队列解决方案设计"", 《福建电脑》, pages 9 - 12 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113590708B (en) | 2024-02-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108388479B (en) | Delayed message pushing method and device, computer equipment and storage medium | |
| CN108009642B (en) | Distributed machine learning method and system | |
| CN107092521B (en) | Distributed task scheduling method, device and system | |
| CN109450987B (en) | Number generation method, device and system and storage medium | |
| CN110599659A (en) | Access control permission synchronization method and device | |
| CN113193947B (en) | Method, apparatus, medium, and program product for implementing distributed global ordering | |
| CN111181765A (en) | Task processing method and device | |
| CN114048931B (en) | Replenishment information generation method and device, electronic equipment and computer readable medium | |
| CN109388655A (en) | A kind of method and apparatus of dynamic control of data access | |
| CN114492834A (en) | Training method, training device, training equipment, training system and storage medium | |
| CN113590708A (en) | Adaptive delay consumption method, program product, and storage medium | |
| CN110928746B (en) | Configurable business processing system, method, device and storage medium | |
| CN113157047A (en) | Time adjustment method and device, computer equipment and storage medium | |
| US20160344537A1 (en) | Communication apparatus, communication method, and computer readable medium | |
| CN110071952A (en) | The control method and device of service call amount | |
| CN115129429B (en) | Container application management method and device, electronic equipment and storage medium | |
| CN109284177B (en) | Data updating method and device | |
| CN113703964A (en) | Memory management method and device | |
| CN110941404B (en) | Shared printing method and system of laser printer | |
| CN114612212A (en) | Business processing method, device and system based on risk control | |
| JP2006053737A (en) | Replication system and replication method | |
| CN113672682B (en) | Synchronous frame based quantity synchronization method and synchronization device | |
| US9847919B2 (en) | Data network device discovery optimization to reduce data transfer volume | |
| CN116542324B (en) | Distributed asynchronous protocol method and device for intelligent computing | |
| CN116974825A (en) | Backup method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20231128 Address after: Room 102, 1st Floor, Building 1, No. 2 Chuangye Road, Haidian District, Beijing, 100080 Applicant after: Seashell Housing (Beijing) Technology Co.,Ltd. Address before: 101309 room 24, 62 Farm Road, Erjie village, Yangzhen, Shunyi District, Beijing Applicant before: Beijing fangjianghu Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |