CN113590708B - Adaptive delay consuming method, program product and storage medium - Google Patents
Adaptive delay consuming method, program product and storage medium Download PDFInfo
- Publication number
- CN113590708B CN113590708B CN202110672428.0A CN202110672428A CN113590708B CN 113590708 B CN113590708 B CN 113590708B CN 202110672428 A CN202110672428 A CN 202110672428A CN 113590708 B CN113590708 B CN 113590708B
- Authority
- CN
- China
- Prior art keywords
- task distribution
- delay time
- distribution speed
- master
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 230000003044 adaptive effect Effects 0.000 title claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 14
- 238000004590 computer program Methods 0.000 claims description 9
- 230000003111 delayed effect Effects 0.000 claims 1
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000011144 upstream manufacturing Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides an adaptive delay consumption method, a program product and a storage medium, wherein the method comprises the following steps: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed. According to the self-adaptive delay consumption method, the program product and the storage medium, the data synchronization is completed, the time of master-slave delay of the database can be automatically detected, the synchronization speed is self-adaptively adjusted according to the master-slave delay degree, the system stability is improved, the timeliness is high, and the manpower consumption is saved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to an adaptive delay consumption method, a program product, and a storage medium.
Background
In the existing data synchronization system, data are generally landed in storage services such as mysql, and in order to ensure availability of the storage services, the storage services generally adopt a master-slave architecture, so that a phenomenon that a master-slave delay of a database is too high due to too high synchronization speed may occur in a synchronization process. The storage service of the master-slave architecture is realized by adopting a master database and a slave database. Master-slave latency refers to the time difference between the master database and the slave database when the same received data is stored post-consumer. In order to reduce the master-slave delay, the synchronization speed is generally manually reduced after receiving the master-slave delay alarm of the database, so as to reduce the delay degree, and the manual operation increases the complexity of solving the problem and reduces the timeliness.
Disclosure of Invention
The invention provides an adaptive delay consumption method, a program product and a storage medium aiming at the problems existing in the prior art.
The invention provides a self-adaptive delay consumption method, which comprises the following steps: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed.
According to the self-adaptive delay consumption method provided by the invention, the master-slave delay time of the database system is detected, and the method specifically comprises the following steps: updating a preset database record in the main database, and storing the current updating time in a preset field; inquiring a field value of the preset field obtained from a database, and calculating an absolute value of a difference value between the current updating time and the field value, thereby obtaining the master-slave delay time.
According to the self-adaptive delay consumption method provided by the invention, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, and the self-adaptive delay consumption method specifically comprises the following steps: if the master-slave delay time is greater than or equal to the preset maximum delay time, reducing the task distribution speed until the task distribution speed reaches a preset minimum task distribution speed; and if the master-slave delay time is smaller than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed.
According to the self-adaptive delay consumption method provided by the invention, the initial value of the task distribution speed is set as the preset maximum task distribution speed.
According to the self-adaptive delay consumption method provided by the invention, the task distribution speed is reduced, and the self-adaptive delay consumption method specifically comprises the following steps: and reducing the task distribution speed to 1/2 of the original speed.
According to the self-adaptive delay consumption method provided by the invention, the task distribution speed is increased, and the self-adaptive delay consumption method specifically comprises the following steps: and increasing the task distribution speed by 2 times.
According to the self-adaptive delay consumption method provided by the invention, the acquisition of the event to be consumed specifically comprises the following steps: and acquiring the event to be consumed from the kafka message queue.
The present invention also provides an adaptive delay consuming apparatus comprising: an event collection module for: acquiring an event to be consumed, and adding the event to be consumed into a task pool; the task distribution module is used for: detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; event consuming module for: and taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the adaptive delay consumption method as described in any of the above when executing the program.
The present invention also provides a computer program product comprising computer programs/instructions which when executed by a processor implement the steps of the adaptive delay consuming method as described in any of the above.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the adaptive delay consumption method as described in any of the above.
According to the self-adaptive delay consuming method, the program product and the storage medium, the event to be consumed is acquired and added into the task pool, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, the event to be consumed in the task pool is stored into the task queue at the task distribution speed, the event to be consumed is taken out from the task queue, the event to be consumed is consumed, the data synchronization is completed, the master-slave delay time of the database can be automatically detected, the synchronization speed is adaptively adjusted according to the master-slave delay degree, the system stability is improved, the timeliness is high, and the manpower consumption is saved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of an adaptive delay consuming method provided by the invention;
FIG. 2 is a schematic diagram of an adaptive delay consuming device according to the present invention;
fig. 3 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The adaptive delay consuming method, program product and storage medium of the present invention are described below in conjunction with fig. 1-3.
Fig. 1 is a schematic flow chart of the adaptive delay consumption method provided by the invention. As shown in fig. 1, the method includes:
step 101, obtaining an event to be consumed, and adding the event to be consumed into a task pool.
In a data synchronization system, an upstream system sends a consumption message to a downstream system, which is typically stored in a message queue. The event to be consumed may be obtained from a message queue, the event to be consumed including a message that has not yet been consumed. After the event to be consumed is acquired, the event to be consumed is added into a task pool for subsequent processing.
Step 102, detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed.
The master-slave delay time of the database system is detected, for example, the master-slave delay time of the database system is detected according to a preset time period. The task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, the master-slave delay time does not exceed the preset maximum delay time, the task distribution speed can be adjusted according to the magnitude relation between the master-slave delay time and the preset maximum delay time, and the events to be consumed in the task pool are stored in the task queue based on the task distribution speed, so that task distribution processing is realized.
And step 103, taking out the event to be consumed from the task queue, and performing consumption processing on the event to be consumed.
And carrying out consumption processing on the event to be consumed to realize data synchronization of the upstream system and the downstream system. And when the consumption is carried out, the event to be consumed is taken out from the task queue, and the event to be consumed is consumed. Adjusting the task distribution speed of the task queue is equivalent to adjusting the consumption speed, namely adjusting the data synchronization speed. And distributing the consumption events in the task pool based on the task distribution speed, consuming after the task distribution is completed, and storing a database after the task distribution is completed. And generating master-slave delay time in the data storage process, and obtaining task distribution speed according to the relationship between the master-slave delay time and the preset maximum delay time. Therefore, a closed loop of control can be formed, so that a data synchronization process (consumption process) and a database storage process have a constraint relationship, and the consumption progress can be adaptively adjusted, so that master-slave delay is optimized.
According to the self-adaptive delay consumption method, the event to be consumed is obtained and added into the task pool, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, the event to be consumed in the task pool is stored into the task queue at the task distribution speed, the event to be consumed is taken out from the task queue, and the event to be consumed is consumed, so that the data synchronization is completed, the time of master-slave delay of the database can be automatically detected, the synchronization speed is adjusted according to the master-slave delay degree, the stability of the system is improved, the timeliness is high, and the manpower consumption is saved.
According to the self-adaptive delay consumption method provided by the invention, the master-slave delay time of the database system is detected, and the method specifically comprises the following steps: updating a preset database record in the main database, and storing the current updating time in a preset field; inquiring a field value of the preset field obtained from a database, and calculating an absolute value of a difference value between the current updating time and the field value, thereby obtaining the master-slave delay time.
The synchronous storage of the master database and the slave database can be realized in a mode of the prior art. The data and fields stored in the master and slave databases are identical. The master-slave delay time of the database system can be detected at regular time. When the master-slave delay time of the database system is detected, a preset database record in the master database can be updated, and the current update time is stored in a preset field. If the master database is updated, the slave database is updated similarly, and the data is stored in the same field, namely the preset field. Thus, after the update of the master database is completed, the slave database is queried for a field value of a preset field, which indicates the update time of a preset database record in the slave database. Since the update may not have been done up to date from the database at this time, the field value will be less than the current update time. If the update has been made up-to-date from the database, the field value will be greater than the current update time. Whether the field value of the slave database is larger or smaller than the current update time, the absolute value of the difference between the current update time and the field value of the preset field in the slave database can reflect the master-slave delay time, and therefore the absolute value of the difference between the current update time and the field value is calculated, and the master-slave delay time is obtained.
The preset database record may be a database record written in advance by newly building a table in the master database and the slave database.
According to the self-adaptive delay consumption method, the preset database record in the main database is updated, and the current update time is stored in the preset field; inquiring a field value of a preset field obtained from a database, and calculating the absolute value of the difference between the current update time and the field value, so that the master-slave delay time is obtained, and the quick and reliable obtaining of the master-slave delay time is realized.
According to the self-adaptive delay consumption method provided by the invention, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, and the self-adaptive delay consumption method specifically comprises the following steps: if the master-slave delay time is greater than or equal to the preset maximum delay time, reducing the task distribution speed until the task distribution speed reaches a preset minimum task distribution speed; and if the master-slave delay time is smaller than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed.
A preset maximum task distribution speed and a preset minimum task distribution speed may be preset to represent a maximum limit and a minimum limit of task distribution, that is, the task distribution speed should be between the preset minimum task distribution speed and the preset maximum task distribution speed.
When the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, if the master-slave delay time is greater than or equal to the preset maximum delay time, the task distribution speed is reduced until the task distribution speed reaches the preset minimum task distribution speed, and the mode of reducing the task distribution speed can be set according to actual needs. If the master-slave delay time is smaller than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed, wherein the mode of increasing the task distribution speed can be set according to actual needs.
According to the self-adaptive delay consumption method, the task distribution speed is reduced until the task distribution speed reaches the preset minimum task distribution speed by the fact that the master-slave delay time is larger than or equal to the preset maximum delay time; and when the master-slave delay time is smaller than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed, ensuring that the task distribution speed is within a preset interval range, and balancing the task distribution speed and the master-slave delay time, thereby balancing the consumption progress and the master-slave delay.
According to the self-adaptive delay consumption method provided by the invention, the initial value of the task distribution speed is set as the preset maximum task distribution speed.
Setting an initial value (starting speed in one period) of the task distribution speed as a preset maximum task distribution speed, and if the master-slave delay time is smaller than the maximum delay time, performing task distribution at the preset maximum task distribution speed, thereby improving the event consumption efficiency on the basis of meeting the master-slave delay requirement.
According to the self-adaptive delay consumption method, the initial value of the task distribution speed is set to be the preset maximum task distribution speed, so that the event consumption efficiency is improved on the basis of meeting the master-slave delay requirement.
According to the self-adaptive delay consumption method provided by the invention, the task distribution speed is reduced, and the self-adaptive delay consumption method specifically comprises the following steps: and reducing the task distribution speed to 1/2 of the original speed.
When the master-slave delay time is greater than or equal to the preset maximum delay time and the task distribution speed is reduced, the task distribution speed can be reduced to 1/2 of the original task distribution speed. For example, if the task distribution speed before adjustment is ns, ns is adjusted to ns/2. And then, calculating master-slave delay time again under the task distribution speed, and if the master-slave delay time is still greater than or equal to the preset maximum delay time, adjusting ns/2 to ns/4, and the like until the task distribution speed reaches the preset minimum task distribution speed.
According to the invention, when the task distribution speed is reduced, the task distribution speed is reduced to 1/2 of the original task distribution speed, so that the simplicity of task distribution speed adjustment is improved.
According to the self-adaptive delay consumption method provided by the invention, the task distribution speed is increased, and the self-adaptive delay consumption method specifically comprises the following steps: and increasing the task distribution speed by 2 times.
When the master-slave delay time is smaller than the preset maximum delay time and the task distribution speed is increased, the task distribution speed can be increased to 2 times of the original task distribution speed. For example, if the task distribution speed before adjustment is ns, ns is adjusted to 2ns. And then calculating master-slave delay time again under the task distribution speed, if the master-slave delay time is still smaller than the preset maximum delay time, adjusting 2ns to 4ns, and the like until the task distribution speed reaches the preset maximum task distribution speed.
When the task distribution speed is increased, the task distribution speed is increased by 2 times, so that the simplicity of task distribution speed adjustment is improved.
According to the self-adaptive delay consumption method provided by the invention, the acquisition of the event to be consumed specifically comprises the following steps: and acquiring the event to be consumed from the kafka message queue.
The kafka consumption event may be utilized and may be retrieved from the kafka message queue when retrieved.
According to the self-adaptive delay consumption method provided by the invention, the event to be consumed is acquired from the kafka message queue, so that the convenience of event consumption is improved.
The following describes the flow of the adaptive delay consumption method according to the present invention by way of a specific example. The self-adaptive delay consumption method provided by the invention comprises the following steps:
1. acquiring an upstream event to be consumed, and adding the event to be consumed into a task pool, for example: and starting an event collection process, acquiring an event to be consumed from a message queue such as kafka and adding the event to be consumed into a task pool.
2. Detecting the time of master-slave delay of a database system, such as: starting a monitoring process, updating one record in the master library at regular time, storing the updated time in an mtime field, inquiring the mtime field value T of the record from the slave library, and calculating the absolute value of the difference between the current time and T, namely the master-slave delay time T.
3. The tasks in the task pool are put into a task queue, and the task distribution speed is automatically adjusted according to the master-slave delay time of the database, for example: presetting a maximum distribution speed max, a minimum distribution speed min, starting a task distribution process, setting the starting speed to ns=max, putting tasks in a task pool into a task queue according to ns, and starting circulation, wherein if master-slave delay time T > =maximum delay time mt, ns=ns/2, until the minimum distribution speed min is reached; if T < mt, ns=2ns, until the maximum dispensing speed max is reached.
4. Acquiring an event from the task queue, and performing corresponding processing on the event, for example: and starting a consumer process, taking out the task from the task queue, and landing the data by the consumer task.
The adaptive delay consuming device provided by the invention is described below, and the adaptive delay consuming device described below and the adaptive delay consuming method described above can be referred to correspondingly.
Fig. 2 is a schematic structural diagram of an adaptive delay consuming device according to the present invention. As shown in fig. 2, the apparatus includes: an event collection module 10, a task distribution module 20, and an event consumption module 30, wherein: the event collection module 10 is configured to: acquiring an event to be consumed, and adding the event to be consumed into a task pool; the task distribution module 20 is configured to: detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; the event consuming module 30 is configured to: and taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed.
According to the self-adaptive delay consumption method, the event to be consumed is obtained and added into the task pool, the task distribution speed is adjusted according to the relation between the master-slave delay time and the preset maximum delay time, the event to be consumed in the task pool is stored into the task queue at the task distribution speed, the event to be consumed is taken out from the task queue, and the event to be consumed is consumed, so that the data synchronization is completed, the time of master-slave delay of the database can be automatically detected, the synchronization speed is adjusted according to the master-slave delay degree, the stability of the system is improved, the timeliness is high, and the manpower consumption is saved.
According to the present invention, when the task distribution module 20 is configured to detect the master-slave delay time of the database system, the task distribution module is specifically configured to: updating a preset database record in the main database, and storing the current updating time in a preset field; inquiring a field value of the preset field obtained from a database, and calculating an absolute value of a difference value between the current updating time and the field value, thereby obtaining the master-slave delay time.
According to the self-adaptive delay consumption method, the preset database record in the main database is updated, and the current update time is stored in the preset field; inquiring a field value of a preset field obtained from a database, and calculating the absolute value of the difference between the current update time and the field value, so that the master-slave delay time is obtained, and the quick and reliable obtaining of the master-slave delay time is realized.
According to the adaptive delay consuming device provided by the present invention, the task distribution module 20 is specifically configured to: if the master-slave delay time is greater than or equal to the preset maximum delay time, reducing the task distribution speed until the task distribution speed reaches a preset minimum task distribution speed; and if the master-slave delay time is smaller than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed.
According to the self-adaptive delay consumption method, the task distribution speed is reduced until the task distribution speed reaches the preset minimum task distribution speed by the fact that the master-slave delay time is larger than or equal to the preset maximum delay time; and when the master-slave delay time is smaller than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed, ensuring that the task distribution speed is within a preset interval range, and balancing the task distribution speed and the master-slave delay time, thereby balancing the consumption progress and the master-slave delay.
According to the self-adaptive delay consumption device provided by the invention, the initial value of the task distribution speed is set to be the preset maximum task distribution speed.
According to the self-adaptive delay consumption method, the initial value of the task distribution speed is set to be the preset maximum task distribution speed, so that the event consumption efficiency is improved on the basis of meeting the master-slave delay requirement.
According to the invention, the adaptive delay consumption device for reducing the task distribution speed comprises the following specific steps: and reducing the task distribution speed to 1/2 of the original speed.
According to the invention, when the task distribution speed is reduced, the task distribution speed is reduced to 1/2 of the original task distribution speed, so that the simplicity of task distribution speed adjustment is improved.
According to the present invention, the adaptive delay consuming device increases the task distribution speed, and specifically includes: and increasing the task distribution speed by 2 times.
When the task distribution speed is increased, the task distribution speed is increased by 2 times, so that the simplicity of task distribution speed adjustment is improved.
According to the self-adaptive delay consumption device provided by the invention, the acquisition of the event to be consumed specifically comprises the following steps: and acquiring the event to be consumed from the kafka message queue.
According to the self-adaptive delay consumption method provided by the invention, the event to be consumed is acquired from the kafka message queue, so that the convenience of event consumption is improved.
Fig. 3 is a schematic structural diagram of an electronic device provided by the present invention, and as shown in fig. 3, the electronic device may include: processor 310, communication interface (Communications Interface) 320, memory 330 and communication bus 340, wherein processor 310, communication interface 320, memory 330 accomplish communication with each other through communication bus 340. The processor 310 may invoke logic instructions in the memory 330 to perform an adaptive delay consuming method comprising: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed.
Further, the logic instructions in the memory 330 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the adaptive delay consumption method provided by the above methods, the method comprising: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the adaptive delay consumption methods provided above, the method comprising: acquiring an event to be consumed, and adding the event to be consumed into a task pool; detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed; and taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (7)
1. An adaptive delayed consumption method, comprising:
acquiring an event to be consumed, and adding the event to be consumed into a task pool;
detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed;
taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed; the detecting the master-slave delay time of the database system specifically comprises the following steps:
updating a preset database record in the main database, and storing the current updating time in a preset field; inquiring a field value of the preset field obtained from a database, and calculating an absolute value of a difference value between the current updating time and the field value, so as to obtain the master-slave delay time; the task distribution speed adjusting method according to the relation between the master-slave delay time and the preset maximum delay time specifically comprises the following steps:
if the master-slave delay time is greater than or equal to the preset maximum delay time, reducing the task distribution speed until the task distribution speed reaches a preset minimum task distribution speed;
and if the master-slave delay time is smaller than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed.
2. The adaptive delay consuming method of claim 1 wherein an initial value of the task distribution speed is set to the preset maximum task distribution speed.
3. The adaptive delay consuming method of claim 2, wherein said reducing the task distribution speed comprises:
and reducing the task distribution speed to 1/2 of the original speed.
4. The adaptive delay consuming method of claim 2, wherein said increasing the task distribution speed comprises:
and increasing the task distribution speed by 2 times.
5. The adaptive delay consuming method of claim 1, wherein the acquiring the event to be consumed specifically comprises:
and acquiring the event to be consumed from the kafka message queue.
6. An adaptive delay consuming device comprising:
an event collection module for: acquiring an event to be consumed, and adding the event to be consumed into a task pool;
the task distribution module is used for: detecting master-slave delay time of a database system, adjusting task distribution speed according to the relation between the master-slave delay time and preset maximum delay time, and storing the events to be consumed in the task pool into a task queue at the task distribution speed;
event consuming module for: taking out the event to be consumed from the task queue, and carrying out consumption processing on the event to be consumed;
the task distribution module is used for detecting master-slave delay time of the database system, and is specifically used for: updating a preset database record in the main database, and storing the current updating time in a preset field; inquiring a field value of the preset field obtained from a database, and calculating an absolute value of a difference value between the current updating time and the field value, so as to obtain the master-slave delay time;
the task distribution module is used for adjusting the task distribution speed according to the relation between the master-slave delay time and the preset maximum delay time, and is specifically used for: if the master-slave delay time is greater than or equal to the preset maximum delay time, reducing the task distribution speed until the task distribution speed reaches a preset minimum task distribution speed; and if the master-slave delay time is smaller than the preset maximum delay time, increasing the task distribution speed until the task distribution speed reaches the preset maximum task distribution speed.
7. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of the adaptive delay consumption method according to any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110672428.0A CN113590708B (en) | 2021-06-17 | 2021-06-17 | Adaptive delay consuming method, program product and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110672428.0A CN113590708B (en) | 2021-06-17 | 2021-06-17 | Adaptive delay consuming method, program product and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113590708A CN113590708A (en) | 2021-11-02 |
| CN113590708B true CN113590708B (en) | 2024-02-20 |
Family
ID=78243851
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110672428.0A Active CN113590708B (en) | 2021-06-17 | 2021-06-17 | Adaptive delay consuming method, program product and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113590708B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106648904A (en) * | 2017-01-09 | 2017-05-10 | 大连理工大学 | Self-adaptive rate control method for stream data processing |
| CN109298951A (en) * | 2018-08-21 | 2019-02-01 | 平安科技（深圳）有限公司 | The execution method and terminal device of Celery task |
| CN109743137A (en) * | 2019-01-10 | 2019-05-10 | 浙江小泰科技有限公司 | A kind of Distributed Delay message queue processing system for supporting to update |
| CN110012039A (en) * | 2018-01-04 | 2019-07-12 | 华北电力大学 | Task distribution and power control scheme in a kind of car networking based on ADMM |
| CN110196761A (en) * | 2019-04-15 | 2019-09-03 | 北京达佳互联信息技术有限公司 | Delay task processing method and processing device |
| CN110636130A (en) * | 2019-09-23 | 2019-12-31 | 上海钧正网络科技有限公司 | Delayed message processing method and device, computer equipment and storage medium |
| CN111796946A (en) * | 2020-06-22 | 2020-10-20 | 深圳壹账通智能科技有限公司 | Data processing method, device, equipment and storage medium |
| CN111813513A (en) * | 2020-06-24 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Real-time task scheduling method, device, equipment and medium based on distribution |
| CN112073526A (en) * | 2020-09-14 | 2020-12-11 | 苏州颂康智能科技有限公司 | Distributed IOT remote monitoring system based on 5G |
| CN112104519A (en) * | 2020-08-06 | 2020-12-18 | 北京健康之家科技有限公司 | Delayed message delivery method and device, storage medium and computer equipment |
| CN112612607A (en) * | 2020-12-16 | 2021-04-06 | 平安消费金融有限公司 | Data processing method, device, computer equipment and medium based on message queue |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8631411B1 (en) * | 2009-07-21 | 2014-01-14 | The Research Foundation For The State University Of New York | Energy aware processing load distribution system and method |
| US10284389B2 (en) * | 2015-10-21 | 2019-05-07 | Arris Enterprises Llc | High availability for distributed network services in an extended bridge |
| US10977260B2 (en) * | 2016-09-26 | 2021-04-13 | Splunk Inc. | Task distribution in an execution node of a distributed execution environment |
-
2021
- 2021-06-17 CN CN202110672428.0A patent/CN113590708B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106648904A (en) * | 2017-01-09 | 2017-05-10 | 大连理工大学 | Self-adaptive rate control method for stream data processing |
| CN110012039A (en) * | 2018-01-04 | 2019-07-12 | 华北电力大学 | Task distribution and power control scheme in a kind of car networking based on ADMM |
| CN109298951A (en) * | 2018-08-21 | 2019-02-01 | 平安科技（深圳）有限公司 | The execution method and terminal device of Celery task |
| CN109743137A (en) * | 2019-01-10 | 2019-05-10 | 浙江小泰科技有限公司 | A kind of Distributed Delay message queue processing system for supporting to update |
| CN110196761A (en) * | 2019-04-15 | 2019-09-03 | 北京达佳互联信息技术有限公司 | Delay task processing method and processing device |
| CN110636130A (en) * | 2019-09-23 | 2019-12-31 | 上海钧正网络科技有限公司 | Delayed message processing method and device, computer equipment and storage medium |
| CN111796946A (en) * | 2020-06-22 | 2020-10-20 | 深圳壹账通智能科技有限公司 | Data processing method, device, equipment and storage medium |
| CN111813513A (en) * | 2020-06-24 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Real-time task scheduling method, device, equipment and medium based on distribution |
| CN112104519A (en) * | 2020-08-06 | 2020-12-18 | 北京健康之家科技有限公司 | Delayed message delivery method and device, storage medium and computer equipment |
| CN112073526A (en) * | 2020-09-14 | 2020-12-11 | 苏州颂康智能科技有限公司 | Distributed IOT remote monitoring system based on 5G |
| CN112612607A (en) * | 2020-12-16 | 2021-04-06 | 平安消费金融有限公司 | Data processing method, device, computer equipment and medium based on message queue |
Non-Patent Citations (2)
| Title |
|---|
| "Optimal Scheduling of VMs in Queueing Cloud Computing Systems With a Heterogeneous Workload";Mian Guo;《IEEE Access ( Volume: 6)》;第15178-15191页 * |
| "互联网延迟队列解决方案设计";李祎 等;《福建电脑》;第9-12页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113590708A (en) | 2021-11-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109102245B (en) | Processing method, system and device for approval process | |
| CN108509603B (en) | A kind of adaptive dynamic dispatching method and system of data warehouse | |
| US20140281038A1 (en) | Terminal and application synchronization method thereof | |
| CN113590708B (en) | Adaptive delay consuming method, program product and storage medium | |
| CN114492834A (en) | Training method, training device, training equipment, training system and storage medium | |
| CN113141410A (en) | Dynamically adjusted QPS control method, system, device and storage medium | |
| CN116258197B (en) | Distributed training acceleration method and system based on parameter calculation and communication scheduling | |
| CN110888739B (en) | Distributed processing method and device for delayed tasks | |
| CN116501465A (en) | Automatic driving task execution method and device, electronic equipment and storage medium | |
| CN115129429B (en) | Container application management method and device, electronic equipment and storage medium | |
| CN114422440B (en) | Gray scale distribution method and device, electronic equipment and storage medium | |
| CN110781387A (en) | Pushing method of stock market snapshot, stock trading system and storage medium | |
| CN110555600A (en) | method, device, equipment and storage medium for preventing forbidden components in modular BOM from being misused | |
| WO2017206620A1 (en) | Method and apparatus for setting state of application program | |
| CN111381932B (en) | Method, device, electronic equipment and storage medium for triggering application program change | |
| CN113703964A (en) | Memory management method and device | |
| CN110673942B (en) | Asynchronous process ending method and device of cloud platform and cloud platform server | |
| CN103118072B (en) | A kind of method and system of database update | |
| US9847919B2 (en) | Data network device discovery optimization to reduce data transfer volume | |
| CN113377861B (en) | Reconstruction method, device, equipment and storage medium of distributed storage system | |
| CN112199201B (en) | Delayed task processing method, device and equipment | |
| CN109273122A (en) | System parameter method of adjustment, device and terminal device under ELPO operating condition | |
| CN113704022B (en) | Periodic asynchronous remote copying method, device, equipment and storage medium | |
| CN106921646B (en) | Service processing method and device | |
| CN115237968A (en) | Node management method, device, equipment and storage medium in database system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20231128 Address after: Room 102, 1st Floor, Building 1, No. 2 Chuangye Road, Haidian District, Beijing, 100080 Applicant after: Seashell Housing (Beijing) Technology Co.,Ltd. Address before: 101309 room 24, 62 Farm Road, Erjie village, Yangzhen, Shunyi District, Beijing Applicant before: Beijing fangjianghu Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |