CN116166451B - Dynamic adjustment method, system and device for topic number and storage medium - Google Patents
Dynamic adjustment method, system and device for topic number and storage medium Download PDFInfo
- Publication number
- CN116166451B CN116166451B CN202211586160.XA CN202211586160A CN116166451B CN 116166451 B CN116166451 B CN 116166451B CN 202211586160 A CN202211586160 A CN 202211586160A CN 116166451 B CN116166451 B CN 116166451B
- Authority
- CN
- China
- Prior art keywords
- topics
- speed
- adjustment
- index
- backlog data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 230000008602 contraction Effects 0.000 claims abstract description 18
- 230000005540 biological transmission Effects 0.000 claims description 10
- 230000006870 function Effects 0.000 description 11
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a method, a system, a device and a storage medium for dynamically adjusting the number of topics. The method comprises the following steps: acquiring a speed index and a backlog data volume index of a theme; obtaining the adjustment quantity of the number of subjects through a preset adjustment strategy according to the speed index and the backlog data quantity index; the adjustment quantity is used for representing the expansion quantity or the contraction quantity for adjusting the quantity of the subjects; and performing expansion and contraction operation on the number of the topics according to the adjustment quantity of the number of the topics, so as to realize dynamic adjustment of the number of the topics. According to the embodiment of the application, the adjustment quantity of the number of the topics is calculated through the speed index and the backlog data quantity index of the message topics, so that the number of the topics can be dynamically adjusted, the dynamic balance of the number of the topics is realized, the delay and backlog phenomena of data are relieved, and the customer experience is improved. The method can be widely applied to the technical field of computers.
Description
Technical Field
The application relates to the technical field of computers, in particular to a method, a system, a device and a storage medium for dynamically adjusting the number of topics.
Background
In the related art, when a service consumes and transmits a message in a message queue, the number of corresponding message topics is generally fixed, and when the message volume increases, the throughput increases, and a large amount of data synchronization link delay and backlog are easily generated.
Disclosure of Invention
The present application aims to solve at least one of the technical problems existing in the prior art to a certain extent.
It is therefore an object of the present application to provide a method, system, apparatus and storage medium for fast and accurate dynamic adjustment of the number of topics.
In order to achieve the technical purpose, the technical scheme adopted by the embodiment of the application comprises the following steps:
in one aspect, an embodiment of the present application provides a method for dynamically adjusting the number of topics, including the following steps:
the method for dynamically adjusting the number of the topics in the embodiment of the application comprises the following steps: acquiring a speed index and a backlog data volume index of a theme; obtaining the adjustment quantity of the number of subjects through a preset adjustment strategy according to the speed index and the backlog data quantity index; the adjustment quantity is used for representing the expansion quantity or the contraction quantity for adjusting the quantity of the subjects; and performing expansion and contraction operation on the number of the topics according to the adjustment quantity of the number of the topics, so as to realize dynamic adjustment of the number of the topics. According to the method and the device for predicting the number of the topics, the number of the topics can be dynamically adjusted through the speed index and the backlog data quantity index of the message topics, dynamic balance of the number of the topics is achieved, delay and backlog of data are relieved, and customer experience is improved.
In addition, the method for dynamically adjusting the number of topics according to the embodiment of the application can also have the following additional technical characteristics:
further, in the method for dynamically adjusting the number of topics according to the embodiment of the present application, the speed index includes a sending speed and a consuming speed, the backlog data volume index includes a sending backlog data volume and a consuming backlog data volume, and the obtaining, according to the speed index and the backlog data volume index, an adjustment amount of the number of topics through a preset adjustment policy includes:
obtaining a first difference value according to the consumption backlog data volume and the transmission backlog data volume; the first difference is the difference between the consumption backlog data amount and the transmission backlog data amount;
obtaining a first numerical value according to the first difference value and the consumption speed; the first value is a quotient of the first difference and the consumption speed;
obtaining a second difference value according to the sending speed and the consumption speed; the second difference is the difference between the sending speed and the consumption speed;
obtaining a second numerical value according to the second difference value and the consumption speed, wherein the second numerical value is the quotient of the second difference value and the consumption speed;
obtaining the adjustment quantity of the number of subjects according to the first numerical value and the second numerical value; the amount of adjustment of the number of subjects is the sum of the first value and the second value.
Further, in an embodiment of the present application, the scaling operation is performed on the number of topics according to the adjustment amount of the number of topics, including:
if the adjustment quantity is the expansion quantity, increasing the consumer group to consume;
and if the adjustment quantity is the contraction quantity, reducing the message sending quantity of the corresponding theme.
Further, in one embodiment of the present application, the scaling operation is performed on the number of topics according to the adjustment amount of the number of topics, including the following steps:
if the adjustment quantity is the expansion quantity and the number of topics is smaller than a first threshold value, creating new topics to realize dynamic adjustment of the number of topics; the first threshold is used for representing the number of topics allowed by the corresponding cluster;
if the adjustment quantity is the expansion quantity and the number of the topics is larger than a first threshold value, creating a new cluster, and dynamically adjusting the number of the topics through the new cluster.
Further, in one embodiment of the present application, the method further comprises:
if the new cluster is successfully created, a new theme is created in the new cluster.
Further, in an embodiment of the present application, the acquiring the speed index and the backlog data volume index of the subject includes:
acquiring accumulated time length;
and if the accumulated time length is equal to the preset time length, acquiring a speed index and a backlog data volume index of the theme, and recalculating the accumulated time length.
Further, in an embodiment of the present application, before the step of obtaining the adjustment amount of the number of subjects according to the speed index and the backlog data amount index by presetting an adjustment policy, the method includes:
judging whether the backlog data quantity index is larger than a second threshold value or not; the second threshold is used to characterize the amount of backlog data allowed.
In another aspect, an embodiment of the present application provides a system for dynamically adjusting the number of topics, including:
the first module is used for acquiring the speed index and the backlog data volume index of the theme;
the second module is used for obtaining the adjustment quantity of the number of subjects through a preset adjustment strategy according to the speed index and the backlog data quantity index; the adjustment quantity is used for representing the expansion quantity or the contraction quantity for adjusting the quantity of the subjects;
and the third module is used for performing expansion and contraction operation on the number of the topics according to the adjustment quantity of the number of the topics, so as to realize dynamic adjustment on the number of the topics.
In another aspect, an embodiment of the present application provides a device for dynamically adjusting the number of topics, including:
at least one processor;
at least one memory for storing at least one program;
the at least one program, when executed by the at least one processor, causes the at least one processor to implement any of the methods of dynamically adjusting the number of topics described above.
In another aspect, embodiments of the present application provide a storage medium having stored therein a processor-executable program which, when executed by a processor, is configured to implement a method for dynamically adjusting the number of any of the above-described topics.
According to the embodiment of the application, the adjustment quantity of the number of the topics is predicted through the speed index and the backlog data quantity index of the message topics, so that the number of the topics can be dynamically adjusted, the dynamic balance of the number of the topics is realized, the delay and backlog phenomena of data are relieved, and the customer experience is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following description is made with reference to the accompanying drawings of the embodiments of the present application or the related technical solutions in the prior art, and it should be understood that the drawings in the following description are only for convenience and clarity of describing some embodiments in the technical solutions of the present application, and other drawings may be obtained according to these drawings without the need of inventive labor for those skilled in the art.
FIG. 1 is a schematic flow chart of an embodiment of a method for dynamically adjusting the number of topics provided by the present application;
FIG. 2 is a flow chart of another embodiment of a method for dynamically adjusting the number of topics provided by the present application;
FIG. 3 is a schematic diagram of one embodiment of a dynamic adjustment system for the number of topics provided by the present application;
FIG. 4 is a schematic diagram of an embodiment of a dynamic adjustment device for the number of subjects provided by the present application.
Detailed Description
Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the application. The step numbers in the following embodiments are set for convenience of illustration only, and the order between the steps is not limited in any way, and the execution order of the steps in the embodiments may be adaptively adjusted according to the understanding of those skilled in the art.
In the related art, when a service consumes and transmits messages, the number of corresponding message topics is usually fixed, and when the message quantity is increased, the throughput is increased, and a large amount of data synchronization link delay and backlog are easy to generate.
Based on this, an embodiment of the present application proposes a method, a system, an apparatus and a storage medium for dynamically adjusting the number of topics, and a method and a system for dynamically adjusting the number of topics according to an embodiment of the present application are described in detail below with reference to the accompanying drawings.
Referring to fig. 1, a method for dynamically adjusting the number of topics is provided in the embodiment of the present application, and the method for dynamically adjusting the number of topics in the embodiment of the present application may be applied to a terminal, a server, software running in a terminal or a server, etc. The terminal may be, but is not limited to, a tablet computer, a notebook computer, a desktop computer, etc. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, basic cloud computing services such as big data and artificial intelligent platforms. The method for dynamically adjusting the number of the topics in the embodiment of the application mainly comprises the following steps:
s110: acquiring a speed index and a backlog data volume index of a theme;
s120: obtaining the adjustment quantity of the number of subjects through a preset adjustment strategy according to the speed index and the backlog data quantity index; the adjustment quantity is used for representing the expansion quantity or the contraction quantity for adjusting the quantity of the subjects;
s130: and performing expansion and contraction operation on the number of the topics according to the adjustment quantity of the number of the topics, so as to realize dynamic adjustment of the number of the topics.
In the embodiment of the application, the speed index and the backlog data quantity index of the theme are obtained by collecting the monitoring data of a plurality of message queue clusters. According to the obtained speed index and backlog data quantity index of the topics, the adjustment quantity of the number of the topics is calculated through the preset adjustment strategy provided by the application, so that the number of the topics of the message queue is dynamically adjusted, the distribution accuracy is improved, and the resource utilization rate is improved.
Optionally, in the method for dynamically adjusting the number of topics in the embodiment of the present application, the speed index includes a sending speed and a consuming speed, the backlog data volume index includes a sending backlog data volume and a consuming backlog data volume, and the obtaining, according to the speed index and the backlog data volume index, an adjustment volume of the number of topics through a preset adjustment policy includes:
obtaining a first difference value according to the consumption backlog data volume and the transmission backlog data volume; the first difference is the difference between the consumption backlog data amount and the transmission backlog data amount;
obtaining a first numerical value according to the first difference value and the consumption speed; the first value is a quotient of the first difference and the consumption speed;
obtaining a second difference value according to the sending speed and the consumption speed; the second difference is the difference between the sending speed and the consumption speed;
obtaining a second numerical value according to the second difference value and the consumption speed, wherein the second numerical value is the quotient of the second difference value and the consumption speed;
obtaining the adjustment quantity of the number of subjects according to the first numerical value and the second numerical value; the amount of adjustment of the number of subjects is the sum of the first value and the second value.
In this step, the preset adjustment measurement provided by the embodiment of the present application may be calculated by the following formula: and (consumption backlog data quantity-sending backlog data quantity)/consumption speed+ (sending speed-consumption speed)/consumption speed, and calculating to obtain the adjustment quantity of the number of the topics, wherein the real-time adjustment of the number of the topics can be realized through the adjustment quantity, and the resource allocation efficiency is improved.
Optionally, in an embodiment of the present application, the scaling operation is performed on the number of topics according to the adjustment amount of the number of topics, including:
if the adjustment quantity is the expansion quantity, increasing the consumer group to consume;
and if the adjustment quantity is the contraction quantity, reducing the message sending quantity of the corresponding theme.
Referring to fig. 2, in one embodiment of the present application, if the adjustment amount is an expansion amount, an expansion operation is required for the number of subjects. Specifically, the lateral dynamic scaling service acquires the number of topics which need to be increased, namely the number of topics; creating a topic and submitting the topic to a control message queue, namely controlling the MQ; the issuing service and the consuming service acquire the topic event, if the event is the capacity expansion topic, the consumption of the corresponding consumer group is directly increased, and if the event is the reduction of the number of topic, the sending service stops sending the message towards the reduced topic, and the consuming service continues to consume. In some possible implementations, when the scaling is performed, the lateral dynamic scaling service determines whether the scaled-down topic has no message, based on the collected index data, and performs the deletion. And if the adjustment quantity is the contraction quantity and the corresponding theme has no message, deleting the corresponding theme.
Optionally, in an embodiment of the present application, the scaling operation is performed on the number of topics according to the adjustment amount of the number of topics, including:
if the adjustment quantity is the expansion quantity and the number of topics is smaller than a first threshold value, creating new topics to realize dynamic adjustment of the number of topics; the first threshold is used for representing the number of topics allowed by the corresponding cluster;
if the adjustment quantity is the expansion quantity and the number of the topics is larger than a first threshold value, creating a new cluster, and dynamically adjusting the number of the topics through the new cluster.
Optionally, in one embodiment of the present application, the method further includes:
if the new cluster is successfully created, a new theme is created in the new cluster.
In the embodiment of the application, for the expansion operation of the number of topics, if the number of topics can be expanded in the message queue cluster, the expansion and contraction service automatically creates one topic, then sends notification events to producers and consumers, and increases the creation of producer groups and consumer groups of the corresponding topics. Conversely, when the theme cannot be newly added in the MQ cluster, the MQ scaling service sends an operator operation k8s for creating the MQ cluster, and automatically creates the MQ cluster. In particular, the MQ scaling service periodically checks whether the cluster created by k8s is complete (illustratively, by taking the status of the corresponding pod and establishing a test topic, when sending a message, consuming a message is successful, it is considered that the cluster creation is successful). When the creation of the new MQ cluster is completed, a message broadcasting the completion of the creation of the new MQ is sent, and after the consumer and producer receive, the new MQ cluster is dynamically added and locally connected, and the TOPIC is dynamically created. Accordingly, by expanding multiple MQ clusters, the TOPIC previously performed at a single MQ cluster is dynamically expanded to a cross MQ cluster level (and the number of MQ clusters can be automatically scaled).
Optionally, in one embodiment of the present application, the acquiring the speed index and the backlog data volume index of the subject includes:
acquiring accumulated time length;
and if the accumulated time length is equal to the preset time length, acquiring a speed index and a backlog data volume index of the theme, and recalculating the accumulated time length.
Specifically, a person skilled in the art may perform the operation of obtaining the monitoring index by adding the time attribute, thereby executing the method for adjusting the number of topics provided in the embodiment of the present application. The setting of the time period may be real-time, a fixed local time period, or a non-fixed time period. According to the using frequency or importance degree of the system in different time periods, the time periods with different interval lengths are set, and diversified selections are provided while the system requirements are met.
Optionally, before the step of obtaining the adjustment quantity of the number of subjects according to the speed index and the backlog data quantity index through a preset adjustment strategy, the method further includes:
judging whether the backlog data quantity index is larger than a second threshold value or not; the second threshold is used to characterize the amount of backlog data allowed.
Specifically, a second threshold may be set, and when the backlog data amount indicator is greater than the second threshold, the data amount is indicated to be increased, and at this time, dynamic adjustment of the number of topics is performed to control the trend of increasing the data amount. Specifically, the backlog data amount index may adopt a transmission backlog data amount, or may adopt a consumption backlog data amount, or may adopt both the transmission backlog data amount and the consumption backlog data amount.
In summary, it can be known that the embodiment of the application predicts the adjustment amount of the number of the topics through the speed index and the backlog data amount index of the message topics, can dynamically adjust the number of the topics, realizes the dynamic balance of the number of the topics, is beneficial to relieving the delay and backlog phenomena of data and improves the customer experience.
Next, a dynamic adjustment system for the number of subjects proposed according to an embodiment of the present application will be described with reference to fig. 3.
FIG. 3 is a schematic diagram of a system for dynamically adjusting the number of subjects in accordance with an embodiment of the present application, the system comprising:
a first module 310, configured to obtain a speed indicator and a backlog data volume indicator of a topic;
a second module 320, configured to obtain an adjustment amount of the number of topics by presetting an adjustment policy according to the speed index and the backlog data amount index; the adjustment quantity is used for representing the expansion quantity or the contraction quantity for adjusting the quantity of the subjects;
and a third module 330, configured to perform scaling operation on the number of topics according to the adjustment amount of the number of topics, so as to dynamically adjust the number of topics.
It can be seen that the content in the above method embodiment is applicable to the system embodiment, and the functions specifically implemented by the system embodiment are the same as those of the method embodiment, and the beneficial effects achieved by the method embodiment are the same as those achieved by the method embodiment.
Referring to fig. 4, an embodiment of the present application provides a dynamic adjustment device for a number of topics, including:
at least one processor 410;
at least one memory 420 for storing at least one program;
the at least one program, when executed by the at least one processor 410, causes the at least one processor 410 to implement a method of dynamically adjusting the number of topics.
Similarly, the content in the above method embodiment is applicable to the embodiment of the present device, and the functions specifically implemented by the embodiment of the present device are the same as those of the embodiment of the above method, and the beneficial effects achieved by the embodiment of the above method are the same as those achieved by the embodiment of the above method.
The embodiment of the present application also provides a computer-readable storage medium in which a program executable by the processor 410 is stored, the program executable by the processor 410 being for performing the above-described dynamic adjustment method of the number of subjects when executed by the processor 410.
Similarly, the content in the above method embodiment is applicable to the present storage medium embodiment, and the specific functions of the present storage medium embodiment are the same as those of the above method embodiment, and the achieved beneficial effects are the same as those of the above method embodiment.
In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present application are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.
Furthermore, while the application is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the functions and/or features may be integrated in a single physical device and/or software module or may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present application. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the application as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the application, which is to be defined in the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in the form of a software product stored in a storage medium, including several programs for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable programs for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with a program execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the programs from the program execution system, apparatus, or device and execute the programs. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the program execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
In the foregoing description of the present specification, reference has been made to the terms "one embodiment/example", "another embodiment/example", "certain embodiments/examples", and the like, means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the application, the scope of which is defined by the claims and their equivalents.
While the preferred embodiment of the present application has been described in detail, the present application is not limited to the embodiments described above, and various equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the present application, and these equivalent modifications and substitutions are intended to be included in the scope of the present application as defined in the appended claims.
Claims (9)
1. A method for dynamically adjusting the number of subjects, comprising the steps of:
acquiring a speed index and a backlog data volume index of a theme;
obtaining the adjustment quantity of the number of subjects through a preset adjustment strategy according to the speed index and the backlog data quantity index; the adjustment quantity is used for representing the expansion quantity or the contraction quantity for adjusting the quantity of the subjects;
performing expansion and contraction operation on the number of topics according to the adjustment quantity of the number of topics, and dynamically adjusting the number of topics;
the speed index comprises a sending speed and a consumption speed, the backlog data volume index comprises a sending backlog data volume and a consumption backlog data volume, and the adjusting quantity of the number of subjects is obtained through a preset adjusting strategy according to the speed index and the backlog data volume index, and the method comprises the following steps:
obtaining a first difference value according to the consumption backlog data volume and the transmission backlog data volume; the first difference is the difference between the consumption backlog data amount and the transmission backlog data amount;
obtaining a first numerical value according to the first difference value and the consumption speed; the first value is a quotient of the first difference and the consumption speed;
obtaining a second difference value according to the sending speed and the consumption speed; the second difference is the difference between the sending speed and the consumption speed;
obtaining a second numerical value according to the second difference value and the consumption speed, wherein the second numerical value is the quotient of the second difference value and the consumption speed;
obtaining the adjustment quantity of the number of subjects according to the first numerical value and the second numerical value; the amount of adjustment of the number of subjects is the sum of the first value and the second value.
2. The method for dynamically adjusting the number of topics according to claim 1, wherein the scaling the number of topics according to the adjustment amount of the number of topics comprises:
if the adjustment quantity is the expansion quantity, increasing the consumer group to consume;
and if the adjustment quantity is the contraction quantity, reducing the message sending quantity of the corresponding theme.
3. The method for dynamically adjusting the number of topics according to claim 1, wherein the scaling the number of topics according to the adjustment amount of the number of topics comprises:
if the adjustment quantity is the expansion quantity and the number of topics is smaller than a first threshold value, creating new topics to realize dynamic adjustment of the number of topics; the first threshold is used for representing the number of topics allowed by the corresponding cluster;
if the adjustment quantity is the expansion quantity and the number of the topics is larger than a first threshold value, creating a new cluster, and dynamically adjusting the number of the topics through the new cluster.
4. A method of dynamically adjusting the number of subjects according to claim 3, further comprising:
if the new cluster is successfully created, a new theme is created in the new cluster.
5. The method for dynamically adjusting the number of topics according to claim 1, wherein the step of obtaining the speed index and the backlog data amount index of the topics comprises the steps of:
acquiring accumulated time length;
and if the accumulated time length is equal to the preset time length, acquiring a speed index and a backlog data volume index of the theme, and recalculating the accumulated time length.
6. The method for dynamically adjusting the number of subjects according to claim 1, wherein before the step of obtaining the adjustment amount of the number of subjects by a preset adjustment strategy according to the speed index and the backlog data amount index, the method further comprises:
judging whether the backlog data quantity index is larger than a second threshold value or not; the second threshold is used to characterize the amount of backlog data allowed.
7. A system for dynamically adjusting the number of subjects, comprising:
the first module is used for acquiring the speed index and the backlog data volume index of the theme;
the second module is used for obtaining the adjustment quantity of the number of subjects through a preset adjustment strategy according to the speed index and the backlog data quantity index; the adjustment quantity is used for representing the expansion quantity or the contraction quantity for adjusting the quantity of the subjects;
the third module is used for performing expansion and contraction operation on the number of the topics according to the adjustment quantity of the number of the topics, so as to realize dynamic adjustment of the number of the topics;
the speed index includes a sending speed and a consuming speed, the backlog data volume index includes a sending backlog data volume and a consuming backlog data volume, and the second module is further configured to:
obtaining a first difference value according to the consumption backlog data volume and the transmission backlog data volume; the first difference is the difference between the consumption backlog data amount and the transmission backlog data amount;
obtaining a first numerical value according to the first difference value and the consumption speed; the first value is a quotient of the first difference and the consumption speed;
obtaining a second difference value according to the sending speed and the consumption speed; the second difference is the difference between the sending speed and the consumption speed;
obtaining a second numerical value according to the second difference value and the consumption speed, wherein the second numerical value is the quotient of the second difference value and the consumption speed;
obtaining the adjustment quantity of the number of subjects according to the first numerical value and the second numerical value; the amount of adjustment of the number of subjects is the sum of the first value and the second value.
8. A dynamic adjustment device for a number of subjects, comprising:
at least one processor;
at least one memory for storing at least one program;
the at least one program, when executed by the at least one processor, causes the at least one processor to implement the method of dynamically adjusting the number of topics as set forth in any of claims 1 to 6.
9. A computer-readable storage medium having stored therein a program executable by a processor, characterized in that: the processor executable program when executed by a processor is for implementing a method of dynamically adjusting the number of topics as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211586160.XA CN116166451B (en) | 2022-12-09 | 2022-12-09 | Dynamic adjustment method, system and device for topic number and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211586160.XA CN116166451B (en) | 2022-12-09 | 2022-12-09 | Dynamic adjustment method, system and device for topic number and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116166451A CN116166451A (en) | 2023-05-26 |
| CN116166451B true CN116166451B (en) | 2023-11-07 |
Family
ID=86410317
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211586160.XA Active CN116166451B (en) | 2022-12-09 | 2022-12-09 | Dynamic adjustment method, system and device for topic number and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116166451B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968430A (en) * | 2018-09-28 | 2020-04-07 | 北京京东尚科信息技术有限公司 | Message queue-based processing method and message queue |
| CN112612607A (en) * | 2020-12-16 | 2021-04-06 | 平安消费金融有限公司 | Data processing method, device, computer equipment and medium based on message queue |
| CN113138860A (en) * | 2020-01-17 | 2021-07-20 | 中国移动通信集团浙江有限公司 | Message queue management method and device |
| CN114168367A (en) * | 2021-12-09 | 2022-03-11 | 中国建设银行股份有限公司 | Method and system for solving queue backlog through batch tasks |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180260819A1 (en) * | 2017-03-09 | 2018-09-13 | Capital One Services, Llc | Systems and methods for real time message processing using an event driven framework |
-
2022
- 2022-12-09 CN CN202211586160.XA patent/CN116166451B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110968430A (en) * | 2018-09-28 | 2020-04-07 | 北京京东尚科信息技术有限公司 | Message queue-based processing method and message queue |
| CN113138860A (en) * | 2020-01-17 | 2021-07-20 | 中国移动通信集团浙江有限公司 | Message queue management method and device |
| CN112612607A (en) * | 2020-12-16 | 2021-04-06 | 平安消费金融有限公司 | Data processing method, device, computer equipment and medium based on message queue |
| CN114168367A (en) * | 2021-12-09 | 2022-03-11 | 中国建设银行股份有限公司 | Method and system for solving queue backlog through batch tasks |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116166451A (en) | 2023-05-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109947668B (en) | Method and device for storing data | |
| US7631034B1 (en) | Optimizing node selection when handling client requests for a distributed file system (DFS) based on a dynamically determined performance index | |
| CN101394307B (en) | Method, apparatus and system for on-line user amount statistic | |
| US20200351207A1 (en) | Method and system of limiting traffic | |
| CN104243405A (en) | Request processing method, device and system | |
| US20020174217A1 (en) | System and method for predicting network performance | |
| CN112445857A (en) | Resource quota management method and device based on database | |
| CN106933868A (en) | A kind of method and data server for adjusting data fragmentation distribution | |
| CN106791930B (en) | Video processing method and device | |
| CN115211092B (en) | Message pulling method and device and computer storage medium | |
| CN111858067A (en) | Data processing method and device | |
| CN104202305A (en) | Transcoding processing method and device, server | |
| CN106936867A (en) | The response method and device of a kind of service request | |
| CN114490078A (en) | Dynamic capacity reduction and expansion method, device and equipment for micro-service | |
| CN116166451B (en) | Dynamic adjustment method, system and device for topic number and storage medium | |
| CN111796935B (en) | Consumption instance distribution method and system for calling log information | |
| CN116055401A (en) | Message processing method, device, equipment and storage medium | |
| CN114143263B (en) | Method, equipment and medium for limiting current of user request | |
| CN112367384B (en) | Kafka cluster-based dynamic speed limiting method and device and computer equipment | |
| CN110838989A (en) | Method and device for network current limiting based on token | |
| CN112994934B (en) | Data interaction method, device and system | |
| CN111737297B (en) | Method and device for processing link aggregation call information | |
| CN101753607A (en) | Server and working method thereof | |
| CN110460663B (en) | Data distribution method and device among distributed nodes, server and storage medium | |
| CN113835905A (en) | Message queue load balancing method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |