CN117851045A - Resource scheduling method, device and medium for message data processing - Google Patents
Resource scheduling method, device and medium for message data processing Download PDFInfo
- Publication number
- CN117851045A CN117851045A CN202311724550.3A CN202311724550A CN117851045A CN 117851045 A CN117851045 A CN 117851045A CN 202311724550 A CN202311724550 A CN 202311724550A CN 117851045 A CN117851045 A CN 117851045A
- Authority
- CN
- China
- Prior art keywords
- message
- threads
- consumer
- server
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000012545 processing Methods 0.000 title claims abstract description 39
- 238000004519 manufacturing process Methods 0.000 claims abstract description 61
- 238000012544 monitoring process Methods 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 10
- 238000004891 communication Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000000903 blocking effect Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 241000234282 Allium Species 0.000 description 1
- 235000002732 Allium cepa var. cepa Nutrition 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000000843 powder Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a resource scheduling method, a device and a medium for message data processing. Creating a message queue according to preset configuration information; monitoring the message queue state, and calculating the message production capacity and the message consumption capacity required by meeting the message queue state according to the message queue state; and adjusting the number of the producer threads and the consumer threads according to the matched producer threads and consumer threads of the message production capacity and the message consumption capacity. The method and the device can dynamically expand the capacity of producers and consumers and rapidly process a large amount of data.
Description
Technical Field
The present invention relates to the field of message data processing technologies, and in particular, to a method, an apparatus, and a medium for scheduling resources for message data processing.
Background
The message queue is an infrastructure for constructing the distributed internet application, and the loose coupling architecture design can improve the usability and the expandability of the system, so that the method is an optimal design scheme suitable for modern applications.
The production and consumption cost capability of data mainly depends on the thread number set by an application program, when the production capability is low after a large amount of data is in a message, the messages are piled up, the consumption capability reaches the bottleneck, the asynchronous processing capability of a system is greatly reduced, the expansion of the production and consumption capability cannot be dynamically carried out, the large data amount cannot be rapidly processed, and the multi-core performance of a server cannot be fully utilized. The data is piled up for a large amount of time and even drags down the whole distributed system.
Disclosure of Invention
The invention provides a resource scheduling method, a device and a medium for message data processing, which are used for solving the technical problems that the existing message queue middleware can not dynamically expand the production and consumption capacity, can not rapidly process large data volume and can not fully utilize the multi-core performance of a server.
According to an aspect of the present invention, there is provided a resource scheduling method for message data processing, including:
creating a message queue according to preset configuration information;
monitoring the message queue state, and calculating the message production capacity and the message consumption capacity required by meeting the message queue state according to the message queue state;
and adjusting the number of the producer threads and the consumer threads according to the matched producer threads and consumer threads of the message production capacity and the message consumption capacity.
Optionally, the method further comprises: acquiring idle resource information of a current server;
said adjusting the number of said producer threads and said consumer threads according to said matched producer threads and said consumer threads of said message production capability and message consumption capability, comprising:
and under the limitation of the idle resources of the server, adjusting the number of the producer threads and the consumer threads according to the matched producer threads and consumer threads of the message production capacity and the message consumption capacity.
Optionally, the calculating the message production capability and the message consumption capability required to meet the message queue status according to the message queue status includes:
and calculating the currently required message production capacity and message consumption capacity according to the comparison result of the message data volume of the current message queue and a preset threshold value.
Optionally, said adjusting the number of said producer threads and said consumer threads according to said matched producer threads and said consumer threads of said message production capability and message consumption capability under the limitation of said server free resources includes:
if the thread number corresponding to the message consumption capability is smaller than the thread number of the idle resources of the server, increasing the consumer thread number corresponding to the message consumption capability;
and if the thread number corresponding to the message consumption capability is larger than the thread number of the idle resources of the server, taking the thread number of the idle resources of the server as the consumer thread.
Optionally, if the number of threads corresponding to the message consumption capability is smaller than the number of threads of the idle resources of the server, increasing the number of threads of the consumer corresponding to the message consumption capability; and if the number of threads corresponding to the message consumption capability is greater than the number of threads of the server idle resources, taking the number of threads of the server idle resources as the consumer threads, the method further comprises:
and if no accumulated message data in the message queue is detected, releasing the added consumer thread.
Optionally, the method further comprises:
when the message data volume in the message queue is monitored to be larger than an overload threshold value, increasing the server resource;
and increasing the number of the consumer threads for consumption of the message data according to the newly added server resources.
Optionally, the method further comprises:
and when no accumulated message data in the message queue is detected, releasing the consumer thread added from the newly added server resource.
According to another aspect of the present invention, there is provided a message data processing resource scheduling apparatus comprising:
the configuration module is used for creating a message queue according to preset configuration information;
the calculation module is used for monitoring the message queue state and calculating the message production capacity and the message consumption capacity required by meeting the message queue state according to the message queue state;
and the adjusting module is used for adjusting the number of the producer threads and the consumer threads according to the matched producer threads and consumer threads of the message production capacity and the message consumption capacity.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the resource scheduling method of message data processing according to any one of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute a resource scheduling method for message data processing according to any one of the embodiments of the present invention.
According to the technical scheme of the embodiment of the invention, the message production capacity and the message consumption capacity required by meeting the message queue state are calculated according to the message queue state by monitoring the message queue state; according to the producer threads and the consumer threads matched with the message production capacity and the message consumption capacity, the number of the producer threads and the number of the consumer threads are adjusted, so that the capacity of the producer and the consumer can be dynamically expanded, a large amount of data can be rapidly processed, and the multi-core performance of the server can be fully utilized.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a resource scheduling method for message data processing according to a first embodiment of the present invention;
fig. 2 is a flowchart of a resource scheduling method for message data processing according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a resource scheduling device for message data processing according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device implementing a resource scheduling method for message data processing according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For a better understanding of the skilled person, some of the terms of art are described as follows:
the producer: i.e., the provider, sender of the message, note that in the art, the nature here generally refers to a node rather than a natural person.
Message: may be data or management instructions, etc.
Message queues: message queues are communication means used to asynchronously process a series of incoming messages. The message queue may be implemented by a message queue server.
The consumer: i.e., the user, recipient of the message, note that in the art, the nature herein generally refers to a node rather than a natural person.
Thread: the minimum unit of operation scheduling of the operating system is contained in the process, is the actual operation unit in the process, and generally refers to a single-order control flow in the process, and multiple threads can be concurrent in one process, and each thread performs different tasks in parallel. Each piece of the behavior trace data and the task request sent in the embodiment of the application can be processed based on threads.
Example 1
Fig. 1 is a flowchart of a resource scheduling method for message data processing according to an embodiment of the present invention. As shown in fig. 1, the method includes:
s110, creating a message queue according to preset configuration information;
the scheduling system in the application can create a message queue according to the pre-configured configuration information, and the message queue can comprise one or more than one message queue. Wherein the configuration information is configured by a manager in advance according to the environment.
S120, monitoring the state of the message queue, and calculating the message production capacity and the message consumption capacity required by meeting the state of the message queue according to the state of the message queue;
the method and the device can monitor the state of the message queue in real time, and can monitor the state of the message queue at regular time. The method for regularly monitoring the state of the message queue can adopt a heartbeat monitoring method, a scheduling system sends a monitoring signal to the message queue every 5 seconds, and then the fed-back data containing the state of the message queue is obtained.
The status data of the message queue comprises the status of the current message queue, such as idle, normal, blocking, or data overload, wherein blocking refers to that the message data exceeds a normal value, and data overload refers to that the message data volume exceeds the system capacity; the status data of the message queue may also include the amount of message data contained in the current message queue.
The scheduling system may determine the current message production capability and message consumption capability based on the current state of the message queue. For example, when the state of the current message queue is idle, it indicates that the current message production capability is poor and the message consumption capability is strong, and it is necessary to increase the producer thread; when the state of the current message queue is normal, the current message production capacity and the message consumption capacity are balanced, and the number of producer threads and consumer threads is not required to be adjusted; when the current message queue is blocked, the current message has strong message production capacity and weak message consumption capacity, and a consumer thread needs to be added; when the state of the current message queue is data overload, the current message consumption capability is insufficient to digest the data in the current message queue, and server resources need to be added to the system to acquire more consumer threads for consumption of the data. In another embodiment, the current message production capability and message consumption capability, and the required message production capability and message consumption capability, may also be determined according to the data interval in which the message data amount included in the current message queue is located.
S130, adjusting the number of the producer threads and the consumer threads according to the matched producer threads and consumer threads of the message production capacity and the message consumption capacity.
The dispatching system can respectively initialize a producer thread and a consumer thread according to the information of the current message queue; and can obtain the initial message production capability and message consumption capability size. When the further required message production capacity and message consumption capacity is obtained according to the current message queue status, the producer or consumer thread may be added or new server resources may be added according to the required message production capacity and message consumption capacity.
According to the technical scheme of the embodiment of the invention, the message production capacity and the message consumption capacity required by meeting the message queue state are calculated according to the message queue state by monitoring the message queue state; according to the producer threads and the consumer threads matched with the message production capacity and the message consumption capacity, the number of the producer threads and the number of the consumer threads are adjusted, so that the capacity of the producer and the consumer can be dynamically expanded, a large amount of data can be rapidly processed, and the multi-core performance of the server can be fully utilized.
Example two
Fig. 2 is a flowchart of a resource scheduling method for message data processing according to a second embodiment of the present invention. As shown in fig. 2, the method includes:
s210, creating a message queue according to preset configuration information.
S220, acquiring idle resource information of the current server.
The scheduling system can acquire idle resource information of the current server in real time, for example, a current available thread of the server can be acquired, and the current available thread can be used for scheduling of the scheduling system, so that the green onion powder utilizes the multi-core performance of the server.
S230, calculating the currently required message production capacity and message consumption capacity according to the comparison result of the message data volume of the current message queue and a preset threshold value.
It should be noted that, the present application may monitor the message data amount of the message queue with timing by using the heartbeat monitoring method, for example, send a heartbeat monitoring signal to the message queue every 5 seconds, so as to calculate the current message data amount or the current state of the message queue according to the information fed back by the message queue.
The scheduling system can judge the current message production capacity and message consumption capacity and the required message production capacity and message consumption capacity according to the data interval of the message data quantity contained in the current message queue. For example, a plurality of data intervals may be set, and if the data interval of the message queue with the message data amount in the normal state is [ M-a, M), when the message data amount in the message queue is [ M, m+b), it indicates that the current message consumption capability is insufficient, and a consumer thread needs to be added; when the message data amount of the message queue is [ M, M+b), the current message consumption capability is insufficient, and one unit of message consumption capability needs to be added, and the one unit of message consumption capability can be assumed to be the data processing capability of one consumer thread; when the message data amount of the message queue is [ M, M+nb), indicating that the current message consumption capability is insufficient, and increasing the message consumption capability of n units, namely increasing n consumer threads, is needed; when the message data amount of the message queue is zero, the current production capacity is insufficient, the message production capacity of one unit can be increased, and the message production capacity of one unit can be assumed to be the data processing capacity of one producer thread. The present example is only one example, and the congestion state of the current message queue may be actually determined according to the comparison result of the message data amount and the preset threshold value, so as to determine the required message production capability and message consumption capability.
In one example, the currently required message production capability and message consumption capability may also be determined based on the state of the current message queue. For example, when the state of the current message queue is idle, it indicates that the current message production capability is poor and the message consumption capability is strong, and it is necessary to step up one producer thread with 1 as step size; when the state of the current message queue is normal, the current message production capacity and the message consumption capacity are balanced, and the number of producer threads and consumer threads is not required to be adjusted; when the current message queue is blocked, the current message has strong production capacity and weak message consumption capacity, and the consumer thread needs to be gradually increased by taking 1 as a step length; when the state of the current message queue is data overload, the current message consumption capability is insufficient to digest the data in the current message queue, and server resources need to be added to the system to acquire more consumer threads for consumption of the data. The step of gradually increasing one consumer thread by 1 step is represented by increasing one consumer thread when the current message production capability is strong and the message consumption capability is weak, and after a certain period of time, for example, after the duration of a heartbeat monitoring period of 5s, if the current message production capability is still monitored and the message consumption capability is weak, continuing to increase one consumer thread until the state of the current message queue is normal, and not increasing the consumer thread any more.
S240, under the limitation of the idle resources of the server, the number of the producer threads and the consumer threads is adjusted according to the matched producer threads and consumer threads of the message production capacity and the message consumption capacity.
It should be noted that, in the case of configured server resources, the available server resources are limited, that is, the number of producer threads or consumer threads cannot be increased without limitation, and the number of producer threads and consumer threads is adjusted according to the producer threads and consumer threads matched with the message production capability and the message consumption capability, which further includes:
s241, if the thread number corresponding to the message consumption capability is smaller than the thread number of the idle resources of the server, increasing the consumer thread number corresponding to the message consumption capability;
s242, if the number of threads corresponding to the message consumption capability is greater than the number of threads of the idle resources of the server, the number of threads of the idle resources of the server is used as the consumer thread.
Wherein, when the current message consumption capability is insufficient, a consumer thread needs to be added for consumption; still can assume that the message data amount of the message queue is the data interval of the normal state is [ M-a, M), then increase a consumer thread when the message data amount of the message queue is [ M, M+b); when the message data amount of the message queue is [ M, M+nb), indicating that the current message consumption capability is insufficient, adding n consumer threads is needed, wherein n cannot be larger than the thread number of the idle resources of the current server; when the message data amount of the message queue is zero, the current insufficient production capacity is indicated, the producer threads can be sequentially increased, and the increased producer threads should be smaller than the thread number of the idle resources of the current server.
S243, if no accumulated message data in the message queue is detected, releasing the added consumer thread.
It should be noted that, when it is detected that there is no message data accumulated in the message queue, the message throughput is smaller than the message consumption capability, and then all the consumer threads obtained from the thread number of the current server idle resource need to be released for other applications.
And S250, increasing the server resources when the message data volume in the message queue is monitored to be larger than an overload threshold value.
It should be noted that, when the message data amount in the current message queue is greater than the overload threshold or the state in the current message queue is data overload, that is, the message consumption capability of the server idle resource that can be scheduled by the current scheduling system is insufficient to digest the data in the current message queue, server resources need to be added to the system to obtain more consumer threads for consuming the data. The server resource is added by staff in the background management, or more server resources are configured in advance, and when the server resource needs to be added, the server which is configured in advance is accessed without modifying program codes.
And S260, increasing the number of the consumer threads for consumption of the message data according to the newly added server resources.
It should be noted that, for the increase of the newly added server resource, the consumer thread may be gradually increased by taking 1 as a step length, after a preset period, for example, after the duration of the heartbeat monitoring period of 5s, if it is still monitored that the message data amount of the current message queue is greater than the overload threshold value or the state in the current message queue is data overload, one consumer thread is continuously increased until the state of the current message queue is normal, and the consumer thread is not increased any more.
S270, when no accumulated message data in the message queue is detected, releasing the consumer thread added from the newly added server resource.
It should be noted that, when no message data has been accumulated in the monitored message queue, the consumer threads acquired from the newly added server resource may be released gradually, or all the consumer threads acquired from the newly added server resource may be released at one time.
The message data processing resource scheduling algorithm of the invention manages the message queue middleware commonly used in the market, and dynamically expands the production and consumption capacity by monitoring the message accumulation condition of the corresponding message queue through heartbeat. According to the core number of the server where the application service is located, analysis and calculation are carried out on a plurality of dimensions such as single message processing capability, under the condition that an application program does not issue and code is not changed, the appropriate amount of production and consumption is rapidly increased when the data volume is large, the message consumption capability is greatly improved in a short time, the resources of the server are fully utilized, and after the data processing accumulated in a queue is completed, the application program releases the server resources.
Example III
Fig. 3 is a schematic structural diagram of a message data processing resource scheduling device according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes:
a configuration module 310, configured to create a message queue according to preset configuration information;
a calculation module 320, configured to monitor the message queue status, and calculate a message production capability and a message consumption capability required to satisfy the message queue status according to the message queue status;
an adjustment module 330, configured to adjust the number of producer threads and consumer threads according to the matched producer threads and consumer threads of the message production capability and message consumption capability.
The calculating module 320 is further configured to calculate the currently required message production capability and message consumption capability according to a comparison result between the message data amount of the message queue and a preset threshold.
The configuration module 330 is further configured to adjust the number of the producer threads and the consumer threads according to the matched producer threads and consumer threads of the message production capability and message consumption capability under the limitation of the server idle resources.
The configuration module 330 is further configured to increase the number of threads of the consumer corresponding to the message consumption capability when the number of threads corresponding to the message consumption capability is less than the number of threads of the idle resource of the server; and when the thread number corresponding to the message consumption capability is larger than the thread number of the idle resources of the server, taking the thread number of the idle resources of the server as the consumer thread.
The message data processing resource scheduling device provided by the embodiment of the invention can execute the message data processing resource scheduling device method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example IV
Fig. 4 shows a schematic diagram of the structure of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a resource scheduling method for message data processing.
In some embodiments, a resource scheduling method of message data processing may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the resource scheduling method of message data processing described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured in any other suitable way (e.g. by means of firmware) to perform a resource scheduling method of message data processing.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for scheduling resources for message data processing, comprising:
creating a message queue according to preset configuration information;
monitoring the message queue state, and calculating the message production capacity and the message consumption capacity required by meeting the message queue state according to the message queue state;
and adjusting the number of the producer threads and the consumer threads according to the matched producer threads and consumer threads of the message production capacity and the message consumption capacity.
2. The method as recited in claim 1, further comprising:
acquiring idle resource information of a current server;
said adjusting the number of said producer threads and said consumer threads according to said matched producer threads and said consumer threads of said message production capability and message consumption capability, comprising:
and under the limitation of the idle resources of the server, adjusting the number of the producer threads and the consumer threads according to the matched producer threads and consumer threads of the message production capacity and the message consumption capacity.
3. The method of claim 2, wherein said calculating message production capacity and message consumption capacity required to satisfy said message queue status from said message queue status comprises:
and calculating the currently required message production capacity and message consumption capacity according to the comparison result of the message data volume of the current message queue and a preset threshold value.
4. A method according to claim 3, wherein said adjusting the number of said producer and consumer threads based on said matched producer and consumer threads of said message production and message consumption capabilities under the limitation of said server free resources comprises:
if the thread number corresponding to the message consumption capability is smaller than the thread number of the idle resources of the server, increasing the consumer thread number corresponding to the message consumption capability;
and if the thread number corresponding to the message consumption capability is larger than the thread number of the idle resources of the server, taking the thread number of the idle resources of the server as the consumer thread.
5. The method of claim 4, wherein the number of threads corresponding to the message consumption capability is increased if the number of threads corresponding to the message consumption capability is less than the number of threads of the server idle resources; and if the number of threads corresponding to the message consumption capability is greater than the number of threads of the server idle resources, taking the number of threads of the server idle resources as the consumer threads, the method further comprises:
and if no accumulated message data in the message queue is detected, releasing the added consumer thread.
6. The method according to claim 1, wherein the method further comprises:
when the message data volume in the message queue is monitored to be larger than an overload threshold value, increasing the server resource;
and increasing the number of the consumer threads for consumption of the message data according to the newly added server resources.
7. The method of claim 6, wherein the method further comprises:
and when no accumulated message data in the message queue is detected, releasing the consumer thread added from the newly added server resource.
8. A message data processing resource scheduling apparatus, comprising:
the configuration module is used for creating a message queue according to preset configuration information;
the calculation module is used for monitoring the message queue state and calculating the message production capacity and the message consumption capacity required by meeting the message queue state according to the message queue state;
and the adjusting module is used for adjusting the number of the producer threads and the consumer threads according to the matched producer threads and consumer threads of the message production capacity and the message consumption capacity.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the resource scheduling method of message data processing of any one of claims 1-7.
10. A computer readable storage medium storing computer instructions for causing a processor to execute the resource scheduling method for message data processing according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311724550.3A CN117851045A (en) | 2023-12-14 | 2023-12-14 | Resource scheduling method, device and medium for message data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311724550.3A CN117851045A (en) | 2023-12-14 | 2023-12-14 | Resource scheduling method, device and medium for message data processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117851045A true CN117851045A (en) | 2024-04-09 |
Family
ID=90528046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311724550.3A Pending CN117851045A (en) | 2023-12-14 | 2023-12-14 | Resource scheduling method, device and medium for message data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117851045A (en) |
-
2023
- 2023-12-14 CN CN202311724550.3A patent/CN117851045A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP4113299A2 (en) | Task processing method and device, and electronic device | |
| CN115378859B (en) | Method, apparatus, device, medium and product for determining limit state information | |
| CN114201278A (en) | Task processing method, task processing device, electronic device, and storage medium | |
| CN114448989B (en) | Method, device, electronic equipment, storage medium and product for adjusting message distribution | |
| CN113360266B (en) | Task processing method and device | |
| CN113742057A (en) | Task execution method and device | |
| CN112860401A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN116594563A (en) | Distributed storage capacity expansion method and device, electronic equipment and storage medium | |
| CN112887407A (en) | Job flow control method and device for distributed cluster | |
| CN116661960A (en) | Batch task processing method, device, equipment and storage medium | |
| CN117851045A (en) | Resource scheduling method, device and medium for message data processing | |
| CN115543416A (en) | Configuration updating method and device, electronic equipment and storage medium | |
| CN113971083A (en) | Task scheduling method, device, equipment, medium and product | |
| CN113971082A (en) | Task scheduling method, device, equipment, medium and product | |
| CN110380991A (en) | A kind of IOCP mechanism and the Internet of Things Network Communication acceleration system based on eFPGA and IOCP | |
| CN114598705B (en) | Message load balancing method, device, equipment and medium | |
| CN115442432B (en) | Control method, device, equipment and storage medium | |
| CN116048791B (en) | Regulation and control method and device of test node, electronic equipment and storage medium | |
| CN113220554B (en) | Method and apparatus for detecting performance of program code | |
| CN114661496A (en) | Message processing method, device, electronic equipment, storage medium and product | |
| CN115801718A (en) | Message processing method and device, electronic equipment and message processing system | |
| CN117082083A (en) | Data storage method, device and medium based on distributed Internet of things architecture | |
| CN116954922A (en) | Distributed storage method, device, equipment and medium | |
| CN117785403A (en) | Heartbeat task execution device for equipment, electronic equipment and storage medium | |
| CN114722006A (en) | Data transmission control method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |