CN111124710A - Data processing method and electronic equipment - Google Patents
Data processing method and electronic equipment Download PDFInfo
- Publication number
- CN111124710A CN111124710A CN201911299428.XA CN201911299428A CN111124710A CN 111124710 A CN111124710 A CN 111124710A CN 201911299428 A CN201911299428 A CN 201911299428A CN 111124710 A CN111124710 A CN 111124710A
- Authority
- CN
- China
- Prior art keywords
- message
- consumption
- storage unit
- group
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a data processing method and electronic equipment, wherein the method comprises the following steps: detecting message consumption information of a plurality of storage units, wherein the message consumption information comprises the consumption condition of message consumers in a first consumption group to messages produced by a message producer; judging the consumption balance degree corresponding to the storage unit according to the message consumption information; when the consumption balance degree exceeds a preset balance threshold value, adjusting the association relationship between the storage unit and the message producer and/or the association relationship between the storage unit and the message consumer based on the message consumption information, wherein the method comprises the following steps: and utilizing the second consumption group to perform load balancing operation on the storage units with the consumption balance degree exceeding the preset balancing threshold value in the process of using the first consumption group to perform message consumption. The method can utilize the second consumption group to process the backlogged messages in the related storage units in time, thereby rapidly solving the unbalanced phenomenon of message consumption.
Description
Technical Field
The present disclosure relates to the field of data processing, and in particular, to a data processing method and an electronic device.
Background
In a system of an electronic device, intermediate data needs to be transferred to realize processing of data streams, including processes of production, storage, consumption and the like of messages, after a plurality of message producers produce messages, the messages are stored in a plurality of storage units, and a plurality of message consumers consume the messages stored in the storage units to realize processing of the data. For example, Kafka is an open source message queue, and in many systems Kafka is required as middleware for message distribution and buffering. However, the number of storage units (Partition) most suitable for the current use environment cannot be determined initially in the use process, and as the number of processed messages increases, the number of existing storage units (Partition) cannot meet the requirement, and particularly, the messages are stored in a plurality of storage units extremely unevenly, so that the messages cannot be processed by corresponding message consumers (consumers) in time, which may cause the existing messages to be accumulated in the storage units (Partition), resulting in poor data processing and even errors.
Disclosure of Invention
An object of the embodiments of the present application is to provide a data processing method and an electronic device, where the method can adjust a processing manner of a message in a system, so that consumption of the message is balanced, thereby improving processing efficiency of the data and avoiding data loss.
In order to solve the technical problem, the embodiment of the application adopts the following technical scheme: a method of data processing, comprising:
detecting message consumption information of a plurality of storage units, wherein the message consumption information comprises the consumption condition of message consumers in a first consumption group to messages produced by a message producer;
judging the consumption balance degree corresponding to the storage unit according to the message consumption information;
if the consumption balance degree exceeds a preset balance threshold, adjusting the association relationship between the storage unit and the message producer and/or the association relationship between the storage unit and the message consumer based on the message consumption information, wherein the adjusting comprises: and utilizing a second consumption group to perform load balancing operation on the storage units with the consumption balance degree exceeding the preset balance threshold value in the process of using the first consumption group to perform message consumption.
Preferably, the performing, by using the second consumption group, a load balancing operation on the storage unit in which the consumption balance degree exceeds the preset balance threshold in the process of using the first consumption group to perform message consumption includes:
setting the storage unit corresponding to the consumption balance degree exceeding a preset balance threshold value as a target storage unit;
and consuming the messages stored in the target storage unit by using the message consumers in the selected second consumption group, wherein at least part of the messages stored in the target storage unit are transferred to other storage units meeting preset storage conditions.
Preferably, the consumption balance is specifically a message lag value, and correspondingly, the preset balance threshold is a preset lag standard value, and the method further includes:
acquiring a message lag value corresponding to each storage unit;
and determining the storage unit with the message lag value exceeding a preset lag standard value as the target storage unit.
Preferably, the consuming the message stored in the target storage unit by the message consumer in the selected second consumption group includes:
acquiring a first offset corresponding to the first consumption group, wherein the first offset represents message consumers in the first consumption group and consumes the position information of the messages in the storage unit;
determining a second offset of the second consumption group based on the first offset, wherein the second offset characterizes message consumers in the second consumption group to consume the position information of the message in the storage unit;
and establishing an association relation between the message consumers in the second consumption group and the target storage unit based on the second offset so as to consume the messages stored in the target storage unit by using the message consumers in the second consumption group.
Preferably, the consuming, by the message consumers in the second consumption group, the message stored in the target storage unit includes:
establishing an association between the target storage unit and the selected message producer to determine an association between the selected message producer and the message consumer in the second consumption group.
Preferably, the method further comprises:
and correspondingly adjusting the first offset according to the consumption result of the message consumer in the second consumption group consuming the message stored in the target storage unit.
Preferably, when the consumption balance degree exceeds a preset balance threshold, the adjusting the association relationship between the storage unit and the message producer and/or the association relationship between the storage unit and the message consumer based on the message consumption information further includes:
designating one or more specific message consumers among the message consumers when a current offset in the storage unit exceeds a preset equalization threshold;
the messages stored in the storage unit are adjusted by the particular message consumer.
Preferably, the adjusting, by the specific message consumer, the message stored in the storage unit includes:
the messages stored in the storage unit are adjusted in a polling and/or random manner.
An embodiment of the present application further provides an electronic device, including:
a detection module configured to detect message consumption information of a plurality of storage units, wherein the message consumption information includes consumption of messages produced by message producers by message consumers in a first consumption group;
the judging module is configured to judge the consumption balance degree corresponding to the storage unit according to the message consumption information;
a processing module configured to adjust an association relationship between the storage unit and the message producer and/or an association relationship between the storage unit and the message consumer based on the message consumption information if the consumption balance degree exceeds a preset balance threshold, wherein the processing module comprises: and utilizing a second consumption group to perform load balancing operation on the storage units with the consumption balance degree exceeding the preset balance threshold value in the process of using the first consumption group to perform message consumption.
Preferably, the processing module is further configured to:
setting the storage unit corresponding to the consumption balance degree exceeding a preset balance threshold value as a target storage unit;
and consuming the messages stored in the target storage unit by using the message consumers in the selected second consumption group, wherein at least part of the messages stored in the target storage unit are transferred to other storage units meeting preset storage conditions.
The beneficial effects of the embodiment of the application are that: the method can adjust the processing mode of the messages in the system, and particularly can utilize the specific second consumption group to process the messages stored in the related storage units in time under the condition that the imbalance phenomenon occurs in the processing process of the messages, so that the imbalance phenomenon of the consumption of the messages is rapidly solved, the data processing efficiency is improved, and the data loss is avoided.
Drawings
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present application;
FIG. 2 is a flowchart of one embodiment of step S3 of FIG. 1 according to the present application;
FIG. 3 is a flowchart of one embodiment of step S32 of FIG. 2 according to the present application;
FIG. 4 is a flowchart of another embodiment of step S3 of FIG. 1 according to the present application;
FIG. 5 is a flow chart of a first consumption group performing consumption of a message according to an embodiment of the present application;
FIG. 6 is a flowchart of one embodiment of consuming a message based on a second consumption group, according to an embodiment of the present application;
FIG. 7 is a flowchart of another embodiment of step S3 of FIG. 1 according to the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Various aspects and features of the present application are described herein with reference to the drawings.
It will be understood that various modifications may be made to the embodiments of the present application. Accordingly, the foregoing description should not be construed as limiting, but merely as exemplifications of embodiments. Those skilled in the art will envision other modifications within the scope and spirit of the application.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with a general description of the application given above and the detailed description of the embodiments given below, serve to explain the principles of the application.
These and other characteristics of the present application will become apparent from the following description of preferred forms of embodiment, given as non-limiting examples, with reference to the attached drawings.
It should also be understood that, although the present application has been described with reference to some specific examples, a person of skill in the art shall certainly be able to achieve many other equivalent forms of application, having the characteristics as set forth in the claims and hence all coming within the field of protection defined thereby.
The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.
Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it is to be understood that the disclosed embodiments are merely exemplary of the application, which can be embodied in various forms. Well-known and/or repeated functions and constructions are not described in detail to avoid obscuring the application of unnecessary or unnecessary detail. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present application in virtually any appropriately detailed structure.
The specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," which may each refer to one or more of the same or different embodiments in accordance with the application.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application, which may be applied in a process in which middleware (e.g., Kafka) processes data. The data has a corresponding message Producer (Producer), message Consumer (Consumer), and after the message is produced by the message Producer, the message is sent to a Topic containing one or more storage units (Partition) specified in the middleware (e.g., Kafka), and a storage unit is determined to store the message. After a message consumer subscribes to the Topic and starts, a middleware (such as Kafka) can dynamically appoint a certain storage unit in the Topic to enable the message consumer to consume (process the message), and the data processing method implemented by the application can adjust the processing mode of the message in the system, so that the consumption of the message is balanced, and the data processing efficiency is improved. As shown in fig. 1, the method comprises the steps of:
s1, message consumption information of the plurality of storage units is detected, wherein the message consumption information comprises the consumption condition of the message consumers in the first consumption group to the message produced by the message producer.
In this embodiment, a plurality of storage units are provided to store a message generated by a message producer, and the stored message may be consumed by a message consumer, that is, the message is processed. The message consumption information in this embodiment includes a relevant situation about a message that is produced by a message producer and stored in a storage unit, and also includes a situation that a message consumer consumes a message stored in a storage unit, and the like. If the first consumption group and the second consumption group can be set, the first consumption group can consume the messages in the storage unit under the state of normal processing of the messages, wherein the message consumption information comprises the consumption of the message producer by each message consumer in the first consumption group and the consumption condition of the message producer by the message consumer in the first consumption group.
And S2, judging the consumption balance degree corresponding to the storage unit according to the message consumption information.
The consumption balance degree can represent whether the messages stored in the storage unit are balanced in the process of being processed (including aspects of processing efficiency, processing mode and the like), for example, whether data backlog in the storage unit is caused by too large data quantity or whether the messages are backlogged in the storage unit due to untimely processing of the messages by message consumers, and the consumption balance degree can represent the process balance degree of production, storage and consumption of the messages in a quantized form.
S3, if the consumption balance degree exceeds a preset balance threshold, adjusting the association between the storage unit and the message producer and/or the association between the storage unit and the message consumer based on the message consumption information, including: and utilizing a second consumption group to perform load balancing operation on the storage units with the consumption balance degree exceeding the preset balance threshold value in the process of using the first consumption group to perform message consumption.
If the consumption balance degree exceeds the preset balance threshold, it indicates that the produced message is stored in the storage unit and the message is not processed in time (for example, the produced message is stored in a large number of storage units, and the message in one or more of the storage units is not processed in time), thereby resulting in an imbalance of consumption, and a load balancing operation needs to be performed on the imbalance phenomenon. The specific value of the preset equalization threshold can be set according to the actual use environment and historical experience, such as the storage capacity of the storage unit, the production capacity of the message producer and the consumption capacity of the message consumer according to one of the following aspects. In this embodiment, when the consumption balance degree exceeds the preset balance threshold, the association relationship between the storage unit and the message producer is adjusted based on the message consumption information, and/or the association relationship between the storage unit and the message consumer is adjusted. The association relationship between the storage unit and the message producer may be information associated between a specific message producer and a storage unit corresponding to the specific message producer, for example, information produced by a first message producer needs to be stored in a first storage unit associated with the first message producer; the association relationship between the storage unit and the message consumer may be information associated between a specific message consumer and the storage unit corresponding to the specific message consumer, for example, the first message consumer needs to consume the information stored in the first storage unit. If the consumption balance degree exceeds a preset balance threshold value, the association relation can be adjusted. For example, in the process of working together by a plurality of storage units, the message in the first storage unit can not be consumed in time, and the second message consumer can be invoked to consume the message in the first storage unit which causes the message backlog. In this embodiment, in the process of consuming messages by the first consumption group, if the consumption balance degree exceeds the preset balance threshold, the second consumption group different from the first consumption group is selected to perform load balancing operation on the storage unit with the consumption balance degree exceeding the preset balance threshold, and the second consumption group does not work (only the first consumption group performs message consumption operation) when the consumption balance degree does not exceed the preset balance threshold, but only works when the consumption balance degree exceeds the preset balance threshold, so that the storage unit with the message backlog can be timely consumed, and each storage unit can perform balanced work.
The method can adjust the processing mode of the messages in the system, and particularly can utilize the specific second consumption group to process the messages stored in the related storage units in time under the condition that the imbalance phenomenon occurs in the processing process of the messages, so that the imbalance phenomenon of the consumption of the messages is rapidly solved, the data processing efficiency is improved, and the data loss is avoided.
In an embodiment of the present application, as shown in fig. 2, the performing, by using a second consumption group, a load balancing operation on the storage unit in which the consumption balance degree exceeds the preset balance threshold in the process of using the first consumption group to perform message consumption includes the following steps:
and S31, setting the storage unit corresponding to the consumption balance degree exceeding the preset balance threshold value as a target storage unit.
The target storage unit can be an object needing attention and processing, and the production and consumption conditions of the message in the target storage unit enable the consumption balance degree to exceed a preset balance threshold value, so that load balance processing is needed. For example, in the process of consuming the target storage unit, the first consuming group generates a backlog of messages (the consuming rate of the first consuming group is slower than the message producing speed of the message producer, and the message cannot be consumed in time, which results in the backlog), and further the second consuming group needs to process (consume) the message to the target storage unit.
And S32, consuming the messages stored in the target storage unit by using the message consumers in the selected second consumption group, wherein the consumption comprises the step of unloading at least part of the messages stored in the target storage unit to other storage units meeting preset storage conditions.
The second consumption group may not perform the operation of consuming the message when no load imbalance occurs, but only initiate consumption of the target storage unit when the load imbalance occurs (that is, the consumption balance degree exceeds the preset balance threshold), so as to achieve the target of load balancing, where the process includes unloading at least part of the messages stored in the target storage unit to other storage units that satisfy the preset storage condition. The preset storage condition may be a condition set based on an actual working scene of the storage unit, such as a blank storage area being larger than a certain value, or directly regarding a storage unit which is not used or stores data as a storage unit satisfying the preset storage condition. The other storage units satisfying the preset storage condition may be storage units associated with the second consumption group, or storage units associated with other consumption groups, and are not limited herein. In this embodiment, when the message consumers in the second consumption group consume the messages stored in the target storage unit, at least a part of the messages stored in the target storage unit may be transferred to other storage units satisfying the preset storage condition, so that the number of messages accumulated in the target storage unit for sale is reduced, and data processing balance of the system is achieved. In addition, on one hand, the message consumers in the second consumption group can respectively store the messages in different target storage units in other storage units meeting the preset storage condition; on the other hand, the message consumers in the second consumption group can store the messages in the same target storage unit in a plurality of other storage units meeting the preset storage condition separately, so that the load balancing operation can be flexibly performed.
In an embodiment of the present application, the consumption balance is specifically a message lag value, and correspondingly, the preset balance threshold is a preset lag criterion value, and the method further includes the following steps:
acquiring a message lag value corresponding to each storage unit;
and determining the storage unit with the message lag value exceeding a preset lag standard value as the target storage unit.
In particular, a message LAG value (LAG) characterizes the relevance of a message being processed by a LAG. In general, after a message producer produces a message, the message is stored in a storage unit and can be consumed (disposed) by a message consumer in time, and the corresponding message lag value is lower at the moment; if the message is heavily backlogged in the memory location and cannot be processed in a timely manner, the corresponding message lag value will be higher. In this embodiment, each storage unit corresponds to a respective message lag value, and if the message lag value of a certain storage unit exceeds a preset lag standard value, the storage unit is determined as a target storage unit and needs to be processed.
In an embodiment, with reference to fig. 6, the backlog degrees of the messages of the multiple storage units may be sorted respectively, and then the storage unit with the larger backlog degree is selected as the target storage unit, for example, sorting according to the message LAG value (LAG) of each storage unit, subtracting the message LAG values of the multiple storage units two by two to obtain an intermediate value, comparing the intermediate value with a specific value Q, and selecting the storage unit with the intermediate value greater than Q as the target storage unit (for example, P storage units).
In an embodiment of the present application, as shown in fig. 3, the consuming the message stored in the target storage unit by the message consumer in the selected second consumption group includes the following steps:
s321, obtaining a first offset corresponding to the first consumption group, wherein the first offset represents the message consumers in the first consumption group, and consumes the position information of the message in the storage unit.
In one embodiment, the first consumption group has a plurality of storage units, and the first offset may be information of the location of the message consumer consuming to a specific storage unit (including the specific location of the message in the storage unit consumed); in another embodiment, the first consumption group corresponds to a memory location, and the first offset may be specific location information of the message consumer processing the message of the memory location. The messages in the storage unit may be sequentially arranged to wait for consumption by the message consumer, and with reference to fig. 5, for example, when the first offset is M, and after the consumption of N messages is processed, the first offset is changed to M + N, and when the first offset is obtained, it can be known where the message consumed by the message consumer in the first consumption group is specifically processed, which is a quantized value.
S322, determining a second offset of the second consumption group based on the first offset, wherein the second offset characterizes message consumers in the second consumption group, and consumes the position information of the message in the storage unit.
Based on the first offset, the second consuming group can know from which position to consume (process the message), and it is ensured that no processing of the message is missed, for example, based on the position of the message already processed by the first consuming group, and then it is determined from which position to consume the second consuming group, that is, a second offset is determined, and the second offset can be changed along with the processing of the message.
S323, based on the second offset, establishing an association relationship between the message consumers in the second consumption group and the target storage unit, so as to consume the messages stored in the target storage unit by using the message consumers in the second consumption group.
The target storage unit is a storage unit to be processed, and after the incidence relation between the message consumers in the second consumption group and the target storage unit is established, the second consumption group can consume (process) the messages stored in the target storage unit, so that the problem of message backlog of the target storage unit is solved, and further load balancing is realized. In addition, the process ensures that the message is not in danger of being missed or even lost in any message in the target storage unit in the process of being processed by the second consumption group.
In an embodiment of the application, said consuming, by a message consumer in the second consumption group, a message stored in the target storage unit includes: establishing an association between the target storage unit and the selected message producer to determine an association between the selected message producer and the message consumer in the second consumption group.
Specifically, during the process of processing the messages stored in the target storage unit by using the message consumers in the second consumption group, the user may select one or more specific message producers to establish an association relationship with the target storage unit, so that the selected message producer may store the messages produced by the selected message producer in the target storage unit, for example, select one specific message producer with a smaller message production speed or amount to establish an association relationship with the target storage unit, or after selecting one or more specific message producers, not allow other message producers to reproduce the stored messages in the target storage unit. Because the message consumer also has an incidence relation with the target storage unit, the incidence relation between the selected message producer and the message consumer in the second consumption group can be established, so that the processing paths of the production, storage and consumption of the message are optimized, and the processing efficiency is improved.
In one embodiment of the present application, the method further comprises the steps of: and correspondingly adjusting the first offset according to the consumption result of the message consumer in the second consumption group consuming the message stored in the target storage unit.
In an embodiment, with reference to fig. 5 and fig. 6, the L may be divided into P components (e.g., Σ L0 — Lp ═ L), and then Li (a component thereof) is allocated to P message consumers, so as to correspondingly adjust the first offset corresponding to the first consumption group, e.g., adjust the first offset from M + N to M + N + Li. Therefore, the consumption accuracy of the message consumers in the first consumption group is ensured, and messages to be processed (consumed) in the target storage unit cannot be missed.
In an embodiment of the application, as shown in fig. 4, when the consumption balance degree exceeds a preset balance threshold, adjusting the association between the storage unit and the message producer and/or the association between the storage unit and the message consumer based on the message consumption information further includes:
s33, when the current offset in the storage unit exceeds the preset balance threshold value, one or more specific message consumers are appointed in the message consumers;
s34, adjusting the message stored in the storage unit by the specific message consumer.
Specifically, when the current offset in the storage unit exceeds a preset balance threshold, a specific message consumer meeting a preset condition may be selected to consume a message causing load imbalance (backlog), for example, a message consumer with strong processing capability is selected as the specific message consumer, or an idle message consumer is selected as the specific message consumer, so that the message stored in the storage unit can be adjusted.
In an embodiment of the present application, as shown in fig. 7, the adjusting, by the specific message consumer, the message stored in the storage unit includes: the messages stored in the storage unit are adjusted in a polling and/or random manner.
In one embodiment, a plurality of storage units can be operated by a specific message consumer, and if the messages stored in the plurality of storage units are overstocked, the storage units can be selected by the specific message consumers in a polling and/or random manner, and then the selected storage units are processed, so that load balancing is simply and quickly realized. In another embodiment, a plurality of specific message consumers are selected, each polling for consumption of messages causing load imbalance (backlog) (possibly in one storage unit or multiple storage units), or randomly consuming messages causing load imbalance (backlog).
An embodiment of the present application further provides an electronic device, which can process a message, and as shown in fig. 8, the electronic device includes:
a detection module configured to detect message consumption information of the plurality of storage units, wherein the message consumption information includes consumption of messages produced by the message producer by message consumers in the first consumption group.
In this embodiment, a plurality of storage units are provided to store a message generated by a message producer, and the stored message may be consumed by a message consumer, that is, the message is processed. The message consumption information in this embodiment includes a relevant situation about a message that is produced by a message producer and stored in a storage unit, and also includes a situation that a message consumer consumes a message stored in a storage unit, and the like. If the first consumption group and the second consumption group can be set, the first consumption group can consume the messages in the storage unit under the state of normal processing of the messages, wherein the message consumption information comprises the consumption of the message producer by each message consumer in the first consumption group and the consumption condition of the message producer by the message consumer in the first consumption group.
And the judging module is configured to judge the consumption balance degree corresponding to the storage unit according to the message consumption information.
The consumption balance degree can represent whether the messages stored in the storage unit are balanced in the process of being processed (including aspects of processing efficiency, processing mode and the like), for example, whether data backlog in the storage unit is caused by too large data quantity or whether the messages are backlogged in the storage unit due to untimely processing of the messages by message consumers, and the consumption balance degree can represent the process balance degree of production, storage and consumption of the messages in a quantized form.
A processing module configured to adjust an association relationship between the storage unit and the message producer and/or an association relationship between the storage unit and the message consumer based on the message consumption information if the consumption balance degree exceeds a preset balance threshold, wherein the processing module comprises: and utilizing a second consumption group to perform load balancing operation on the storage units with the consumption balance degree exceeding the preset balance threshold value in the process of using the first consumption group to perform message consumption.
If the consumption balance degree exceeds the preset balance threshold, it indicates that the produced message is stored in the storage unit and the message is not processed in time (for example, the produced message is stored in a large number of storage units, and the message in one or more of the storage units is not processed in time), thereby resulting in an imbalance of consumption, and a processing module is required to perform a load balancing operation on the imbalance phenomenon. The specific value of the preset equalization threshold can be set according to the actual use environment and historical experience, such as the storage capacity of the storage unit, the production capacity of the message producer and the consumption capacity of the message consumer according to one of the following aspects. In this embodiment, when the consumption balance degree exceeds the preset balance threshold, the processing module adjusts the association relationship between the storage unit and the message producer and/or the association relationship between the storage unit and the message consumer based on the message consumption information. The association relationship between the storage unit and the message producer may be information associated between a specific message producer and a storage unit corresponding to the specific message producer, for example, information produced by a first message producer needs to be stored in a first storage unit associated with the first message producer; the association relationship between the storage unit and the message consumer may be information associated between a specific message consumer and the storage unit corresponding to the specific message consumer, for example, the first message consumer needs to consume the information stored in the first storage unit. If the consumption balance degree exceeds a preset balance threshold value, the processing module can adjust the association relation. For example, in the process of working together by a plurality of storage units, the message in the first storage unit can not be consumed in time, and the second message consumer can be invoked to consume the message in the first storage unit which causes the message backlog. In this embodiment, in the process of consuming messages by the first consumption group, if the consumption balance degree exceeds the preset balance threshold, the processing module selects the second consumption group different from the first consumption group to perform load balancing operation on the storage unit with the consumption balance degree exceeding the preset balance threshold, and the second consumption group does not work (only the first consumption group performs message consumption operation) when the consumption balance degree does not exceed the preset balance threshold, but only works when the consumption balance degree exceeds the preset balance threshold, so that the storage unit with message backlog can be timely consumed by the storage unit with the message backlog, and each storage unit can work in a balanced manner.
The electronic equipment can adjust the processing mode of the messages in the system, and particularly can utilize the specific second consumption group to process the messages stored in the related storage units in time under the condition that an imbalance phenomenon occurs in the processing process of the messages, so that the imbalance phenomenon of the consumption of the messages is rapidly solved, the data processing efficiency is improved, and data loss is avoided.
In one embodiment of the present application, the processing module is further configured to:
and setting the storage unit corresponding to the consumption balance degree exceeding a preset balance threshold value as a target storage unit.
The target storage unit can be an object needing attention and processing, and the production and consumption conditions of the message in the target storage unit enable the consumption balance degree to exceed a preset balance threshold value, so that load balance processing is needed. For example, in the process of consuming the target storage unit, the first consuming group generates a backlog of messages (the consuming rate of the first consuming group is slower than the message producing speed of the message producer, and the message cannot be consumed in time, which results in the backlog), and further the second consuming group needs to process (consume) the message to the target storage unit.
And consuming the messages stored in the target storage unit by using the message consumers in the selected second consumption group, wherein at least part of the messages stored in the target storage unit are transferred to other storage units meeting preset storage conditions.
The processing module may enable the second consumption group not to perform the operation of consuming the message under the condition that no load imbalance occurs, but only start consuming the target storage unit under the condition that the load imbalance occurs (that is, the consumption balance degree exceeds a preset balance threshold), so as to achieve the target of load balancing, where the process includes unloading at least part of the messages stored in the target storage unit to other storage units that satisfy a preset storage condition. The preset storage condition may be a condition set based on an actual working scene of the storage unit, such as a blank storage area being larger than a certain value, or directly regarding a storage unit which is not used or stores data as a storage unit satisfying the preset storage condition. The other storage units satisfying the preset storage condition may be storage units associated with the second consumption group, or storage units associated with other consumption groups, and are not limited herein. In this embodiment, when the processing module consumes the message stored in the target storage unit by using the message consumer in the second consumption group, at least a part of the message stored in the target storage unit may be transferred to another storage unit satisfying the preset storage condition, so that the number of messages accumulated in the target storage unit for sale is reduced, and data processing balance of the system is achieved. In addition, on one hand, the message consumers in the second consumption group can respectively store the messages in different target storage units in other storage units meeting the preset storage condition; on the other hand, the message consumers in the second consumption group can store the messages in the same target storage unit in a plurality of other storage units meeting the preset storage condition separately, so that the load balancing operation can be flexibly performed.
In an embodiment of the application, the consumption balance is specifically a message lag value, and correspondingly, the preset balance threshold is a preset lag criterion value, and the processing module is further configured to:
acquiring a message lag value corresponding to each storage unit;
and determining the storage unit with the message lag value exceeding a preset lag standard value as the target storage unit.
In particular, a message LAG value (LAG) characterizes the relevance of a message being processed by a LAG. In general, after a message producer produces a message, the message is stored in a storage unit and can be consumed (disposed) by a message consumer in time, and the corresponding message lag value is lower at the moment; if the message is heavily backlogged in the memory location and cannot be processed in a timely manner, the corresponding message lag value will be higher. In this embodiment, each storage unit corresponds to a respective message lag value, and if the message lag value of a certain storage unit exceeds a preset lag standard value, the storage unit is determined as a target storage unit by the processing module, and thus needs to be processed.
In an embodiment, the processing module may sort the backlog degrees of the messages of the plurality of storage units, respectively, and then select a storage unit with a larger backlog degree as a target storage unit, for example, sort the backlog degrees of the messages of the plurality of storage units according to the message LAG value (LAG) of each storage unit, subtract the message LAG values of the plurality of storage units two by two to obtain an intermediate value, compare the intermediate value with a specific value Q, and select a storage unit with the intermediate value greater than Q as the target storage unit (for example, P storage units).
In one embodiment of the present application, the processing module is further configured to:
and acquiring a first offset corresponding to the first consumption group, wherein the first offset represents the message consumers in the first consumption group and consumes the position information of the messages in the storage unit.
Determining a second offset for the second consumption group based on the first offset, wherein the second offset characterizes message consumers in the second consumption group consuming location information of messages in the storage unit.
And establishing an association relation between the message consumers in the second consumption group and the target storage unit based on the second offset so as to consume the messages stored in the target storage unit by using the message consumers in the second consumption group.
Specifically, in one embodiment, a first consumption group has a plurality of storage units, and the first offset may be information of a location where a message consumer consumes to a specific storage unit (including a specific location where a message in the storage unit is consumed); in another embodiment, the first consumption group corresponds to a memory location, and the first offset may be specific location information of the message consumer processing the message of the memory location. The messages in the storage unit may be sequentially arranged to wait for consumption by the message consumer, for example, when the first offset is M, after the consumption of N messages is processed, the first offset is changed to M + N, and when the first offset is obtained, it is known where the message consumed by the message consumer in the first consumption group is specifically processed, which is a quantized value.
The processing module can make the second consumption group know the position from which to start consumption (process the message) based on the first offset, and ensure that no processing of the message is missed, for example, based on the position of the message already processed by the first consumption group, and further determine the position from which the second consumption group starts consumption, that is, determine the second offset, which can be changed along with the processing of the message.
The target storage unit is a storage unit to be processed, and after the incidence relation between the message consumers in the second consumption group and the target storage unit is established, the messages stored in the target storage unit can be consumed (processed) through the second consumption group, so that the problem of message backlog of the target storage unit is solved, and further load balancing is realized. In addition, the process ensures that the message is not in danger of being missed or even lost in any message in the target storage unit in the process of being processed by the second consumption group.
In one embodiment of the present application, the processing module is further configured to: establishing an association between the target storage unit and the selected message producer to determine an association between the selected message producer and the message consumer in the second consumption group.
In one embodiment of the present application, the processing module is further configured to: and correspondingly adjusting the first offset according to the consumption result of the message consumer in the second consumption group consuming the message stored in the target storage unit.
In one embodiment of the present application, the electronic device further comprises a second processing module configured to: designating one or more specific message consumers among the message consumers when a current offset in the storage unit exceeds a preset equalization threshold; the messages stored in the storage unit are adjusted by the particular message consumer.
In one embodiment of the present application, the second processing module is further configured to: the messages stored in the storage unit are adjusted in a polling and/or random manner.
The above embodiments are only exemplary embodiments of the present application, and are not intended to limit the present application, and the protection scope of the present application is defined by the claims. Various modifications and equivalents may be made by those skilled in the art within the spirit and scope of the present application and such modifications and equivalents should also be considered to be within the scope of the present application.
Claims (10)
1. A method of data processing, comprising:
detecting message consumption information of a plurality of storage units, wherein the message consumption information comprises the consumption condition of message consumers in a first consumption group to messages produced by a message producer;
judging the consumption balance degree corresponding to the storage unit according to the message consumption information;
if the consumption balance degree exceeds a preset balance threshold, adjusting the association relationship between the storage unit and the message producer and/or the association relationship between the storage unit and the message consumer based on the message consumption information, wherein the adjusting comprises: and utilizing a second consumption group to perform load balancing operation on the storage units with the consumption balance degree exceeding the preset balance threshold value in the process of using the first consumption group to perform message consumption.
2. The data processing method according to claim 1, wherein the performing, by using the second consumption group, a load balancing operation on the storage unit in which the consumption balance degree exceeds the preset balance threshold value during the message consumption by using the first consumption group comprises:
setting the storage unit corresponding to the consumption balance degree exceeding a preset balance threshold value as a target storage unit;
and consuming the messages stored in the target storage unit by using the message consumers in the selected second consumption group, wherein at least part of the messages stored in the target storage unit are transferred to other storage units meeting preset storage conditions.
3. A data processing method according to claim 2, wherein the consumption balance is a message lag value, and correspondingly, the preset balance threshold is a preset lag criterion value, the method further comprising:
acquiring a message lag value corresponding to each storage unit;
and determining the storage unit with the message lag value exceeding a preset lag standard value as the target storage unit.
4. The data processing method of claim 2, wherein said consuming the message stored in the target storage unit with the message consumer in the selected second consumption group comprises:
acquiring a first offset corresponding to the first consumption group, wherein the first offset represents message consumers in the first consumption group and consumes the position information of the messages in the storage unit;
determining a second offset of the second consumption group based on the first offset, wherein the second offset characterizes message consumers in the second consumption group to consume the position information of the message in the storage unit;
and establishing an association relation between the message consumers in the second consumption group and the target storage unit based on the second offset so as to consume the messages stored in the target storage unit by using the message consumers in the second consumption group.
5. The data processing method of claim 4, wherein consuming, with the message consumers in the second consumption group, the messages stored in the target storage unit comprises:
establishing an association between the target storage unit and the selected message producer to determine an association between the selected message producer and the message consumer in the second consumption group.
6. The data processing method of claim 4, the method further comprising:
and correspondingly adjusting the first offset according to the consumption result of the message consumer in the second consumption group consuming the message stored in the target storage unit.
7. The data processing method according to claim 1, wherein if the consumption balance degree exceeds a preset balance threshold, the adjusting the association relationship between the storage unit and the message producer and/or the association relationship between the storage unit and the message consumer based on the message consumption information further comprises:
designating one or more specific message consumers among the message consumers when a current offset in the storage unit exceeds a preset equalization threshold;
the messages stored in the storage unit are adjusted by the particular message consumer.
8. The data processing method of claim 7, wherein said adjusting the message stored in the storage unit by the particular message consumer comprises:
the messages stored in the storage unit are adjusted in a polling and/or random manner.
9. An electronic device, comprising:
a detection module configured to detect message consumption information of a plurality of storage units, wherein the message consumption information includes consumption of messages produced by message producers by message consumers in a first consumption group;
the judging module is configured to judge the consumption balance degree corresponding to the storage unit according to the message consumption information;
a processing module configured to adjust an association relationship between the storage unit and the message producer and/or an association relationship between the storage unit and the message consumer based on the message consumption information if the consumption balance degree exceeds a preset balance threshold, wherein the processing module comprises: and utilizing a second consumption group to perform load balancing operation on the storage units with the consumption balance degree exceeding the preset balance threshold value in the process of using the first consumption group to perform message consumption.
10. The electronic device of claim 9, the processing module further configured to:
setting the storage unit corresponding to the consumption balance degree exceeding a preset balance threshold value as a target storage unit;
and consuming the messages stored in the target storage unit by using the message consumers in the selected second consumption group, wherein at least part of the messages stored in the target storage unit are transferred to other storage units meeting preset storage conditions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911299428.XA CN111124710A (en) | 2019-12-17 | 2019-12-17 | Data processing method and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911299428.XA CN111124710A (en) | 2019-12-17 | 2019-12-17 | Data processing method and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111124710A true CN111124710A (en) | 2020-05-08 |
Family
ID=70499133
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911299428.XA Pending CN111124710A (en) | 2019-12-17 | 2019-12-17 | Data processing method and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111124710A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112612631A (en) * | 2020-12-29 | 2021-04-06 | 杭州涂鸦信息技术有限公司 | Message queue accumulation detection method and related device |
| CN114448989A (en) * | 2022-01-26 | 2022-05-06 | 北京百度网讯科技有限公司 | Method, device, electronic equipment, storage medium and product for adjusting message distribution |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108509299A (en) * | 2018-03-29 | 2018-09-07 | 努比亚技术有限公司 | Message treatment method, equipment and computer readable storage medium |
| US20190129771A1 (en) * | 2017-10-26 | 2019-05-02 | Nokia Solutions And Networks Oy | Balanced message distribution in distributed message handling systems |
| CN110502402A (en) * | 2019-08-28 | 2019-11-26 | 中国联合网络通信集团有限公司 | Message treatment method, equipment and terminal device |
-
2019
- 2019-12-17 CN CN201911299428.XA patent/CN111124710A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190129771A1 (en) * | 2017-10-26 | 2019-05-02 | Nokia Solutions And Networks Oy | Balanced message distribution in distributed message handling systems |
| CN108509299A (en) * | 2018-03-29 | 2018-09-07 | 努比亚技术有限公司 | Message treatment method, equipment and computer readable storage medium |
| CN110502402A (en) * | 2019-08-28 | 2019-11-26 | 中国联合网络通信集团有限公司 | Message treatment method, equipment and terminal device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112612631A (en) * | 2020-12-29 | 2021-04-06 | 杭州涂鸦信息技术有限公司 | Message queue accumulation detection method and related device |
| CN112612631B (en) * | 2020-12-29 | 2024-04-02 | 杭州涂鸦信息技术有限公司 | Message queue accumulation detection method and related device |
| CN114448989A (en) * | 2022-01-26 | 2022-05-06 | 北京百度网讯科技有限公司 | Method, device, electronic equipment, storage medium and product for adjusting message distribution |
| CN114448989B (en) * | 2022-01-26 | 2024-04-05 | 北京百度网讯科技有限公司 | Method, device, electronic equipment, storage medium and product for adjusting message distribution |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108021487B (en) | GPU (graphics processing Unit) graphic processing performance monitoring and analyzing method | |
| CN111124710A (en) | Data processing method and electronic equipment | |
| CN107247540B (en) | Application icon adjusting method and device | |
| RU2018142269A (en) | SYSTEM AND METHOD OF ORGANIZING A QUEUE BASED ON EXPECTATION | |
| CN105516347A (en) | Method and device for load balance allocation of streaming media server | |
| CN103312825A (en) | Method and device for data distribution and storage | |
| CN111045808A (en) | Distributed network task scheduling method and device | |
| CN110597126A (en) | Intelligent security check method, device, equipment and storage medium | |
| CN106059940B (en) | A kind of flow control methods and device | |
| CN111083062A (en) | Weight mechanism-based current limiting method and device, computer equipment and storage medium | |
| CN114448989B (en) | Method, device, electronic equipment, storage medium and product for adjusting message distribution | |
| CN106649042B (en) | A kind of processing method and processing unit of cpu load information | |
| CN108694083B (en) | Data processing method and device for server | |
| CN109039389B (en) | Low-power-consumption Bluetooth transmission method and device | |
| CN111262783A (en) | Dynamic routing method and device | |
| US20150006678A1 (en) | Electronic device and method for managing network bandwidth distribution | |
| CN101515888A (en) | Port mapping method and device | |
| JP2007074607A (en) | Data processing circuit for reducing power consumption by using clock control | |
| CN107247638B (en) | Data backup method, device and system | |
| CN115883391A (en) | Single-user service selection method and device | |
| CN107590686A (en) | A kind for the treatment of method and apparatus of ad-request | |
| CN104754636B (en) | A kind of detection method and communication equipment of cell load status | |
| CN105653831A (en) | Electronic card game card type recommendation method and device | |
| CN109510855B (en) | Event distribution system, method and device | |
| CN112379978A (en) | Back-end service calling method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |