CN115640359A - Message synchronization control method and device, electronic equipment and storage medium - Google Patents
Message synchronization control method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115640359A CN115640359A CN202211346305.9A CN202211346305A CN115640359A CN 115640359 A CN115640359 A CN 115640359A CN 202211346305 A CN202211346305 A CN 202211346305A CN 115640359 A CN115640359 A CN 115640359A
- Authority
- CN
- China
- Prior art keywords
- data
- time window
- prediction
- data synchronization
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000004519 manufacturing process Methods 0.000 claims abstract description 22
- 230000008569 process Effects 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012549 training Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 5
- 230000003993 interaction Effects 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a message synchronization control method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a historical data pushing sequence of a data production object, and predicting to obtain a predicted data pushing sequence in a prediction time period based on the historical data pushing sequence; determining a data synchronization time window within a prediction time period based on the predicted data push sequence; and generating control information of data synchronization based on the data synchronization time window, and sending the control information of data synchronization to the data consumption object so that the data consumption object performs data synchronization in the data synchronization time window. According to the invention, the data synchronization time window in the prediction time range is obtained through prediction of the historical data pushing sequence, and the historical data pushing sequence of the data production object is taken as a reference, so that the data consumption object can flexibly adjust the consumption plan according to the data pushing condition, meanwhile, the waiting time of the data consumption object is reduced, and the consumption of system resources is effectively reduced.
Description
Technical Field
The present invention relates to the field of network information interaction technologies, and in particular, to a method and an apparatus for controlling message synchronization, an electronic device, and a storage medium.
Background
Generally, a large-and-medium-sized enterprise will construct a plurality of information systems, and data synchronization is realized among the plurality of information systems through a message queue.
The existing method for synchronizing data between a producer and a consumer through a Kafka cluster is to manually set the start-stop time period of the process of the consumer according to the transaction running condition and human experience.
However, the manual setting of the start-stop time according to the transaction operation condition and the human experience leads to the fact that the consumption plan cannot be dynamically adjusted according to the real producer data pushing condition, in addition, the start-stop time parameter is a statically configured parameter, after the setting is completed, the configured time parameter needs to be manually modified and cannot be changed in the operation process, the flexibility is poor, and the flexible adjustment of the consumption plan is difficult to achieve.
Disclosure of Invention
The invention provides a message synchronization control method and device, electronic equipment and a storage medium, which are used for flexibly adjusting a consumption plan according to a data pushing condition.
According to an aspect of the present invention, there is provided a method for controlling message synchronization, including:
acquiring a historical data pushing sequence of a data production object, and predicting to obtain a predicted data pushing sequence in a prediction time period based on the historical data pushing sequence;
determining a data synchronization time window within a prediction time period based on the predicted data push sequence;
and generating control information of data synchronization based on the data synchronization time window, and sending the control information of data synchronization to a data consumption object so that the data consumption object performs data synchronization in the data synchronization time window.
According to another aspect of the present invention, there is provided a message synchronization control apparatus, comprising:
the prediction data pushing sequence prediction module is used for acquiring a historical data pushing sequence of a data production object and predicting to obtain a prediction data pushing sequence in a prediction time period based on the historical data pushing sequence;
the data synchronization time window determining module is used for determining a data synchronization time window in a prediction time period based on the prediction data pushing sequence;
the data synchronization module is used for generating data synchronization control information based on the data synchronization time window and sending the data synchronization control information to a data consumption object so that the data consumption object performs data synchronization in the data synchronization time window.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the method of controlling message synchronization according to any of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement a control method for message synchronization according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the technical scheme of the embodiment of the invention, the data synchronization time window in the prediction time range is obtained through prediction of the historical data pushing sequence, the data consumption object sets the start-stop time of data synchronization according to the predicted data synchronization time window, and the historical data pushing sequence of the data production object is taken as a reference, so that the data consumption object can flexibly adjust the consumption plan according to the data pushing condition, meanwhile, the waiting time of the data consumption object is reduced, and the consumption of system resources is effectively reduced.
It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present invention, nor are they intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a message synchronization control method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a message synchronization mechanism according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a control apparatus for message synchronization according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example one
Fig. 1 is a flowchart of a message synchronization control method according to an embodiment of the present invention, where this embodiment is applicable to data synchronization between a producer and a consumer through a Kafka cluster, and the method may be executed by a message synchronization control device, where the message synchronization control device may be implemented in hardware and/or software, and the message synchronization control device may be configured in an electronic device according to an embodiment of the present invention. As shown in fig. 1, the method includes:
s110, obtaining a historical data pushing sequence of the data production object, and predicting to obtain a predicted data pushing sequence in a prediction time period based on the historical data pushing sequence.
The data production object refers to a system for producing data, namely a producer; the historical data pushing sequence refers to a sequence of historical production data which is generated by a producer process of the data production object and pushed to the message queue, and specifically, the historical data pushing sequence for acquiring the data production object may be a historical data pushing sequence for acquiring any message queue by the production object. In the embodiment, a historical data pushing sequence of a production object to any message queue is obtained, and the data pushing sequence in a prediction time period is predicted based on the historical data pushing sequence to obtain a predicted data pushing sequence; the predicted data pushing sequence refers to a sequence of predicted production data that is pushed to the message queue by the data production object within a prediction time period, and optionally, the prediction time period may be 24 hours, which is not limited herein.
On the basis of the foregoing embodiment, optionally, the predicting a predicted data pushing sequence within a prediction time period based on the historical data pushing sequence includes: and inputting the historical data pushing sequence into a pushing prediction model to obtain a predicted data pushing sequence in the prediction time period.
The push prediction model is a pre-trained prediction model used for predicting a predicted data push sequence within a prediction time period, and specifically, the push prediction model includes, but is not limited to, a support vector machine model, a naive bayes model, a decision tree model, and the like, and is not limited herein. In this embodiment, the historical data push sequence is input into the push prediction model, and the push prediction model outputs the predicted data push sequence within the prediction time period.
When the push prediction model is used for the first time, a historical data statistical set is constructed based on a historical data push sequence, the historical data statistical set is used as a training sample set to train the initial push prediction model, and the push prediction model is obtained; the historical data statistics set includes each time node in the historical pushing time and the historical pushing data amount corresponding to each time node, specifically, the time node includes but is not limited to a time node using a minute as a node, and this is not limited here. It will be appreciated that in the initial training of the push prediction model, the training parameters of the initial push prediction model are randomly initialized.
On the basis of the above embodiment, the method further includes: and acquiring an actual data push sequence in a prediction time period, updating the push prediction model based on the actual data push sequence and the historical data push sequence to obtain an updated push prediction model, wherein the updated push prediction model is used for predicting to obtain a predicted data push sequence of the next prediction time period.
The actual data pushing sequence refers to a sequence of historical production data which is actually generated by a producer process of the data production object in the prediction time period and pushed to the message queue. In the embodiment, an actual data push sequence in a prediction time period is obtained, a historical data statistical set is constructed based on the actual data push sequence and the historical data push sequence, and the historical data statistical set is used as a training sample set to train and update a push prediction model to obtain an updated push prediction model; and the updated push prediction model is used for predicting a predicted data push sequence of the next prediction time period.
And S120, determining a data synchronization time window in the prediction time period based on the prediction data push sequence.
The data synchronization time window refers to a time range for performing data synchronization within a prediction time period, specifically, the data synchronization time window includes a start time node and a stop time node of the data synchronization, exemplarily, the data synchronization time window is represented by [ L, R ], L represents the start time node of the data synchronization, and R represents the stop time node of the data synchronization. In this embodiment, the data synchronization time window within the prediction time period is determined based on the prediction data push sequence.
On the basis of the foregoing embodiment, optionally, the predicted data pushing sequence includes a pushed data amount of each time node in the predicted time period; the determining a data synchronization time window within a prediction time period based on the predicted data push sequence comprises: determining a pushing peak time node in the prediction time period based on the pushing data volume of each time node in the prediction time period; and determining a data synchronization time window based on the push peak time node, wherein the sum of the push data amount corresponding to each time node in the data synchronization time window meets a preset condition.
The pushing peak time node refers to a time node corresponding to a pushing data volume peak in each time node in the prediction time period. In this embodiment, a push peak time node in the prediction time period is obtained based on the push data volume of each time node in the prediction time period in the prediction data push sequence, and a data synchronization time window is determined with the push peak time node as a reference; illustratively, assume a push peak time node is t max In t with max Defining an initial data synchronization time window [ L, R ] for a reference]L and R respectively denote boundary time nodes of a data synchronization time window, initialBoth L and R of the data synchronization time window are t max 。
It should be noted that the sum of the pushed data amounts corresponding to the time nodes in the data synchronization time window satisfies a preset condition.
On the basis of the above embodiment, optionally, the preset condition is: and the sum of the pushed data amounts corresponding to the time nodes in the data synchronization time window is greater than or equal to a data amount threshold, and the data amount threshold is determined based on the threshold proportion of the total pushed data amount of the time nodes in the prediction time period.
Specifically, the preset condition is that the sum of the pushed data amounts corresponding to each time node in the data synchronization time window is greater than or equal to the product of the threshold of the total pushed data amount and a proportionality coefficient, wherein the threshold of the total pushed data amount is the sum of the pushed data amounts of each time node in the predicted data pushing sequence, and the value range of the proportionality coefficient is (0, 1).
On the basis of the foregoing embodiment, optionally, the determining a data synchronization time window based on the push peak time node includes: and setting an initial time window with the window length of zero on the pushing peak time node, and expanding the initial time window until the sum of the pushed data amount corresponding to each time node in the data synchronization time window obtained by expansion meets a preset condition.
The initial time window refers to an initial data synchronization time window set according to the push peak time node, and the window length of the initial time window is zero. In this embodiment, an initial time window with a window length of zero is set on the push peak time node, and the initial time window is expanded towards both sides until the sum of the pushed data amounts corresponding to each time node in the data synchronization time window obtained through expansion is greater than or equal to the product of the threshold of the pushed total data amount of each time node in the prediction time period and the proportionality coefficient.
On the basis of the foregoing embodiment, optionally, the expanding the initial time window includes: for a data synchronization time window in an expansion process, comparing the pushed data volumes corresponding to a first boundary time node and a second boundary time node of the data synchronization time window respectively, and determining a comparison result; determining an expansion boundary based on the comparison result, and expanding the expansion boundary towards an expansion direction to obtain a data synchronization time window obtained by current expansion; verifying the sum of the pushed data amount corresponding to each time node in the data synchronization time window obtained by current expansion; and if the pushed data amount corresponding to each time node in the data synchronization time window obtained by current expansion meets a preset condition, stopping the expansion of the data synchronization time window.
The first boundary time node and the second boundary time node are boundary time nodes of the data synchronization time window, and for example, if the data synchronization time window is [ L, R ], the first boundary time node may be represented by L and the second boundary time node may be represented by R. In this embodiment, for a data synchronization time window in an expansion process, push data volumes corresponding to a first boundary time node and a second boundary time node of the data synchronization time window are compared, and a comparison result is determined; if the comparison result is that the pushed data amount corresponding to the first boundary time node is larger than or equal to the pushed data amount corresponding to the second boundary time node, determining an expanded boundary according to the comparison result, and expanding the expanded boundary towards the first boundary time node; if the comparison result is that the pushed data amount corresponding to the first boundary time node is smaller than the pushed data amount corresponding to the second boundary time node, determining an expanded boundary according to the comparison result, and expanding the expanded boundary towards the second boundary time node; wherein the extended boundary comprises a first boundary time node and a second boundary time node of the current data synchronization time window.
Exemplarily, assume S LR Represents the sum of the pushed data amount corresponding to each time node in the data synchronization time window, n L Indicating the amount of pushed data, n, corresponding to the first boundary time node R Indicating the amount of the pushed data corresponding to the second boundary time node, and respectively moving towards the first boundary time node direction and the second edgeThe time node direction extends the boundary and then accumulates the data synchronization time window [ L, R ]]The amount of push data of (a), namely: l = L-1, R = R +1, if there is n L >＝n R Then S is LR ＝S LR +n L Then, continuously moving L to the left, and keeping R still; if there is n L <n R Then S is LR ＝S LR +n R Then continue to shift right R, L is stationary.
In this embodiment, the sum of the pushed data amounts corresponding to the time nodes in the data synchronization time window obtained by the current expansion is verified, and if the pushed data amounts corresponding to the time nodes in the data synchronization time window obtained by the current expansion meet a preset condition, the expansion of the data synchronization time window is stopped.
Exemplary, suppose S LR Representing the sum of the pushed data amount corresponding to each time node in the data synchronization time window, S representing the threshold value of the pushed total data amount, K representing the proportionality coefficient, and K being more than 0 and less than or equal to 1, if S is LR <S X K, if the sum of the pushed data volumes corresponding to the time nodes in the data synchronization time window does not reach the data volume threshold value, continuing to expand the data synchronization time window; if S LR >If not, the sum of the pushed data amounts corresponding to the time nodes in the data synchronization time window reaches a data amount threshold, stopping expanding the data synchronization time window, recording the starting and ending positions of the data synchronization time window obtained by expansion, and recording the starting and ending positions as [ L ] s ,R e ]。
In some embodiments, optionally, a time offset vector is obtained, and the data synchronization time window obtained by the expansion is expanded for the second time based on the time offset vector to obtain a final data synchronization time window.
The time offset vector is a time offset amount preset to avoid prediction error. In this embodiment, a time offset vector is obtained, and a data synchronization time window obtained by expansion is expanded for the second time based on the time offset vector, so as to obtain a final data synchronization time window. According to the embodiment, the data synchronization time window is subjected to secondary expansion through the preset time offset, so that the prediction deviation caused by few training samples is avoided, and the accuracy of the predicted data synchronization time window is improved.
Illustratively, assuming that the preset time offset is A, the time window [ L ] is synchronized for the expanded data s ,R e ]Performing secondary expansion to obtain final data synchronization time window L s -A,R e +A]。
S130, generating control information of data synchronization based on the data synchronization time window, and sending the control information of the data synchronization to a data consumption object so that the data consumption object performs data synchronization in the data synchronization time window.
The control information refers to information for controlling a data consuming object to perform data synchronization, and specifically, the control information includes a start message and a stop message; illustratively, the start message includes a start time of the consumer process of the data consumption object, the stop message includes a stop time of the consumer process of the data consumption object, and illustratively, the start time may be L s A, the stop time may be R e + A; a data consumption object refers to a system that obtains data, i.e., a consumer. In this embodiment, control information for data synchronization, that is, a start time message and a stop time message, is generated based on the final data synchronization time window; and sending the control information of the data synchronization to a data consumption object, receiving the control information by the data consumption object, setting the starting time in the starting time message as the next self-starting time based on the starting time message, setting the stopping time in the stopping time message as the next stopping time based on the stopping time message, obtaining the next data synchronization time window, and carrying out data synchronization in the set next data synchronization time window.
According to the technical scheme, the data synchronization time window in the prediction time range is obtained through prediction of the historical data pushing sequence, the data consumption object sets the start-stop time of data synchronization according to the predicted data synchronization time window, the historical data pushing sequence of the data production object is used as a reference, so that the data consumption object can flexibly adjust the consumption plan according to the data pushing condition, meanwhile, the waiting time of the data consumption object is reduced, and the consumption of system resources is effectively reduced.
Example two
Fig. 2 is a schematic structural diagram of a message synchronization mechanism provided in the second embodiment of the present invention, and this embodiment is a preferred embodiment provided on the basis of the foregoing embodiments. As shown in fig. 2, the method includes:
1. producer pushing data to Kafka cluster
Suppose a batch server deploys i producer processes, each producer process corresponds to only one message queue Q i Let P i Denotes the ith producer process, P i Production push data D i Then push data D i Push to message queue Q i In (1).
It should be noted that in the actual message synchronization, one producer process may correspond to multiple message queues, but the implementation of the present technical solution is not affected.
2. Historical push data statistical analysis
2.1 queue Q of messages on batch Server i Constructing a historical data statistics set T i ，T i For recording message queue Q i The amount of data received per minute, exemplary, T i ＝{(t j ,n j )|0<j<=1440}, j denotes the jth minute-by-time node in 24 hours of the day, t j Denotes the jth time node, n j Denotes the t-th j-1 Time node to tth j The pushed data volume of each time node.
2.2 initializing the push prediction model M i ，M i For existing machine learning models, e.g. support vector machine models, M i Not limited to a machine learning algorithm, with a set of historical data statistics T i Training model M as a set of training samples i 。
In addition, the model M is used for the first time i When the model is in use, the training parameters of the model are initialized randomly; when the model is reused subsequently, the model parameters trained last time can be loadedAnd continuing training.
2.3 Using model M i Predicting the pushed data amount in the next 24 hours (corresponding to the predicted time period in the above embodiment), calculating the sum of the pushed data amounts of each time node in 24 hours, making S represent the sum of the pushed data amounts of each time node in 24 hours, and acquiring a pushed peak time node, which is recorded as: (t) max ,n max )，t max Representing the amount of push peak data n present max Time node corresponding to the time.
2.4 by t max As a reference, an initial data synchronization time window [ L, R ] is defined]At the beginning, the first boundary time node L and the second boundary time node R are both t max . Order S LR Representing the sum of the pushed data volumes, n, of time nodes within a data synchronization time window L Represents the amount of push data, n, corresponding to the first boundary time node L R Indicating the amount of push data corresponding to the second boundary time node R, n at the beginning L ＝n R ＝S LR ＝n max . Shift L to the left and R to the right, then accumulate the data sync time windows [ L, R]The amount of data of (c), namely: l = L-1, R = R +1, if there is n L >＝n R Then S is LR ＝S LR +n L Then, continuously moving L to the left, and keeping R still; if there is n L <n R Then S is LR ＝S LR +n R Then continue to shift right R, L is stationary.
2.5 determination of S LR Whether the pushed data amount of (2) reaches a preset data amount threshold value. Let K denote a predetermined scaling factor, 0<K<If S is =1 LR <S, K, if the pushed data volume of the data synchronization time window does not reach the data volume threshold of the total predicted data volume, continuing to expand the data synchronization time window, and repeatedly executing 2.4; if S LR >And = S × K, the pushed data amount of the data synchronization time window reaches the data amount threshold of the predicted data total amount, the data synchronization time window is stopped to move, the starting position and the ending position of the data synchronization time window at the moment are recorded, and the recording is as follows: [ L ] s ,R e ]。
2.6 because in the initial stage of model training, training samplesThe method has less cost, is easy to generate time prediction deviation, and synchronizes a time window [ L ] for data obtained by prediction s ,R e ]Performing secondary expansion, and adding a predefined time offset A, wherein the data synchronization time window is as follows: [ L ] s -A，R e +A]。L s A is as the consumer process start time, R e + O is the consumer process stop time.
3. Sending control messages
The time data synchronization time window [ L ] is obtained in step 2.6 s —O,R e +O]After that, the control node sends a start message C start To message queue Q i In which C is start Time of (1) is L s -a; control node sends stop message C end To message queue Q i In which C is end Time in (1) is R e +A。
4. Consumer consumption data
4.1 when the consumer receives the Start message C during Normal consumption start When it is, C is start The next self-starting time is set according to the starting time in the method, and similarly, when the consumer receives the message C in the consumption process end When it is, C end The next stop time is set.
4.2 Consumer Process reach C end The time of day can be regarded as the completion of the data consumption task of the day, and the producer sends a message queue Q i Pushing the historical push data of the current day forms a new training sample set T i Starting again with step 2.2, a new training sample set T is used i Training update push prediction model M i And predicting to obtain the start-stop time in the next prediction time period.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a control apparatus for message synchronization according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes:
the predicted data pushing sequence prediction module 310 is configured to obtain a historical data pushing sequence of a data production object, and predict, based on the historical data pushing sequence, a predicted data pushing sequence within a prediction time period;
the data synchronization time window determination module 320 is configured to determine a data synchronization time window within a prediction time period based on the predicted data push sequence;
the data synchronization module 330 is configured to generate control information for data synchronization based on the data synchronization time window, and send the control information for data synchronization to a data consumption object, so that the data consumption object performs data synchronization in the data synchronization time window.
On the basis of the foregoing embodiment, optionally, the predicted data pushing sequence includes a pushed data amount of each time node in the predicted time period; the data synchronization time window determining module 320 is specifically configured to determine a push peak time node in the prediction time period based on the pushed data amount of each time node in the prediction time period; and determining a data synchronization time window based on the push peak time node, wherein the sum of the push data amount corresponding to each time node in the data synchronization time window meets a preset condition.
On the basis of the foregoing embodiment, optionally, the data synchronization time window determining module 320 includes a data synchronization time window determining unit, where the data synchronization time window determining unit is configured to set an initial time window with a window length of zero on the push peak time node, and extend the initial time window until a sum of pushed data amounts corresponding to each time node in the data synchronization time window obtained by extension meets a preset condition.
On the basis of the foregoing embodiment, optionally, the data synchronization time window determining unit is further configured to compare, for a data synchronization time window in an expansion process, the pushed data amounts corresponding to the first boundary time node and the second boundary time node of the data synchronization time window, respectively, and determine a comparison result; determining an expansion boundary based on the comparison result, and expanding the expansion boundary towards an expansion direction to obtain a data synchronization time window obtained by current expansion; verifying the sum of the pushed data amount corresponding to each time node in the data synchronization time window obtained by current expansion; and if the pushed data amount corresponding to each time node in the data synchronization time window obtained by current expansion meets a preset condition, stopping the expansion of the data synchronization time window.
On the basis of the foregoing embodiment, optionally, the apparatus further includes a data synchronization time window secondary expansion module, specifically configured to obtain a time offset vector, and perform secondary expansion on the data synchronization time window obtained by expansion based on the time offset vector, so as to obtain a final data synchronization time window.
On the basis of the foregoing embodiment, optionally, the preset condition is: and the sum of the pushed data amounts corresponding to each time node in the data synchronization time window is greater than or equal to a data amount threshold, and the data amount threshold is determined based on the threshold proportion of the pushed total data amount of each time node in the prediction time period.
On the basis of the foregoing embodiment, optionally, the prediction data push sequence prediction module 310 is specifically configured to input the historical data push sequence into a push prediction model to obtain a prediction data push sequence in the prediction time period; correspondingly, the device further comprises a push prediction model updating module, which is specifically configured to obtain an actual data push sequence in a prediction time period, update the push prediction model based on the actual data push sequence and the historical data push sequence to obtain an updated push prediction model, and the updated push prediction model is used to predict and obtain a predicted data push sequence of a next prediction time period.
The message synchronization control device provided by the embodiment of the invention can execute the message synchronization control method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example four
Fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 may also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
In some embodiments, the control method of message synchronization may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the above-described message synchronization control method may be performed. Alternatively, in other embodiments, the processor 11 may be configured by any other suitable means (e.g. by means of firmware) to perform the control method of message synchronization.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
The computer program for implementing the message synchronization control method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
EXAMPLE five
An embodiment of the present invention further provides a computer-readable storage medium, where a computer instruction is stored in the computer-readable storage medium, where the computer instruction is used to enable a processor to execute a control method for message synchronization, where the method includes: acquiring a historical data pushing sequence of a data production object, and predicting to obtain a predicted data pushing sequence in a prediction time period based on the historical data pushing sequence; determining a data synchronization time window within a prediction time period based on the predicted data push sequence; and generating control information of data synchronization based on the data synchronization time window, and sending the control information of the data synchronization to the data consumption object so as to enable the data consumption object to carry out data synchronization in the data synchronization time window.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above, reordering, adding or deleting steps, may be used. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for controlling message synchronization, comprising:
acquiring a historical data pushing sequence of a data production object, and predicting to obtain a predicted data pushing sequence in a prediction time period based on the historical data pushing sequence;
determining a data synchronization time window within a prediction time period based on the predicted data push sequence;
and generating control information of data synchronization based on the data synchronization time window, and sending the control information of data synchronization to a data consumption object so that the data consumption object performs data synchronization in the data synchronization time window.
2. The method of claim 1, wherein the predicted data push sequence comprises a pushed data amount for each time node within the prediction time period;
determining a data synchronization time window within a prediction time period based on the predicted data push sequence, comprising:
determining a pushing peak time node in the prediction time period based on the pushing data volume of each time node in the prediction time period;
and determining a data synchronization time window based on the push peak time node, wherein the sum of the push data amount corresponding to each time node in the data synchronization time window meets a preset condition.
3. The method of claim 2, wherein the determining a data synchronization time window based on the push peak time node comprises:
and setting an initial time window with the window length of zero on the pushing peak time node, and expanding the initial time window until the sum of the pushed data amount corresponding to each time node in the data synchronization time window obtained by expansion meets a preset condition.
4. The method of claim 3, wherein said expanding said initial time window comprises:
for a data synchronization time window in an expansion process, comparing the pushed data volumes corresponding to a first boundary time node and a second boundary time node of the data synchronization time window respectively, and determining a comparison result;
determining an expansion boundary based on the comparison result, and expanding the expansion boundary to an expansion direction to obtain a data synchronization time window obtained by current expansion;
verifying the sum of the pushed data amount corresponding to each time node in the data synchronization time window obtained by current expansion;
and if the pushed data amount corresponding to each time node in the data synchronization time window obtained by current expansion meets a preset condition, stopping the expansion of the data synchronization time window.
5. The method of claim 3, further comprising:
and acquiring a time offset vector, and performing secondary expansion on the data synchronization time window obtained by expansion based on the time offset vector to obtain a final data synchronization time window.
6. The method according to claim 2, wherein the preset condition is: and the sum of the pushed data amounts corresponding to the time nodes in the data synchronization time window is greater than or equal to a data amount threshold, and the data amount threshold is determined based on the threshold proportion of the total pushed data amount of the time nodes in the prediction time period.
7. The method of claim 1, wherein predicting a predicted data push sequence within a prediction time period based on the historical data push sequence comprises:
inputting the historical data pushing sequence into a pushing prediction model to obtain a predicted data pushing sequence in the prediction time period;
correspondingly, the method further comprises the following steps:
and acquiring an actual data pushing sequence in a prediction time period, updating the pushing prediction model based on the actual data pushing sequence and the historical data pushing sequence to obtain an updated pushing prediction model, wherein the updated pushing prediction model is used for predicting to obtain a predicted data pushing sequence of the next prediction time period.
8. A control apparatus for message synchronization, comprising:
the prediction data pushing sequence prediction module is used for acquiring a historical data pushing sequence of a data production object and predicting to obtain a prediction data pushing sequence in a prediction time period based on the historical data pushing sequence;
the data synchronization time window determining module is used for determining a data synchronization time window in a prediction time period based on the prediction data pushing sequence;
the data synchronization module is used for generating data synchronization control information based on the data synchronization time window and sending the data synchronization control information to a data consumption object so that the data consumption object performs data synchronization in the data synchronization time window.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of controlling message synchronization of any one of claims 1-7.
10. A computer-readable storage medium storing computer instructions for causing a processor to implement the method of controlling message synchronization according to any one of claims 1 to 7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211346305.9A CN115640359A (en) | 2022-10-31 | 2022-10-31 | Message synchronization control method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211346305.9A CN115640359A (en) | 2022-10-31 | 2022-10-31 | Message synchronization control method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115640359A true CN115640359A (en) | 2023-01-24 |
Family
ID=84946960
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211346305.9A Pending CN115640359A (en) | 2022-10-31 | 2022-10-31 | Message synchronization control method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115640359A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117931484A (en) * | 2024-03-22 | 2024-04-26 | 中国人民解放军国防科技大学 | Message consumption method, device, equipment and storage medium based on sliding window |
-
2022
- 2022-10-31 CN CN202211346305.9A patent/CN115640359A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117931484A (en) * | 2024-03-22 | 2024-04-26 | 中国人民解放军国防科技大学 | Message consumption method, device, equipment and storage medium based on sliding window |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114298322B (en) | Federal learning method and apparatus, system, electronic device, and computer readable medium | |
| CN114065863A (en) | Method, device and system for federal learning, electronic equipment and storage medium | |
| CN115640359A (en) | Message synchronization control method and device, electronic equipment and storage medium | |
| CN114065864A (en) | Federal learning method, federal learning device, electronic device, and storage medium | |
| CN112631775A (en) | Model training method and device, electronic equipment and computer readable storage medium | |
| CN115358411A (en) | Data processing method, device, equipment and medium | |
| CN114818913A (en) | Decision generation method and device | |
| CN113361574A (en) | Training method and device of data processing model, electronic equipment and storage medium | |
| CN114841341B (en) | Image processing model training and image processing method, device, equipment and medium | |
| CN116523249A (en) | Production line determining method, device, equipment and storage medium | |
| CN115757047A (en) | Cloud platform state synchronization method, multi-cloud management platform and storage medium | |
| CN115719093A (en) | Distributed training method, device, system, storage medium and electronic equipment | |
| CN114051057B (en) | Cloud equipment queuing time determination method and device, electronic equipment and medium | |
| CN114141236B (en) | Language model updating method and device, electronic equipment and storage medium | |
| CN115890684A (en) | Robot scheduling method, device, equipment and medium | |
| CN115391158A (en) | Time delay determination method, system and device and electronic equipment | |
| CN113112311B (en) | Method for training causal inference model and information prompting method and device | |
| CN114611609A (en) | Graph network model node classification method, device, equipment and storage medium | |
| CN114217933A (en) | Multi-task scheduling method, device, equipment and storage medium | |
| CN114048863A (en) | Data processing method, data processing device, electronic equipment and storage medium | |
| CN113807397A (en) | Training method, device, equipment and storage medium of semantic representation model | |
| CN113313049A (en) | Method, device, equipment, storage medium and computer program product for determining hyper-parameters | |
| CN117453358A (en) | Optimization method and device for offline scheduling task, electronic equipment and storage medium | |
| CN116821046B (en) | Automatic synchronization method, device, equipment and medium for data among heterogeneous units | |
| CN115730000A (en) | Medical data integration method, device, equipment and medium based on data lake |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |