CN117609089A - Data compensation system, method, equipment and medium - Google Patents
Data compensation system, method, equipment and medium Download PDFInfo
- Publication number
- CN117609089A CN117609089A CN202311666636.5A CN202311666636A CN117609089A CN 117609089 A CN117609089 A CN 117609089A CN 202311666636 A CN202311666636 A CN 202311666636A CN 117609089 A CN117609089 A CN 117609089A
- Authority
- CN
- China
- Prior art keywords
- compensation
- data
- service
- strategy
- data packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000012545 processing Methods 0.000 claims abstract description 44
- 238000012216 screening Methods 0.000 claims abstract description 17
- 230000003993 interaction Effects 0.000 claims abstract description 15
- 238000013500 data storage Methods 0.000 claims description 27
- 238000012544 monitoring process Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 8
- 238000011161 development Methods 0.000 abstract description 9
- 230000000694 effects Effects 0.000 abstract description 9
- 230000008569 process Effects 0.000 description 20
- 230000002159 abnormal effect Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000011835 investigation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/362—Software debugging
- G06F11/3644—Software debugging by instrumenting at runtime
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a data compensation system, a method, equipment and a medium, comprising the following steps: the data interaction module and the data compensation module; the data interaction module is used for receiving a plurality of data packets to be compensated and processed sent by each service line, and sequentially distributing the data packets to a plurality of message queues according to the service lines and the time sequence; the data packet comprises a compensation service type and a compensation strategy, and the compensation service type corresponds to the compensation strategy; and the data compensation module is used for screening at least one data packet from each message queue according to the compensation service type and carrying out data compensation processing according to the corresponding compensation strategy. The invention independently encapsulates the data interaction module and the data compensation module of the data compensation system, obviously shortens the development efficiency of the data compensation service demands of various compensation strategies corresponding to different compensation service types of each service line, avoids the repeated realization of common compensation logic and flow by developers, and improves the data compensation effect.
Description
Technical Field
The present invention relates to the field of data transmission technologies, and in particular, to a data compensation system, method, device, and medium.
Background
With the popularity of the internet, various websites, such as travel websites, often need to perform data compensation according to the business requirements of various business lines. In the development process of the existing server system, the problem of data loss in the process of each service or data storage is unavoidable, and if important data loss is involved, serious consequences are caused, so that a corresponding data compensation system can be developed to compensate the lost data.
Currently, the existing compensation system technology usually scans data to be compensated by starting a timing job, the compensation strategy is usually to compensate once at fixed time intervals, and the whole process has no corresponding alarm mechanism. When the data compensation logic changes, a developer needs to repeatedly realize the common compensation logic and flow, so that the development efficiency is low and the data compensation effect is poor.
Disclosure of Invention
The invention aims to overcome the defects of low development efficiency and poor data compensation effect caused by repeated realization of common compensation logic and flow by developers when the data compensation logic changes in the prior art, and provides a data compensation system, a method, equipment and a medium.
The invention solves the technical problems by the following technical scheme:
in a first aspect, the present invention provides a data compensation system comprising: the data interaction module and the data compensation module;
the data interaction module is used for receiving a plurality of data packets to be compensated and processed sent by each service line, and sequentially distributing the data packets to a plurality of message queues according to a time sequence; the data packet comprises a compensation service type and a compensation strategy, the compensation service type corresponds to the compensation strategy, and the service line corresponds to the message queue;
and the data compensation module is used for screening at least one data packet from each message queue according to the compensation service type and carrying out data compensation processing according to the corresponding compensation strategy.
Preferably, the compensation service type comprises a database compensation service and a message compensation service, the compensation strategy comprises a first strategy library and a second strategy library, and the data compensation module comprises a first compensation unit and a second compensation unit;
the first compensation unit is used for determining a first compensation strategy from the first strategy library and carrying out data compensation processing on the data packet corresponding to the database compensation service according to the first compensation strategy;
the second compensation unit is configured to determine a second compensation policy from the second policy repository, and perform data compensation processing on the data packet corresponding to the message compensation service according to the second compensation policy.
Preferably, the data compensation module further comprises a first data storage unit and a second data storage unit;
the first data storage unit is used for storing first compensation information of the data packet corresponding to the database compensation service;
and the second data storage unit is used for storing second compensation information of the data packet corresponding to the message compensation service.
Preferably, the first compensation unit is specifically configured to:
screening the data packets corresponding to the database compensation service from all the message queues to serve as first data packets, and storing the first data packets in the first data storage unit;
and acquiring the first data packet from the first data storage unit according to a set rule, and performing data compensation processing according to the first compensation strategy.
Preferably, the second compensation unit is specifically configured to:
screening from all of the message queues
Selecting the data packet corresponding to the message compensation service as a second data packet;
and carrying out data compensation processing on the second data packet according to the second compensation strategy.
Preferably, the data compensation module further comprises a monitoring alarm unit;
and the monitoring alarm unit is used for sending out first early warning information when the data compensation processing of the data packet fails.
Preferably, the monitoring alarm unit is further configured to send second early warning information outwards when the number of times of compensation of the data packet exceeds a preset threshold.
In a second aspect, the present invention provides a data compensation method, applied to a server, where the data compensation method includes:
receiving a plurality of data packets to be compensated and transmitted by each service line, and sequentially distributing the data packets to a plurality of message queues according to a time sequence; the data packet comprises a compensation service type and a compensation strategy, the compensation service type corresponds to the compensation strategy, and the service line corresponds to the message queue;
and screening at least one data packet from each message queue according to the compensation service type, and carrying out data compensation processing according to the corresponding compensation strategy.
In a third aspect, the present invention provides an electronic device comprising a processor, a memory and a computer program stored on the memory and executable on the processor, the computer program implementing the data compensation method according to the third aspect when executed by the processor.
In a fourth aspect, the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data compensation method according to the third aspect.
The invention has the positive progress effects that: the data interaction module is used for receiving a plurality of data packets to be compensated and transmitted by each service line, and sequentially distributing the data packets to a plurality of message queues according to the time sequence according to the service lines; the data packet comprises a compensation service type and a compensation strategy, and the compensation service type corresponds to the compensation strategy; and the data compensation module is used for screening at least one data packet from each message queue according to the compensation service type and carrying out data compensation processing according to the corresponding compensation strategy. The invention independently encapsulates the data interaction module and the data compensation module of the data compensation system, obviously shortens the development efficiency of the data compensation service demands of various compensation strategies corresponding to different compensation service types of each service line, avoids the repeated realization of common compensation logic and flow by developers, and improves the data compensation effect.
Drawings
Fig. 1 is a schematic block diagram of a data compensation system according to embodiment 1 of the present invention.
Fig. 2 is a diagram illustrating an application example of the data compensation system according to embodiment 1 of the present invention.
Fig. 3 is a first flowchart of a data compensation method according to embodiment 2 of the present invention.
Fig. 4 is a second flowchart of the data compensation method of embodiment 2 of the present invention.
Fig. 5 is an application example diagram of the data compensation method of embodiment 2 of the present invention.
Fig. 6 is a schematic hardware structure of an electronic device according to embodiment 3 of the present invention.
Detailed Description
The invention is further illustrated by means of the following examples, which are not intended to limit the scope of the invention.
Example 1
The present embodiment provides a data compensation system, as shown in fig. 1, by way of example and not limitation, comprising: a data interaction module 110 and a data compensation module 120;
the data interaction module 110 is configured to receive a plurality of data packets to be compensated and sent by each service line, and sequentially allocate the data packets to a plurality of message queues according to a time sequence; the data packet comprises a compensation service type and a compensation strategy, the compensation service type corresponds to the compensation strategy, and the service line corresponds to the message queue;
the data compensation module 120 is configured to screen at least one data packet from each message queue according to the compensation service type, and perform data compensation processing according to a corresponding compensation policy.
The data interaction module 110 may obtain a plurality of data packets to be compensated and sent by the service line 1, the service line 2, the service line 3, the service line 4 and the service line 5, where the service line 1 corresponds to sending hotel service data, the service line 2 corresponds to sending ticket service data, the service line 3 corresponds to sending group trip product service data, the service line 4 corresponds to free running product service data, and the service line 5 corresponds to sending play product service data. The data packets sent by the service line 1 are distributed to the message queue 1 according to the time sequence, the data packets sent by the service line 2 are distributed to the message queue 2 according to the time sequence, the data packets sent by the service line 3 are distributed to the message queue 3 according to the time sequence, the data packets sent by the service line 4 are distributed to the message queue 4 according to the time sequence, and the data packets sent by the service line 5 are distributed to the message queue 5 according to the time sequence.
The data compensation module 120 performs data compensation in batches according to the compensation service type marked by each data packet and the compensation policy corresponding to the compensation service type, or performs data compensation simultaneously according to the compensation policy corresponding to the compensation service type in a multithreading manner.
In one embodiment, the compensation service type includes a database compensation service and a message compensation service, the compensation policy includes a first policy bank and a second policy bank, and the data compensation module 120 includes a first compensation unit 121 and a second compensation unit 122;
a first compensation unit 121, configured to determine a first compensation policy from a first policy base, and perform data compensation processing on the data packet corresponding to the database compensation service according to the first compensation policy;
the second compensation unit 122 is configured to determine a second compensation policy from the second policy repository, and perform data compensation processing on the data packet corresponding to the message compensation service according to the second compensation policy.
Each service line sets an identifier corresponding to a compensation policy for each data packet in advance, and the first compensation unit 121 screens the first compensation policy from a plurality of compensation policies in the first policy base according to the identifier, and performs data compensation processing according to the first compensation policy. The second compensation unit 122 screens out a second compensation policy from a plurality of compensation policies in the second policy repository according to the identification, and performs data compensation processing according to the second compensation policy. The method ensures that when the first compensation strategy or the second compensation strategy is changed, the requirement change of the data compensation service can be met only by independently redeveloping the data compensation module, so that a developer is prevented from repeatedly realizing common compensation logic and flow, the service development time is shortened, and the data compensation effect is improved.
In one embodiment, the data compensation module 120 further includes a first data storage unit 123 and a second data storage unit 124;
a first data storage unit 123 for storing first compensation information of a data packet corresponding to the database compensation service;
and a second data storage unit 124 for storing second compensation information of the data packet corresponding to the message compensation service.
The first compensation information and the second compensation information include, but are not limited to, compensation data, the number of times of compensation, a compensation state, and a compensation process. The compensation state comprises a compensation success or a compensation failure, and the compensation process comprises a compensation process not started, a compensation process started and ended. The method realizes dynamic monitoring of the data compensation condition of the data packet and abnormal alarm of the abnormal condition in the data compensation process, and improves the data compensation efficiency.
In one embodiment, the first compensation unit 121 is specifically configured to:
screening data packets corresponding to the database compensation service from all message queues as first data packets, and storing the first data packets in the first data storage unit 123;
the first data packet obtained from the first data storage unit 123 according to the set rule performs data compensation processing according to the first compensation policy.
After storing the data packets corresponding to the database compensation service in the first data storage unit 123, the data compensation process may be performed by acquiring one first data packet from the first data storage unit 123 at a time, or by acquiring a plurality of first data packets from the first data storage unit 123 at a time. In this embodiment, the setting rule includes the number value of the first data packet for each data compensation, the compensation processing time interval, and the like. The first compensation strategy can comprise compensation methods such as compensation times setting, compensation content constraint conditions, compensation duration setting and the like.
In one embodiment, the second compensation unit 122 is specifically configured to:
screening data packets corresponding to the message compensation service from all message queues to serve as second data packets;
and carrying out data compensation processing on the second data packet according to a second compensation strategy.
And carrying out data compensation processing on the second data packet directly screened from the message queue by using a second compensation strategy, wherein the second compensation strategy is set by the compensation times and the compensation result, and the embodiment is not particularly limited.
In one embodiment, the data compensation module 120 further includes a monitor alarm unit 125;
the monitoring alarm unit 125 is configured to send the first early warning information to the outside when the data compensation processing of the data packet fails.
In one embodiment, the monitoring alarm unit 125 is further configured to send the second early warning information outwards when the number of times of compensation of the data packet exceeds a preset threshold.
Each time when the data compensation processing of the data packet is unsuccessful, first early warning information is sent outwards to prompt the compensation thread to interrupt; if the preset compensation times are 3 times, when the compensation times of the data packet exceed 3 times, second early warning information is sent outwards to prompt the end of the compensation thread. The method provides various compensation strategies or algorithms to meet various compensation scene requirements, dynamically monitors the compensation process, facilitates the timely investigation of abnormal conditions by developers, and improves the compensation business efficiency.
In one embodiment, as shown in fig. 2, the data interaction module acquires data packets of various compensation scenes sent by each service line by using a compensation component and distributes the data packets to a data compensation queue, wherein the compensation scenes include but are not limited to storage compensation scenes and external calling compensation scenes, and the compensation service module performs data compensation processing on all the data packets for database compensation service and message compensation service, and when abnormal conditions are met, the compensation monitoring alarm information is triggered. The data compensation system is characterized in that the compensation action and the compensation service logic are decoupled and coupled, the modules are independent, the compensation action is refined and independent into a common compensation system, two compensation driving modes (message queue driving and timing task driving) are provided, multiple compensation retry strategies (conventional, ladder and custom compensation retry strategies and the like) are provided, an alarm module can configure an alarm for compensation content, and the system problem can be found in advance by the alarm when an alarm condition is met.
In this embodiment, a data compensation system is provided, where a data interaction module is configured to receive a plurality of data packets to be compensated and sent by each service line, and sequentially allocate the data packets to a plurality of message queues according to a time sequence according to the service lines; the data packet comprises a compensation service type and a compensation strategy, and the compensation service type corresponds to the compensation strategy; and the data compensation module is used for screening at least one data packet from each message queue according to the compensation service type and carrying out data compensation processing according to the corresponding compensation strategy. The invention independently encapsulates the data interaction module and the data compensation module of the data compensation system, obviously shortens the development efficiency of the data compensation service demands of various compensation strategies corresponding to different compensation service types of each service line, avoids the repeated realization of common compensation logic and flow by developers, and improves the data compensation effect.
Example 2
The present embodiment provides a data compensation method, as shown in fig. 3, by way of example and not limitation, including:
s10, receiving a plurality of data packets to be compensated and processed sent by each service line, and sequentially distributing the data packets to a plurality of message queues according to a time sequence; the data packet includes a compensation service type and a compensation policy, the compensation service type and the compensation policy correspond to each other, and the service line corresponds to the message queue.
S20, screening at least one data packet from each message queue according to the compensation service type, and carrying out data compensation processing according to the corresponding compensation strategy.
For the step S10, a plurality of data packets to be compensated and processed sent by the service line 1, the service line 2, the service line 3, the service line 4 and the service line 5 may be obtained, where the service line 1 corresponds to sending hotel service data, the service line 2 corresponds to sending ticket service data, the service line 3 corresponds to sending group trip product service data, the service line 4 corresponds to free running product service data, and the service line 5 corresponds to sending play product service data. The data packets sent by the service line 1 are distributed to the message queue 1 according to the time sequence, the data packets sent by the service line 2 are distributed to the message queue 2 according to the time sequence, the data packets sent by the service line 3 are distributed to the message queue 3 according to the time sequence, the data packets sent by the service line 4 are distributed to the message queue 4 according to the time sequence, and the data packets sent by the service line 5 are distributed to the message queue 5 according to the time sequence.
For the above step S20, the data compensation module 120 performs data compensation in batches according to the compensation policy corresponding to the compensation service type according to the compensation service type marked by each data packet, or performs data compensation simultaneously according to the compensation policy corresponding to the compensation service type in a multi-thread manner.
In one embodiment, as shown in fig. 4, the compensation service type includes a database compensation service, the compensation policy includes a first policy repository, and step S20 includes:
s21, determining a first compensation strategy from a first strategy library, and carrying out data compensation processing on a data packet corresponding to the database compensation service according to the first compensation strategy;
after step S21, the data compensation method further includes:
s31, storing first compensation information of a data packet corresponding to the database compensation service.
The compensation service type further includes a message compensation service, the compensation policy includes a second policy repository, and step S20 includes:
s22, determining a second compensation strategy from the second strategy library, and carrying out data compensation processing on the data packet corresponding to the message compensation service according to the second compensation strategy.
After step S21, the data compensation method further includes:
and S32, storing second compensation information of the data packet corresponding to the message compensation service.
Each service line sets an identifier corresponding to the compensation strategy for each data packet in advance, screens out a first compensation strategy from a plurality of compensation strategies in a first strategy library according to the identifier, and performs data compensation processing according to the first compensation strategy. And screening a second compensation strategy from a plurality of compensation strategies in the second strategy library according to the identification, and carrying out data compensation processing according to the second compensation strategy. The method ensures that when the first compensation strategy or the second compensation strategy is changed, the requirement change of the data compensation service can be met only by independently redeveloping the data compensation module, so that a developer is prevented from repeatedly realizing common compensation logic and flow, the service development time is shortened, and the data compensation effect is improved.
The first compensation information and the second compensation information include, but are not limited to, compensation data, the number of times of compensation, a compensation state, and a compensation process. The compensation state comprises a compensation success or a compensation failure, and the compensation process comprises a compensation process not started, a compensation process started and ended. The method realizes dynamic monitoring of the data compensation condition of the data packet and abnormal alarm of the abnormal condition in the data compensation process, and improves the data compensation efficiency.
In one embodiment, step S21 specifically includes:
screening data packets corresponding to the database compensation service from all the message queues to serve as first data packets, and storing the first data packets in a first data storage unit;
and acquiring the first data packet from the first data storage unit according to the set rule, and performing data compensation processing according to a first compensation strategy.
After the data packets corresponding to the database compensation service are stored in the first data storage unit, one first data packet may be acquired from the first data storage unit at a time to perform data compensation processing, or a plurality of first data packets may be acquired from the first data storage unit at a time to perform data compensation processing simultaneously. In this embodiment, the setting rule includes a number value of first data packets for each data compensation, a compensation processing time interval, and the like, and the first compensation policy may include compensation methods such as compensation times setting, compensation content constraint conditions, and compensation duration setting.
In one embodiment, step S22 specifically includes:
screening data packets corresponding to the message compensation service from all message queues to serve as second data packets;
and carrying out data compensation processing on the second data packet according to a second compensation strategy.
And carrying out data compensation processing on the second data packet directly screened from the message queue by using a second compensation strategy, wherein the second compensation strategy is set by the compensation times and the compensation result, and the embodiment is not particularly limited.
In one embodiment, as shown in fig. 4, the data compensation method further includes:
and S40, when the data compensation processing of the data packet fails, the first early warning information is sent outwards.
And S50, when the compensation times of the data packet exceeds a preset threshold value, sending second early warning information outwards.
Each time when the data compensation processing of the data packet is unsuccessful, first early warning information is sent outwards to prompt the compensation thread to interrupt; if the preset compensation times are 3 times, when the compensation times of the data packet exceed 3 times, second early warning information is sent outwards to prompt the end of the compensation thread. The method provides various compensation strategies or algorithms to meet various compensation scene requirements, dynamically monitors the compensation process, facilitates the timely investigation of abnormal conditions by developers, and improves the compensation business efficiency.
In one embodiment, as shown in fig. 5, the first compensation policy for the database compensation service may include setting a timing task, a database query process, a database compensation process, and a database update process, and the second compensation policy for the message compensation service may include setting a number of compensation retries, whether the failure exceeds the number of retries, whether the compensation is successful, and a database save process. The method ensures that when the first compensation strategy or the second compensation strategy is changed, the requirement change of the data compensation service can be met only by singly and repeatedly modifying the compensation strategy.
In this embodiment, a data compensation method is provided, which performs data compensation processing on a plurality of data packets sent by each service line in a message queue according to a compensation service type and a compensation policy, so as to significantly shorten development efficiency of data compensation service requirements of a plurality of compensation policies corresponding to different compensation service types of each service line, avoid developers from repeatedly implementing common compensation logic and flow, and improve data compensation effects.
Example 3
Fig. 6 is a schematic structural diagram of an electronic device according to the present embodiment. The electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the data compensation method of embodiment 2 when executing the program. The electronic device 90 shown in fig. 6 is merely an example and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 6, the electronic device 90 may be embodied in the form of a general purpose computing device, which may be a server device, for example. Components of the electronic device 90 may include, but are not limited to: the at least one processor 91, the at least one memory 92, a bus 93 connecting the different system components, including the memory 92 and the processor 91.
The bus 93 includes a data bus, an address bus, and a control bus.
The memory 92 may include volatile memory such as Random Access Memory (RAM) 921 and/or cache memory 922, and may further include Read Only Memory (ROM) 923.
Memory 92 may also include a program/utility 925 having a set (at least one) of program modules 924, such program modules 924 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The processor 91 executes various functional applications and data processing, such as the data compensation method of embodiment 2 of the present invention, by running a computer program stored in the memory 92.
The electronic device 90 may also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). Such communication may occur through an input/output (I/O) interface 95. Also, model-generating device 90 may also communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the internet via network adapter 96. As shown in fig. 6, the network adapter 96 communicates with other modules of the model-generating device 90 via a bus 93. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in connection with the model-generating device 90, including, but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, data backup storage systems, and the like.
It should be noted that although several units/modules or sub-units/modules of an electronic device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module in accordance with embodiments of the present invention. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.
Example 4
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data compensation method of embodiment 2.
More specifically, among others, readable storage media may be employed including, but not limited to: portable disk, hard disk, random access memory, read only memory, erasable programmable read only memory, optical storage device, magnetic storage device, or any suitable combination of the foregoing.
In a possible embodiment, the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps of implementing the data compensation method of embodiment 2, when the program product is run on the terminal device.
Wherein the program code for carrying out the invention may be written in any combination of one or more programming languages, the program code may execute entirely on the user device, partly on the user device, as a stand-alone software package, partly on the user device, partly on a remote device or entirely on the remote device.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the principles and spirit of the invention, but such changes and modifications fall within the scope of the invention.
Claims (10)
1. A data compensation system, for application to a server, the data compensation system comprising: the data interaction module and the data compensation module;
the data interaction module is used for receiving a plurality of data packets to be compensated and processed sent by each service line, and sequentially distributing the data packets to a plurality of message queues according to a time sequence; the data packet comprises a compensation service type and a compensation strategy, the compensation service type corresponds to the compensation strategy, and the service line corresponds to the message queue;
and the data compensation module is used for screening at least one data packet from each message queue according to the compensation service type and carrying out data compensation processing according to the corresponding compensation strategy.
2. The data compensation system of claim 1, wherein the compensation service type comprises a database compensation service and a message compensation service, the compensation policy comprises a first policy repository and a second policy repository, and the data compensation module comprises a first compensation unit and a second compensation unit;
the first compensation unit is used for determining a first compensation strategy from the first strategy library and carrying out data compensation processing on the data packet corresponding to the database compensation service according to the first compensation strategy;
the second compensation unit is configured to determine a second compensation policy from the second policy repository, and perform data compensation processing on the data packet corresponding to the message compensation service according to the second compensation policy.
3. The data compensation system of claim 2, wherein the data compensation module further comprises a first data storage unit and a second data storage unit;
the first data storage unit is used for storing first compensation information of the data packet corresponding to the database compensation service;
and the second data storage unit is used for storing second compensation information of the data packet corresponding to the message compensation service.
4. A data compensation system according to claim 3, wherein the first compensation unit is specifically configured to:
screening the data packets corresponding to the database compensation service from all the message queues to serve as first data packets, and storing the first data packets in the first data storage unit;
and acquiring the first data packet from the first data storage unit according to a set rule, and performing data compensation processing according to the first compensation strategy.
5. A data compensation system according to claim 3, wherein the second compensation unit is specifically configured to:
screening the data packets corresponding to the message compensation service from all the message queues to be used as second data packets;
and carrying out data compensation processing on the second data packet according to the second compensation strategy.
6. The data compensation system of claim 3, wherein the data compensation module further comprises a monitoring alarm unit;
and the monitoring alarm unit is used for sending out first early warning information when the data compensation processing of the data packet fails.
7. The data compensation system of claim 6, wherein,
and the monitoring alarm unit is also used for sending second early warning information outwards when the compensation times of the data packet exceed a preset threshold value.
8. The data compensation method is characterized by being applied to a server, and comprises the following steps:
receiving a plurality of data packets to be compensated and transmitted by each service line, and sequentially distributing the data packets to a plurality of message queues according to a time sequence; the data packet comprises a compensation service type and a compensation strategy, the compensation service type corresponds to the compensation strategy, and the service line corresponds to the message queue;
and screening at least one data packet from each message queue according to the compensation service type, and carrying out data compensation processing according to the corresponding compensation strategy.
9. An electronic device, comprising: a processor, a memory and a computer program stored on the memory for execution on the processor, which when executed by the processor implements the data compensation method of claim 8.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the data compensation method according to claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311666636.5A CN117609089A (en) | 2023-12-06 | 2023-12-06 | Data compensation system, method, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311666636.5A CN117609089A (en) | 2023-12-06 | 2023-12-06 | Data compensation system, method, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117609089A true CN117609089A (en) | 2024-02-27 |
Family
ID=89949767
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311666636.5A Pending CN117609089A (en) | 2023-12-06 | 2023-12-06 | Data compensation system, method, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117609089A (en) |
-
2023
- 2023-12-06 CN CN202311666636.5A patent/CN117609089A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8239868B2 (en) | Computer system, servers constituting the same, and job execution control method and program | |
| US8868986B2 (en) | Relevant alert delivery in a distributed processing system with event listeners and alert listeners | |
| EP1750200A2 (en) | System and method for executing job step, and computer product | |
| US10503558B2 (en) | Adaptive resource management in distributed computing systems | |
| US9495201B2 (en) | Management of bottlenecks in database systems | |
| US9535754B1 (en) | Dynamic provisioning of computing resources | |
| CN109558260B (en) | Kubernetes fault elimination system, method, equipment and medium | |
| US20160371122A1 (en) | File processing workflow management | |
| US11132191B2 (en) | Software and firmware updates of computing systems | |
| US7937499B1 (en) | Methods and apparatus for dynamically switching between polling and interrupt mode for a ring buffer of a network interface card | |
| CN112118315A (en) | Data processing system, method, device, electronic equipment and storage medium | |
| US8280197B1 (en) | Managed background loading of image resources in resource-constrained devices | |
| US20110302302A1 (en) | Adaptive mobile cloud system using private virtual intance and construction method thereof | |
| CN111858007A (en) | Task scheduling method and device based on message middleware | |
| US11334436B2 (en) | GPU-based advanced memory diagnostics over dynamic memory regions for faster and efficient diagnostics | |
| CA3129982A1 (en) | Method and system for accessing distributed block storage system in kernel mode | |
| US9507637B1 (en) | Computer platform where tasks can optionally share per task resources | |
| CN111400041A (en) | Server configuration file management method and device and computer readable storage medium | |
| CN114721807A (en) | Batch business task execution method, device, equipment, medium and program product | |
| US20220398073A1 (en) | System and method for intelligent update flow across inter and intra update dependencies | |
| CN111800511B (en) | Synchronous login state processing method, system, equipment and readable storage medium | |
| EP2110748A2 (en) | Cluster control apparatus, control system, control method, and control program | |
| US20070174836A1 (en) | System for controlling computer and method therefor | |
| CN113485812A (en) | Partition parallel processing method and system based on large data volume task | |
| JP2003173272A (en) | Information processing system, information processor and maintenance center |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |