CN113176951A - Message copying method, device, equipment and computer readable medium - Google Patents
Message copying method, device, equipment and computer readable medium Download PDFInfo
- Publication number
- CN113176951A CN113176951A CN202110425821.XA CN202110425821A CN113176951A CN 113176951 A CN113176951 A CN 113176951A CN 202110425821 A CN202110425821 A CN 202110425821A CN 113176951 A CN113176951 A CN 113176951A
- Authority
- CN
- China
- Prior art keywords
- message
- messages
- time interval
- configuration information
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 230000010076 replication Effects 0.000 claims abstract description 37
- 238000001914 filtration Methods 0.000 claims description 34
- 238000004891 communication Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 7
- 230000003362 replicative effect Effects 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 14
- 238000012545 processing Methods 0.000 description 19
- 238000010586 diagram Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to the field of software engineering technologies, and in particular, to a method, an apparatus, a device, and a computer readable medium for message replication. The method is used for copying the message from the message source to the copy database according to the message pulling time interval, and comprises the following steps: acquiring configuration information of the message pulling time interval, wherein the configuration information at least comprises residual storage message capacity information; selecting target messages to be copied to a copy database from the messages pulled in the message pulling time interval according to the acquired information of the capacity of the residual storage messages; sending the selected target message to the replication database. The method of the embodiment of the invention greatly reduces the pressure of huge online flow on the system, and improves the storage efficiency, flexibility and accuracy of the message copying process.
Description
Technical Field
The present invention relates to the field of software engineering technologies, and in particular, to a method, an apparatus, a device, and a computer readable medium for message replication.
Background
The main functions of the message replication platform include a traffic storage process and a playback process, wherein the traffic storage is specific to different data sources (such as a database, a log, asynchronous messages and the like) and the operation modes of the traffic storage are different.
In the process of implementing the invention, the inventor finds that: the method for acquiring the online flow from the asynchronous message has the problems of overlarge online flow, overlarge system pressure, incapability of freely controlling the message receiving process and the like because a monitoring mode is adopted for receiving the message, and the configuration of a data switch is read from a database.
In view of the above problems, no effective solution has been proposed, and a new message copying method is urgently needed to solve the above problems.
Disclosure of Invention
In view of the above, in order to solve the technical problem or at least partially solve the technical problem, embodiments of the present invention provide a message copying method, device, apparatus, and computer-readable medium.
In a first aspect, an embodiment of the present invention provides a message copying method, configured to copy a message from a message source to a copy database at a message pull time interval, where the method includes:
acquiring configuration information of the message pulling time interval, wherein the configuration information at least comprises residual storage message capacity information;
selecting target messages to be copied to a copy database from the messages pulled in the message pulling time interval according to the acquired information of the capacity of the residual storage messages;
sending the selected target message to the replication database.
Optionally, the sending the selected target message to the replication database includes:
and updating the configuration information in the replication database to obtain updated configuration information, and using the updated configuration information as the configuration information of the next message pull time interval.
Optionally, the obtaining the configuration information of the message pull time interval includes:
sending a configuration information request to the replication database;
receiving the configuration information returned from the replicated database.
Optionally, the configuration information further includes a filtering condition,
selecting the target message to be copied to the copy database from the messages pulled in the message pulling time interval according to the acquired information of the capacity of the residual storage messages, wherein the selecting step comprises the following steps:
and selecting the messages which meet the filtering condition and the quantity of which does not exceed the residual storage message capacity information from the messages pulled in the message pulling time interval as target messages to be copied into a copy database.
Optionally, the filter condition is a specified keyword.
Optionally, the selecting, from the messages pulled in the message pulling time interval, messages that meet the filtering condition and whose number does not exceed the information of the remaining storage message capacity as target messages to be copied to a copy database includes:
selecting messages meeting the filtering condition from the messages pulled in the message pulling time interval;
and if the number of the messages meeting the filtering condition is larger than the number indicated by the residual storage message capacity information, reserving the messages with the number indicated by the residual storage message capacity information from the messages meeting the filtering condition as target messages to be copied into a copy database.
Optionally, the selecting, from the messages pulled in the message pulling time interval, messages that meet the filtering condition and whose number does not exceed the number indicated by the remaining storage message capacity information as target messages to be copied to a copy database includes:
setting an initial value of a counter equal to the number of indications of the remaining storage message capacity information;
determining one of the messages pulled in the message pulling time interval, if the message meets the filtering condition, regarding the message as a target message to be copied into a copy database, subtracting 1 from the value of the counter, and then performing judgment on whether the next message in the messages pulled in the message pulling time interval meets the filtering condition.
Optionally, the configuration information includes a message source identification,
selecting a target message to be copied to a copy database from the messages pulled from the message source within the message pulling time interval according to the acquired information of the capacity of the residual storage messages, wherein the selecting step comprises the following steps:
pulling a message from a message source corresponding to the message source identification within the message pull time interval;
and selecting the target message to be copied to the copy database from the message pulled by the message source corresponding to the message source identifier according to the acquired residual storage message capacity information.
Optionally, the pulling a message from a message source corresponding to the message source identifier in the message pull time interval includes:
determining a pulling mode corresponding to the message source identifier;
and pulling the message from the message source corresponding to the message source identifier according to the determined pulling mode.
Optionally, the message pull time interval is set by at least one of the following ways:
set to a fixed time interval;
setting the message pulling time interval according to a preset time function;
setting the message pull time interval based on the number of messages from the message source pulled in the last pull time interval;
and setting the message pulling time interval based on the number of messages indicated by the remaining storage message capacity information in the last pulling time interval.
In a second aspect, an embodiment of the present invention provides a message copying apparatus for copying a message from a message source to a copy database at a message pull time interval, including:
a configuration information obtaining unit, configured to obtain configuration information of the message pull time interval, where the configuration information at least includes information of remaining storage message capacity;
a target message selection unit, which selects the target message to be copied to the copy database according to the acquired information of the capacity of the residual storage message from the messages pulled in the message pulling time interval;
and the target message sending unit is used for sending the selected target message to the replication database.
In a third aspect, an embodiment of the present invention provides an electronic device, including a memory, a processor, a communication interface, and a communication bus, where the memory stores a computer program that is executable on the processor, and the memory and the processor communicate with the communication interface through the communication bus, and when the processor executes the computer program, the steps of the method according to any embodiment of the present invention are implemented.
In a fourth aspect, the present invention provides a computer-readable medium storing non-volatile program code executable by a processor, the program code causing the processor to execute the method according to any one of the embodiments of the present invention.
The technical scheme provided by the embodiment of the invention has the following beneficial effects: by copying the message from the message source to the copy database according to the message pulling time interval, the configuration does not need to be frequently read in real-time monitoring in the message copying process, so that the problems of overlarge database pressure and inconsistent data caused by message copying in the conventional monitoring mode are solved, the pressure of huge online flow on a system is greatly reduced, and the efficiency, the flexibility and the accuracy of message storage are improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the related art, the drawings used in the description of the embodiments or the related art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
Fig. 1 is a schematic diagram of a hardware environment of an alternative message replication method according to an embodiment of the present invention;
fig. 2 is a flowchart of an alternative message copying method according to an embodiment of the present invention;
FIG. 3 is a block diagram of an alternative message replication system provided in accordance with an embodiment of the present invention;
fig. 4 is a block diagram of an alternative message copying apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an alternative electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in themselves. Thus, "module" and "component" may be used in a mixture.
For the understanding of the embodiments of the present invention, the system architecture related to the embodiments of the present invention will be described first with reference to the accompanying drawings.
Referring to fig. 1, a schematic diagram of a hardware environment of an optional message replication method according to an embodiment of the present invention is provided.
The message replication method can be applied to a hardware environment formed by the terminal 101 and the server 103 as shown in fig. 1, and the traffic playback system can be mounted on the terminal or the server. As shown in fig. 1, a server 103 is connected to a terminal 101 through a network, which may be used to provide services for the terminal or a client installed on the terminal, and a database 105 may be provided on the server or separately from the server, and is used to provide data storage services for the server 103, and the network includes but is not limited to: wide area network, metropolitan area network, or local area network, and the terminal 101 includes but is not limited to a PC, a cell phone, a tablet computer, and the like.
The message replication method in the embodiment of the present invention may be executed by the server 103, or may be executed by both the server 103 and the terminal 101.
As shown in fig. 2, the message copying method according to the embodiment of the present invention copies a message from a message source to a copy database at a message pull time interval, in order to avoid a problem of high database pressure caused by performing a read determination and a drop determination under a determination condition every time a message is monitored in a conventional monitoring manner, and includes the following steps:
step S202, obtaining configuration information of the message pulling time interval, wherein the configuration information at least comprises residual storage message capacity information;
step S204, selecting target messages to be copied to a copy database from the messages pulled in the message pulling time interval according to the acquired information of the capacity of the residual storage messages;
step S206, sending the selected target message to the copy database.
The above steps are described in detail below.
Specifically, the message may be flow data of the platform, and optionally, for example, the message may be order information, for example, at least one field of a plurality of fields including an order number, a recipient, a product number, a recipient address, a mobile phone number, and the like. Typically, the sender of the message is the producer and the recipient is the consumer, both of which are related by the subject of the message. The data format of the message is customized by a producer, and the intermediate platform for transmitting the message sends the message according to the specified template format. Wherein the fields may be arranged in a different order.
Optionally, the configuration information is stored in a database, for example, a mysql database, for the configuration information obtaining unit to read and call through the reading device.
Alternatively, the message pull time interval of the pull message may be set to a fixed time interval of 1 second, 5 seconds, 1 minute, or the like according to the platform processing requirement, or may be set to a dynamic time interval that dynamically changes with the passage of time. In addition, the message pull time interval may also be set according to the platform processing capability or the platform processing requirement of the message replication platform.
In the prior art, the synchronization time difference may cause the actual amount of stored data to be larger than the configured threshold switch value, and whether to store the data is determined by reading the data from the buffer, and the buffer during reading is not the latest configuration, so the actual number of drops may be larger than the number of configurations, and the number may not be controllable. The embodiment of the invention copies the message from the message source to the copy database according to the message pulling time interval, thereby fundamentally and effectively solving the problems.
Specifically, when the server of the message replication platform starts, the task is executed by setting an appropriate timing through a timer, and the traffic pulling operation is executed at intervals. Therefore, the configuration is not required to be read every time a message is consumed, the consistency of the data volume and the switch configuration can be ensured, the operation times of the database are greatly reduced, the speed of reading the memory cache is increased, the data storage efficiency of the message copying platform is improved, the controllability of the processing capacity of the platform can be realized, and the uncontrollable conditions of the platform copying overload operation and the like can not occur.
In addition, preferably, in the method according to the embodiment of the present invention, the template format of the message may be set by the message replication platform, and the pulling methods of the message may be different or the same according to different traffic sources.
According to an embodiment of the present invention, the message copying method, wherein the step S206 further includes: step S206A, updating the configuration information in the replication database to obtain updated configuration information, and using the updated configuration information as the configuration information of the next message pull time interval.
Therefore, the configuration information of the next message pulling time interval is set in response to the configuration information in the updated database, the flow copying process of the platform can be mastered in time, and the adjustment can be performed in time according to the processing capacity and the processing task of the platform. The improvement of the overall processing efficiency of the platform is ensured.
According to an embodiment of the present invention, the message copying method, wherein the step S204 includes: step S204A, sending a configuration information request to the replication database; step S204B, receiving the configuration information returned from the replication database.
Since it is set that the replicated database feeds back the corresponding configuration information in response to the configuration information request, it is not necessary to provide feedback of the configuration information if the configuration information request is not received. Compared with the prior real-time answering mode in the prior art, the embodiment of the invention does not need to uninterruptedly read the configuration information from the database, thereby greatly reducing the processing pressure of the replication platform and further improving the overall replication capacity of the platform.
According to an embodiment of the present invention, the configuration information further includes a filtering condition, wherein step S204 further includes: step S204C, selecting, from the messages pulled in the message pulling time interval, messages that meet the filtering condition and whose number does not exceed the information of the remaining storage message capacity, as target messages to be copied to the copy database.
According to an embodiment of the invention, the message replication method is characterized in that the filtering condition is a specified keyword. And if the pulled message contains the keywords specified in the configuration information and the number of the keywords does not exceed the information of the residual storage message capacity, copying the message as a target message.
According to the embodiment of the invention, the messages which meet the filtering condition and are not more than the residual storage message capacity information are selected from the pulled messages, and other unnecessary data except the target message can be directly removed, so that the quantity of the overall copied data of the platform is greatly reduced, the copying platform has larger message copying capability, and more data to be copied can be copied with the residual capacity.
According to an embodiment of the present invention, the message copying method, wherein step S204C includes: step S204Ca, selecting the messages meeting the filtering condition from the messages pulled in the message pulling time interval; step S204Cb, if the number of messages meeting the filtering condition is greater than the number indicated by the remaining storage message capacity information, retaining the messages of the number indicated by the remaining storage message capacity information from the messages meeting the filtering condition as the target messages to be copied into the copy database.
In the embodiment of the present invention, if the number of messages meeting the filtering condition is less than or equal to the number indicated by the remaining storage message capacity information, it goes without saying that all of the messages meeting the filtering condition may be stored in the database as target messages. However, once the number of messages meeting the filtering condition is greater than the number indicated by the remaining storage message capacity information, the copy action is not performed any more. In this case, the embodiment of the present invention effectively avoids the situation of wasteful consumption of the remaining storage message capacity by setting the number of messages indicated by the remaining storage message capacity information to be reserved from the messages meeting the filtering condition, thereby sufficiently improving the entire replication capacity and processing efficiency of the platform.
According to an embodiment of the present invention, the message copying method, wherein step S204C includes: step S204Cc, setting an initial value of a counter equal to the number indicated by the remaining storage message capacity information; step S204Cd, determining one of the messages pulled in the message pull time interval, if the message meets the filtering condition, taking the message as a target message to be copied into the copy database, subtracting 1 from the counter value, and then performing a determination on whether the next message in the messages pulled in the message pull time interval meets the filtering condition.
Therefore, the number of a batch of messages pulled in the same pulling time interval can be correspondingly set according to the residual storage message capacity information, then whether the batch of messages meet the filtering condition is detected one by one, if the batch of messages meet the filtering condition, the batch of messages are copied, and if the batch of messages do not meet the filtering condition, the batch of messages are directly removed, so that the whole storage capacity of the copy platform is further effectively and orderly utilized, and the processing capacity of the copy platform is improved.
According to an embodiment of the present invention, the message copying method, wherein the configuration information includes a message source identifier, and step S204 includes: step S204D, pulling the message from the message source corresponding to the message source identification in the message pulling time interval; step S204E, selecting a target message to be copied to the copy database according to the acquired information about the remaining capacity of the storage message from the message pulled by the message source corresponding to the message source identifier.
Therefore, the message source identification is set in the configuration information to clearly indicate which message source each pulled message comes from, the messages can be regrouped according to different message sources, and the message source from which the copied message originally comes can be still clear after copying.
According to an embodiment of the present invention, the message copying method, wherein step S204D includes: step S204Da, determining the pull mode corresponding to the message source identification; step S204Db, pulling the message from the message source corresponding to the message source identifier according to the determined pulling manner.
Thus, the embodiment of the invention can adopt different pulling modes for the messages from different message sources, thereby avoiding the low processing efficiency and the low capacity caused by adopting the same pulling mode for all the messages.
According to an embodiment of the present invention, the message pull time interval is set by at least one of the following methods: set to a fixed time interval; setting the message pulling time interval according to a preset time function; setting the message pull time interval based on the number of messages from the message source pulled in the last pull time interval; and setting the message pulling time interval based on the number of messages indicated by the remaining storage message capacity information in the last pulling time interval.
The embodiment of the invention can flexibly set the message pulling time interval which is fixed or changed in time according to the actual requirement of the platform or the dynamic processing condition of the platform, so that the processing of the message copying platform can be timely adjusted according to the current task quantity and the saturated state of the processing capacity, thereby comprehensively improving the processing efficiency and the capacity of the copying platform in the whole copying process.
In addition, as shown in fig. 3, an alternative message replication system structure is provided according to an embodiment of the present invention.
Fig. 3 shows a structure of a message duplication system according to an embodiment of the present invention. In this embodiment, the message copy system includes: a reading device 310, a buffering device 320, a timer device 330, and a copy writing device 340. The service server includes an interface 350 for the consumer to enter a message replication request. The reading device 310 is mainly used to receive a message copy request from the interface 350 and to initiate reading and timing reading. The buffer device 320 is mainly used to store the configuration information and the data amount as a buffer in the message copying process. The timer device 330 is mainly used for setting a message pull period to periodically pull a message for message replication. The copy writing device 340 is mainly used for judging and screening the data needing to be copied to the database in each message pulling period, and copying the data needing to be copied to the database as copy target data after screening, thereby completing message copying of the platform. The interface 350 is primarily used to receive service requests, i.e., message duplication requests. The message replication request is from the client or other server to which the service server is connected.
According to another aspect of the embodiments of the present invention, as shown in fig. 4, there is provided a message copying apparatus for copying a message from a message source to a copy database at a message pull time interval, including:
a configuration information obtaining unit 401, configured to obtain configuration information of the message pull time interval, where the configuration information at least includes information of remaining storage message capacity;
a target message selecting unit 403, configured to select a target message to be copied to the copy database according to the acquired remaining storage message capacity information from the messages pulled within the message pulling time interval;
a target message sending unit 405 that sends the selected target message to the replication database.
It should be noted that the configuration information obtaining unit 401 in this embodiment may be configured to execute step S202 in this embodiment of the present invention, the target message selecting unit 403 in this embodiment may be configured to execute step S204 in this embodiment of the present invention, and the target message sending unit 405 in this embodiment may be configured to execute step S206 in this embodiment of the present invention.
It should be noted here that the modules described above are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the above embodiments. It should be noted that the modules described above as a part of the apparatus may operate in a hardware environment as shown in fig. 1, and may be implemented by software or hardware.
According to another aspect of the embodiments of the present invention, the present invention provides an electronic device, as shown in fig. 5, including: the computer program comprises a memory 501, a processor 503, a communication interface 505 and a communication bus 507, wherein a computer program capable of running on the processor 503 is stored in the memory 501, the memory 501 and the processor 503 communicate with each other through the communication interface 505 and the communication bus 507, and the steps of the method are realized when the processor 503 executes the computer program.
The memory and the processor in the electronic equipment are communicated with the communication interface through a communication bus. The communication bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Alternatively, the memory may be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
There is also provided, in accordance with yet another aspect of an embodiment of the present invention, a computer-readable medium having non-volatile program code executable by a processor.
Optionally, in an embodiment of the present invention, a computer readable medium is configured to store program code for the processor to perform the following steps:
step S202, obtaining configuration information of the message pulling time interval, wherein the configuration information at least comprises residual storage message capacity information;
step S204, selecting target messages to be copied to a copy database from the messages pulled in the message pulling time interval according to the acquired information of the capacity of the residual storage messages;
step S206, sending the selected target message to the copy database.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment is not described herein again.
When the embodiment of the invention is specifically realized, reference can be made to the various embodiments, and corresponding technical effects are achieved.
It is to be understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For a hardware implementation, the Processing units may be implemented within one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof.
For a software implementation, the techniques described herein may be implemented by means of units performing the functions described herein. The software codes may be stored in a memory and executed by a processor. The memory may be implemented within the processor or external to the processor.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and in actual implementation, there may be other divisions, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially implemented or make a contribution to the prior art, or may be implemented in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk. It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (13)
1. A message replication method for replicating messages from a message source to a replication database at message pull intervals, comprising:
acquiring configuration information of the message pulling time interval, wherein the configuration information at least comprises residual storage message capacity information;
selecting target messages to be copied to a copy database from the messages pulled in the message pulling time interval according to the acquired information of the capacity of the residual storage messages;
sending the selected target message to the replication database.
2. The message replication method of claim 1, wherein said sending the selected target message to the replication database comprises:
and updating the configuration information in the replication database to obtain updated configuration information, and using the updated configuration information as the configuration information of the next message pull time interval.
3. The message copying method according to claim 1, wherein the obtaining the configuration information of the message pull time interval comprises:
sending a configuration information request to the replication database;
receiving the configuration information returned from the replicated database.
4. The message copying method of claim 1, wherein the configuration information further includes a filtering condition,
selecting the target message to be copied to the copy database from the messages pulled in the message pulling time interval according to the acquired information of the capacity of the residual storage messages, wherein the selecting step comprises the following steps:
and selecting the messages which meet the filtering condition and the quantity of which does not exceed the residual storage message capacity information from the messages pulled in the message pulling time interval as target messages to be copied into a copy database.
5. The message duplication method of claim 4 wherein the filter condition is a specified keyword.
6. The message copying method according to claim 4, wherein the selecting, from the messages pulled in the message pull time interval, messages that meet the filtering condition and whose number does not exceed the remaining storage message capacity information as target messages to be copied into a copy database comprises:
selecting messages meeting the filtering condition from the messages pulled in the message pulling time interval;
and if the number of the messages meeting the filtering condition is larger than the number indicated by the residual storage message capacity information, reserving the messages with the number indicated by the residual storage message capacity information from the messages meeting the filtering condition as target messages to be copied into a copy database.
7. The message copying method according to claim 4, wherein the selecting, from the messages pulled in the message pull time interval, messages that meet the filtering condition and whose number does not exceed the number indicated by the remaining storage message capacity information as target messages to be copied into a copy database comprises:
setting an initial value of a counter equal to the number of indications of the remaining storage message capacity information;
determining one of the messages pulled in the message pulling time interval, if the message meets the filtering condition, regarding the message as a target message to be copied into a copy database, subtracting 1 from the value of the counter, and then performing judgment on whether the next message in the messages pulled in the message pulling time interval meets the filtering condition.
8. The message copying method of claim 1, wherein the configuration information comprises a message source identification,
selecting a target message to be copied to a copy database from the messages pulled from the message source within the message pulling time interval according to the acquired information of the capacity of the residual storage messages, wherein the selecting step comprises the following steps:
pulling a message from a message source corresponding to the message source identification within the message pull time interval;
and selecting the target message to be copied to the copy database from the message pulled by the message source corresponding to the message source identifier according to the acquired residual storage message capacity information.
9. The message copying method of claim 8, wherein the pulling a message from a message source corresponding to the message source identification within the message pull time interval comprises:
determining a pulling mode corresponding to the message source identifier;
and pulling the message from the message source corresponding to the message source identifier according to the determined pulling mode.
10. The message copying method according to claim 1, wherein the message pull interval is set by at least one of:
set to a fixed time interval;
setting the message pulling time interval according to a preset time function;
setting the message pull time interval based on the number of messages from the message source pulled in the last pull time interval;
and setting the message pulling time interval based on the number of messages indicated by the remaining storage message capacity information in the last pulling time interval.
11. A message copying apparatus for copying a message from a message source to a copy database at a message pull interval, comprising:
a configuration information obtaining unit, configured to obtain configuration information of the message pull time interval, where the configuration information at least includes information of remaining storage message capacity;
a target message selection unit, which selects the target message to be copied to the copy database according to the acquired information of the capacity of the residual storage message from the messages pulled in the message pulling time interval;
and the target message sending unit is used for sending the selected target message to the replication database.
12. An electronic device comprising a memory, a processor, a communication interface and a communication bus, a computer program being executable on the processor being stored in the memory, the memory and the processor communicating via the communication bus and the communication interface, wherein the steps of the method of any of claims 1 to 10 are implemented when the computer program is executed by the processor.
13. A computer readable medium storing non-volatile program code executable by a processor, wherein the program code causes the processor to perform the method of any of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110425821.XA CN113176951A (en) | 2021-04-20 | 2021-04-20 | Message copying method, device, equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110425821.XA CN113176951A (en) | 2021-04-20 | 2021-04-20 | Message copying method, device, equipment and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113176951A true CN113176951A (en) | 2021-07-27 |
Family
ID=76924391
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110425821.XA Pending CN113176951A (en) | 2021-04-20 | 2021-04-20 | Message copying method, device, equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113176951A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050251547A1 (en) * | 2004-05-05 | 2005-11-10 | International Business Machines (Ibm) Corporation | Self-optimizing workload distribution among virtual storage controllers |
| US7325042B1 (en) * | 2002-06-24 | 2008-01-29 | Microsoft Corporation | Systems and methods to manage information pulls |
| CN102004743A (en) * | 2009-09-02 | 2011-04-06 | 中国银联股份有限公司 | System and method for copying data among heterogeneous databases |
| US20120126995A1 (en) * | 2010-11-23 | 2012-05-24 | Corinex Communications Corp. | System and method for communicating over power lines |
| US20150188862A1 (en) * | 2013-12-27 | 2015-07-02 | Entefy Inc. | Apparatus and Method for Multi-Format Communication Composition |
| CN107454144A (en) * | 2017-07-03 | 2017-12-08 | 深圳市金立通信设备有限公司 | Information processing method and server |
| CN108881354A (en) * | 2017-05-09 | 2018-11-23 | 腾讯科技（深圳）有限公司 | A kind of pushed information storage method, device, server and computer storage medium |
| CN109067643A (en) * | 2018-09-26 | 2018-12-21 | 中国平安财产保险股份有限公司 | Answering method, device, computer equipment and storage medium based on keyword |
-
2021
- 2021-04-20 CN CN202110425821.XA patent/CN113176951A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7325042B1 (en) * | 2002-06-24 | 2008-01-29 | Microsoft Corporation | Systems and methods to manage information pulls |
| US20050251547A1 (en) * | 2004-05-05 | 2005-11-10 | International Business Machines (Ibm) Corporation | Self-optimizing workload distribution among virtual storage controllers |
| CN102004743A (en) * | 2009-09-02 | 2011-04-06 | 中国银联股份有限公司 | System and method for copying data among heterogeneous databases |
| US20120126995A1 (en) * | 2010-11-23 | 2012-05-24 | Corinex Communications Corp. | System and method for communicating over power lines |
| US20150188862A1 (en) * | 2013-12-27 | 2015-07-02 | Entefy Inc. | Apparatus and Method for Multi-Format Communication Composition |
| CN108881354A (en) * | 2017-05-09 | 2018-11-23 | 腾讯科技（深圳）有限公司 | A kind of pushed information storage method, device, server and computer storage medium |
| CN107454144A (en) * | 2017-07-03 | 2017-12-08 | 深圳市金立通信设备有限公司 | Information processing method and server |
| CN109067643A (en) * | 2018-09-26 | 2018-12-21 | 中国平安财产保险股份有限公司 | Answering method, device, computer equipment and storage medium based on keyword |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10484464B2 (en) | Connection control device, connection control system, and non-transitory computer readable medium | |
| WO2017166679A1 (en) | Method and apparatus for managing storage space of applications, and electronic device | |
| CN106547615B (en) | Management method and management module for background application | |
| KR101871383B1 (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| CN108737132B (en) | Alarm information processing method and device | |
| CN112511580B (en) | Message pushing method, device, storage medium and equipment | |
| CN113608842B (en) | Container cluster and component management method, device, system and storage medium | |
| CN112838980B (en) | Message processing method, system, device, electronic equipment and storage medium | |
| CN107329832B (en) | Data receiving method and device | |
| CN113176951A (en) | Message copying method, device, equipment and computer readable medium | |
| CN112631808B (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN113296977B (en) | Message processing method and device | |
| EP4033722A1 (en) | Event notification method, system, server device, and computer storage medium | |
| CN109450874B (en) | User account monitoring method and device | |
| CN109144788B (en) | Method, device and system for reconstructing OSD | |
| CN109062642B (en) | Control message notification method and device | |
| CN112052101A (en) | Broadcast processing method and device and computer system | |
| CN112581150A (en) | Information management method, device, equipment and storage medium | |
| CN112579189A (en) | Configuration file updating method and device | |
| CN109189738A (en) | Choosing method, the apparatus and system of main OSD in a kind of distributed file system | |
| CN108229917B (en) | Project data analysis method and system | |
| CN113094228B (en) | Task monitoring method and device, equipment and storage medium | |
| US20080183768A1 (en) | System, method and program for monitoring local mail replicas and local directory replicas for e-mail | |
| CN113572824B (en) | Dynamic updating method and device for timeout duration, electronic equipment and storage medium | |
| CN113495888B (en) | Identification management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant before: Jingdong Digital Technology Holding Co., Ltd |
|
| CB02 | Change of applicant information |