CN113360487A - Message duplication eliminating method and related device - Google Patents
Message duplication eliminating method and related device Download PDFInfo
- Publication number
- CN113360487A CN113360487A CN202110620329.8A CN202110620329A CN113360487A CN 113360487 A CN113360487 A CN 113360487A CN 202110620329 A CN202110620329 A CN 202110620329A CN 113360487 A CN113360487 A CN 113360487A
- Authority
- CN
- China
- Prior art keywords
- message
- node
- messages
- distributed system
- consumption
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000004519 manufacturing process Methods 0.000 claims abstract description 48
- 238000004590 computer program Methods 0.000 claims description 11
- 230000000694 effects Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 6
- 230000008859 change Effects 0.000 description 3
- 230000003252 repetitive effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000000889 atomisation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The application discloses a message duplicate removal method, which comprises the following steps: a message production node of the distributed system writes a message to the first set; when the message consumption nodes of the distributed system read the messages of the second set, the message production nodes write the messages to the second set so that the message consumption nodes read the messages from the first set; wherein the first set and the second set are both sets of non-repeating value data structures. Through the switching operation between the first set and the second set, the messages in the sets are subjected to duplicate removal according to the data structure characteristics of the sets, and the accuracy of message duplicate removal is improved. The application also discloses a message duplication removal device, a server and a computer readable storage medium, which have the effects.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a message duplication eliminating method, a message duplication eliminating apparatus, a server, and a computer-readable storage medium.
Background
With the continuous development of information technology, more and more messages are produced in the internet, and more nodes or devices consume the messages. There is a need for efficient management of transmitted messages. Further, in the face of a large number of messages, repeated messages are easy to occur, which not only increases the burden of hardware, but also leads to redundant transmission of messages. Therefore, a message needs to be deduplicated.
In the related art, a bloom filter is generally used for deduplication processing, or external key-value storage is introduced for deduplication processing. However, as the data volume continuously becomes larger, some high concurrency scenes exist, and the deduplication operations cannot strictly perform deduplication under the high concurrency condition, so that the deduplication accuracy is reduced.
Therefore, how to improve the accuracy of message deduplication is a key issue that those skilled in the art are concerned about.
Disclosure of Invention
The application aims to provide a message deduplication method, a message deduplication device, a server and a computer readable storage medium, so as to solve the problem that the message deduplication accuracy is low.
In order to solve the above technical problem, the present application provides a message deduplication method, including:
a message production node of the distributed system writes a message to the first set;
when the message consumption nodes of the distributed system read the messages of the second set, the message production nodes write the messages to the second set so that the message consumption nodes read the messages from the first set;
wherein the first set and the second set are both sets of non-repeating value data structures.
Optionally, the writing of the message to the first set by the message production node of the distributed system includes:
the message production node of the distributed system determines the current write-in set as the first set according to the acquired set control identification;
writing a message to the first set.
Optionally, when the message consumption node of the distributed system finishes reading the message of the second set, the message production node writes the message to the second set, so that the message consumption node reads the message from the first set, including:
and when the message consumption node of the distributed system finishes reading the message of the second set, the message consumption node switches the set control identifier and reads the message from the first set so that the message production node writes the message to the second set.
Optionally, the set control flag is a self-increment value;
correspondingly, the determining, by the message production node of the distributed system, that the current write set is the first set according to the acquired set control identifier includes:
the message production node of the distributed system takes 2 for complementation to the self-increment numerical value to obtain a remainder;
when the remainder is 0, the current write set is the first set.
Optionally, when the message consumption node of the distributed system finishes reading the message of the second set, the message production node writes the message to the second set, so that the message consumption node reads the message from the first set, including:
when the message consumption node of the distributed system finishes reading the message of the second set, the message consumption node increases the self-increment value by one to obtain a new self-increment value;
the message consumption node determines that the current reading set is the first set according to the value obtained by adding one to the new self-increment value, and reads the message from the first set;
and the message production node determines that the current writing set is the second set according to the new self-increment value and writes a message to the second set.
Optionally, the method further includes:
and when the message consumption nodes of the distributed system finish reading the messages of the first set, the message production nodes write the messages to the first set so that the message consumption nodes read the messages from the second set.
Optionally, the no-repeat value data structure is a set data structure.
The present application further provides a message deduplication apparatus, including:
a message writing module for writing a message to the first set;
a switching set writing module, configured to write a message to a second set when a message consuming node of the distributed system finishes reading a message of the second set, so that the message consuming node reads the message from the first set; wherein the first set and the second set are both sets of non-repeating value data structures.
The present application further provides a server, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the message deduplication method as described above when executing the computer program.
The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the message deduplication method as described above.
The application provides a message duplicate removal method, which comprises the following steps: a message production node of the distributed system writes a message to the first set; when the message consumption nodes of the distributed system read the messages of the second set, the message production nodes write the messages to the second set so that the message consumption nodes read the messages from the first set; wherein the first set and the second set are both sets of non-repeating value data structures.
The message is written into the first set by the message production node of the distributed system, then the message is written into the second set when the message of the second set is read, so that the consumption node reads the message from the first set, one set is used for writing the message and the other set is used for reading the message in the message stream, when the message is read, the two sets are switched, and the two sets are both sets without a repeated value data structure, so that the repeated message in the set of the written message is avoided, the message is further deduplicated, and the accuracy of message deduplication is improved due to the characteristic of no repeated value of the data structure.
The present application further provides a message deduplication device, a server, and a computer-readable storage medium, which have the above effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a message deduplication method according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a message deduplication apparatus according to an embodiment of the present application.
Detailed Description
The core of the application is to provide a message duplication eliminating method, a message duplication eliminating device, a server and a computer readable storage medium, so as to solve the problem of low message duplication eliminating accuracy.
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the related art, a bloom filter is generally used for deduplication processing, or an external key-value storage is introduced for deduplication processing. However, as the data volume continuously becomes larger, some high concurrency scenes exist, and the deduplication operations cannot strictly perform deduplication under the high concurrency condition, so that the deduplication accuracy is reduced.
Therefore, the application provides a message deduplication method, a message is written to a first set through a message production node of a distributed system, then when the message of a second set is read, the message is written to the second set, so that a consumption node reads the message from the first set, one set is used for writing the message and the other set is used for reading the message in a message stream, when the message is read, the two sets are switched, and the two sets are both sets without a repeated value data structure, so that repeated messages in the sets of the written messages are avoided, and further the messages are deduplicated.
The following describes a message deduplication method provided by the present application by using an embodiment.
Referring to fig. 1, fig. 1 is a flowchart of a message deduplication method according to an embodiment of the present application.
In this embodiment, the method may include:
s101, writing a message to a first set by a message production node of a distributed system;
it can be seen that this step is directed to the message producing node of the distributed system writing a message to the first set. Wherein the first set is a set to which a current message production node needs to write a message, that is, a current write set.
In this embodiment, there are at least two sets. While one set is being written to a message, the other set is being read from the message. Therefore, the present embodiment also needs to determine the set of messages that need to be written.
Further, the step may include:
step 1, determining that a current write-in set is a first set by a message production node of a distributed system according to an acquired set control identifier;
and 2, writing a message to the first set.
It can be seen that the present embodiment mainly describes how to write a message to the first set. In this alternative, the message production node determines that the current write set is the first set according to the acquired set control identifier. The set control flag is a flag indicating which set is the current write set or the current read set. The flag may be an automatic switching flag, may be a self-increment value, or may be a change between 0 and 1 of a signal bit. It can be seen that the specific form of the set control flag is not limited in this embodiment. Finally, a message is written to the first set.
S102, when the message consumption nodes of the distributed system read the messages of the second set, the message production nodes write the messages to the second set so that the message consumption nodes read the messages from the first set;
wherein the first set and the second set are both sets of non-repetitive value data structures.
On the basis of S101, this step aims at that when the message consumption node of the distributed system finishes reading the message of the second set, the message production node writes the message to the second set so that the message consumption node reads the message from the first set. That is, the message producing node writes messages to the first set and the message consuming node reads messages from the second set before the message consuming node finishes reading messages of the second set. And when the message in the second set is completely read by the message consumption node, the message production node is switched with the set operated by the message consumption node. At this point, there are no messages in the second set, and multiple messages have been written in the first set. The message producing node writes messages to the second set and the message consuming node reads messages from the first set.
Wherein, the first set and the second set are both sets of non-repetitive value data structures. That is, the duplicate value is not present in the set stored in the data structure, and the messages in the set are indirectly deduplicated. In addition, due to the characteristic of the aggregated data structure, the phenomenon of missing duplicate removal can be avoided in the high-concurrency scene, and the accuracy of duplicate removal is improved.
Further, on the basis of the set control flag in the previous step, this step may include:
and when the message consumption node of the distributed system finishes reading the message of the second set, the message consumption node switches the set control identifier and reads the message from the first set so that the message production node writes the message to the second set.
It can be seen that, in other words, in this embodiment, when the message consumption node of the distributed system finishes reading the message of the second set, the message consumption node needs to continuously read the message at this time. Therefore, it is necessary to switch the set of message producing node and message consuming node operations so that the message consuming node continuously reads messages from the first set. And for the message production node and the message consumption node, the functions of the first set and the second set can be switched through the set control identification. Therefore, when the message consumption node of the distributed system finishes reading the message of the second set, the set control identification is directly switched, so that the message consumption node reads the message from the first set and the message production node writes the message into the second set.
Further, the aggregate control in the last alternative is identified as a self-increasing value. Therefore, the step of determining, by the message production node of the distributed system in S101 in this embodiment, that the current write set is the first set according to the acquired set control identifier may include:
step 1, a message production node of a distributed system obtains a remainder by using 2 for a self-increment numerical value;
and step 2, when the remainder is 0, the current write set is the first set.
It can be seen that in the present embodiment, the current write set is determined by the self-increment value and the remainder of 2. Where the value of the auto-increment is a positive integer, the remainder of the value with 2 is either 0 or 1, and thus either 0 or 1 when the value of the auto-increment changes. Further, it can be set that when the remainder is 0, the current write set is the first set, and when the remainder is 1, the current write set is the second set. Therefore, the control of the current write-in set is realized through the self-increment numerical value in the alternative scheme, so that the processing process can be effectively controlled, and the control effect is improved.
Further, S102 of this embodiment may include:
step 1, when the message consumption node of the distributed system finishes reading the message of the second set, the message consumption node increases the self-increment value by one to obtain a new self-increment value;
step 2, the message consumption node determines that the current reading set is a first set according to the value obtained by adding one to the new self-increment value, and reads the message from the first set;
and 3, the message production node determines the current writing set as a second set according to the new self-increment value and writes the message into the second set.
It can be seen that this alternative is primarily illustrative of how messages are written to the second set. In the alternative scheme, when the message consumption node of the distributed system finishes reading the message of the second set, the message consumption node increases the self-increment value by one to obtain a new self-increment value; then, the message consumption node determines that the current reading set is a first set according to the value obtained by adding one to the new self-increment value, and reads the message from the first set; and finally, the message production node determines that the current writing set is a second set according to the new self-increment value, and writes the message into the second set.
Further, the no-duplication value data structure in the present embodiment may be a set data structure.
In addition, the operation of switching between the first set and the second set is only described once in this embodiment, and it is conceivable that values between the first set and the second set are switched cyclically until the message transmission is completed in actual operation. Accordingly, the present embodiment may further include:
when the message consumption nodes of the distributed system finish reading the messages of the first set, the message production nodes write the messages to the first set so that the message consumption nodes read the messages from the second set.
In summary, in this embodiment, a message is written to a first set by a message production node of a distributed system, and then the message is written to a second set when the message of the second set is read, so that a consuming node reads the message from the first set, thereby implementing that one set is used for writing the message and one set is used for reading the message in a message stream, and when the message is read, the two sets are switched, and further the two sets are both sets of a non-duplicate value data structure, thereby avoiding the occurrence of duplicate messages in the set of the written message, and further performing deduplication on the message.
The message deduplication method provided by the present application is further described below by a specific embodiment.
In this embodiment, duplicate removal is switched by double set reading and writing, the model may set two set sets for a duplicate removal message stream in redis, one set is responsible for writing messages, and the other set is responsible for reading messages. In addition, a long value n is set in the redis, and the atomization increment of the n value is used for switching the reading and writing of the cluster. Among them, redis is a data structure server.
In the distributed system, all nodes producing messages determine a write set according to the n% 2 value, and write messages into the write set, and according to the characteristics of the set, the same messages do not exist in 2 parts, so that repeated messages in the current write set can be removed.
All the consuming nodes determine a reading set according to the (n + 1)% 2 value, and send out messages from the pop in the reading set, when the messages in the reading set are empty, the first pop which is found to be empty requests to directly perform self-increment processing on the n value, and locks the n value in one time domain, so that excessive switching when no message exists is prevented, stable and ordered switching of the reading and writing set under a concurrent environment can be maintained, and subsequent reading and writing requests of the n value self-increment can change along with the change of the n value.
Specifically, the method in this embodiment may include:
step 1, deploying a redis service, and creating a message write set, a message read set and a control value n which are exclusive to a message flow according to a message flow name;
step 2, the message production node determines a message write set according to the control value n% 2 and writes the message;
step 3, the message consumption node determines a message reading set according to the control value (n + 1)% 2 and reads the message;
step 4, any message consumption node finds that the read message is empty and exceeds a read-write switching threshold value, and adds 1 to the value n so as to switch a message write set and a message read set and record a switching time point; otherwise, the consumption node waits for the threshold residual time;
and 5, repeating the processes from the step 2 to the step 4.
It can be seen that, in this embodiment, a message is written to a first set by a message production node of a distributed system, and then when the message of a second set is read, the message is written to the second set, so that a consumption node reads the message from the first set, thereby implementing that one set is used for writing the message and one set is used for reading the message in a message stream, and when the message is read, the two sets are switched, and further the two sets are both sets of a non-duplicate value data structure, thereby avoiding the occurrence of duplicate messages in the set of the written message, and further performing deduplication on the message.
In the following, the message deduplication device provided in the embodiment of the present application is introduced, and the message deduplication device described below and the message deduplication method described above may be referred to correspondingly.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a message deduplication apparatus according to an embodiment of the present application.
In this embodiment, the apparatus may include:
a message writing module 100 for writing a message to the first set;
a switching set writing module 200, configured to write a message to the second set when the message consuming node of the distributed system finishes reading the message of the second set, so that the message consuming node reads the message from the first set; wherein the first set and the second set are both sets of non-repetitive value data structures.
Optionally, the message writing module 100 is specifically used for
Optionally, the switching set writing module 200 is specifically configured to
Optionally, the set control flag is a self-increment value; correspondingly, the message writing module 100 is specifically configured to use 2 for the self-increment value to obtain a remainder; when the remainder is 0, the current write set is the first set.
Optionally, the switching set writing module 200 is specifically configured to, when the message consuming node of the distributed system finishes reading the message of the second set, add one to the self-increment value by the message consuming node to obtain a new self-increment value; the message consumption node determines that the current reading set is a first set according to the value obtained by adding one to the new self-increment value, and reads the message from the first set; and the message production node determines that the current write set is a second set according to the new self-increment value and writes the message into the second set.
Optionally, the apparatus may further include:
and the circulation module is used for writing the message to the first set by the message production node when the message consumption node of the distributed system finishes reading the message of the first set so that the message consumption node can read the message from the second set.
Optionally, the no-duplicate value data structure is a set data structure.
An embodiment of the present application further provides a server, including:
a memory for storing a computer program;
a processor for implementing the steps of the message deduplication method as described in the above embodiments when executing the computer program.
An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the message deduplication method according to the foregoing embodiment.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
A message deduplication method, a message deduplication apparatus, a server, and a computer-readable storage medium provided by the present application are described in detail above. The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
Claims (10)
1. A method for message deduplication, comprising:
a message production node of the distributed system writes a message to the first set;
when the message consumption nodes of the distributed system read the messages of the second set, the message production nodes write the messages to the second set so that the message consumption nodes read the messages from the first set;
wherein the first set and the second set are both sets of non-repeating value data structures.
2. The message deduplication method of claim 1, wherein the message production node of the distributed system writes the message to the first set, comprising:
the message production node of the distributed system determines the current write-in set as the first set according to the acquired set control identification;
writing a message to the first set.
3. The message deduplication method of claim 2, wherein when a message consumption node of the distributed system finishes reading a second set of messages, the message production node writes a message to the second set so that the message consumption node reads a message from the first set, comprising:
and when the message consumption node of the distributed system finishes reading the message of the second set, the message consumption node switches the set control identifier and reads the message from the first set so that the message production node writes the message to the second set.
4. The message deduplication method of claim 2, wherein the set control flag is a self-incrementing number;
correspondingly, the determining, by the message production node of the distributed system, that the current write set is the first set according to the acquired set control identifier includes:
the message production node of the distributed system takes 2 for complementation to the self-increment numerical value to obtain a remainder;
when the remainder is 0, the current write set is the first set.
5. The message deduplication method of claim 4, wherein when a message consumption node of the distributed system finishes reading a second set of messages, the message production node writes a message to the second set so that the message consumption node reads a message from the first set, comprising:
when the message consumption node of the distributed system finishes reading the message of the second set, the message consumption node increases the self-increment value by one to obtain a new self-increment value;
the message consumption node determines that the current reading set is the first set according to the value obtained by adding one to the new self-increment value, and reads the message from the first set;
and the message production node determines that the current writing set is the second set according to the new self-increment value and writes a message to the second set.
6. The message deduplication method of any one of claims 1 to 5, further comprising:
and when the message consumption nodes of the distributed system finish reading the messages of the first set, the message production nodes write the messages to the first set so that the message consumption nodes read the messages from the second set.
7. The message deduplication method of claim 6, wherein the no-repetition value data structure is a set data structure.
8. A message deduplication apparatus, comprising:
a message writing module for writing a message to the first set;
a switching set writing module, configured to write a message to a second set when a message consuming node of the distributed system finishes reading a message of the second set, so that the message consuming node reads the message from the first set; wherein the first set and the second set are both sets of non-repeating value data structures.
9. A server, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the message deduplication method as claimed in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the message deduplication method as recited in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110620329.8A CN113360487A (en) | 2021-06-03 | 2021-06-03 | Message duplication eliminating method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110620329.8A CN113360487A (en) | 2021-06-03 | 2021-06-03 | Message duplication eliminating method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113360487A true CN113360487A (en) | 2021-09-07 |
Family
ID=77531737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110620329.8A Pending CN113360487A (en) | 2021-06-03 | 2021-06-03 | Message duplication eliminating method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113360487A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106960054A (en) * | 2017-04-01 | 2017-07-18 | 北京奇虎科技有限公司 | The access method and device of data file |
| CN108076103A (en) * | 2016-11-18 | 2018-05-25 | 北京优朋普乐科技有限公司 | A kind of message treatment method, system and message producer's cluster |
| CN108769099A (en) * | 2018-04-02 | 2018-11-06 | 武汉斗鱼网络科技有限公司 | A kind of implementation method of the message duplicate removal of message-oriented middleware |
| CN111984198A (en) * | 2020-08-25 | 2020-11-24 | 北京一起教育信息咨询有限责任公司 | Message queue implementation method and device and electronic equipment |
| CN112015563A (en) * | 2020-09-02 | 2020-12-01 | 中国平安财产保险股份有限公司 | Message queue switching method and device, electronic equipment and storage medium |
-
2021
- 2021-06-03 CN CN202110620329.8A patent/CN113360487A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108076103A (en) * | 2016-11-18 | 2018-05-25 | 北京优朋普乐科技有限公司 | A kind of message treatment method, system and message producer's cluster |
| CN106960054A (en) * | 2017-04-01 | 2017-07-18 | 北京奇虎科技有限公司 | The access method and device of data file |
| CN108769099A (en) * | 2018-04-02 | 2018-11-06 | 武汉斗鱼网络科技有限公司 | A kind of implementation method of the message duplicate removal of message-oriented middleware |
| CN111984198A (en) * | 2020-08-25 | 2020-11-24 | 北京一起教育信息咨询有限责任公司 | Message queue implementation method and device and electronic equipment |
| CN112015563A (en) * | 2020-09-02 | 2020-12-01 | 中国平安财产保险股份有限公司 | Message queue switching method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107391628B (en) | Data synchronization method and device | |
| KR101505263B1 (en) | Method for de-duplicating data and apparatus therefor | |
| CN111881133B (en) | Storage bucket management method and device, computer equipment and readable storage medium | |
| US20050246362A1 (en) | System and method for dynamci log compression in a file system | |
| DE102016013248A1 (en) | Reference block accumulation in a reference quantity for deduplication in storage management | |
| CN102867035B (en) | A kind of distributed file system cluster high availability method and device | |
| US11029891B2 (en) | Hybrid distributed storage system to dynamically modify storage overhead and improve access performance | |
| JP2008217209A (en) | Difference snapshot management method, computer system and nas computer | |
| CN110875885B (en) | Message processing method, server, terminal, system and storage medium | |
| CN107797770B (en) | Method and device for synchronizing disk state information | |
| CN109471843B (en) | Metadata caching method, system and related device | |
| CN103150149A (en) | Method and device for processing redo data of database | |
| CN110618974A (en) | Data storage method, device, equipment and storage medium | |
| CN111737212A (en) | Method and equipment for improving performance of distributed file system | |
| CN107423425B (en) | Method for quickly storing and inquiring data in K/V format | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| CN115114232A (en) | Method, device and medium for enumerating historical version objects | |
| KR20150035876A (en) | Method for de-duplicating data and apparatus therefor | |
| CN109302449B (en) | Data writing method, data reading device and server | |
| CN112463880B (en) | Block chain data storage method and related device | |
| CN110019347B (en) | Data processing method and device of block chain and terminal equipment | |
| CN113360487A (en) | Message duplication eliminating method and related device | |
| CN110658993A (en) | Snapshot rollback method, device, equipment and storage medium | |
| CN114791904A (en) | Persistent compression method and device for bloom filter | |
| CN112463048B (en) | Compact processing speed adjusting method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210907 |
|
| RJ01 | Rejection of invention patent application after publication |