CN114157611B - Message de-duplication method, device and storage medium - Google Patents
Message de-duplication method, device and storage medium Download PDFInfo
- Publication number
- CN114157611B CN114157611B CN202111536438.8A CN202111536438A CN114157611B CN 114157611 B CN114157611 B CN 114157611B CN 202111536438 A CN202111536438 A CN 202111536438A CN 114157611 B CN114157611 B CN 114157611B
- Authority
- CN
- China
- Prior art keywords
- message
- duplication
- deduplication
- engine
- duplicate removal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 238000012545 processing Methods 0.000 claims abstract description 40
- 238000004590 computer program Methods 0.000 claims description 16
- 230000004044 response Effects 0.000 claims description 5
- 238000013461 design Methods 0.000 abstract description 4
- 230000000007 visual effect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/31—Flow control; Congestion control by tagging of packets, e.g. using discard eligibility [DE] bits
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a message deduplication method, a device and a storage medium, wherein the method comprises the steps of receiving a message and judging whether the message needs to be subjected to deduplication processing or not; when needed, adding a duplicate removal message header into the message; inquiring all the duplicate removal engines according to the duplicate removal message header carried by the message and judging whether the duplicate removal engines which process the message exist or not; if the duplicate removal engine which processes the message exists, the message is sent to the duplicate removal engine which processes the message; if the de-duplication engine which processes the message does not exist, the message is sent to the corresponding de-duplication engine according to the de-duplication index corresponding to the message. The invention can flexibly realize flow duplicate removal processing, has better expandability and can reduce hardware design cost.
Description
Technical Field
The present invention relates to the field of network technologies, and in particular, to a message duplication removing method, and an apparatus and a storage medium for implementing the message duplication removing method.
Background
In networks, data traffic varies widely. The visualization of network traffic is a trend in the development of network maintenance guarantees, as required by network performance analysis. The underlying basis of the visual analysis of the network traffic is to collect and capture the data messages transmitted in real time in the network, and perform statistical analysis according to the collected data messages, and in the process of data collection, message deduplication is usually required, where the message deduplication refers to the removal of duplicate messages.
Network data packet capturing (commonly called packet capturing) generally adopts two modes, one is a mode of adopting port mirror image of network equipment, and the other is a mode of bypassing a link through a mode of light splitting, high-resistance, an Ethernet (electric port) splitter and the like. Data message a sent from client (client) to Server (Server) is transmitted through network. For a network visualization system, a plurality of data acquisition points are generally distributed end to end in the whole network, and messages A1, A2, A3, A4 and A5 can be captured at each acquisition point, wherein the messages are related to an original message a and are called homologous messages. Message de-duplication is to remove redundant parts in the homologous message, and reserve a part of data message to output to the network visual analysis system at the back end. The front end collects the captured data packet and sends the captured data packet to the network visual analysis tool at the back end for further decoding, and repeated data packet collection can bring influence to the back end analysis tool, for example, the processing pressure can be increased, the complexity of service processing is increased, thus the duplicate removal processing is needed, and the data packet after the duplicate removal is forwarded to the analysis tool.
However, in the prior art, the deduplication method is various, such as a Netflow-based message deduplication method, and another such as an FPGA-based message deduplication method. For the message deduplication method based on Netflow, the method is limited by the quantity of Netflow flow table resources and the hardware integrated circuit, the quantity of the flow capable of deduplication is not too large (the Netflow flow table resources are limited), the expandability is poor (the Netflow table resources are limited to the hardware integrated circuit), the development period is long, and the cost is high (the corresponding ASIC chip is required to be specially designed). For the message deduplication method based on the FPGA, the maximum deduplication capability is limited by the flow bandwidth of the FPGA, and the expandability is not flexible enough although the expandability is increased compared with the Netflow scheme.
Disclosure of Invention
The invention aims to provide a message deduplication method which can flexibly realize flow deduplication processing, has better expandability and can reduce hardware design cost. Meanwhile, a device and a storage medium for realizing the message deduplication method are also provided.
In order to achieve the above objective, the present invention provides a message duplication removal method, where the message duplication removal method includes:
receiving a message and judging whether the message needs to be subjected to duplicate removal processing or not;
in response to the message needing to be subjected to de-duplication processing, adding a de-duplication message header into the message, wherein the de-duplication message header comprises a stream number field for identifying the flow to which the message belongs and an index field for identifying a de-important index;
inquiring all the duplicate removal engines according to the duplicate removal message header carried by the message and judging whether the duplicate removal engines which process the message exist or not;
responding to the judging result, if a de-duplication engine which processes the message exists, sending the message to the de-duplication engine which processes the message; if the de-duplication engine which processes the message does not exist, the message is sent to the corresponding de-duplication engine according to the de-duplication index corresponding to the message.
Preferably, the determining whether the message needs to be subjected to deduplication processing includes:
and matching the information carried by the message through the ACL, and determining that the message needs to be subjected to duplication removal processing when the message is matched.
Preferably, the deduplication index field is selected from one or more of deduplication priority, deduplication rate and time delay, wherein the deduplication priority is used for identifying the deduplication level of the traffic; the de-duplication rate is used to identify the minimum de-duplication rate required for the traffic; the delay is used to identify the highest delay used in deduplication.
Preferably, each deduplication engine corresponds to an information table, and the information table is used for recording current state information and inherent attribute parameter information of the deduplication engine, wherein the current state information comprises a current message throughput rate and a flow number corresponding to the processed flow; the inherent attribute parameter information comprises a maximum message throughput rate, a deduplication time delay and a deduplication rate.
Preferably, querying all the deduplication engines according to the deduplication header carried by the message and judging whether the deduplication engines processing the message exist or not includes:
inquiring whether the stream numbers corresponding to the messages are recorded in the information tables corresponding to all the duplicate removal engines;
and in response to the record of the stream number corresponding to the message in the information table, determining that the message is processed by the deduplication engine corresponding to the information table.
Preferably, the method further comprises:
and when the de-duplication engine is overloaded, the message stream is sent to other de-duplication engines which are idle and meet the de-duplication index, wherein the overload is that the current message throughput rate of the de-duplication engine is close to or greater than the maximum message throughput rate of the de-duplication engine, and the idle is that the current message throughput rate of the de-duplication engine is lower than the maximum message throughput rate of the de-duplication engine.
Preferably, all the deduplication engines process the messages in parallel.
The invention also discloses a message de-duplication device, which comprises:
the de-duplication engine modules are used for de-duplication processing the message;
the message matching module is used for receiving the message and judging whether the message needs to be subjected to duplicate removal processing or not;
the message editing module is used for responding to the message to be subjected to de-duplication treatment, and adding a de-duplication message header into the message, wherein the de-duplication message header comprises a stream number field for identifying the flow to which the message belongs and an index field for identifying a de-duplication index;
the searching module is used for inquiring all the duplicate removal engines according to the duplicate removal message header carried by the message and judging whether the duplicate removal engines which process the message exist or not;
and the shunt module is used for responding to the judging result, sending the message to the de-duplication engine which processes the message when the de-duplication engine which processes the message exists, and sending the message to the corresponding de-duplication engine according to the de-duplication index corresponding to the message when the de-duplication engine which processes the message does not exist.
The invention also discloses a message de-duplication device, which comprises:
at least one processor;
and at least one memory coupled to the at least one processor and storing a computer program for execution by the at least one processor, the computer program, when executed by the at least one processor, causing the apparatus to perform the method described above.
The invention also discloses a computer readable storage medium having stored thereon a computer program which when executed by a machine implements the method described above.
The beneficial effects of the invention are as follows:
the invention can flexibly realize flow duplicate removal processing, has better expandability and can reduce hardware design cost.
Drawings
FIG. 1 is a flow chart of a message de-duplication method according to an embodiment of the present invention;
fig. 2 is a diagram of a de-duplicated header format.
Fig. 3 is a block diagram of a message de-duplication apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
The flow deduplication scheme adopted in the prior art has the defects of poor expandability and inflexibility, so that the flow deduplication processing can be realized more flexibly, and the expandability is better.
As shown in fig. 1, the method for message deduplication disclosed by the invention comprises the following steps:
s100, receiving a message and judging whether the message needs to be subjected to duplicate removal processing or not;
specifically, firstly judging whether the received message needs to be subjected to the de-duplication processing, further performing the de-duplication processing on the message needing to be subjected to the de-duplication processing, and performing the forwarding processing on the message needing not to be subjected to the de-duplication processing. In practice, the information carried in the message may be matched through ACL (Access Control Lists, access control list) to determine whether the message needs to be subjected to deduplication processing. The ACL herein is an access control technology based on packet filtering, which can perform matching filtering on a data packet according to a set packet matching policy, where the packet matching policy is generally formed by information carried in a packet, such as a source address, a destination address, a port number, and so on. When the messages are matched through the ACL, the duplicate removal processing is needed for the matched messages, and the duplicate removal processing is not needed for the unmatched messages.
S200, in response to the message needing to be subjected to de-duplication processing, adding a de-duplication message header into the message, wherein the de-duplication message header comprises a stream number for marking the flow to which the message belongs and an index field for marking a de-important index;
specifically, after the message to be subjected to the de-duplication processing is matched through the ACL, the message is further edited so as to add a de-duplication message header in the message, wherein the de-duplication message header is used for distinguishing different flows and marking a de-duplication index of the flows. In implementation, the de-duplication message header is preferably arranged before the original message, and of course, the position of the de-duplication message header can be set according to actual requirements.
Further, as shown in fig. 2, the format of the deduplication header includes a deduplication priority field, a deduplication rate field, a delay field, and a stream number field. The priority field occupies 1 byte and is used for identifying the priority of the flow, and the priority treatment with high priority of the flow is carried out; the deduplication rate field occupies 1 byte, and is used for identifying the minimum deduplication rate required by the traffic; the delay field occupies 2 bytes and is used for identifying the highest delay which can be used when the flow is de-duplicated; the flow number field occupies 2 bytes and is used to identify the flow to which the message belongs, i.e. which flow the message belongs. In implementation, the messages in the same flow are added with the same duplicate removal message header.
S300, inquiring all the deduplication engines according to the deduplication message header carried by the message and judging whether the deduplication engines which process the message exist or not;
specifically, the invention adopts a plurality of different deduplication engines to perform message deduplication processing, and the plurality of different deduplication engines can process in parallel. The multiple different de-duplication engines can meet different de-duplication indexes, namely, the different de-duplication engines have different performance parameters according to the characteristics of the de-duplication engines, wherein the performance parameters comprise time delay, de-duplication rate, reliability, throughput and the like, and the de-duplication engines can be suitable for different flows, so that flexible de-duplication processing of the flows can be realized. Meanwhile, the deduplication engines can be flexibly added or deleted according to actual requirements, and the expandability is higher. The deduplication engine may be an FPGA-based deduplication engine, a NetFlow-based deduplication engine, or a CPU-based deduplication engine. That is, the deduplication engine used in the present invention may be at least two of the above deduplication engines, and of course, the present invention is not limited to the above deduplication engines, and other engines capable of implementing deduplication processing may be used.
After the message is added with the duplicate removal message header, the splitting process is further performed, namely, the fact that the message needs to be subjected to the duplicate removal process in which duplicate removal engine is determined. When in implementation, all the deduplication engines are queried according to the deduplication message header carried by the message, so as to determine whether the deduplication engine which processes the message exists. When searching the corresponding de-duplication engine, firstly, analyzing the de-duplication message header carried by the message to obtain the stream number in the de-duplication message header. After the stream numbers are acquired, the information tables corresponding to all the deduplication engines are further queried to determine which information table has the stream numbers. When the stream number exists in a certain information table, the message is processed by the de-duplication engine corresponding to the confidence table. When the stream number does not exist in all the information tables, the message is processed by none of the deduplication engines.
In this embodiment, each deduplication engine corresponds to an information table, where the information table is used to record some information of the deduplication engine, and the information table may record current state information and inherent attribute parameter information of the deduplication engine, where the current state information includes a current message throughput rate and a flow number corresponding to the processed flow; the inherent attribute parameter information includes maximum message throughput rate, de-duplication delay and de-duplication rate. The message throughput rate herein refers to the number of messages that the deduplication engine is capable of handling per second. In practice, each deduplication engine will record the corresponding flow number in the corresponding information table for the processed traffic.
S400, responding to the judging result, and if a de-duplication engine which processes the message exists, sending the message to the de-duplication engine which processes the message; if the de-duplication engine which processes the message does not exist, the message is sent to the corresponding de-duplication engine according to the de-duplication index corresponding to the message.
Specifically, when it is determined that there is a deduplication engine that has processed the message, that is, the flow recorded in the information table corresponding to the deduplication engine and the message belong to the same flow, the message is sent to the deduplication engine that has processed the message. When it is determined that there is no deduplication engine that has processed the message, a deduplication engine that satisfies a corresponding deduplication criterion is selected from a plurality of deduplication engines for the message, e.g., for a message with low latency requirements, a NetFlow-based deduplication engine may be selected.
Further, when each deduplication engine receives more messages in a short time, it is preferable to process the message with high deduplication priority. Meanwhile, when the current message throughput rate of the deduplication engine is close to or exceeds the maximum message throughput rate, the deduplication engine is in an overload state, the traffic processed by the deduplication engine is further distributed to other deduplication engines which are in an idle state and meet the criterion of the deduplication importance, the overload is that the current message throughput rate of the deduplication engine is close to or greater than the maximum message throughput rate of the deduplication engine, and the idle state is that the current message throughput rate of the deduplication engine is lower than the maximum message throughput rate of the deduplication engine.
As shown in FIG. 3, the invention also discloses a message deduplication device, which comprises a message matching module, a message editing module, a message parsing module, a shunting module and a plurality of deduplication engine modules. The message matching module is used for receiving the message and judging whether the message needs to be subjected to duplicate removal processing or not; the message editing module is used for responding to the message to be subjected to de-duplication treatment, and adding a de-duplication message header into the message, wherein the de-duplication message header comprises a stream number field for identifying the stream to which the message belongs and an index field for identifying a de-important index; the searching module is used for searching all the deduplication engines according to the information carried by the deduplication message header in the message and judging whether the deduplication engines which process the message exist or not; the distribution module is used for responding to the judging result, sending the message to the de-duplication engine which processes the message when the de-duplication engine which processes the message exists, and sending the message to the corresponding de-duplication engine according to the de-duplication index corresponding to the message when the de-duplication engine which processes the message does not exist.
Specifically, the message matching module matches the information carried by the message through the ACL to determine whether the message needs to be subjected to duplication removal processing, which is described in detail above and will not be repeated here. And after the message matching module matches out the message needing to be subjected to the duplicate removal processing, sending the message to the message editing module. The message editing module edits the message, adds a duplicate removal message header to the message, and the duplicate removal message header is specifically described above and will not be described in detail herein.
After the message adds the duplicate removal message header, the message is sent to a searching module, the searching module analyzes the duplicate removal message header carried by the message to obtain a stream number, and further searches an information table corresponding to each duplicate removal engine module to determine whether the duplicate removal engine module which has processed the message exists. When the corresponding deduplication engine module is found, the message is sent to the deduplication engine module for deduplication processing. When the corresponding de-duplication engine module is not found, the message is sent to the corresponding de-duplication engine module according to the de-duplication index carried in the de-duplication message header.
By adopting a plurality of different de-duplication engine modules, different de-duplication indexes can be satisfied, and the de-duplication engine module is applicable to different flows, so that flexible de-duplication treatment of the flows can be realized. Meanwhile, the multiple deduplication engine modules can be flexibly added or deleted according to different scenes and different performance requirements, and the expandability is good. When a new de-duplication engine module is added, a connection is established between the new de-duplication engine module and the shunting module, and performance parameters (such as time delay, de-duplication rate, etc.) of the de-duplication engine module need to be associated to the shunting module, so that the shunting module can select a corresponding de-duplication engine module according to the de-duplication index carried by the message.
The invention also discloses a message deduplication device, which comprises a memory, a processor and a calculator program, wherein the calculator program is stored in the processor and can run in the processor, such as a message deduplication program and the like. When the processor executes the computer program, the steps of the method can be realized, and the steps of receiving the message and judging whether the message needs to be subjected to the duplicate removal processing or not are realized.
The computer program may be divided into one or more units, which are stored in and executed by the memory, to accomplish the present invention, for example. Wherein one or more elements may be a series of computer program instruction segments capable of performing a specified function, the computer program instruction segments being adapted to describe the execution of a computer program in said message deduplication apparatus.
The invention also discloses a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and when the computer program is executed by a processor, the method can be realized. Wherein the computer program comprises computer program code, which may be in the form of source code, executable files, or in some intermediate form, etc., the computer readable medium may comprise any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), etc.
The invention can flexibly realize flow duplicate removal processing, has better expandability and can reduce hardware design cost.
While the foregoing has been disclosed in the specification and drawings, it will be apparent to those skilled in the art that various substitutions and modifications may be made without departing from the spirit of the invention, and it is intended that the scope of the invention be limited not by the specific embodiments disclosed, but by the appended claims.
Claims (9)
1. The message de-duplication method is characterized by comprising the following steps:
receiving a message and judging whether the message needs to be subjected to duplicate removal processing or not;
in response to the message needing to be subjected to the de-duplication treatment, adding a de-duplication message header in the message, wherein the de-duplication message header comprises a stream number field for identifying the flow to which the message belongs and an index field for identifying a de-duplication index, and the de-duplication index field is selected from one or more of de-duplication priority, de-duplication rate and time delay, wherein the de-duplication priority is used for identifying the de-duplication level of the flow, the de-duplication rate is used for identifying the minimum de-duplication rate required by the flow, and the time delay is used for identifying the highest time delay used during de-duplication;
inquiring all the duplicate removal engines according to the duplicate removal message header carried by the message and judging whether the duplicate removal engines which process the message exist or not;
responding to the judging result, if a de-duplication engine which processes the message exists, sending the message to the de-duplication engine which processes the message; if the de-duplication engine which processes the message does not exist, the message is sent to the corresponding de-duplication engine according to the de-duplication index corresponding to the message.
2. The method for message deduplication according to claim 1, wherein the determining whether the message needs to be deduplicated comprises:
and matching the information carried by the message through the ACL, and determining that the message needs to be subjected to duplication removal processing when the message is matched.
3. The method for message deduplication according to claim 1, wherein each deduplication engine corresponds to an information table, the information table is used for recording current state information and inherent attribute parameter information of the deduplication engine, and the current state information comprises a current message throughput rate and a flow number corresponding to the processed flow; the inherent attribute parameter information comprises a maximum message throughput rate, a deduplication time delay and a deduplication rate.
4. The method for message deduplication according to claim 3, wherein querying all deduplication engines according to the deduplication header carried by the message and judging whether the deduplication engines processing the message exist comprises:
inquiring whether the stream numbers corresponding to the messages are recorded in the information tables corresponding to all the duplicate removal engines;
and in response to the record of the stream number corresponding to the message in the information table, determining that the message is processed by the deduplication engine corresponding to the information table.
5. The method for message deduplication according to claim 1, further comprising:
and when the de-duplication engine is overloaded, the message stream is sent to other de-duplication engines which are idle and meet the de-duplication index, wherein the overload is that the current message throughput rate of the de-duplication engine is close to or greater than the maximum message throughput rate of the de-duplication engine, and the idle is that the current message throughput rate of the de-duplication engine is lower than the maximum message throughput rate of the de-duplication engine.
6. The method for message deduplication according to claim 1, wherein all deduplication engines process the message requiring deduplication in parallel.
7. A message de-duplication apparatus for implementing the message de-duplication method of any one of claims 1 to 6, comprising:
the de-duplication engine modules are used for de-duplication processing the message;
the message matching module is used for receiving the message and judging whether the message needs to be subjected to duplicate removal processing or not;
the message editing module is used for responding to the message to be subjected to the de-duplication treatment, adding a de-duplication message header into the message, wherein the de-duplication message header comprises a stream number field for identifying the flow to which the message belongs and an index field for identifying a de-duplication index, the de-duplication index field is selected from one or more of de-duplication priority, de-duplication rate and time delay, the de-duplication priority is used for identifying the de-duplication level of the flow, the de-duplication rate is used for identifying the minimum de-duplication rate required by the flow, and the time delay is used for identifying the highest time delay used during de-duplication;
the searching module is used for inquiring all the duplicate removal engines according to the duplicate removal message header carried by the message and judging whether the duplicate removal engines which process the message exist or not;
and the shunt module is used for responding to the judging result, sending the message to the de-duplication engine which processes the message when the de-duplication engine which processes the message exists, and sending the message to the corresponding de-duplication engine according to the de-duplication index corresponding to the message when the de-duplication engine which processes the message does not exist.
8. The message deduplication device is characterized by comprising:
at least one processor;
and at least one memory coupled to the at least one processor and storing a computer program for execution by the at least one processor, the computer program, when executed by the at least one processor, causing the apparatus to perform the method of any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon a computer program which, when executed by a machine, implements the method according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111536438.8A CN114157611B (en) | 2021-12-15 | 2021-12-15 | Message de-duplication method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111536438.8A CN114157611B (en) | 2021-12-15 | 2021-12-15 | Message de-duplication method, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114157611A CN114157611A (en) | 2022-03-08 |
| CN114157611B true CN114157611B (en) | 2023-12-08 |
Family
ID=80451167
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111536438.8A Active CN114157611B (en) | 2021-12-15 | 2021-12-15 | Message de-duplication method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114157611B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115086195B (en) * | 2022-06-09 | 2024-02-02 | 北京锐安科技有限公司 | Method, device, equipment and medium for determining message de-duplication time of shunt equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1996939A (en) * | 2006-12-29 | 2007-07-11 | 华为技术有限公司 | Method for message access control, forwarding engine and communication device |
| CN102664815A (en) * | 2012-05-21 | 2012-09-12 | 华为技术有限公司 | Load sharing method, device and system for message traffic |
| CN106557571A (en) * | 2016-11-23 | 2017-04-05 | 福建亿榕信息技术有限公司 | A kind of data duplicate removal method and device based on K V storage engines |
| CN109150684A (en) * | 2018-07-20 | 2019-01-04 | 新华三技术有限公司 | Message processing method, device, communication equipment and computer readable storage medium |
| CN111770023A (en) * | 2020-06-28 | 2020-10-13 | 湖南有马信息技术有限公司 | Message duplicate removal method and device based on FPGA and FPGA chip |
| CN113055127A (en) * | 2021-03-17 | 2021-06-29 | 网宿科技股份有限公司 | Data message duplicate removal and transmission method, electronic equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2473409B (en) * | 2011-01-17 | 2013-03-13 | Quantum Corp | Cloud-based de-duplication |
| WO2013070792A1 (en) * | 2011-11-07 | 2013-05-16 | Nexgen Storage, Inc. | Primary data storage system with staged deduplication |
| US10162554B2 (en) * | 2016-08-03 | 2018-12-25 | Samsung Electronics Co., Ltd. | System and method for controlling a programmable deduplication ratio for a memory system |
-
2021
- 2021-12-15 CN CN202111536438.8A patent/CN114157611B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1996939A (en) * | 2006-12-29 | 2007-07-11 | 华为技术有限公司 | Method for message access control, forwarding engine and communication device |
| CN102664815A (en) * | 2012-05-21 | 2012-09-12 | 华为技术有限公司 | Load sharing method, device and system for message traffic |
| CN106557571A (en) * | 2016-11-23 | 2017-04-05 | 福建亿榕信息技术有限公司 | A kind of data duplicate removal method and device based on K V storage engines |
| CN109150684A (en) * | 2018-07-20 | 2019-01-04 | 新华三技术有限公司 | Message processing method, device, communication equipment and computer readable storage medium |
| CN111770023A (en) * | 2020-06-28 | 2020-10-13 | 湖南有马信息技术有限公司 | Message duplicate removal method and device based on FPGA and FPGA chip |
| CN113055127A (en) * | 2021-03-17 | 2021-06-29 | 网宿科技股份有限公司 | Data message duplicate removal and transmission method, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| "海量数据的重复数据删除中元数据管理关键技术研究";周炳;《中国博士学位论文全文数据库》(第第7期期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114157611A (en) | 2022-03-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111770023B (en) | Message duplicate removal method and device based on FPGA and FPGA chip | |
| CN112491652B (en) | Network flow sample processing method and device for testing | |
| CN106972985B (en) | Method for accelerating data processing and forwarding of DPI (deep packet inspection) equipment and DPI equipment | |
| EP3211834B1 (en) | Fast packet retrieval based on flow id and metadata | |
| CN114157611B (en) | Message de-duplication method, device and storage medium | |
| US10148596B2 (en) | Data flow statistics collection method, system, and apparatus | |
| CN112486914A (en) | Data packet storage and fast check method and system | |
| CN103595577B (en) | System and method for monitoring whether traffic of interconnection interfaces between ISPs exceeds threshold valve | |
| CN116340055B (en) | Data recovery method and system for solid state disk | |
| CN112968842A (en) | Novel network flow acquisition and analysis method and system | |
| CN114244781B (en) | Message de-duplication processing method and device based on DPDK | |
| CN112822077B (en) | Method and system for measuring total network flow in data center network and packet loss detection method | |
| CN114676130A (en) | Time series data storage method, computing equipment and storage medium | |
| CN111131479B (en) | Flow processing method and device and flow divider | |
| CN111865723A (en) | Network data acquisition system based on big data | |
| CN114374622B (en) | Shunting method based on fusion shunting equipment and fusion shunting equipment | |
| CN114095383B (en) | Network flow sampling method and system and electronic equipment | |
| CN114598493B (en) | Network traffic collection method | |
| CN108366024A (en) | Message forwarding method and device | |
| CN113347036A (en) | Method and system for realizing cloud environment bypass monitoring by utilizing public cloud storage | |
| JP2017216613A (en) | Transfer device and transfer method | |
| CN112565821A (en) | Data processing method and device, security gateway and storage device | |
| US11106656B2 (en) | Method and apparatus for a software-seamed and augmented view of an asynchronous network fabric | |
| CN112560877A (en) | Service classification method and device and Internet system | |
| CN115580546B (en) | Data subscription method and device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |