CN113472609A - Data repeated transmission marking system for wireless communication - Google Patents
Data repeated transmission marking system for wireless communication Download PDFInfo
- Publication number
- CN113472609A CN113472609A CN202110745357.2A CN202110745357A CN113472609A CN 113472609 A CN113472609 A CN 113472609A CN 202110745357 A CN202110745357 A CN 202110745357A CN 113472609 A CN113472609 A CN 113472609A
- Authority
- CN
- China
- Prior art keywords
- data
- repeated
- submodule
- repetition rate
- region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 43
- 238000004891 communication Methods 0.000 title claims abstract description 28
- 238000012544 monitoring process Methods 0.000 claims abstract description 56
- 238000013467 fragmentation Methods 0.000 claims abstract description 45
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 45
- 238000004140 cleaning Methods 0.000 claims abstract description 40
- 238000013508 migration Methods 0.000 claims abstract description 18
- 230000005012 migration Effects 0.000 claims abstract description 18
- 238000012217 deletion Methods 0.000 claims description 68
- 239000012634 fragment Substances 0.000 claims description 67
- 230000037430 deletion Effects 0.000 claims description 65
- 238000000034 method Methods 0.000 claims description 16
- 230000004044 response Effects 0.000 claims description 12
- 238000004088 simulation Methods 0.000 claims description 9
- 238000012790 confirmation Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 claims description 7
- 230000003252 repetitive effect Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000004020 conductor Substances 0.000 description 1
- 230000005684 electric field Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Abstract
The invention discloses a data repeated transmission marking system for wireless communication, which comprises a target migration data content reading module, a repeated data transmission area position positioning module, a repeated data area fragmentation analysis module, a data repeated counting monitoring unit and a repeated and miscellaneous data cleaning module, wherein the target migration data content reading module is used for scanning data to be transmitted and monitoring whether repeated information exists in the data to be transmitted, the repeated data transmission area position positioning module is used for positioning and marking the repeated area of the data to be transmitted, the repeated data area fragmentation analysis module is used for monitoring the data repetition rate after segmenting the area where the repeated data is positioned, the data repeated counting monitoring unit is used for predicting the reasonability of the repeated data, the repeated and miscellaneous data cleaning module is used for cleaning and deleting the repeated data in the data to be transmitted, aims to perform monitoring analysis on repeated data and manually confirm and delete the repeated data after confirming the data.
Description
Technical Field
The invention relates to the field of communication, in particular to a data repeated transmission marking system for wireless communication.
Background
Wireless communication refers to long-distance transmission communication between a plurality of nodes without propagation via conductors or cables, and can be performed by radio, radio and the like.
Wireless communication includes a variety of stationary, mobile, and portable applications such as two-way radios, cell phones, personal digital assistants, and wireless networks. Other examples of wireless radio communication are GPS, garage door remote control, wireless mouse, etc.
Most wireless communication technologies use radio, including Wi-fi, which is only a few meters away, and also include deep space networks, which communicate with the seaman number 1, and which are more than a few million kilometers away. However, some wireless communication technologies do not use radio, but use other electromagnetic wave wireless technologies, such as light, magnetic field, electric field, etc.
Wireless data transmission is an essential part of mobile computing. There are many technologies available, the difference being availability, coverage and performance in the area, and in some cases, the user must be able to deploy multiple wiring patterns and switch between different modes. In order to simplify the time spent by the user, the connection management software, or the MVPN, can be used to process several connections into a single secured virtual network,
Wi-Fi is a wireless local area network that allows portable computing devices to connect to the Internet in a simple manner, and is close to wired networks by the IEEE 802.11a, b, g, n, etc. Wi-Fi has become the de facto standard of hotspots in homes, offices, and public spaces. Some businesses charge for Wi-Fi once a month and some businesses offer it free of charge because providing Wi-Fi can increase sales of their products.
At present, in data sending, because the sent data are collected in real time, a large amount of repeated data can be sent, a client side can receive a large amount of repeated data for backup, and memory consumption can be caused.
Disclosure of Invention
The present invention is directed to a system and method for repeatedly sending and marking data for wireless communication, so as to solve the problems in the prior art.
In order to achieve the purpose, the invention provides the following technical scheme:
a data repeated sending and marking system for wireless communication comprises a target migration data content reading module, a repeated data transmission area position positioning module, a repeated data area fragmentation analysis module, a data repeated counting monitoring unit and a repeated redundant miscellaneous data cleaning module, wherein the target migration data content reading module, the repeated data transmission area position positioning module, the repeated data area fragmentation analysis module, the data repeated counting monitoring unit and the repeated redundant miscellaneous data cleaning module are sequentially connected through an intranet;
the target migration data content reading module is used for scanning data to be sent and monitoring whether repeated information exists in the data, the repeated data transmission region position positioning module is used for positioning and marking a repeated region of the data to be sent, the repeated data region fragmentation analysis module is used for monitoring the data repetition rate after the region where the repeated data is positioned is segmented, the data repetition counting monitoring unit is used for reasonably predicting existence of the repeated data, and the repeated and redundant data cleaning module is used for cleaning and deleting the repeated data in the data to be sent.
By adopting the technical scheme: the target migration data content reading module comprises a data information pre-scanning submodule to be sent and a repeated transmission data information counting submodule, wherein the data information pre-scanning submodule to be sent is used for scanning repeated data of data to be sent and sending the scanned data to the repeated transmission data information counting submodule, and the repeated transmission data information counting submodule is used for counting the scanned data, confirming whether the scanned data contains repeated data or not and marking the scanned repeated data in advance.
By adopting the technical scheme: the repeated data transmission region position positioning module comprises a duplicate data retrieval marking submodule and a data repeated node positioning submodule, wherein the duplicate data retrieval marking submodule is used for marking a data information duplicate contained in data to be transmitted after retrieval, the data duplicate is not used as a data repeated statistical range, and the data repeated node positioning submodule is used for positioning a repeated region monitored by data, marking a repeated region in the data to be transmitted and transmitting the marked region to the repeated data region fragmentation analysis module.
By adopting the technical scheme: the repeated data region fragmentation analysis module comprises a repeated data region cutting submodule and a fragmentation repetition rate comparison analysis submodule, wherein the repeated data region cutting submodule is used for cutting a data repetition region positioned by the data repetition node positioning submodule, thinning the repetition region, comparing the thinned repetition region with a reference source data segment, and the fragmentation repetition rate comparison analysis submodule is used for analyzing the repetition rate after comparing the cut repetition region with the reference source data segment.
By adopting the technical scheme: the fragmentation repetition rate comparison analysis submodule sets a certain repetition region as P0, the source data segment referred by the repetition region P0 as Ln, the repetition region P0 is compared with the source data segment referred to by the repetition region as Ln and then is cut, and the repetition data region cutting submodule sets the repetition region P0 to be cut into Y segments, wherein different segments are P segments1、p2、p3、…、pn-1、pnThe repetition rate of the different segments compared with the reference source data Ln is K1、K2、K3、…、Kn-1、KnWherein, K is setnThe repetition rate is more than or equal to 60 percent, and the repetition rate K isnSegmentation data of > 60% are marked.
By adopting the technical scheme: setting the repetition rate K of the comparison between different segments and the reference source data LnnThe number of fragments is not less than 60% C0, C0 >)
Judging that the repetition rate between the repetition region P0 and the reference source data segment Ln is high when C0 <
Re-detecting the repetition rate of the comparison between different segments and the reference source data Ln, and extracting the repetition rate pnE [40%, 60%), the number of extracted fragment data is set to C1, and the fragment data is set to pl1、pl2、pl3、…、plnThe extracted fragment data is re-segmented, and the re-segmented data is set to p21、p22、p23、…、p2nWherein the number of the re-segmented data segments is C2, C2 ≧ 2C 1, and the repetition rate for detecting the comparison of the re-segmented data with the reference source data segment Ln is set as Km1、Km2、Km3、…、KmnExtracting KmnData for fragments of not less than 80%, KmnAnd judging that the repetition rate is high for more than or equal to 80 percent of fragment data, and marking.
By adopting the technical scheme: the data repetition counting monitoring unit comprises a high repetition rate data acquisition and statistics submodule and a data integrity prediction submodule after simulated deletion, wherein the high repetition rate data acquisition and statistics submodule is used for acquiring high repetition rate data fragments marked by a fragmentation repetition rate comparison analysis submodule, carrying out centralized statistics to form a list and sending the list to the data integrity prediction submodule after simulated deletion, the data integrity prediction submodule after simulated deletion is used for deleting data with high repetition rate in data to be sent one by one, analyzing whether the data integrity is influenced after the high repetition rate data is deleted or not, when the fragment data with high repetition rate is deleted, the data integrity is larger than a set threshold value, and sending a data deletion application with high repetition rate to the repeated redundant miscellaneous data cleaning module.
By adopting the technical scheme: the repeated and miscellaneous data cleaning module comprises a communication returning unit and a manual response submodule, wherein the communication returning unit is used for obtaining a deletion request of the high repetition rate fragment data counted by the data repetition counting monitoring unit, summarizing the deletion request and sending the deletion request to the manual response submodule, and the manual response submodule is used for manually accessing and manually judging the deletion request.
A data repeat transmission tagging method for wireless communication, characterized by:
s1: scanning data to be transmitted by using a target migration data content reading module, monitoring whether repeated information exists in the data to be transmitted, scanning repeated data of the data to be transmitted by using a data information pre-scanning submodule to be transmitted, transmitting the scanned data to a repeated transmission data information counting submodule, counting the scanned data by using the repeated transmission data information counting submodule, confirming whether the scanned data has the repeated data, and marking the scanned repeated data in advance;
s2: the repeated region of the data to be transmitted is positioned and marked by using a repeated data transmission region position positioning module, a duplicate data retrieval marking submodule retrieves and marks a data information duplicate contained in the data to be transmitted, the data duplicate is not used as a data duplicate statistical range, a data duplicate node positioning submodule positions the repeated region monitored by the data, marks the repeated region in the data to be transmitted, and transmits the marked region to a repeated data region fragmentation analysis module;
s3: a repeated data region fragmentation analysis module is used for monitoring the data repetition rate after the region where the repeated data is positioned is fragmented, a repeated data region cutting submodule is used for cutting the data repetition region where the data repetition node positioning submodule is positioned, the repeated region is refined, the refined repeated region is compared with the reference source data segment, and a fragmentation repetition rate comparison analysis submodule is used for analyzing the repetition rate after the cut repeated region is compared with the reference source data segment;
s4: the method comprises the steps that a data repetition counting monitoring unit is used for carrying out reasonability prediction on repeated data, a high repetition rate data obtaining and counting submodule obtains high repetition rate data segments marked by a fragmentation repetition rate comparison analysis submodule, centralized counting is carried out to form a list, the list is sent to a data integrity prediction submodule after simulated deletion, the data integrity prediction submodule deletes data with high repetition rate in data to be sent one by one after simulated deletion, whether the data integrity is influenced after the high repetition rate data are deleted or not is analyzed, when the fragmented data with high repetition rate are deleted, the data integrity is larger than a set threshold value, and a data deletion application with high repetition rate is sent to a repeated and miscellaneous data cleaning module;
s5: the data redundancy and redundancy cleaning module is used for cleaning and deleting the repeated data in the data to be sent, the communication returning unit obtains the deleting request of the high-repetition-rate fragment data counted by the data repetition counting monitoring unit, the deleting request is collected and sent to the manual response submodule, and the manual response submodule is manually accessed to manually judge the deleting request.
By adopting the technical scheme: in step S4, the presence rationality of the repetitive data is predicted by using the data repetition count monitoring unit, the high repetition rate data obtaining and counting submodule obtains the fragmentation repetition rate and compares the high repetition rate data fragments marked by the analysis submodule, centralized counting is performed to form a list, the list is sent to the data integrity prediction submodule after simulated deletion, the data integrity prediction submodule after simulated deletion deletes the data with high repetition rate one by one in the data to be sent, whether the data integrity is affected after the high repetition rate data is deleted is analyzed, when the fragment data with high repetition rate is deleted, the data integrity is greater than a set threshold, the data deletion application with high repetition rate is sent to the repeated miscellaneous data cleaning module, and the method further includes the following steps:
the high repetition rate data segment marked by the set fragmentation repetition rate contrast analysis submodule is Q1、Q2、Q3、…、Qn-1、QnThe data integrity prediction submodule performs one-by-one simulated deletion on the high-repetition-rate data segments after simulated deletion, monitors the integrity of the current data to be sent one by one while performing one-by-one simulated deletion, and sets the data integrity to be A when the high-repetition-rate data segments are subjected to one-by-one simulated deletion1、A2、A3、…、An-1、AnWhen monitoring An=100%, judging that the integrity of the data to be sent is not influenced after the segment data is deleted, sending the segment data to a repeated redundant and miscellaneous data cleaning module for manual confirmation, and deleting the segment data after the segment data is monitored to be AnLess than 85%, determining that the deleted data has influence on the integrity of the data to be transmitted, and not processing the data when monitoring AnC, determining the content of the current fragment data according to the content of 85,100 percent, and monitoring the data consistency Z after the current fragment data are deleted in a simulation mode for multiple times1、Z2、Z3、…、Zn-1、ZnWherein, in the step (A),
calculating to obtain the data after the current fragment data is subjected to analog deletionConsistency, set pair AnSecondary data integrity for E [85,100)% fragment data monitoring is BnAccording to the formula:
calculating to obtain the simulated deletion AnSecondary data integrity as monitored by fragment data of e [85,100)% when Bn=100%, judging that the integrity of the data to be sent is not influenced after the segment data is deleted, sending the segment data to a repeated redundant and miscellaneous data cleaning module for manual confirmation, and deleting the segment data when the segment data is deleted, wherein when the segment data is deleted, the segment data is deletednNot equal to 100%, judging that the completeness of the data to be transmitted is influenced after the fragment data is deleted, and not processing the fragment data.
Compared with the prior art, the invention has the beneficial effects that: the invention aims to monitor and analyze the repeated data, and manually confirm and delete the repeated data after confirming the data;
the data transmission method comprises the steps that a target migration data content reading module is used for scanning data to be transmitted and monitoring whether repeated information exists, a repeated data transmission region position positioning module is used for positioning and marking repeated regions of the data to be transmitted, a repeated data region fragmentation analysis module is used for monitoring data repetition rate after the regions where the repeated data are positioned are fragmented, a data repetition counting monitoring unit is used for reasonably predicting existence of the repeated data, and a repeated and redundant data cleaning module is used for cleaning and deleting the repeated data in the data to be transmitted.
Drawings
In order that the present invention may be more readily and clearly understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings.
FIG. 1 is a block diagram of a data retransmission flag system for wireless communication according to the present invention;
FIG. 2 is a diagram illustrating specific steps of a method for repeatedly sending and marking data for wireless communication according to the present invention;
fig. 3 is a schematic diagram illustrating an implementation method of a data retransmission flag method for wireless communication according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1 to 3, in an embodiment of the present invention, a data repeat transmission tagging system and method for wireless communication includes a target migration data content reading module, a repeated data transmission region position locating module, a repeated data region fragmentation analysis module, a data repeat count monitoring unit, and a repeat redundant data cleaning module, where the target migration data content reading module, the repeated data transmission region position locating module, the repeated data region fragmentation analysis module, the data repeat count monitoring unit, and the repeat redundant data cleaning module are sequentially connected via an intranet;
the target migration data content reading module is used for scanning data to be sent and monitoring whether repeated information exists in the data, the repeated data transmission region position positioning module is used for positioning and marking a repeated region of the data to be sent, the repeated data region fragmentation analysis module is used for monitoring the data repetition rate after the region where the repeated data is positioned is segmented, the data repetition counting monitoring unit is used for reasonably predicting existence of the repeated data, and the repeated and redundant data cleaning module is used for cleaning and deleting the repeated data in the data to be sent.
By adopting the technical scheme: the target migration data content reading module comprises a data information pre-scanning submodule to be sent and a repeated transmission data information counting submodule, wherein the data information pre-scanning submodule to be sent is used for scanning repeated data of data to be sent and sending the scanned data to the repeated transmission data information counting submodule, and the repeated transmission data information counting submodule is used for counting the scanned data, confirming whether the scanned data contains repeated data or not and marking the scanned repeated data in advance.
By adopting the technical scheme: the repeated data transmission region position positioning module comprises a duplicate data retrieval marking submodule and a data repeated node positioning submodule, wherein the duplicate data retrieval marking submodule is used for marking a data information duplicate contained in data to be transmitted after retrieval, the data duplicate is not used as a data repeated statistical range, and the data repeated node positioning submodule is used for positioning a repeated region monitored by data, marking a repeated region in the data to be transmitted and transmitting the marked region to the repeated data region fragmentation analysis module.
By adopting the technical scheme: the repeated data region fragmentation analysis module comprises a repeated data region cutting submodule and a fragmentation repetition rate comparison analysis submodule, wherein the repeated data region cutting submodule is used for cutting a data repetition region positioned by the data repetition node positioning submodule, thinning the repetition region, comparing the thinned repetition region with a reference source data segment, and the fragmentation repetition rate comparison analysis submodule is used for analyzing the repetition rate after comparing the cut repetition region with the reference source data segment.
By adopting the technical scheme: the fragmentation repetition rate comparison analysis submodule sets a certain repetition region as P0, the source data segment referred by the repetition region P0 as Ln, the repetition region P0 is compared with the source data segment referred to by the repetition region as Ln and then is cut, and the repetition data region cutting submodule sets the repetition region P0 to be cut into Y segments, wherein different segments are P segments1、p2、p3、…、pn-1、pnThe repetition rate of the different segments compared with the reference source data Ln is K1、K2、K3、…、Kn-1、KnWherein, K is setnThe repetition rate is more than or equal to 60 percent, and the repetition rate K isnSegmentation data of > 60% are marked.
By adopting the technical scheme: setting different segmentsRepetition rate K compared to reference source data LnnThe number of fragments is not less than 60% C0, C0 >)
Judging that the repetition rate between the repetition region P0 and the reference source data segment Ln is high when C0 <
Re-detecting the repetition rate of the comparison between different segments and the reference source data Ln, and extracting the repetition rate pnE [40%, 60%), the number of extracted fragment data is set to C1, and the fragment data is set to pl1、pl2、pl3、…、plnThe extracted fragment data is re-segmented, and the re-segmented data is set to p21、p22、p23、…、p2nWherein the number of the re-segmented data segments is C2, C2 ≧ 2C 1, and the repetition rate for detecting the comparison of the re-segmented data with the reference source data segment Ln is set as Km1、Km2、Km3、…、KmnExtracting KmnData for fragments of not less than 80%, KmnAnd judging that the repetition rate is high for more than or equal to 80 percent of fragment data, and marking.
By adopting the technical scheme: the data repetition counting monitoring unit comprises a high repetition rate data acquisition and statistics submodule and a data integrity prediction submodule after simulated deletion, wherein the high repetition rate data acquisition and statistics submodule is used for acquiring high repetition rate data fragments marked by a fragmentation repetition rate comparison analysis submodule, carrying out centralized statistics to form a list and sending the list to the data integrity prediction submodule after simulated deletion, the data integrity prediction submodule after simulated deletion is used for deleting data with high repetition rate in data to be sent one by one, analyzing whether the data integrity is influenced after the high repetition rate data is deleted or not, when the fragment data with high repetition rate is deleted, the data integrity is larger than a set threshold value, and sending a data deletion application with high repetition rate to the repeated redundant miscellaneous data cleaning module.
By adopting the technical scheme: the repeated and miscellaneous data cleaning module comprises a communication returning unit and a manual response submodule, wherein the communication returning unit is used for obtaining a deletion request of the high repetition rate fragment data counted by the data repetition counting monitoring unit, summarizing the deletion request and sending the deletion request to the manual response submodule, and the manual response submodule is used for manually accessing and manually judging the deletion request.
A data repeat transmission tagging method for wireless communication, characterized by:
s1: scanning data to be transmitted by using a target migration data content reading module, monitoring whether repeated information exists in the data to be transmitted, scanning repeated data of the data to be transmitted by using a data information pre-scanning submodule to be transmitted, transmitting the scanned data to a repeated transmission data information counting submodule, counting the scanned data by using the repeated transmission data information counting submodule, confirming whether the scanned data has the repeated data, and marking the scanned repeated data in advance;
s2: the repeated region of the data to be transmitted is positioned and marked by using a repeated data transmission region position positioning module, a duplicate data retrieval marking submodule retrieves and marks a data information duplicate contained in the data to be transmitted, the data duplicate is not used as a data duplicate statistical range, a data duplicate node positioning submodule positions the repeated region monitored by the data, marks the repeated region in the data to be transmitted, and transmits the marked region to a repeated data region fragmentation analysis module;
s3: a repeated data region fragmentation analysis module is used for monitoring the data repetition rate after the region where the repeated data is positioned is fragmented, a repeated data region cutting submodule is used for cutting the data repetition region where the data repetition node positioning submodule is positioned, the repeated region is refined, the refined repeated region is compared with the reference source data segment, and a fragmentation repetition rate comparison analysis submodule is used for analyzing the repetition rate after the cut repeated region is compared with the reference source data segment;
s4: the method comprises the steps that a data repetition counting monitoring unit is used for carrying out reasonability prediction on repeated data, a high repetition rate data obtaining and counting submodule obtains high repetition rate data segments marked by a fragmentation repetition rate comparison analysis submodule, centralized counting is carried out to form a list, the list is sent to a data integrity prediction submodule after simulated deletion, the data integrity prediction submodule deletes data with high repetition rate in data to be sent one by one after simulated deletion, whether the data integrity is influenced after the high repetition rate data are deleted or not is analyzed, when the fragmented data with high repetition rate are deleted, the data integrity is larger than a set threshold value, and a data deletion application with high repetition rate is sent to a repeated and miscellaneous data cleaning module;
s5: the data redundancy and redundancy cleaning module is used for cleaning and deleting the repeated data in the data to be sent, the communication returning unit obtains the deleting request of the high-repetition-rate fragment data counted by the data repetition counting monitoring unit, the deleting request is collected and sent to the manual response submodule, and the manual response submodule is manually accessed to manually judge the deleting request.
By adopting the technical scheme: in step S4, the presence rationality of the repetitive data is predicted by using the data repetition count monitoring unit, the high repetition rate data obtaining and counting submodule obtains the fragmentation repetition rate and compares the high repetition rate data fragments marked by the analysis submodule, centralized counting is performed to form a list, the list is sent to the data integrity prediction submodule after simulated deletion, the data integrity prediction submodule after simulated deletion deletes the data with high repetition rate one by one in the data to be sent, whether the data integrity is affected after the high repetition rate data is deleted is analyzed, when the fragment data with high repetition rate is deleted, the data integrity is greater than a set threshold, the data deletion application with high repetition rate is sent to the repeated miscellaneous data cleaning module, and the method further includes the following steps:
the high repetition rate data segment marked by the set fragmentation repetition rate contrast analysis submodule is Q1、Q2、Q3、…、Qn-1、QnThe data integrity prediction submodule performs one-by-one simulated deletion on the high-repetition-rate data segments after simulated deletion, monitors the integrity of the current data to be sent one by one while performing one-by-one simulated deletion, and sets the data integrity to be A when the high-repetition-rate data segments are subjected to one-by-one simulated deletion1、A2、A3、…、An-1、AnWhen monitoring An=100%, judging that the integrity of the data to be sent is not influenced after the segment data is deleted, sending the segment data to a repeated redundant and miscellaneous data cleaning module for manual confirmation, and deleting the segment data after the segment data is monitored to be AnLess than 85%, determining that the deleted data has influence on the integrity of the data to be transmitted, and not processing the data when monitoring AnC, determining the content of the current fragment data according to the content of 85,100 percent, and monitoring the data consistency Z after the current fragment data are deleted in a simulation mode for multiple times1、Z2、Z3、…、Zn-1、ZnWherein, in the step (A),
calculating the data consistency of the current fragment data after the simulated deletion, and setting a pair AnSecondary data integrity for E [85,100)% fragment data monitoring is BnAccording to the formula:
calculating to obtain the simulated deletion AnSecondary data integrity as monitored by fragment data of e [85,100)% when Bn=100%, judging that the integrity of the data to be sent is not influenced after the segment data is deleted, sending the segment data to a repeated redundant and miscellaneous data cleaning module for manual confirmation, and deleting the segment data when the segment data is deleted, wherein when the segment data is deleted, the segment data is deletednNot equal to 100%, judging that the completeness of the data to be transmitted is influenced after the fragment data is deleted, and not processing the fragment data.
Example 1: defining conditions, setting a repeated data region cutting submodule to cut the repeated region P0 into 4 segments, wherein different segments are P1、p2、p3、p4、p5The repetition rate of the different segments compared with the reference source data Ln is 70%, 67%, 51%, 57%, 49%, wherein K is setnMore than or equal to 60 percent is high in repetition rate, 70 percent more than or equal to 60 percent, 67 percent more than or equal to 60 percent, p1、p2Marking the segmented data;
repetition rate K of different segments compared to reference source data LnnThe number of fragments of not less than 60% is 2,
re-detecting the repetition rate of the comparison between different segments and the reference source data Ln, and extracting the repetition rate pnE [40%, 60%), the number of extracted fragment data is set to 3, and the fragment data is set to pl1、pl2、pl3The extracted fragment data is re-segmented, and the re-segmented data is set to p21、p22、p23、p24、p25、p26Wherein the number of the data segments to be re-segmented is 6, 6=2 × 3, the repetition rate of the comparison between the detected re-segmented data and the reference source data segment Ln is set to 89%, 45%, 97%, 39%, 55%, 87%, and p2 is extracted1、p23、p26Data of fragments of (1), KmnThe repetition rate is judged to be high for fragment data of more than or equal to 80 percent, and p2 is used1、p23、p26Fragment data is tagged.
Example 2: the limiting conditions are as follows: setting the high-repetition-rate data segment marked by the fragmentation repetition rate contrast analysis submodule as Q1、Q2、Q3、Q4The data integrity prediction submodule performs one-by-one simulated deletion on the high repetition rate data segments after simulated deletion, monitors the integrity of the current data to be sent one by one while performing one-by-one simulated deletion, sets the data integrity to be 100%, 89%, 100% and 79% when the high repetition rate data segments are subjected to one-by-one simulated deletion, and monitors out the simulated deletion Q1、Q3The integrity of the data to be sent after the data segment is 100 percent, and Q is judged1、Q3And after the fragment data is deleted, the integrity of the data to be sent is not influenced, and the fragment data is sent to a repeated redundant and miscellaneous data cleaning module for manual confirmation and then deleted.
Example 3: defining conditions, setting the high-repetition-rate data segment marked by the fragmentation repetition rate contrast analysis submodule as Q1、Q2、Q3、Q4Simulation deletionAnd the post-data integrity prediction submodule carries out one-by-one simulation deletion on the high-repetition-rate data segments, monitors the integrity of the current data to be sent one by one while carrying out one-by-one simulation deletion, sets the data integrity to be 100%, 89%, 100% and 79% when the high-repetition-rate data segments are subjected to one-by-one simulation deletion, and monitors out simulation deletion Q4The integrity of data to be sent after the data fragment is 79%, 79% < 85%, and the fragment data Q is judged4After deletion, the integrity of the data to be sent is influenced, and the fragment data is not processed.
Example 4: a limiting condition, the limiting condition: setting the high-repetition-rate data segment marked by the fragmentation repetition rate contrast analysis submodule as Q1、Q2、Q3、Q4The data integrity prediction submodule performs one-by-one simulated deletion on the high-repetition-rate data segments after simulated deletion, monitors the integrity of the current data to be sent one by one while performing one-by-one simulated deletion, sets the data integrity to be 100%, 89%, 100% and 79% when the high-repetition-rate data segments are subjected to one-by-one simulated deletion, and monitors the simulated deletion data segments Q2The integrity of data to be sent is 89%, 89% belongs to 85,100%, and the current fragment data Q is monitored for multiple times2The deleted data consistency is simulated as 91%, 89%, 94%, 90%, wherein,
calculating the data consistency of the current fragment data after the simulated deletion, and setting a pair AnSecondary data integrity for E [85,100)% fragment data monitoring is BnAccording to the formula:
calculating to obtain the simulated deletion AnSecondary data integrity 98.5, B for E [85,100)% fragment data monitoringnNot equal to 100%, judging that the completeness of the data to be transmitted is influenced after the fragment data is deleted, and not processing the fragment data.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (2)
1. A data repeat transmission tagging system for wireless communication, characterized by: the system comprises a target migration data content reading module, a repeated data transmission region position positioning module, a repeated data region fragmentation analysis module, a repeated data repetition counting monitoring unit and a repeated redundant and miscellaneous data cleaning module, wherein the target migration data content reading module, the repeated data transmission region position positioning module, the repeated data region fragmentation analysis module, the repeated data repetition counting monitoring unit and the repeated redundant and miscellaneous data cleaning module are sequentially connected through an intranet;
the target migration data content reading module is used for scanning data to be sent and monitoring whether repeated information exists in the data, the repeated data transmission region position positioning module is used for positioning and marking a repeated region of the data to be sent, the repeated data region fragmentation analysis module is used for monitoring the data repetition rate after the region where the repeated data is positioned is segmented, the data repetition counting monitoring unit is used for reasonably predicting existence of the repeated data, and the repeated and redundant data cleaning module is used for cleaning and deleting the repeated data in the data to be sent;
the target migration data content reading module comprises a data information pre-scanning submodule to be sent and a repeated transmission data information counting submodule, wherein the data information pre-scanning submodule to be sent is used for scanning repeated data of data to be sent and sending the scanned data to the repeated transmission data information counting submodule;
the data repeated transmission marking method of the marking system comprises the following steps:
s1: scanning data to be transmitted by using a target migration data content reading module, monitoring whether repeated information exists in the data to be transmitted, scanning repeated data of the data to be transmitted by using a data information pre-scanning submodule to be transmitted, transmitting the scanned data to a repeated transmission data information counting submodule, counting the scanned data by using the repeated transmission data information counting submodule, confirming whether the scanned data has the repeated data, and marking the scanned repeated data in advance;
s2: the repeated region of the data to be transmitted is positioned and marked by using a repeated data transmission region position positioning module, a duplicate data retrieval marking submodule retrieves and marks a data information duplicate contained in the data to be transmitted, the data duplicate is not used as a data duplicate statistical range, a data duplicate node positioning submodule positions the repeated region monitored by the data, marks the repeated region in the data to be transmitted, and transmits the marked region to a repeated data region fragmentation analysis module;
s3: a repeated data region fragmentation analysis module is used for monitoring the data repetition rate after the region where the repeated data is positioned is fragmented, a repeated data region cutting submodule is used for cutting the data repetition region where the data repetition node positioning submodule is positioned, the repeated region is refined, the refined repeated region is compared with the reference source data segment, and a fragmentation repetition rate comparison analysis submodule is used for analyzing the repetition rate after the cut repeated region is compared with the reference source data segment;
s4: the method comprises the steps that a data repetition counting monitoring unit is used for carrying out reasonability prediction on repeated data, a high repetition rate data obtaining and counting submodule obtains high repetition rate data segments marked by a fragmentation repetition rate comparison analysis submodule, centralized counting is carried out to form a list, the list is sent to a data integrity prediction submodule after simulated deletion, the data integrity prediction submodule deletes data with high repetition rate in data to be sent one by one after simulated deletion, whether the data integrity is influenced after the high repetition rate data are deleted or not is analyzed, when the fragmented data with high repetition rate are deleted, the data integrity is larger than a set threshold value, and a data deletion application with high repetition rate is sent to a repeated and miscellaneous data cleaning module;
s5: the data redundancy and redundancy cleaning module is used for cleaning and deleting the repeated data in the data to be sent, the communication returning unit obtains the deleting request of the high-repetition-rate fragment data counted by the data repetition counting monitoring unit, the deleting request is collected and sent to the manual response submodule, and the manual response submodule is manually accessed to manually judge the deleting request.
2. A data repeat request tagging system for wireless communication according to claim 1 further comprising: in step S4, the presence rationality of the repetitive data is predicted by using the data repetition count monitoring unit, the high repetition rate data obtaining and counting submodule obtains the fragmentation repetition rate and compares the high repetition rate data fragments marked by the analysis submodule, centralized counting is performed to form a list, the list is sent to the data integrity prediction submodule after simulated deletion, the data integrity prediction submodule after simulated deletion deletes the data with high repetition rate one by one in the data to be sent, whether the data integrity is affected after the high repetition rate data is deleted is analyzed, when the fragment data with high repetition rate is deleted, the data integrity is greater than a set threshold, the data deletion application with high repetition rate is sent to the repeated miscellaneous data cleaning module, and the method further includes the following steps:
the high repetition rate data segment marked by the set fragmentation repetition rate contrast analysis submodule is Q1、Q2、Q3、…、Qn-1、QnThe data integrity prediction submodule performs one-by-one simulated deletion on the high-repetition-rate data segments after simulated deletion, monitors the integrity of the current data to be sent one by one while performing one-by-one simulated deletion, and sets the data integrity to be A when the high-repetition-rate data segments are subjected to one-by-one simulated deletion1、A2、A3、…、An-1、AnWhen monitoring An=100%, judging that the integrity of the data to be sent is not influenced after the segment data is deleted, sending the segment data to a repeated redundant and miscellaneous data cleaning module for manual confirmation, and deleting the segment data after the segment data is monitored to be AnLess than 85%, determining that the deleted data has influence on the integrity of the data to be transmitted, and not processing the data when monitoring AnC, determining the content of the current fragment data according to the content of 85,100 percent, and monitoring the data consistency Z after the current fragment data are deleted in a simulation mode for multiple times1、Z2、Z3、…、Zn-1、ZnWherein, in the step (A),
calculating the data consistency of the current fragment data after the simulated deletion, and setting a pair AnSecondary data integrity for E [85,100)% fragment data monitoring is BnThe meanings expressed by Znmin and Znmax refer to the minimum value and the maximum value of data continuity after the current fragment data are monitored for multiple times and deleted in a simulation way; according to the formula:
calculating to obtain the simulated deletion AnSecondary data integrity as monitored by fragment data of e [85,100)% when Bn=100%, judging that the integrity of the data to be sent is not influenced after the segment data is deleted, sending the segment data to a repeated redundant and miscellaneous data cleaning module for manual confirmation, and deleting the segment data when the segment data is deleted, wherein when the segment data is deleted, the segment data is deletednNot equal to 100%, judging that the completeness of the data to be transmitted is influenced after the fragment data is deleted, and not processing the fragment data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110745357.2A CN113472609B (en) | 2020-05-25 | 2020-05-25 | Data repeated sending marking system for wireless communication |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010447145.1A CN111628909B (en) | 2020-05-25 | 2020-05-25 | Data repeated sending marking system and method for wireless communication |
| CN202110745357.2A CN113472609B (en) | 2020-05-25 | 2020-05-25 | Data repeated sending marking system for wireless communication |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010447145.1A Division CN111628909B (en) | 2020-05-25 | 2020-05-25 | Data repeated sending marking system and method for wireless communication |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113472609A true CN113472609A (en) | 2021-10-01 |
| CN113472609B CN113472609B (en) | 2024-03-19 |
Family
ID=72272447
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110745357.2A Active CN113472609B (en) | 2020-05-25 | 2020-05-25 | Data repeated sending marking system for wireless communication |
| CN202010447145.1A Active CN111628909B (en) | 2020-05-25 | 2020-05-25 | Data repeated sending marking system and method for wireless communication |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010447145.1A Active CN111628909B (en) | 2020-05-25 | 2020-05-25 | Data repeated sending marking system and method for wireless communication |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN113472609B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1206315A (en) * | 1997-04-10 | 1999-01-27 | 索尼公司 | Transmission speed predicting apparatus and transmission speed predicting method |
| CN101784996A (en) * | 2007-06-29 | 2010-07-21 | 赛帕顿有限公司 | Emulated storage system |
| US20100246472A1 (en) * | 2009-03-25 | 2010-09-30 | Wi-Lan, Inc. | System and method for proactive repeat transmission of data over an unreliable transmission medium |
| US20110231362A1 (en) * | 2010-03-16 | 2011-09-22 | Deepak Attarde | Extensible data deduplication system and method |
| CN102221982A (en) * | 2011-06-13 | 2011-10-19 | 北京卓微天成科技咨询有限公司 | Method and system for implementing deletion of repeated data on block-level virtual storage equipment |
| CN102323958A (en) * | 2011-10-27 | 2012-01-18 | 上海文广互动电视有限公司 | Data de-duplication method |
| CN102831222A (en) * | 2012-08-24 | 2012-12-19 | 华中科技大学 | Differential compression method based on data de-duplication |
| CN103189867A (en) * | 2012-10-30 | 2013-07-03 | 华为技术有限公司 | Duplicated data search method and equipment |
| CN103580956A (en) * | 2013-11-05 | 2014-02-12 | 北京锐安科技有限公司 | Method and device for detecting data integrity |
| US20150026135A1 (en) * | 2013-07-17 | 2015-01-22 | International Business Machines Corporation | Adaptive similarity search resolution in a data deduplication system |
| CN105808169A (en) * | 2016-03-14 | 2016-07-27 | 联想(北京)有限公司 | Data deduplication method, apparatus and system |
| US20180139118A1 (en) * | 2016-11-15 | 2018-05-17 | At&T Intellectual Property I, L.P. | Recovering a replica in an operator in a data streaming processing system |
| CN109597574A (en) * | 2018-11-27 | 2019-04-09 | 深圳市酷开网络科技有限公司 | Distributed data storage method, server and readable storage medium storing program for executing |
| CN109739841A (en) * | 2018-12-14 | 2019-05-10 | 国网江苏省电力有限公司电力科学研究院 | A kind of integration system being directed to electrical equipment online supervision repeated data and method |
| CN111046030A (en) * | 2019-12-11 | 2020-04-21 | 姚欣 | Data elimination integration system for big data |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678158B (en) * | 2013-12-26 | 2016-11-02 | 中国科学院信息工程研究所 | A kind of data layout optimization method and system |
| CN107229660A (en) * | 2016-03-25 | 2017-10-03 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of data deduplication |
| CN109104732B (en) * | 2018-06-13 | 2021-06-01 | 珠海格力电器股份有限公司 | Data sending method and device and intelligent electric appliance |
-
2020
- 2020-05-25 CN CN202110745357.2A patent/CN113472609B/en active Active
- 2020-05-25 CN CN202010447145.1A patent/CN111628909B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1206315A (en) * | 1997-04-10 | 1999-01-27 | 索尼公司 | Transmission speed predicting apparatus and transmission speed predicting method |
| CN101784996A (en) * | 2007-06-29 | 2010-07-21 | 赛帕顿有限公司 | Emulated storage system |
| US20100246472A1 (en) * | 2009-03-25 | 2010-09-30 | Wi-Lan, Inc. | System and method for proactive repeat transmission of data over an unreliable transmission medium |
| US20110231362A1 (en) * | 2010-03-16 | 2011-09-22 | Deepak Attarde | Extensible data deduplication system and method |
| CN102221982A (en) * | 2011-06-13 | 2011-10-19 | 北京卓微天成科技咨询有限公司 | Method and system for implementing deletion of repeated data on block-level virtual storage equipment |
| CN102323958A (en) * | 2011-10-27 | 2012-01-18 | 上海文广互动电视有限公司 | Data de-duplication method |
| CN102831222A (en) * | 2012-08-24 | 2012-12-19 | 华中科技大学 | Differential compression method based on data de-duplication |
| CN103189867A (en) * | 2012-10-30 | 2013-07-03 | 华为技术有限公司 | Duplicated data search method and equipment |
| US20150026135A1 (en) * | 2013-07-17 | 2015-01-22 | International Business Machines Corporation | Adaptive similarity search resolution in a data deduplication system |
| CN103580956A (en) * | 2013-11-05 | 2014-02-12 | 北京锐安科技有限公司 | Method and device for detecting data integrity |
| CN105808169A (en) * | 2016-03-14 | 2016-07-27 | 联想(北京)有限公司 | Data deduplication method, apparatus and system |
| US20180139118A1 (en) * | 2016-11-15 | 2018-05-17 | At&T Intellectual Property I, L.P. | Recovering a replica in an operator in a data streaming processing system |
| CN109597574A (en) * | 2018-11-27 | 2019-04-09 | 深圳市酷开网络科技有限公司 | Distributed data storage method, server and readable storage medium storing program for executing |
| CN109739841A (en) * | 2018-12-14 | 2019-05-10 | 国网江苏省电力有限公司电力科学研究院 | A kind of integration system being directed to electrical equipment online supervision repeated data and method |
| CN111046030A (en) * | 2019-12-11 | 2020-04-21 | 姚欣 | Data elimination integration system for big data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113472609B (en) | 2024-03-19 |
| CN111628909B (en) | 2021-08-20 |
| CN111628909A (en) | 2020-09-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101637042B (en) | Wireless cell monitoring method, its device, and its program | |
| CN104657437A (en) | Monitoring method and monitoring device for promotion status data | |
| Zhao et al. | Temporal‐spatial distribution nature of traffic and base stations in cellular networks | |
| CN115345466A (en) | Power distribution network online evaluation system based on network frame topology and multi-source data fusion | |
| US20220103442A1 (en) | Internet of things operations monitoring system | |
| US8634965B1 (en) | Computer-based method for power system state estimation | |
| CN111628909B (en) | Data repeated sending marking system and method for wireless communication | |
| Asghar et al. | Correlation-based cell degradation detection for operational fault detection in cellular wireless base-stations | |
| CN113591803A (en) | Intelligent remote inspection method and system for power distribution room based on edge cloud cooperation technology | |
| CN111490991B (en) | Multiple server connection request system and method based on communication equipment | |
| Di Cicco et al. | Machine learning-based line-of-sight prediction in urban manhattan-like environments | |
| CN112116511A (en) | State monitoring method and device for urban rail transit system | |
| Alves et al. | A novel approach for user equipment indoor/outdoor classification in mobile networks | |
| CN109982373B (en) | LTE frequent fallback point analysis method and device, computing device and storage medium | |
| Bessho et al. | Sensing Street-level Crowd Density by Observing Public Bluetooth Low Energy Advertisements from Contact Tracing Applications | |
| CN115330140A (en) | Building risk prediction method based on data mining and prediction system thereof | |
| CN115981192A (en) | Industrial network based cooperative control and prejudgment method | |
| CN115379456A (en) | Power 5G network risk assessment method and device, electronic equipment and storage medium | |
| CN114124555A (en) | Message playback method and device, electronic equipment and computer readable medium | |
| CN117560706B (en) | Root cause analysis method, root cause analysis device, electronic equipment and storage medium | |
| CN112020087A (en) | Tunnel fault monitoring method and device and computing equipment | |
| Ortega‐Sicilia et al. | Design and evaluation of ANR algorithm for LTE real scenario with high interference | |
| CN112307075A (en) | User relationship identification method and device | |
| CN111787557B (en) | Information exchange return system and method based on wireless communication | |
| US11528211B2 (en) | Data inspection system for inspecting operating situations of multiple IoT devices in a plurality of target IoT systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |