CN111614786A - System and method for processing data at high speed by remote server based on block chain - Google Patents
System and method for processing data at high speed by remote server based on block chain Download PDFInfo
- Publication number
- CN111614786A CN111614786A CN202010506120.4A CN202010506120A CN111614786A CN 111614786 A CN111614786 A CN 111614786A CN 202010506120 A CN202010506120 A CN 202010506120A CN 111614786 A CN111614786 A CN 111614786A
- Authority
- CN
- China
- Prior art keywords
- data
- submodule
- remote server
- module
- classification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/45—Clustering; Classification
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
- H04L63/1466—Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Multimedia (AREA)
- Computer Hardware Design (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a high-speed processing system and a method for data by a remote server based on a block chain, the system comprises a data shunting and collecting and buffering module, an invalid data real-time intercepting module, a data uniform storage and classification module, a plurality of sampling point extraction modules and a block chain platform, wherein the data shunting and collecting and buffering module is used for establishing a buffering and shunting region for data sent to the remote server, the invalid data real-time intercepting module is used for intercepting invalid data sent to the remote server, the data uniform storage and classification module is used for classifying and classifying screened data and then storing the screened data, the data plurality of sampling point extraction modules are used for collecting partial data of the data, classifying and dividing the data according to the collection result, the block chain platform is used for coordinating all the modules and accessing a third-party client side for data query, the method aims to establish a server buffer and distribution area, screen different data in real time and avoid the server from processing repeated invalid data.
Description
Technical Field
The invention relates to the field of data processing, in particular to a system and a method for processing data at a high speed by a remote server based on a block chain.
Background
Data is the presentation and carrier of information, which may be symbols, words, numbers, speech, images, video, etc. The information is the connotation of the data, and the information is loaded on the data and makes meaningful explanation on the data. Data and information are inseparable, and information is expressed in dependence on data, which is vivid and embodies the information. The data is a symbol, is physical, and the information is data which is obtained after processing the data and has influence on decision, and is logical and conceptual; data is a representation of information, which is a meaningful representation of data. The data is the expression and carrier of the information, and the information is the connotation of the data and is the relation of shape and quality. Data itself has no meaning, and data only becomes information when it affects the behavior of an entity.
Data is a form of expression for facts, concepts, or instructions that may be processed by human or automated means. After the data is interpreted and given a certain meaning, it becomes information. The data processing is the collection, storage, retrieval, processing, transformation and transmission of data.
The basic purpose of data processing is to extract and derive valuable, meaningful data for certain people from large, possibly chaotic, unintelligible amounts of data.
Data processing is the basic link of system engineering and automatic control. Data processing is throughout various fields of social production and social life. The development of data processing technology and the breadth and depth of application thereof greatly influence the progress of the development of human society
At present, when a server processes data, the data are directly received, but the overload of the data processing amount of the server can occur under the condition of excessive data, the server cannot process the data, the efficiency is low, the server can also receive the data in a full disk mode, and the redundant and miscellaneous data are increased.
Disclosure of Invention
The invention aims to provide a system and a method for processing data at a high speed by a remote server based on a block chain, so as to solve the problems in the prior art.
In order to achieve the purpose, the invention provides the following technical scheme:
a remote server high-speed data processing system based on a block chain comprises a data shunting and collecting buffer module, an invalid data real-time intercepting module, a data uniform storage and classification module, a plurality of data sampling point extracting modules and a block chain platform, wherein the data shunting and collecting buffer module, the invalid data real-time intercepting module, the data uniform storage and classification module and the data plurality of sampling point extracting modules are sequentially connected through an intranet, and the block chain platform and the data uniform storage and classification module are mutually connected through the intranet;
the data distributing and streaming collection buffer module is used for establishing a buffer and a distribution area for data sent to a remote server, the invalid data real-time interception module is used for intercepting invalid data sent to the remote server, the data unified storage and classification module is used for storing screened data after classifying and dividing, the data sampling point extraction modules are used for collecting partial data of the data, classifying and dividing the data according to a collection result, and the block chain platform is used for coordinating all the modules and accessing a third-party client to inquire the data.
By adopting the technical scheme: the data distribution flow acquisition and buffering module comprises a plurality of buffer area establishing sub-modules and a data distribution flow introduction and arrangement sub-module, the buffer area establishing sub-modules are used for establishing a plurality of buffer areas when the remote server receives data, a large amount of data is guided to enter the buffer areas for buffering, and the data distribution flow introduction and arrangement sub-module is used for arranging the data of the buffer areas and then distributing the data to the remote server for data processing.
By adopting the technical scheme: the invalid data real-time intercepting module comprises a target repeated data reading and monitoring submodule, a data segment repetition rate comparison and analysis submodule and an invalid IP address intercepting submodule, wherein the target repeated data reading and monitoring submodule is used for reading data sent to a remote server, monitoring whether the sent data has repeated data or not, marking the monitored repeated data and the target compared data, sending a marking result to the data segment repetition rate comparison and analysis submodule, the data segment repetition rate comparison and analysis submodule segments the monitored repeated data, analyzes the repetition rate of the segmented repeated data and the target compared data, and the invalid IP address intercepting submodule is used for detecting the IP address of a data sending device and monitoring the validity of the IP address, and when the current IP address is monitored to belong to the invalid IP address, all data sent by the equipment are intercepted.
By adopting the technical scheme: the data segment repetition rate comparison and analysis submodule performs repetition rate analysis on the monitored repeated data and the target comparison data, the data segment repetition rate comparison and analysis submodule is set to segment the monitored repeated data, and the repeated data segment is set to be L1、L2、L3、…、Ln-1、LnSetting the current target contrast data to L0, and setting the repetition rate of each segment to L0
According to the formula:
the formula I is as follows:
the formula II is as follows:
when the monitored repeated data meet the first formula and the second formula, judging that the repetition rate of the current monitored data and the target comparison data does not meet the interception requirement, continuously processing the monitored data by the remote server, and when the monitored repeated data meet the first formula and meet the second formula, processing the monitored data by the remote server
And after the data segment repetition rate comparison analysis submodule marks the segment data, the remote server continues to process the monitoring data, and when the monitored repeated data does not satisfy the formula I and the formula II, the remote server intercepts the monitoring data.
By adopting the technical scheme: the data uniform storage and classification module comprises a data packet corresponding division submodule and a multi-classification extraction and call-in submodule, wherein the data packet corresponding division submodule is used for dividing data received inside the remote server into different data packets, the cached data volume of each data packet is the same, the multi-classification extraction and call-in submodule is used for performing fixed-point sampling on a plurality of data in each data packet, different data classes are formed according to sampling data, and the counted data classes are sent to a plurality of sampling point extraction modules of the data.
By adopting the technical scheme: the data multiple sampling point extraction module comprises a target data difference point analysis and confirmation submodule and a different classification data extraction submodule, the target data difference point analysis and confirmation submodule is used for extracting data in a data packet, analyzing different classification ratios in the extracted data, confirming the classification of the data according to the different classification ratios, the different classification data extraction submodule is used for confirming the classification ratio of each section of data in the data packet confirmed by the target data difference point analysis and confirmation submodule, confirming the classification of each section of data, subdividing the data packet of each section of data according to different classifications, and sending the subdivided data packet to the multi-classification extraction and calling submodule for summarizing.
By adopting the technical scheme: the block chain platform comprises an effective data real-time backup submodule and a data query access submodule, wherein the effective data real-time backup submodule is used for carrying out real-time backup on effective data stored in the data uniform storage and classification module, and the data query access submodule is used for accessing the block chain platform at a third-party client side to query data information in the system.
A high-speed data processing method of a remote server based on a block chain comprises the following steps:
s1: the data streaming acquisition and buffering module is used for establishing a buffering and shunting area for data sent to a remote server;
s2: the invalid data real-time interception module is used for intercepting invalid data sent to a remote server;
s3: the data unified storage and classification module is used for classifying and classifying the screened data and then storing the classified and classified data;
s4: the data extraction modules are used for collecting partial data of the data and classifying the data according to the collection result;
s5: and coordinating all modules by using the block chain platform, and accessing a third-party client to perform data query.
By adopting the technical scheme: the processing method further comprises the following steps:
s1-1, establishing a plurality of buffer areas when the remote server receives data by using a plurality of buffer area establishing sub-modules, guiding a large amount of data into the plurality of buffer areas for buffering, and guiding the data of the plurality of buffer areas into a sorting sub-module for sorting and then guiding the data into the remote server for data processing;
s2-1: reading data sent to a remote server by using a target repeated data reading monitoring submodule, monitoring whether the sent data has repeated data or not, marking the monitored repeated data and target comparison data, sending a marking result to a data segmentation repetition rate comparison and analysis submodule, segmenting the monitored repeated data by using the data segmentation repetition rate comparison and analysis submodule, carrying out repetition rate analysis on the segmented repeated data and the target comparison data, detecting an IP address of a data sending device by using an invalid IP address interception submodule, monitoring the validity of the IP address, and when the current IP address is monitored to belong to the invalid IP address, completely intercepting the data sent by the device;
s3-1: the data packet corresponding division submodule is used for dividing different data packets of data received in the remote server, the cached data volume of each data packet is the same, the multi-classification extraction calling submodule carries out fixed point sampling on a plurality of data in each data packet, different data categories are formed according to the sampled data, and the counted data categories are sent to a plurality of sampling point extraction modules of the data;
s4-1: extracting data in the data packet by using a target data difference point analysis and confirmation submodule, analyzing different class ratios in the extracted data, confirming the class of the data according to the different class ratios, determining the class ratio of each segment of data in the data packet confirmed by the target data difference point analysis and confirmation submodule by using a different class data extraction submodule, confirming the class of each segment of data, re-dividing each segment of data into data packets according to different classes, and sending the divided data packets to a multi-class extraction and call-in submodule for gathering;
s5-1: and the data query access sub-module accesses the block chain platform at a third-party client to query the data information in the system.
By adopting the technical scheme: in the step S4-1, the target data difference point analyzing and confirming submodule is used to extract data inside the data packet, different category ratios in the extracted data are analyzed, categories of the data are confirmed according to the different category ratios, the different classification data extracting submodule confirms the category ratio of each data in the data packet confirmed by the target data difference point analyzing and confirming submodule to confirm the category of each data, each data is re-divided into data packets according to different categories, and the divided data packets are sent to the multi-classification extraction call-in submodule for gathering, which further includes the following steps:
the data categories collected by the multi-classification extraction calling sub-module comprise: the data processing method comprises the following steps that text data, image data, voice data, video data and comprehensive data are judged, wherein the current data is the comprehensive data if one section of data comprises the text data, the image data, the voice data and the video data;
setting a target data difference point analysis and confirmation submodule to extract data A in a data packet, extracting different types of data in the data A, setting data with three different types in the current data A, wherein the data with three different types respectively account for D1%, D2% and D3% in the data A, judging that the type of the current data A is the data type with the accounting for D1% when D1% is more than or equal to 60%, and judging that the current data A is comprehensive data when D1% is less than 60%, D2% is less than 60% and D3% is less than 60%;
the data with two different types in the current data A are set, the proportion of the data in the data A respectively comprises D1% and D2%, when D1% is greater than D2%, the type of the current data A is judged to be the data type with the proportion of D1%, and when D1% is D2%, the current data A is judged to be comprehensive data.
Compared with the prior art, the invention has the beneficial effects that: the invention aims to establish a server buffer and a shunting area, shunt received data, screen different data in real time and avoid the server from processing repeated data and invalid data;
the data distribution and stream collection buffer module is used for establishing a buffer and a stream distribution area for data sent to a remote server, the invalid data real-time interception module is used for intercepting invalid data sent to the remote server, the data unified storage and classification module is used for classifying and classifying screened data and then storing the classified and classified data, the data sampling point extraction modules are used for collecting partial data of the data, classifying and classifying the data according to a collection result, and the block chain platform is used for coordinating all the modules and accessing a third-party client to inquire the data.
Drawings
In order that the present invention may be more readily and clearly understood, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings.
FIG. 1 is a block chain-based block structure diagram of a high-speed data processing system by a remote server according to the present invention;
FIG. 2 is a schematic diagram illustrating steps of a block chain-based method for processing data at high speed by a remote server according to the present invention;
FIG. 3 is a diagram illustrating specific steps of a block chain-based method for processing data at high speed by a remote server according to the present invention;
fig. 4 is a schematic diagram of an implementation method of the remote server for high-speed data processing based on the blockchain according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1 to 4, in the embodiment of the present invention, a system and a method for processing data at a high speed by a remote server based on a block chain are provided, where the system includes a data shunting and collecting buffer module, an invalid data real-time intercepting module, a data uniform storage and classification module, a plurality of sampling point extracting modules, and a block chain platform, where the data shunting and collecting buffer module, the invalid data real-time intercepting module, the data uniform storage and classification module, and the plurality of sampling point extracting modules are sequentially connected through an intranet, and the block chain platform and the data uniform storage and classification module are connected to each other through the intranet;
the data distributing and streaming collection buffer module is used for establishing a buffer and a distribution area for data sent to a remote server, the invalid data real-time interception module is used for intercepting invalid data sent to the remote server, the data unified storage and classification module is used for storing screened data after classifying and dividing, the data sampling point extraction modules are used for collecting partial data of the data, classifying and dividing the data according to a collection result, and the block chain platform is used for coordinating all the modules and accessing a third-party client to inquire the data.
By adopting the technical scheme: the data distribution flow acquisition and buffering module comprises a plurality of buffer area establishing sub-modules and a data distribution flow introduction and arrangement sub-module, the buffer area establishing sub-modules are used for establishing a plurality of buffer areas when the remote server receives data, a large amount of data is guided to enter the buffer areas for buffering, and the data distribution flow introduction and arrangement sub-module is used for arranging the data of the buffer areas and then distributing the data to the remote server for data processing.
By adopting the technical scheme: the invalid data real-time intercepting module comprises a target repeated data reading and monitoring submodule, a data segment repetition rate comparison and analysis submodule and an invalid IP address intercepting submodule, wherein the target repeated data reading and monitoring submodule is used for reading data sent to a remote server, monitoring whether the sent data has repeated data or not, marking the monitored repeated data and the target compared data, sending a marking result to the data segment repetition rate comparison and analysis submodule, the data segment repetition rate comparison and analysis submodule segments the monitored repeated data, analyzes the repetition rate of the segmented repeated data and the target compared data, and the invalid IP address intercepting submodule is used for detecting the IP address of a data sending device and monitoring the validity of the IP address, and when the current IP address is monitored to belong to the invalid IP address, all data sent by the equipment are intercepted.
By adopting the technical scheme: the data segment repetition rate comparison and analysis submodule performs repetition rate analysis on the monitored repeated data and the target comparison data, the data segment repetition rate comparison and analysis submodule is set to segment the monitored repeated data, and the repeated data segment is set to be L1、L2、L3、…、Ln-1、LnSetting the current target contrast data to L0, and setting the repetition rate of each segment to L0
According to the formula:
the formula I is as follows:
the formula II is as follows:
when the monitored repeated data meets the first formula and the second formula, the remote server judges that the repetition rate of the current monitored data and the target comparison data does not meet the interception requirement, the remote server continues to process the monitored data, when the monitored repeated data meets the first formula,and is
And after the data segment repetition rate comparison analysis submodule marks the segment data, the remote server continues to process the monitoring data, and when the monitored repeated data does not satisfy the formula I and the formula II, the remote server intercepts the monitoring data.
By adopting the technical scheme: the data uniform storage and classification module comprises a data packet corresponding division submodule and a multi-classification extraction and call-in submodule, wherein the data packet corresponding division submodule is used for dividing data received inside the remote server into different data packets, the cached data volume of each data packet is the same, the multi-classification extraction and call-in submodule is used for performing fixed-point sampling on a plurality of data in each data packet, different data classes are formed according to sampling data, and the counted data classes are sent to a plurality of sampling point extraction modules of the data.
By adopting the technical scheme: the data multiple sampling point extraction module comprises a target data difference point analysis and confirmation submodule and a different classification data extraction submodule, the target data difference point analysis and confirmation submodule is used for extracting data in a data packet, analyzing different classification ratios in the extracted data, confirming the classification of the data according to the different classification ratios, the different classification data extraction submodule is used for confirming the classification ratio of each section of data in the data packet confirmed by the target data difference point analysis and confirmation submodule, confirming the classification of each section of data, subdividing the data packet of each section of data according to different classifications, and sending the subdivided data packet to the multi-classification extraction and calling submodule for summarizing.
By adopting the technical scheme: the block chain platform comprises an effective data real-time backup submodule and a data query access submodule, wherein the effective data real-time backup submodule is used for carrying out real-time backup on effective data stored in the data uniform storage and classification module, and the data query access submodule is used for accessing the block chain platform at a third-party client side to query data information in the system.
A high-speed data processing method of a remote server based on a block chain comprises the following steps:
s1: the data streaming acquisition and buffering module is used for establishing a buffering and shunting area for data sent to a remote server;
s2: the invalid data real-time interception module is used for intercepting invalid data sent to a remote server;
s3: the data unified storage and classification module is used for classifying and classifying the screened data and then storing the classified and classified data;
s4: the data extraction modules are used for collecting partial data of the data and classifying the data according to the collection result;
s5: and coordinating all modules by using the block chain platform, and accessing a third-party client to perform data query.
By adopting the technical scheme: the processing method further comprises the following steps:
s1-1, establishing a plurality of buffer areas when the remote server receives data by using a plurality of buffer area establishing sub-modules, guiding a large amount of data into the plurality of buffer areas for buffering, and guiding the data of the plurality of buffer areas into a sorting sub-module for sorting and then guiding the data into the remote server for data processing;
s2-1: reading data sent to a remote server by using a target repeated data reading monitoring submodule, monitoring whether the sent data has repeated data or not, marking the monitored repeated data and target comparison data, sending a marking result to a data segmentation repetition rate comparison and analysis submodule, segmenting the monitored repeated data by using the data segmentation repetition rate comparison and analysis submodule, carrying out repetition rate analysis on the segmented repeated data and the target comparison data, detecting an IP address of a data sending device by using an invalid IP address interception submodule, monitoring the validity of the IP address, and when the current IP address is monitored to belong to the invalid IP address, completely intercepting the data sent by the device;
s3-1: the data packet corresponding division submodule is used for dividing different data packets of data received in the remote server, the cached data volume of each data packet is the same, the multi-classification extraction calling submodule carries out fixed point sampling on a plurality of data in each data packet, different data categories are formed according to the sampled data, and the counted data categories are sent to a plurality of sampling point extraction modules of the data;
s4-1: extracting data in the data packet by using a target data difference point analysis and confirmation submodule, analyzing different class ratios in the extracted data, confirming the class of the data according to the different class ratios, determining the class ratio of each segment of data in the data packet confirmed by the target data difference point analysis and confirmation submodule by using a different class data extraction submodule, confirming the class of each segment of data, re-dividing each segment of data into data packets according to different classes, and sending the divided data packets to a multi-class extraction and call-in submodule for gathering;
s5-1: and the data query access sub-module accesses the block chain platform at a third-party client to query the data information in the system.
By adopting the technical scheme: in the step S4-1, the target data difference point analyzing and confirming submodule is used to extract data inside the data packet, different category ratios in the extracted data are analyzed, categories of the data are confirmed according to the different category ratios, the different classification data extracting submodule confirms the category ratio of each data in the data packet confirmed by the target data difference point analyzing and confirming submodule to confirm the category of each data, each data is re-divided into data packets according to different categories, and the divided data packets are sent to the multi-classification extraction call-in submodule for gathering, which further includes the following steps:
the data categories collected by the multi-classification extraction calling sub-module comprise: the data processing method comprises the following steps that text data, image data, voice data, video data and comprehensive data are judged, wherein the current data is the comprehensive data if one section of data comprises the text data, the image data, the voice data and the video data;
setting a target data difference point analysis and confirmation submodule to extract data A in a data packet, extracting different types of data in the data A, setting data with three different types in the current data A, wherein the data with three different types respectively account for D1%, D2% and D3% in the data A, judging that the type of the current data A is the data type with the accounting for D1% when D1% is more than or equal to 60%, and judging that the current data A is comprehensive data when D1% is less than 60%, D2% is less than 60% and D3% is less than 60%;
the data with two different types in the current data A are set, the proportion of the data in the data A respectively comprises D1% and D2%, when D1% is greater than D2%, the type of the current data A is judged to be the data type with the proportion of D1%, and when D1% is D2%, the current data A is judged to be comprehensive data.
Example 1: and under the limited condition, the data segmentation repetition rate comparison and analysis submodule performs repetition rate analysis on the monitored repeated data and the target comparison data, the data segmentation repetition rate comparison and analysis submodule is set to segment the monitored repeated data, and the segmented repeated data is set to be L1、L2、L3、L4Setting the current target contrast data to L0, setting the repetition rate of each segment to 0.2, 0.37, 0.45, 0.51, according to the formula:
the formula I is as follows:
the formula II is as follows: 0.51 x 100 percent, 51 percent to 55 percent
And the monitored repeated data meets the first formula and the second formula, the repetition rate of the current monitored data and the target comparison data is judged not to meet the interception requirement, and the remote server continues to process the monitored data.
Example 2: and under the limited condition, the data segmentation repetition rate comparison and analysis submodule performs repetition rate analysis on the monitored repeated data and the target comparison data, the data segmentation repetition rate comparison and analysis submodule is set to segment the monitored repeated data, and the segmented repeated data is set to be L1、L2、L3、L4Is set whenThe pre-target contrast data is L0, and the repetition rate of each segment is set to 0.11, 0.64, 0.31 according to the formula:
the formula I is as follows:
the formula II is as follows: 0.64 x 100% > -64% > 55%
The monitored repeated data meets the formula I, 55% < 0.64 × 100% > -64% < 65% >, and after the segmented data is marked by the data segment repetition rate comparison and analysis submodule, the remote server continues to process the monitored data.
Example 3: and defining conditions, setting a target data difference point analysis and confirmation submodule to extract data A in the data packet, extracting different types of data in the data A, setting three types of data with different types in the current data A, wherein the three types of data comprise text type data, image type data and video type data, the percentage of the three types of data in the data A comprises 20%, 18% and 62%, 62% is more than or equal to 60%, and judging that the type of the current data A is video type data.
Example 4: and defining conditions, setting a target data difference point analysis and confirmation submodule to extract data A in the data packet, extracting different types of data in the data A, setting two types of data with different types in the current data A, wherein the two types of data comprise text type data and voice type data, the percentage of the two types of data in the data A respectively comprises 47 percent, 53 percent and 53 percent > 47 percent, and judging that the type of the current data A is the voice type data.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (10)
1. A remote server is to data high-speed processing system based on block chain, characterized by that: the system comprises a data shunting and collecting buffer module, an invalid data real-time intercepting module, a data uniform storage and classification module, a plurality of data sampling point extracting modules and a block chain platform, wherein the data shunting and collecting buffer module, the invalid data real-time intercepting module, the data uniform storage and classification module and the plurality of data sampling point extracting modules are sequentially connected through an intranet;
the data distributing and streaming collection buffer module is used for establishing a buffer and a distribution area for data sent to a remote server, the invalid data real-time interception module is used for intercepting invalid data sent to the remote server, the data unified storage and classification module is used for storing screened data after classifying and dividing, the data sampling point extraction modules are used for collecting partial data of the data, classifying and dividing the data according to a collection result, and the block chain platform is used for coordinating all the modules and accessing a third-party client to inquire the data.
2. The system for processing data at high speed by the remote server based on the block chain as claimed in claim 1, wherein: the data distribution flow acquisition and buffering module comprises a plurality of buffer area establishing sub-modules and a data distribution flow introduction and arrangement sub-module, the buffer area establishing sub-modules are used for establishing a plurality of buffer areas when the remote server receives data, a large amount of data is guided to enter the buffer areas for buffering, and the data distribution flow introduction and arrangement sub-module is used for arranging the data of the buffer areas and then distributing the data to the remote server for data processing.
3. The system for processing data at high speed by the remote server based on the block chain as claimed in claim 1, wherein: the invalid data real-time intercepting module comprises a target repeated data reading and monitoring submodule, a data segment repetition rate comparison and analysis submodule and an invalid IP address intercepting submodule, wherein the target repeated data reading and monitoring submodule is used for reading data sent to a remote server, monitoring whether the sent data has repeated data or not, marking the monitored repeated data and the target compared data, sending a marking result to the data segment repetition rate comparison and analysis submodule, the data segment repetition rate comparison and analysis submodule segments the monitored repeated data, analyzes the repetition rate of the segmented repeated data and the target compared data, and the invalid IP address intercepting submodule is used for detecting the IP address of a data sending device and monitoring the validity of the IP address, and when the current IP address is monitored to belong to the invalid IP address, all data sent by the equipment are intercepted.
4. A block chain based remote server to data high speed processing system according to claim 3, characterized in that: the data segment repetition rate comparison and analysis submodule performs repetition rate analysis on the monitored repeated data and the target comparison data, the data segment repetition rate comparison and analysis submodule is set to segment the monitored repeated data, and the repeated data segment is set to be L1、L2、L3、…、Ln-1、LnSetting the current target contrast data to L0, and setting the repetition rate of each segment to L0
According to the formula:
the formula I is as follows:
the formula II is as follows:
when the monitored repeated data meets the first formula and the second formula, judging that the repetition rate of the current monitored data and the target comparison data does not meet the requirementIntercepting the requirement, the remote server continuously processes the monitoring data, and when the monitored repeated data meets the formula I, the monitored repeated data meets the formula I
And after the data segment repetition rate comparison analysis submodule marks the segment data, the remote server continues to process the monitoring data, and when the monitored repeated data does not satisfy the formula I and the formula II, the remote server intercepts the monitoring data.
5. The system for processing data at high speed by the remote server based on the block chain as claimed in claim 1, wherein: the data uniform storage and classification module comprises a data packet corresponding division submodule and a multi-classification extraction and call-in submodule, wherein the data packet corresponding division submodule is used for dividing data received inside the remote server into different data packets, the cached data volume of each data packet is the same, the multi-classification extraction and call-in submodule is used for performing fixed-point sampling on a plurality of data in each data packet, different data classes are formed according to sampling data, and the counted data classes are sent to a plurality of sampling point extraction modules of the data.
6. The system for processing data at high speed by the remote server based on the block chain as claimed in claim 5, wherein: the data multiple sampling point extraction module comprises a target data difference point analysis and confirmation submodule and a different classification data extraction submodule, the target data difference point analysis and confirmation submodule is used for extracting data in a data packet, analyzing different classification ratios in the extracted data, confirming the classification of the data according to the different classification ratios, the different classification data extraction submodule is used for confirming the classification ratio of each section of data in the data packet confirmed by the target data difference point analysis and confirmation submodule, confirming the classification of each section of data, subdividing the data packet of each section of data according to different classifications, and sending the subdivided data packet to the multi-classification extraction and calling submodule for summarizing.
7. The system for processing data at high speed by the remote server based on the block chain as claimed in claim 1, wherein: the block chain platform comprises an effective data real-time backup submodule and a data query access submodule, wherein the effective data real-time backup submodule is used for carrying out real-time backup on effective data stored in the data uniform storage and classification module, and the data query access submodule is used for accessing the block chain platform at a third-party client side to query data information in the system.
8. A high-speed data processing method of a remote server based on a block chain is characterized in that:
s1: the data streaming acquisition and buffering module is used for establishing a buffering and shunting area for data sent to a remote server;
s2: the invalid data real-time interception module is used for intercepting invalid data sent to a remote server;
s3: the data unified storage and classification module is used for classifying and classifying the screened data and then storing the classified and classified data;
s4: the data extraction modules are used for collecting partial data of the data and classifying the data according to the collection result;
s5: and coordinating all modules by using the block chain platform, and accessing a third-party client to perform data query.
9. The method for processing data at high speed by the remote server based on the blockchain as claimed in claim 8, wherein: the processing method further comprises the following steps:
s1-1, establishing a plurality of buffer areas when the remote server receives data by using a plurality of buffer area establishing sub-modules, guiding a large amount of data into the plurality of buffer areas for buffering, and guiding the data of the plurality of buffer areas into a sorting sub-module for sorting and then guiding the data into the remote server for data processing;
s2-1: reading data sent to a remote server by using a target repeated data reading monitoring submodule, monitoring whether the sent data has repeated data or not, marking the monitored repeated data and target comparison data, sending a marking result to a data segmentation repetition rate comparison and analysis submodule, segmenting the monitored repeated data by using the data segmentation repetition rate comparison and analysis submodule, carrying out repetition rate analysis on the segmented repeated data and the target comparison data, detecting an IP address of a data sending device by using an invalid IP address interception submodule, monitoring the validity of the IP address, and when the current IP address is monitored to belong to the invalid IP address, completely intercepting the data sent by the device;
s3-1: the data packet corresponding division submodule is used for dividing different data packets of data received in the remote server, the cached data volume of each data packet is the same, the multi-classification extraction calling submodule carries out fixed point sampling on a plurality of data in each data packet, different data categories are formed according to the sampled data, and the counted data categories are sent to a plurality of sampling point extraction modules of the data;
s4-1: extracting data in the data packet by using a target data difference point analysis and confirmation submodule, analyzing different class ratios in the extracted data, confirming the class of the data according to the different class ratios, determining the class ratio of each segment of data in the data packet confirmed by the target data difference point analysis and confirmation submodule by using a different class data extraction submodule, confirming the class of each segment of data, re-dividing each segment of data into data packets according to different classes, and sending the divided data packets to a multi-class extraction and call-in submodule for gathering;
s5-1: and the data query access sub-module accesses the block chain platform at a third-party client to query the data information in the system.
10. The method for processing data at high speed by the remote server based on the blockchain according to claim 9, wherein the method comprises the following steps: in the step S4-1, the target data difference point analyzing and confirming submodule is used to extract data inside the data packet, different category ratios in the extracted data are analyzed, categories of the data are confirmed according to the different category ratios, the different classification data extracting submodule confirms the category ratio of each data in the data packet confirmed by the target data difference point analyzing and confirming submodule to confirm the category of each data, each data is re-divided into data packets according to different categories, and the divided data packets are sent to the multi-classification extraction call-in submodule for gathering, which further includes the following steps:
the data categories collected by the multi-classification extraction calling sub-module comprise: the data processing method comprises the following steps that text data, image data, voice data, video data and comprehensive data are judged, wherein the current data is the comprehensive data if one section of data comprises the text data, the image data, the voice data and the video data;
setting a target data difference point analysis and confirmation submodule to extract data A in a data packet, extracting different types of data in the data A, setting data with three different types in the current data A, wherein the data with three different types respectively account for D1%, D2% and D3% in the data A, judging that the type of the current data A is the data type with the accounting for D1% when D1% is more than or equal to 60%, and judging that the current data A is comprehensive data when D1% is less than 60%, D2% is less than 60% and D3% is less than 60%;
the data with two different types in the current data A are set, the proportion of the data in the data A respectively comprises D1% and D2%, when D1% is greater than D2%, the type of the current data A is judged to be the data type with the proportion of D1%, and when D1% is D2%, the current data A is judged to be comprehensive data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010506120.4A CN111614786A (en) | 2020-06-05 | 2020-06-05 | System and method for processing data at high speed by remote server based on block chain |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010506120.4A CN111614786A (en) | 2020-06-05 | 2020-06-05 | System and method for processing data at high speed by remote server based on block chain |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111614786A true CN111614786A (en) | 2020-09-01 |
Family
ID=72197076
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010506120.4A Withdrawn CN111614786A (en) | 2020-06-05 | 2020-06-05 | System and method for processing data at high speed by remote server based on block chain |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111614786A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116466883A (en) * | 2023-04-12 | 2023-07-21 | 上海威固信息技术股份有限公司 | Distributed multi-source information acquisition and storage system and storage method thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070033402A1 (en) * | 2005-08-05 | 2007-02-08 | Williams Robert J | System and method for pre-loading personal media device content |
| CN106484857A (en) * | 2016-10-09 | 2017-03-08 | 珠海经济特区远宏科技有限公司大连分公司 | Data collecting system and its method |
| CN109639592A (en) * | 2018-12-11 | 2019-04-16 | 武汉奥浦信息技术有限公司 | A kind of rapid data analysis method and device based on ten thousand megastream amounts |
| CN110046264A (en) * | 2019-04-02 | 2019-07-23 | 云南大学 | A kind of automatic classification method towards mobile phone document |
| CN111061790A (en) * | 2019-12-13 | 2020-04-24 | 江苏智谋科技有限公司 | Information acquisition system and method for customer data management |
-
2020
- 2020-06-05 CN CN202010506120.4A patent/CN111614786A/en not_active Withdrawn
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070033402A1 (en) * | 2005-08-05 | 2007-02-08 | Williams Robert J | System and method for pre-loading personal media device content |
| CN106484857A (en) * | 2016-10-09 | 2017-03-08 | 珠海经济特区远宏科技有限公司大连分公司 | Data collecting system and its method |
| CN109639592A (en) * | 2018-12-11 | 2019-04-16 | 武汉奥浦信息技术有限公司 | A kind of rapid data analysis method and device based on ten thousand megastream amounts |
| CN110046264A (en) * | 2019-04-02 | 2019-07-23 | 云南大学 | A kind of automatic classification method towards mobile phone document |
| CN111061790A (en) * | 2019-12-13 | 2020-04-24 | 江苏智谋科技有限公司 | Information acquisition system and method for customer data management |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116466883A (en) * | 2023-04-12 | 2023-07-21 | 上海威固信息技术股份有限公司 | Distributed multi-source information acquisition and storage system and storage method thereof |
| CN116466883B (en) * | 2023-04-12 | 2024-01-30 | 上海威固信息技术股份有限公司 | Distributed multi-source information acquisition and storage system and storage method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106021508A (en) | Sudden event emergency information mining method based on social media | |
| CN108199863B (en) | Network traffic classification method and system based on two-stage sequence feature learning | |
| CN102420723A (en) | Anomaly detection method for various kinds of intrusion | |
| EP3048539A1 (en) | Method and apparatus for recognizing junk messages | |
| CN112422531A (en) | CNN and XGboost-based network traffic abnormal behavior detection method | |
| CN112019500B (en) | Encrypted traffic identification method based on deep learning and electronic device | |
| CN112528277A (en) | Hybrid intrusion detection method based on recurrent neural network | |
| CN114915575B (en) | Network flow detection device based on artificial intelligence | |
| CN106649338B (en) | Information filtering strategy generation method and device | |
| CN112884121A (en) | Traffic identification method based on generation of confrontation deep convolutional network | |
| CN113128626A (en) | Multimedia stream fine classification method based on one-dimensional convolutional neural network model | |
| CN111614786A (en) | System and method for processing data at high speed by remote server based on block chain | |
| Barozzi et al. | Filtering images extracted from social media in the response phase of emergency events | |
| CN114650229A (en) | Network encryption traffic classification method and system based on three-layer model SFTF-L | |
| CN116232696A (en) | Encryption traffic classification method based on deep neural network | |
| CN114970680A (en) | CNN + LSTM-based flow terminal real-time identification method and device | |
| CN117914599A (en) | Mobile network malicious traffic identification method based on graph neural network | |
| CN117650935A (en) | Interference flow identification method based on service application classification model | |
| CN112910606B (en) | Data sending system adopting sectional type data structure | |
| CN112468509A (en) | Deep learning technology-based automatic flow data detection method and device | |
| CN114884894B (en) | Semi-supervised network traffic classification method based on transfer learning | |
| CN113746707B (en) | Encrypted traffic classification method based on classifier and network structure | |
| CN113037551B (en) | Quick identification and positioning method for sensitive-related services based on traffic slice | |
| CN113919493A (en) | Fraud behavior analysis system based on neural network model | |
| CN113886704A (en) | Information generation and aggregation system based on network viewpoint |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200901 |
|
| WW01 | Invention patent application withdrawn after publication |