CN116466883A - Distributed multi-source information acquisition and storage system and storage method thereof - Google Patents
Distributed multi-source information acquisition and storage system and storage method thereof Download PDFInfo
- Publication number
- CN116466883A CN116466883A CN202310390367.8A CN202310390367A CN116466883A CN 116466883 A CN116466883 A CN 116466883A CN 202310390367 A CN202310390367 A CN 202310390367A CN 116466883 A CN116466883 A CN 116466883A
- Authority
- CN
- China
- Prior art keywords
- data
- data content
- module
- screening
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 72
- 238000013523 data management Methods 0.000 claims abstract description 62
- 238000013500 data storage Methods 0.000 claims abstract description 27
- 238000012544 monitoring process Methods 0.000 claims abstract description 10
- 238000012216 screening Methods 0.000 claims description 109
- 230000005540 biological transmission Effects 0.000 claims description 27
- 238000011156 evaluation Methods 0.000 claims description 24
- 230000000694 effects Effects 0.000 claims description 21
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000012217 deletion Methods 0.000 claims description 4
- 230000037430 deletion Effects 0.000 claims description 4
- 230000006855 networking Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0653—Monitoring storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data storage, in particular to a distributed multi-source information acquisition and storage system and a storage method thereof, wherein the distributed multi-source information acquisition and storage system comprises a data management layer, a data processing layer and a monitoring layer; the data required to be stored by a user is uploaded through the data management layer and is temporarily stored in the data management layer, the data is sent to the data processing layer in a set period, the data content of the source data management layer is differentiated and stored through the data processing layer, and the data content is differentiated and stored according to the differentiating result.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a distributed multi-source information acquisition and storage system and a storage method thereof.
Background
The data storage object includes temporary files generated during processing of the data stream or information to be searched during processing. Data is recorded in a certain format on a storage medium inside or outside a computer. The data store is named, which reflects the constitutive meaning of the information features. The data flow reflects the data flowing in the system and shows the characteristics of dynamic data; the data store reflects data that is stationary in the system and features static data.
However, in the current data storage technology, centralized data storage is often used as the main technology, the data stored by the user is only stored in a distinguishing way by constructing folders, the data storage system does not have a function of autonomously distinguishing and storing the data, and when the data storage amount is large, the operation of searching the required data in the stored data is difficult.
Disclosure of Invention
Technical problem to be solved
The invention provides a distributed multi-source information acquisition and storage system and a storage method thereof, aiming at the defects existing in the prior art, solving the problems that the prior data storage technology is mainly concentrated in centralized data storage, data stored by a user are differentiated and stored only by constructing folders, the data storage system does not have the function of autonomously differentiating and storing the processing of the data, and the operation of searching the needed data in the stored data is difficult when the data storage capacity is large.
Technical proposal
In order to achieve the above purpose, the invention is realized by the following technical scheme:
in a first aspect, a distributed multi-source information acquisition and storage system includes a data management layer, a data processing layer, and a monitoring layer;
uploading data required to be stored by a user through a data management layer, temporarily storing the data in the data management layer, sending the data to a data processing layer in a set period, distinguishing the data content of a source data management layer through the data processing layer, distinguishing and storing the data content according to distinguishing results, and intervening a monitoring layer into the data processing layer to monitor and analyze the activity of each data content in real time by distinguishing the stored data in the data processing layer;
the data processing layer comprises a transmission module, a screening module and a storage module, wherein the transmission module is used for receiving the data content sent by the data management layer and transmitting the data content to the screening module, the screening module is used for setting data content screening conditions, screening the data content by applying the screening conditions, and the storage module is used for receiving the screened data content and storing the data content in a distinguishing way according to the data content screening result;
when the screening module in the data processing layer operates, the attribute of the data content is synchronously judged through the following formula, the screening is used for setting when the screening condition of the data content is set, the attribute of the data content is applied, and the calculation formula is as follows:
wherein: n (N) select Determining attributes of the data content; a is the size of image data present in the data content; b is the size of character data existing in the data content; c, replacing audio-visual data existing in the data content; k is the size of the data content; best (a，b，c) The attribute data with the largest proportion in the data content composed of all attribute data;
the data attribute of each group of data content transmitted to the screening module by the transmission module is obtained through the formula, and when the screening module screens the data content, the screening module applies the corresponding attribute of each data content to carry out screening processing.
Further, the data management layer comprises a receiving module and a setting module, the receiving module is used for receiving the data content which is required to be stored and is uploaded in the data management layer by a user through the electronic equipment, and the setting module is used for setting the sending period of the data content received by the receiving module to the data processing layer;
the setting module is further used for acquiring the distinguishing processing state of the received data content by the data processing layer in real time, wherein the state comprises the operation of sending the data content to the data processing layer by the data management layer, the sending period set in the setting module is set through manual editing of a user at a system end, the setting range of the sending period is 2-20 s, the state comprises the distinguishing processing in progress and the distinguishing processing ending, the data processing layer preferentially follows the distinguishing processing state of the received data content acquired by the setting module in real time, the sending period is followed when the state is the distinguishing processing ending, the distinguishing processing of the received data content is waited for when the state is the distinguishing processing in progress, and the sending period is followed after the distinguishing processing ending.
Further, after the data content uploaded by the user temporarily stored in the data management layer is sent to the data processing layer through the setting period, the corresponding data content is deleted in the data management layer synchronously, a system end user in the data management layer manually sets the resetting period, and the data management layer autonomously completes the resetting operation according to the resetting period.
Further, when the data management layer sends the data content to the data processing layer, the system is in an offline state, and when the receiving module in the data management layer receives the data content uploaded by the user, the system is in a networking state, the data management layer collects data in a system connection network to perform network security evaluation, and the evaluation result is expressed by the following formula:
wherein: p (P) avg The network security evaluation coefficient is used; p (P) i The probability of the vulnerability in the system connection network being successfully utilized in various attack paths possibly existing is high; n is the number of paths that the vulnerability can attack; i is an attack path;
wherein n and i are data management layers for collecting data in a system connection network, P avg The smaller the value of the (C) is, the more safe the current network tends to, and when the network security evaluation is carried out, the network security evaluation is carried out by the user-defined editing and setting of the evaluation threshold value of the system terminal.
Furthermore, the transmission module is set with a rated transmission amount of the data content by a system end user, continuously receives the data content sent by the data management layer, and transmits the data content sent by the data management layer to the screening module when the received data content reaches the set rated transmission amount of the data content.
Further, the screening module is composed of a plurality of screening layers, each group of screening layers is provided with data content screening conditions, the data content acquired by the screening module is sequentially transmitted to each group of screening layers, and when the data content passes through each group of screening layers, the data content corresponding to the data content screening conditions of the screening layers is marked by the screening conditions;
wherein, the corresponding marks of each data content are not less than two groups, the data content screening conditions are set and modified by manual editing of a system end user, and the screening layer and the corresponding data content screening conditions are initially set as follows:
a first layer: image data; a second layer: character data; third layer: video and audio data; fourth layer: data content less than 10M; fifth layer: data content of more than or equal to 10M and less than 100M; sixth layer: data content of 100M or more.
Furthermore, the storage module is provided with a data storage interval with the quantity which is matched with the screening layer and the data content screening condition corresponding to the screening layer in an initial setting mode, and the storage module stores the data content through the storage interval arranged in the storage module;
the data storage interval set in the storage module is initially set as follows:
a first storage section: storing less than 10M of image data; a second storage section: storing character data smaller than 10M; a third storage section: storing video and audio data smaller than 10M; fourth storage section: storing image data of 10M or more and less than 100M; fifth storage section: storing character data with the size of more than or equal to 10M and less than 100M; sixth storage section: storing more than or equal to 10M and less than 100M of video and audio data; seventh storage section: storing 100M or more of image data; eighth storage section: storing character data of 100M or more; a ninth storage section: storing the video and audio data with the size of more than or equal to 100M.
Further, the monitoring layer comprises a recording module and a reducing module, wherein the recording module is used for recording the reading times of the data content stored in the storage module by a user, the reducing module is used for analyzing the activity of the data content stored in the storage module, and deleting and reducing the data content stored in the storage module through the activity of the data content;
when a user reads the data content stored in the storage module, the user searches the data content through the corresponding mark of the data content, the activity evaluation threshold of the data content is manually edited and set by the user at the system end in the abatement module, the abatement module performs deletion abatement operation on the data content stored in the storage module, the activity evaluation threshold of the data content is applied to delete the abatement data content, the activity of the data content is the ratio of the number of times the data content is read by the user to the time when the data content exists in the storage module, and the deletion abatement target data content of the abatement module is the data content which exists in the storage module for more than six months.
Furthermore, the receiving module is electrically connected with a setting module through a medium, the setting module is electrically connected with a transmission module, a screening module and a storage module through the medium, and the storage module is electrically connected with a recording module and a reducing module through the medium.
In a second aspect, a distributed multi-source information collection and storage method includes the following steps:
step 1: receiving data content and temporarily storing the data content;
step 2: screening and distinguishing the data temporarily stored in the step 1;
step 3: constructing a data storage space, and setting a storage interval in the data storage space;
step 4: and sending the data content subjected to screening and distinguishing to each storage section of the data storage space according to the screening and distinguishing result, and completing the distinguishing storage of the data content.
Advantageous effects
Compared with the known public technology, the technical scheme provided by the invention has the following beneficial effects:
1. the invention provides a distributed multi-source information acquisition and storage system, which can identify data in a data receiving stage in the operation process, so that the data content is further distinguished according to the data attribute identification result of the data content, and the distinguished data content is further distinguished, configured and stored in a mode of setting a storage interval in a storage space, so that the storage management of the data content is realized, the fault tolerance of the data content during storage is effectively improved, and convenience is brought to a certain extent when a subsequent user reads required data in the storage space.
2. In the invention, when the system is operated, the network state of the system when receiving the data content can be detected in real time, the process of data transmission is ensured to be safer, the risk of stealing the transmitted data content is avoided, in addition, in the later stage of managing the data by storing the data through the system, the data content stored in the system can be adaptively deleted, on one hand, the space for storing the data content in the managed system is further saved, and on the other hand, the speed of obstructing the operation of the system by excessive zombie data in the system is avoided.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is evident that the drawings in the following description are only some embodiments of the present invention and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.
FIG. 1 is a schematic diagram of a distributed multi-source information collection and storage system;
fig. 2 is a flow chart of a distributed multi-source information collection and storage method.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It will be apparent that the described embodiments are some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention is further described below with reference to examples.
Example 1
The distributed multi-source information acquisition and storage system of the embodiment, as shown in fig. 1, comprises a data management layer, a data processing layer and a monitoring layer;
uploading data required to be stored by a user through a data management layer, temporarily storing the data in the data management layer, sending the data to a data processing layer in a set period, distinguishing the data content of a source data management layer through the data processing layer, distinguishing and storing the data content according to distinguishing results, and intervening a monitoring layer into the data processing layer to monitor and analyze the activity of each data content in real time by distinguishing the stored data in the data processing layer;
the data processing layer comprises a transmission module, a screening module and a storage module, wherein the transmission module is used for receiving the data content sent by the data management layer and transmitting the data content to the screening module, the screening module is used for setting data content screening conditions, screening the data content by applying the screening conditions, and the storage module is used for receiving the screened data content and distinguishing and storing the data content according to the data content screening result;
when a screening module in the data processing layer operates, the attribute of the data content is synchronously judged through the following formula, the screening is used for setting when setting the screening condition of the data content, the attribute of the data content is applied, and the calculation formula is as follows:
wherein: n (N) select Determining attributes of the data content; a is a graph existing in data contentImage data size; b is the size of character data existing in the data content; c, replacing audio-visual data existing in the data content; k is the size of the data content; best (a，b，c) The attribute data with the largest proportion in the data content composed of all attribute data;
each group of data content transmitted to the screening module by the transmission module is obtained by the formula, and when the screening module screens the data content, the screening module applies the corresponding attribute of each data content to carry out screening treatment;
the data management layer comprises a receiving module and a setting module, wherein the receiving module is used for receiving data contents which are uploaded in the data management layer by a user through the electronic equipment and are required to be stored, and the setting module is used for setting a sending period of the data contents received by the receiving module to the data processing layer;
the sending period set in the setting module is set through manual editing of a user at a system end, the setting range of the sending period is 2 s-20 s, the setting module is further used for acquiring distinguishing processing states of the received data content of the data processing layer in real time, the states comprise the proceeding of distinguishing processing and the ending of distinguishing processing, the data management layer sends the data content to the data processing layer, the operation of the data processing layer in real time acquired by the setting module is preferentially followed, the sending period is followed when the state is the distinguishing processing ending, the distinguishing processing of the received data content of the data processing layer is waited when the state is the distinguishing processing, and the sending period is followed after the distinguishing processing is ended;
the monitoring layer comprises a recording module and a reducing module, wherein the recording module is used for recording the reading times of the data content stored in the storage module by a user, the reducing module is used for analyzing the activity of the data content stored in the storage module, and deleting and reducing the data content stored in the storage module through the activity of the data content;
when a user reads the data content stored in the storage module, searching the data content through a corresponding mark of the data content, manually editing and setting an activity evaluation threshold of the data content by a system end user in the abatement module, and selecting an activity evaluation threshold of the data content for deleting and abatement operation of the data content stored in the storage module by the abatement module, wherein the activity of the data content is the ratio of the number of times the data content is read by the user to the time when the data content exists in the storage module, and the deleting and abatement target data content of the abatement module is the data content which exists in the storage module for more than six months;
the receiving module is electrically connected with a setting module through a medium, the setting module is electrically connected with a transmission module, a screening module and a storage module through the medium, and the storage module is electrically connected with a recording module and a reducing module through the medium.
In this embodiment, the receiving module receives the data content to be stored uploaded by the user through the electronic device in the data management layer, the setting module operates to set a sending period for sending the data content received by the receiving module to the data processing layer, the transmitting module receives the data content sent by the data management layer and transmits the data content to the screening module, the synchronous screening module sets a data content screening condition, the screening condition is applied to screen the data content, the storage module operates to receive the screened data content, the data content is stored in a distinguishing mode according to the data content screening result, finally the recording module records the reading times of the data content stored in the storage module by the user, the subtracting module analyzes the activity of the data content stored in the storage module, and the data content stored in the storage module is deleted and subtracted through the data content activity.
Example 2
On the aspect of implementation, on the basis of embodiment 1, this embodiment further specifically describes a distributed multi-source information collection and storage system in embodiment 1 with reference to fig. 1:
after the data content uploaded by the user temporarily stored in the data management layer is sent to the data processing layer through the setting period, the corresponding data content is deleted in the data management layer synchronously, a system end user in the data management layer manually sets a resetting period, and the data management layer autonomously completes resetting operation according to the resetting period.
By this arrangement, the problem of accumulating too much useless cache data due to long-term operation in the data management layer can be avoided.
As shown in fig. 1, when the data management layer sends data content to the data processing layer, the system is in an offline state, and when the receiving module in the data management layer receives data content uploaded by a user, the system is in a networking state, the data management layer collects data in a system connection network, and performs network security evaluation, and an evaluation result is represented by the following formula:
wherein: p (P) avg The network security evaluation coefficient is used; p (P) i The probability of the vulnerability in the system connection network being successfully utilized in various attack paths possibly existing is high; n is the number of paths that the vulnerability can attack; i is an attack path;
wherein n and i are data management layers for collecting data in a system connection network, P avg The smaller the value of the (C) is, the more safe the current network tends to, and when the network security evaluation is carried out, the network security evaluation is carried out by the user-defined editing and setting of the evaluation threshold value of the system terminal.
Through the formula calculation, the network security of the system application can be monitored in real time in the running process of the system, so that the user and the system end user are assisted in completing the data uploading operation in a safer environment, and the data uploading operation is ensured to be safer.
Example 3
On the aspect of implementation, on the basis of embodiment 1, this embodiment further specifically describes a distributed multi-source information collection and storage system in embodiment 1 with reference to fig. 1:
the transmission module is used for continuously receiving the data content sent by the data management layer, and transmitting the data content sent by the data management layer to the screening module when the received data content reaches the set rated data content.
The rated transmission amount of the data content is the data content capacity of each transmission when the transmission module receives the data content and transmits the data content to the screening module.
As shown in fig. 1, the screening module is composed of a plurality of screening layers, each set of screening layers is provided with a data content screening condition, the data content acquired by the screening module is sequentially transmitted to each set of screening layers, and when the data content passes through each set of screening layers, the data content corresponding to the data content screening condition of the screening layer is marked by the screening condition;
wherein, the corresponding marks of each data content are not less than two groups, the data content screening conditions are set and modified by manual editing of a system end user, and the screening layer and the corresponding data content screening conditions are initially set as follows:
a first layer: image data; a second layer: character data; third layer: video and audio data; fourth layer: data content less than 10M; fifth layer: data content of more than or equal to 10M and less than 100M; sixth layer: data content of 100M or more.
As shown in fig. 1, a data storage section with a number adapted to the screening layer and the corresponding data content screening condition is set in the storage module, and the storage module stores the data content through the storage section set in the storage module;
the data storage interval set in the storage module is initially set as follows:
a first storage section: storing less than 10M of image data; a second storage section: storing character data smaller than 10M; a third storage section: storing video and audio data smaller than 10M; fourth storage section: storing image data of 10M or more and less than 100M; fifth storage section: storing character data with the size of more than or equal to 10M and less than 100M; sixth storage section: storing more than or equal to 10M and less than 100M of video and audio data; seventh storage section: storing 100M or more of image data; eighth storage section: storing character data of 100M or more; a ninth storage section: storing the video and audio data with the size of more than or equal to 100M.
Example 4
In a specific implementation aspect, on the basis of embodiment 1, this embodiment further specifically describes a distributed multi-source information collection and storage system in embodiment 1 with reference to fig. 2:
a distributed multi-source information acquisition and storage method comprises the following steps:
step 1: receiving data content and temporarily storing the data content;
step 2: screening and distinguishing the data temporarily stored in the step 1;
step 3: constructing a data storage space, and setting a storage interval in the data storage space;
step 4: and sending the data content subjected to screening and distinguishing to each storage section of the data storage space according to the screening and distinguishing result, and completing the distinguishing storage of the data content.
In summary, through the system in the above embodiment, attribute identification can be performed on data in the data receiving stage, so that the data content is further distinguished according to the data attribute identification result of the data content, and further, the distinguished data content is further distinguished, configured and stored in a manner of setting a storage interval in the storage space, so that storage management of the data content is realized, the fault tolerance of the data content during storage is effectively improved, and convenience is brought to a certain extent when a subsequent user reads required data in the storage space; in addition, when the system is operated, the network state of the system when receiving the data content can be detected in real time, the process of data transmission is ensured to be safer, the risk of stealing the transmitted data content is avoided, in addition, in the later stage of data storage and management through the system, the data content stored in the system can be adaptively deleted, on one hand, the space for storing the data content in the managed system is further saved, and on the other hand, the speed of obstructing the operation of the system due to excessive zombie data in the system is avoided.
The above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. The distributed multi-source information acquisition and storage system is characterized by comprising a data management layer, a data processing layer and a monitoring layer;
uploading data required to be stored by a user through a data management layer, temporarily storing the data in the data management layer, sending the data to a data processing layer in a set period, distinguishing the data content of a source data management layer through the data processing layer, distinguishing and storing the data content according to distinguishing results, and intervening a monitoring layer into the data processing layer to monitor and analyze the activity of each data content in real time by distinguishing the stored data in the data processing layer;
the data processing layer comprises a transmission module, a screening module and a storage module, wherein the transmission module is used for receiving the data content sent by the data management layer and transmitting the data content to the screening module, the screening module is used for setting data content screening conditions, screening the data content by applying the screening conditions, and the storage module is used for receiving the screened data content and storing the data content in a distinguishing way according to the data content screening result;
when the screening module in the data processing layer operates, the attribute of the data content is synchronously judged through the following formula, the screening is used for setting when the screening condition of the data content is set, the attribute of the data content is applied, and the calculation formula is as follows:
wherein: n (N) select Determining attributes of the data content; a is the size of image data present in the data content; b in the data contentThe size of the character data present; c, replacing audio-visual data existing in the data content; k is the size of the data content;the attribute data with the largest proportion in the data content composed of all attribute data;
the data attribute of each group of data content transmitted to the screening module by the transmission module is obtained through the formula, and when the screening module screens the data content, the screening module applies the corresponding attribute of each data content to carry out screening processing.
2. The distributed multi-source information collection and storage system according to claim 1, wherein the data management layer comprises a receiving module and a setting module, the receiving module is used for receiving data content to be stored uploaded by a user in the data management layer through the electronic device, and the setting module is used for setting a sending period of the data content received by the receiving module to be sent to the data processing layer;
the setting module is further used for acquiring the distinguishing processing state of the received data content by the data processing layer in real time, wherein the state comprises the operation of sending the data content to the data processing layer by the data management layer, the sending period set in the setting module is set through manual editing of a user at a system end, the setting range of the sending period is 2-20 s, the state comprises the distinguishing processing in progress and the distinguishing processing ending, the data processing layer preferentially follows the distinguishing processing state of the received data content acquired by the setting module in real time, the sending period is followed when the state is the distinguishing processing ending, the distinguishing processing of the received data content is waited for when the state is the distinguishing processing in progress, and the sending period is followed after the distinguishing processing ending.
3. The distributed multi-source information collection and storage system according to claim 1, wherein after the data content uploaded by the user temporarily stored in the data management layer is sent to the data processing layer through the set period, the corresponding data content is deleted in the data management layer in synchronization, the system end user in the data management layer manually sets the reset period, and the data management layer autonomously completes the reset operation according to the reset period.
4. The distributed multi-source information collection and storage system according to claim 1, wherein when the data management layer sends data content to the data processing layer, the system is in an offline state, and when the receiving module in the data management layer receives data content uploaded by a user, the system is in a networking state, the data management layer collects data in a system connection network to perform network security evaluation, and the evaluation result is expressed by the following formula:
wherein: p (P) avg The network security evaluation coefficient is used; p (P) i The probability of the vulnerability in the system connection network being successfully utilized in various attack paths possibly existing is high; n is the number of paths that the vulnerability can attack; i is an attack path;
wherein n and i are data management layers for collecting data in a system connection network, P avg The smaller the value of the (C) is, the more safe the current network tends to, and when the network security evaluation is carried out, the network security evaluation is carried out by the user-defined editing and setting of the evaluation threshold value of the system terminal.
5. The distributed multi-source information collection and storage system according to claim 1, wherein the transmission module sets a rated transmission amount of data content through a user at a system end, the transmission module continuously receives the data content sent by the data management layer, and when the received data content reaches the set rated transmission amount of the data content, the transmission module transmits the data content sent by the data management layer received by the transmission module to the screening module.
6. The distributed multi-source information acquisition and storage system according to claim 1, wherein the screening module comprises a plurality of screening layers, each set of screening layers is provided with a data content screening condition, the data content acquired by the screening module is sequentially transmitted to each set of screening layers, and when the data content passes through each set of screening layers, the data content corresponding to the data content screening condition of the screening layer is marked by the screening condition;
wherein, the corresponding marks of each data content are not less than two groups, the data content screening conditions are set and modified by manual editing of a system end user, and the screening layer and the corresponding data content screening conditions are initially set as follows:
a first layer: image data; a second layer: character data; third layer: video and audio data; fourth layer: data content less than 10M; fifth layer: data content of more than or equal to 10M and less than 100M; sixth layer: data content of 100M or more.
7. The distributed multi-source information acquisition and storage system according to claim 1, wherein the storage module is provided with a data storage section with the number which is initially set to be suitable for the screening layer and the corresponding data content screening condition, and the storage module stores the data content through the storage section which is internally set;
the data storage interval set in the storage module is initially set as follows:
a first storage section: storing less than 10M of image data; a second storage section: storing character data smaller than 10M; a third storage section: storing video and audio data smaller than 10M; fourth storage section: storing image data of 10M or more and less than 100M; fifth storage section: storing character data with the size of more than or equal to 10M and less than 100M; sixth storage section: storing more than or equal to 10M and less than 100M of video and audio data; seventh storage section: storing 100M or more of image data; eighth storage section: storing character data of 100M or more; a ninth storage section: storing the video and audio data with the size of more than or equal to 100M.
8. The distributed multi-source information collection and storage system according to claim 1, wherein the monitoring layer comprises a recording module and a subtraction module, the recording module is used for recording the number of times the data content stored in the storage module is read by a user, the subtraction module is used for analyzing the activity of the data content stored in the storage module, and deleting and subtracting the data content stored in the storage module through the activity of the data content;
when a user reads the data content stored in the storage module, the user searches the data content through the corresponding mark of the data content, the activity evaluation threshold of the data content is manually edited and set by the user at the system end in the abatement module, the abatement module performs deletion abatement operation on the data content stored in the storage module, the activity evaluation threshold of the data content is applied to delete the abatement data content, the activity of the data content is the ratio of the number of times the data content is read by the user to the time when the data content exists in the storage module, and the deletion abatement target data content of the abatement module is the data content which exists in the storage module for more than six months.
9. The distributed multi-source information collection and storage system according to claim 1, wherein the receiving module is electrically connected with a setting module through a medium, the setting module is electrically connected with a transmission module, a screening module and a storage module through a medium, and the storage module is electrically connected with a recording module and a reduction module through a medium.
10. A method for collecting and storing distributed multi-source information, which is an implementation method for a distributed multi-source information collecting and storing system according to any one of claims 1-9, and is characterized by comprising the following steps:
step 1: receiving data content and temporarily storing the data content;
step 2: screening and distinguishing the data temporarily stored in the step 1;
step 3: constructing a data storage space, and setting a storage interval in the data storage space;
step 4: and sending the data content subjected to screening and distinguishing to each storage section of the data storage space according to the screening and distinguishing result, and completing the distinguishing storage of the data content.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310390367.8A CN116466883B (en) | 2023-04-12 | 2023-04-12 | Distributed multi-source information acquisition and storage system and storage method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310390367.8A CN116466883B (en) | 2023-04-12 | 2023-04-12 | Distributed multi-source information acquisition and storage system and storage method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116466883A true CN116466883A (en) | 2023-07-21 |
| CN116466883B CN116466883B (en) | 2024-01-30 |
Family
ID=87180117
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310390367.8A Active CN116466883B (en) | 2023-04-12 | 2023-04-12 | Distributed multi-source information acquisition and storage system and storage method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116466883B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001306435A (en) * | 2000-02-14 | 2001-11-02 | Matsushita Electric Ind Co Ltd | Data transmission system |
| KR20130019509A (en) * | 2011-08-17 | 2013-02-27 | 주식회사에어플러그 | Method for displaying section information of contents, contents reproducing apparatus and contents providing system |
| CN110941592A (en) * | 2019-11-26 | 2020-03-31 | 维沃移动通信有限公司 | Data management method and mobile terminal |
| CN111614786A (en) * | 2020-06-05 | 2020-09-01 | 易盼红 | System and method for processing data at high speed by remote server based on block chain |
| CN114912153A (en) * | 2022-05-20 | 2022-08-16 | 杭州市肿瘤医院 | Scientific research data informatization management platform based on cloud computing |
| CN115019008A (en) * | 2022-05-30 | 2022-09-06 | 深圳市鸿普森科技股份有限公司 | Intelligent 3D model design analysis service management platform |
| CN115086038A (en) * | 2022-06-16 | 2022-09-20 | 国网新疆电力有限公司信息通信公司 | Information security storage method and system based on big data |
| CN115277853A (en) * | 2022-07-04 | 2022-11-01 | 阿里巴巴（中国）有限公司 | Data processing method and device based on edge cloud |
| CN115952563A (en) * | 2023-03-10 | 2023-04-11 | 云南贝蝉物联网科技有限公司 | Data security communication system based on Internet of things |
-
2023
- 2023-04-12 CN CN202310390367.8A patent/CN116466883B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001306435A (en) * | 2000-02-14 | 2001-11-02 | Matsushita Electric Ind Co Ltd | Data transmission system |
| KR20130019509A (en) * | 2011-08-17 | 2013-02-27 | 주식회사에어플러그 | Method for displaying section information of contents, contents reproducing apparatus and contents providing system |
| CN110941592A (en) * | 2019-11-26 | 2020-03-31 | 维沃移动通信有限公司 | Data management method and mobile terminal |
| CN111614786A (en) * | 2020-06-05 | 2020-09-01 | 易盼红 | System and method for processing data at high speed by remote server based on block chain |
| CN114912153A (en) * | 2022-05-20 | 2022-08-16 | 杭州市肿瘤医院 | Scientific research data informatization management platform based on cloud computing |
| CN115019008A (en) * | 2022-05-30 | 2022-09-06 | 深圳市鸿普森科技股份有限公司 | Intelligent 3D model design analysis service management platform |
| CN115086038A (en) * | 2022-06-16 | 2022-09-20 | 国网新疆电力有限公司信息通信公司 | Information security storage method and system based on big data |
| CN115277853A (en) * | 2022-07-04 | 2022-11-01 | 阿里巴巴（中国）有限公司 | Data processing method and device based on edge cloud |
| CN115952563A (en) * | 2023-03-10 | 2023-04-11 | 云南贝蝉物联网科技有限公司 | Data security communication system based on Internet of things |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116466883B (en) | 2024-01-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114143020B (en) | Rule-based network security event association analysis method and system | |
| CN111078513B (en) | Log processing method, device, equipment, storage medium and log alarm system | |
| CN108337482A (en) | The storage method and system of monitor video | |
| CN113407886A (en) | Network crime platform identification method, system, device and computer storage medium | |
| CN108614837A (en) | File stores and the method and device of retrieval | |
| CN115757363B (en) | Multi-level management method and system for three-dimensional cadastral database | |
| CN116634106A (en) | Monitoring safety system based on enterprise campus monitoring | |
| CN116340055A (en) | Data recovery method and system for solid state disk | |
| CN116466883B (en) | Distributed multi-source information acquisition and storage system and storage method thereof | |
| CN112672086B (en) | Audio and video equipment data acquisition, analysis, early warning system | |
| CN114489477A (en) | Decentralized distributed storage method based on block chain | |
| CN113711619B (en) | Multimedia data storage method, device, equipment, storage medium and program product | |
| CN117115718B (en) | Government affair video data processing method, system and computer readable storage medium | |
| CN110858837A (en) | Network management and control method and device and electronic equipment | |
| CN112235254A (en) | Rapid identification method for Tor network bridge in high-speed backbone network | |
| CN114863364B (en) | Security detection method and system based on intelligent video monitoring | |
| CN110909380A (en) | Abnormal file access behavior monitoring method and device | |
| US11398091B1 (en) | Repairing missing frames in recorded video with machine learning | |
| CN116074543A (en) | Live broadcast data supervision system based on live broadcast interaction platform | |
| US20040093413A1 (en) | Selecting and managing time specified segments from a large continuous capture of network data | |
| CN111666453B (en) | Video management and retrieval method and device, electronic equipment and storage medium | |
| CN111966859A (en) | Video data processing method and device and readable storage medium | |
| CN113595958A (en) | Safety detection system and method for Internet of things equipment | |
| CN115858458B (en) | Financial accounting data archiving system based on data statistics | |
| CN112052110B (en) | Storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |