CN117032587B - Optical storage integrated information management system based on distributed architecture - Google Patents
Optical storage integrated information management system based on distributed architecture Download PDFInfo
- Publication number
- CN117032587B CN117032587B CN202311248667.9A CN202311248667A CN117032587B CN 117032587 B CN117032587 B CN 117032587B CN 202311248667 A CN202311248667 A CN 202311248667A CN 117032587 B CN117032587 B CN 117032587B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- transmission
- information
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 43
- 230000003287 optical effect Effects 0.000 title claims abstract description 13
- 238000007726 management method Methods 0.000 claims abstract description 66
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000011084 recovery Methods 0.000 claims abstract description 13
- 230000007246 mechanism Effects 0.000 claims abstract description 7
- 230000008569 process Effects 0.000 claims abstract description 6
- 238000012545 processing Methods 0.000 claims abstract description 4
- 230000005540 biological transmission Effects 0.000 claims description 106
- 238000011157 data evaluation Methods 0.000 claims description 31
- 230000000694 effects Effects 0.000 claims description 22
- 238000004458 analytical method Methods 0.000 claims description 18
- 238000004422 calculation algorithm Methods 0.000 claims description 18
- 238000004364 calculation method Methods 0.000 claims description 18
- 238000005516 engineering process Methods 0.000 claims description 14
- 238000007906 compression Methods 0.000 claims description 12
- 230000006835 compression Effects 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 12
- 238000013500 data storage Methods 0.000 claims description 11
- 238000013144 data compression Methods 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 9
- 230000010365 information processing Effects 0.000 claims description 8
- 238000012550 audit Methods 0.000 claims description 7
- 238000009826 distribution Methods 0.000 claims description 6
- 230000006872 improvement Effects 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 6
- 238000012423 maintenance Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 5
- 238000013461 design Methods 0.000 claims description 5
- 230000010354 integration Effects 0.000 claims description 5
- 230000002452 interceptive effect Effects 0.000 claims description 4
- 238000012986 modification Methods 0.000 claims description 4
- 230000004048 modification Effects 0.000 claims description 4
- 238000012216 screening Methods 0.000 claims description 4
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 230000005856 abnormality Effects 0.000 claims description 3
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000005520 cutting process Methods 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000003745 diagnosis Methods 0.000 claims description 3
- 238000012360 testing method Methods 0.000 claims description 3
- 238000012795 verification Methods 0.000 claims description 3
- 238000013441 quality evaluation Methods 0.000 claims 3
- 238000013523 data management Methods 0.000 abstract description 7
- 241000282414 Homo sapiens Species 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/14—Protection against unauthorised use of memory or access to memory
- G06F12/1408—Protection against unauthorised use of memory or access to memory by using cryptography
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/14—Protection against unauthorised use of memory or access to memory
- G06F12/1458—Protection against unauthorised use of memory or access to memory by checking the subject access rights
- G06F12/1483—Protection against unauthorised use of memory or access to memory by checking the subject access rights using an access-table, e.g. matrix or list
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0622—Securing storage systems in relation to access
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Human Computer Interaction (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an optical storage integrated information management system based on a distributed architecture, which particularly relates to the technical field of data management, and comprises the following components: the system adopts a distributed architecture, has high availability and fault tolerance, can automatically switch to other available nodes when a certain node fails, ensures stable operation of the system, can horizontally expand as required, namely, adds more nodes to process more data and user requests, improves the performance and throughput of the system, meets the requirement of large-scale data processing by fully utilizing distributed computing and storage resources, adopts a multi-level data backup and disaster recovery mechanism, ensures the reliability and safety of the data, and prevents data loss and system failure.
Description
Technical Field
The invention relates to the technical field of data management, in particular to an optical storage integrated information management system based on a distributed architecture.
Background
The existing information management system can intensively manage and store a large amount of data, is convenient for organizing and searching information, can improve the working efficiency, reduce manual operation and errors, can update data in real time, can realize data sharing among multiple users, promotes collaboration and communication, can reduce the use and storage of paper files, saves space and resources, and brings convenience to the life of human beings;
however, the above system still has a certain problem in the use process, the existing information management system faces security threats such as data leakage, hacking or unauthorized access under certain conditions, and may have problems of slow operation, long response time and the like, which affect the working efficiency of users, and the existing information management system lacks flexibility, cannot adapt to specific requirements of different organizations or users, cannot be effectively integrated with other systems, and causes data exchange difficulty, and lacks a data analysis function, cannot provide deep analysis and insight on data, and has poor expandability, high single-point fault risk and a plurality of inconveniences when performing user information data management.
Disclosure of Invention
In order to overcome the defects of the prior art, the optical storage integrated information management system based on the distributed architecture disclosed by the invention performs distributed storage, encryption transmission and authority setting on information data uploaded by a user, calculates the data quality, transmission effect index, data security index and comprehensive quality index of the user information data, screens out data information and timestamp information with unqualified data comprehensive quality index, and manages and repairs the system to solve the problems in the background technology.
In order to achieve the above purpose, the present invention provides the following technical solutions: comprising the following steps: the system comprises a data storage module, a data security module, a data statistics module, a data calculation module, a data evaluation module, a system management module and a user interface module.
And a data storage module: the system is used for storing data information uploaded by a user, realizing the operations of slicing, de-duplication and compression of data by adopting a distributed storage mode, and transmitting the stored data to a data security module;
and a data security module: the data statistics module is used for protecting the safety and authority control of the data by adopting a data encryption algorithm and an access control list method, including encryption and audit operation of the data, and transmitting the data subjected to the safety encryption to the data statistics module;
and a data statistics module: the data computing module is used for computing the basic state of the transmission data according to the encrypted transmission data, and particularly comprises the transmission state of information data, the data security state and the data state of the transmission completion, and transmitting the counted data information to the data computing module;
and a data calculation module: the data quality, the transmission effect index, the data safety index and the comprehensive quality index of the transmission data are calculated according to the basic state of the acquired user data information, and the comprehensive quality index is transmitted to the data evaluation module;
and a data evaluation module: the system comprises a system management module, a data evaluation index, a data node information processing module and a data transmission status information processing module, wherein the system management module is used for comparing and judging the data evaluation index with a preset value, evaluating the comprehensive quality of data, screening out data node information with the comprehensive quality lower than the preset value and feeding back unqualified data transmission status information to the system management module;
and a system management module: the system comprises a data evaluation module, a data transmission state information acquisition module, a data transmission state information processing module and a data transmission state information processing module, wherein the data transmission state information acquisition module is used for receiving data transmission state information issued by the data evaluation module, and utilizes an automatic management and monitoring technology to manage configuration and maintenance of a system, including node management, task scheduling and fault recovery operation;
a user interface module: the method is used for providing a user interface, and a user completes data uploading, inquiring and analyzing operations through an intuitive graphical interface and interactive design.
Preferably, the data storage module is configured to store data information uploaded by a user, and implement operations of slicing, deduplication, and compression of data in a distributed storage manner, where the data storage module specifically includes:
data slicing unit: slicing the data uploaded by the user according to the rule of the file size, and cutting the large file into a plurality of smaller data blocks;
data deduplication unit: performing de-duplication operation on each data block by using a hash algorithm, performing hash calculation on the data block by using MD5 or SHA-1, taking a hash value as a unique identifier, checking whether the same data block exists, storing one copy when the same data block exists, and recording the reference count;
data compression algorithm: compressing the data block by using an LZ77 algorithm;
a storage management unit: and storing the fragmented, de-duplicated and compressed data blocks into a distributed storage system.
Preferably, the data security module is used for protecting security and authority control of data by adopting a data encryption algorithm and an access control list method, and comprises encryption and audit operations of the data, and the data security module specifically comprises:
a data encryption unit: performing encryption operation on data to be protected by using an AES algorithm;
a key management unit: storing the generated key in a storage medium using a key management system storage mechanism, and periodically updating and backing up the key;
audit operation unit: recording an operation log of a user, including access, modification and deletion operations of data, storing the operation log, and subsequently auditing and tracking access and operation record use of the data;
preferably, the data statistics module is configured to calculate, according to encrypted transmission data, a basic state of the transmission data, including in particular a transmission state of information data, a data security status, and a data state of transmission completion, where the data statistics module specifically includes:
a data recording unit: recording the time stamp and the data size in the data transmission process, and analyzing the recorded information;
a data state acquisition unit: the data state is obtained according to the analysis result, which concretely comprises the following steps: the total amount of transmission data, the number of transmission misses, the amount of erroneous data, the amount of repeated data, the amount of invalid data, the frequency of data update, the system memory capacity, the used capacity, the data transmission rate, the user access rate, the data compression rate, the compression ratio, the transmission data size, the encryption time, the decryption time, the key length, the key generation time.
Preferably, the data calculating module is configured to calculate, according to the basic state of the collected user data information, data quality, transmission effect index, data security index and comprehensive quality index of the transmission data, where the data calculating module specifically includes:
a data quality calculation unit: the method is used for calculating the data quality of the user according to the data state in the process of calculating the user information data transmission, and comprises the following steps:wherein a is n For transmitting data total amount s n For transmitting the missing quantity, d n Is the error data quantity f n For repeating data volume g n For failing data volume, h n Updating the frequency for the data;
a transmission effect index calculation unit: according to the data state information of the transmission completion, calculating the transmission effect index of the user information data as follows:wherein q is n For the system storage capacity, w n For the used capacity e n Data transmission rate, t n For user access rate, y n Data compression rate, u n Is the compression ratio;
a data security index calculation unit: according to the data security condition of the user information data, calculating the security index of the transmission data as follows:；
the comprehensive quality index calculating unit: according to the calculated data quality, transmission effect index and data security index, the integrated quality index of the transmission data is calculated as follows:。
preferably, the data evaluation module is configured to compare and judge a data evaluation index with a preset value, evaluate the comprehensive quality of the data, and screen out data information with the comprehensive quality lower than the preset value, where the data evaluation module specifically includes:
data comparison unit: the calculated comprehensive quality index Z of the transmission data n And a preset value Y n Comparing and judging the size relationship of the two;
a data judging unit: when Z is n <Y n When the comprehensive quality index judgment result of the transmission data is unqualified, and the timestamp information and the transmission state data of the transmission data are screened out;
and a data feedback unit: and feeding back the acquired data to a system management module.
Preferably, the system management module is configured to receive data status information issued by the data evaluation module, and manage configuration and maintenance of a system by using an automated management and monitoring technology, including configuration management, quality improvement, and fault recovery operations, where the system management module specifically includes:
quality improvement unit: carrying out detailed analysis on the data information, including data missing analysis, data error analysis and data abnormality analysis, carrying out data cleaning on low-quality data nodes by utilizing the technologies of removing repeated data, processing missing data and repairing error data, and filling missing values by using interpolation and regression methods;
and a fault recovery unit: the fault and abnormal conditions in the system are monitored in real time by using a monitoring tool and a system log, diagnosis of the fault is carried out by checking the system log, checking hardware equipment and a network connection test technology, and corresponding recovery operation is adopted, and the method comprises the following steps: restarting the node and repairing network connection;
configuration management unit: and (3) formulating a permission distribution strategy, ensuring reasonable distribution and control of the permissions according to the user roles and the hierarchy of the permissions, determining the access and operation resources of each user role, and defining the access permissions of different user roles to the resources by using an identity verification mechanism and an access control list.
Preferably, the user interface module is used for providing a user interface, and a user completes data uploading, querying and analyzing operations through an intuitive graphical interface and interactive design, and the user interface module specifically comprises:
an interface determination unit: determining the functionality that the user interface needs to provide, including: data uploading, inquiring, analyzing, data quality displaying, data transmission effect displaying and data security index displaying;
function integration unit: through the API interface, the user interface and the back end are subjected to data interaction and communication, and a user finishes uploading information data, inquiring information data and maintaining and managing a system through the intelligent terminal.
The invention has the technical effects and advantages that:
the invention dispersedly stores the data on a plurality of nodes through a distributed storage technology, improves the capacity and expandability of the system, simultaneously reduces the risk of single point faults, adopts a multi-level data security strategy, including data encryption, authority control, access control and the like, ensures the confidentiality and integrity of the data, prevents the data from being revealed and tampered, and provides the visual display and report generation functions of the data through carrying out statistical analysis on the data stored in the system.
Drawings
Fig. 1 is a block diagram of a system architecture of the present invention.
Fig. 2 is a flow chart of the system of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The embodiment provides a light storage integrated information management system based on a distributed architecture as shown in fig. 1, which comprises: the system comprises a data storage module, a data security module, a data statistics module, a data calculation module, a data evaluation module, a system management module and a user interface module.
The data storage module is used for storing data information uploaded by a user, a distributed storage mode is adopted to realize slicing, deduplication and compression operation of data, stored data is transmitted to the data security module, the data security module is used for protecting the security and authority control of the data by adopting a data encryption algorithm and an access control list method, encryption and audit operation of the data are included, the data after being subjected to security encryption is transmitted to the data statistics module, the data statistics module is used for counting the basic state of the transmitted data according to the encrypted data, the data security state specifically comprises the transmission state, the data security state and the data state of the transmitted data, the counted data information is transmitted to the data calculation module, the data calculation module is used for calculating the data quality of the transmitted data, the transmission effect index, the data security index and the comprehensive quality index according to the basic state of the acquired user data information, the comprehensive quality index is transmitted to the data evaluation module, the data evaluation module is used for comparing the data evaluation index with a preset value, the comprehensive quality of the data is made, the data node information with the comprehensive quality of the data is screened out, the data node information with the comprehensive quality lower than the preset value is lower than the comprehensive quality, the data node information is calculated, the data node information is not lower than the data node information is calculated, the data node information is not is distributed, the data management system is used for managing system is not is arranged, the system is arranged, the user interface is arranged and a user interface is used for managing the system is used for managing the data management system is used, and is used for managing and is used for receiving and is used for a user, and is used for a system is connected to manage, and a system is used for a system is connected to manage and a system, query and analysis operations.
The implementation is different from the prior art in that the data computing module, the data evaluation module and the system management module are added with computing functions, the data quality, the transmission effect index, the data safety index and the comprehensive quality index of the system management information data are obtained through analysis on the data state and the transmission storage state of the user data, the data evaluation module is added with a comparison judging function, the judgment result is obtained through comparison evaluation on the comprehensive quality index of the management data, the system management module utilizes an automatic management and monitoring technology according to the judgment result, the configuration and maintenance of the management system are convenient for an administrator to manage and maintain the system, the normal operation of the user information data management is guaranteed, and the whole process is not possessed by the prior art.
As shown in fig. 2, the present embodiment provides a method flowchart of an optical storage integrated information management system based on a distributed architecture, which specifically includes the following steps:
101. the data information uploaded by the user is stored through the data storage module, the operations of slicing, de-duplication and compression of the data are realized by adopting a distributed storage mode, and the specific steps of the data storage module for data storage are as follows:
a1, data slicing: slicing the data uploaded by the user according to the rule of the file size, and cutting the large file into a plurality of smaller data blocks;
a2, data deduplication: performing de-duplication operation on each data block by using a hash algorithm, performing hash calculation on the data block by using MD5 or SHA-1, taking a hash value as a unique identifier, checking whether the same data block exists, storing one copy when the same data block exists, and recording the reference count;
further, when performing deduplication, performing hash calculation on each data block, performing hash calculation on the data block by using an MD5 or SHA-1 algorithm, generating a unique hash value, comparing the generated hash value with an existing hash table, checking whether the same hash value exists, if the hash value exists, saving a copy of one data block, increasing a reference count, if the hash value does not exist, adding the hash value into the hash table, saving the data block, and setting the reference count to 1;
a3, data compression: compressing the data block by using an LZ77 algorithm;
further, when data compression is performed, a sliding window and a search buffer area are required to be initialized, the sliding window is used for storing scanned data, the search buffer area is used for storing a current data block to be compressed, a character is taken out of the search buffer area and used as a current matching character, the sliding window is searched for the longest character string which is the same as the current matching character, namely the longest matching, an instruction is generated according to the length and the distance of the longest matching, the length and the distance information are included, and the instruction is output to a compressed data stream;
a4, storage management: storing the fragmented, de-duplicated and compressed data blocks into a distributed storage system;
further, dividing the fragmented, de-duplicated and compressed data blocks into a plurality of data block groups, generating a unique identifier for each data block group, uploading each data block group to a distributed storage system, selecting a plurality of storage nodes for backup, and recording the identifier and the storage position of each data block group.
102. The data security module is used for protecting the security and authority control of data by utilizing a data encryption algorithm and an access control list method, and comprises the following specific steps of data encryption and key management:
b1, data encryption: performing encryption operation on data to be protected by using an AES algorithm;
further, dividing the data to be encrypted into data blocks with fixed lengths, carrying out encryption operation on each data block, and encrypting the data blocks by using an AES algorithm and a selected key;
b2, key management: storing the generated key in a storage medium using a key management system storage mechanism, and periodically updating and backing up the key;
b3, auditing operation: recording an operation log of a user, including access, modification and deletion operations of data, storing the operation log, and subsequently auditing and tracking access and operation record use of the data;
103. the data statistics module is used for counting the basic state of the transmission data according to the encrypted transmission data, and specifically comprises the transmission state of information data, the data security state and the data state of the transmission completion, and the data statistics module specifically comprises:
c1, data recording: recording the time stamp and the data size in the data transmission process, and analyzing the recorded information;
and C2, acquiring a data state: the data state is obtained according to the analysis result, which concretely comprises the following steps: the total amount of transmission data, the number of transmission misses, the amount of erroneous data, the amount of repeated data, the amount of invalid data, the frequency of data update, the system memory capacity, the used capacity, the data transmission rate, the user access rate, the data compression rate, the compression ratio, the transmission data size, the encryption time, the decryption time, the key length, the key generation time.
104. According to the basic state of the acquired user data information, a data computing module computes the data quality, the transmission effect index, the data safety index and the comprehensive quality index of the transmission data, wherein the data computing module computes the data quality, the transmission effect index, the data safety index and the comprehensive quality index by the following specific steps:
d1, calculating data quality: the method is used for calculating the data quality of the user according to the data state in the process of calculating the user information data transmission, and comprises the following steps:wherein a is n For transmitting data total amount s n For transmitting the missing quantity, d n Is the error data quantity f n For repeating data volume g n For failing data volume, h n Updating the frequency for the data;
d2, calculating a transmission effect index: according to the data state information of the transmission completion, calculating the transmission effect index of the user information data as follows:wherein q is n For the system storage capacity, w n For the used capacity e n Data transmission rate, t n For user access rate, y n Data compression rate, u n Is the compression ratio;
d3, calculating a data security index: according to the data security condition of the user information data, calculating the security index of the transmission data as follows:；
and D4, calculating a comprehensive quality index: according to the calculated data quality, transmission effect index and data security index, the integrated quality index of the transmission data is calculated as follows:。
105. comparing and judging the data evaluation index with a preset value through a data evaluation module, evaluating the comprehensive quality of the data, and screening out data information with the comprehensive quality lower than the preset value, wherein the data evaluation module specifically comprises:
e1, data comparison: the calculated comprehensive quality index Z of the transmission data n And a preset value Y n Comparing and judging the size relationship of the two;
further, the magnitude of the preset value is related to the level of the data management quality requirement, wherein the preset value Y n As a simple example only, the size of a specific value needs to be actually defined according to the actual situation;
e2, judging data: when Z is n <Y n When the comprehensive quality index judgment result of the transmission data is unqualified, and the timestamp information and the transmission state data of the transmission data are screened out;
e3, data feedback: and feeding back the acquired data to a system management module.
106. The system management module receives the data state information issued by the data evaluation module, and manages the configuration and maintenance of the system by utilizing an automatic management and monitoring technology, wherein the configuration management, quality improvement and fault recovery operations are included, and the system management module performs the specific steps of system fault recovery and configuration management:
f1, quality improvement: carrying out detailed analysis on the data information, including data missing analysis, data error analysis and data abnormality analysis, carrying out data cleaning on low-quality data nodes by utilizing the technologies of removing repeated data, processing missing data and repairing error data, and filling missing values by using interpolation and regression methods;
further, when filling up the missing value, determining the feature or variable where the missing value is located and the position of the missing value, selecting a proper interpolation or regression method according to the feature and position of the missing value, estimating the missing value by using an interpolation function according to the existing data points, wherein the specific interpolation function and method depend on the selected interpolation method, establishing a regression model by using the existing data points for the regression method, and predicting the missing value according to the model;
f2, fault recovery: the fault and abnormal conditions in the system are monitored in real time by using a monitoring tool and a system log, diagnosis of the fault is carried out by checking the system log, checking hardware equipment and a network connection test technology, and corresponding recovery operation is adopted, and the method comprises the following steps: restarting the node and repairing network connection;
f3, configuration management: and (3) formulating a permission distribution strategy, ensuring reasonable distribution and control of the permissions according to the user roles and the hierarchy of the permissions, determining the access and operation resources of each user role, and defining the access permissions of different user roles to the resources by using an identity verification mechanism and an access control list.
107. The user interface module is used for providing a user interface, and a user finishes data uploading, inquiring and analyzing operations through an intuitive graphical interface and interaction design, wherein the user interface module comprises the following specific steps of man-machine interaction and function integration:
g1, interface determination: determining the functionality that the user interface needs to provide, including: data uploading, inquiring, analyzing, data quality displaying, data transmission effect displaying and data security index displaying;
g2, function integration: through the API interface, the user interface and the rear end are subjected to data interaction and communication, and a user finishes uploading information data, inquiring information data and maintaining and managing a system through the intelligent terminal;
further, when the functions are integrated, the developed API interface needs to be configured on the server so that the developed API interface can be accessed through a network, then an API calling function is integrated in the intelligent terminal application, a user can perform data interaction and communication with a back-end API through the application, the user selects a file or inputs data through the intelligent terminal application, and then the data is uploaded to the back-end through the API calling.
Finally: the foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (8)
1. The utility model provides a light stores up integration information management system based on distributed architecture which characterized in that: comprising the following steps:
and a data storage module: the system is used for storing data information uploaded by a user, realizing the operations of slicing, de-duplication and compression of data by adopting a distributed storage mode, and transmitting the stored data to a data security module;
and a data security module: the data statistics module is used for protecting the safety and authority control of the data by adopting a data encryption algorithm and an access control list method, including encryption and audit operation of the data, and transmitting the data subjected to the safety encryption to the data statistics module;
and a data statistics module: the data computing module is used for computing the basic state of the transmission data according to the encrypted transmission data, and particularly comprises the transmission state of information data, the data security state and the data state of the transmission completion, and transmitting the counted data information to the data computing module;
and a data calculation module: the system comprises a data evaluation module, a data quality evaluation module, a data transmission module, a data safety module, a data quality evaluation module, a data safety module and a data quality evaluation module, wherein the data quality, the transmission effect index and the data safety index of transmission data are calculated according to the basic state of acquired user data information;
and a data evaluation module: the system comprises a system management module, a data evaluation index, a data node information processing module and a data transmission status information processing module, wherein the system management module is used for comparing and judging the data evaluation index with a preset value, evaluating the comprehensive quality of data, screening out data node information with the comprehensive quality lower than the preset value and feeding back unqualified data transmission status information to the system management module;
and a system management module: the system comprises a data evaluation module, a data transmission state information acquisition module, a data transmission state information processing module and a data transmission state information processing module, wherein the data transmission state information acquisition module is used for receiving data transmission state information issued by the data evaluation module, and utilizes an automatic management and monitoring technology to manage configuration and maintenance of a system, including node management, task scheduling and fault recovery operation;
a user interface module: the method is used for providing a user interface, and a user completes data uploading, inquiring and analyzing operations through an intuitive graphical interface and interactive design.
2. The integrated optical storage information management system based on a distributed architecture as claimed in claim 1, wherein: the data storage module is used for storing data information uploaded by a user, and realizes the operations of slicing, de-duplication and compression of data by adopting a distributed storage mode, and specifically comprises:
data slicing unit: slicing the data uploaded by the user according to the rule of the file size, and cutting the large file into a plurality of smaller data blocks;
data deduplication unit: performing de-duplication operation on each data block by using a hash algorithm, performing hash calculation on the data block by using MD5 and SHA-1, taking a hash value as a unique identifier, checking, storing one copy when the same data block exists, and recording the reference count;
a data compression unit: compressing the data block by using an LZ77 algorithm;
a storage management unit: and storing the fragmented, de-duplicated and compressed data blocks into a distributed storage system.
3. The integrated optical storage information management system based on a distributed architecture as claimed in claim 1, wherein: the data security module is used for protecting the security and authority control of data by adopting a data encryption algorithm and an access control list method, and comprises encryption and audit operations of the data, and the data security module specifically comprises:
a data encryption unit: performing encryption operation on data to be protected by using an AES algorithm;
a key management unit: storing the generated key in a storage medium using a key management system storage mechanism, and periodically updating and backing up the key;
audit operation unit: and recording an operation log of the user, including access, modification and deletion operations of the data, storing the operation log, and subsequently auditing and tracking the access and operation record use of the data.
4. The integrated optical storage information management system based on a distributed architecture as claimed in claim 1, wherein: the data statistics module is used for counting the basic state of the transmission data according to the encrypted transmission data, and specifically comprises the transmission state of information data, the data security state and the data state of the transmission completion, and the data statistics module specifically comprises:
a data recording unit: recording the time stamp and the data size in the data transmission process, and analyzing the recorded information;
a data state acquisition unit: the data state is obtained according to the analysis result, which concretely comprises the following steps: the total amount of transmission data, the number of transmission misses, the amount of erroneous data, the amount of repeated data, the amount of invalid data, the frequency of data update, the system memory capacity, the used capacity, the data transmission rate, the user access rate, the data compression rate, the compression ratio, the transmission data size, the encryption time, the decryption time, the key length, the key generation time.
5. The integrated optical storage information management system based on a distributed architecture as claimed in claim 1, wherein: the data calculation module is used for calculating the data quality, the transmission effect index, the data security index and the comprehensive quality index of transmission data according to the basic state of the acquired user data information, and specifically comprises the following steps:
a data quality calculation unit: the method is used for calculating the data quality of the user according to the data state in the process of calculating the user information data transmission, and comprises the following steps:wherein a is n For transmitting data total amount s n For transmitting the missing quantity, d n Is the error data quantity f n For repeating data volume g n For failing data volume, h n Updating the frequency for the data;
a transmission effect index calculation unit: according to the data state information of the transmission completion, calculating the transmission effect index of the user information data as follows:wherein q is n For the system storage capacity, w n For the used capacity e n Data transmission rate, t n For user access rate, y n Data compression rate, u n Is the compression ratio;
a data security index calculation unit: according to the data security condition of the user information data, calculating the security index of the transmission data as follows:；
the comprehensive quality index calculating unit: according to the calculated data quality, transmission effect index and data security index, the integrated quality index of the transmission data is calculated as follows:。
6. the integrated optical storage information management system based on a distributed architecture as claimed in claim 1, wherein: the data evaluation module is used for comparing and judging the data evaluation index with a preset value, evaluating the comprehensive quality of the data, and screening out data information with the comprehensive quality lower than the preset value, and the data evaluation module specifically comprises:
data comparison unit: the calculated comprehensive quality index Z of the transmission data n And a preset value Y n Comparing and judging the size relationship of the two;
a data judging unit: when Z is n <Y n When the comprehensive quality index judgment result of the transmission data is unqualified, and the timestamp information and the transmission state data of the transmission data are screened out;
and a data feedback unit: and feeding back the acquired data to a system management module.
7. The integrated optical storage information management system based on a distributed architecture as claimed in claim 1, wherein: the system management module is used for receiving the data state information issued by the data evaluation module, managing the configuration and maintenance of the system by utilizing an automatic management and monitoring technology, and comprises configuration management, quality improvement and fault recovery operation, and the system management module specifically comprises:
quality improvement unit: carrying out detailed analysis on the data information, including data missing analysis, data error analysis and data abnormality analysis, carrying out data cleaning on low-quality data nodes by utilizing the technologies of removing repeated data, processing missing data and repairing error data, and filling missing values by using interpolation and regression methods;
and a fault recovery unit: the fault and abnormal conditions in the system are monitored in real time by using a monitoring tool and a system log, diagnosis of the fault is carried out by checking the system log, checking hardware equipment and a network connection test technology, and corresponding recovery operation is adopted, and the method comprises the following steps: restarting the node and repairing network connection;
configuration management unit: and (3) formulating a permission distribution strategy, ensuring reasonable distribution and control of the permissions according to the user roles and the hierarchy of the permissions, determining the access and operation resources of each user role, and defining the access permissions of different user roles to the resources by using an identity verification mechanism and an access control list.
8. The integrated optical storage information management system based on a distributed architecture as claimed in claim 1, wherein: the user interface module is used for providing a user interface, and a user finishes data uploading, inquiring and analyzing operations through an intuitive graphical interface and interactive design, and specifically comprises the following steps:
an interface determination unit: determining the functionality that the user interface needs to provide, including: data uploading, inquiring, analyzing, data quality displaying, data transmission effect displaying and data security index displaying;
function integration unit: through the API interface, the user interface and the back end are subjected to data interaction and communication, and a user finishes uploading information data, inquiring information data and maintaining and managing a system through the intelligent terminal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311248667.9A CN117032587B (en) | 2023-09-26 | 2023-09-26 | Optical storage integrated information management system based on distributed architecture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311248667.9A CN117032587B (en) | 2023-09-26 | 2023-09-26 | Optical storage integrated information management system based on distributed architecture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117032587A CN117032587A (en) | 2023-11-10 |
| CN117032587B true CN117032587B (en) | 2024-01-09 |
Family
ID=88632009
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311248667.9A Active CN117032587B (en) | 2023-09-26 | 2023-09-26 | Optical storage integrated information management system based on distributed architecture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117032587B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106203164A (en) * | 2016-07-01 | 2016-12-07 | 何钟柱 | The big Data Resources Management System of information security based on trust computing and cloud computing |
| CN108881415A (en) * | 2018-05-31 | 2018-11-23 | 广州亿程交通信息集团有限公司 | Distributed big data analysis system in real time |
| CN113553381A (en) * | 2021-07-28 | 2021-10-26 | 中建材信息技术股份有限公司 | Distributed data management system based on novel pipeline scheduling algorithm |
| CN116720752A (en) * | 2023-08-07 | 2023-09-08 | 济宁金虹装配式建筑科技有限公司 | Assembled building quality information supervision system based on big data |
-
2023
- 2023-09-26 CN CN202311248667.9A patent/CN117032587B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106203164A (en) * | 2016-07-01 | 2016-12-07 | 何钟柱 | The big Data Resources Management System of information security based on trust computing and cloud computing |
| CN108881415A (en) * | 2018-05-31 | 2018-11-23 | 广州亿程交通信息集团有限公司 | Distributed big data analysis system in real time |
| CN113553381A (en) * | 2021-07-28 | 2021-10-26 | 中建材信息技术股份有限公司 | Distributed data management system based on novel pipeline scheduling algorithm |
| CN116720752A (en) * | 2023-08-07 | 2023-09-08 | 济宁金虹装配式建筑科技有限公司 | Assembled building quality information supervision system based on big data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117032587A (en) | 2023-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8131677B2 (en) | System and method for effecting information governance | |
| TWI406152B (en) | Storing log data efficiently while supporting querying | |
| TWI434190B (en) | Storing log data efficiently while supporting querying to assist in computer network security | |
| US10678619B2 (en) | Unified logs and device statistics | |
| US9286319B2 (en) | Method, system and serving node for data backup and restoration | |
| CN104301360B (en) | A kind of method of logdata record, log server and system | |
| US11593029B1 (en) | Identifying a parent event associated with child error states | |
| US7100008B2 (en) | Long term data protection system and method | |
| US20090193064A1 (en) | Method and system for access-rate-based storage management of continuously stored data | |
| CN101133413A (en) | Backup information management | |
| US11475132B2 (en) | Systems and methods for protecting against malware attacks | |
| CN111680900A (en) | Work order issuing method and device, electronic equipment and storage medium | |
| CN111522499A (en) | Operation and maintenance data reading device and reading method thereof | |
| Zhang et al. | SimEDC: A simulator for the reliability analysis of erasure-coded data centers | |
| Zhang et al. | A simulation analysis of reliability in erasure-coded data centers | |
| CN117221088A (en) | Computer network intensity detection system and device | |
| CN117032587B (en) | Optical storage integrated information management system based on distributed architecture | |
| JP2013041574A (en) | Information processing system operation management device, operation management method and operation management program | |
| US12014045B2 (en) | Creation and use of an efficiency set to estimate an amount of data stored in a data set of a storage system having one or more characteristics | |
| CN116089427A (en) | Management method and system for multi-medium fusion storage of electronic files | |
| Li et al. | Fast Proactive Repair in Erasure-Coded Storage: Analysis, Design, and Implementation | |
| Rao | Data duplication using Amazon Web Services cloud storage | |
| US11645333B1 (en) | Garbage collection integrated with physical file verification | |
| KR102432530B1 (en) | System for reporting of digital evidence by sorting data collection from object disk | |
| CN116450734B (en) | Distributed storage method for development and construction digital twin data of industrial park |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |