CN117041256A - Network data transmission and storage method and system - Google Patents
Network data transmission and storage method and system Download PDFInfo
- Publication number
- CN117041256A CN117041256A CN202311286195.6A CN202311286195A CN117041256A CN 117041256 A CN117041256 A CN 117041256A CN 202311286195 A CN202311286195 A CN 202311286195A CN 117041256 A CN117041256 A CN 117041256A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- node
- disk
- scrambling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000005540 biological transmission Effects 0.000 title claims abstract description 34
- 238000004590 computer program Methods 0.000 claims description 10
- 238000013500 data storage Methods 0.000 claims description 7
- 238000012937 correction Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 5
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims 2
- 230000009193 crawling Effects 0.000 abstract description 7
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/065—Replication mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
- H04L63/145—Countermeasures against malicious traffic the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/40—Network security protocols
Abstract
The application belongs to the technical field of network data transmission and storage, and provides a network data transmission and storage method and a system, wherein disk array scrambling operation is carried out once in every node data scrambling time period in all storage server nodes; the control server performs node data scrambling operation once every other node data scrambling time period; the node data scrambling time period is dynamically adjusted, so that an illegal intrusion visitor cannot climb the number of the network in the storage server according to the crawler script program, and the possibility of crawling after the data is recovered is reduced; meanwhile, proper load balance is ensured, loss generated by high-frequency reading and writing of a magnetic disk is reduced, network time delay is increased, and network congestion is avoided.
Description
Technical Field
The application belongs to the technical field of network transmission, and particularly relates to a network data transmission and storage method and system.
Background
In a communication network, especially when a large amount of data acquired by an ad hoc network is transmitted, due to the real-time requirement of the data, the data which are not acquired in time and are partially from different acquisition nodes often need to be cleaned, converted and standardized, and after the consistency and usability of the data are ensured, the data are also required to be repaired so as to enable the data to be rapidly and synchronously complete, so that the reliability and the integrity of the data are improved under a huge data scale.
In the storage of massive large data transmitted by a communication network, the large data is huge in quantity and difficult to collect, process and analyze. Big data storage is the persistence of these data sets into a computer. The popularity of big data cloud storage is higher and higher, and undoubtedly the privacy of the information is also leaked, the data security problem caused by the high popularity of the big data cloud storage is not neglected, the security of the big data cloud storage is difficult to ensure, the existing big data cloud storage method mainly comprises the steps of automatically configuring storage equipment and space of the storage equipment through virtualization, the security of the big data cloud storage is to be improved, some illegal intrusion visitors can easily climb or copy data through crawlers according to the logical position of the virtualized cloud storage, and great hidden trouble is brought to the information security of big data users.
Disclosure of Invention
The application aims to provide a network data transmission and storage method and system, which are used for solving one or more technical problems in the prior art and at least providing a beneficial selection or creation condition.
In order to achieve the above object, according to an aspect of the present application, there is provided a network data transmission and storage method including the steps of:
the communication network system at least comprises a control server and Max storage server nodes, wherein Max is a positive integer greater than or equal to 3;
the N storage server node is called as an N node, the N+1th storage server node is called as an N+1th node, and so on; n=1 … Max; n is the node sequence number;
when N is even, the data in all storage disks in the N-th storage server node are empty;
the storage server nodes are physical servers for storing big data, each storage server node comprises a plurality of storage disks, and each storage disk comprises a first storage disk, a second storage disk and a third storage disk; wherein the second storage disk is an empty spare disk; the first storage disk and the third storage disk are disks for storing big data.
The control server comprises a data index file, wherein the data index file comprises the running time of all storage server nodes, the starting running time of the control server, the running time of first disk array scrambling operation in all storage server nodes, the first time of node data scrambling operation of the control server, the initial serial number ID of all storage server nodes, the current serial number ID of all storage server nodes and the number of disk array scrambling operation in all storage server nodes.
The method comprises the following steps:
s100: performing disk array scrambling operation once every one node data scrambling time period in all storage server nodes;
s200: the control server performs node data scrambling operation once every other node data scrambling time period;
in S100, the method for scrambling the disk array includes:
s101: copying the data of the first storage disk to a second storage disk, and physically deleting all the data in the first storage disk;
s102: copying the data of the third storage disk to the first storage disk, and physically deleting all the data in the third storage disk;
s103: copying the data of the second storage disk to a third storage disk, and physically deleting all the data in the second storage disk.
Further, the method further comprises the following steps: s104: and adding 1 to the number of disk array scrambling operations of the corresponding storage server node in the data index file of the storage server node.
In order to prevent illegal intrusion visitors from recovering and reading disk data of storage server nodes, and simultaneously carrying out load balancing on writing of each storage server node so as to greatly ensure the security of the storage server, the application provides the following calculation method of node data scrambling time period:
further, the calculation method of the node data scrambling time period comprises the following steps:
forming a sequence as a writing sequence WrList1 by all time lengths of the last written data or read data in each storage server node; the difference values between every two of all time lengths in the WrList1 are calculated in sequence, a sequence WrList2 is formed by the non-zero difference values, and the node data scrambling time period is set as the average value of all the difference values in the WrList 2.
Each difference value in WrList2 means a difference between the time spent for writing data or reading data of the storage server node, and may be greatly different due to a process and an access heat difference of a disk of the storage server node, so that the difference value is too large, and load balancing is affected.
However, since the algorithm is relatively fixed, an illegal intrusion visitor may reversely push out the node data scrambling time period according to the cracked algorithm and the script, so as to recover the disk before the period is finished according to the preset script program, thereby maliciously crawling the disk data, in order to further solve the problem, reduce the possibility of crawling after the recovery of the data and reduce the loss generated by high-frequency reading and writing of the disk, the application proposes the following preferred method:
preferably, the method for calculating the node data scrambling time period comprises the following steps:
forming a sequence as a writing sequence WrList1 by all time lengths of the last written data or read data in each storage server node; sequentially and respectively calculating the difference values between every two of all time lengths in the WrList1, forming a sequence WrList2 by each non-zero difference value,
calculating the earliest starting time corresponding to each time length in the WrList1 as TS1; calculating the latest starting time corresponding to each time length in the WrList1 as TS2; calculating the earliest ending time corresponding to each time length in the WrList1 as TE1; calculating the latest end time corresponding to each time length in the WrList1 as TE2;
taking the time length between TS1 and TS2 as TSG; taking the duration between TE1 and TE2 as TEG;
taking smaller values in TSG and TEG as numerator, taking larger values in TSG and TEG as denominator, taking the ratio of the numerator to the denominator as a time period correction coefficient K, and taking the average value of all the differences in WrList2 as Mean; when TSG is less than or equal to TEG, the node data scrambling time period is set to be mean+mean multiplied by K, otherwise, the node data scrambling time period is set to be Mean-Mean multiplied by K.
The beneficial effects are as follows: the method has the advantages that the ratio of the starting time difference to the ending time difference between the time when each storage server node last writes data or reads data is used as a time period correction coefficient K, when TSG is smaller than or equal to TEG, the ending time difference is larger than the starting time difference, the high-frequency reading and writing of a magnetic disk is easy to generate, the magnetic disk of the storage server is damaged, larger data congestion is generated, and therefore network time delay is indirectly increased, the product of the time period correction coefficient K and the average value of the difference is needed to be added, so that the frequency of node data scrambling is reduced, the loss generated by the high-frequency reading and writing of the magnetic disk is reduced, the network time delay is increased, and network congestion is avoided; when the TSG is larger than the TEG, the starting time difference value is smaller than the ending time difference value, and then the product of the time period correction coefficient K and the average value of the difference values is subtracted, so that the frequency of node data scrambling is increased, and the possibility of crawling after the data is recovered is reduced; by dynamically adjusting the node data scrambling time period, an illegal intrusion visitor cannot climb the data of the network storage server according to the crawler script program, and proper load balancing is ensured.
Preferably, the time of the node data scrambling time period is set to 10 minutes by default, and can be manually adjusted within the range of 1 to 240 minutes.
Further, in S200, the method of node data scrambling operation is:
let N be the node sequence number, N epsilon [1, max ], max is the node number; and in the value range of N, carrying out the following node data scrambling operation on each node:
s201: transmitting the data of the N node to the N+1st node, and physically deleting the data in all storage disks in the N node;
s202: transmitting the data of the N+2th node to the N node, and physically deleting the data in all storage disks in the N+2th node;
s203: transmitting the data of the (N+1) th node to the (N+2) th node, and physically deleting the data in all storage disks in the (N+1) th node;
s204: and exchanging the current serial numbers ID of the storage server nodes represented by the N node and the N+2 node in the data index file of the storage server node.
Further, when a data access request occurs, reading a data index file of the storage server node, positioning the data index file to the storage server node of the actual data storage according to the current number ID of the storage server node, positioning the number of times of disk array scrambling operation to the number ID of the actual data storage disk, and performing disk reading.
The physical deletion means that the index area of the data is deleted together with the data of the data area, and is not recoverable.
The application also provides a network data transmission and storage system, which comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to run in units of the following system:
the disk array scrambling main unit is used for performing disk array scrambling operation once every one disk array scrambling period in all storage server nodes;
the disk array scrambles the first unit, is used for copying the data of the first storage disk to the second storage disk, and delete all data in the first storage disk physically;
the disk array scrambling second unit is used for copying the data of the third storage disk to the first storage disk and physically deleting all the data in the third storage disk;
the disk array scrambling third unit is used for copying the data of the second storage disk to the third storage disk and physically deleting all the data in the second storage disk;
a disk array scrambling fourth unit, configured to add 1 to a number of disk array scrambling operations of a corresponding storage server node in the data index file of the storage server node;
the node data scrambling main unit is used for controlling the server to perform node data scrambling operation once every other node data scrambling time period;
the node data scrambling first unit is used for transmitting the data of the N node to the N+1st node and physically deleting the data in all storage disks in the N node;
the node data scrambling second unit is used for transmitting the data of the (N+2) th node to the (N+2) th node and physically deleting the data in all storage disks in the (N+2) th node;
the node data scrambling third unit is used for transmitting the data of the (N+1) th node to the (N+2) th node and physically deleting the data in all storage disks in the (N+1) th node;
and the node data scrambling fourth unit is used for exchanging the current serial numbers ID of the storage server nodes represented by the N node and the N+2 node in the data index file of the storage server node.
The beneficial effects of the application are as follows: the application provides a network data transmission storage method and a system, which dynamically adjust the node data scrambling time period, so that an illegal intrusion visitor cannot climb the number of storage servers of a network according to a crawler script program, and the possibility of crawling after the data is recovered is reduced; meanwhile, proper load balance is ensured, loss generated by high-frequency reading and writing of a magnetic disk is reduced, network time delay is increased, and network congestion is avoided.
Drawings
The above and other features of the present application will become more apparent from the detailed description of the embodiments thereof given in conjunction with the accompanying drawings, in which like reference characters designate like or similar elements, and it is apparent that the drawings in the following description are merely some examples of the present application, and other drawings may be obtained from these drawings without inventive effort to those of ordinary skill in the art, in which:
FIG. 1 is a flow chart of a method for network data transmission and storage;
fig. 2 is a diagram showing a structure of a network data transmission storage system.
Detailed Description
The conception, specific structure, and technical effects produced by the present application will be clearly and completely described below with reference to the embodiments and the drawings to fully understand the objects, aspects, and effects of the present application. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other.
Referring to fig. 1, a flowchart of a network data transmission and storage method is shown, and a method for storing network data transmission and storage according to an embodiment of the present application is described below with reference to fig. 1, where the method includes the following steps:
the communication network system at least comprises a control server and Max storage server nodes, wherein Max is a positive integer greater than or equal to 3;
the N storage server node is called as an N node, the N+1th storage server node is called as an N+1th node, and so on; n=1 … Max; n is the node sequence number;
when N is even, the data in all storage disks in the N-th storage server node are empty;
the storage server nodes are physical servers for storing big data, each storage server node comprises a plurality of storage disks, and each storage disk comprises a first storage disk, a second storage disk and a third storage disk; wherein the second storage disk is an empty spare disk; the first storage disk and the third storage disk are disks for storing big data.
Wherein the storage server is typically a stand-alone unit. Sometimes they are designed as 4U racks. Alternatively, they may consist of a storage unit and a server located nearby. The two boxes can then be installed in parallel in the cabinet. Specifically, the storage server is a SunFire X4100 server.
The control server is GNAC, and the GNAC is connected in series in the network, one Ethernet port of the GNAC is connected with a gateway or a server group, and the other Ethernet port is connected with a local area network switch. The network bridge mode is strictly managed, is easy to access and basically does not influence the original network structure. In addition, a dual-machine hot standby mechanism is provided, if one control server fails, the control server is automatically switched to the other control server for management, and the reliability of management is ensured. The GNAC is connected to the mirror image port, the mirror image source port of the switch is connected with the gateway or the server group, and the mirror image destination port of the switch is connected with the GNAC. The bypass mode of the GNAC adopts a monitoring mechanism to implement network control, is simple and convenient to deploy, does not influence the original network structure, and has no worry of network single-point faults.
The control server comprises a data index file, wherein the data index file comprises the running time of all storage server nodes, the starting running time of the control server, the running time of first disk array scrambling operation in all storage server nodes, the first time of node data scrambling operation of the control server, the initial serial number ID of all storage server nodes, the current serial number ID of all storage server nodes and the number of disk array scrambling operation in all storage server nodes.
The method comprises the following steps:
s100: performing disk array scrambling operation once every one node data scrambling time period in all storage server nodes;
s200: the control server performs node data scrambling operation once every other node data scrambling time period;
in S100, the method for scrambling the disk array includes:
s101: copying the data of the first storage disk to a second storage disk, and physically deleting all the data in the first storage disk;
s102: copying the data of the third storage disk to the first storage disk, and physically deleting all the data in the third storage disk;
s103: copying the data of the second storage disk to a third storage disk, and physically deleting all the data in the second storage disk.
Further, the method further comprises the following steps: s104: and adding 1 to the number of disk array scrambling operations of the corresponding storage server node in the data index file of the storage server node.
In order to prevent illegal intrusion visitors from recovering and reading disk data of storage server nodes, and simultaneously carrying out load balancing on writing of each storage server node so as to greatly ensure the security of the storage server, the application provides the following calculation method of node data scrambling time period:
further, the calculation method of the node data scrambling time period comprises the following steps:
forming a sequence as a writing sequence WrList1 by all time lengths of the last written data or read data in each storage server node; the difference values between every two of all time lengths in the WrList1 are calculated in sequence, a sequence WrList2 is formed by the non-zero difference values, and the node data scrambling time period is set as the average value of all the difference values in the WrList 2.
Each difference value in WrList2 means a difference between the time spent for writing data or reading data of the storage server node, and may be greatly different due to a process and an access heat difference of a disk of the storage server node, so that the difference value is too large, and load balancing is affected.
However, since the algorithm is relatively fixed, an illegal intrusion visitor may reversely push out the node data scrambling time period according to the cracked algorithm and the script, so as to recover the disk before the period is finished according to the preset script program, thereby maliciously crawling the disk data, in order to further solve the problem, reduce the possibility of crawling after the recovery of the data and reduce the loss generated by high-frequency reading and writing of the disk, the application proposes the following preferred method:
preferably, the method for calculating the node data scrambling time period comprises the following steps:
forming a sequence as a writing sequence WrList1 by all time lengths of the last written data or read data in each storage server node; sequentially and respectively calculating the difference values between every two of all time lengths in the WrList1, forming a sequence WrList2 by each non-zero difference value,
calculating the earliest starting time corresponding to each time length in the WrList1 as TS1; calculating the latest starting time corresponding to each time length in the WrList1 as TS2; calculating the earliest ending time corresponding to each time length in the WrList1 as TE1; calculating the latest end time corresponding to each time length in the WrList1 as TE2;
taking the time length between TS1 and TS2 as TSG; taking the duration between TE1 and TE2 as TEG;
taking smaller values in TSG and TEG as numerator, taking larger values in TSG and TEG as denominator, taking the ratio of the numerator to the denominator as a time period correction coefficient K, and taking the average value of all the differences in WrList2 as Mean; when TSG is less than or equal to TEG, the node data scrambling time period is set to be mean+mean multiplied by K, otherwise, the node data scrambling time period is set to be Mean-Mean multiplied by K.
Preferably, the time of the node data scrambling time period is set to 10 minutes by default.
Further, in S200, the method of node data scrambling operation is:
let N be the node sequence number, N epsilon [1, max ], max is the node number; and in the value range of N, carrying out the following node data scrambling operation on each node:
s201: transmitting the data of the N node to the N+1st node, and physically deleting the data in all storage disks in the N node;
s202: transmitting the data of the N+2th node to the N node, and physically deleting the data in all storage disks in the N+2th node;
s203: transmitting the data of the (N+1) th node to the (N+2) th node, and physically deleting the data in all storage disks in the (N+1) th node;
s204: and exchanging the current serial numbers ID of the storage server nodes represented by the N node and the N+2 node in the data index file of the storage server node.
Further, when a data access request occurs, reading a data index file of the storage server node, positioning the data index file to the storage server node of the actual data storage according to the current number ID of the storage server node, positioning the number of times of disk array scrambling operation to the number ID of the actual data storage disk, and performing disk reading.
The physical deletion is to delete the index area of the data together with the data of the data area, and erase and clear the magnetic storage area used for storing the file.
Fig. 2 is a block diagram of a network data transmission and storage system according to an embodiment of the present application, where the network data transmission and storage system according to the embodiment includes: a processor, a memory and a computer program stored in the memory and executable on the processor, the processor implementing the steps in one of the network data transmission storage system embodiments described above when the computer program is executed.
The system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to run in units of the following system:
the disk array scrambling main unit is used for performing disk array scrambling operation once every one disk array scrambling period in all storage server nodes;
the disk array scrambles the first unit, is used for copying the data of the first storage disk to the second storage disk, and delete all data in the first storage disk physically;
the disk array scrambling second unit is used for copying the data of the third storage disk to the first storage disk and physically deleting all the data in the third storage disk;
the disk array scrambling third unit is used for copying the data of the second storage disk to the third storage disk and physically deleting all the data in the second storage disk;
a disk array scrambling fourth unit, configured to add 1 to a number of disk array scrambling operations of a corresponding storage server node in the data index file of the storage server node;
the node data scrambling main unit is used for controlling the server to perform node data scrambling operation once every other node data scrambling time period;
the node data scrambling first unit is used for transmitting the data of the N node to the N+1st node and physically deleting the data in all storage disks in the N node;
the node data scrambling second unit is used for transmitting the data of the (N+2) th node to the (N+2) th node and physically deleting the data in all storage disks in the (N+2) th node;
the node data scrambling third unit is used for transmitting the data of the (N+1) th node to the (N+2) th node and physically deleting the data in all storage disks in the (N+1) th node;
and the node data scrambling fourth unit is used for exchanging the current serial numbers ID of the storage server nodes represented by the N node and the N+2 node in the data index file of the storage server node.
The network data transmission storage system can be operated in computing equipment such as a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The network data transmission storage system may include, but is not limited to, a processor, a memory. It will be appreciated by those skilled in the art that the example is merely an example of a network data transmission storage system and is not limiting of a network data transmission storage system, and may include more or fewer components than examples, or may combine certain components, or different components, e.g., the network data transmission storage system may further include input and output devices, network access devices, buses, etc.
The processor may be a central processing unit (CentralProcessingUnit, CPU), other general purpose processors, digital signal processors (DigitalSignalProcessor, DSP), application specific integrated circuits (ApplicationSpecificIntegratedCircuit, ASIC), field programmable gate arrays (Field-ProgrammableGateArray, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like that is a control center of an operating system of the network data transmission storage system, and that connects various parts of the entire operating system of the network data transmission storage system using various interfaces and lines.
The memory may be used to store the computer program and/or modules, and the processor may implement various functions of the network data transmission storage system by executing or executing the computer program and/or modules stored in the memory and invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SmartMediaCard, SMC), secure digital (SecureDigital, SD) card, flash card (FlashCard), at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
Although the present application has been described in considerable detail and with particularity with respect to several described embodiments, it is not intended to be limited to any such detail or embodiment or any particular embodiment so as to effectively cover the intended scope of the application. Furthermore, the foregoing description of the application has been presented in its embodiments contemplated by the inventors for the purpose of providing a useful description, and for the purposes of providing a non-essential modification of the application that may not be presently contemplated, may represent an equivalent modification of the application.
Claims (10)
1. A method for network data transmission and storage, the method comprising the steps of:
performing disk array scrambling operation once every one node data scrambling time period in all storage server nodes; the control server performs node data scrambling operation once every other node data scrambling time period;
the method for scrambling the disk array comprises the following steps: copying the data of the first storage disk to a second storage disk, and physically deleting all the data in the first storage disk; copying the data of the third storage disk to the first storage disk, and physically deleting all the data in the third storage disk; copying the data of the second storage disk to a third storage disk, and physically deleting all the data in the second storage disk.
2. A network data transmission storage method according to claim 1, characterized in that it comprises at least a control server, a plurality of storage server nodes in a communication network system; the N storage server node is called as an N node, the N+1th storage server node is called as an N+1th node, and so on; n=1 … Max; n is the node sequence number; and when N is even, the data in all storage disks in the Nth storage server node are empty.
3. The network data transmission and storage method according to claim 1, wherein the storage server nodes are physical servers for storing big data, each storage server node comprises a plurality of storage disks, and the storage disks comprise a first storage disk, a second storage disk and a third storage disk; wherein the second storage disk is an empty spare disk; the first storage disk and the third storage disk are disks for storing big data.
4. The network data transmission and storage method according to claim 1, wherein the control server includes a data index file, and the data index file includes a running time of all storage server nodes, a starting running time of the control server, a first disk array scrambling operation running time of all storage server nodes, a first node data scrambling operation time of the control server, an initial number ID of all storage server nodes, a current number ID of all storage server nodes, and a number of disk array scrambling operations of all storage server nodes.
5. The network data transmission and storage method according to claim 1, wherein the node data scrambling time period calculating method comprises the steps of: forming a sequence as a writing sequence WrList1 by all time lengths of the last written data or read data in each storage server node; the difference values between every two of all time lengths in the WrList1 are calculated in sequence, a sequence WrList2 is formed by the non-zero difference values, and the node data scrambling time period is set as the average value of all the difference values in the WrList 2.
6. The network data transmission and storage method according to claim 5, wherein the method for calculating the node data scrambling time period is replaced with:
forming a sequence as a writing sequence WrList1 by all time lengths of the last written data or read data in each storage server node; sequentially and respectively calculating the difference values between every two of all the time lengths in the WrList1, forming a sequence WrList2 by using each non-zero difference value, and calculating the earliest starting time corresponding to each time length in the WrList1 as TS1; calculating the latest starting time corresponding to each time length in the WrList1 as TS2; calculating the earliest ending time corresponding to each time length in the WrList1 as TE1; calculating the latest end time corresponding to each time length in the WrList1 as TE2;
taking the time length between TS1 and TS2 as TSG; taking the duration between TE1 and TE2 as TEG;
taking smaller values in TSG and TEG as numerator, taking larger values in TSG and TEG as denominator, taking the ratio of the numerator to the denominator as a time period correction coefficient K, and taking the average value of all the differences in WrList2 as Mean; when TSG is less than or equal to TEG, the node data scrambling time period is set to be mean+mean multiplied by K, otherwise, the node data scrambling time period is set to be Mean-Mean multiplied by K.
7. The method for storing network data according to claim 1, wherein in S200, the method for scrambling node data is as follows:
let N be the node sequence number, N epsilon [1, max ], max is the node number; and in the value range of N, carrying out the following node data scrambling operation on each node:
s201: transmitting the data of the N node to the N+1st node, and physically deleting the data in all storage disks in the N node;
s202: transmitting the data of the N+2th node to the N node, and physically deleting the data in all storage disks in the N+2th node;
s203: transmitting the data of the (N+1) th node to the (N+2) th node, and physically deleting the data in all storage disks in the (N+1) th node;
s204: and exchanging the current serial numbers ID of the storage server nodes represented by the N node and the N+2 node in the data index file of the storage server node.
8. The network data transmission and storage method according to claim 1, wherein when a data access request occurs, a data index file of a storage server node is read, the storage server node is located to an actual data storage according to a current number ID of the storage server node, and the number of disk array scrambling operations is located to an actual data storage disk number ID, so that disk reading is performed.
9. The network data transmission and storage method of claim 1, wherein the physical deletion means that the index area of the data is deleted together with the data of the data area, and is unrecoverable.
10. A network data transfer storage system, the network data transfer storage system comprising: a processor, a memory and a computer program stored in the memory and executable on the processor, the processor implementing the steps in a network data transmission storage method according to any one of claims 1 to 9 when the computer program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311286195.6A CN117041256B (en) | 2023-10-08 | 2023-10-08 | Network data transmission and storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311286195.6A CN117041256B (en) | 2023-10-08 | 2023-10-08 | Network data transmission and storage method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117041256A true CN117041256A (en) | 2023-11-10 |
| CN117041256B CN117041256B (en) | 2024-02-02 |
Family
ID=88639881
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311286195.6A Active CN117041256B (en) | 2023-10-08 | 2023-10-08 | Network data transmission and storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117041256B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5542065A (en) * | 1995-02-10 | 1996-07-30 | Hewlett-Packard Company | Methods for using non-contiguously reserved storage space for data migration in a redundant hierarchic data storage system |
| JP2011008583A (en) * | 2009-06-26 | 2011-01-13 | Toshiba Corp | Disk array controller and disk array device |
| CN105808150A (en) * | 2014-12-29 | 2016-07-27 | 先智云端数据股份有限公司 | Solid state disk caching system for hybrid storage device |
| WO2017162179A1 (en) * | 2016-03-23 | 2017-09-28 | 北京书生国际信息技术有限公司 | Load rebalancing method and apparatus for use in storage system |
| CN107346342A (en) * | 2017-07-13 | 2017-11-14 | 上海携程商务有限公司 | A kind of file call method calculated based on storage and system |
| US20200133587A1 (en) * | 2018-10-26 | 2020-04-30 | EMC IP Holding Company LLC | Method, device, and computer readable storage medium for managing redundant array of independent disks |
| CN114265834A (en) * | 2021-12-27 | 2022-04-01 | 山东浪潮科学研究院有限公司 | Distributed time sequence database storage engine method, device and medium |
| US20220215103A1 (en) * | 2021-01-07 | 2022-07-07 | Nxp B.V. | Data processing system and method for protecting data in the data processing system |
| WO2023068463A1 (en) * | 2021-10-21 | 2023-04-27 | 서울대학교산학협력단 | Storage device system for quantum circuit simulation |
-
2023
- 2023-10-08 CN CN202311286195.6A patent/CN117041256B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5542065A (en) * | 1995-02-10 | 1996-07-30 | Hewlett-Packard Company | Methods for using non-contiguously reserved storage space for data migration in a redundant hierarchic data storage system |
| JP2011008583A (en) * | 2009-06-26 | 2011-01-13 | Toshiba Corp | Disk array controller and disk array device |
| CN105808150A (en) * | 2014-12-29 | 2016-07-27 | 先智云端数据股份有限公司 | Solid state disk caching system for hybrid storage device |
| WO2017162179A1 (en) * | 2016-03-23 | 2017-09-28 | 北京书生国际信息技术有限公司 | Load rebalancing method and apparatus for use in storage system |
| CN107346342A (en) * | 2017-07-13 | 2017-11-14 | 上海携程商务有限公司 | A kind of file call method calculated based on storage and system |
| US20200133587A1 (en) * | 2018-10-26 | 2020-04-30 | EMC IP Holding Company LLC | Method, device, and computer readable storage medium for managing redundant array of independent disks |
| US20220215103A1 (en) * | 2021-01-07 | 2022-07-07 | Nxp B.V. | Data processing system and method for protecting data in the data processing system |
| WO2023068463A1 (en) * | 2021-10-21 | 2023-04-27 | 서울대학교산학협력단 | Storage device system for quantum circuit simulation |
| CN114265834A (en) * | 2021-12-27 | 2022-04-01 | 山东浪潮科学研究院有限公司 | Distributed time sequence database storage engine method, device and medium |
Non-Patent Citations (1)
| Title |
|---|
| 倪云竹;李志蜀;胡大裟;: "基于改进型遗传算法的存储子系统动态负载均衡", 中南大学学报(自然科学版), no. 08, pages 97 - 106 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117041256B (en) | 2024-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5571786B2 (en) | Method, system, and program for deduplicating data in a distributed environment including sources and targets | |
| US10997127B2 (en) | Preventing inefficient recalls in a hierarchical storage management (HSM) system | |
| US11093387B1 (en) | Garbage collection based on transmission object models | |
| US10169358B2 (en) | Data deduplication using a small hash table | |
| US10628298B1 (en) | Resumable garbage collection | |
| US11397749B2 (en) | Asynchronous replication of in-scope table data | |
| US11210003B2 (en) | Method, device and computer program product for restoring data based on replacing child node identifiers with parent node identifier | |
| US8347052B2 (en) | Initializing of a memory area | |
| US11289115B1 (en) | Copying a storage tape | |
| US10395145B2 (en) | Deduplication ratio estimation using representative images | |
| CN107798063B (en) | Snapshot processing method and snapshot processing device | |
| US10956042B2 (en) | Tiering data compression within a storage system | |
| JP2021522577A (en) | Host-aware update write method, system, and computer program | |
| US10042570B2 (en) | Tape backup and restore in a disk storage environment with intelligent data placement | |
| CN105808451B (en) | Data caching method and related device | |
| US10691349B2 (en) | Mitigating data loss | |
| CN117041256B (en) | Network data transmission and storage method and system | |
| US10747726B2 (en) | Deduplication ratio estimation using an expandable basis set | |
| AU2021268828B2 (en) | Secure data replication in distributed data storage environments | |
| US11640339B2 (en) | Creating a backup data set | |
| US10078553B2 (en) | Point in time copy technique using a block level of granularity | |
| US20210208989A1 (en) | Enterprise-scale time series graphite backend infrastructure | |
| US11163642B2 (en) | Methods, devices and computer readable medium for managing a redundant array of independent disks | |
| CN113297003A (en) | Method, electronic device and computer program product for managing backup data | |
| US20210383006A1 (en) | Distribution of user specific data elements in a replication environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |