CN117093576B - Data storage method for reducing storage transfer time - Google Patents
Data storage method for reducing storage transfer time Download PDFInfo
- Publication number
- CN117093576B CN117093576B CN202311339956.XA CN202311339956A CN117093576B CN 117093576 B CN117093576 B CN 117093576B CN 202311339956 A CN202311339956 A CN 202311339956A CN 117093576 B CN117093576 B CN 117093576B
- Authority
- CN
- China
- Prior art keywords
- data
- target
- transfer
- node
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012546 transfer Methods 0.000 title claims abstract description 256
- 238000000034 method Methods 0.000 title claims abstract description 86
- 238000013500 data storage Methods 0.000 title claims abstract description 37
- 230000008569 process Effects 0.000 claims description 49
- 230000005540 biological transmission Effects 0.000 claims description 24
- 238000005457 optimization Methods 0.000 claims description 9
- 230000005484 gravity Effects 0.000 claims description 7
- 230000008030 elimination Effects 0.000 claims description 6
- 238000003379 elimination reaction Methods 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 5
- 230000000694 effects Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 3
- 239000000203 mixture Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000001364 causal effect Effects 0.000 description 1
- 230000006854 communication Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Probability & Statistics with Applications (AREA)
- Quality & Reliability (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data storage method for reducing storage transfer time, which belongs to the technical field of data storage, and comprises the following steps: obtaining original data, removing repeated data and useless data in the original data, and obtaining target data; determining a transfer target for the target data, and determining a final transfer format of the target data based on the transfer target; determining a transfer phase and phase parameters of a current transfer mode according to a final transfer format and a current transfer format of target data; optimizing the transfer stage and stage parameters based on a preset transfer logic, acquiring a target transfer mode, and storing target data by using the target transfer mode. The method solves the problems that the existing data storage method is to store data in a classified mode according to data parameters such as data uploading time and the like, and the data is not stored in an overlong mode due to the fact that different data storage formats are adopted according to a data storage target.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a data storage method for reducing storage transfer time.
Background
At present, along with development of science and technology, demands of internet enterprises on data storage are higher and higher, meanwhile, along with gradual expansion of internet financial modes, demands of existing users on timeliness of data are higher and higher, storage and transfer time of users on different types of data are different, for example, storage time of information data such as bank cards is shorter, storage time of some video file data or text file data is longer, but the existing data storage method is used for storing data in a classified mode according to data parameters such as data uploading time and the like, and different data storage formats are not adopted according to transfer targets of the data, so that the data transfer and storage time is overlong.
Therefore, the invention provides a data storage method for reducing the storage transfer time.
Disclosure of Invention
The invention provides a data storage method for reducing storage transfer time, which is characterized in that repeated data and useless data in original data are removed, target data are obtained, a transfer target is determined, a final transfer format of the target data is determined based on the transfer target, transfer stages and stage parameters of a current transfer mode are determined according to the final transfer format and the current transfer format of the target data, the transfer stages and the stage parameters are optimized based on preset transfer logic, the target transfer mode is obtained, and the target data are stored, so that the problem that the data transfer storage time is overlong due to the fact that the existing data storage method in the background technology is used for classifying and storing the data according to the data parameters such as data uploading time and the like and different data storage formats are not adopted according to the transfer target of the data is solved.
The invention provides a data storage method for reducing storage transfer time, which comprises the following steps:
step 1: obtaining original data, removing repeated data and useless data in the original data, and obtaining target data;
step 2: determining a transfer target for the target data, and determining a final transfer format of the target data based on the transfer target;
step 3: determining a transfer phase and phase parameters of a current transfer mode according to a final transfer format and a current transfer format of target data;
step 4: optimizing the transfer stage and stage parameters based on a preset transfer logic, acquiring a target transfer mode, and storing target data by using the target transfer mode.
Preferably, the obtaining the original data, eliminating the repeated data and the useless data in the original data, and obtaining the target data includes:
acquiring original data from a data source, and counting the original data;
acquiring a plurality of data items of the original data based on the statistical result, and performing first elimination by taking a first repeated data item in the plurality of data items as repeated data;
judging the association degree between the data items according to the data attribute between the data items, and performing second elimination by taking a second data item with the association degree smaller than a preset threshold value as useless data;
and acquiring the original data after the first eliminating process and the second eliminating process and taking the original data as target data.
Preferably, determining the spool target for the target data, determining a final spool format of the target data based on the spool target, includes:
acquiring a data transfer request of a client, and determining a transfer path of target data in the client according to the transfer request;
determining a dump target for target data based on the dump path;
determining the configuration information of the transfer requirement for the target data based on the transfer target of the target data;
determining a transfer strategy corresponding to the target data according to the transfer demand configuration information of the target data;
and determining the final transfer format of the target data according to the transfer strategy.
Preferably, after the data transfer request of the client is acquired, the method further includes:
acquiring the request sending time of the data transfer request;
judging whether the request sending time of the data transfer request is in a preset time period or not;
if yes, a first storage security policy for the target data is obtained;
if not, a second storage security policy for the target data is obtained.
Preferably, determining the transfer phase and the phase parameter of the current transfer mode according to the final transfer format and the current transfer format of the target data includes:
acquiring a first transfer process of a final transfer format of target data and a second transfer process of a current transfer format;
determining a third transfer process for transferring the target data from the current transfer format to a final transfer format according to the first transfer process and the second transfer process;
acquiring a phase transfer characteristic corresponding to the third transfer process;
and determining phase parameters according to the phase transfer characteristics.
Preferably, optimizing the transfer stage and the stage parameter based on the preset transfer logic, obtaining a target transfer mode, and storing the target data by using the target transfer mode, including:
acquiring target logic parameter information of a preset dump logic;
extracting target unloading step and target stage parameters based on the target logic parameter information;
performing first comparison on the target transfer step and the transfer stage to obtain a process deviation degree;
performing second comparison on the target stage parameter and the stage parameter to obtain a parameter difference value;
optimizing the transfer stage and stage parameters based on the process deviation degree and the parameter difference value, acquiring a target transfer mode, and storing target data by using the target transfer mode.
Preferably, the determining the association degree between the data items according to the data attribute between the data items includes:
analyzing the relation between every two data items, and establishing a first relation network;
recording the position of each data item on the first relation network as a corresponding first network node;
acquiring a plurality of first network nodes contained in a first relation network, and respectively acquiring node relations between each target first network node and the rest target second network nodes in the first relation network;
based on the node relation, combining the data items with the connection effect to form a data pair;
performing alignment training on a plurality of data items contained in each data pair;
acquiring data attributes of the aligned data items, and comparing information to acquire attribute difference points between two adjacent data items;
determining the difference proportion corresponding to the attribute difference point between two adjacent data items;
and determining the association degree between the data items according to the difference specific gravity.
Preferably, in the process of storing the target data by using the target transfer mode, the method further includes:
acquiring the flow variation condition of a relevant network layer of a client when receiving target data;
evaluating the stability coefficient of each network layer according to the flow variation condition of the relevant network layer of the client when receiving the target data;
and evaluating the probability of success of the client to the transmission of the target data according to the stability coefficient of each network layer:
；
wherein Q is expressed as the probability of successful transmission of the target data by the client,expressed as performance parameters of the client, M expressed as the number of relevant network layers in the client, j expressed as the j-th present backbone network layer,/and%>Expressed as steady state factor, ">Stability factor denoted as j-th related network layer,>the operating strength denoted as j-th related network layer;
and determining whether the probability of success of the transmission of the client to the target data is greater than or equal to a preset probability, if so, not carrying out subsequent operation, and if not, carrying out network optimization on a relevant network layer of the client to ensure successful transmission of the target data.
Preferably, network optimization is performed on the relevant network layer of the client to ensure successful transmission of the target data, including:
acquiring operation data and node attribute data of each node in a relevant network layer of a client;
determining a first target node set with the node computing resource occupation amount larger than a first preset value in each node based on the operation data of each node;
determining node attribute parameters of each node based on the node attribute data of each node;
determining a second target node set with the node attribute parameter larger than a second preset value in each node based on the node attribute parameter of each node, and determining a third target node based on the second target node set and the first target node set;
and determining the running state of the third target node based on the running data and the node attribute data of the third target node, and performing optimization processing on the third target node to ensure successful transmission of the target data when the running state is determined to meet the preset running state.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, serve to explain the invention. In the drawings:
FIG. 1 is a flowchart of a data storage method for reducing a storage transfer time according to an embodiment of the present invention;
FIG. 2 is a flowchart of a data storage method for reducing the storage transfer time according to an embodiment of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described below with reference to the accompanying drawings, it being understood that the preferred embodiments described herein are for illustration and explanation of the present invention only, and are not intended to limit the present invention.
Example 1:
the invention provides a data storage method for reducing storage transfer time, as shown in fig. 1, the method comprises the following steps:
step 1: obtaining original data, removing repeated data and useless data in the original data, and obtaining target data;
step 2: determining a transfer target for the target data, and determining a final transfer format of the target data based on the transfer target;
step 3: determining a transfer phase and phase parameters of a current transfer mode according to a final transfer format and a current transfer format of target data;
step 4: optimizing the transfer stage and stage parameters based on a preset transfer logic, acquiring a target transfer mode, and storing target data by using the target transfer mode.
In this embodiment, the raw data is data in a user database, which may or may not be in machine readable form, and is in various forms, such as text data, image data, audio data, or a mixture of several data.
In this embodiment, the duplicate data is a duplicate between data, and can be said to be a phenomenon in which the same data is stored in different data files.
In this embodiment, the garbage data refers to erroneous or meaningless data.
In this embodiment, the dump target refers to, for example, whether the data can be edited finally, if it needs to be edited multiple times, it can be dumped into Word form, and if it does not need to be edited, it can be dumped into PDF form.
The dump target may also be to ensure reliability, security, scalability, and manageability of the data store.
In this embodiment, the spool format may be pdf, html, txt, gif or jpg.
In this embodiment, the transfer mode includes: copy storage, distributed storage, and USB flash disk storage.
Completely transferring: means that the data in the whole database is totally backed up again;
the difference is transferred and stored: is a dump of the modified or deleted records in all files after the last full dump;
incremental transfer: copying the whole file changed after the last dump.
In this embodiment, the dump phase includes: and receiving data, determining a transfer target, determining a final transfer format, and transferring and storing.
In this embodiment, the phase parameters refer to, for example, received data, and then the parameters include: the received data size, the time of reception, and the form of reception.
In this embodiment, the preset dump logic is a rule that is prescribed in advance and needs to be adhered to in the process of dump, for example, the amount of target data received at one time cannot exceed 100.
The beneficial effects of the technical scheme are as follows: the method has the advantages that the useless data in the original data are removed repeatedly, the target data are obtained, the final transfer format is determined according to the transfer target, the final format can be determined rapidly, the transfer speed is accelerated, meanwhile, the transfer stage and the stage parameter of the current transfer mode are determined, the transfer stage and the stage parameter are optimized based on the preset transfer logic, the target transfer mode is obtained, the target data can be stored in an accurate target transfer mode, the time required by data transfer is saved, and the data transfer process is simplified and accurate.
Example 2:
the invention provides a data storage method for reducing storage transfer time, as shown in figure 2, obtaining original data, eliminating repeated data and useless data in the original data, and obtaining target data, comprising the following steps:
s01: acquiring original data from a data source, and counting the original data;
s02: acquiring a plurality of data items of the original data based on the statistical result, and performing first elimination by taking a first repeated data item in the plurality of data items as repeated data;
s03: judging the association degree between the data items according to the data attribute between the data items, and performing second elimination by taking a second data item with the association degree smaller than a preset threshold value as useless data;
s04: and acquiring the original data after the first eliminating process and the second eliminating process and taking the original data as target data.
In this embodiment, the raw data is data in a user database, which may or may not be in machine readable form, and is in various forms, such as text data, image data, audio data, or a mixture of several data.
In this embodiment, the first culling refers to culling of repeated data.
In this embodiment, the degree of association refers to the degree of association between data.
In this embodiment, the second culling refers to culling data that has no actual meaning or error.
In this embodiment, the preset threshold is 90%.
The beneficial effects of the technical scheme are as follows: by acquiring the target data, removing the repeated data and the useless data in the target data and taking the removed data as the target data, larger backup capacity can be provided, longer-time data retention can be realized, continuous verification of the backup data can be ensured, the data recovery service level is improved, and data disaster recovery and the like can be conveniently realized.
Example 3:
the invention provides a data storage method for reducing storage transfer time, which determines a transfer target of target data, and determines a final transfer format of the target data based on the transfer target, and comprises the following steps:
acquiring a data transfer request of a client, and determining a transfer path of target data in the client according to the transfer request;
determining a dump target for target data based on the dump path;
determining the configuration information of the transfer requirement for the target data based on the transfer target of the target data;
determining a transfer strategy corresponding to the target data according to the transfer demand configuration information of the target data;
and determining the final transfer format of the target data according to the transfer strategy.
In this embodiment, the transfer path may be transferred from the mobile phone to the computer, or may be transferred from the computer to the usb disk.
In this embodiment, the dump target refers to, for example, whether the data can be edited finally, if it needs to be edited multiple times, it can be dumped into Word form, and if it does not need to be edited, it can be dumped into PDF form.
In this embodiment, the dump demand configuration information includes: source database and target database related information.
In this embodiment, the dump policy is, for example, to obtain a message queue executor, parse the target to-be-dumped data by using JSONObject, and then dump the parsed target to-be-dumped data to the target storage space according to the message queue executor.
In this embodiment, the transfer format may be PPT format, pdf format, JPG format, PNG format, MP4 format.
The beneficial effects of the technical scheme are as follows: by determining the transfer path of the target data in the client, and determining the transfer target and transfer requirement configuration information, a transfer strategy is obtained, a final transfer format is determined, and an appropriate transfer format can be selected, so that a transfer result is more targeted.
Example 4:
the invention provides a data storage method for reducing storage transfer time, which further comprises the following steps after a data transfer request of a client is acquired:
acquiring the request sending time of the data transfer request;
judging whether the request sending time of the data transfer request is in a preset time period or not;
if yes, a first storage security policy for the target data is obtained;
if not, a second storage security policy for the target data is obtained.
In this embodiment, the request issue time may be 9 am 20 minutes 25 seconds.
In this embodiment, the preset time period may be 8.m. 20.00.min.00.s — 11.m. 20.00.min.00.s.
In this embodiment, the first storage security policy means that if the request sending time is within a preset time period, only password verification is needed.
In this embodiment, the second storage security policy means that if the request sending time is not within the preset time period, password authentication+face authentication is required.
The beneficial effects of the technical scheme are as follows: by judging whether the sending time is within the preset time period or not and selecting a proper storage security policy, the security of the stored data can be ensured.
Example 5:
the invention provides a data storage method for reducing storage transfer time, which determines transfer phase and phase parameters of a current transfer mode according to a final transfer format and a current transfer format of target data, and comprises the following steps:
acquiring a first transfer process of a final transfer format of target data and a second transfer process of a current transfer format;
determining a third transfer process for transferring the target data from the current transfer format to a final transfer format according to the first transfer process and the second transfer process;
acquiring a phase transfer characteristic corresponding to the third transfer process;
and determining phase parameters according to the phase transfer characteristics.
In this embodiment, the first transfer process refers to a process of transferring the target data into a final transfer format, for example: receiving data, removing repeated data and useless data, determining a transfer target, determining a final transfer format, transferring and storing.
In this embodiment, the second transfer process refers to a process of the current transfer format of the target data, for example: receive data-save and save.
In this embodiment, the third transfer process refers to a process of transferring from the current transfer format to the final transfer format.
In this embodiment, the stage transfer feature refers to a feature of each stage in the transfer process, such as the process of eliminating data, the data is reduced, and the remaining data is automatically rearranged.
The beneficial effects of the technical scheme are as follows: the first transfer process of the final transfer format and the second transfer process of the current transfer format are determined, the third transfer process transferred from the current transfer format to the final transfer format is determined, and the phase transfer characteristics are obtained, so that the phase parameters are determined, the transfer process can be visualized, the transfer process is convenient to observe and control, and the transfer efficiency is improved.
Example 6:
the invention provides a data storage method for reducing storage transfer time, which optimizes transfer stage and stage parameters based on preset transfer logic, obtains a target transfer mode, stores target data by using the target transfer mode, and comprises the following steps:
acquiring target logic parameter information of a preset dump logic;
extracting target unloading step and target stage parameters based on the target logic parameter information;
performing first comparison on the target transfer step and the transfer stage to obtain a process deviation degree;
performing second comparison on the target stage parameter and the stage parameter to obtain a parameter difference value;
optimizing the transfer stage and stage parameters based on the process deviation degree and the parameter difference value, acquiring a target transfer mode, and storing target data by using the target transfer mode.
In this embodiment, the preset dump logic is a rule that is prescribed in advance and needs to be adhered to in the process of dump, for example, the amount of target data received at one time cannot exceed 100.
In this embodiment, the logic parameter information means, for example, that the speed of receiving data is 50 pieces/second.
In this embodiment, the process deviation means whether the restocking stage deviates from the target restocking step, and if the deviation is too large, the greater the deviation.
In this embodiment, the parameter difference is 5 seconds, for example, the target phase parameter receiving data time is 5 seconds, and the phase parameter receiving data time is 10 seconds, then the parameter difference is 5 seconds.
In this embodiment, the transfer mode includes: copy storage, distributed storage, and USB flash disk storage.
The beneficial effects of the technical scheme are as follows: the target transfer step and the target stage parameters are extracted and respectively compared with the transfer stage and the stage parameters, the process deviation degree and the parameter difference value are obtained, the transfer stage and the stage parameters are optimized, the target storage mode is obtained, the proper storage mode can be rapidly and accurately determined, the rapidity of data storage is ensured, and the storage time is saved.
Example 7:
the invention provides a data storage method for reducing storage transfer time, which judges the association degree between each data item according to the data attribute between each data item, and comprises the following steps:
analyzing the relation between every two data items, and establishing a first relation network;
recording the position of each data item on the first relation network as a corresponding first network node;
acquiring a plurality of first network nodes contained in a first relation network, and respectively acquiring node relations between each target first network node and the rest target second network nodes in the first relation network;
based on the node relation, combining the data items with the connection effect to form a data pair;
performing alignment training on a plurality of data items contained in each data pair;
acquiring data attributes of the aligned data items, and comparing information to acquire attribute difference points between two adjacent data items;
determining the difference proportion corresponding to the attribute difference point between two adjacent data items;
and determining the association degree between the data items according to the difference specific gravity.
In this embodiment, the first relationship network refers to a relationship network formed by independent or related relationships among data, such as sub-data, data factors, and data items, where sub-data 1 is connected to sub-data 2, and sub-data 2 is connected to sub-data 3 and 4, so that the relationship network can be formed.
In this embodiment, the names of the data items include a number, an alias, a profile, a length, a type, and a value range of the data items.
In this embodiment, the first network node refers to a node participating in the communication process, and the network node is a point where multiple lines/paths intersect or connect, and may be an intersection point.
Wherein each data item corresponds to a node on the relational network, for example, a data item corresponds to an access node.
In this embodiment, the second network node refers to all nodes except the selected target node.
In this embodiment, the node relationship includes: parent node, child node, sibling node.
In this embodiment, the linking effect means that in an interconnected system, a small initial data item may produce other corresponding linkages, which may be continuous or may simply cause a causal response.
For example, the type of a data item determines the range of values for the data item.
In this embodiment, a data pair refers to a collection of data elements that are identical in nature or identical in attribute.
In this embodiment, alignment training refers to arranging data of each type in a certain rule in memory space, instead of arranging immediately after each other, for example, requiring that the address of an object of a certain type must be a multiple of a certain value K (2, 4, 8).
In this embodiment, the attribute difference point refers to which differences exist between the respective attributes of two data items, and if any data items have an operation constraint, and if any, there is no operation constraint.
In this embodiment, the differential specific gravity refers to whether the specific gravity of the attribute differential point is large or small in all the differential points, and if the differential specific gravity is too large, the association degree of the two data items is small, wherein the differential specific gravity is calculated as follows:
；
wherein L01 represents the absolute value of the data item difference corresponding to the adjacent two data items; l02 represents the sum of all data difference absolute values aligned; lmax represents the maximum absolute value of all data difference absolute values aligned;represents the fine tuning coefficient and takes a value of 0.1.
The beneficial effects of the technical scheme are as follows: through marking the first network node on the constructed relation network by each data line, the relation between the target first network node and the rest target second network nodes is obtained, data pairs are formed, and alignment training is carried out, so that the data can be tidier and convenient to observe, meanwhile, information comparison is carried out on the data attributes, attribute difference points between adjacent data items are obtained, the degree of association between the data items is determined, the data characteristics between the data items can be effectively known, and objective and real description of the data relation is facilitated.
Example 8:
the invention provides a data storage method for reducing storage transfer time, which further comprises the following steps of:
acquiring the flow variation condition of a relevant network layer of a client when receiving target data;
evaluating the stability coefficient of each network layer according to the flow variation condition of the relevant network layer of the client when receiving the target data;
and evaluating the probability of success of the client to the transmission of the target data according to the stability coefficient of each network layer:
；
wherein Q is expressed as the probability of successful transmission of the target data by the client,expressed as performance parameters of the client, M expressed as the number of relevant network layers in the client, j expressed as the j-th present backbone network layer,/and%>Expressed as steady state factor, ">Stability factor denoted as j-th related network layer,>the operating strength denoted as j-th related network layer;
and determining whether the probability of success of the transmission of the client to the target data is greater than or equal to a preset probability, if so, not carrying out subsequent operation, and if not, carrying out network optimization on a relevant network layer of the client to ensure successful transmission of the target data.
In this embodiment, the network layer refers to trying to transfer data from a source to a destination through several intermediate nodes, thereby providing the most basic end-to-end data transfer service to the transport layer.
In this embodiment, the performance may be how fast the client is accepting the data.
In this embodiment, the working strength refers to the amount of data received by the network layer in a certain period of time, and the greater the received data, the greater the working strength, and the less the received data, the smaller the representative working strength.
In this embodiment, the flow rate variation refers to the amount of data received in a predetermined time.
The larger the flow fluctuation is, the unstable the network layer is represented, the smaller the stability coefficient is, the smaller the flow fluctuation is, the larger the stability coefficient is represented.
In this embodiment, the preset probability is 50%.
The beneficial effects of the technical scheme are as follows: the data can be effectively transmitted by calculating the probability of success of the client to the transmission of the target data, so that the problem of missing data or data missing caused by unstable network layers of the data transmission terminal is avoided, and the integrity and stability of the data transmission are improved.
Example 9:
the invention provides a data storage method for reducing storage transfer time, which carries out network optimization on a relevant network layer of a client to ensure successful transmission of target data, and comprises the following steps:
acquiring operation data and node attribute data of each node in a relevant network layer of a client;
determining a first target node set with the node computing resource occupation amount larger than a first preset value in each node based on the operation data of each node;
determining node attribute parameters of each node based on the node attribute data of each node;
determining a second target node set with the node attribute parameter larger than a second preset value in each node based on the node attribute parameter of each node, and determining a third target node based on the second target node set and the first target node set;
and determining the running state of the third target node based on the running data and the node attribute data of the third target node, and performing optimization processing on the third target node to ensure successful transmission of the target data when the running state is determined to meet the preset running state.
In this embodiment, the operational data refers to node computing resource data.
In this embodiment, the node attribute data is a node type parameter, a node connection number, such as 5 connections to node a.
In this embodiment, the node attribute parameter is used to characterize the running performance of the node, and the greater the node attribute parameter is, the worse the running performance of the node is.
In this embodiment, the first preset value is 5.
In this embodiment, the first set of target nodes is a set of all nodes having a node resource occupancy greater than 5.
In this embodiment, the second preset value refers to, for example, 21 minutes and 30 seconds at 8 am when reaching a certain node.
In this embodiment, the third set of target nodes refers to nodes in the intersection of the first set of target nodes and the second set of target nodes.
In this embodiment, the node computing resources include: CPU resource occupation data, storage resource occupation data and network resource occupation data.
In this embodiment, the operating state may be on-stream, running, stopping, stopped.
In this embodiment, the preset status condition is that it is being started.
The beneficial effects of the technical scheme are as follows: and determining a first target point set and a second target point set by acquiring operation data and node attribute data of nodes in the network layer, so as to determine a third target point set, and optimizing the third target point set when the third target point set meets a preset operation state, so that the transmission of target data can be ensured, and the transmission probability and the transmission time can be improved.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (8)
1. A data storage method for reducing storage transfer time, the method comprising:
step 1: obtaining original data, removing repeated data and useless data in the original data, and obtaining target data;
step 2: determining a transfer target for the target data, and determining a final transfer format of the target data based on the transfer target;
step 3: determining a transfer phase and phase parameters of a current transfer mode according to a final transfer format and a current transfer format of target data;
step 4: optimizing a transfer stage and stage parameters based on a preset transfer logic, acquiring a target transfer mode, and storing target data by using the target transfer mode;
the method for optimizing the transfer stage and the stage parameters based on the preset transfer logic, obtaining a target transfer mode, and storing target data by using the target transfer mode comprises the following steps:
acquiring target logic parameter information of a preset dump logic;
extracting target unloading step and target stage parameters based on the target logic parameter information;
performing first comparison on the target transfer step and the transfer stage to obtain a process deviation degree;
performing second comparison on the target stage parameter and the stage parameter to obtain a parameter difference value;
optimizing the transfer stage and stage parameters based on the process deviation degree and the parameter difference value, acquiring a target transfer mode, and storing target data by using the target transfer mode.
2. The data storage method for reducing a storage transfer time according to claim 1, wherein acquiring the original data, removing the duplicate data and the useless data in the original data, and acquiring the target data, comprises:
acquiring original data from a data source, and counting the original data;
acquiring a plurality of data items of the original data based on the statistical result, and performing first elimination by taking a first repeated data item in the plurality of data items as repeated data;
judging the association degree between the data items according to the data attribute between the data items, and performing second elimination by taking a second data item with the association degree smaller than a preset threshold value as useless data;
and acquiring the original data after the first eliminating process and the second eliminating process and taking the original data as target data.
3. The data storage method for reducing storage transfer time of claim 1, wherein determining a transfer target for the target data, and determining a final transfer format of the target data based on the transfer target, comprises:
acquiring a data transfer request of a client, and determining a transfer path of target data in the client according to the transfer request;
determining a dump target for target data based on the dump path;
determining the configuration information of the transfer requirement for the target data based on the transfer target of the target data;
determining a transfer strategy corresponding to the target data according to the transfer demand configuration information of the target data;
and determining the final transfer format of the target data according to the transfer strategy.
4. The data storage method for reducing storage transfer time according to claim 3, further comprising, after obtaining the data transfer request of the client:
acquiring the request sending time of the data transfer request;
judging whether the request sending time of the data transfer request is in a preset time period or not;
if yes, a first storage security policy for the target data is obtained;
if not, a second storage security policy for the target data is obtained.
5. The data storage method for reducing storage transfer time according to claim 1, wherein determining transfer phases and phase parameters of a current transfer mode according to a final transfer format and a current transfer format of target data comprises:
acquiring a first transfer process of a final transfer format of target data and a second transfer process of a current transfer format;
determining a third transfer process for transferring the target data from the current transfer format to a final transfer format according to the first transfer process and the second transfer process;
acquiring a phase transfer characteristic corresponding to the third transfer process;
and determining phase parameters according to the phase transfer characteristics.
6. The data storage method for reducing a storage transfer time according to claim 2, wherein determining the degree of association between the respective data items based on the data attributes between the respective data items, comprises:
analyzing the relation between every two data items, and establishing a first relation network;
recording the position of each data item on the first relation network as a corresponding first network node;
acquiring a plurality of first network nodes contained in a first relation network, and respectively acquiring node relations between each target first network node and the rest target second network nodes in the first relation network;
based on the node relation, combining the data items with the connection effect to form a data pair;
performing alignment training on a plurality of data items contained in each data pair;
acquiring data attributes of the aligned data items, and comparing information to acquire attribute difference points between two adjacent data items;
determining the difference proportion corresponding to the attribute difference point between two adjacent data items;
and determining the association degree between the data items according to the difference specific gravity.
7. The data storage method for reducing a storage transfer time according to claim 1, further comprising, in storing the target data by using the target transfer method:
acquiring the flow variation condition of a relevant network layer of a client when receiving target data;
evaluating the stability coefficient of each network layer according to the flow variation condition of the relevant network layer of the client when receiving the target data;
and evaluating the probability of success of the client to the transmission of the target data according to the stability coefficient of each network layer:
；
wherein Q is expressed as the probability of successful transmission of the target data by the client,expressed as performance parameters of the client, M expressed as the number of relevant network layers in the client, j expressed as the j-th present backbone network layer,/and%>Expressed as steady state factor, ">Stability factor denoted as j-th related network layer,>the operating strength denoted as j-th related network layer;
and determining whether the probability of success of the transmission of the client to the target data is greater than or equal to a preset probability, if so, not carrying out subsequent operation, and if not, carrying out network optimization on a relevant network layer of the client to ensure successful transmission of the target data.
8. The data storage method for reducing storage transfer time according to claim 7, wherein performing network optimization on the relevant network layer of the client to ensure successful transmission of the target data comprises:
acquiring operation data and node attribute data of each node in a relevant network layer of a client;
determining a first target node set with the node computing resource occupation amount larger than a first preset value in each node based on the operation data of each node;
determining node attribute parameters of each node based on the node attribute data of each node;
determining a second target node set with the node attribute parameter larger than a second preset value in each node based on the node attribute parameter of each node, and determining a third target node based on the second target node set and the first target node set;
and determining the running state of the third target node based on the running data and the node attribute data of the third target node, and performing optimization processing on the third target node to ensure successful transmission of the target data when the running state is determined to meet the preset running state.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311339956.XA CN117093576B (en) | 2023-10-17 | 2023-10-17 | Data storage method for reducing storage transfer time |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311339956.XA CN117093576B (en) | 2023-10-17 | 2023-10-17 | Data storage method for reducing storage transfer time |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117093576A CN117093576A (en) | 2023-11-21 |
| CN117093576B true CN117093576B (en) | 2024-01-05 |
Family
ID=88771965
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311339956.XA Active CN117093576B (en) | 2023-10-17 | 2023-10-17 | Data storage method for reducing storage transfer time |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117093576B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3144821A1 (en) * | 2015-09-18 | 2017-03-22 | Frank Sax | Method for generating electronic documents |
| CN115098574A (en) * | 2022-06-27 | 2022-09-23 | 招商银行股份有限公司 | Data unloading method, system, terminal device and storage medium for multiple data sources |
| CN115314566A (en) * | 2022-07-26 | 2022-11-08 | 北京金山云网络技术有限公司 | Data transmission method and device, electronic equipment and storage medium |
-
2023
- 2023-10-17 CN CN202311339956.XA patent/CN117093576B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3144821A1 (en) * | 2015-09-18 | 2017-03-22 | Frank Sax | Method for generating electronic documents |
| CN115098574A (en) * | 2022-06-27 | 2022-09-23 | 招商银行股份有限公司 | Data unloading method, system, terminal device and storage medium for multiple data sources |
| CN115314566A (en) * | 2022-07-26 | 2022-11-08 | 北京金山云网络技术有限公司 | Data transmission method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117093576A (en) | 2023-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101009516B (en) | A method, system and device for data synchronization | |
| US11392600B2 (en) | Techniques for in memory key range searches | |
| US11971880B2 (en) | Techniques for in-memory data searching | |
| CN106446168B (en) | A kind of load client realization method of Based on Distributed data warehouse | |
| CN112988683A (en) | Data processing method and device, electronic equipment and storage medium | |
| WO2022083436A1 (en) | Data processing method and apparatus, and device and readable storage medium | |
| CN111753016A (en) | Data processing method, device, system and computer readable storage medium | |
| WO2022048357A1 (en) | Transaction endorsement method and apparatus, and storage medium | |
| US20230334009A1 (en) | Data archiving method and apparatus, device, and storage medium | |
| CN112381649A (en) | Transaction consensus method, device and equipment based on block chain | |
| CN103503388B (en) | A kind of distributed queue's message read method and equipment, system | |
| CN117093576B (en) | Data storage method for reducing storage transfer time | |
| CN112333291A (en) | File retransmission prevention method | |
| US9697089B2 (en) | Backup method and storage medium having backup program stored therein | |
| EP3349416B1 (en) | Relationship chain processing method and system, and storage medium | |
| CN109635193B (en) | Book reading sharing platform | |
| CN114676229B (en) | Technical improvement major repair project file management system and management method | |
| CN115599295A (en) | Node capacity expansion method and device of storage system | |
| CN117376364A (en) | Data processing method and related equipment | |
| CN112131180B (en) | Data reporting method, device and storage medium | |
| CN115167913A (en) | Operating system layering method, computing device and storage medium | |
| JPH11127147A (en) | Information contradiction discrimination, correction device and method in storage exchange electronic conference system and medium with recorded with information contradiction and correction program | |
| CN112347046A (en) | Method for acquiring creation time of file in distributed system | |
| CN112541027B (en) | Data processing method and device | |
| EP4369658A2 (en) | Method of storing blockchain transaction data using fountain codes and apparatus for the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |