CN116974468A - Equipment data storage management method and system based on big data - Google Patents
Equipment data storage management method and system based on big data Download PDFInfo
- Publication number
- CN116974468A CN116974468A CN202310878824.8A CN202310878824A CN116974468A CN 116974468 A CN116974468 A CN 116974468A CN 202310878824 A CN202310878824 A CN 202310878824A CN 116974468 A CN116974468 A CN 116974468A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- equipment
- storage position
- migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 124
- 238000007726 management method Methods 0.000 title claims abstract description 113
- 238000012544 monitoring process Methods 0.000 claims abstract description 37
- 230000005540 biological transmission Effects 0.000 claims abstract description 25
- 230000005012 migration Effects 0.000 claims description 83
- 238000013508 migration Methods 0.000 claims description 83
- 238000000034 method Methods 0.000 claims description 18
- 238000004458 analytical method Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 6
- 230000010354 integration Effects 0.000 claims description 6
- 238000012216 screening Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 230000009471 action Effects 0.000 description 3
- 238000013523 data management Methods 0.000 description 3
- 239000002699 waste material Substances 0.000 description 3
- 238000004140 cleaning Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0623—Securing storage systems in relation to content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0653—Monitoring storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Abstract
The invention discloses a device data storage management method and system based on big data, which relate to the technical field of device data storage management and comprise the following steps: s1: for the equipment in the access equipment data storage management system, equipment numbers are allocated, and uploading data of each equipment is analyzed in a mode of reading configuration files; s2: monitoring storage management operation of equipment data, acquiring storage management operation of corresponding data of each equipment in an equipment data storage system, and integrating the acquired storage management operation; s3: updating the actual storage capacity of each storage position, acquiring the data access operation of each storage position, extracting the data storage time of the equipment data in each storage position, and judging whether the stored data in each storage position is migrated or not; s4: and managing the data transmission channels among the storage positions according to the judging result of whether the stored data in each storage position is migrated or not.
Description
Technical Field
The invention relates to the technical field of equipment data storage management, in particular to an equipment data storage management method and system based on big data.
Background
With the development of big data technology, the size and complexity of device data are also increasing. Because the equipment data has the characteristics of mass, continuity, diversity, isomerism and the like, the data needs to be collected and processed in real time, and the data flow is filtered and screened in real time so as to extract key information and provide efficient and reliable storage and management.
The efficiency of storage management systems is critical to data storage management, which currently mainly takes into account the performance of the storage system to cope with ever-increasing amounts of data and access requests. However, life cycle management on data use conditions in data storage management is not perfect, and currently, judgment on a device data storage position by selecting a data type or a history storage position of each device data in history storage data is mainly considered for automatic data storage, but the judgment mode of the storage position is too one-sided and cannot meet the actual requirement of a data storage management system. When the storage position of the data of the device is not reasonable enough, the data management difficulty or the system performance reduction may be caused, so that the device frequently reads and writes the data, the load and the loss of the device are increased, and even the data access efficiency of the system may be affected.
Therefore, in order to solve the above problems or part of the problems, the present invention provides a device data storage management method and system based on big data.
Disclosure of Invention
The invention aims to provide a device data storage management method and system based on big data, so as to solve the problems in the background technology.
In order to solve the technical problems, the invention provides the following technical scheme: a device data storage management method based on big data comprises the following steps:
s1: for the equipment in the access equipment data storage management system, equipment numbers are allocated, and uploading data of each equipment is analyzed in a mode of reading configuration files;
s2: monitoring storage management operation of equipment data, acquiring storage management operation of corresponding data of each equipment in an equipment data storage system, and integrating the acquired storage management operation;
s3: updating the actual storage capacity of each storage position, acquiring the data access operation of each storage position, extracting the data storage time of the equipment data in each storage position, and judging whether the stored data in each storage position is migrated or not;
s4: and managing the data transmission channels among the storage positions according to the judging result of whether the stored data in each storage position is migrated or not so as to ensure the smooth transmission and storage of the data.
Further, the step S1 includes:
step S1-1: respectively allocating different equipment numbers to each equipment of the connection system based on corresponding working contents;
step S1-2: and transmitting the equipment data to a processing platform, and analyzing the equipment data according to a predefined configuration file.
Further, the step S2 includes:
step S2-1: monitoring the data set uploaded by each device, and acquiring all storage management operations of the data uploaded by each numbering device in a device storage management system to respectively acquire a storage operation set of each data storage corresponding to the data uploaded by each numbering device;
step S2-2: integrating the storage operation sets of the devices in each device data set to obtain { C } 1 ,C 2 ，...，C n The storage operation set with the highest repeated occurrence number in each numbering device is regarded as a first storage operation set corresponding to the distribution of each device data set, and each storage operation in the first storage operation set is set as one storage operation node corresponding to each device data set;
step S2-3: integrating storage positions corresponding to the first storage operation set of each device data set to obtain a storage operation node C at any one i Storage information Z relating to its device data set Ci The method comprises the steps of carrying out a first treatment on the surface of the Wherein C is i Representing a store operation of an ith operation node, Z Ci Storage location information corresponding to a storage operation of the i-th operation node;
step S2-4: for each device data setAcquiring all operation nodes of the corresponding storage positions, and integrating the storage position data corresponding to each acquired device data set to obtain storage operation information of each device data set for each storage position in the device data storage management system; the storage operation information includes a storage operation path R of each device data at each storage location, r=z C1 →...→Z Cj →...→Z Cm Wherein Z is C1 、...、Z Cj 、...、Z Cm Storage location information corresponding to storage operations of the 1 st, i., j, m operation nodes are respectively indicated. The integrated storage location information may be used for subsequent data management operations.
Further, the step S3 includes:
step S3-1: when the data storage operation of the data of each device is finished, the storage capacity change in each storage position is updated in real time;
by analyzing the storage capacity of the storage locations, it is convenient to determine whether data cleaning, archiving, or migration is required. Therefore, the use of the storage position can be effectively managed, and data overflow or resource waste is avoided.
Step S3-2: setting the data storage time of any device data S corresponding to a data storage position P to be longer than D, collecting access operation generated in each data storage process of the device data S, and sequencing according to the storage operation paths of the device data in each storage position to obtain an accessed operation set of the device data S in a device data storage management system about each storage position;
step S3-3: setting a storage location in a device data storage management system to p= { P 1 ,P 2 ，...，P h Y which is less than a preset threshold value for the number of operations to be accessed 1 The device data storage locations of (1) are recorded as a first storage location set, and the number of accessed operations exceeds y of a preset threshold value 2 Is noted as a second set of storage locations;
step S3-4: judging the equipment data of which the accessed operation times do not belong to the preset threshold range, and analyzing whether the data migration is performed comprises the following steps:
step S3-4-1: analyzing the storage time of the equipment data in the current storage position, if the data storage time of the current equipment data exceeds a preset threshold T, executing the next analysis, otherwise, temporarily not performing data migration;
step S3-4-2: analyzing the current storage position of the equipment data, and judging whether the current data storage position can meet the access frequency of the current equipment data;
if the storage position of the current device data belongs to the first storage position set, executing the next analysis when the storage position of the device data is a high-speed storage medium; otherwise, not carrying out data migration;
if the storage position of the current device data belongs to the second storage position set, executing the next analysis when the storage position of the device data is a low-speed storage medium; otherwise, not carrying out data migration;
step S3-4-3: analyzing the change rule of the accessed times of the device data in any time period of the current storage position according to the following formula:
A(t)＝∑[η(i)*A(t-i)]+∑[μ(j)*λ(t-v)]+c；
wherein A (t) represents the number of times the device data is accessed in the current storage position within time t, eta (i) represents the weight of the previous i time points, mu (j) represents the weight of the jth error term, lambda (t-v) is the jth error term, and c represents a constant which is preset by a related technician;
monitoring the predicted value of the accessed frequency of the equipment data in the current storage position, and according to the variation trend of the accessed frequency of the equipment data in the current storage position, when A (t) is in the completion time of completing data migration, reaching a preset threshold range y 1 Or y 2 If the data is not migrated, the data is not migrated; otherwise, data migration is carried out, and the next step is executed;
preferably, the amount of resources required for migration of device data may also be analyzed.
Further, the step S4 includes:
step S4-1: marking the front storage according to the storage operation path of the device data, if any storage position in the storage operation path of the device data, which is in front of the current storage position, meets the data migration condition of the device data, wherein the position has priority in the storage position selection of the data migration;
if a front storage position in a storage operation path of the device data meets the data migration requirement and the backup of the current device data exists in the position, the device data is not migrated, only the device data in the current storage position is deleted, and an access address of the device data is modified to the front storage position so as to reduce the workload of a system;
step S4-2: if the storage position which does not meet the data migration of the device data is in front of the current storage position in the storage operation path of the device data, screening the storage position types meeting the current accessed and storage time requirements of the device data in the device data storage management system;
step S4-3: analyzing the available storage capacity of the storage positions obtained by screening in the step S4-2, and judging whether the available storage capacity is larger than the storage occupation amount required by the equipment data or not so that the actual storage capacity of each storage position can meet the storage requirement; managing data transmission among the storage positions according to the judgment result; and managing the data transmission channels among the storage positions to ensure smooth transmission and storage of data. In addition, operations such as data backup, data migration, data replication and the like may be involved, and security and reliability of device data need to be ensured.
Step S4-4: and feeding back the data storage position selection of the automatic data storage according to the data migration result, and judging the influence of the addition of the historical data migration on the selection of the automatic data storage position of the equipment number corresponding to the equipment data in the equipment data storage management system.
By combining the influence of data migration in the history storage management data during the automated data storage, the reliability of the automated data storage management can be further improved.
A big data based device data storage management system, the system comprising: the device comprises a device account management module, a data storage operation acquisition module, a data migration discrimination module and a storage position management module;
the device account management module is used for recording the devices of each access system and distributing corresponding device numbers based on the device types or service contents corresponding to the devices;
the data storage operation acquisition module is used for monitoring the storage management operation of the uploading data of each device and acquiring the storage operation data in the storage process;
the data migration judging module is used for replacing storage related information of each device data and judging migration of stored device data in each storage position;
the storage position management module is used for monitoring the storage capacity in each storage position in real time and managing the storage and transmission of the equipment data according to the real-time storage capacity condition in each storage position and the migration judgment result of the stored data in each storage position.
Further, the data storage operation acquisition module comprises an equipment data identification unit, a storage operation monitoring unit and a storage operation integration unit;
the device data identification unit is used for identifying each device accessed to the device data storage management system, and receiving the data uploaded by the device with the system authority through identifying the device interface or the device number;
the storage operation monitoring unit is used for monitoring the storage operation of the data uploaded by each device in the device data storage management system and acquiring the storage operation corresponding to each device data;
the storage operation integration unit is used for integrating the storage operation of the acquired device data with the storage information corresponding to each operation.
Further, the data migration discriminating module comprises a stored information analyzing unit and a data migration discriminating unit;
the storage information analysis unit is used for acquiring the equipment data related storage information required by data migration discrimination and respectively analyzing the storage information affecting the data migration discrimination;
the data migration judging unit is used for judging whether the equipment data of the current storage position are migrated according to the storage information which influences the data migration judgment, and transmitting the judging result to the storage position management module.
Further, the storage position management module comprises a storage position monitoring unit, a data transmission management unit and a storage data feedback unit;
the storage position monitoring unit is used for monitoring each storage position in the equipment data storage management system and updating the real-time storage capacity of each storage position;
the data transmission management unit is used for managing data transmission among all storage positions according to the data migration discrimination result;
the data storage feedback unit is used for feeding back the data storage position selection of the automatic data storage according to the data migration judging result, and further, when the automatic data storage is carried out, the influence on the data migration in the historical storage management data is required to be combined.
Compared with the prior art, the invention has the following beneficial effects:
the invention records the equipment of each access system through the equipment account management module, and allocates corresponding equipment numbers based on the equipment types or service contents corresponding to the equipment; so as to facilitate the management and tracking of each device and determine the attribution of the device data;
monitoring the storage management operation of the uploading data of each device through a data storage operation acquisition module, and acquiring the storage operation data in the storage process; the method is convenient for knowing the data storage condition of each device in real time, thereby effectively monitoring the storage operation of the uploaded data and finding out the storage abnormality in time;
replacing storage related information of each device data through a data migration judging module, and judging migration of stored device data in each storage position; analyzing the accessed condition of the stored data in the system, analyzing the rationality of the device data in the current storage position by combining with the influence factors such as a storage operation path, adjusting the storage position of the device data in time, and migrating the device data can ensure the control of the data storage cost, reduce the resource waste of the storage space and improve the utilization rate of storage resources;
the storage capacity in each storage position is monitored in real time through the storage position management module, and the storage and transmission of the equipment data are managed according to the real-time storage capacity condition in each storage position and the migration judgment result of the stored data in each storage position. The influence of data migration on the data storage of the equipment is considered, the reliability of automatic data storage management is further improved, the actual requirement of a data storage management system is met, and the data access efficiency of the system is improved.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, serve to explain the invention. In the drawings:
FIG. 1 is a schematic block diagram of a big data based device data storage management system according to the present invention;
fig. 2 is a flow chart of a device data storage management method based on big data according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention is further described with reference to fig. 1, 2 and embodiments.
Example 1: as shown in fig. 1, the present embodiment provides a device data storage management system based on big data, the system including: the device comprises a device account management module, a data storage operation acquisition module, a data migration discrimination module and a storage position management module;
the device account management module is used for recording the devices of each access system and distributing corresponding device numbers based on the device types or service contents corresponding to the devices;
the data storage operation acquisition module is used for monitoring the storage management operation of the uploading data of each device and acquiring the storage operation data in the storage process; the data storage operation acquisition module comprises an equipment data identification unit, a storage operation monitoring unit and a storage operation integration unit;
the device data identification unit is used for identifying each device accessed into the device data storage management system, and receiving the data uploaded by the device with the system authority through identifying the device interface or the device number;
the storage operation monitoring unit is used for monitoring the storage operation of the data uploaded by each device in the device data storage management system and acquiring the storage operation corresponding to each device data;
the storage operation integration unit is used for integrating the storage operation of the acquired device data with the storage information corresponding to each operation.
The data migration judging module is used for replacing storage related information of each device data and judging migration of stored device data in each storage position; the data migration discriminating module comprises a stored information analyzing unit and a data migration discriminating unit;
the storage information analysis unit is used for acquiring the equipment data related storage information required by data migration discrimination and respectively analyzing the storage information affecting the data migration discrimination;
the data migration discriminating unit is used for discriminating whether the equipment data of the current storage position is migrated or not according to the storage information which influences the data migration discrimination, and transmitting the discrimination result to the storage position management module.
The storage position management module is used for monitoring the storage capacity in each storage position in real time and managing the storage and transmission of the equipment data according to the real-time storage capacity condition in each storage position and the migration judgment result of the stored data in each storage position; the storage position management module comprises a storage position monitoring unit, a data transmission management unit and a storage data feedback unit;
the storage position monitoring unit is used for monitoring each storage position in the equipment data storage management system and updating the real-time storage capacity of each storage position;
the data transmission management unit is used for managing data transmission among the storage positions according to the data migration discrimination result;
and the stored data feedback unit is used for feeding back the data storage position selection of the automatic data storage according to the data migration discrimination result.
Example 2: as shown in fig. 2, the present embodiment provides a device data storage management method based on big data, which is implemented based on a device data storage management system based on big data in the embodiment, and specifically includes the following steps:
s1: for the equipment in the access equipment data storage management system, equipment numbers are allocated, and uploading data of each equipment is analyzed in a mode of reading configuration files;
step S1-1: respectively allocating different equipment numbers to each equipment of the connection system based on corresponding working contents;
step S1-2: and transmitting the equipment data to a processing platform, and analyzing the equipment data according to a predefined configuration file.
S2: monitoring storage management operation of equipment data, acquiring storage management operation of corresponding data of each equipment in an equipment data storage system, and integrating the acquired storage management operation;
step S2-1: monitoring the data set uploaded by each device, and acquiring all storage management operations of the data uploaded by each numbering device in a device storage management system to respectively acquire a storage operation set of each data storage corresponding to the data uploaded by each numbering device;
the storage management operation of the equipment data can be obtained by monitoring the methods such as system logs, API calls, data auditing or custom setting event triggers and the like;
step S2-2: integrating the storage operation sets of the devices in each device data set to obtain { C } 1 ,C 2 ，...，C n The storage operation set with the highest repeated occurrence number in each numbering device is regarded as a first storage operation set corresponding to the distribution of each device data set, and each storage operation in the first storage operation set is set as one storage operation node corresponding to each device data set;
step S2-3: integrating storage positions corresponding to the first storage operation set of each device data set to obtain a storage operation node C at any one i Storage information Z relating to its device data set Ci The method comprises the steps of carrying out a first treatment on the surface of the Wherein C is i Representing a store operation of an ith operation node, Z Ci Storage location information corresponding to a storage operation of the i-th operation node;
step S2-4: acquiring all operation nodes of storage positions corresponding to the equipment data sets, and integrating the acquired storage position data corresponding to the equipment data sets to obtain storage operation information of the equipment data sets for the storage positions in the equipment data storage management system; the storage operation information includes a storage operation path R of each device data at each storage location, r=z C1 →...→Z Cj →...→Z Cm Wherein Z is C1 、...、Z Cj 、...、Z Cm Storage location information corresponding to storage operations of the 1 st, i., j, m operation nodes are respectively indicated.
The integrated storage location information may be used for subsequent data management operations such as backup and restore, data migration, or optimizing storage resource allocation.
S3: updating the actual storage capacity of each storage position, acquiring the data access operation of each storage position, extracting the data storage time of the equipment data in each storage position, and judging whether the stored data in each storage position is migrated or not;
step S3-1: when the data storage operation of the data of each device is finished, the storage capacity change in each storage position is updated in real time;
by analyzing the storage capacity of a storage location, such as a hard disk, database, etc., it is convenient to determine whether data cleaning, archiving, or migration is required. Therefore, the use of the storage position can be effectively managed, and data overflow or resource waste is avoided.
Step S3-2: setting the data storage time of any device data S corresponding to a data storage position P to be longer than D, collecting access operation generated in each data storage process of the device data S, and sequencing according to the storage operation paths of the device data in each storage position to obtain an accessed operation set of the device data S in a device data storage management system about each storage position;
step S3-3: setting a storage location in a device data storage management system to p= { P 1 ,P 2 ，...，P h Y which is less than a preset threshold value for the number of operations to be accessed 1 The device data storage locations of (1) are recorded as a first storage location set, and the number of accessed operations exceeds y of a preset threshold value 2 Is noted as a second set of storage locations; if there is device data S Pk Storage position P k The number of accesses exceeding a preset threshold y 2 Wherein k is [1, h ]]The method comprises the steps of carrying out a first treatment on the surface of the For the equipment data S Pk Marking a previous storage position of a storage operation path R of each storage position;
step S3-4: judging the equipment data of which the accessed operation times do not belong to the preset threshold range, and analyzing whether the data migration is performed comprises the following steps:
step S3-4-1: analyzing the storage time of the equipment data in the current storage position, if the data storage time of the current equipment data exceeds a preset threshold T, executing the next analysis, otherwise, temporarily not performing data migration;
step S3-4-2: analyzing the current storage position of the equipment data, and judging whether the current data storage position can meet the access frequency of the current equipment data;
if the storage position of the current device data belongs to the first storage position set, executing the next analysis when the storage position of the device data is a high-speed storage medium; otherwise, not carrying out data migration;
if the storage position of the current device data belongs to the second storage position set, executing the next analysis when the storage position of the device data is a low-speed storage medium; otherwise, not carrying out data migration;
for example, for y, the number of accessed operations exceeds a preset threshold 2 The device data storage location of (2) is recorded as a second storage location set, and if the current storage location is a high-speed storage medium, for example, a Solid State Disk (SSD) or a memory database which can provide lower access delay and faster data retrieval speed is not migrated; if the current storage location is a low-speed storage medium, such as a tape library or cold storage with lower storage cost, performing the next analysis;
step S3-4-3: analyzing the change rule of the accessed times of the device data in any time period of the current storage position according to the following formula:
A(t)＝∑[η(i)*A(t-i)]+∑[μ(j)*λ(t-v)]+c；
wherein A (t) represents the number of times the device data is accessed in the current storage position within time t, eta (i) represents the weight of the previous i time points, mu (j) represents the weight of the jth error term, lambda (t-v) is the jth error term, and c represents a constant which is preset by a related technician;
monitoring the predicted value of the accessed frequency of the equipment data in the current storage position, and according to the variation trend of the accessed frequency of the equipment data in the current storage position, when A (t) is in the completion time of completing data migration, reaching a preset threshold range y 1 Or y 2 If the data is not migrated, the data is not migrated; whether or notThen, data migration is carried out, and the next step is executed;
s4: and managing the data transmission channels among the storage positions according to the judging result of whether the stored data in each storage position is migrated or not.
Step S4-1: marking the front storage according to the storage operation path of the device data, if any storage position in the storage operation path of the device data, which is in front of the current storage position, meets the data migration condition of the device data, wherein the position has priority in the storage position selection of the data migration;
if a front storage position in a storage operation path of the device data meets the data migration requirement and the backup of the current device data exists in the position, the device data is not migrated, only the device data in the current storage position is deleted, and an access address of the device data is modified to the front storage position so as to reduce the workload of a system;
step S4-2: if the storage position which does not meet the data migration of the device data is in front of the current storage position in the storage operation path of the device data, screening the storage position types meeting the current accessed and storage time requirements of the device data in the device data storage management system;
step S4-3: analyzing the available storage capacity of the storage positions obtained by screening in the step S4-2, and judging whether the available storage capacity is larger than the storage occupation amount required by the equipment data or not so that the actual storage capacity of each storage position can meet the storage requirement; managing data transmission among the storage positions according to the judgment result; and managing the data transmission channels among the storage positions to ensure smooth transmission and storage of data. In addition, operations such as data backup, data migration, data replication and the like may be involved, and security and reliability of device data need to be ensured.
Step S4-4: and feeding back the data storage position selection of the automatic data storage according to the data migration result, and judging the influence of the addition of the historical data migration on the selection of the automatic data storage position of the equipment number corresponding to the equipment data in the equipment data storage management system.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Finally, it should be noted that: the foregoing description is only a preferred embodiment of the present invention, and the present invention is not limited thereto, but it is to be understood that modifications and equivalents of some of the technical features described in the foregoing embodiments may be made by those skilled in the art, although the present invention has been described in detail with reference to the foregoing embodiments. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A device data storage management method based on big data is characterized in that: the method comprises the following steps:
s1: for the equipment in the access equipment data storage management system, equipment numbers are allocated, and uploading data of each equipment is analyzed in a mode of reading configuration files;
s2: monitoring storage management operation of equipment data, acquiring storage management operation of corresponding data of each equipment in an equipment data storage system, and integrating the acquired storage management operation;
s3: updating the actual storage capacity of each storage position, acquiring the data access operation of each storage position, extracting the data storage time of the equipment data in each storage position, and judging whether the stored data in each storage position is migrated or not;
s4: and managing the data transmission channels among the storage positions according to the judging result of whether the stored data in each storage position is migrated or not.
2. The big data based device data storage management method of claim 1, wherein: the S1 comprises the following steps:
step S1-1: respectively allocating different equipment numbers to each equipment of the connection system based on corresponding working contents;
step S1-2: and transmitting the equipment data to a processing platform, and analyzing the equipment data according to a predefined configuration file.
3. The big data based device data storage management method of claim 1, wherein: the step S2 comprises the following steps:
step S2-1: monitoring the data set uploaded by each device, and acquiring all storage management operations of the data uploaded by each numbering device in a device storage management system to respectively acquire a storage operation set of each data storage corresponding to the data uploaded by each numbering device;
step S2-2: integrating the storage operation sets of the devices in each device data set to obtain { C } 1 ,C 2 ，...，C n The storage operation set with the highest repeated occurrence number in each numbering device is regarded as a first storage operation set corresponding to the distribution of each device data set, and each storage operation in the first storage operation set is set as one storage operation node corresponding to each device data set;
step S2-3: integrating storage positions corresponding to the first storage operation set of each device data set to obtain a storage operation node C at any one i Storage information Z relating to its device data set Ci The method comprises the steps of carrying out a first treatment on the surface of the Wherein C is i Representing a store operation of an ith operation node, Z Ci Storage location information corresponding to a storage operation of the i-th operation node;
step S2-4: acquiring all operation nodes of storage positions corresponding to the equipment data sets, and integrating the acquired storage position data corresponding to the equipment data sets to obtain storage operation information of the equipment data sets for the storage positions in the equipment data storage management system; the storage operation information includes a storage operation path R of each device data at each storage location, r=z C1 →...→Z Cj →...→Z Cm Wherein Z is C1 、...、Z Cj 、...、Z Cm Storage location information corresponding to storage operations of the 1 st, i., j, m operation nodes are respectively indicated.
4. The big data based device data storage management method of claim 1, wherein: the step S3 comprises the following steps:
step S3-1: when the data storage operation of the data of each device is finished, the storage capacity change in each storage position is updated in real time;
step S3-2: setting the data storage time of any device data S corresponding to a data storage position P to be longer than D, collecting access operation generated in each data storage process of the device data S, and sequencing according to the storage operation paths of the device data in each storage position to obtain an accessed operation set of the device data S in a device data storage management system about each storage position;
step S3-3: setting a storage location in a device data storage management system to p= { P 1 ,P 2 ，...，P h Y which is less than a preset threshold value for the number of operations to be accessed 1 The device data storage locations of (1) are recorded as a first storage location set, and the number of accessed operations exceeds y of a preset threshold value 2 Is noted as a second set of storage locations;
step S3-4: judging the equipment data of which the accessed operation times do not belong to the preset threshold range, and analyzing whether the data migration is performed comprises the following steps:
step S3-4-1: analyzing the storage time of the equipment data in the current storage position, if the data storage time of the current equipment data exceeds a preset threshold T, executing the next analysis, otherwise, temporarily not performing data migration;
step S3-4-2: analyzing the current storage position of the equipment data, and judging whether the current data storage position can meet the access frequency of the current equipment data;
if the storage position of the current device data belongs to the first storage position set, executing the next analysis when the storage position of the device data is a high-speed storage medium; otherwise, not carrying out data migration;
if the storage position of the current device data belongs to the second storage position set, executing the next analysis when the storage position of the device data is a low-speed storage medium; otherwise, not carrying out data migration;
step S3-4-3: analyzing the change rule of the accessed times of the device data in any time period of the current storage position according to the following formula:
A(t)＝∑[η(i)*A(t-i)]+∑[μ(j)*λ(t-v)]+c；
wherein A (t) represents the number of times the device data is accessed in the current storage position within time t, eta (i) represents the weight of the previous i time points, mu (j) represents the weight of the jth error term, lambda (t-v) is the jth error term, and c represents a constant which is preset by a related technician;
monitoring the predicted value of the accessed frequency of the equipment data in the current storage position, and according to the variation trend of the accessed frequency of the equipment data in the current storage position, when A (t) is in the completion time of completing data migration, reaching a preset threshold range y 1 Or y 2 If the data is not migrated, the data is not migrated; otherwise, data migration is carried out, and the next step is executed.
5. The big data based device data storage management method of claim 1, wherein: the step S4 comprises the following steps:
step S4-1: marking the front storage according to the storage operation path of the device data, if any storage position in the storage operation path of the device data, which is in front of the current storage position, meets the data migration condition of the device data, wherein the position has priority in the storage position selection of the data migration;
step S4-2: if the storage position which does not meet the data migration of the device data is in front of the current storage position in the storage operation path of the device data, screening the storage position types meeting the current accessed and storage time requirements of the device data in the device data storage management system;
step S4-3: analyzing the available storage capacity of the storage position obtained by screening in the step S4-2, and judging whether the available storage capacity is larger than the required storage occupation amount of the equipment data; managing data transmission among the storage positions according to the judgment result;
step S4-4: and feeding back the data storage position selection of the automatic data storage according to the data migration result, and judging the influence of the addition of the historical data migration on the selection of the automatic data storage position of the equipment number corresponding to the equipment data in the equipment data storage management system.
6. A big data based device data storage management system, characterized in that: the system comprises: the device comprises a device account management module, a data storage operation acquisition module, a data migration discrimination module and a storage position management module;
the device account management module is used for recording the devices of each access system and distributing corresponding device numbers based on the device types or service contents corresponding to the devices;
the data storage operation acquisition module is used for monitoring the storage management operation of the uploading data of each device and acquiring the storage operation data in the storage process;
the data migration judging module is used for replacing storage related information of each device data and judging migration of stored device data in each storage position;
the storage position management module is used for monitoring the storage capacity in each storage position in real time and managing the storage and transmission of the equipment data according to the real-time storage capacity condition in each storage position and the migration judgment result of the stored data in each storage position.
7. The big data based device data storage management system of claim 6, wherein: the data storage operation acquisition module comprises an equipment data identification unit, a storage operation monitoring unit and a storage operation integration unit;
the device data identification unit is used for identifying each device accessed to the device data storage management system, and receiving the data uploaded by the device with the system authority through identifying the device interface or the device number;
the storage operation monitoring unit is used for monitoring the storage operation of the data uploaded by each device in the device data storage management system and acquiring the storage operation corresponding to each device data;
the storage operation integration unit is used for integrating the storage operation of the acquired device data with the storage information corresponding to each operation.
8. The big data based device data storage management system of claim 6, wherein: the data migration discriminating module comprises a stored information analyzing unit and a data migration discriminating unit;
the storage information analysis unit is used for acquiring the equipment data related storage information required by data migration discrimination and respectively analyzing the storage information affecting the data migration discrimination;
the data migration judging unit is used for judging whether the equipment data of the current storage position are migrated according to the storage information which influences the data migration judgment, and transmitting the judging result to the storage position management module.
9. The big data based device data storage management system of claim 6, wherein: the storage position management module comprises a storage position monitoring unit, a data transmission management unit and a storage data feedback unit;
the storage position monitoring unit is used for monitoring each storage position in the equipment data storage management system and updating the real-time storage capacity of each storage position;
the data transmission management unit is used for managing data transmission among all storage positions according to the data migration discrimination result;
and the stored data feedback unit is used for feeding back the data storage position selection of the automatic data storage according to the data migration discrimination result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310878824.8A CN116974468B (en) | 2023-07-18 | 2023-07-18 | Equipment data storage management method and system based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310878824.8A CN116974468B (en) | 2023-07-18 | 2023-07-18 | Equipment data storage management method and system based on big data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116974468A true CN116974468A (en) | 2023-10-31 |
| CN116974468B CN116974468B (en) | 2024-02-20 |
Family
ID=88480800
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310878824.8A Active CN116974468B (en) | 2023-07-18 | 2023-07-18 | Equipment data storage management method and system based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116974468B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140189196A1 (en) * | 2013-01-02 | 2014-07-03 | International Business Machines Corporation | Determining weight values for storage devices in a storage tier to use to select one of the storage devices to use as a target storage to which data from a source storage is migrated |
| US20160034197A1 (en) * | 2014-08-04 | 2016-02-04 | Fujitsu Limited | Data migration method and data migration device |
| WO2017092480A1 (en) * | 2015-12-04 | 2017-06-08 | 华为技术有限公司 | Data migration method and device |
| CN107340975A (en) * | 2017-07-13 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of method and device of file storage |
| US20180088870A1 (en) * | 2016-09-23 | 2018-03-29 | EMC IP Holding Company LLC | Method and device for storage management |
| CN111367469A (en) * | 2020-02-16 | 2020-07-03 | 苏州浪潮智能科技有限公司 | Layered storage data migration method and system |
| CN115562870A (en) * | 2022-10-25 | 2023-01-03 | 北京京航计算通讯研究所 | Method for constructing task node resources of cluster |
| CN115826877A (en) * | 2023-01-20 | 2023-03-21 | 中国华能集团清洁能源技术研究院有限公司 | Data object migration method and device in big data environment |
-
2023
- 2023-07-18 CN CN202310878824.8A patent/CN116974468B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140189196A1 (en) * | 2013-01-02 | 2014-07-03 | International Business Machines Corporation | Determining weight values for storage devices in a storage tier to use to select one of the storage devices to use as a target storage to which data from a source storage is migrated |
| US20160034197A1 (en) * | 2014-08-04 | 2016-02-04 | Fujitsu Limited | Data migration method and data migration device |
| WO2017092480A1 (en) * | 2015-12-04 | 2017-06-08 | 华为技术有限公司 | Data migration method and device |
| US20180088870A1 (en) * | 2016-09-23 | 2018-03-29 | EMC IP Holding Company LLC | Method and device for storage management |
| CN107340975A (en) * | 2017-07-13 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of method and device of file storage |
| CN111367469A (en) * | 2020-02-16 | 2020-07-03 | 苏州浪潮智能科技有限公司 | Layered storage data migration method and system |
| CN115562870A (en) * | 2022-10-25 | 2023-01-03 | 北京京航计算通讯研究所 | Method for constructing task node resources of cluster |
| CN115826877A (en) * | 2023-01-20 | 2023-03-21 | 中国华能集团清洁能源技术研究院有限公司 | Data object migration method and device in big data environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116974468B (en) | 2024-02-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US5325505A (en) | Intelligent storage manager for data storage apparatus having simulation capability | |
| CN101615143B (en) | Method and device for diagnosing memory leak | |
| EP1471441B1 (en) | Performance management of a computer system with a database management system | |
| US20090228669A1 (en) | Storage Device Optimization Using File Characteristics | |
| CN101673192B (en) | Method for time-sequence data processing, device and system therefor | |
| CN101443761A (en) | QOS-enabled lifecycle management for file systems | |
| CN109918448A (en) | A kind of cloud storage data classification method based on user behavior | |
| US20100199058A1 (en) | Data Set Size Tracking and Management | |
| CN110175100B (en) | Storage disk fault prediction method and prediction system | |
| CN106598501A (en) | Data migration device and method for storage automatic hierarchy | |
| CN110175070B (en) | Distributed database management method, device, system, medium and electronic equipment | |
| CN111125171A (en) | Monitoring data access method, device, equipment and readable storage medium | |
| CN111008107A (en) | Big data cluster log storage method, device, equipment and storage medium | |
| CN116974468B (en) | Equipment data storage management method and system based on big data | |
| CN106899436A (en) | A kind of cloud platform failure predication diagnostic system | |
| CN113778964B (en) | Recording device for storing multiple temporary storage files and management method of temporary storage files | |
| CN116091175B (en) | Transaction information data management system and method based on big data | |
| EP0352462B1 (en) | Method and apparatus for calculating disk-access footprints | |
| CN116661685A (en) | Hierarchical storage method and system for object storage metadata of business behavior awareness | |
| CN115374065B (en) | File cleaning method and system based on cloud platform log record monitoring | |
| CN116244085A (en) | Kubernetes cluster container group scheduling method, device and medium | |
| CN114675956B (en) | Method for configuration and scheduling of Pod between clusters based on Kubernetes | |
| CN110932935A (en) | Resource control method, device, equipment and computer storage medium | |
| CN115993932A (en) | Data processing method, device, storage medium and electronic equipment | |
| CN109828718B (en) | Disk storage load balancing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |