CN113342273A - Cache-based big data storage method and system - Google Patents

Cache-based big data storage method and system Download PDF

Info

Publication number
CN113342273A
CN113342273A CN202110645921.3A CN202110645921A CN113342273A CN 113342273 A CN113342273 A CN 113342273A CN 202110645921 A CN202110645921 A CN 202110645921A CN 113342273 A CN113342273 A CN 113342273A
Authority
CN
China
Prior art keywords
data
time interval
group
cache region
management platform
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110645921.3A
Other languages
Chinese (zh)
Other versions
CN113342273B (en
Inventor
郝忆冰
李晓磊
郝博森
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing Cloud Micro Software Co ltd
Original Assignee
Beijing Jiuhe Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jiuhe Technology Co ltd filed Critical Beijing Jiuhe Technology Co ltd
Priority to CN202110645921.3A priority Critical patent/CN113342273B/en
Publication of CN113342273A publication Critical patent/CN113342273A/en
Application granted granted Critical
Publication of CN113342273B publication Critical patent/CN113342273B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0656—Data buffering arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/544—Buffers; Shared memory; Pipes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/06—Energy or water supply
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
    • Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
    • Y04S10/00—Systems supporting electrical power generation, transmission or distribution
    • Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Economics (AREA)
  • Water Supply & Treatment (AREA)
  • Public Health (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Primary Health Care (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Human Computer Interaction (AREA)
  • Remote Monitoring And Control Of Power-Distribution Networks (AREA)

Abstract

The invention provides a big data storage method and a big data storage system of a power monitoring system. The method comprises the following steps: sending the acquired data to a data management platform according to the power monitoring system; the data management platform stores data sent by all the power monitoring systems to a first data cache region, and stores the data sent by the power monitoring systems to a storage module of the data management platform through the first data cache region. The system comprises modules corresponding to the method steps.

Description

Cache-based big data storage method and system
Technical Field
The invention provides a cache-based big data storage method, and belongs to the technical field of data storage.
Background
In the technical field of power distribution, the safety of a power monitoring system is directly related to power supply, and the power monitoring system is a guarantee for normal operation of daily life and production of people. The existing power monitoring system has a large amount of data, and often in the data storage process, the problem of low data storage efficiency occurs because a large amount of data needs to be stored at the same moment, or data congestion occurs because the data amount is large at the same moment in the storage process, and then the data storage failure is caused.
Disclosure of Invention
The invention provides a cache-based big data storage method, which is used for solving the problems of low data rough storage efficiency and high storage failure rate in the big data storage process of the conventional power monitoring system:
the invention provides a big data storage method of a power monitoring system, which comprises the following steps:
sending the acquired data to a data management platform according to the power monitoring system;
the data management platform stores data sent by all the power monitoring systems to a first data cache region, and stores the data sent by the power monitoring systems to a storage module of the data management platform through the first data cache region.
Further, the data management platform stores data sent by all the power monitoring systems to a first data cache region, and stores the data sent by the powered monitoring systems to a storage module of the data management platform through the first data cache region, including:
the data management platform stores data sent by all the power monitoring systems to a first data cache region;
dividing all data into a plurality of data groups according to the types of the collected data in the first data cache region, setting the data sending time interval of each data group, and sending each data in the data groups to a data storage module of a data management platform for data storage according to the data sending time interval corresponding to each data group.
Further, dividing all data into a plurality of data groups according to the types of the collected data in the first data cache region, including:
dividing all data in the first data cache region into n data groups according to the type of the data collected by the power monitoring system, wherein the number n of the data groups is obtained through the following formula;
Figure BDA0003109175330000011
Figure BDA0003109175330000012
wherein Q represents the number of data types in each data group; z represents the total number of data types; m represents the total number of data storage modules in the data management platform; int () represents a rounding function;
when the data collected by the power monitoring system are sent to the first data cache region, the data are directly stored into the corresponding data group.
Further, the setting of the data transmission time interval of each data group includes:
extracting the number n of the data groups, and simultaneously extracting the number m of data storage modules in the data management platform;
acquiring the data sending time interval corresponding to each data group by using the number n of the data groups and the number m of the data storage modules through the following formula:
Figure BDA0003109175330000021
wherein, TiA data transmission time interval representing the ith data group; i denotes the serial number of each data set, i is 1,2, … …, n; t isajThe storage time required for storing a group of data of the jth data storage module is represented; t isbmaxRepresenting the maximum value of the data transmission time interval in each data type in each data group; t isbminIndicating the minimum value of the data transmission time interval in each data type in each data group; t isbtA data transmission time interval representing the t-th data type within each data group.
Further, the method further comprises: when the power monitoring system is additionally provided with the data acquisition sub-modules and further increases the data acquisition quantity, the data generated by the additionally arranged data acquisition sub-modules are uniformly sent to a second data cache region, and the data stored in the second data cache region are grouped on the basis of the additionally arranged data acquisition sub-modules to obtain a plurality of data groups;
setting a data sending time interval for each data group according to the following formula, and sending the data cached in the second data cache region to a data storage module of the data management platform according to the data sending time interval:
Figure BDA0003109175330000022
wherein, Tr2A data transmission time interval representing the r-th data group in the second data buffer area; t isamaxThe maximum value of time consumed by the data storage module for storing single batch of data is represented; t isaminThe minimum time consumed by the data storage module for storing single batch of data is represented; x represents the number of data types contained in each data group in the second data cache region; t isexAnd the data sending time interval of the x-th data type in each data group in the second data buffer area is shown.
A cache-based big data storage system, the system comprising:
the data sending module is used for sending the acquired data to the data management platform according to the power monitoring system;
and the data cache module is used for storing the data sent by all the power monitoring systems to a first data cache region by the data management platform and storing the data sent by the power monitoring systems to a storage module of the data management platform through the first data cache region.
Further, the data caching module comprises:
the data management platform is used for storing data sent by all the power monitoring systems to a first data cache region;
and the grouping module is used for dividing all data into a plurality of data groups according to the types of the acquired data in the first data cache region, setting the data sending time interval of each data group, and sending each data in the data groups to the data storage module of the data management platform for data storage according to the data sending time interval corresponding to each data group.
Further, the grouping module includes:
the data grouping module is used for dividing all data in the first data cache region into n data groups according to the type of the data collected by the power monitoring system, wherein the number n of the data groups is obtained through the following formula;
Figure BDA0003109175330000031
Figure BDA0003109175330000032
wherein Q represents the number of data types in each data group; z represents the total number of data types; m represents the total number of data storage modules in the data management platform; int () represents a rounding function;
and the grouped data sending module is used for directly storing the data into the corresponding data group when the data collected by the power monitoring system is sent into the first data cache region.
Further, the grouping module further comprises:
the extraction module is used for extracting the number n of the data groups and extracting the number m of the data storage modules in the data management platform;
the time interval acquisition module is used for acquiring the data sending time interval corresponding to each data group by using the number n of the data groups and the number m of the data storage modules through the following formula:
Figure BDA0003109175330000033
wherein, TiA data transmission time interval representing the ith data group; i denotes the serial number of each data set, i is 1,2, … …, n; t isajThe storage time required for storing a group of data of the jth data storage module is represented; t isbmaxRepresenting the maximum value of the data transmission time interval in each data type in each data group; t isbminIndicating the minimum value of the data transmission time interval in each data type in each data group; t isbtA data transmission time interval representing the t-th data type within each data group.
Further, the system further comprises: the data processing module is added and used for uniformly sending data generated by the added data acquisition sub-modules to a second data cache region when the data acquisition sub-modules are added to the power monitoring system and further increasing the data acquisition number, and grouping the data stored in the second data cache region on the basis of the added data acquisition sub-units to obtain a plurality of data groups;
wherein the data processing module comprises:
a time interval setting module, configured to set a data transmission time interval for each data group according to the following formula:
Figure BDA0003109175330000041
wherein, Tr2A data transmission time interval representing the r-th data group in the second data buffer area; t isamaxThe maximum value of time consumed by the data storage module for storing single batch of data is represented; t isaminThe minimum time consumed by the data storage module for storing single batch of data is represented; x represents the number of data types contained in each data group in the second data cache region; t isexIndicating the sending time interval of the data of the x-th data type in each data group in the second data cache region;
and the second data sending module is used for sending the data cached in the second data caching area to the data storage module of the data management platform according to the data sending time interval.
The invention has the beneficial effects that:
according to the cache-based big data storage method provided by the invention, the data sent by the power monitoring system is cached by setting the cache area, and the big data is stored in batches by grouping and storing at wrong time, so that the problem of reduction of data storage efficiency caused by data congestion due to the fact that the data are stored at the same time is effectively avoided. Meanwhile, the efficiency of data storage and the success rate of data storage are effectively improved.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
fig. 2 is a system block diagram of the system of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.
As shown in fig. 1, a big data storage method for a power monitoring system according to an embodiment of the present invention includes:
s1, sending the acquired data to a data management platform according to the power monitoring system;
s2, the data management platform stores the data sent by all the power monitoring systems to a first data cache region, and stores the data sent by the power monitoring systems to a storage module of the data management platform through the first data cache region.
The effect of the above technical scheme is as follows: the data sent by the power monitoring system are cached by setting the cache area, and the big data are stored in batches by grouping time-staggered storage, so that the problem that the data storage efficiency is reduced due to data congestion caused by the fact that the data are stored at the same time is effectively avoided. Meanwhile, the efficiency of data storage and the success rate of data storage are effectively improved.
In an embodiment of the present invention, the data management platform stores data sent by all power monitoring systems in a first data cache region, and stores data sent by the powered monitoring systems in a storage module of the data management platform through the first data cache region, including:
s201, the data management platform stores data sent by all the power monitoring systems to a first data cache region;
s202, dividing all data into a plurality of data groups according to the types of the collected data in the first data cache region, setting the data sending time interval of each data group, and sending each data in the data groups to a data storage module of the data management platform for data storage according to the data sending time interval corresponding to each data group.
Wherein, divide all data into a plurality of data groups according to the kind of the data of gathering in said first data buffer area, include:
s2021a, dividing all data in the first data cache region into n data groups according to the types of the data collected by the power monitoring system, wherein the number n of the data groups is obtained through the following formula;
Figure BDA0003109175330000051
Figure BDA0003109175330000052
wherein Q represents the number of data types in each data group; z represents the total number of data types; m represents the total number of data storage modules in the data management platform; int () represents a rounding function;
and S2022a, when the data collected by the power monitoring system is sent to the first data buffer area, directly storing the data into the corresponding data group.
Wherein the setting of the data transmission time interval of each data group includes:
s2021b, extracting the number n of the data groups, and extracting the number m of the data storage modules in the data management platform;
s2022b, obtaining the data sending time interval corresponding to each data group by using the number n of data groups and the number m of data storage modules according to the following formula:
Figure BDA0003109175330000061
wherein, TiA data transmission time interval representing the ith data group; i denotes the serial number of each data set, i is 1,2, … …, n; t isajThe storage time required for storing a group of data of the jth data storage module is represented; t isbmaxRepresenting the maximum value of the data transmission time interval in each data type in each data group; t isbminIndicating the minimum value of the data transmission time interval in each data type in each data group; t isbtA data transmission time interval representing the t-th data type within each data group.
The effect of the above technical scheme is as follows: by the method, the matching degree of the data packets and the number of the data storage modules is higher, and the capacity saturation rate of the second data storage module is effectively improved. Meanwhile, the grouping is carried out through the formula, so that the matching degree and the matching degree between the data grouping and the number of the data storage modules can be effectively improved. And the data storage module is used for correspondingly storing the data packets in the subsequent data storage process. On the other hand, the data transmission time interval is set through the formula, so that the power monitoring data transmission between each data packet can be completely staggered and complementarily overlapped, and the problem that the load of the data storage module is increased due to the mutual overlapping of the data stored in the previous data packet of the next batch and the data stored in the unfinished data packet of the previous batch because the data transmission time interval is set to be too short is avoided. The load of the data storage and writing process of the data storage module is effectively controlled through the sending time interval, data congestion caused by the saturation of the load in the writing process is effectively prevented, and the data storage efficiency and the storage success rate are further improved.
In one embodiment of the present invention, the method further comprises: when the power monitoring system is additionally provided with the data acquisition sub-modules and further increases the data acquisition quantity, the data generated by the additionally arranged data acquisition sub-modules are uniformly sent to a second data cache region, and the data stored in the second data cache region are grouped on the basis of the additionally arranged data acquisition sub-modules to obtain a plurality of data groups;
setting a data sending time interval for each data group according to the following formula, and sending the data cached in the second data cache region to a data storage module of the data management platform according to the data sending time interval:
Figure BDA0003109175330000062
wherein, Tr2A data transmission time interval representing the r-th data group in the second data buffer area; t isamaxThe maximum value of time consumed by the data storage module for storing single batch of data is represented; t isaminThe minimum time consumed by the data storage module for storing single batch of data is represented; x represents the number of data types contained in each data group in the second data cache region; t isexAnd the data sending time interval of the x-th data type in each data group in the second data buffer area is shown.
The effect of the above technical scheme is as follows: because the electric power monitoring system can increase data acquisition and data monitoring project according to actual demand, therefore, in order to guarantee that the data storage mode is not influenced, a second data cache region is arranged to store newly-added data types and projects in a wrong time, and meanwhile, the data transmission time interval obtained through the formula can effectively guarantee that the data transmission among all data groups is in a wrong time, so that the data transmission in a wrong time is not overlapped and influenced, and the data storage efficiency and the data storage success rate are improved.
An embodiment of the present invention provides a cache-based big data storage system, as shown in fig. 2, the system includes:
the data sending module is used for sending the acquired data to the data management platform according to the power monitoring system;
and the data cache module is used for storing the data sent by all the power monitoring systems to a first data cache region by the data management platform and storing the data sent by the power monitoring systems to a storage module of the data management platform through the first data cache region.
Wherein, the data cache module comprises:
the data management platform is used for storing data sent by all the power monitoring systems to a first data cache region;
and the grouping module is used for dividing all data into a plurality of data groups according to the types of the acquired data in the first data cache region, setting the data sending time interval of each data group, and sending each data in the data groups to the data storage module of the data management platform for data storage according to the data sending time interval corresponding to each data group.
Wherein the grouping module comprises:
the data grouping module is used for dividing all data in the first data cache region into n data groups according to the type of the data collected by the power monitoring system, wherein the number n of the data groups is obtained through the following formula;
Figure BDA0003109175330000071
Figure BDA0003109175330000072
wherein Q represents the number of data types in each data group; z represents the total number of data types; m represents the total number of data storage modules in the data management platform; int () represents a rounding function;
and the grouped data sending module is used for directly storing the data into the corresponding data group when the data collected by the power monitoring system is sent into the first data cache region.
Wherein the grouping module further comprises:
the extraction module is used for extracting the number n of the data groups and extracting the number m of the data storage modules in the data management platform;
the time interval acquisition module is used for acquiring the data sending time interval corresponding to each data group by using the number n of the data groups and the number m of the data storage modules through the following formula:
Figure BDA0003109175330000081
wherein, TiA data transmission time interval representing the ith data group; i denotes the serial number of each data set, i is 1,2, … …, n; t isajThe storage time required for storing a group of data of the jth data storage module is represented; t isbmaxRepresenting the maximum value of the data transmission time interval in each data type in each data group; t isbminIndicating the minimum value of the data transmission time interval in each data type in each data group; t isbtA data transmission time interval representing the t-th data type within each data group.
Wherein the system further comprises: the data processing module is added and used for uniformly sending data generated by the added data acquisition sub-modules to a second data cache region when the data acquisition sub-modules are added to the power monitoring system and further increasing the data acquisition number, and grouping the data stored in the second data cache region on the basis of the added data acquisition sub-units to obtain a plurality of data groups;
the data processing module comprises:
a time interval setting module, configured to set a data transmission time interval for each data group according to the following formula:
Figure BDA0003109175330000082
wherein, Tr2Indicating the r-th number in the second data buffer areaA data transmission time interval of the group; t isamaxThe maximum value of time consumed by the data storage module for storing single batch of data is represented; t isaminThe minimum time consumed by the data storage module for storing single batch of data is represented; x represents the number of data types contained in each data group in the second data cache region; t isexIndicating the sending time interval of the data of the x-th data type in each data group in the second data cache region;
and the second data sending module is used for sending the data cached in the second data caching area to the data storage module of the data management platform according to the data sending time interval.
The effect of the above technical scheme is as follows: the data sent by the power monitoring system are cached by setting the cache area, and the big data are stored in batches by grouping time-staggered storage, so that the problem that the data storage efficiency is reduced due to data congestion caused by the fact that the data are stored at the same time is effectively avoided. Meanwhile, the efficiency of data storage and the success rate of data storage are effectively improved.
By the method, the matching degree of the data packets and the number of the data storage modules is higher, and the capacity saturation rate of the second data storage module is effectively improved. Meanwhile, the grouping is carried out through the formula, so that the matching degree and the matching degree between the data grouping and the number of the data storage modules can be effectively improved. And the data storage module is used for correspondingly storing the data packets in the subsequent data storage process. Because the electric power monitoring system can increase data acquisition and data monitoring project according to actual demand, therefore, in order to guarantee that the data storage mode is not influenced, a second data cache region is arranged to store newly-added data types and projects in a wrong time, and meanwhile, the data transmission among all data groups can be effectively guaranteed to be transmitted in a wrong time, so that the data transmission among the data groups is stored in a wrong time in a memorable manner, the overlapping and the mutual influence of the data transmission among the data groups can not be caused, and the data storage efficiency and the data storage success rate are improved.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims (10)

1. A big data storage method of a power monitoring system, which is characterized by comprising the following steps:
sending the acquired data to a data management platform according to the power monitoring system;
the data management platform stores data sent by all the power monitoring systems to a first data cache region, and stores the data sent by the power monitoring systems to a storage module of the data management platform through the first data cache region.
2. The method of claim 1, wherein the data management platform stores data sent by all power monitoring systems in a first data cache region, and stores data sent by the powered monitoring systems in a storage module of the data management platform through the first data cache region, and the method comprises:
the data management platform stores data sent by all the power monitoring systems to a first data cache region;
dividing all data into a plurality of data groups according to the types of the collected data in the first data cache region, setting the data sending time interval of each data group, and sending each data in the data groups to a data storage module of a data management platform for data storage according to the data sending time interval corresponding to each data group.
3. The method of claim 2, wherein dividing all data into a plurality of data groups according to the types of the collected data in the first data buffer area comprises:
dividing all data in the first data cache region into n data groups according to the type of the data collected by the power monitoring system, wherein the number n of the data groups is obtained through the following formula;
Figure FDA0003109175320000011
Figure FDA0003109175320000012
wherein Q represents the number of data types in each data group; z represents the total number of data types; m represents the total number of data storage modules in the data management platform; int () represents a rounding function;
when the data collected by the power monitoring system are sent to the first data cache region, the data are directly stored into the corresponding data group.
4. The method of claim 2, wherein setting the data transmission time interval for each data group comprises:
extracting the number n of the data groups, and simultaneously extracting the number m of data storage modules in the data management platform;
acquiring the data sending time interval corresponding to each data group by using the number n of the data groups and the number m of the data storage modules through the following formula:
Figure FDA0003109175320000013
wherein, TiA data transmission time interval representing the ith data group; i denotes the serial number of each data set, i is 1,2, … …, n; t isajThe storage time required for storing a group of data of the jth data storage module is represented; t isbmaxRepresenting the maximum value of the data transmission time interval in each data type in each data group; t isbminIndicating the minimum value of the data transmission time interval in each data type in each data group; t isbtRepresenting the t-th data type within each data groupThe data transmission time interval of (2).
5. The method of claim 1, further comprising: when the power monitoring system is additionally provided with the data acquisition sub-modules and further increases the data acquisition quantity, the data generated by the additionally arranged data acquisition sub-modules are uniformly sent to a second data cache region, and the data stored in the second data cache region are grouped on the basis of the additionally arranged data acquisition sub-modules to obtain a plurality of data groups;
setting a data sending time interval for each data group according to the following formula, and sending the data cached in the second data cache region to a data storage module of the data management platform according to the data sending time interval:
Figure FDA0003109175320000021
wherein, Tr2A data transmission time interval representing the r-th data group in the second data buffer area; t isamaxThe maximum value of time consumed by the data storage module for storing single batch of data is represented; t isaminThe minimum time consumed by the data storage module for storing single batch of data is represented; x represents the number of data types contained in each data group in the second data cache region; t isexAnd the data sending time interval of the x-th data type in each data group in the second data buffer area is shown.
6. A cache-based big data storage system, the system comprising:
the data sending module is used for sending the acquired data to the data management platform according to the power monitoring system;
and the data cache module is used for storing the data sent by all the power monitoring systems to a first data cache region by the data management platform and storing the data sent by the power monitoring systems to a storage module of the data management platform through the first data cache region.
7. The system of claim 6, wherein the data caching module comprises:
the data management platform is used for storing data sent by all the power monitoring systems to a first data cache region;
and the grouping module is used for dividing all data into a plurality of data groups according to the types of the acquired data in the first data cache region, setting the data sending time interval of each data group, and sending each data in the data groups to the data storage module of the data management platform for data storage according to the data sending time interval corresponding to each data group.
8. The system of claim 7, wherein the grouping module comprises:
the data grouping module is used for dividing all data in the first data cache region into n data groups according to the type of the data collected by the power monitoring system, wherein the number n of the data groups is obtained through the following formula;
Figure FDA0003109175320000031
Figure FDA0003109175320000032
wherein Q represents the number of data types in each data group; z represents the total number of data types; m represents the total number of data storage modules in the data management platform; int () represents a rounding function;
and the grouped data sending module is used for directly storing the data into the corresponding data group when the data collected by the power monitoring system is sent into the first data cache region.
9. The system of claim 7, wherein the grouping module further comprises:
the extraction module is used for extracting the number n of the data groups and extracting the number m of the data storage modules in the data management platform;
the time interval acquisition module is used for acquiring the data sending time interval corresponding to each data group by using the number n of the data groups and the number m of the data storage modules through the following formula:
Figure FDA0003109175320000033
wherein, TiA data transmission time interval representing the ith data group; i denotes the serial number of each data set, i is 1,2, … …, n; t isajThe storage time required for storing a group of data of the jth data storage module is represented; t isbmaxRepresenting the maximum value of the data transmission time interval in each data type in each data group; t isbminIndicating the minimum value of the data transmission time interval in each data type in each data group; t isbtA data transmission time interval representing the t-th data type within each data group.
10. The system of claim 6, further comprising: the data processing module is added and used for uniformly sending data generated by the added data acquisition sub-modules to a second data cache region when the data acquisition sub-modules are added to the power monitoring system and further increasing the data acquisition number, and grouping the data stored in the second data cache region on the basis of the added data acquisition sub-units to obtain a plurality of data groups;
wherein the data processing module comprises:
a time interval setting module, configured to set a data transmission time interval for each data group according to the following formula:
Figure FDA0003109175320000041
wherein, Tr2A data transmission time interval representing the r-th data group in the second data buffer area; t isamaxThe maximum value of time consumed by the data storage module for storing single batch of data is represented; t isaminThe minimum time consumed by the data storage module for storing single batch of data is represented; x represents the number of data types contained in each data group in the second data cache region; t isexIndicating the sending time interval of the data of the x-th data type in each data group in the second data cache region;
and the second data sending module is used for sending the data cached in the second data caching area to the data storage module of the data management platform according to the data sending time interval.
CN202110645921.3A 2021-06-09 2021-06-09 Cache-based big data storage method and system Active CN113342273B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110645921.3A CN113342273B (en) 2021-06-09 2021-06-09 Cache-based big data storage method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110645921.3A CN113342273B (en) 2021-06-09 2021-06-09 Cache-based big data storage method and system

Publications (2)

Publication Number Publication Date
CN113342273A true CN113342273A (en) 2021-09-03
CN113342273B CN113342273B (en) 2022-09-09

Family

ID=77475625

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110645921.3A Active CN113342273B (en) 2021-06-09 2021-06-09 Cache-based big data storage method and system

Country Status (1)

Country Link
CN (1) CN113342273B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103439590A (en) * 2013-08-08 2013-12-11 深圳市康必达中创科技有限公司 Electric energy monitoring DSP control system
CN105471958A (en) * 2015-08-03 2016-04-06 汉柏科技有限公司 Data processing system and method of internet of things
CN109941315A (en) * 2019-03-29 2019-06-28 上海仁童电子科技有限公司 A kind of data processing equipment and method
CN111125094A (en) * 2019-11-25 2020-05-08 怀化建南电子科技有限公司 Electric power data storage system based on block chain
CN111198918A (en) * 2020-01-17 2020-05-26 国网福建省电力有限公司 Data processing system based on big data platform and link optimization method
CN112925673A (en) * 2021-02-18 2021-06-08 北京九贺科技有限公司 Big data storage management method and system for power monitoring

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103439590A (en) * 2013-08-08 2013-12-11 深圳市康必达中创科技有限公司 Electric energy monitoring DSP control system
CN105471958A (en) * 2015-08-03 2016-04-06 汉柏科技有限公司 Data processing system and method of internet of things
CN109941315A (en) * 2019-03-29 2019-06-28 上海仁童电子科技有限公司 A kind of data processing equipment and method
CN111125094A (en) * 2019-11-25 2020-05-08 怀化建南电子科技有限公司 Electric power data storage system based on block chain
CN111198918A (en) * 2020-01-17 2020-05-26 国网福建省电力有限公司 Data processing system based on big data platform and link optimization method
CN112925673A (en) * 2021-02-18 2021-06-08 北京九贺科技有限公司 Big data storage management method and system for power monitoring

Also Published As

Publication number Publication date
CN113342273B (en) 2022-09-09

Similar Documents

Publication Publication Date Title
CN111600967B (en) Access pressure-based load balancing system among block chain nodes
CN102929961B (en) Based on the data processing method and the device thereof that build rapid data classification passage
CN105814543A (en) Power balancing to increase workload density and improve energy efficiency
CN102681904B (en) Data syn-chronization dispatching method and device
CN101741885A (en) Distributed system and method for processing task flow thereof
CN112035058B (en) Cloud platform storage system
CN111901131A (en) Flow charging scheduling method, storage medium and computer
CN102970244A (en) Network message processing method of multi-CPU (Central Processing Unit) inter-core load balance
CN109785606A (en) A kind of power information acquisition system acquisition quality analysis method and device
CN109165045A (en) A kind of method and apparatus for the hardware configuration adjusting server
CN113342273B (en) Cache-based big data storage method and system
CN101022414B (en) Message retransmitting method and apparatus
CN103248636A (en) Offline download system and method
EP2477366A1 (en) Data transmission method, apparatus and system
CN111047306A (en) Parallel transaction processing method and device for transaction input set
CN106713512A (en) Efficient data transmission method and system capable of dynamically balancing network bandwidth of Internet of things
CN103888372A (en) Traffic shaping method and data processing equipment
CN102739531A (en) Traffic shaping method and traffic shaping equipment
CN108810074A (en) Data transmission method, apparatus and system, terminal
CN111935032B (en) Data packet sending method and system
CN113344738A (en) Access management method and system suitable for data intensive monitoring
CN112529732A (en) Energy storage unit charging and discharging control method and device, computer equipment and storage medium
CN112261509A (en) Meter reading scheduling method for electricity consumption information acquisition terminal
CN116382863B (en) Intelligent energy-saving scheduling method for data center
CN114069847B (en) Distributed photovoltaic data backfill system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20220819

Address after: 6th Floor-2, No. 2, No. 196, Zhongshan 2nd Road, Yuzhong District, Chongqing 400000

Applicant after: Chongqing Cloud Micro Software Co.,Ltd.

Address before: 100080 1505-270, 15th floor, 1 Zhongguancun Street, Haidian District, Beijing

Applicant before: Beijing Jiuhe Technology Co.,Ltd.

TA01 Transfer of patent application right
GR01 Patent grant
GR01 Patent grant