CN104317800A - Hybrid storage system and method for mass intelligent power utilization data - Google Patents
Hybrid storage system and method for mass intelligent power utilization data Download PDFInfo
- Publication number
- CN104317800A CN104317800A CN201410482773.8A CN201410482773A CN104317800A CN 104317800 A CN104317800 A CN 104317800A CN 201410482773 A CN201410482773 A CN 201410482773A CN 104317800 A CN104317800 A CN 104317800A
- Authority
- CN
- China
- Prior art keywords
- data
- terminal
- file
- mongodb
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 153
- 238000000034 method Methods 0.000 title claims abstract description 34
- 238000004891 communication Methods 0.000 claims abstract description 41
- 238000007726 management method Methods 0.000 claims abstract description 39
- 230000008569 process Effects 0.000 claims abstract description 23
- 238000004458 analytical method Methods 0.000 claims abstract description 19
- 238000004364 calculation method Methods 0.000 claims abstract description 5
- 238000007405 data analysis Methods 0.000 claims abstract description 3
- 238000012545 processing Methods 0.000 claims description 65
- 238000013523 data management Methods 0.000 claims description 18
- 238000002156 mixing Methods 0.000 claims description 16
- 238000012544 monitoring process Methods 0.000 claims description 12
- 238000013507 mapping Methods 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 230000005611 electricity Effects 0.000 claims description 8
- 230000002688 persistence Effects 0.000 claims description 7
- 238000012797 qualification Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 6
- 230000010076 replication Effects 0.000 claims description 5
- 230000008859 change Effects 0.000 claims description 4
- 230000001934 delay Effects 0.000 claims description 4
- 230000008676 import Effects 0.000 claims description 4
- 238000005259 measurement Methods 0.000 claims description 4
- 230000001360 synchronised effect Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000013467 fragmentation Methods 0.000 claims description 3
- 238000006062 fragmentation reaction Methods 0.000 claims description 3
- 230000008520 organization Effects 0.000 claims description 3
- 230000000737 periodic effect Effects 0.000 claims description 3
- 206010048669 Terminal state Diseases 0.000 claims description 2
- 230000015572 biosynthetic process Effects 0.000 claims description 2
- 238000001514 detection method Methods 0.000 claims description 2
- 238000012795 verification Methods 0.000 claims description 2
- 238000004519 manufacturing process Methods 0.000 abstract description 8
- 230000005055 memory storage Effects 0.000 description 4
- 238000010276 construction Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000000151 deposition Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000005684 electric field Effects 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000005194 fractionation Methods 0.000 description 1
- 238000002386 leaching Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000005619 thermoelectricity Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Economics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- Software Systems (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a hybrid storage system and method for mass intelligent power utilization data. The hybrid storage system comprises a front communication server cluster, wherein the front communication server cluster receives an original data frame uploaded by a terminal and stores an original data film obtained after data analysis is carried out into a local disk; the front communication server cluster unloads the original data frame and the original data file to a MongoDB private cloud storage platform; a collection data unloading calculation platform reads the data of the MongoDB private cloud storage platform to process the data, and then the processed data is stored in a production library and an analysis library of the collection data unloading calculation platform; the production library stores service data, and the analysis library analyzes and processes the service data of the production library; and an analysis result is written back to the production library for query. The invention can cope with challenges brought for the storage capability of a front master station for collecting mass data, the collected data is stored in a local cache, the requirements of quick storage and uniform management of the mass data are met, and the storage pressure of a production database is lowered.
Description
Technical field
The present invention relates to a kind of magnanimity intelligent power data mixing storage system and method.
Background technology
Power information acquisition system is as the important component part of intelligent power, the collection to electricity consumption data can be realized, and carry out managing based on the electricity consumption data gathered, analyze, decision-making, improve user power utilization efficiency and optimize power mode, implementing the important handgrip of China's energy-saving and emission-reduction and step price policy, is the important content that intelligent power is built.State Grid Corporation of China deeply advances " Two change ", active construction " three collection five are large " system, the construction object of power information collection " all standing, full collection, full control " was proposed in " 12 " period, realize the comprehensive covering to all power consumers and critical point, realize the Real-time Collection of the important information such as measuring apparatus on-line monitoring and customer charge, electricity, voltage, in time, complete, exactly for related system provides basic data, for the analysis of each link of enterprise operation and management, decision-making provide support.
Along with the realization of " all standing, full collection, full control " construction object, the electric field scope that power information acquisition system covers expands further, the scope that power supply enterprise's power information collection covers specially becomes user from only covering important special line, expand to cover all kinds of special line gradually and specially become the multiple electric field such as user, general industrial and commercial producer, Gong Biantai district, the scale accessing all kinds of acquisition terminal and table meter also increases thereupon, image data amount linear increase, image data presents magnanimity trend.By analyzing image data, these data great majority are source code data, and power information acquisition system was all directly be stored into gathering the source code data of coming up in storage facility located at processing plant after resolving to the storage of image data in the past.
But, on the one hand, power information acquisition system has acquisition tasks and performs the middle and high concurrent feature of period collection, cause the data loading pressure of part-time section larger, there is I/O bottleneck in database, make the related data statistical study business of system can not get carrying out in time, have a strong impact on the search efficiency of related data;
On the other hand, source code data are basic datas of power information acquisition system, vital effect is had to communication traffic statistics, communication log recording, electricity calculating, fault analysis etc., and power information acquisition system cannot be preserved source code data for a long time by strategy restriction in the past, hindered the good application to data.
Summary of the invention
Object of the present invention is exactly to solve the problem, a kind of magnanimity intelligent power data mixing storage system and method are provided, it has the challenge that reply magnanimity image data is brought to preposition main website storage capacity, now by acquired data storage at local cache, then the privately owned cloud storage platform centralized management of MongoDB is stored into, meet the quick storage to mass data and unified management demand, reduce the advantage of the warehouse-in pressure to Production database.
To achieve these goals, the present invention adopts following technical scheme:
A kind of magnanimity intelligent power data mixing storage system, comprise preposition communication server cluster, the privately owned cloud storage platform of MongoDB, image data unloading computing platform, the initial data frame that described preposition communication server cluster receiving terminal is uploaded, the raw data file obtained after Data Analysis is stored into local disk, initial data frame and raw data file adopt the mode of data cross copy backup to dump to the privately owned cloud storage platform of described MongoDB by described preposition communication server cluster afterwards, be stored into the storage facility located at processing plant of the relation data module of described image data unloading computing platform after the data that described image data unloading computing platform reads described MongoDB privately owned cloud storage platform afterwards process and analyze in storehouse, storage facility located at processing plant stores business datum and realizes terminal debugging, terminal parameter setting and image data real-time query manipulation, analyze the business datum of storehouse to storage facility located at processing plant and be further analyzed process, realize gathering success ratio, the statistical study of the online rate of terminal and line loss qualification rate, and analysis result is write back storage facility located at processing plant, for inquiry.
Described preposition communication server cluster comprises communication interface modules, data cache module and repository;
Described communication interface modules comprises telecommunication management interface, condition monitoring interface, task management interface, stipulations parsing interface and Data import interface;
The foundation of the preposition communication server cluster of described telecommunication management Interface realization and terminal room physical connection;
The monitoring of described condition monitoring Interface realization terminal and server state;
The unified management of described task management Interface realization acquisition tasks: task issue with terminal affair on give;
Described stipulations are resolved interface and the initial data frame collected are resolved to original data object;
Data primary object after parsing is loaded in data cache module by described metadata loading interface.
Original data object is stored in local disk by described data cache module in the form of a file;
Described repository is responsible for the organization and management of the monitor message of terminal and server, terminal file information, configuration information and acquisition tasks information.
The privately owned cloud storage platform of described MongoDB comprises data memory module, routing module and configuration module;
The initial data frame that described data memory module storage front server cluster transmission is come and raw data file;
Described routing module deposits information for the concrete data of management configuration module;
Described configuration module for preserving data to place data block and place data block to the mapping of place data fragmentation, and stores all acquisition state and the collection success ratio that need image data measurement point.
Described image data unloading computing platform comprises data processing module and relation data memory module.
Described relation data memory module comprises storage facility located at processing plant and analyzes storehouse;
Described storage facility located at processing plant stores business datum;
Described analysis storehouse, for sharing the pressure of storage facility located at processing plant, is derived for index data query, statistical study and form.
Described data processing module comprises archives buffer memory, data persistence interface and large data management engine, synchronous for realizing archive information, for calculating and the unloading of acquired original data, also for providing whole archive informations of record terminal acquisition state.
Described large data management engine reads raw data file from the privately owned cloud storage platform of MongoDB, and the data for different types of data formulate scheduling and the management of different priorities.
Described archives buffer memory is synchronous archive information from storage facility located at processing plant, is calculated generate business datum by the process of large data management engine.
The business datum that large data management engine generates according to the archives in archives buffer memory by described data persistence interface is stored in relation data memory module.
A kind of magnanimity intelligent power data mixing storage means, comprises the steps:
Step (1): the initial data frame that preposition communication server cluster is uploaded by telecommunication management interface receives end, is resolved to original data object according to terminal number, is deposited into local disk in the form of a file;
Step (2): adopt the mode of data cross copy backup to dump on each server of the privately owned cloud storage platform of MongoDB initial data frame and original data object;
Step (3): the data processing module of image data unloading computing platform reads and processes the data of the privately owned cloud storage platform of MongoDB, be stored into the storage facility located at processing plant of relation data module after forming business datum and analyze in storehouse, storage facility located at processing plant stores business datum and realizes terminal debugging, terminal parameter setting and image data real-time query manipulation, analyze the business datum of storehouse to storage facility located at processing plant and be further analyzed process, realize the statistical study gathering success ratio, the online rate of terminal and line loss qualification rate, and analysis result is write back storage facility located at processing plant, for inquiry.
The step of described step (1) is:
Step (1-1): preposition communication server cluster is set up Session by the telecommunication management interface of communication interface modules with acquisition terminal and is connected, obtain terminal number and the initial data frame of this acquisition terminal, and terminal number is write in Session to carry out the mark of terminal belonging to data to initial data frame, the Session of initial data frame and correspondence is obtained by metadata loading interface, by initial data frame stored in the privately owned cloud storage platform of MongoDB, and from Session, read terminal number corresponding to initial data frame;
Step (1-2): whether inquiry terminal numbering is empty, if just enter step (1-3) for sky; If just do not enter step (1-4) for sky;
Step (1-3): terminal number is empty, then resolve terminal address code corresponding to initial data frame and check terminal address code whether in terminal file table;
If terminal address code is in terminal file table, then according to No. Session of the terminal address code parsed and initial data frame, terminal file table is upgraded, setting terminal state, resolves to original data object by initial data frame simultaneously, then enters step (1-5);
If terminal address code is not in archives table, proves that terminal is unauthorized terminal, abandon initial data frame, terminate;
Step (1-4): terminal file numbering be empty, identifies the control code of initial data frame, address code and function code by stipulations parsing interface, and initial data frame is resolved to original data object; Enter step (1-5); Described original data object comprises data unique identification, markers, acquisition quality code, terminal freeze-off time and value;
Step (1-5): original data object is stored into local disk in the form of a file, upgrades the log sheet of the privately owned cloud storage platform of MongoDB, is write in repository by document base information simultaneously.
The step of described step (1-5) is: the original data object after resolving be stored in the form of a file in local disk, original data object stores hereof with the form of key-value pair, comprise data unique identification, markers, acquisition quality code, terminal freeze-off time and magnitude information, when file reaches the file size that presets or file does not change within a very long time, then put file for writing state, and by it stored in the privately owned cloud storage platform of MongoDB, the start time of uploading of file is write repository, cycle detection file status, until all raw data files in local disk file have all been uploaded, by files passe deadline write repository, and delete to reserve the real-time image data given of disk space storage terminal from local disk by uploading successful file.
The document base information of described step (1-5) comprise reference number of a document, preposition communication server numbering, file fullpath, filename, file whether deleted, high-low pressure mark, freeze mark, file declustering mark, file type and VIP mark curve day.
Described high-low pressure mark comprises high pressure, low pressure, transformer station and electricity determining by heat.
Freeze described curve day mark comprise freeze day, curve, the moon freeze, freeze day of checking meter and thermoelectricity data.
Described file declustering mark comprises the file after the file and fractionation do not split.
Described file type and VIP mark comprise ordinary file and VIP file.
The step of described step (2) is:
Step (2-1): raw data file state in preposition communication server cluster periodic monitor repository, by file status be and the files passe do not uploaded to the data memory module of the privately owned cloud storage platform of MongoDB; The deadline of uploading of file is write repository;
The privately owned cloud storage platform of step (2-2): MongoDB is by disposing MongoDB cluster, the initial data frame that preposition communication server cluster send and original data object file are stored in the data memory module of the server of the privately owned cloud storage platform of MongoDB, if terminal quantity constantly increases the storage demand making existing server can not meet ever-increasing raw data file, then dynamically increase new server, the storage capacity of raising system, initial data frame and original data object are deposited with json file layout, exist with key-value pair form hereof,
Step (2-3): carry out burst storage to data file on every station server of MongoDB privately owned cloud storage platform, each burst comprises some data blocks, if certain data block exceeds restriction size, then generates new multiple data blocks, a burst is a copy set, use the mode of cross replication backup, under multiple copies of same copy set are deployed in different servers, under the copy cross part of different burst being deployed on same station server simultaneously, finally configuration module less for the server cluster consumption of natural resource of privately owned for MongoDB cloud storage platform and routing module are deployed on each server, dispose multiple configuration module and routing module guarantees that the privately owned cloud storage platform of whole MongoDB is when a server delays machine, the ruuning situation of whole server cluster can not be affected.
The initial data frame of described step (2-2), the every a line in file stores an initial data frame, and each initial data frame comprises affiliated terminal iidentification, up-downgoing Data Identification and concrete Frame,
The original data object of described step (2-2), comprises data unique identification, markers, acquisition quality code, terminal freeze-off time and value.
Described routing module is responsible for the deposit position that query configuration module finds data, described configuration module saves two mapping relations, one is data and the mapping relations of data block depositing these data, and another is data block and the mapping relations of burst depositing this data block.
The step of described step (3) is:
Step (3-1): the data processing module of image data unloading computing platform is by reading the repository of preposition communication server cluster, and the file location information of data conversion storage is not carried out in inquiry;
Step (3-2): the file that do not carry out data conversion storage of data processing module from MongoDB privately owned cloud storage platform in read step (3-1) of image data unloading computing platform, and file is read start time write repository;
Step (3-3): data processing module obtains the archive information of the file not carrying out data conversion storage from data cache module, specifies different priority by large data management engine to dissimilar data, and carries out dispatching and managing; As carried out for transformer station, high voltage customer, low-voltage customer scheduling and the management that the different types of data such as voltage, load, electric current, table code data formulate different priorities;
Step (3-4): file is read the time write repository terminated, the business datum through large data management engine process is stored in relational database; The process of described large data management engine comprises: carry out corresponding rate verification, PT, CT, comprehensive multiplying power calculating, electricity calculating and always add group power calculation, forming business datum, by the business datum of formation stored in the storage facility located at processing plant of relational database and analysis storehouse,
Storage facility located at processing plant stores business datum and realizes terminal debugging, terminal parameter setting and image data real-time query manipulation,
Analyze the business datum of storehouse to storage facility located at processing plant and be further analyzed process, realize the statistical study gathering success ratio, the online rate of terminal and line loss qualification rate, and analysis result is write back storage facility located at processing plant, for inquiry.
The reading process of described step (3-2) refers to:
Routing module in the privately owned cloud storage platform of data processing unit access MongoDB of image data unloading computing platform, routing module obtains the configuration information of raw data file by configuration module, find deposit raw data file burst and burst on corresponding concrete data block location, from data block, read out raw data file line by line.
Beneficial effect of the present invention:
(1) the magnanimity intelligent power data mixing memory storage of the present invention's design, the challenge that reply magnanimity image data brings to preposition main website storage capacity, now by acquired data storage at local cache, then the privately owned cloud storage platform centralized management of MongoDB is stored into, meet the quick storage to mass data and unified management demand, reduce the warehouse-in pressure to Production database.
(2) the privately owned cloud storage platform of MongoDB of the present invention's structure, by distributed server disposition pattern, share the load pressure of individual server, the mode of usage data cross replication backup ensures the security of data, in the ever-increasing situation of terminal, increased the storage capacity of system by the mode dynamically increasing new server, ensure permanent, the reliable memory of image data.
(3) the present invention design image data unloading computing platform in relation data memory module adopt data divide storehouse technology, relational database be divided into storage facility located at processing plant and analyze storehouse, the complicated applications such as the statistical study of former dependence storage facility located at processing plant are migrated to and analyzes in storehouse, ensure that the data acquisition warehouse-in that storage facility located at processing plant is single and basic application function, reduce storage facility located at processing plant pressure, and be statistical study, the senior operating portion such as data query affixes one's name to independent resource, promote the efficiency of sophisticated statistical application and data processing, advanced inquiry analytic function response speed is significantly promoted, effectively alleviate storage facility located at processing plant pressure.
(4) on every station server of MongoDB privately owned cloud storage platform, carry out burst storage to data file, each burst comprises some data blocks, when data block exceeds restriction size, is just split into two data blocks.A burst is a copy set, use the mode of cross replication backup, under multiple copies of same copy set are deployed in different servers, ensure the accurate safety of data, the copy cross part of different burst is deployed under same station server to make full use of every resource of server simultaneously, finally configuration module less for privately owned cloud storage server cluster consumption of natural resource and routing module are deployed on each node server, dispose multiple configuration module and routing module guarantees that the ruuning situation of whole MongoDB privately owned cloud storage platform whole cluster when a server delays machine can not be affected, routing module is responsible for the deposit position that query configuration module finds data, configuration module saves two mapping relations, the mapping relations being data and all leaving in those data blocks in, another is the mapping relations which burst is data block all leave in.
Accompanying drawing explanation
Fig. 1 is magnanimity intelligent power data mixing memory storage overall architecture;
Fig. 2 is that magnanimity intelligent power data store overall flow;
Fig. 3 is magnanimity intelligent power data local disk buffer memory flow process;
Fig. 4 is the privately owned cloud Stored Procedure of magnanimity intelligent power data MongoDB;
Fig. 5 is image data unloading calculation process.
Embodiment
Below in conjunction with accompanying drawing and embodiment, the invention will be further described.
The embodiment of the present invention provides a kind of magnanimity intelligent power data mixing storage means and device.The storage of magnanimity intelligent power data mixing is the local disk image data sent from acquisition terminal being first persisted to preposition main website, then concentrated stored in the privately owned cloud storage platform of MongoDB, and by the storage condition of magnanimity intelligent power data write repository, data processing unit reads the raw data file in the privately owned cloud storage platform of MongoDB, then a series of calculating is carried out, form business datum, stored in relational database.
Below in conjunction with drawings and Examples, the present invention is further elaborated.
With reference to figure 1, be a kind of magnanimity intelligent power of the present invention data mixing memory storage, this device comprises:
A kind of magnanimity intelligent power data mixing memory storage, relates to preposition communication server cluster 100, the privately owned cloud storage platform 200 of MongoDB, image data unloading computing platform 300.
Preposition communication server cluster 100 comprises communication interface modules 101, data cache module 102, repository 103.
Communication interface modules 101 comprises telecommunication management interface, condition monitoring interface, task management interface, stipulations resolve interface, Data import interface etc., the foundation of the preposition communication server cluster of telecommunication management Interface realization and terminal room physical connection, the monitoring of condition monitoring Interface realization terminal and server state information, task management Interface realization acquisition tasks issues, the unified management of task such as terminal affair to send, stipulations are resolved interface and the initial data frame collected are resolved to original data object, data primary object after parsing is loaded in data cache module by Data import interface.
Original data object is stored in local disk by data cache module 102 in the form of a file.
The organization and management of the relevant informations such as repository 103 primary responsibility terminal and server state monitoring information, terminal file information, configuration information, acquisition tasks information and terminal Information Monitoring.
The privately owned cloud storage platform 200 of MongoDB comprises data memory module 201, routing module 202, configuration module 203.
Data memory module 201 stores the initial data frame and raw data file that front server cluster transmission comes.
In routing module 202 management configuration module, initial data frame and raw data file deposits information.
Configuration module 203 preserves data to place data block and place data block to the mapping of place data fragmentation, and stores all log informations such as acquisition state, collection success ratio needing image data measurement point.
Image data unloading computing platform 300 comprises data processing module 301, relation data memory module 302.
Data processing module 301 comprises archives buffer memory, large data management engine, data persistence interface.Archives buffer memory realizes terminal file information and loads and synchronously, provide whole archive informations of record terminal acquisition state.Large data management engine reads raw data file from the privately owned cloud storage platform of MongoDB, scheduling and the management of different priorities is formulated for different types of data, calculate for the voltage of transformer station, high voltage customer, low-voltage customer, load, electric current, table code data and store the management and running of formulating different stage, generating business datum.Business datum is stored in relation data memory module 302 according to the archives in archives buffer memory by data persistence interface.
Relation data memory module 302 comprises storage facility located at processing plant and analyzes storehouse, storage facility located at processing plant stores all business datums associated with the query, analyze storehouse for sharing the pressure of storage facility located at processing plant, lay particular emphasis on the complicated applications analyses such as achievement data inquiry, statistical study, form derivation, analyze after storehouse has been analyzed and analysis result is write back storage facility located at processing plant.
As shown in Figure 2, a kind of magnanimity intelligent power data mixing storage means mainly comprises the local disk buffer memory of magnanimity intelligent power data, the privately owned cloud of MongoDB of magnanimity intelligent power data stores, the unloading of magnanimity intelligent power data calculates three parts.
As shown in Figure 3, wherein, the local disk buffer memory of magnanimity intelligent power data is achieved in that
(1) acquisition terminal carries out three-way handshake by the Socket port that front server cluster module is open and sets up TCP and connect, generate Session object, when terminal number being put into Session object for resolution data, and on send the initial data frame such as electric energy indicating value, voltage, electric current, power be made up of control code, address code, function code, frame sequence territory, data cell mark, data markers, data content.Main website obtains No. Session of initial data frame and the correspondence thereof that terminal is sent, and obtains terminal number from Session;
(2) by the initial data frame that gets stored in the privately owned cloud storage platform of MongoDB, judge whether the terminal number obtained from Session is empty, if not for idle running is to (7), as dallying to (3);
(3) the terminal address code that initial data frame is corresponding is parsed;
(4) according to the terminal address code inquiry terminal archives table that parses, inquiry terminal whether in archives table, if inquiry terminal does not go to (6) in archives table, as gone to (5) in archives table;
(5) upgrade terminal file according to No. Session of the terminal address code parsed and message, the state of setting terminal is that terminal is reached the standard grade or state keeps, and goes to (7);
(6) prove that terminal is unauthorized terminal, abandon initial data frame;
(7) initial data frame is resolved, identify the information such as control code, address code, function code, frame sequence territory, data cell mark, data markers, data content, form the original data object be made up of key-value key-value pair.
(8) persistence process is done to original data object, raw data write local disk file, original data object is present in local disk file with the form of key-value key-value pair, when file reaches any one in following three conditions, is then set to by file and writes mark.
file size has exceeded the threshold value of setting;
file does not change within long enough a period of time, overtime threshold value;
file from set up and be in write state always, but be through the size threshold value that the sufficiently long time does not still reach file, exceeded time threshold.
(9) by essential information write repositories such as the write time of file, the states of file, simultaneously by the log sheet of the configuration module in the privately owned cloud storage platform of the acquisition state of image data measurement point write MongoDB;
(10) timing query configuration storehouse, the state of image data file is monitored, by file status be write and the files passe not being uploaded to MongoDB privately owned cloud storage platform in the privately owned cloud storage platform of MongoDB, and by files passe time and the deadline of uploading write repository.
(11) upload successful file and can store certain hour at local disk, the time is configurable, after file time-out, and the automatic deleted file of local disk and catalogue.
As shown in Figure 4, the privately owned cloud storage of the MongoDB of magnanimity intelligent power data is achieved in that
(1) the privately owned cloud of MongoDB stores and receives initial data frame and raw data file, it is stored in MongoDB server cluster with the form of key-value key-value pair, when storing, burst is carried out to data, data block storage in each burst, when storing data, routing node judges whether the data block in current slice exceeds restriction size, limit size just split into two data blocks if exceeded, configuration node is recorded data and is stored into the mapping to place data block of the mapping of burst and burst.
(2) bursts are a copy set, the mode of usage data cross replication backup, by multiple node deployments of same copy set under different servers, ensure the accurate safety of data.
(3) under the copy cross part of different burst node being deployed on same station server, to make full use of every resource of server.
(4) configuration module less for cloud storage cluster consumption of natural resource and routing module are deployed on each node server, dispose multiple configuration module and routing module guarantees that the ruuning situation of whole MongoDB privately owned cloud storage platform whole cluster when a server delays machine can not be affected.
As shown in Figure 5, the unloading of magnanimity intelligent power data calculates and is achieved in that
(1) fileinfo of data conversion storage calculating is not carried out in data processing unit periodic monitor repository, with reference to this part fileinfo, adopt socket communication mode, based on the standard commands of Tcp/Ip, extracted by this part file in MongoDB, leaching process is as follows:
Read the routing module in the privately owned cloud storage platform of MongoDB according to the unique identification not carrying out data object in data conversion storage file, obtain the burst at data object place and the data block at place by routing module, then on its position, read corresponding data.
(2) file extracted is submitted to large data management engine by data processing unit, large data management engine formulates scheduling and the management of different priorities for different types of data, carries out the statistics of the voltage of transformer station, high voltage customer, low-voltage customer etc., power, electric current, table code data.
(3) statistics processed, verify and calculate accordingly, form power consumption, always add the business datums such as group power, electric current, voltage.
(4) final business datum submitted to storage facility located at processing plant simultaneously and analyze storehouse, storage facility located at processing plant realizes the business operation higher to ageing requirement such as terminal debugging, terminal parameter setting, the inquiry of image data real-time, analyze the advanced statistical analysis that storehouse realizes gathering success ratio, the online rate of terminal, line loss qualification rate etc., and analysis result is written back in storage facility located at processing plant for inquiry.
The present invention obtains application in national grid science and technology item " improving the intelligent power technical research of Power marketing management and countermeasure level ", effect is good, effectively alleviate the warehouse-in pressure (Production database server cpu load can long-time stable below 40%) of storage facility located at processing plant, by the privately owned cloud storage platform of design MongoDB, magnanimity source code data obtain and store reliably and with long-term, platform day data renewal amount more than 3,000,000,000 times, dayfile memory space is more than 1000G.Simultaneously, communication traffic is added up, communication log recording, electricity calculates, the statistical study business such as fault analysis all need by having come the inquiry of magnanimity source code data, the privately owned cloud storage platform of MongoDB stores source code data with document form, the fast query of source code data can be met, and then improve statistical study application efficiency, compared with use relation data library storage source code data, efficiency data query significantly promotes, as used during relation data library storage data, 20ms is needed to 2,000 ten thousand data queries, and when using the privately owned cloud storage platform of MongoDB to store data, inquire about 2,000 ten thousand data and only need 4ms.
By reference to the accompanying drawings the specific embodiment of the present invention is described although above-mentioned; but not limiting the scope of the invention; one of ordinary skill in the art should be understood that; on the basis of technical scheme of the present invention, those skilled in the art do not need to pay various amendment or distortion that creative work can make still within protection scope of the present invention.
Claims (10)
1. a magnanimity intelligent power data mixing storage system, it is characterized in that, comprise preposition communication server cluster, the privately owned cloud storage platform of MongoDB, image data unloading computing platform, the initial data frame that described preposition communication server cluster receiving terminal is uploaded, the raw data file obtained after Data Analysis is stored into local disk, initial data frame and raw data file adopt the mode of data cross copy backup to dump to the privately owned cloud storage platform of described MongoDB by described preposition communication server cluster afterwards, be stored into the storage facility located at processing plant of the relation data module of described image data unloading computing platform after the data that described image data unloading computing platform reads described MongoDB privately owned cloud storage platform afterwards process and analyze in storehouse, storage facility located at processing plant stores business datum and realizes terminal debugging, terminal parameter setting and image data real-time query manipulation, analyze the business datum of storehouse to storage facility located at processing plant and be further analyzed process, realize gathering success ratio, the statistical study of the online rate of terminal and line loss qualification rate, and analysis result is write back storage facility located at processing plant, for inquiry.
2. a kind of magnanimity intelligent power data mixing storage system as claimed in claim 1, is characterized in that,
Described preposition communication server cluster comprises communication interface modules, data cache module and repository;
Described communication interface modules comprises telecommunication management interface, condition monitoring interface, task management interface, stipulations parsing interface and Data import interface;
The foundation of the preposition communication server cluster of described telecommunication management Interface realization and terminal room physical connection;
The monitoring of described condition monitoring Interface realization terminal and server state;
The unified management of described task management Interface realization acquisition tasks: task issue with terminal affair on give;
Described stipulations are resolved interface and the initial data frame collected are resolved to original data object;
Data primary object after parsing is loaded in data cache module by described metadata loading interface;
Original data object is stored in local disk by described data cache module in the form of a file;
Described repository is responsible for the organization and management of the monitor message of terminal and server, terminal file information, configuration information and acquisition tasks information.
3. a kind of magnanimity intelligent power data mixing storage system as claimed in claim 1, is characterized in that,
The privately owned cloud storage platform of described MongoDB comprises data memory module, routing module and configuration module;
The initial data frame that described data memory module storage front server cluster transmission is come and raw data file;
Described routing module deposits information for the concrete data of management configuration module;
Described configuration module for preserving data to place data block and place data block to the mapping of place data fragmentation, and stores all acquisition state and the collection success ratio that need image data measurement point.
4. a kind of magnanimity intelligent power data mixing storage system as claimed in claim 1, is characterized in that,
Described image data unloading computing platform comprises data processing module and relation data memory module;
Described relation data memory module comprises storage facility located at processing plant and analyzes storehouse;
Described storage facility located at processing plant stores business datum;
Described analysis storehouse, for sharing the pressure of storage facility located at processing plant, is derived for index data query, statistical study and form;
Described data processing module comprises archives buffer memory, data persistence interface and large data management engine, synchronous for realizing archive information, for calculating and the unloading of acquired original data, also for providing whole archive informations of record terminal acquisition state;
Described large data management engine reads raw data file from the privately owned cloud storage platform of MongoDB, and the data for different types of data formulate scheduling and the management of different priorities;
Described archives buffer memory is synchronous archive information from storage facility located at processing plant, is calculated generate business datum by the process of large data management engine;
The business datum that large data management engine generates according to the archives in archives buffer memory by described data persistence interface is stored in relation data memory module.
5. the system as described in above-mentioned arbitrary claim the storage means that adopts, it is characterized in that, comprise the steps:
Step (1): the initial data frame that preposition communication server cluster is uploaded by telecommunication management interface receives end, is resolved to original data object according to terminal number, is deposited into local disk in the form of a file;
Step (2): adopt the mode of data cross copy backup to dump on each server of the privately owned cloud storage platform of MongoDB initial data frame and original data object;
Step (3): the data processing module of image data unloading computing platform reads and processes the data of the privately owned cloud storage platform of MongoDB, be stored into the storage facility located at processing plant of relation data module after forming business datum and analyze in storehouse, storage facility located at processing plant stores business datum and realizes terminal debugging, terminal parameter setting and image data real-time query manipulation, analyze the business datum of storehouse to storage facility located at processing plant and be further analyzed process, realize the statistical study gathering success ratio, the online rate of terminal and line loss qualification rate, and analysis result is write back storage facility located at processing plant, for inquiry.
6. method as claimed in claim 5, it is characterized in that, the step of described step (1) is:
Step (1-1): preposition communication server cluster is set up Session by the telecommunication management interface of communication interface modules with acquisition terminal and is connected, obtain terminal number and the initial data frame of this acquisition terminal, and terminal number is write in Session to carry out the mark of terminal belonging to data to initial data frame, the Session of initial data frame and correspondence is obtained by metadata loading interface, by initial data frame stored in the privately owned cloud storage platform of MongoDB, and from Session, read terminal number corresponding to initial data frame;
Step (1-2): whether inquiry terminal numbering is empty, if just enter step (1-3) for sky; If just do not enter step (1-4) for sky;
Step (1-3): terminal number is empty, then resolve terminal address code corresponding to initial data frame and check terminal address code whether in terminal file table;
If terminal address code is in terminal file table, then according to No. Session of the terminal address code parsed and initial data frame, terminal file table is upgraded, setting terminal state, resolves to original data object by initial data frame simultaneously, then enters step (1-5);
If terminal address code is not in archives table, proves that terminal is unauthorized terminal, abandon initial data frame, terminate;
Step (1-4): terminal file numbering be empty, identifies the control code of initial data frame, address code and function code by stipulations parsing interface, and initial data frame is resolved to original data object; Enter step (1-5); Described original data object comprises data unique identification, markers, acquisition quality code, terminal freeze-off time and value;
Step (1-5): original data object is stored into local disk in the form of a file, upgrades the log sheet of the privately owned cloud storage platform of MongoDB, is write in repository by document base information simultaneously.
7. method as claimed in claim 5, it is characterized in that, the step of described step (2) is:
Step (2-1): raw data file state in preposition communication server cluster periodic monitor repository, by file status be and the files passe do not uploaded to the data memory module of the privately owned cloud storage platform of MongoDB; The deadline of uploading of file is write repository;
The privately owned cloud storage platform of step (2-2): MongoDB is by disposing MongoDB cluster, the initial data frame that preposition communication server cluster send and original data object file are stored in the data memory module of the server of the privately owned cloud storage platform of MongoDB, if terminal quantity constantly increases the storage demand making existing server can not meet ever-increasing raw data file, then dynamically increase new server, the storage capacity of raising system, initial data frame and original data object are deposited with json file layout, exist with key-value pair form hereof,
Step (2-3): carry out burst storage to data file on every station server of MongoDB privately owned cloud storage platform, each burst comprises some data blocks, if certain data block exceeds restriction size, then generates new multiple data blocks, a burst is a copy set, use the mode of cross replication backup, under multiple copies of same copy set are deployed in different servers, under the copy cross part of different burst being deployed on same station server simultaneously, finally configuration module less for the server cluster consumption of natural resource of privately owned for MongoDB cloud storage platform and routing module are deployed on each server, dispose multiple configuration module and routing module guarantees that the privately owned cloud storage platform of whole MongoDB is when a server delays machine, the ruuning situation of whole server cluster can not be affected.
8. method as claimed in claim 5, it is characterized in that, the step of described step (3) is:
Step (3-1): the data processing module of image data unloading computing platform is by reading the repository of preposition communication server cluster, and the file location information of data conversion storage is not carried out in inquiry;
Step (3-2): the file that do not carry out data conversion storage of data processing module from MongoDB privately owned cloud storage platform in read step (3-1) of image data unloading computing platform, and file is read start time write repository;
Step (3-3): data processing module obtains the archive information of the file not carrying out data conversion storage from data cache module, specifies different priority by large data management engine to dissimilar data, and carries out dispatching and managing; As carried out for transformer station, high voltage customer, low-voltage customer scheduling and the management that the different types of data such as voltage, load, electric current, table code data formulate different priorities;
Step (3-4): file is read the time write repository terminated, the business datum through large data management engine process is stored in relational database; The process of described large data management engine comprises: carry out corresponding rate verification, PT, CT, comprehensive multiplying power calculating, electricity calculating and always add group power calculation, forming business datum, by the business datum of formation stored in the storage facility located at processing plant of relational database and analysis storehouse,
Storage facility located at processing plant stores business datum and realizes terminal debugging, terminal parameter setting and image data real-time query manipulation,
Analyze the business datum of storehouse to storage facility located at processing plant and be further analyzed process, realize the statistical study gathering success ratio, the online rate of terminal and line loss qualification rate, and analysis result is write back storage facility located at processing plant, for inquiry.
9. method as claimed in claim 6, is characterized in that,
The step of described step (1-5) is: the original data object after resolving be stored in the form of a file in local disk, original data object stores hereof with the form of key-value pair, comprise data unique identification, markers, acquisition quality code, terminal freeze-off time and magnitude information, when file reaches the file size that presets or file does not change within a very long time, then put file for writing state, and by it stored in the privately owned cloud storage platform of MongoDB, the start time of uploading of file is write repository, cycle detection file status, until all raw data files in local disk file have all been uploaded, by files passe deadline write repository, and delete to reserve the real-time image data given of disk space storage terminal from local disk by uploading successful file,
The document base information of described step (1-5) comprise reference number of a document, preposition communication server numbering, file fullpath, filename, file whether deleted, high-low pressure mark, freeze mark, file declustering mark, file type and VIP mark curve day.
10. method as claimed in claim 7, is characterized in that,
The initial data frame of described step (2-2), the every a line in file stores an initial data frame, and each initial data frame comprises affiliated terminal iidentification, up-downgoing Data Identification and concrete Frame,
The original data object of described step (2-2), comprises data unique identification, markers, acquisition quality code, terminal freeze-off time and value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410482773.8A CN104317800A (en) | 2014-09-19 | 2014-09-19 | Hybrid storage system and method for mass intelligent power utilization data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410482773.8A CN104317800A (en) | 2014-09-19 | 2014-09-19 | Hybrid storage system and method for mass intelligent power utilization data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104317800A true CN104317800A (en) | 2015-01-28 |
Family
ID=52373032
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410482773.8A Pending CN104317800A (en) | 2014-09-19 | 2014-09-19 | Hybrid storage system and method for mass intelligent power utilization data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104317800A (en) |
Cited By (40)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104615736A (en) * | 2015-02-10 | 2015-05-13 | 上海创景计算机系统有限公司 | Quick analysis and storage method of big data based on database |
| CN104657462A (en) * | 2015-02-10 | 2015-05-27 | 北京宇航系统工程研究所 | Massive measurement data near-real-time storage method |
| CN104834985A (en) * | 2015-02-15 | 2015-08-12 | 杭州智光一创科技有限公司 | Synthetic data active information service method for electric power dispatching |
| CN105183371A (en) * | 2015-08-14 | 2015-12-23 | 山东大学 | Migration balancing policy based electricity-consuming information distributed file storage method and apparatus |
| CN105243632A (en) * | 2015-10-26 | 2016-01-13 | 深圳荣亚物联科技有限公司 | Cloud management based air pollution monitoring system and method |
| CN105760452A (en) * | 2016-02-04 | 2016-07-13 | 深圳市嘉力达实业有限公司 | Method and system for collection, processing and storage of high-concurrency mass data |
| CN106250517A (en) * | 2016-08-02 | 2016-12-21 | 广东电网有限责任公司中山供电局 | The storage system and method for regional power grid power consumption data |
| CN106528635A (en) * | 2016-10-12 | 2017-03-22 | 青岛大学 | Two-dimensional data storage system, storage method and query method |
| CN106570151A (en) * | 2016-10-28 | 2017-04-19 | 上海斐讯数据通信技术有限公司 | Data collection processing method and system for mass files |
| CN106776139A (en) * | 2016-12-16 | 2017-05-31 | 贵州中小乾信金融信息服务有限公司 | Suitable for the consumption information backup management system of big data |
| CN106933507A (en) * | 2016-11-08 | 2017-07-07 | 朱寅 | A kind of method that data dump to the subscriber computer specified automatically |
| CN107391764A (en) * | 2017-08-31 | 2017-11-24 | 江西博瑞彤芸科技有限公司 | Business datum querying method |
| CN107506438A (en) * | 2017-08-23 | 2017-12-22 | 福建星瑞格软件有限公司 | A kind of data processing storage method and device for Internet of Things |
| CN107615262A (en) * | 2015-03-31 | 2018-01-19 | 交互智能集团有限公司 | The system and method survived offline |
| CN108074086A (en) * | 2016-11-14 | 2018-05-25 | 咪咕音乐有限公司 | Charge system |
| CN108460070A (en) * | 2017-12-21 | 2018-08-28 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device and equipment based on database |
| CN109145061A (en) * | 2018-07-26 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of method, database and the server of database distribution session object |
| CN109246254A (en) * | 2018-11-29 | 2019-01-18 | 国网重庆市电力公司 | The data acquisition communications platform and communication means for supporting large-scale electric energy table directly to adopt |
| CN109508318A (en) * | 2018-11-15 | 2019-03-22 | 北京金山云网络技术有限公司 | A kind of amount of storage statistical method, device, electronic equipment and readable storage medium storing program for executing |
| CN109977173A (en) * | 2019-04-02 | 2019-07-05 | 山东浪潮云信息技术有限公司 | Distribution, plug-in type data monitoring system and method based on cloud data center |
| CN110309205A (en) * | 2019-07-10 | 2019-10-08 | 上海诚数信息科技有限公司 | A kind of data-optimized extraction system of credit decision and method |
| CN110705914A (en) * | 2019-10-18 | 2020-01-17 | 上海应用技术大学 | Industrial electricity data processing system |
| CN110716899A (en) * | 2019-09-25 | 2020-01-21 | 重庆紫光华山智安科技有限公司 | Method, system, medium and electronic terminal for quickly reading files and pictures |
| CN111143369A (en) * | 2019-12-27 | 2020-05-12 | 合肥美的智能科技有限公司 | Cloud service data unloading method and device, local server and electronic equipment |
| CN111143280A (en) * | 2019-12-31 | 2020-05-12 | 广州广电运通信息科技有限公司 | Data scheduling method, system, device and storage medium |
| CN111737325A (en) * | 2020-05-25 | 2020-10-02 | 南京华盾电力信息安全测评有限公司 | Power data analysis method and device based on big data technology |
| CN111897771A (en) * | 2020-07-24 | 2020-11-06 | 宁夏隆基宁光仪表股份有限公司 | Gas meter message testing method and system based on intelligent analysis model |
| CN106781415B (en) * | 2017-01-20 | 2021-03-02 | 中国电力科学研究院 | Method and system for automatically maintaining communication parameters of centralized meter reading collector |
| CN112542886A (en) * | 2020-12-01 | 2021-03-23 | 广东电网有限责任公司广州供电局 | Integrated control system for power grid |
| CN112988500A (en) * | 2019-12-17 | 2021-06-18 | 上海商汤智能科技有限公司 | Method, device and system for monitoring file storage system |
| CN113129063A (en) * | 2021-04-20 | 2021-07-16 | 国网江西省电力有限公司信息通信分公司 | Electricity charge calculation and distribution method and system based on cloud platform and data center |
| CN113568587A (en) * | 2021-09-22 | 2021-10-29 | 深圳市中天碧姆科技有限公司 | Smart city real-time data processing method, system and storage medium |
| CN113706102A (en) * | 2021-08-25 | 2021-11-26 | 宁夏隆基宁光仪表股份有限公司 | Data processing method based on ELK tool batch production meter |
| CN113849119A (en) * | 2020-08-28 | 2021-12-28 | 京东科技控股股份有限公司 | Storage method, storage device, and computer-readable storage medium |
| CN114218245A (en) * | 2022-02-22 | 2022-03-22 | 国网北京市电力公司 | Data processing method and device and electronic equipment |
| CN114328389A (en) * | 2021-12-31 | 2022-04-12 | 浙江汇鼎华链科技有限公司 | Big data file analysis processing system and method under cloud computing environment |
| CN114326605A (en) * | 2021-11-16 | 2022-04-12 | 中铁十四局集团房桥有限公司 | Tube sheet production informatization management optimization system and optimization method thereof |
| CN115033603A (en) * | 2022-06-16 | 2022-09-09 | 支付宝(杭州)信息技术有限公司 | Data storage method, data query method, data storage device and data query device |
| CN116633952A (en) * | 2023-07-25 | 2023-08-22 | 常州辉途智能科技有限公司 | Data processing system and processing method for pasture |
| CN116701412A (en) * | 2023-08-07 | 2023-09-05 | 中煤科工集团信息技术有限公司 | Mining data storage system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100332479A1 (en) * | 2009-06-30 | 2010-12-30 | Anand Prahlad | Performing data storage operations in a cloud storage environment, including searching, encryption and indexing |
| CN102727127A (en) * | 2012-06-22 | 2012-10-17 | 唐学兵 | Toilet for conveniently cleaning buttocks |
| CN103365971A (en) * | 2013-06-27 | 2013-10-23 | 北京中科金财科技股份有限公司 | Mass data access processing system based on cloud computing |
-
2014
- 2014-09-19 CN CN201410482773.8A patent/CN104317800A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100332479A1 (en) * | 2009-06-30 | 2010-12-30 | Anand Prahlad | Performing data storage operations in a cloud storage environment, including searching, encryption and indexing |
| CN102727127A (en) * | 2012-06-22 | 2012-10-17 | 唐学兵 | Toilet for conveniently cleaning buttocks |
| CN103365971A (en) * | 2013-06-27 | 2013-10-23 | 北京中科金财科技股份有限公司 | Mass data access processing system based on cloud computing |
Non-Patent Citations (2)
| Title |
|---|
| 刘愉 等: "基于MongoDB的EHR存储方案研究与设计", 《中国数字医学》 * |
| 王勇 等: "大型供电企业电能计量自动化系统设计与应用", 《电测与仪表》 * |
Cited By (53)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104657462B (en) * | 2015-02-10 | 2017-12-22 | 北京宇航系统工程研究所 | A kind of magnanimity measurement data quasi real time storage method |
| CN104657462A (en) * | 2015-02-10 | 2015-05-27 | 北京宇航系统工程研究所 | Massive measurement data near-real-time storage method |
| CN104615736A (en) * | 2015-02-10 | 2015-05-13 | 上海创景计算机系统有限公司 | Quick analysis and storage method of big data based on database |
| CN104615736B (en) * | 2015-02-10 | 2017-10-27 | 上海创景计算机系统有限公司 | Big data fast resolving storage method based on database |
| CN104834985A (en) * | 2015-02-15 | 2015-08-12 | 杭州智光一创科技有限公司 | Synthetic data active information service method for electric power dispatching |
| CN107615262B (en) * | 2015-03-31 | 2021-03-23 | 交互智能集团有限公司 | System and method for offline survival |
| CN107615262A (en) * | 2015-03-31 | 2018-01-19 | 交互智能集团有限公司 | The system and method survived offline |
| CN105183371A (en) * | 2015-08-14 | 2015-12-23 | 山东大学 | Migration balancing policy based electricity-consuming information distributed file storage method and apparatus |
| CN105243632A (en) * | 2015-10-26 | 2016-01-13 | 深圳荣亚物联科技有限公司 | Cloud management based air pollution monitoring system and method |
| CN105760452A (en) * | 2016-02-04 | 2016-07-13 | 深圳市嘉力达实业有限公司 | Method and system for collection, processing and storage of high-concurrency mass data |
| CN105760452B (en) * | 2016-02-04 | 2020-03-10 | 深圳市嘉力达实业有限公司 | Method and system for acquiring, processing and storing high-concurrency mass data |
| CN106250517A (en) * | 2016-08-02 | 2016-12-21 | 广东电网有限责任公司中山供电局 | The storage system and method for regional power grid power consumption data |
| CN106528635B (en) * | 2016-10-12 | 2019-04-12 | 青岛大学 | Storage system, storage method and the querying method of 2-D data |
| CN106528635A (en) * | 2016-10-12 | 2017-03-22 | 青岛大学 | Two-dimensional data storage system, storage method and query method |
| CN106570151A (en) * | 2016-10-28 | 2017-04-19 | 上海斐讯数据通信技术有限公司 | Data collection processing method and system for mass files |
| CN106933507A (en) * | 2016-11-08 | 2017-07-07 | 朱寅 | A kind of method that data dump to the subscriber computer specified automatically |
| CN108074086A (en) * | 2016-11-14 | 2018-05-25 | 咪咕音乐有限公司 | Charge system |
| CN106776139A (en) * | 2016-12-16 | 2017-05-31 | 贵州中小乾信金融信息服务有限公司 | Suitable for the consumption information backup management system of big data |
| CN106776139B (en) * | 2016-12-16 | 2020-10-02 | 贵州中小乾信金融信息服务有限公司 | Consumption information backup management system suitable for big data |
| CN106781415B (en) * | 2017-01-20 | 2021-03-02 | 中国电力科学研究院 | Method and system for automatically maintaining communication parameters of centralized meter reading collector |
| CN107506438A (en) * | 2017-08-23 | 2017-12-22 | 福建星瑞格软件有限公司 | A kind of data processing storage method and device for Internet of Things |
| CN107391764A (en) * | 2017-08-31 | 2017-11-24 | 江西博瑞彤芸科技有限公司 | Business datum querying method |
| CN108460070A (en) * | 2017-12-21 | 2018-08-28 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device and equipment based on database |
| CN109145061A (en) * | 2018-07-26 | 2019-01-04 | 阿里巴巴集团控股有限公司 | A kind of method, database and the server of database distribution session object |
| CN109508318A (en) * | 2018-11-15 | 2019-03-22 | 北京金山云网络技术有限公司 | A kind of amount of storage statistical method, device, electronic equipment and readable storage medium storing program for executing |
| CN109246254A (en) * | 2018-11-29 | 2019-01-18 | 国网重庆市电力公司 | The data acquisition communications platform and communication means for supporting large-scale electric energy table directly to adopt |
| CN109977173A (en) * | 2019-04-02 | 2019-07-05 | 山东浪潮云信息技术有限公司 | Distribution, plug-in type data monitoring system and method based on cloud data center |
| CN109977173B (en) * | 2019-04-02 | 2021-07-16 | 浪潮云信息技术股份公司 | Distributed and plug-in data monitoring system and method based on cloud data center |
| CN110309205A (en) * | 2019-07-10 | 2019-10-08 | 上海诚数信息科技有限公司 | A kind of data-optimized extraction system of credit decision and method |
| CN110309205B (en) * | 2019-07-10 | 2021-12-07 | 上海诚数信息科技有限公司 | Credit decision data optimization extraction system and method |
| CN110716899A (en) * | 2019-09-25 | 2020-01-21 | 重庆紫光华山智安科技有限公司 | Method, system, medium and electronic terminal for quickly reading files and pictures |
| CN110705914A (en) * | 2019-10-18 | 2020-01-17 | 上海应用技术大学 | Industrial electricity data processing system |
| CN112988500A (en) * | 2019-12-17 | 2021-06-18 | 上海商汤智能科技有限公司 | Method, device and system for monitoring file storage system |
| CN111143369A (en) * | 2019-12-27 | 2020-05-12 | 合肥美的智能科技有限公司 | Cloud service data unloading method and device, local server and electronic equipment |
| CN111143280A (en) * | 2019-12-31 | 2020-05-12 | 广州广电运通信息科技有限公司 | Data scheduling method, system, device and storage medium |
| CN111143280B (en) * | 2019-12-31 | 2023-10-10 | 广州广电运通信息科技有限公司 | Data scheduling method, system, device and storage medium |
| CN111737325A (en) * | 2020-05-25 | 2020-10-02 | 南京华盾电力信息安全测评有限公司 | Power data analysis method and device based on big data technology |
| CN111897771A (en) * | 2020-07-24 | 2020-11-06 | 宁夏隆基宁光仪表股份有限公司 | Gas meter message testing method and system based on intelligent analysis model |
| CN113849119A (en) * | 2020-08-28 | 2021-12-28 | 京东科技控股股份有限公司 | Storage method, storage device, and computer-readable storage medium |
| CN112542886A (en) * | 2020-12-01 | 2021-03-23 | 广东电网有限责任公司广州供电局 | Integrated control system for power grid |
| CN113129063A (en) * | 2021-04-20 | 2021-07-16 | 国网江西省电力有限公司信息通信分公司 | Electricity charge calculation and distribution method and system based on cloud platform and data center |
| CN113129063B (en) * | 2021-04-20 | 2024-05-03 | 国网江西省电力有限公司信息通信分公司 | Electric charge calculation issuing method and system based on cloud platform and data center platform |
| CN113706102A (en) * | 2021-08-25 | 2021-11-26 | 宁夏隆基宁光仪表股份有限公司 | Data processing method based on ELK tool batch production meter |
| CN113568587A (en) * | 2021-09-22 | 2021-10-29 | 深圳市中天碧姆科技有限公司 | Smart city real-time data processing method, system and storage medium |
| CN114326605A (en) * | 2021-11-16 | 2022-04-12 | 中铁十四局集团房桥有限公司 | Tube sheet production informatization management optimization system and optimization method thereof |
| CN114328389A (en) * | 2021-12-31 | 2022-04-12 | 浙江汇鼎华链科技有限公司 | Big data file analysis processing system and method under cloud computing environment |
| CN114328389B (en) * | 2021-12-31 | 2022-06-17 | 浙江汇鼎华链科技有限公司 | Big data file analysis processing system and method under cloud computing environment |
| CN114218245A (en) * | 2022-02-22 | 2022-03-22 | 国网北京市电力公司 | Data processing method and device and electronic equipment |
| CN115033603A (en) * | 2022-06-16 | 2022-09-09 | 支付宝(杭州)信息技术有限公司 | Data storage method, data query method, data storage device and data query device |
| CN116633952B (en) * | 2023-07-25 | 2023-09-29 | 常州辉途智能科技有限公司 | Data processing system and processing method for pasture |
| CN116633952A (en) * | 2023-07-25 | 2023-08-22 | 常州辉途智能科技有限公司 | Data processing system and processing method for pasture |
| CN116701412A (en) * | 2023-08-07 | 2023-09-05 | 中煤科工集团信息技术有限公司 | Mining data storage system |
| CN116701412B (en) * | 2023-08-07 | 2023-10-31 | 中煤科工集团信息技术有限公司 | Mining data storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104317800A (en) | Hybrid storage system and method for mass intelligent power utilization data | |
| CN110022226B (en) | Object-oriented data acquisition system and acquisition method | |
| CN111125089B (en) | Time sequence data storage method, device, server and storage medium | |
| CN105278373A (en) | Substation integrated information processing system realizing method | |
| CN106503276A (en) | A kind of method and apparatus of the time series databases for real-time monitoring system | |
| CN102779138B (en) | The hard disk access method of real time data | |
| CN102982489A (en) | Power customer online grouping method based on mass measurement data | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN106095828B (en) | A kind of curve data storage scheme based on power information acquisition terminal | |
| CN102158349A (en) | Log management device and method thereof | |
| CN102254024A (en) | Mass data processing system and method | |
| CN103955509A (en) | Quick search method for massive electric power metering data | |
| CN104599032A (en) | Distributed memory power grid construction method and system for resource management | |
| CN103294413B (en) | Support the distributed memory real-time storage device and method of magnanimity acquisition terminal | |
| CN102332004B (en) | Data processing method and system for managing mass data | |
| CN111147959B (en) | Processing method and device for change information of centralized meter reading system equipment and computer equipment | |
| CN104362757A (en) | Transformer substation on-line monitoring system based on IEC61850 | |
| CN110781184A (en) | Data table construction method, device, equipment and storage medium | |
| CN102111285A (en) | Method and system for managing log information | |
| CN103325074A (en) | Electric system real-time data processing method | |
| CN110727406A (en) | Data storage scheduling method and device | |
| CN103823807A (en) | Data de-duplication method, device and system | |
| CN105787090A (en) | Index building method and system of OLAP system of electric data | |
| CN109471860B (en) | Large-scale charging pile data processing method and device for electric vehicle charging network | |
| CN110134690B (en) | Method and system for rapidly storing oversized data of Oracle database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150128 |
|
| WD01 | Invention patent application deemed withdrawn after publication |