CN106126644A - Recorded wave file based on NoSQL data base storage method - Google Patents
Recorded wave file based on NoSQL data base storage method Download PDFInfo
- Publication number
- CN106126644A CN106126644A CN201610465934.1A CN201610465934A CN106126644A CN 106126644 A CN106126644 A CN 106126644A CN 201610465934 A CN201610465934 A CN 201610465934A CN 106126644 A CN106126644 A CN 106126644A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- channel
- storage
- row
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
It is an object of the invention to provide a kind of recorded wave file based on NoSQL data base storage method, recorded wave file is stored by the mode that the present invention uses relevant database and NoSQL data base to combine, relational data library storage record ripple configuration file, NoSQL data base uses HBase database purchase recorder data file.The inventive method includes following content: the 1) storage of record ripple configuration file；1.1) record ripple configuration file list structure definition；1.2) record ripple configuration file is stored；2) storage of recorder data file；2.1) recorder data definition file storage；2.2) recorder data file is stored.
Description
Technical field
The present invention relates to storage and the analysis of recorder data file, be under a kind of big data environment, based on NoSQL data base
A kind of storage method.
Background technology
Along with the fast development in digital information epoch, quantity of information is also in explosive increase situation.Enjoy to the full the mankind
Information that informationization brings, convenience and quick time, also make the digital information resources in the whole world just entering into one unprecedented
The quickly rise period.According to IDC add up, within 2011, global metadata amount has reached 1.8ZB, be equivalent to the whole world produce per capita 200GB with
On data, and also will continue to increase with the speed of annual 50%.Under this surging data tide attacked, each neck of society
Territory also will start its datumization process.No matter academia, business circles or government, all will inevitably enter " during big data
Generation ".As the basic energy resource support system of the world's second-biggest economy, there can be no exception whatever for China electric power industry.
In power industry, the growth of data volume also presents similar situation.In recent years, electric power trade informationization also obtained
Significant progress, China's Electric Power Enterprise Information originates from the sixties in 20th century, from the initial electrical production automation to 80 year
The generation IT application in management construction with financial computerization as representative, then to large scale business enterprise's informatization in recent years, the most adjoint
The all-round construction of intelligent power network of future generation, with Internet of Things and cloud computing, the IT technology of new generation as representative is in power industry
Extensive application, electric power data resource starts sharp increase and reaches a certain scale.In the long run, as China's economic
" barometer " of social development, electric power data closely and widely contacts with economic development with it, it will present unmatched
Straight outside, to China's socio-economic development so that progress of human society also will form the most powerful motive force.
Summary of the invention
It is an object of the invention to provide a kind of recorded wave file based on NoSQL data base storage method, the present invention uses
Recorded wave file is stored by the mode that relevant database and NoSQL data base combine, relational data library storage record ripple
Configuration file, NoSQL data base uses HBase database purchase recorder data file.
Technical scheme is as follows:
A kind of recorded wave file based on NoSQL data base storage method, including following content:
1) storage of ripple configuration file is recorded
1.1) record ripple configuration file list structure definition
Record ripple configuration filename ends up with " .cfg ", and the record ripple configuration file of its storage is structural data, specifically includes
The contents such as sample frequency, the number of channel, line frequency and channel information.These contents use relation library storage, define four for this
Table, is respectively " recorded wave file describes table, recorded wave file frequency meter, quantity of state point table, analog quantity point table ", and each table particular content is such as
Shown in table 1 below-1,1-2,1-3,1-4, corresponding list structure and relation thereof as shown in Figure 1:
Shown in FIG, CFG_FILE_Desc, Sample_Rate_List, Analog_Channel, Status_
Channel is four tables defined in relation storehouse, stores profile information；TimesDig and TimesAna is row race data base
(HBase) tables of data defined in, stores data file information.Configuration file describes table and other several tables are the passes of one-to-many
System, specifically each field description is as shown in the table:
Table 1-1
Table 1-2
Table 1-3
Table 1-4
1.2) storing record ripple configuration file, idiographic flow is:
(a1) the four kinds of tables created in relational database in step 1.1；
(b1) read configuration file, and resolve to the data of each list structure；
(c1) data are inserted, after inserting the data of table CFG_Fi le_Desc structure according to the order of sublist after first parent table
Can produce globally unique GUID, GUID inputs as the critical field of other table, otherwise according to " referential integrity " of table
Definition, can store data failure.
2) storage of recorder data file
2.1) recorder data definition file storage
Recorder data file contains the sampled value after each remote measurement and remote signalling channel compressions, remote measurement and remote signalling passage in record
Defined in ripple configuration file.The partial data of the inquiry of a recorder data file often channel of a data file, adopts
Store with a kind of row race data base of NoSQL data base.According to securing two kinds of passages of remote measure and communication, row race data base wound
Build two tables of data: telemeter TimesDig and remote signalling table TimesAna, as shown in Figure 1.The two table major key use version number+
The mode of GUID+ channel number, convenient passage of pressing searches data；Defined in each record ripple configuration file channel quantity differ,
Therefore channel name+end counted as row name, be stored as multiple row；Owing to row race data press byte storage of array data, therefore
Convert the data into byte array to store.Concrete structure such as table 2 below:
Table 2
2.2) storing recorder data file, idiographic flow is:
(a2) press passage and read recorder data file content；
(b2) produce partition data according to the number of channel and form the array of multiple data；
(c2) in row race data base, row name is set by row is strong, stores data.
The present invention contrasts prior art, has the following advantages:
1. during recorded wave file stores data base, centralized management, add index, facilitate inquiry, change file and deposit
The problem that storage inquiry is slower.
2., during recorded wave file stores data base, define concentration, unified storage mode, facilitate the expansion of memory capacity
Fill, change locally stored restriction.
3. the present invention provides the storage mode of a kind of big data, by extendible row race data, it is possible to store the biggest
The recorded wave file of amount.By these substantial amounts of recorder datas, machine learning, intellectual analysis can be carried out, produce bigger value,
Change recorded wave file remotely to transfer, the artificial old pattern analyzed.
Accompanying drawing explanation
Fig. 1 recorded wave file E-R is schemed.
Detailed description of the invention
The following is a real case according to method flow, the file of parsing is " DRL600C_DRec_119313_
20140531_065437_121.cfg ", contain the establishment of database table, the content such as insertion of data.Data base is utilized to be built
Vertical index, facilitates the inquiry of data, for the intellectual analysis under big data, not in the range of this method describes.
1., according to E-R G-Design, create relation storehouse table, with sql statement as standard, by establishment, there is the number that is mutually related
According to storehouse table, particularly as follows:
A) table CFG_File_Desc is created:
Create table CFG_File_Desc(
Station_Namevarchar(64)NOT NULL,
Recording_Device_IDvarchar(64)not null,
Revision_Year integer,
Tatal_Channel_Count integer,
Analog_Channel_Count integer,
Status_Channel_Count integer,
Line_Frequency FLOAT,
Sample_Rate_Count integer,
File_Start_Time TIMESTAMP,
Trigger_Time TIMESTAMP,
File_Type integer,
Time_Multiplier integer,
GUID serial unique,
HDR_File TEXT,
INF_File TEXT,
PRIMARY KEY(Station_Name,Recording_Device_ID,Trigger_Time)
)
B) table: Sample_Rate_List is created
Create tablesample_rate_list(
GUID integer references CFG_File_Desc(GUID),
Number integer,
Samp_Rate float,
Start_Samp integer,
End_Samp integer,
PRIMARY KEY(GUID,Number)
)
C) Table A nalog_Channel is created
Create tableanalog_channel(
GUID integer references CFG_File_Desc(GUID),
Channel_ID integer,
Channel_Namevarchar(64),
Phase_IDvarchar(2),
Monitored_Componentvarchar(64),
Channel_Unitsvarchar(32),
Channel_Multiplier float,
Channel_Offset float,
Channel_Skew float,
Range_Min_Limit_Value integer,
Range_Max_Limit_Value integer,
Channel_Ratio_Primary float,
Channel_Ratio_Secondary float,
Data_Primary_Secondary integer,
PRIMARY KEY(GUID,Channel_ID)
)
D) table Status_Channel is created
Create table test1.status_channel(
GUID integer references test1.CFG_File_Desc(GUID),
Channel_ID integer,
Channel_Namevarchar(64),
Phase_IDvarchar(2),
Monitored_Componentvarchar(64),
Normal_State integer,
PRIMARY KEY(GUID,Channel_ID)
)
2. read configuration file * .cfg, obtain related content, formation sql insertion statement:
A) data in CFG_File_Desc table are inserted:
insert
intocfg_file_desc(station_name,recording_device_ID,revision_year,
tatal_channel_count,analog_channel_count,status_channel_count,l ine_frequenc
y,sample_rate_count,file_start_time,trigger_time,file_type,time_multip lier,
File_name) values (' Zhejiang. halfway up the hill factory ', ' #8 transformer unit therefore record DRL600A', 1999,288,96,192,
50.000000,2,'2014-05-3106:05:37.001000','2014-05-31 06:05:37.121000',1,
1.000000,'DRL600C_DRec_119313_20140531_065437_121.cfg')
B) obtain the GUID automatically generated, after performing following sql, obtain guid=3658
Select station_device_id fromcfg_file_desc where Station_Name=' Zhejiang.
Halfway up the hill factory ' and Recording_Device_ID='#8 transformer unit thus record DRL600A'and trigger_time='
2014-05-31 06:05:37.121000'
C) insert data in Sample_Rate_List table: have two frequency bands here, turn round and look at and insert two records
insert
intosample_rate_list(station_device_id,number,samp_rate,start_samp,
end_samp)values(3658,1,4800.000000,0,576)
insert
intosample_rate_list(station_device_id,number,samp_rate,start_samp,
end_samp)values(3658,2,4800.000000,577,45504)
D) inserting data in Analog_Channel table, the following is a passage content, actual meeting circular insertion is all logical
Track data.
insert intoanalog_channel(station_device_id,channel_id,channel_name,
phase_id,monitored_Component,channel_units,channel_multiplier,channel_offset,
channel_skew,range_min_limit_value,range_max_limit_value,channel_ratio_
Primary, channel_ratio_secondary, data_primary_secondary) values (3658,0, ' electromotor
Set end voltage Ua', ' A', ' electromotor ', ' V', 0.006200,0.000000,0.000000 ,-32767,32767,
1.000000,1.000000,2)
E) inserting data in Status_Channel table, the following is a passage content, actual meeting circular insertion is all logical
Track data.
insert intostatus_channel(station_device_id,channel_id,channel_name,
Phase_id, moni tored_component, normal_state) values (3658,0, ' main transformer protection AB cabinet main transformer is poor
Dynamic protection act ', ", ", 69317296)
3. create row race database table
Table name TimesDig and TimesAna by definition, calling system interface creates tables of data.
4. insert row race data base
A) row is produced according to definition strong: 1003658001,1 is version number, and 003658 is guid, and 001 is channel number
B) according to configuration file two row names of definition:
i.4800.000000,576
ii.4800.000000,45504
C) it is each column storage
D) it is recycled to next passage and continues storage.
Embodiments of the present invention are not limited to this, under the present invention above-mentioned basic fundamental thought premise, according to this area
Present invention is made the amendment of other various ways, is replaced or change by ordinary technical knowledge and customary means, all falls within
Within the scope of rights protection of the present invention.
Claims (2)
1. a recorded wave file based on NoSQL data base storage method, it is characterised in that include following content:
1) storage of ripple configuration file is recorded；
1.1) record ripple configuration file list structure definition；
Record ripple configuration filename ends up with " .cfg ", and the record ripple configuration file of storage is structural data, and particular content includes adopting
Sample frequency, the number of channel, line frequency and channel information；These contents use relation library storage, define four tables for this, are respectively
" recorded wave file describes table, recorded wave file frequency meter, quantity of state point table, analog quantity point table "；
1.2) storing record ripple configuration file, idiographic flow is:
(a1) create step 1.1 in relational database) in four kinds of tables；
(b1) read configuration file, and resolve to the data of each list structure；
(c1) according to the order of sublist after first parent table, data are inserted, can produce after the data of insertion table CFG_Fi le_Desc structure
Raw globally unique GUID, GUID input as the critical field of other table, otherwise according to " referential integrity " definition of table,
Data failure can be stored；
2) storage of recorder data file
2.1) recorder data definition file storage
Recorder data file contains the sampled value after each remote measurement and remote signalling channel compressions, remote measurement and remote signalling passage and joins at record ripple
Put defined in file.The partial data of the inquiry of a recorder data file often channel of a data file, uses
A kind of row race data base of NoSQL data base stores；According to securing two kinds of passages of remote measure and communication, row race data base create
Two tables of data: telemeter TimesDig and remote signalling table TimesAna.
2.2) recorder data file is stored, idiographic flow is:
(a2) press passage and read recorder data file content；
(b2) produce partition data according to the number of channel and form the array of multiple data；
(c2) in row race data base, row name is set by row is strong, stores data.
Method the most according to claim 1, it is characterised in that: described telemeter TimesDig's and remote signalling table TimesAna
Major key uses the mode of version number's+GUID+ channel number, convenient passage of pressing to search data；Defined in each record ripple configuration file
Channel quantity differ, therefore channel name+end counted as row name, is stored as multiple row；Owing to row race data press byte number
Group storage data, therefore convert the data into byte array and store.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610465934.1A CN106126644A (en) | 2016-06-22 | 2016-06-22 | Recorded wave file based on NoSQL data base storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610465934.1A CN106126644A (en) | 2016-06-22 | 2016-06-22 | Recorded wave file based on NoSQL data base storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106126644A true CN106126644A (en) | 2016-11-16 |
Family
ID=57268705
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610465934.1A Pending CN106126644A (en) | 2016-06-22 | 2016-06-22 | Recorded wave file based on NoSQL data base storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106126644A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109739854A (en) * | 2018-12-27 | 2019-05-10 | 新华三大数据技术有限公司 | A kind of date storage method and device |
| CN110032604A (en) * | 2019-02-02 | 2019-07-19 | 阿里巴巴集团控股有限公司 | Data storage device, transfer device and data bank access method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040010415A1 (en) * | 2000-02-23 | 2004-01-15 | Yoon-Deuk Seo | Apparatus and method for searching digital audio data file from media where digital audio data files are recorded |
| CN103092992A (en) * | 2013-02-17 | 2013-05-08 | 南京师范大学 | Vector data preorder quadtree coding and indexing method based on Key / Value type NoSQL (Not only SQL) |
| CN104750809A (en) * | 2015-03-26 | 2015-07-01 | 中国科学院软件研究所 | Storage method for supporting relation model and blended data of key-value structure |
| CN105488058A (en) * | 2014-09-17 | 2016-04-13 | 华中科技大学 | Fault recording format conversion method for power system |
-
2016
- 2016-06-22 CN CN201610465934.1A patent/CN106126644A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040010415A1 (en) * | 2000-02-23 | 2004-01-15 | Yoon-Deuk Seo | Apparatus and method for searching digital audio data file from media where digital audio data files are recorded |
| CN103092992A (en) * | 2013-02-17 | 2013-05-08 | 南京师范大学 | Vector data preorder quadtree coding and indexing method based on Key / Value type NoSQL (Not only SQL) |
| CN105488058A (en) * | 2014-09-17 | 2016-04-13 | 华中科技大学 | Fault recording format conversion method for power system |
| CN104750809A (en) * | 2015-03-26 | 2015-07-01 | 中国科学院软件研究所 | Storage method for supporting relation model and blended data of key-value structure |
Non-Patent Citations (1)
| Title |
|---|
| 白红伟等: "基于Hadoop的录波数据的存储与访问", 《河北省科学院学报》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109739854A (en) * | 2018-12-27 | 2019-05-10 | 新华三大数据技术有限公司 | A kind of date storage method and device |
| CN110032604A (en) * | 2019-02-02 | 2019-07-19 | 阿里巴巴集团控股有限公司 | Data storage device, transfer device and data bank access method |
| CN110032604B (en) * | 2019-02-02 | 2021-12-07 | 北京奥星贝斯科技有限公司 | Data storage device, translation device and database access method |
| US11226961B2 (en) | 2019-02-02 | 2022-01-18 | Beijing Oceanbase Technology Co., Ltd. | Data storage apparatus, translation apparatus, and database access method |
| US11797533B2 (en) | 2019-02-02 | 2023-10-24 | Beijing Oceanbase Technology Co., Ltd. | Data storage apparatus, translation apparatus, and database access method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107402976B (en) | Power grid multi-source data fusion method and system based on multi-element heterogeneous model | |
| CN103177094B (en) | Cleaning method of data of internet of things | |
| CN104850601B (en) | Police service based on chart database analyzes application platform and its construction method in real time | |
| CN102332030A (en) | Data storing, managing and inquiring method and system for distributed key-value storage system | |
| CN107391502B (en) | Time interval data query method and device and index construction method and device | |
| CN104317966A (en) | Dynamic indexing method applied to quick combined querying of big electric power data | |
| CN103116625A (en) | Volume radio direction finde (RDF) data distribution type query processing method based on Hadoop | |
| CN102609490B (en) | Column-storage-oriented B+ tree index method for DWMS (data warehouse management system) | |
| CN111159180A (en) | Data processing method and system based on data resource directory construction | |
| Wang et al. | Distributed storage and index of vector spatial data based on HBase | |
| CN107766541B (en) | Distribution and utilization global full-volume data transmission and storage method and device, and electronic equipment | |
| CN104317864A (en) | Automatic identification method of information model based on IEC61850 logical nodes | |
| Wang et al. | Massive remote sensing image data management based on HBase and GeoSOT | |
| CN102375827A (en) | Method for fast loading versioned electricity network model database | |
| CN115858513A (en) | Data governance method, data governance device, computer equipment and storage medium | |
| Kang et al. | Research on construction methods of big data semantic model | |
| Liu et al. | Construction of power projects knowledge graph based on graph database Neo4j | |
| CN106126644A (en) | Recorded wave file based on NoSQL data base storage method | |
| CN114443656A (en) | Customizable automated data model analysis tool and use method thereof | |
| CN112540987A (en) | Big data management system of distribution and utilization electricity based on data mart | |
| Bakalov et al. | A network model for the utility domain | |
| Humaira et al. | A survey on automatic mapping of ontology to relational database schema | |
| Li et al. | A data warehouse architecture supporting energy management of intelligent electricity system | |
| Scriney et al. | Efficient cube construction for smart city data | |
| CN103336810A (en) | Power distribution network topology analysis method based on multi-core computer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161116 |