CN104834487B - Remote signalling data compression based on slowly varying dimension form, which is stored and looked into, lacks method of filling a vacancy - Google Patents
Remote signalling data compression based on slowly varying dimension form, which is stored and looked into, lacks method of filling a vacancy Download PDFInfo
- Publication number
- CN104834487B CN104834487B CN201510268933.3A CN201510268933A CN104834487B CN 104834487 B CN104834487 B CN 104834487B CN 201510268933 A CN201510268933 A CN 201510268933A CN 104834487 B CN104834487 B CN 104834487B
- Authority
- CN
- China
- Prior art keywords
- remote signalling
- vacancy
- filename
- data
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention provides a kind of remote signalling data compression storage based on slowly varying dimension form and the method for filling a vacancy that lacks is looked into, including：New and old Documents Comparison step, wherein by the way that compared with deposit file name in database, the remote signalling file for judging access is new file or ancient deed by the filename marked with having time of the remote signalling file of access；Normal storage step, wherein in the case of judging that the remote signalling file of access is new file in new and old Documents Comparison step, increases newest state on existing remote signalling data, and the remote signalling data of generating state change is added in database table；Storage of filling a vacancy step, wherein in the case of judging that the remote signalling file of access is not new file in new and old Documents Comparison step, increases the remote signalling state of past particular point in time in existing history remote signalling data；Filename preserves step, wherein after completing normal storage step or filling a vacancy storage step, the filename of remote signalling data is preserved with common format.
Description
Technical field
The present invention relates to the storage that status information is cut-off in business data field of storage, especially electric system；More
Say body, the present invention relates to a kind of remote signalling data compression storage based on slowly varying dimension form and look into scarce method of filling a vacancy.
Background technology
National grid East China branch manages regional power grid maximum in world wide, breaker, knife in administrative area
The Switch equipments such as lock, grounding switch share more than 70,000, and each equipment is per minute to be produced 1 remote signalling data (switch cut-offs shape
State) D5000 dispatching automations EMS system (Energy Management System) is sent to, scheduling data center obtains this from D5000EMS again
How a little remote signalling datas, timely and effectively store, be a technological difficulties using these data.
Stored for data, most common mode is reference paper content, builds up corresponding table in the database, then
Data are saved into database table by programming using ETL (Extract-Transform-Load, extraction-conversion-loading) instrument
In.For remote signalling data, shown in the example that is preserved as this form storage table such as remote signalling data of table 1 with common format.
Table 1
| ID | t_id | INDICATION |
| 1291121921257 | 2015-3-10 16:33 | 0 |
| 1291121921257 | 2015-3-10 16:34 | 0 |
| 1291121921257 | 2015-3-10 16:35 | 1 |
| 1291121921257 | 2015-3-10 16:36 | 1 |
| 1291121921257 | 2015-3-10 16:37 | 1 |
| 1291121921257 | 2015-3-10 16:38 | 1 |
| 1291121921257 | 2015-3-10 16:39 | 1 |
| 1291121921257 | 2015-3-10 16:40 | 1 |
| 1291121921257 | 2015-3-10 16:41 | 0 |
| 1291121921257 | 2015-3-10 16:42 | 0 |
| 1291121921257 | 2015-3-10 16:43 | 1 |
| 1291121921257 | 2015-3-10 16:44 | 1 |
| 1291121921257 | 2015-3-10 16:45 | 1 |
| 1291121921257 | 2015-3-10 16:46 | 1 |
| 1291121921257 | 2015-3-10 16:47 | 1 |
| 1291121921257 | 2015-3-10 16:48 | 0 |
| 1291121921257 | 2015-3-10 16:49 | 0 |
| 1291121921257 | 2015-3-10 16:50 | 0 |
| 1291121921257 | 2015-3-10 16:51 | 1 |
| 1291121921257 | 2015-3-10 16:33 | 0 |
| 1291121921257 | 2015-3-10 16:34 | 0 |
If remote signalling data is a uncertain changing value, this form can only be used to preserve data, if data volume is few,
It is also very convenient with this form.But remote signalling data measuring point has more than 70,000,1 section file is preserved within 1 minute, if pressing common format,
To preserve daily 100,000,000 a plurality of, the storage and use of data are all inconvenient；Found after analyzing remote signalling data, remote signalling data only has
0th, 1 two kinds of numerical value, and every kind of numerical value (numerical value represents state of cut-offfing, and 0- is disconnected, 1- closures) can all maintain the long period.Due to distant
Letter data has the characteristics that data volume is big, storage is frequent, remote signalling value is fixed, state change is few, through research, finds slowly varying dimension
Comparing the storage of suitable remote signalling data, table 2 is based on 1 data of table, is converted into the slowly varying storage form tieed up after form,
I.e. remote signalling data is with slowly varying dimension checkering storage.
Table 2
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 1 | 1291121921257 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 2 | 1291121921257 | 2015-3-10 16:35 | 2015-3-10 16:41 | 1 |
| 3 | 1291121921257 | 2015-3-10 16:41 | 2015-3-10 16:43 | 0 |
| 4 | 1291121921257 | 2015-3-10 16:43 | 2015-3-10 16:48 | 1 |
| 5 | 1291121921257 | 2015-3-10 16:48 | 2015-3-10 16:51 | 0 |
| 6 | 1291121921257 | 2015-3-10 16:51 | 1 |
As can be seen from the above table, slowly varying dimension is that the remote signalling data of same state is recorded in a line, passes through VALID_
DATE (entry-into-force time) and INVALID_DATE (out-of-service time) records the duration of this state.By using slowly varying
Tie up form and preserve data, remote signalling data amount of storage is greatly compressed, and (in real-time working, switch often works several days, several months
Do not cut-off once within several years even, remote signalling data only just preserves once when cut-offfing change), efficiently solve remote signalling data and deposit
Storage, use problem.
This compression storing data technology of slowly varying dimension different field can use arrive, such as employee work experience (work
Work starts, the end time), overhaul of the equipments (maintenance starts, the end time) etc. is all preserved using similar form.But as remote signalling data
So big data volume, so fast storage time, the requirement of the so high quality of data, general production management system is difficult to see
Arrive, seldom can run into shortage of data inspection, the problems such as data are filled a vacancy.
The content of the invention
The technical problems to be solved by the invention are to be directed to that drawbacks described above exists in the prior art, there is provided one kind is based on slow
The remote signalling data compression of change dimension form, which is stored and looked into, lacks method of filling a vacancy, and slowly varying dimension can be applied to electric power remote signalling number by it
According to storage, and remote signalling data quick storage, shortage of data inspection are solved, point data is lost and the technical barrier such as fills a vacancy, realize remote signalling
The storage of data high compression ratio, high quality so that remote signalling data preserves for a long time, using being possibly realized.
In order to realize above-mentioned technical purpose, according to the present invention, there is provided a kind of remote signalling number based on slowly varying dimension form
The method of filling a vacancy that lacks is stored and looked into according to compression, including：New and old Documents Comparison step：By by the band having time of the remote signalling file of access
For the filename of mark compared with deposit file name in database, the remote signalling file for judging access is new file or old text
Part；Normal storage step：In the case that the remote signalling file for judging to access in new and old Documents Comparison step is new file, existing
Increase newest state on remote signalling data, and the remote signalling data of generating state change is added in database table；Fill a vacancy storage
Step：In the case that the remote signalling file for judging to access in new and old Documents Comparison step is not new file, in existing history remote signalling
Increase the remote signalling state of past particular point in time in data；Filename preserves step：Complete normal storage step or filling a vacancy
After being put in storage step, the filename of remote signalling data is preserved with common format.
Preferably, the method further includes shortage of data inspection processing, and the shortage of data inspection processing includes：Generate distant
The full dose filename of message part, the wherein form of full dose filename form for " implementor name+time+file type ", and are saved into
In full dose table of file name；Filename is lacked according to full dose filename lookup；Carry out missing filename to preserve, wherein file will be lacked
Name is saved in database table.
Preferably, when generating the full dose file of remote signalling file, by all time points in the programming automatic generation review time
And the filename of the three classes remote signalling equipment in D5000EMS.
Preferably, when lacking filename according to full dose filename lookup, using minus sentences in SQL statement, full dose is used
Filename goes to subtract storage filename, has been put in storage filename than the filename that full dose filename lacks, has as lacked filename.
Preferably, remote signalling device id, newest remote signalling value are maintained in memory.
Preferably, when remote signalling data is handled, first by the digital independent in remote signalling file to memory, then in memory with
The newest remote signalling value of history is compared, and the remote signalling value of change finally is stored in database, while by the remote signalling state in memory more
Newly arrive last state.
Preferably, in storage step of filling a vacancy, the data in remote signalling file is read using JAVA programs and are saved in interim
In table, then filling a vacancy for remote signalling is carried out using storing process.
Preferably, in storage step of filling a vacancy, the data that will be filled a vacancy are filtered, wherein the remote signalling that will will be filled a vacancy
The remote signalling value that value was corresponded to remote signalling table in the period is contrasted, and only retains the remote signalling status discovery in the data that will be filled a vacancy
The data of change.
Preferably, in storage step of filling a vacancy, it will fill a vacancy and be a little divided into five types and perform phase for each type
The processing of filling a vacancy answered, wherein five type is the indeclinable point of filling a vacancy of fill a vacancy point, state of no record, in start position
Point of filling a vacancy, in an intermediate position fill a vacancy point and filling a vacancy a little in tail position.
Brief description of the drawings
With reference to attached drawing, and by reference to following detailed description, it will more easily have more complete understanding to the present invention
And be more easily understood its with the advantages of and feature, wherein：
Fig. 1 schematically shows the remote signalling data pressure according to the preferred embodiment of the invention based on slowly varying dimension form
Contracting storage and the overview flow chart for looking into scarce method of filling a vacancy.
Fig. 2 schematically shows the remote signalling data pressure according to the preferred embodiment of the invention based on slowly varying dimension form
Contracting storage and the idiographic flow example for looking into scarce method of filling a vacancy.
Fig. 3 schematically shows new ancient deed according to the preferred embodiment of the invention and judges flow.
Fig. 4 schematically shows missing documents check process according to the preferred embodiment of the invention.
Fig. 5 schematically shows the flow chart of remote signalling data normal storage according to the preferred embodiment of the invention.
Fig. 6 schematically show remote signalling according to the preferred embodiment of the invention fill a vacancy figure displaying.
It should be noted that attached drawing is used to illustrate the present invention, it is not intended to limit the present invention.Note that represent that the attached drawing of structure can
It can be not necessarily drawn to scale.Also, in attached drawing, same or similar element indicates same or similar label.
Embodiment
In order to make present disclosure more clear and understandable, with reference to specific embodiments and the drawings in the present invention
Appearance is described in detail.
Fig. 1 schematically shows the remote signalling data pressure according to the preferred embodiment of the invention based on slowly varying dimension form
Contracting storage and the overview flow chart for looking into scarce method of filling a vacancy.In a particular embodiment, Fig. 2 schematically shows excellent according to the present invention
The remote signalling data compression based on slowly varying dimension form of embodiment is selected to store and look into the idiographic flow example for lacking method of filling a vacancy.It is distant
Letter data is new file or ancient deed by judging, normal flow is respectively adopted or flow of filling a vacancy, and preserve text in storage
Part name, process is as shown in Figure 1, the work completed below to each several part illustrates.
Specifically, as shown in Figure 1, the remote signalling data pressure according to the preferred embodiment of the invention based on slowly varying dimension form
Contracting, which stores and look into the scarce method of filling a vacancy, to be included：
New and old Documents Comparison step S1：By the filename that marks the remote signalling file of access with having time (for example, disconnected
Road device filename may be Breaker201503120145.txt, wherein " 201503120145 " represent data generation when
Between) compared with deposit file name in database, the remote signalling file for judging access is new file or ancient deed；
For example, whether deterministic process can be as shown in figure 3, first check for table dw_r_min_ind_filename, judges table
For sky；It is determined as new file in the case of for sky；Judge incoming file name input_filename in the case of not for sky
It is present in table of file name dw_r_min_ind_ with the presence or absence of in table of file name dw_r_min_ind_filename, and in judgement
It is determined as ancient deed in the case of in filename, is judging to be not present in table of file name dw_r_min_ind_filename
In the case of take out table dw_r_min_ind_filename latest document name db_last_filename, and input_
Filename is compared, and subsequently determines whether input_filename<Db_last_filename, establishment are then determined as ancient deed,
It is invalid then to determine new file.
Normal storage step S2：The remote signalling file that access is judged in new and old Documents Comparison step S1 is the situation of new file
Under, increase newest state on existing remote signalling data, and the remote signalling data of generating state change is added in database table.
Storage step of filling a vacancy S3：The remote signalling file that access is judged in new and old Documents Comparison step S1 is not the feelings of new file
Under condition, increase the remote signalling state (operation of filling a vacancy) of some past particular point in time in existing history remote signalling data；Due to
Remote signalling data is preserved using slowly varying dimension form, can not be inserted directly into new data in the historical data, and is needed to having delayed
The remote signalling data of slow change dimension form is increased, is deleted, changing operation.
Filename preserves step S4：The problems such as inspection, new and old Documents Comparison are lost to solve remote signalling data, are completed normally
After the storage step S2 or storage step S3 that fills a vacancy, need to by the filename of remote signalling data with common format (each filename a line, it is non-
Slowly varying dimension form) preserve.
<Shortage of data inspection>
Remote signalling data is preserved using slowly varying dimension form, it is difficult to shortage of data feelings are directly judged by remote signalling data
Condition, the present invention realize deletion condition inspection, process is as shown in Figure 4 indirectly by checking remote signalling filename.Checking step is as follows：
Generate full dose filename：Remote signalling filename is made of " implementor name+time+file type ", distant in D5000EMS
Message part shares three classes, breaker Breaker201503120145.txt, disconnecting link
Disconnector201503120145.txt, grounding switch grounddisconnector201503120145.txt, in life
During into full dose file, by all time points in the programming automatic generation review time, the filename of three classes remote signalling equipment, and it is saved into
In full dose table of file name.
Search missing filename：Using minus sentences in SQL statement, gone to subtract storage filename with full dose filename, obtained
Filename has been put in storage than the filename that full dose filename lacks, has as lacked file.
Filename is lacked to preserve：Missing filename is saved in database table, is used easy to inquire about.
<Normal storage>
It was found from introduction above, Switch equipment 70,000 a plurality of remote signalling datas of generation per minute, data need to convert when storing
For slowly varying dimension form, this just needs to handle substantial amounts of data in a short time.Observe the remote signalling number of slowly varying dimension form
According to it can be found that only just preserved when remote signalling state changes, and this partial data is considerably less, then processing
The key problem of remote signalling data is just converted into the data for how finding these changes.The present invention is programmed using JAVA, will be newest
The remote signalling data memory-resident of state, and state change is searched in memory, process flow can be as shown in Figure 5.
First, remote signalling device systems ID, D5000ID mapping relations are read from account table, puts in memory, save as account
The MAP objects of ID, map_id；Then, remote signalling data is read from remote signalling data table dw_r_min_ind tables, it is desirable to invalid_
Date=null (last state), and query result is put to the MAP objects in memory, saving as remote signalling data, map_ind；
Map_id and map_ind is memory-resident, and timing is also needed in actual use and carries out account in map_id and database more
Newly, to ensure that map_id is constantly in last state.
Then, remote signalling data is read from remote signalling file, and D5000ID in remote signalling file and mapping table map_id is carried out
Compare, generate the LIST objects of file remote signalling, list_file_ind.
Hereafter, into cyclic process.
In cyclic process, remote signalling value is taken out one by one from list_file_ind first, with database last state map_
Ind is compared, until list_file_ind takes the last item data, subsequently determines whether remote signalling value is not present in map_
ind.In the case where being judged as affirmative, a record, valid_date=document times, invalid_date=are increased newly
Null, indication=list_file_ind.indication, and write the LIST objects to be inserted into, list_insert_
ind；And the remote signalling data map_ind in modification memory, increase map_ind.indication=list_file_ newly
ind.indication.In the case where being judged as negative, judge whether remote signalling value is different from map_ind, if it is different, then more
The new invalid_date=document times recorded, and the record is write into the LIST objects to be updated, list_
update_ind；A newly-increased record, valid_date=document times, invalid_date=null, indication=
List_file_ind.indication, and write the LIST objects to be inserted into, list_insert_ind；Change in memory
Remote signalling data, map_ind.indication=list_file_ind.indication.If identical, do not make any operation,
Continue the comparison work of next remote signalling data.
After cyclic process, changed remote signalling value list_update_ind, list_insert_ind is write into number
According to storehouse, wherein list_update_ind will update the data corresponding record in storehouse, and list_insert_ind increases record newly.
From Fig. 5 flows it can be found that when remote signalling data is handled, the account ID (account identification code) of remote signalling data and remote signalling
Value is memory-resident, first by the digital independent in remote signalling file to memory, is then compared in memory with history remote signalling value
Compared with, the remote signalling value of change is finally stored in database, while by the remote signalling state renewal in memory to last state.
Since the last state of remote signalling, account are memory-residents, when having new file processing every time, directly obtained from memory
Newest remote signalling state is taken, is not required to obtain data from database again, so as to reduce the interaction time with database；Compare remote signalling
During state change, due to carrying out in memory, speed is very fast；When remote signalling data is put in storage, it is changed need to only to preserve state
Data, data volume is considerably less, preserves speed and also greatly improves.By a series of technical finesse, solves remote signalling number well
According to compression handle.
<Fill a vacancy storage>
It is when data check that remote signalling data, which is filled a vacancy, and the remote signalling data at some time point in discovery past is not put in storage, again
Access the process of the historical data.Fill a vacancy for remote signalling data, it with the characteristics of normal storage again it is different.First, lose a little
File will not be very much, are usually several files, also with regard to tens when most；Secondly, remote signalling data, which is filled a vacancy, to be dealt with very
Trouble, it is necessary to according to lose a little time location, remote signalling value progress different disposal, it is relatively more with interacting for database.
Since the time that remote signalling data is lost a little can not can know that in advance, prior pretreatment that JAVA memories compare will be unable into
OK, the advantage that memory compares also just is lost.From behind the present invention filling a vacancy analysis in it can be found that when remote signalling is filled a vacancy need compared with
More database interactive operations, database will repeatedly be connected by being filled a vacancy using JAVA, and execution efficiency is relatively low.By research, find to deposit
The combination of storage process and database is the closest, reading and writing database efficiency higher.In view of these features, the present invention uses
JAVA programs read the data in remote signalling file and are saved in interim table dw_r_min_ind_lost, then using storing
Cheng Jinhang remote signalling is filled a vacancy, and process can be as shown in " storage of filling a vacancy " in Fig. 2, i.e. file data is write to remote signalling table of filling a vacancy
In, table name dw_r_min_ind_lost, common table structure, storage of then filling a vacancy simultaneously shows and fills a vacancy situation.
During storing process is filled a vacancy, the data to be filled a vacancy are filtered first (by the remote signalling value for time of filling a vacancy with
The remote signalling value corresponded in remote signalling table in the period is contrasted, and only retains the data of remote signalling status discovery change), pass through this mistake
Filter, greatly reduces the data volume to be filled a vacancy, then carries out work of filling a vacancy for the data filtered out again, introduces fill a vacancy below
When detailed protocol.
When the slowly varying dimension of remote signalling data is filled a vacancy, the time location filled a vacancy is extremely important, below according to an appearance of filling a vacancy
Position, introduce respective method of filling a vacancy under 5 kinds of scenes.
Scene 1：Record is not present
When filling a vacancy, no record before the point to be filled a vacancy is found, it is newly-increased point to illustrate the point, need to increase a record newly,
As " shown in Fig. 6 scenes 1 ", the situation of change of table is as shown in table 3-1-1~3-1-3 in database：
Before table 3-1-1 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
Table 3-1-2 fills a vacancy data
| RESOURCEID | T_ID | INDICATION | |
| 1 | 011 | 2015-3-10 16:33 | 0 |
After table 3-1-3 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 011 | 2015-3-10 16:33 | 0 |
Scene 2：State does not change
For the indeclinable point of state, filtered out in the filtering before filling a vacancy, in order to which integrality of filling a vacancy is so right
Corresponding processing procedure is also provided in this kind of situation is filled a vacancy.
For the indeclinable point of state, processing method is without any changes, and such as " 5 scenes 2 " are shown, table in database for figure
Situation of change it is as shown in the table：
Before table 3-2-1 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 1 | 021 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 2 | 021 | 2015-3-10 16:35 | 2015-3-10 16:39 | 1 |
| 3 | 021 | 2015-3-10 16:39 | 0 |
Table 3-2-2 fills a vacancy data
| RESOURCEID | T_ID | INDICATION | |
| 1 | 021 | 2015-3-10 16:34 | 0 |
After table 3-2-3 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 1 | 021 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 2 | 021 | 2015-3-10 16:35 | 2015-3-10 16:39 | 1 |
| 3 | 021 | 2015-3-10 16:39 | 0 |
Scene 3：Fill a vacancy a little in start position
When the starting point of a segment record is a little occurred exactly at for filling a vacancy, then the period with reference to generation, follow-up time are needed
Section integrated treatment, sharing three kinds of situations needs to handle.With reference to " Fig. 6 scenes 3 " and data variation in table, introduction, which is filled a vacancy, a little to exist
Processing method during starting point.
Before table 3-3-1 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 1 | 031 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 2 | 031 | 2015-3-10 16:35 | 2015-3-10 16:39 | 1 |
| 3 | 031 | 2015-3-10 16:39 | 0 | |
| 4 | 032 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 5 | 032 | 2015-3-10 16:35 | 2015-3-10 16:36 | 1 |
| 6 | 032 | 2015-3-10 16:36 | 2015-3-10 16:38 | 0 |
| 7 | 032 | 2015-3-10 16:38 | 2015-3-10 16:39 | 1 |
| 8 | 032 | 2015-3-10 16:39 | 0 | |
| 9 | 033 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 10 | 033 | 2015-3-10 16:35 | 2015-3-10 16:36 | 1 |
| 11 | 033 | 2015-3-10 16:36 | 0 |
Table 3-3-2 fills a vacancy data
| RESOURCEID | T_ID | INDICATION | |
| 1 | 031 | 2015-3-10 16:35 | 0 |
| 2 | 032 | 2015-3-10 16:35 | 0 |
| 3 | 033 | 2015-3-10 16:35 | 0 |
After table 3-3-3 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 1 | 031 | 2015-3-10 16:33 | 2015-3-10 16:36 | 0 |
| 2 | 031 | 2015-3-10 16:36 | 2015-3-10 16:39 | 1 |
| 3 | 031 | 2015-3-10 16:39 | 0 | |
| 4 | 032 | 2015-3-10 16:33 | 2015-3-10 16:38 | 0 |
| 7 | 032 | 2015-3-10 16:38 | 2015-3-10 16:39 | 1 |
| 8 | 032 | 2015-3-10 16:39 | 0 | |
| 9 | 033 | 2015-3-10 16:33 | 0 |
When filling a vacancy a generation when a period of time starts, it need to consider three kinds of situations, be analyzed as follows：
031 (RESOURCEID 031)：A generation fill a vacancy in 35 (2015-3-1016:35, it is abbreviated as 35), 35 this section
Time is 35~39, so 0 state delay 1 minute, i.e., 33~36 be 0,36~39 be 1；
032：Fill a vacancy occur 35,35 this be for a period of time 35~36, after state is changed into 0, front and rear state is three sections of 0
Time is linked to be together, so it is 0 to be changed into 33~38, while deletes unnecessary two and records, 5,6 in table 3-3-3 are deleted；
033：Fill a vacancy occur 35,35 this be for a period of time 35~36, after state is changed into 0, front and rear state is three sections of 0
Time is linked to be together, so it is 0 to be changed into 33~NULL, while deletes unnecessary two and records, 10,11 in table 3-3-3 are deleted
Remove；
In above-mentioned three kinds of situations, 032,033 is similar, and difference is 032 follow-up also record, and 033 follow-up no record.
In actual treatment, 032,033 both types are uniformly processed, that is, change 35 previous bar records (33~35), make it
The INVALID_DATE (the 38 of 032,033 NULL) of the latter bar records of INVALID_DATE=35, and delete 35 records (35
~36) and 35 latter bars record (032 36~38,033 36~NULL).
Integrate, occur for filling a vacancy in start position, if the place section time more than 1 minute, carries out state
Delay, if place section just 1 minute, carry out three sections merging.
Scene 4：Fill a vacancy a little in centre position
Occur for filling a vacancy in the centre position of a segment record, then need to be carried out according to the ending time of this section different
Handle, in this example, illustrate respective processing method with three kinds of situations.With reference to " Fig. 6 scenes 4 " and data variation in table,
Introduce a little processing method when middle of filling a vacancy.
Before table 3-4-1 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 1 | 041 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 2 | 041 | 2015-3-10 16:35 | 2015-3-10 16:39 | 1 |
| 3 | 041 | 2015-3-10 16:39 | 0 | |
| 4 | 042 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 5 | 042 | 2015-3-10 16:35 | 2015-3-10 16:38 | 1 |
| 6 | 042 | 2015-3-10 16:38 | 2015-3-10 16:40 | 0 |
| 7 | 042 | 2015-3-10 16:40 | 1 | |
| 8 | 043 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 9 | 043 | 2015-3-10 16:35 | 2015-3-10 16:38 | 1 |
| 10 | 043 | 2015-3-10 16:38 | 0 |
Table 3-4-2 fills a vacancy data
| RESOURCEID | T_ID | INDICATION | |
| 1 | 041 | 2015-3-10 16:37 | 0 |
| 2 | 042 | 2015-3-10 16:37 | 0 |
| 3 | 043 | 2015-3-10 16:37 | 0 |
After table 3-4-3 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 1 | 041 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 2 | 041 | 2015-3-10 16:35 | 2015-3-10 16:37 | 1 |
| 041 | 2015-3-10 16:37 | 2015-3-10 16:38 | 0 | |
| 041 | 2015-3-10 16:38 | 2015-3-10 16:39 | 1 | |
| 3 | 041 | 2015-3-10 16:39 | 0 | |
| 4 | 042 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 5 | 042 | 2015-3-10 16:35 | 2015-3-10 16:37 | 1 |
| 6 | 042 | 2015-3-10 16:37 | 2015-3-10 16:40 | 0 |
| 7 | 042 | 2015-3-10 16:40 | 1 | |
| 8 | 043 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 9 | 043 | 2015-3-10 16:35 | 2015-3-10 16:37 | 1 |
| 10 | 043 | 2015-3-10 16:37 | 0 |
When filling a vacancy a generation among a period of time, it need to consider three kinds of situations, be analyzed as follows：
041：Fill a vacancy occur 37,37 this be 35~39 for a period of time, so 0 state is equivalent to mutation, i.e., 35
Increase the data that 1 section is 0 in~39, show to increase by 2 datas in tables of data, and change 1 data；
042：Fill a vacancy occur 37,37 this be 35~38 for a period of time, since ending 38 was away from 37 only 1 minute, so
After filling a vacancy, 37 and 38~40 merge；
043：Fill a vacancy occur 37,37 this be 35~38 for a period of time, since ending 38 was away from 37 only 1 minute, so
After filling a vacancy, 37 and 38~NULL merges；
In above-mentioned three kinds of situations, 042,043 is similar, and difference is 042 follow-up also record, and 043 follow-up no record.
In actual treatment, 042,043 is uniformly processed, that is, changes 37 place sections (the 35~38 of 042,043), make its INVALID_DATE
=37, and latter bar record (the 38~40 of 042,043 38~NULL) is changed, make its VALID_DATE=37.
Integrate, for a generation of filling a vacancy in centre position, if filled a vacancy a little with the ending time more than 1 minute, increase
Add the new state of 1 minute, if be equal to 1 minute with ending time phase difference, merge with latter section.
Scene 5：Fill a vacancy a little in tail position
For the end position that occurs in a segment record of filling a vacancy, it is contemplated that remote signalling file only represents 1 minute data, then into
The row mutation of 1 minute.With reference to " data variation, introduces a little processing method in afterbody of filling a vacancy in Fig. 6 scenes 5 " and table.
Before table 3-4-1 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 1 | 051 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 2 | 051 | 2015-3-10 16:35 | 1 |
Table 3-4-2 fills a vacancy data
| RESOURCEID | T_ID | INDICATION | |
| 1 | 051 | 2015-3-10 16:38 | 0 |
After table 3-4-3 fills a vacancy
| RESOURCEID | VALID_DATE | INVALID_DATE | INDICATION | |
| 1 | 051 | 2015-3-10 16:33 | 2015-3-10 16:35 | 0 |
| 2 | 051 | 2015-3-10 16:35 | 2015-3-10 16:38 | 1 |
| 051 | 2015-3-10 16:38 | 2015-3-10 16:39 | 0 | |
| 051 | 2015-3-10 16:39 | 1 |
When a generation of filling a vacancy is ending up a period of time, it is analyzed as follows：
051：Fill a vacancy occur 38,38 this be for a period of time 35~NULL, so 0 state exists equivalent to mutation
Increase the data that 1 section is 0 in 35~NULL, show to increase by 2 datas in tables of data, and change 1 data；
The above-mentioned process of filling a vacancy, which will fill a vacancy, is divided into 5 kinds of scenes, and is provided not according to location inside each scene
Same processing method, covers all situations of filling a vacancy, the problem of filling a vacancy for solving the slowly varying dimension of remote signalling data well.
In conclusion preserving remote signalling data using slowly varying dimension form, technological difficulties are that the quick of data handles and deposit
Storage, shortage of data inspection, data are filled a vacancy, the present invention is comprehensive to be compared using memory, storing process is filled a vacancy, remote signalling filename look into it is scarce
The methods of, solve these technical barriers well.
The present invention has following features：
1) technical barrier is overcome：Slowly varying dimension is due to data format, it is difficult to realizes number by table data itself
Checked according to missing, the present invention is by the preservation of remote signalling filename and compares, and realizes the missing inspection of remote signalling data indirectly；Slowly
For change dimension when filling a vacancy, it is inaccurate that any situation of filling a vacancy of omission can all cause remote signalling data to fill a vacancy, and the present invention is according to the position filled a vacancy a little
Put, propose 5 scenes, and situation of filling a vacancy further is segmented according to different contexts under each scene, and propose specific
Fill a vacancy method, so as to completely solve the problem of filling a vacancy of remote signalling data, and be achieved and apply in software project.
2) execution efficiency is high：In normal fill a vacancy, remote signalling data is with data volume is big, state change is few, time requirement is tight
The characteristics of, the present invention is by last state memory-resident, and completion status compares in memory, is finally saved in the state of change
Database, so as to reduce storage data volume, improves the warehouse-in efficiency of remote signalling data；When data are filled a vacancy, by the process of filling a vacancy
In repeatedly increase, delete, change operation and change and performed by storing process, improve execution efficiency.
3) easily controllable details：Either memory, which is can be seen that, from introduction above more still fills a vacancy algorithm all more
Complexity, using JAVA (normal storage, memory compare) and PLSQL, (storage of filling a vacancy, writes storing process and realizes calculation of filling a vacancy the present invention
Method) bilingual progress technology realization, the control implementation procedure that both language can be fine, realizes that data are correctly put in storage.Such as
Fruit uses commercialization ETL softwares, then is difficult to accomplish this Precise control.
For example, the algorithm that the present invention introduces after the application of a period of time, calculates in January, 2015, commonly storage is gentle
Slow change dimension storage data volume, show that common storage data volume is about 3,124,800,000, slowly varying dimension is 280,000, compression
Than for 11160:1, when preserving for a long time, by bigger, (original state must deposit this ratio, follow-up only to preserve variable condition, data
Measure much less).By using slowly varying dimension, the huge compression to remote signalling data is realized, remote signalling data is solved and protects for a long time
Deposit, quick search etc. uses problem.
It is understood that although the present invention is disclosed as above with preferred embodiment, but above-described embodiment and it is not used to
Limit the present invention.For any those skilled in the art, without departing from the scope of the technical proposal of the invention,
Many possible changes and modifications are all made to technical solution of the present invention using the technology contents of the disclosure above, or are revised as
With the equivalent embodiment of change.Therefore, every content without departing from technical solution of the present invention, the technical spirit pair according to the present invention
Any simple modifications, equivalents, and modifications made for any of the above embodiments, still fall within the scope of technical solution of the present invention protection
It is interior.
Claims (8)
1. a kind of remote signalling data compression based on slowly varying dimension form, which is stored and looked into, lacks method of filling a vacancy, it is characterised in that including：
New and old Documents Comparison step：By having been deposited in the filename and database that are marked with having time by the remote signalling file of access
Filename is compared, and the remote signalling file for judging access is new file or ancient deed；
Normal storage step：In the case that the remote signalling file for judging to access in new and old Documents Comparison step is new file,
Having increases newest state on remote signalling data, and the remote signalling data of generating state change is added in database table；
Storage of filling a vacancy step：In the case that the remote signalling file for judging to access in new and old Documents Comparison step is not new file,
Increase the remote signalling state of past particular point in time in existing history remote signalling data；
Filename preserves step：After completing normal storage step or filling a vacancy storage step, by the filename of remote signalling data with general
Logical form preserves；
The remote signalling data compression stores and looks into the scarce method of filling a vacancy and further includes shortage of data inspection processing, the shortage of data inspection
Processing includes：
The full dose filename of remote signalling file is generated, the wherein form of full dose filename is " implementor name+time+file type " group
Into, and be saved into full dose table of file name；
Filename is lacked according to full dose filename lookup；
Carry out missing filename to preserve, wherein missing filename is saved in database table.
2. according to the method described in claim 1, it is characterized in that, wherein generate remote signalling file full dose file when, by journey
Sequence automatically generates the filename of all time points and the three classes remote signalling equipment in D5000EMS in the review time.
3. according to the method described in claim 1, it is characterized in that, when lacking filename according to full dose filename lookup, make
With minus sentences in SQL statement, gone to subtract storage filename with full dose filename, be put in storage filename than full dose filename
The filename lacked, as lacks filename.
4. according to the method described in claim 1, it is characterized in that, remote signalling data is maintained in memory.
5. according to the method described in claim 4, it is characterized in that, when remote signalling data is handled, first by the number in remote signalling file
According to memory is read, then in memory compared with the newest remote signalling value of history, the remote signalling value of change is finally stored in data
Storehouse, while the remote signalling state in memory is updated to last state.
6. according to the method described in claim 1, it is characterized in that, in storage step of filling a vacancy, read using JAVA programs distant
Data in message part are simultaneously saved in interim table, then carry out filling a vacancy for remote signalling using storing process.
7. according to the method described in claim 1, it is characterized in that, in storage step of filling a vacancy, to the data that will fill a vacancy into
Row filtering, is contrasted wherein the remote signalling value that will be filled a vacancy is corresponded to the remote signalling value in the period with remote signalling table, and only retaining will
The data of remote signalling status discovery change in the data to be filled a vacancy.
8. according to the method described in claim 1, it is characterized in that, in storage step of filling a vacancy, it will fill a vacancy and be a little divided into five kinds
Type and corresponding processing of filling a vacancy is performed for each type, wherein fill a vacancy point, state of five type for no record
Indeclinable point of filling a vacancy, the point of filling a vacancy in start position, in an intermediate position fill a vacancy point and the benefit in tail position
Shortcoming.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510268933.3A CN104834487B (en) | 2015-05-24 | 2015-05-24 | Remote signalling data compression based on slowly varying dimension form, which is stored and looked into, lacks method of filling a vacancy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510268933.3A CN104834487B (en) | 2015-05-24 | 2015-05-24 | Remote signalling data compression based on slowly varying dimension form, which is stored and looked into, lacks method of filling a vacancy |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104834487A CN104834487A (en) | 2015-08-12 |
| CN104834487B true CN104834487B (en) | 2018-05-04 |
Family
ID=53812403
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510268933.3A Active CN104834487B (en) | 2015-05-24 | 2015-05-24 | Remote signalling data compression based on slowly varying dimension form, which is stored and looked into, lacks method of filling a vacancy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104834487B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457401B (en) * | 2019-07-08 | 2022-11-08 | 南京苏宁软件技术有限公司 | Data storage method and device, computer equipment and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101149829A (en) * | 2006-09-22 | 2008-03-26 | 阿里巴巴公司 | Client data centralized processing method and system |
-
2015
- 2015-05-24 CN CN201510268933.3A patent/CN104834487B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101149829A (en) * | 2006-09-22 | 2008-03-26 | 阿里巴巴公司 | Client data centralized processing method and system |
Non-Patent Citations (2)
| Title |
|---|
| 基于CIM模型的电网企业统一信息共享平台研究与设计;张韧;《中国优秀硕士学位论文全文数据库 信息科技辑》;20120415;第2012卷(第4期);第4.2.2，4.3.1-4.3.2节，附图4-25 * |
| 基于银行数据仓库系统关键技术的研究与实现;吴莹;《中国优秀硕士学位论文全文数据库 信息科技辑》;20101215;第2010卷(第12期);第5.4.1-5.4.2节，附图5.5 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104834487A (en) | 2015-08-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101504664B (en) | Apparatus and method for extracting, converting and loading total source data | |
| CN104462141B (en) | Method, system and the storage engines device of a kind of data storage and inquiry | |
| CN1992632B (en) | Communication network warning method and warning system | |
| CN102426525B (en) | Panoramic modeling method of multi-application system | |
| CN105975502A (en) | Method for realizing incremental data extract based on CDC (Change Data Capture) mode | |
| CN104317928A (en) | Service ETL (extraction-transformation-loading) method and service ETL system both based on distributed database | |
| CN109471905A (en) | A kind of block chain index method for supporting time range and range of attributes compound query | |
| CN106453482A (en) | Internet of things middleware system and Internet of things system | |
| CN110019157A (en) | Safety in production business datum analysis method based on Internet of Things | |
| CN107463706A (en) | A kind of storage of magnanimity recorder data and parsing method and system based on Hadoop | |
| CN107798062A (en) | A kind of transformer station's historical data unifies storage method and system | |
| CN104504030A (en) | Indexing method for power dispatching automation mass messages | |
| CN106155838A (en) | A kind of database back-up data restoration methods and device | |
| CN100354861C (en) | Data compresser,data decompresser and data managing system | |
| CN105787090A (en) | Index building method and system of OLAP system of electric data | |
| CN104834487B (en) | Remote signalling data compression based on slowly varying dimension form, which is stored and looked into, lacks method of filling a vacancy | |
| CN111488325A (en) | Meteorological big data aggregation method based on Hadoop architecture | |
| CN108334582B (en) | Method for compressing historical data in real-time database | |
| CN105159820A (en) | Transmission method and device of system log data | |
| CN104601374A (en) | Network fault processing method and device for numerical control machine tool | |
| CN107707346A (en) | A kind of key storage based on database, acquisition methods | |
| CN106933783A (en) | A kind of method and device on the intelligent extraction date from text | |
| CN111221967A (en) | Language data classification storage system based on block chain architecture | |
| CN109886318A (en) | A kind of information processing method, device and computer readable storage medium | |
| CN114896235A (en) | Data management method under big data environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |