CN105045530A - Data recording and storing method - Google Patents

Data recording and storing method Download PDF

Info

Publication number
CN105045530A
CN105045530A CN201510370991.7A CN201510370991A CN105045530A CN 105045530 A CN105045530 A CN 105045530A CN 201510370991 A CN201510370991 A CN 201510370991A CN 105045530 A CN105045530 A CN 105045530A
Authority
CN
China
Prior art keywords
packet
data
stored
coding
record
Prior art date
Application number
CN201510370991.7A
Other languages
Chinese (zh)
Other versions
CN105045530B (en
Inventor
邓志峰
单晟
王雨
Original Assignee
株洲南车时代电气股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株洲南车时代电气股份有限公司 filed Critical 株洲南车时代电气股份有限公司
Priority to CN201510370991.7A priority Critical patent/CN105045530B/en
Publication of CN105045530A publication Critical patent/CN105045530A/en
Application granted granted Critical
Publication of CN105045530B publication Critical patent/CN105045530B/en

Links

Abstract

The invention discloses a data recording and storing method, which comprises the steps of: 1, when a data packet to be stored is received, assigning continuous codes to all data contents representing different meanings in the data packet to be stored; 2, judging whether the data packet to be stored is a first data packet that is received, if yes, sequentially performing recording and storage on the data contents in the data packet to be stored and the corresponding codes, and if no, turning to execute the step 3; and 3, sequentially comparing the data contents of the data packet to be stored with data contents of a previously received data packet according to the corresponding codes, and performing recording and storage according to comparing results, wherein when each data content is compared, if the comparing result is in a preset range, the recording and storage are not performed, or the corresponding data contents and codes are recorded and stored. According to the method disclosed by the invention, recording and storage of the same data contents can be reduced; and the method has the advantages of simplicity for implementing storage, small occupied storage space, high storage efficiency and easiness for implementing data reduction.

Description

A kind of data recording and storing method

Technical field

The present invention relates to real-time data acquisition technical field of memory, particularly relate to a kind of data recording and storing method being applicable to real-time data memory.

Background technology

At present in real-time data acquisition stores, the record of packet is stored and normally stores completely by raw content order, as packet 1, packet 2, packet 3 ought be received successively ... then finally be stored as packet 1, packet 2, packet 3 successively ..., and each packet is raw data packets completely.In process of real-time data acquisition, often there is a large amount of duplicate contents between packet, between packet, there is larger correlativity, then can cause the record of a large amount of same data content according to above-mentioned record storage mode, thus consume too much storage space, cause the waste of storage space.

The at present existing practitioner scheme that proposes to adopt repeating data to judge to remove repeating data, solve the problem that identical data repeats to record, but judgement and the storing process of such scheme one side repeating data are comparatively complicated, add the complexity that data store, affect storage efficiency and be not easy to realize data convert; On the other hand, such scheme normally based on data itself repeat judge, identical data may represented by implication completely different, and in real-time data acquisition, the required important information content gathered of the implication correspondence of data, thus such scheme the collection not being suitable for real time data store.

Summary of the invention

The technical problem to be solved in the present invention is just: the technical matters existed for prior art, the invention provides a kind of data recording and storing method being applicable to real-time data acquisition and storing, the record that can reduce identical content data stores, and store realize simple, take that storage space is few, storage efficiency is high and easily realize data convert.

For solving the problems of the technologies described above, the technical scheme that the present invention proposes is:

A kind of data recording and storing method, step comprises:

1) when receiving packet to be stored, for representing in packet to be stored that every data content of different implication distributes continuous print coding;

2) judge whether packet to be stored is the first packet received, if so, data content every in packet to be stored and corresponding coding are carried out record successively and stores; If not, proceed to execution step 3);

3) packet to be stored is compared according to corresponding coding successively with every data content in the upper packet received, and carry out record storage according to comparative result; When wherein each compares, if comparative result is in preset range, does not then carry out record and store; Otherwise record stores corresponding data content and coding.

As a further improvement on the present invention, be in packet to be stored, represent that the concrete steps that every data content distribution continuous print of different implication is encoded are in described step 1):

1.1) packet to be stored is divided into multinomial data content according to represented different implications;

1.2) carry out continuous programming code according to the regular coding form preset, and correspondence distributes to every data content of packet to be stored.

As a further improvement on the present invention, described step 2) in every data content with coding corresponding carry out recording the concrete steps stored be:

2.1) sequence number that also stored record starts mark, corresponding first record bag is set;

2.2) obtain the Section 1 data content of first coding and packet to be stored, store described first coding, Section 1 data content successively;

2.3) obtain the next item down data content of next coding and packet to be stored, store the coding and data content that acquire successively, return and perform step 2.3), until the record completing all data contents in packet to be stored stores;

2.4) also stored record is set and terminates mark.

As a further improvement on the present invention, the concrete steps of described step 3) are:

3.1) also stored record is set and starts the sequence number of mark, corresponding current record bag;

3.2) get the data content corresponding to first coding in packet to be stored, a upper packet received respectively, and calculate the difference of two data contents taken out;

3.3) if the difference of described two data contents is greater than pre-set threshold value, then the data content of respective items in present encoding and packet to be stored is stored successively;

3.4) get the data content corresponding to next coding in packet to be stored, a upper packet received respectively, and calculate the difference of two data contents taken out, return and perform step 3.3); Until complete the comparison of all item data contents, the record completing packet to be stored stores;

3.5) also stored record is set and terminates mark.

As a further improvement on the present invention, the described preset range of adjustment is also comprised in described step 3) to adjust the step of the data content of required record storage.

As a further improvement on the present invention, also comprise data convert step 4), concrete steps are:

4.1) the record memory location of locating first packet is reduced, and obtains every data content of first data packet recording address book stored successively, and reduction obtains first packet content;

4.2) the record memory location of locating next packet is reduced, and every data content of record storage is obtained successively according to coding, if wherein there is discontinuous coding, then from a upper packet, obtain the data content corresponding to Unrecorded coding; Return and perform step 4.2), until complete the reduction of all packets.

As a further improvement on the present invention, described step 4.2) in the record memory location concrete steps of carrying out reducing of the next packet in location be:

4.21) locate the position of next packet and reduce as current data packet, encode and determine whether the first coding in described regular coding form for first that obtains the storage of current data packet record, if so, obtaining corresponding data content; Otherwise judge each the non-record coding between first coding of acquisition and described first coding, and obtain described data content corresponding to each non-record coding successively from every content of a upper packet;

4.22) obtain the next one coding that record stores, and encode whether continuously according to described regular coding form judgement and upper one, if so, obtain corresponding data content; Otherwise judge and upper one encode between each non-record coding, and data content corresponding to each non-record coding described in obtaining successively from every content of a upper packet; Return and perform step 4.22), until complete the reduction of all data contents, reduction obtains current data packet content.

Compared with prior art, the invention has the advantages that:

1) the present invention is when packet stores, by judging that the similarity system design of every data content in packet and a upper packet determines whether to store, if similarity within the specific limits, does not store, and only store the data content producing compared with a upper packet and necessarily change, thus effectively can reduce the storage of repeating data, the storage space taken needed for minimizing; Simultaneously by the identical data process between adjacent data bag, the change procedure between packet effectively can be recorded;

2) the present invention by packet to be stored according to the similarity between data content, determine not carry out recording the data stored, effectively can reduce the identical repeating data record of implication to store, ensure the integrality that data content stores simultaneously, can be applicable in real-time data acquisition storage;

3) first the present invention encodes to different pieces of information content when receiving packet to be stored, by data content every in different coding identification data packet, the similarity of encoding and judging between the packet of front and back can be combined thus easily, to determine the data needing to store;

4) the present invention combines coding from the data that record stores, and can determine the data division do not stored easily, can find this data division easily simultaneously, realize data convert fast and accurately from a upper packet restored;

5) the present invention judges to adjust the pre-set threshold value that required record stores the data of required record, thus can adjust the change size of record data according to the actual requirements further by adjustment.

Accompanying drawing explanation

Fig. 1 is the realization flow schematic diagram of the present embodiment data recording and storing method.

Fig. 2 is the principle schematic of data packet coding mode in the present embodiment.

Fig. 3 is the record storage principle schematic diagram of first packet in the present embodiment.

Fig. 4 is the record storage principle schematic diagram of non-first packet in the specific embodiment of the invention.

Embodiment

Below in conjunction with Figure of description and concrete preferred embodiment, the invention will be further described, but protection domain not thereby limiting the invention.

As shown in Figure 1, the present embodiment data recording and storing method, step comprises:

1) when receiving packet to be stored, for representing in packet to be stored that every data content of different implication distributes continuous print coding;

2) judge whether packet to be stored is the first packet received, if so, data content every in packet to be stored and corresponding coding are carried out record successively and stores; If not, proceed to execution step 3);

3) packet to be stored is compared according to corresponding coding successively with every data content in the upper packet received, and carry out record storage according to comparative result; When wherein each compares, if comparative result is in preset range, does not then carry out record and store; Otherwise record stores corresponding data content and coding.

The present embodiment is when packet stores, by judging that the similarity system design of every data content and a upper packet in packet determines whether to store, if similarity within the specific limits, does not store, and only store the content producing compared with a upper packet and necessarily change, thus effectively can reduce the storage of repeating data, the storage space taken needed for minimizing; Simultaneously by the identical data process between adjacent data bag, the change procedure between packet effectively can be recorded.

In the present embodiment, for representing in packet to be stored that the concrete steps of every data content distribution continuous print coding of different implication are in step 1):

1.1) packet to be stored is divided into multinomial data content according to represented different implications;

1.2) carry out continuous programming code according to the regular coding form preset, and correspondence distributes to every data content of packet to be stored.

Packet to be stored divides according to the implication corresponding to data content by the present embodiment, as speed, displacement etc., determine whether as identical data according to the similarity between same data content, the record that can reduce the identical repeating data of implication stores, ensure the integrality that data content stores simultaneously, thus can effectively be applicable in real-time data acquisition storage.

In the present embodiment, for the packet each to be stored received according to Unified coding form all first allocated code, with different data content every in flag data bag, identical content adopts identical coding.As shown in Figure 2, the present embodiment is encoded according to the digital encoding scheme increased progressively, when receiving packet to be stored, in packet, every data content correspondence distributes a numerical coding, thus data content corresponding in packet can be determined by coding.The present embodiment is by encoding to different pieces of information content, by data content every in different coding identification data packet, the similarity of encoding and judging between the packet of front and back can be combined thus easily, to determine the data needing to store, simultaneously by the corresponding relation of encoding and between data content, also the data division do not stored can be determined easily, so that realize data convert.

In the present embodiment, when receiving first packet, storing all data contents, every storage content being stored according to the mode corresponding with coding simultaneously.Step 2) in first packet when storing, every data content with encode corresponding carry out recording the concrete steps stored be:

2.1) sequence number that also stored record starts mark, corresponding first record bag is set;

2.2) obtain the Section 1 data content of first coding and packet to be stored, store described first coding, Section 1 data content successively;

2.3) obtain the next item down data content of next coding and packet to be stored, store the coding and data content that acquire successively, return and perform step 2.3), until the record completing all data contents in packet to be stored stores;

2.4) also stored record is set and terminates mark.

In the present embodiment, each packet to be stored received is according to the sequence number of reception order setting correspondence, the corresponding sequence number 2 of sequence number as corresponding in first packet 1, second packet, by that analogy.The ordinal relation between packet can be determined by the sequence number of correspondence, then can carry out order restoring according to sequence number when data convert.Shown in Figure 3, obtain the record format 1 of first packet according to the method described above, record stores content and is followed successively by: record starts mark, sequence number 1, coding 1, content (data content) 1 ... end of record (EOR) identifies.By record, mark, end of record (EOR) identify and determine that first packet institute records the content of storage, and every data content coding that all correspondence is respective, can locate the reduction of every data content and every data content easily.

In the present embodiment, when receiving non-first packet, then the packet received is caught up with a packet and carry out similarity-rough set, to determine the data content that will store.The concrete steps of the present embodiment step 3) are:

3.1) also stored record is set and starts the sequence number of mark, corresponding current record bag;

3.2) get the data content corresponding to first coding in packet to be stored, a upper packet received respectively, and calculate the difference of two data contents taken out;

3.3) if the difference of two data contents is greater than pre-set threshold value, then the data content of respective items in present encoding and packet to be stored is stored successively;

3.4) get the data content corresponding to next coding in packet to be stored, a upper packet received respectively, and calculate the difference of two data contents taken out, return and perform step 3.3); Until complete the comparison of all item data contents, the record completing packet to be stored stores;

3.5) also stored record is set and terminates mark.

In the present embodiment, also comprise adjustment preset range in step 3) to adjust the step of the data content of required record storage.The present embodiment step 3.3) in adjust the delta data of required record particular by adjustment pre-set threshold value, when pre-set threshold value is 0, then records all item number certificates of each packet, all changes between adjacent data bag can be recorded.

As shown in Figure 4, when the present embodiment second packet (bag 2) compares with every data content in first packet (wrapping 1), wherein solid line represents that comparing difference is greater than pre-set threshold value, namely change is greater than pre-set threshold value, dotted line represents that comparing difference is less than threshold values, namely change is less than pre-set threshold value, data content 2 as can be seen from Figure, data content 5, and the change of data content 6 is greater than preset difference value, when then record stores, by data content 2, data content 5, the coding of data content 6 and correspondence stores successively, obtain the record format 2 shown in left hand view.

In the present embodiment, also comprise data convert step 4), concrete steps are:

4.1) the record memory location of locating first packet is reduced, and obtains every data content of first data packet recording address book stored successively, and reduction obtains first packet content;

4.2) the record memory location of locating next packet is reduced, and every data content of record storage is obtained successively according to coding, if wherein there is discontinuous coding, then from a upper packet, obtain the data content corresponding to Unrecorded coding; Return and perform step 4.2), until complete the reduction of all packets.

During the present embodiment data convert, the first packet of first reduction record storage, during first packet reduction, is then determined to restore all item data contents according to the record format 1 of first packet behind memory location by sequence number; Restore second packet (bag 2), the data content changing in former packet and be greater than pre-set threshold value can be gone out by direct-reduction according to record format 2, i.e. data content 2, data content 5, data content 6, all the other Unrecorded data contents then obtain from first packet according to coding, the content of second packet can be restored completely, the reduction process of follow-up data bag by that analogy, can be reduced next packet by the upper packet restored.The data division do not stored can be determined in conjunction with coding easily from the data that record stores, this data division can be found easily from a upper packet restored simultaneously, realize data convert fast and accurately.

In the present embodiment, step 4.2) in the record memory location of the next packet in the location concrete steps of carrying out reducing be:

4.21) locate the position of next packet and reduce as current data packet, encode and determine whether the first coding in regular coding form for first that obtains next data packet recording address book stored, if so, obtaining corresponding data content; Otherwise judge each the non-record coding between first coding of acquisition and first coding, and from every content of a upper packet, obtain the data content corresponding to each non-record coding successively;

4.22) obtain the next one coding that record stores, and encode whether continuously according to described regular coding form judgement and upper one, if so, obtain corresponding data content; Otherwise judge and upper one encode between each non-record coding, and from every content of a upper packet, obtain the data content corresponding to each non-record coding successively; Return and perform step 4.22), until complete the reduction of all data contents, reduction obtains current data packet content.

The present invention can be applicable in real-time data acquisition storage, also can be applied in the data storage of other types according to the actual requirements, stores to reduce the identical repeating data of implication.

Above-mentioned just preferred embodiment of the present invention, not does any pro forma restriction to the present invention.Although the present invention discloses as above with preferred embodiment, but and be not used to limit the present invention.Therefore, every content not departing from technical solution of the present invention, according to the technology of the present invention essence to any simple modification made for any of the above embodiments, equivalent variations and modification, all should drop in the scope of technical solution of the present invention protection.

Claims (7)

1. a data recording and storing method, is characterized in that, step comprises:
1) when receiving packet to be stored, for representing in packet to be stored that every data content of different implication distributes continuous print coding;
2) judge whether packet to be stored is the first packet received, if so, data content every in packet to be stored and corresponding coding are carried out record successively and stores; If not, proceed to execution step 3);
3) packet to be stored is compared according to corresponding coding successively with every data content in the upper packet received, and carry out record storage according to comparative result; When wherein each compares, if comparative result is in preset range, does not then carry out record and store; Otherwise record stores corresponding data content and coding.
2. data recording and storing method according to claim 1, is characterized in that, for representing in packet to be stored that the concrete steps of every data content distribution continuous print coding of different implication are in described step 1):
1.1) packet to be stored is divided into multinomial data content according to represented different implications;
1.2) carry out continuous programming code according to the regular coding form preset, and correspondence distributes to every data content of packet to be stored.
3. data recording and storing method according to claim 2, is characterized in that, described step 2) in every data content with coding corresponding carry out recording the concrete steps stored be:
2.1) sequence number that also stored record starts mark, corresponding first record bag is set;
2.2) obtain the Section 1 data content of first coding and packet to be stored, store described first coding, Section 1 data content successively;
2.3) obtain the next item down data content of next coding and packet to be stored, store the coding and data content that acquire successively, return and perform step 2.3), until the record completing all data contents in packet to be stored stores;
2.4) also stored record is set and terminates mark.
4. data recording and storing method according to claim 3, is characterized in that, the concrete steps of described step 3) are:
3.1) also stored record is set and starts the sequence number of mark, corresponding current record bag;
3.2) get the data content corresponding to first coding in packet to be stored, a upper packet received respectively, and calculate the difference of two data contents taken out;
3.3) if the difference of described two data contents is greater than pre-set threshold value, then the data content of respective items in present encoding and packet to be stored is stored successively;
3.4) get the data content corresponding to next coding in packet to be stored, a upper packet received respectively, and calculate the difference of two data contents taken out, return and perform step 3.3); Until complete the comparison of all item data contents, the record completing packet to be stored stores;
3.5) also stored record is set and terminates mark.
5. according to the data recording and storing method in Claims 1 to 4 described in any one, it is characterized in that, also comprise the described preset range of adjustment in described step 3) to adjust the step of the data content of required record storage.
6. according to the data recording and storing method in Claims 1 to 4 described in any one, it is characterized in that, also comprise data convert step 4), concrete steps are:
4.1) the record memory location of locating first packet is reduced, and obtains every data content of first data packet recording address book stored successively, and reduction obtains first packet content;
4.2) the record memory location of locating next packet is reduced, and every data content of record storage is obtained successively according to coding, if wherein there is discontinuous coding, then from a upper packet, obtain the data content corresponding to Unrecorded coding; Return and perform step 4.2), until complete the reduction of all packets.
7. data recording and storing method according to claim 6, is characterized in that, described step 4.2) in the record memory location concrete steps of carrying out reducing of the next packet in location be:
4.21) locate the position of next packet and reduce as current data packet, encode and determine whether the first coding in described regular coding form for first that obtains the storage of current data packet record, if so, obtaining corresponding data content; Otherwise judge each the non-record coding between first coding of acquisition and described first coding, and obtain described data content corresponding to each non-record coding successively from every content of a upper packet;
4.22) obtain the next one coding that record stores, and encode whether continuously according to described regular coding form judgement and upper one, if so, obtain corresponding data content; Otherwise judge and upper one encode between each non-record coding, and data content corresponding to each non-record coding described in obtaining successively from every content of a upper packet; Return and perform step 4.22), until complete the reduction of all data contents, reduction obtains current data packet content.
CN201510370991.7A 2015-06-30 2015-06-30 A kind of data recording and storing method CN105045530B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510370991.7A CN105045530B (en) 2015-06-30 2015-06-30 A kind of data recording and storing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510370991.7A CN105045530B (en) 2015-06-30 2015-06-30 A kind of data recording and storing method

Publications (2)

Publication Number Publication Date
CN105045530A true CN105045530A (en) 2015-11-11
CN105045530B CN105045530B (en) 2018-02-16

Family

ID=54452101

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510370991.7A CN105045530B (en) 2015-06-30 2015-06-30 A kind of data recording and storing method

Country Status (1)

Country Link
CN (1) CN105045530B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009080546A (en) * 2007-09-25 2009-04-16 Brother Ind Ltd Content distribution and storage system, duplicate data storage number tabulation method, node device, and node processing program
CN101706825A (en) * 2009-12-10 2010-05-12 华中科技大学 Replicated data deleting method based on file content types
CN102810108A (en) * 2011-06-02 2012-12-05 英业达股份有限公司 Method for processing repeated data
CN103617260A (en) * 2013-11-29 2014-03-05 华为技术有限公司 Index generation method and device for repeated data deletion

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009080546A (en) * 2007-09-25 2009-04-16 Brother Ind Ltd Content distribution and storage system, duplicate data storage number tabulation method, node device, and node processing program
CN101706825A (en) * 2009-12-10 2010-05-12 华中科技大学 Replicated data deleting method based on file content types
CN102810108A (en) * 2011-06-02 2012-12-05 英业达股份有限公司 Method for processing repeated data
CN103617260A (en) * 2013-11-29 2014-03-05 华为技术有限公司 Index generation method and device for repeated data deletion

Also Published As

Publication number Publication date
CN105045530B (en) 2018-02-16

Similar Documents

Publication Publication Date Title
CN104871249B (en) The adaptive error correcting code of data-storage system
US9851917B2 (en) Method for de-duplicating data and apparatus therefor
CN104166606B (en) File backup method and main storage device
US8943384B2 (en) Using a soft decoder with hard data
CN103336785B (en) A kind of distributed storage method based on network code and device thereof
CN102419766B (en) Data redundancy and file operation methods based on Hadoop distributed file system (HDFS)
CN102332029B (en) Hadoop-based mass classifiable small file association storage method
CN101855636B (en) Data processing equipment and data processing method
CN102694625A (en) Polarization code decoding method for cyclic redundancy check assistance
CN104380267B (en) Data compression/decompression device
CN101855620B (en) Data processing apparatus and method of processing data
WO2014131148A1 (en) Method for encoding minimal storage regenerating codes and repairing storage nodes
US20120185612A1 (en) Apparatus and method of delta compression
CN102244518B (en) The hard-wired system and method for parallel decompression
EP3361409B1 (en) Data deduplication method and apparatus
US8375182B2 (en) System and method for segmenting a data stream
US10082982B2 (en) Data backup method and apparatus, data restoration method and apparatus, and server
CN103473277B (en) The Snapshot Method and device of file system
WO2018161760A8 (en) Blockchain data storage system and method
CN106681934A (en) Method and system for rubbish recycling of solid state disk, and solid state disk controller
JP5955378B2 (en) Encoding method and decoding method
US8265407B2 (en) Method for coding and decoding 3D data implemented as a mesh model
US8362931B2 (en) Compression and decompression of mass spectrometry data
CN103019884B (en) Memory page de-weight method and memory page de-weight device based on virtual machine snapshot
US8892809B2 (en) Data compression and encoding in a memory system

Legal Events

Date Code Title Description
PB01 Publication
C06 Publication
SE01 Entry into force of request for substantive examination
C10 Entry into substantive examination
GR01 Patent grant
GR01 Patent grant