CN108551490A - A kind of industry flow data coding/decoding system and method - Google Patents

A kind of industry flow data coding/decoding system and method Download PDF

Info

Publication number
CN108551490A
CN108551490A CN201810457697.3A CN201810457697A CN108551490A CN 108551490 A CN108551490 A CN 108551490A CN 201810457697 A CN201810457697 A CN 201810457697A CN 108551490 A CN108551490 A CN 108551490A
Authority
CN
China
Prior art keywords
data
cache
grades
grade
server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201810457697.3A
Other languages
Chinese (zh)
Other versions
CN108551490B (en
Inventor
张宁
黄璜
刘润虎
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xijing University
Original Assignee
Xijing University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xijing University filed Critical Xijing University
Priority to CN201810457697.3A priority Critical patent/CN108551490B/en
Publication of CN108551490A publication Critical patent/CN108551490A/en
Application granted granted Critical
Publication of CN108551490B publication Critical patent/CN108551490B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/568Storing data temporarily at an intermediate stage, e.g. caching
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/565Conversion or adaptation of application format or content
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/568Storing data temporarily at an intermediate stage, e.g. caching
    • H04L67/5682Policies or rules for updating, deleting or replacing the stored data
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/16Threshold monitoring

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Memory System Of A Hierarchy Structure (AREA)

Abstract

It is positive integer the invention discloses a kind of industrial flow data coding/decoding system and method, a bottom data modules, a, a >=1 is used for the bottom data of collecting device, and bottom data is sent to b grades of cache modules;B grades of cache modules, b=1 ... L, L >=2, wherein, 1 grade of cache module is used to receive the bottom data of a bottom data modules transmission, higher level's cache module is respectively used to receive the data that adjacent subordinate's cache module is sent, these data are stored in the storage space of corresponding b grades of cache module, and the data that L grades of cache modules receive are packaged as data packet and are sent to server;Data channel module is used to bottom data being transferred to b grades of cache modules;And server, it is used to receive the data packet of L grades of cache modules transmission, and be decoded.The system and method for the present invention can carry out the acquisition of mass data in limited spatial cache, it is ensured that acquisition or the data transmitted it is complete and correct.

Description

A kind of industry flow data coding/decoding system and method
Technical field
The invention belongs to industrial big data technical fields, and in particular to a kind of industry flow data coding/decoding system and side Method.
Background technology
Intelligence manufacture based on the technologies such as cloud computing, big data, artificial intelligence has become current or even futurity industry The important directions of boundary's development.In intelligence manufacture, industrial big data technology is key technology.It is efficient to collect magnanimity bottom number According to structure industrial data model is the basic ideas of industrial big data technology development further according to model-aided business decision.
And in industrial big data technology implementation process, most critical is sought to having generation frequency height, single data Measure it is small, there is the bottom industrial data of the features such as strict sequential order, total amount of data is big to be acquired and arrange.
Currently, the method and few for acquiring and arranging about bottom industry big data in the world, and it is most of among these Method is for a certain particular device, and the method with certain universality proposes not yet.
Invention content
The object of the present invention is to provide a kind of industrial flow data coding/decoding systems and method, the system and method to solve The problem of bottom industry big data is acquired and arranged, can be acquired coding and decoding to industrial big data, convenient for big number According to arrangement, have universality.
In order to achieve the above object, the present invention provides a kind of industrial flow data coding/decoding system, which includes:The A bottom data modules, a are positive integer, and a >=1 is used for the bottom data of collecting device, and the bottom data is sent To b grades of cache modules;B grades of cache modules, b=1 ... L, L >=2, wherein 1 grade of cache module is used to receive the bottoms a The bottom data that layer data module is sent, higher level's cache module are respectively used to receive the number that adjacent subordinate's cache module is sent According to these data are stored in the storage space of corresponding b grades of cache module, and the data that L grades of cache modules receive are packaged as Data packet is sent to server;Data channel module is used to for the bottom data to be transferred to the b grades caching mould Block;And server, it is used to receive the data packet that the L grade cache modules are sent, and be decoded.
Wherein, b grades cache module includes:More than two buffer units, according to writing for the bottom data Enter situation, stores the bottom data between the buffer unit in a manner of cycle alternation.
Preferably, the bottom data includes:Appointing in sensing data, current data, director data and program data Meaning is a kind of.
Preferably, in 1 grade of cache module, the bottom data is converted into basic data, is stored in institute In the storage space for the 1 grade of cache module stated.
The basic data is sequentially stored according to acquisition time in the storage space of the b grade cache modules;Institute The storage space for the b grade cache modules stated for its all buffer unit for being included the sum of storage space.
The storage space of each buffer unit in the b grade cache modules is equal, and equal to the server The threshold value of the integral multiple of file storage minimum unit, each buffer unit is less than its respective storage space.
Preferably, when the shortest time of b grades cache module write-in is more than or equal to the program execution of b grades of cache modules Between with b grades of cache modules transmit the sum of maximum duration to b+1 grades of cache modules.
The bottom data is passed to rate to 1 grade of cache module and should be transmitted to b+1 grades of cache modules with b grades of cache modules Rate matches.
The present invention also provides a kind of industrial flow data coding-decoding methods, and this method is using industrial flow data as mentioned Coding/decoding system, this method include:
Step 1:Bottom data is established with b grades of cache modules by corresponding data channel and is connected, the data channel It is responsible for acquiring in bottom data sample delivery to 1 grade of cache module according to program setting time point;
Step 2:The data channel sampling is programmed:For the same equipment, the data channel is to each Bottom data sampling clock is consistent, and is ensured under arbitrary sampling time point to all bottom datas while being acquired;
Step 3:1 grade of cache module receives the bottom data of the data channel acquisition, and by the bottom number According to basic data write-in caching is converted to, each basic data distributes corresponding spatial cache;
Step 4:When sampling starts, the data channel acquires the same equipment according to the program setting moment and corresponds to successively All bottom datas, and every bottom data is converted into basic data, is written to the caching list of 1 grade of cache module Whether the corresponding spatial cache of items basic data, continue to write to according to the threshold decision data of spatial cache in member, if write-in More than threshold value, then stop being written, basic data is packaged and is sent to the b+1 grade cache modules, which is carried out It empties, line position of going forward side by side sets switching, is continued to write to switching to another buffer unit, which enters standby mode;It repeats Step 4 is until sampling terminates;
Step 5:The b+1 grade cache modules receive and are written the basic data of the b grade cache modules transmission, write In the buffer unit for entering the b+1 grade cache modules, whether continued to write to according to the threshold decision data of spatial cache, if writing Enter to be more than threshold value, then stop being written, the data of transmission are packaged and are sent to the b+2 grade cache modules or server, to this Buffer unit is emptied, and line position of going forward side by side sets switching, is continued to write to switching to another buffer unit, which enters Wait for pattern;Step 5 is repeated until L grades of cache modules send data to server and terminate;
Step 6:The server is that the data of every equipment distribute storage space, and establishes for every equipment corresponding Data packet, for storing total data of this equipment within sometime;
Step 7:The server operation data coding/decoding method obtains parsing data.
Wherein, the spatial cache of the cache module is all made of sequential memory structure.
Preferably, in the step 4 and 5, the cache module when being sampled and being transmitted, adopt respectively by record Sample and output time are ranked up basic data according to the sampling and output time when carrying out sequential storage.
The data that the b grade cache modules transmit successively to b+1 grades of cache modules include:Sampling time data, output Time data, task number and basic data;The basic data is transmitted according to clooating sequence.
Preferably for the same equipment, the data channel keeps one to the sample frequency of all bottom datas It causes, the sample frequency ranging from 1~109Hz。
Preferably, as L=2, basic data is stored in the form of data line and is sent to by described 1 grade of cache module Level 2 cache memory module, the level 2 cache memory module is received data line and is stored in the form of data block, and is sent to the clothes Business device, the server receive data line and are simultaneously stored in the form of data packet.
In the step 6, the data packet according to first data line of each data block sampling time It is front and back that storage is ranked up to data block therein.
Preferably, in the step 7, the server is decoded operation according to setting program, by data from It is converted into the data structure applied to analysis decision convenient for the data structure of transmission, which includes:
Step 8:After data decoding starts, the data packet of the server reading device establishes relevant folder, statement Value is empty variable, is expressed as QUEST;
Step 9:The data in data packet are successively read by storage order, and judge whether the task number of the data is sky, Deterministic process is as follows:
When the task number of the data is empty, then without subsequent operation, next data are skipped to, repeat above-mentioned judgement;
When the task number of the data is not empty, position of the data in the data packet is judged, deterministic process is such as Under:
(1) when the data are first data, then variable QUEST is enabled to be equal to the task number of the data, and described Corresponding assignment folder is established in file, and the corresponding essence data files of basic data are established in the assignment folder Folder, and corresponding basic data is written;
(2) when the data are not first data, whether judgment variable QUEST is equal to the task number of the data;
When variable QUEST is not equal to the task number of the data, corresponding assignment file is established in the file Folder establishes the corresponding essence data files folder of basic data in the assignment folder, and corresponding basic data is written;
When variable QUEST is equal to the task number of the data, corresponding base is written in corresponding essence data files folder Plinth data.
Step 10:When any one process completes write operation to the basic data, judge whether to run through All data in current data packet;When data are not run through, 8 and 9 are repeated the above steps, until all data are run through, it is described Server operation terminate.
Preferably, the basic data includes:Data sequence and corresponding data label;The data mark Label is with the sampling time of each basic data calculated characteristic values;The data sequence is according to sampling time elder generation's heel row The bottom data of row;When carrying out data transmission, first transmission data label, then transmit basic data.
Preferably, this method also includes:
When Sampling interrupt occur in b grades of cache module samplings, then it is currently written into before setting interruption when sampling is restarted Buffer unit terminates write-in, and its data is transmitted to adjacent higher level's cache module, is carried out at the same time position switching, data is made to write Enter into another buffer unit;The break period is calculated according to gathered data amount and frequency acquisition, and the break period is remembered It is written in the data before interrupting for output time, in the data after the sampling time write-in interruption that sampling is restarted, and in Label is added on the disconnected task number for front and back data occur;
It is terminated when there is sampling, then terminates 1 grade of cache module being currently written into, and by data cached transmission upward step by step, Until L grades of cache modules, and the end of record time is output time, and mark is added on the task number for terminating corresponding data Note;
When sampling out current task switching, then stop the buffer unit being currently written into, and its data superior is cached into mould Block transmits, while switching buffer unit and carrying out data write-in;
It is interrupted when b grades of cache modules are to b+1 grades of cache modules or server transmission data, then when transmission is restarted The data label finally received is searched in b+1 grades of cache modules or server first, it is then that the data label is slow with b grades Data in storing module compare, and find the corresponding Data Position of the data label, carry out following operation:
The data for transmitting label in b+1 grades of cache modules or server are deleted, and are again turned on transmission, by the data mark Label and its later data continue to be transferred in b+1 grades of cache modules or server;Or
The point of interruption for finding out the corresponding basic data of the data label in b grades of cache modules or server, is then opened again Transmission is opened, the later basic data of the point of interruption is continued to be transferred in b+1 grades of cache modules or server;It is transmitted when interrupting Be not basic data, then transmit the data of entire b grades of cache module or server again, the processing after interruption should ensure that b+1 Grade cache module or server receive correct and complete data;
When that can not continue to continuously transmit since the break period is long, then stop b grades of cache modules to b+1 grades of cache modules Or server transmission data, until the data storage capacity of the data cell for the b grade cache modules being currently written exceeds threshold value, The data cell of b grades of cache modules is opened to b+1 grades of cache modules or the complete data of server transport, and is passed before and after interruption Plus label on the task number of defeated data.
The industrial flow data coding/decoding system and method for the present invention solves and acquires and arrange to bottom industry big data The problem of, it has the following advantages:
(1) system and method for the invention, can be acquired coding and decoding to industrial big data, convenient for big data It arranges, there is universality;
(2) system and method for the invention uses between buffer unit the storage bottom data in a manner of cycle alternation, full The big feature of the data volume of foot big data, can carry out the acquisition of mass data in limited spatial cache;
(3) system and method for the invention, the spatial cache of cache module are all made of sequential memory structure, ensure that number According to accuracy, will not be chaotic between data;
(4) system and method for the invention, interruption, the termination etc. for acquiring or transmitting to data have specific processing side Formula ensure that the complete and correct of the data of acquisition or transmission.
Description of the drawings
Fig. 1 is that the industrial big data of the present invention acquires the structural schematic diagram of coded system.
Fig. 2 is the data transmission schematic diagram of bottom data-level cache-L2 cache of the present invention.
Fig. 3 is the data transmission schematic diagram of level cache-L2 cache-server of the present invention.
Fig. 4 is the server data decoding process figure of the present invention.
Specific implementation mode
Below in conjunction with drawings and examples, the following further describes the technical solution of the present invention.
A kind of industry big data acquisition coded system, as shown in Figure 1, the industrial big data for the present invention acquires coded system Structural schematic diagram, which includes:A bottom data modules, a are positive integer, and a >=1 is used for the bottom number of collecting device According to, and bottom data is sent to b grades of cache modules;B grades of cache modules, b=1 ... L, L >=2, wherein 1 grade of cache module Bottom data for receiving the transmission of a bottom data modules, higher level's cache module are respectively used to receive adjacent subordinate's caching The data that module is sent, these data are stored in the storage space of corresponding b grades of cache module, and L grades of cache modules receive Data be packaged as data packet and be sent to server;Data channel module is used to bottom data being transferred to b grades of caching moulds Block;And server, it is used to receive the data packet of L grades of cache modules transmission, and be decoded.
Wherein, b grades of cache modules include:More than two buffer units, according to the write-in situation of bottom data, the caching Between unit in a manner of cycle alternation storage bottom data.
Wherein, bottom data includes:Any one in sensing data, current data, director data and program data, this A little data are all to be obtained from equipment, and be converted into digital signal or analog signal is acquired.
A kind of industry big data acquisition coding method, is implemented using above-mentioned data acquisition coded system, such as Fig. 2 institutes Show, be the data transmission schematic diagram of bottom data-level cache-L2 cache of the present invention, as shown in figure 3, being of the invention The data transmission schematic diagram of level cache-L2 cache-server, as shown in figure 4, for the server data decoded stream of the present invention Cheng Tu, this approach includes the following steps:
Step 1 ':Bottom data is established by data channel and level cache in connection structure, each bottom data and spy Fixed data channel corresponds, and data channel is responsible for acquiring bottom data sample delivery to one according to program setting time point In grade caching;
Step 2 ':Data channel sampling is programmed:For the same equipment, data channel adopts every bottom data Sample clock is consistent, and is ensured under arbitrary sampling time point to all bottom datas while being acquired;For the same equipment, number All bottom data sample frequencys are consistent according to channel, sample frequency range desirable 1~109Hz;
Step 3 ':1 grade of caching (1 grade of cache module) receives the bottom data of data channel acquisition, and bottom data is turned It is changed to basic data write-in caching, 1 grade of caching is divided into two parts, A and 1 grade of caching B (buffer unit) of 1 grade of caching, each section All it is that each basic data distributes one-to-one specific spatial cache, each spatial cache uses sequential memory structure, basis Data a (a is positive integer, a >=4) is successively stored in spatial cache successively according to the sampling time:
Step 4 ':When sampling starts, 1 grade first caching A records sampling time N, and then data channel is according to program setting Moment disposably acquires the corresponding all bottom datas of the same equipment successively, and every bottom data is converted to basic data It is written to the corresponding spatial cache of items basic data in 1 grade of caching A simultaneously;Once 1 grade caching A written data amounts be equal to or When person is more than the threshold value of its setting, implement the switching of data writing position immediately according to program setting, so that data is no longer written 1 grade and delay It deposits A but every basic data is written to the corresponding spatial cache of items basic data in 1 grade of caching B, record the sampling time M;Meanwhile it caching the data in A by 1 grade and being packaged into【Data line】It is transferred to level 2 cache memory, and records output time N ';1 grade slow Data are deposited in A to after level 2 cache memory end of transmission, empty level cache A, and set 1 grade of caching A and enter standby mode;Once 1 grade When caching threshold value of the B written data amounts equal to or more than its setting, implement data writing position immediately according to program setting Switching makes data that 1 grade of caching B no longer be written but every basic data is written to every basic data in 1 grade of caching A and is corresponded to Spatial cache, record sampling time K;Meanwhile it caching the data in B by 1 grade and being packaged into【Data line】Level 2 cache memory is transferred to, And record output time M ';Data are to after level 2 cache memory end of transmission, emptying 1 grade of caching B in 1 grade of caching A, and set 1 grade it is slow It deposits B and enters standby mode;And so on until sampling terminates;
Step 5 ':Level 2 cache memory receive 1 grade caching transmission data, 1 grade it is data cached with【Data line】Form be written 2 grades Caching, level 2 cache memory and 1 grade of caching are equally divided into two parts, L2 cache A and L2 cache B;
Specifically, it when level 2 cache memory receives 1 grade of caching transmission data, is written to first in level 2 cache memory A, once 1 grade of caching A When written data amount is equal to or more than the threshold value of its setting, implement the switching of data writing position immediately according to program setting, Make data level 2 cache memory A is no longer written but by 1 grade it is data cached be written in level 2 cache memory B, meanwhile, by the number in level 2 cache memory A According to being packaged into【Data block】It is transferred to server;Data empty level 2 cache memory A to after server transport in level 2 cache memory A, and Setting level 2 cache memory A enters standby mode;When level 2 cache memory B written data amounts are equal to or more than the threshold value of its setting, Implement the switching of data writing position immediately according to program setting, data is made level 2 cache memory A to be no longer written but data cached by 1 grade It is written in level 2 cache memory A, meanwhile, the data in level 2 cache memory B are packaged into【Data block】It is transferred to server;In level 2 cache memory A Data empty level 2 cache memory B, and set level 2 cache memory B and enter standby mode to after level 2 cache memory end of transmission;And so on until Transmission ending;
Step 6 ':If 1 grade caches the level 2 cache memory A/B for terminating and being currently written into level 2 cache memory data transfer termination, and will Level 2 cache memory data are packaged to server transport.
Step 7 ':Server is that the data of every equipment distribute storage space, and establishes for every equipment corresponding【Number According to packet】, for storing the total data in this equipment certain time;【Data packet】According to its correspond to equipment relevant information into Row name, the corresponding L2 cache of every equipment will【Data block】It is passed to this equipment【Data packet】In,【Data packet】It presses According to each【Data block】First【Data line】Sampling time it is front and back right【Data block】It is ranked up storage.
Step 8 ':Data decoding method is carried out in server internal, and server is adopted according to the multiple devices data being collected into Parsing data are obtained with data decoding method, are included the following steps (see Fig. 4):
(1) server is decoded operation after having received a certain amount of data according to program setting, by data from conveniently The data structure of transmission is converted into the data structure that can apply to analysis decision;
(2) after data decoding starts, server reads in equipment first【Data packet】It is denoted as【Data packet Q】, and build The vertical file named with this device-dependent message is denoted as【File DIR-Q】, state that a value is that empty variable is denoted as QUEST is read in order【Data packet Q】In one【Data block】It is denoted as【Data block W】, read in order【Data block W】In One【Data line】It is denoted as【Data line E】, judge【Data line E】Whether middle task number is empty;
If【Data line E】Task number is sky, then without subsequent operation, directly skips【Data line E】It reads in order again 【Data block W】In【Data line E】It is next【Data line】;
If【Data line E】Task number is not sky, then judges【Data line E】Whether be【Data packet Q】In first【Data Block】In the first row, if【Data line E】It is【Data packet Q】In first【Data block】In the first row, then enable variable QUEST It is equal to【Data line E】Task number, and【File DIR-Q】Middle foundation【Data line E】The file of task number name is denoted as【Text Part presss from both sides DIR-Q- task numbers】, then【File DIR-Q- task numbers】It is middle to establish the file named with every basic data label It is denoted as【1 file of basic data】、【2 file of basic data】……【Basic data n files】, then by the number of every basic data The file corresponding to it is written to according to sequence;If【Data line E】It is not【Data packet Q】In first【Data block】In first Row, then whether judgment variable QUEST is equal to【Data line E】Task number, if variable QUEST is not equal to【Data line E】Task number, then And【File DIR-Q】Middle foundation【Data line E】The file of current task number name is denoted as【File DIR-Q- tasks Number】, then【File DIR-Q- task numbers】The middle file named with every basic data label of establishing is denoted as【Basic data 1 File】、【2 file of basic data】……【Basic data n files】, the data sequence of every basic data is then written to it Corresponding file;If variable QUEST is equal to【Data line E】Task number then will【Data line E】The data of middle items basic data Sequence is added to【File DIR-Q- task numbers】The tail portion of the corresponding file of middle items basic data label;
(3) when any of the above process completes write operation to essence data files, will judge whether to run through current【Number According to packet】It is lower current【Data block】It is all【Data line】;
If not running through current【Data packet】It is lower current【Data block】It is all【Data line】, then read in again current【Number According to packet】It is lower current【Data block】It is next【Data line】;
If having run through current【Data packet】It is lower current【Data block】It is all【Data line】, then judge whether to run through current 【Data packet】Under it is all【Data block】;If not running through current【Data packet】Under it is all【Data block】, then read in and work as again Before【Data packet】It is next【Data block】;It has run through current【Data packet】Under it is all【Data block】, then judge whether to read It is complete all【Data packet】;If not running through all【Data packet】, then equipment is read in again【Data packet】, and establish with this The file of platform device-dependent message name states a value for empty variable QUEST;If having run through all【Data packet】, Then server operation terminates.
Further, method of the invention further includes:If server data encounters when decoding<Level cache Sampling interrupt mark Note>、<Level cache is marked to L2 cache Transmission>、<L2 cache is to server transport interrupt flag>Three kinds of labels, It is then automatically separated out task before and after stage casing, which is conducive to processing of the later stage to data.
Above-mentioned sampling time N is one group with corresponding output time N ', and sampling time M is one with corresponding output time M ' Group, and so on.
It is above-mentioned【Data line】In data according to sampling time, output time, task number, basic data 1, basic data 2 ... basic data n's is ranked sequentially (as shown in Figure 1);Each basic data can be divided into data label and data sequence two again The data sequence of part, each basic data is the data to sort according to sampling time sequencing.
Above-mentioned 1 grade of caching gross storage space is respectively the sum of the storage spaces A and 1 grade of caching B of 1 grade of caching, 1 grade of caching A and 1 The storage spaces grade caching B are equal and equal to the integral multiple of server file storage minimum unit;A and 1 grade of caching B's of 1 grade of caching Threshold value be set to be slightly less than respective storage space, with ensure data completely receive without overflowing.
It should ensure that 1 grade of caching write-in shortest time is more than or equal to 1 grade of caching program execution time and 1 by software and hardware collocation Grade is data cached to transmit the sum of maximum duration to level 2 cache memory, and bottom data is passed to rate by data channel to 1 grade of caching should be with 1 grade data cached to match to level 2 cache memory transmission rate, and should ensure that system overall performance is optimal as far as possible.
In step 4 ' in, further, method of the invention further includes:If there is Sampling interrupt, when sampling is restarted The caching A/B that setting is currently written into before interrupting terminates write-in, and its data is transmitted to level 2 cache memory, while switching makes data write Enter to caching B/A;The break period is calculated according to gathered data amount and frequency acquisition, and the time is denoted as output time and is write Before entering interruption【Data line】, after the time write-in interruption that sampling is restarted【Data line】Sampling time, and interrupt occur before Afterwards【Data line】It is added on task number<1 grade of caching Sampling interrupt label>.
In step 4 ' in, further, method of the invention further includes:If sampling occur to terminate, termination is currently written into Level cache A/B, and by level cache data to L2 cache transmit, while the end of record time be denoted as output time, and It is corresponding terminating【Data line】It is added on task number<1 grade of caching sampling end mark>.
In step 4 ' in, further, method of the invention further includes:If going out current task switching when sampling, stop just It is transmitted to level 2 cache memory in the caching A/B of write-in, and by its data, while switching makes data be written to caching B/A, at this time【Data Row】Task number changes.
In step 4 ' in, further, method of the invention further includes:If being not carried out task when sampling,【Data line】 Task number is written as sky.
In step 4 ' in, as described above, 1 grade when caching to level 2 cache memory transmission data, with【Data line】For basic unit. Further, method of the invention further includes:【Data line】When transmission, sampling time data are first passed, then pass output time number According to then transformation task number, is finally transmitted by the sequence of basic data 1, basic data 2 ... .. basic datas n one by one again Collected basic data;When basic data is transmitted, label is first passed, the data of data sequence, data are passed according still further to sequencing Data are arranged according to chronological order in sequence.
In step 4 ' in, further, method of the invention further includes:If 1 grade when caching to level 2 cache memory transmission data It interrupts, then the basic data label finally received is searched in level 2 cache memory first when transmission is restarted, then by the mark It signs and is compared with the data in 1 grade of caching, find out the label corresponding data position, then implement following two schemes:
(1) data of label will be given to delete in level 2 cache memory, and is again turned on transmission, by the label and its later data Continue to be transferred to level 2 cache memory;
(2) point of interruption for continuing will to find out data under the data label in 1 grade of caching, is then again turned on transmission, by this The later data of the point of interruption continue to be transferred to level 2 cache memory;It is non-basic data to be transmitted when if interrupting, and transmits entire 1 again The data of grade caching, the processing of interruption should ensure that level 2 cache memory received【Data line】It is correct and complete.
In step 4 ' in, further, method of the invention further includes:If leading to 1 grade of caching since the break period is long It can not be further continued for continuously transmitting to level 2 cache memory, then system is dished out<1 grade caches to level 2 cache memory Transmission exception>, it is simultaneously stopped 1 grade caches to level 2 cache memory transmission data, when 1 grade of caching A/B data storage capacity being currently written exceeds threshold value, then 1 grade of caching A/B of secondary unlatching is complete to level 2 cache memory【Data line】Transmission, and transmit before and after interruption【Data line】Task number Upper addition<1 grade caches to level 2 cache memory Transmission label>.
Above-mentioned level 2 cache memory gross storage space is respectively the sum of the storage spaces level 2 cache memory A and level 2 cache memory B, level 2 cache memory A and 2 The storage spaces grade caching B are equal and equal to the integral multiple of server file storage minimum unit;Level 2 cache memory A's and level 2 cache memory B Threshold value be set to be slightly less than respective storage space, with ensure data completely receive without overflowing.
It should ensure that the shortest time is written more than or equal to level 2 cache memory program execution time and 2 in level 2 cache memory by software and hardware collocation Grade is data cached to transmit the sum of maximum duration to level 2 cache memory, 1 grade data cached be passed to rate to level 2 cache memory should be with level 2 cache memory number Match according to server transport rate, and should ensure that system overall performance is optimal as far as possible.
In step 5 ' in, further, method of the invention further includes:【Data block】In data be according to sampling when Between sort【Data line】, when L2 cache prepares to server transport, program is run to adjacent【Data line】Sampling Time and output time judge, if in the presence of before【Data line】Output time be not equal to after【Data line】Sampling time, Then dish out<Data line is discontinuously abnormal>, and recording exceptional position;For convenience【Data block】Number, each【Data block】All it is arranged One corresponding label, label be with【Data block】In first【Data line】Sampling time calculated characteristic value.
In step 5 ' in, when L2 cache is to server transmission data, with【Data block】For basic unit.In step 5 ' In, further, method of the invention further includes:【Data block】When transmission, according to【Data block】In【Data line】Sampling when Between sequencing transmit one by one;【Data line】Transmission first pass sampling time data, then transmit out time data, then pass task Number finally transmits collected basic number one by one by the sequence of basic data 1, basic data 2 ... .. basic datas n again According to;When basic data is transmitted, first pass label, the data of data sequence passed according still further to sequencing, in data sequence data be by According to chronological order arrangement.
In step 5 ' in, further, method of the invention further includes:Go out when if level 2 cache memory is to server transmission data It now interrupts, then searches and finally receive in the server first when transmission is restarted【Data line】Sampling time and should【Number According to row】Last basic data label, then the label and the data in level 2 cache memory are compared, when finding out the sampling Between it is corresponding【Data line】The basic data finally transmitted with the data line, then implementing following three kinds of schemes (below will be last It receives【Data line】It is denoted as【Data line X】, should【Data line】The basic data finally transmitted be denoted as【Basic data Y】):
It (1) will be in server【Data line X】It deletes, and is again turned on transmission, it will【Data line X】And its it is later【Number According to row】Continue to be transferred to server in order;
It (2) will be in server【Data line X】's【Basic data Y】It deletes, and is again turned on transmission, it will【Data line X】In 【Basic data Y】And its later basic data and【Data line X】Later【Data line】Continue to be transferred to server in order;
(3) continue to find out in level 2 cache memory【Data line X】's【Basic data Y】The point of interruption of lower data, is then opened again Transmission is opened, it will【Basic data Y】The later data of the point of interruption,【Data line X】In【Basic data Y】Later basic data and 【Data line X】Later【Data line】Continue to be transferred to server in order;The processing of interruption should ensure that server received 【Data block】It is correct and complete.
In step 5 ' in, further, method of the invention further includes:If leading to level 2 cache memory since the break period is long It can not be further continued for continuously transmitting to server, then system is dished out<Level 2 cache memory is to server transport abnormal interruption>, it is simultaneously stopped 2 Grade is cached to server transmission data, when the level 2 cache memory A/B data storage capacities being currently written exceed threshold value, again It is complete to server to open level 2 cache memory A/B【Data block】Transmission, and transmit before and after interruption【Data block】It is added on label <Level 2 cache memory is to server transport interrupt flag>.
In conclusion the industrial flow data coding/decoding system and method, the system and method for the present invention can be to industry Big data is acquired and handles, and has universality, disclosure satisfy that the big feature of the data volume of big data, can be limited Spatial cache in carry out mass data acquisition, and ensure that acquisition or transmission data it is complete and correct.
Although present disclosure is discussed in detail by above preferred embodiment, but it should be appreciated that above-mentioned Description is not considered as limitation of the present invention.After those skilled in the art have read the above, for the present invention's A variety of modifications and substitutions all will be apparent.Therefore, protection scope of the present invention should be limited to the appended claims.

Claims (10)

1. a kind of industry flow data coding/decoding system, which is characterized in that the system includes:
A bottom data modules, a are positive integer, a >=1, are used for the bottom data of collecting device, and by the bottom number According to being sent to b grades of cache modules;
B grades of cache modules, b=1 ... L, L >=2, wherein 1 grade of cache module is used to receive a bottom data modules The bottom data of transmission, higher level's cache module are respectively used to receive the data that adjacent subordinate's cache module is sent, these data It is stored in the storage space of corresponding b grades of cache module, the data that L grades of cache modules receive are packaged as data packet and are sent to Server;
Data channel module is used to for the bottom data to be transferred to the b grade cache modules;And
Server is used to receive the data packet that the L grade cache modules are sent, and is decoded;
The b grade cache modules include:More than two buffer units should according to the write-in situation of the bottom data The bottom data is stored between buffer unit in a manner of cycle alternation.
2. industry flow data coding/decoding system according to claim 1, which is characterized in that the bottom data packet It includes:Any one in sensing data, current data, director data and program data.
3. industry flow data coding/decoding system according to claim 1, which is characterized in that in 1 grade of caching mould In block, the bottom data is converted into basic data, is stored in the storage space of 1 grade of cache module;
The basic data is sequentially stored according to acquisition time in the storage space of the b grade cache modules;The b Grade cache module storage space for its all buffer unit for being included the sum of storage space;
The storage space of each buffer unit in the b grade cache modules is equal, and equal to the file of the server The integral multiple of minimum unit is stored, the threshold value of each buffer unit is less than its respective storage space.
4. the industrial flow data coding/decoding system according to any one of claim 1-3, which is characterized in that described The shortest time of b grades of cache modules write-in be more than or equal to b grade cache modules program execution time and b grades of cache modules to b+1 The sum of grade cache module transmission maximum duration;
The bottom data is passed to rate to 1 grade of cache module should be fast to b+1 grades of cache module transmission with b grades of cache modules Rate matches.
5. a kind of industry flow data coding-decoding method, which is characterized in that this method is used such as any one of claim 1-4 The industrial flow data coding/decoding system, this method include:
Step 1:Bottom data is established with b grades of cache modules by corresponding data channel and is connected, and the data channel is responsible for It is acquired in bottom data sample delivery to 1 grade of cache module according to program setting time point;
Step 2:The data channel sampling is programmed:For the same equipment, the data channel is to every bottom Layer data sampling clock is consistent, and is ensured under arbitrary sampling time point to all bottom datas while being acquired;
Step 3:1 grade of cache module receives the bottom data of the data channel acquisition, and the bottom data is turned It is changed to basic data write-in caching, each basic data distributes corresponding spatial cache;
Step 4:When sampling starts, the data channel acquires the corresponding institute of the same equipment successively according to the program setting moment There is bottom data, and every bottom data is converted into basic data, is written in the buffer unit of 1 grade of cache module Whether the corresponding spatial cache of every basic data, continue to write to according to the threshold decision data of spatial cache, if write-in is more than Threshold value then stops being written, and basic data is packaged and is sent to the b+1 grade cache modules, is emptied to the buffer unit, Line position of going forward side by side sets switching, is continued to write to switching to another buffer unit, which enters standby mode;Repeat step 4 Until sampling terminates;
Step 5:The b+1 grade cache modules receive and are written the basic data of the b grade cache modules transmission, and institute is written In the buffer unit for the b+1 grade cache modules stated, whether continued to write to according to the threshold decision data of spatial cache, if write-in is big In threshold value, then stop being written, the data of transmission are packaged and are sent to the b+2 grade cache modules or server, to the caching Unit is emptied, and line position of going forward side by side sets switching, is continued to write to switching to another buffer unit, which enters waiting mould Formula;Step 5 is repeated until L grades of cache modules send data to server and terminate;
Step 6:The server is that the data of every equipment distribute storage space, and establish corresponding number for every equipment According to packet, for storing this equipment in sometime interior total data;
Step 7:The server operation data coding/decoding method obtains parsing data;
The spatial cache of the cache module is all made of sequential memory structure.
6. industry flow data coding-decoding method according to claim 5, which is characterized in that in the step 4 and 5 In, the cache module records sampling and output time, the root when carrying out sequential storage respectively when being sampled and being transmitted Basic data is ranked up according to the sampling and output time;The b grade cache modules transmit successively to b+1 grades of cache modules Data include:Sampling time data, output time data, task number and basic data;The basic data according to Clooating sequence is transmitted.
7. industry flow data coding-decoding method according to claim 6, which is characterized in that for the same equipment, institute The data channel stated is consistent the sample frequency of all bottom datas, the sample frequency ranging from 1~109Hz。
8. industry flow data coding-decoding method according to claim 7, which is characterized in that as L=2, described 1 grade Basic data is stored in the form of data line and is sent to level 2 cache memory module by cache module, and the level 2 cache memory module receives Data line is simultaneously stored in the form of data block, and is sent to the server, and the server receives data line and with number It is stored according to the form of packet;
In the step 6, before and after sampling time of the data packet according to first data line of each data block Storage is ranked up to data block therein.
9. the industrial flow data coding-decoding method according to any one of claim 5-8, which is characterized in that described Step 7 in, the server is decoded operation according to setting program, by data from convenient for transmission data structure conversion For the data structure applied to analysis decision, which includes:
Step 8:After data decoding starts, the data packet of the server reading device establishes relevant folder, and statement value is Empty variable, is expressed as QUEST;
Step 9:The data in data packet are successively read by storage order, and judge whether the task number of the data is empty, is judged Process is as follows:
(1) when the task number of the data is empty, then without subsequent operation, next data is skipped to, repeat above-mentioned judgement;
(2) when the task number of the data is not empty, judge position of the data in the data packet, deterministic process is such as Under:
(3) when the data are first data, then variable QUEST is enabled to be equal to the task number of the data, and in the file Corresponding assignment folder is established in folder, and the corresponding essence data files folder of basic data is established in the assignment folder, and Corresponding basic data is written;
(4) when the data are not first data, whether judgment variable QUEST is equal to the task number of the data;
When variable QUEST is not equal to the task number of the data, corresponding assignment folder is established in the file, The corresponding essence data files folder of basic data is established in the assignment folder, and corresponding basic data is written;
When variable QUEST is equal to the task number of the data, corresponding basic number is written in corresponding essence data files folder According to.
Step 10:When any one process completes write operation to the basic data, judge whether to run through current All data in data packet;When data are not run through, 8 and 9 are repeated the above steps, until running through all data, the clothes Business device operation terminates.
10. industry flow data coding-decoding method according to claim 9, which is characterized in that the basic data packet It includes:Data sequence and corresponding data label;The data label is to be calculated with the sampling time of each basic data The characteristic value gone out;The data sequence is the bottom data according to the sampling time successively arrangement;When carrying out data transmission, first Transmission data label, then transmit basic data;
This method also includes:
When Sampling interrupt occur in b grades of cache module samplings, then the caching being currently written into before interrupting is set when sampling is restarted Unit terminates write-in, and its data is transmitted to adjacent higher level's cache module, is carried out at the same time position switching, data is made to be written to In another buffer unit;The break period is calculated according to gathered data amount and frequency acquisition, and the break period is denoted as defeated Go out in the data that the time is written to before interrupting, in the data after the sampling time write-in interruption that sampling is restarted, and is interrupting out Label is added on the task number of existing front and back data;
It is terminated when there is sampling, then 1 grade of cache module being currently written into is terminated, and by data cached transmission upward step by step, until L Grade cache module, and the end of record time is output time, and label is added on the task number for terminating corresponding data;
When sampling out current task switching, then stop the buffer unit being currently written into, and its data superior cache module is passed It is defeated, while switching buffer unit and carrying out data write-in;
It is interrupted when b grades of cache modules are to b+1 grades of cache modules or server transmission data, then first when transmission is restarted The data label finally received is searched in b+1 grades of cache modules or server, then by the data label and b grades of caching moulds Data in the block compare, and find the corresponding Data Position of the data label, carry out following operation:
(1) data for transmitting label in b+1 grades of cache modules or server are deleted, and is again turned on transmission, by the data mark Label and its later data continue to be transferred in b+1 grades of cache modules or server;Or
(2) point of interruption for finding out the corresponding basic data of the data label in b grades of cache modules or server, is then again turned on Transmission, the later basic data of the point of interruption is continued to be transferred in b+1 grades of cache modules or server;It is transmitted when interrupting It is not basic data, then transmits the data of entire b grades of cache module or server again, the processing after interruption should ensure that b+1 grades Cache module or server receive correct and complete data;
When that can not continue to continuously transmit since the break period is long, then stop b grades of cache modules to b+1 grades of cache modules or clothes Device transmission data of being engaged in is opened until the data storage capacity of the data cell for the b grade cache modules being currently written exceeds threshold value The data cell of b grades of cache modules is transmitted before and after interruption to b+1 grades of cache modules or the complete data of server transport Plus label on the task number of data.
CN201810457697.3A 2018-05-14 2018-05-14 Industrial stream data coding and decoding system and method Active CN108551490B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810457697.3A CN108551490B (en) 2018-05-14 2018-05-14 Industrial stream data coding and decoding system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810457697.3A CN108551490B (en) 2018-05-14 2018-05-14 Industrial stream data coding and decoding system and method

Publications (2)

Publication Number Publication Date
CN108551490A true CN108551490A (en) 2018-09-18
CN108551490B CN108551490B (en) 2021-06-18

Family

ID=63494770

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810457697.3A Active CN108551490B (en) 2018-05-14 2018-05-14 Industrial stream data coding and decoding system and method

Country Status (1)

Country Link
CN (1) CN108551490B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109379437A (en) * 2018-11-21 2019-02-22 苏州金螳螂文化发展股份有限公司 Output Velocity Dealing Method and system based on data buffer storage
CN109522140A (en) * 2018-11-23 2019-03-26 江西洪都航空工业集团有限责任公司 Communication Card receives data method
CN109889456A (en) * 2018-12-29 2019-06-14 厦门海为科技有限公司 A kind of data transmission method, device, equipment, system and storage medium
CN111385332A (en) * 2018-12-29 2020-07-07 顺丰科技有限公司 Internet of things equipment, Internet of things platform access method and equipment
CN112002115A (en) * 2020-08-05 2020-11-27 中车工业研究院有限公司 Data acquisition method and data processor

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1852318A (en) * 2006-04-19 2006-10-25 华中科技大学 Distributed multi-stage buffer storage system suitable for object network storage
CN101937475A (en) * 2010-08-27 2011-01-05 中国电子科技集团公司第四十一研究所 Multi-rate multi-channel data acquisition method
US20120110113A1 (en) * 2009-07-02 2012-05-03 Aim To G Co., Ltd. Cooperative Caching Method and Contents Providing Method Using Request Apportioning Device
CN102497450A (en) * 2011-12-28 2012-06-13 北京华电天仁电力控制技术有限公司 Two-stage-system-based distributed data compression processing method
CN103034538A (en) * 2012-12-24 2013-04-10 珠海拓普智能电气股份有限公司 Multi-level cache processing method of drive program in embedded type operation system
CN104699626A (en) * 2013-12-10 2015-06-10 中兴通讯股份有限公司 Terminal internal memory processing method, device and terminal
CN104935680A (en) * 2015-06-18 2015-09-23 中国互联网络信息中心 Recursive domain name service system and method of multi-level shared cache
CN105979218A (en) * 2016-06-13 2016-09-28 中南大学 High-definition real-time monitoring video efficient transmission method
CN106155936A (en) * 2015-04-01 2016-11-23 华为技术有限公司 A kind of buffer replacing method and relevant apparatus
CN106200642A (en) * 2016-08-05 2016-12-07 上海电力学院 The two-wheeled balance car of a kind of realizing route automatic Memory and reproduction independently goes on patrol method
CN107273522A (en) * 2015-06-01 2017-10-20 明算科技(北京)股份有限公司 Towards the data-storage system and data calling method applied more

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1852318A (en) * 2006-04-19 2006-10-25 华中科技大学 Distributed multi-stage buffer storage system suitable for object network storage
US20120110113A1 (en) * 2009-07-02 2012-05-03 Aim To G Co., Ltd. Cooperative Caching Method and Contents Providing Method Using Request Apportioning Device
CN101937475A (en) * 2010-08-27 2011-01-05 中国电子科技集团公司第四十一研究所 Multi-rate multi-channel data acquisition method
CN102497450A (en) * 2011-12-28 2012-06-13 北京华电天仁电力控制技术有限公司 Two-stage-system-based distributed data compression processing method
CN103034538A (en) * 2012-12-24 2013-04-10 珠海拓普智能电气股份有限公司 Multi-level cache processing method of drive program in embedded type operation system
CN104699626A (en) * 2013-12-10 2015-06-10 中兴通讯股份有限公司 Terminal internal memory processing method, device and terminal
CN106155936A (en) * 2015-04-01 2016-11-23 华为技术有限公司 A kind of buffer replacing method and relevant apparatus
CN107273522A (en) * 2015-06-01 2017-10-20 明算科技(北京)股份有限公司 Towards the data-storage system and data calling method applied more
CN104935680A (en) * 2015-06-18 2015-09-23 中国互联网络信息中心 Recursive domain name service system and method of multi-level shared cache
CN105979218A (en) * 2016-06-13 2016-09-28 中南大学 High-definition real-time monitoring video efficient transmission method
CN106200642A (en) * 2016-08-05 2016-12-07 上海电力学院 The two-wheeled balance car of a kind of realizing route automatic Memory and reproduction independently goes on patrol method

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109379437A (en) * 2018-11-21 2019-02-22 苏州金螳螂文化发展股份有限公司 Output Velocity Dealing Method and system based on data buffer storage
CN109379437B (en) * 2018-11-21 2022-02-01 苏州金螳螂文化发展股份有限公司 Speed processing method and system based on data caching
CN109522140A (en) * 2018-11-23 2019-03-26 江西洪都航空工业集团有限责任公司 Communication Card receives data method
CN109522140B (en) * 2018-11-23 2023-10-20 江西洪都航空工业集团有限责任公司 Method for receiving data by communication board card
CN109889456A (en) * 2018-12-29 2019-06-14 厦门海为科技有限公司 A kind of data transmission method, device, equipment, system and storage medium
CN111385332A (en) * 2018-12-29 2020-07-07 顺丰科技有限公司 Internet of things equipment, Internet of things platform access method and equipment
CN109889456B (en) * 2018-12-29 2023-09-05 厦门海为科技有限公司 Data transmission method, device, equipment, system and storage medium
CN112002115A (en) * 2020-08-05 2020-11-27 中车工业研究院有限公司 Data acquisition method and data processor
CN112002115B (en) * 2020-08-05 2021-04-23 中车工业研究院有限公司 Data acquisition method and data processor

Also Published As

Publication number Publication date
CN108551490B (en) 2021-06-18

Similar Documents

Publication Publication Date Title
CN108551490A (en) A kind of industry flow data coding/decoding system and method
CN102521406B (en) Distributed query method and system for complex task of querying massive structured data
CN103995855B (en) The method and apparatus of data storage
CN102831405B (en) Method and system for outdoor large-scale object identification on basis of distributed and brute-force matching
CN102253820A (en) Stream type repetitive data detection method
CN107733696A (en) A kind of machine learning and artificial intelligence application all-in-one dispositions method
CN102880854A (en) Distributed processing and Hash mapping-based outdoor massive object identification method and system
CN106294826A (en) A kind of company-data Query method in real time and system
CN107957848A (en) Processing method and storage device are deleted again
CN104765701B (en) Data access method and equipment
CN107454144A (en) Information processing method and server
CN109240607A (en) A kind of file reading and device
CN108257375B (en) Power grid meter reading method and system
CN104899249B (en) Reliable index upgrade system and method under a kind of mass data
CN110018997A (en) A kind of mass small documents storage optimization method based on HDFS
CN108090186A (en) A kind of electric power data De-weight method on big data platform
CN103136225B (en) A kind of method and system of Internet picture conversion
CN112905571B (en) Train rail transit sensor data management method and device
CN101783814A (en) Metadata storing method for mass storage system
CN103543959A (en) Method and device for mass data caching
CN108959614A (en) A kind of snapshot management method, system, device, equipment and readable storage medium storing program for executing
CN109144431A (en) Caching method, device, equipment and the storage medium of data block
CN108228752A (en) Data full dose deriving method, data distribution device and data export node
CN111124313A (en) Data reading and writing method and device for power acquisition terminal and electronic equipment
CN106021574A (en) Data storage replication method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant