CN108551490B - Industrial stream data coding and decoding system and method - Google Patents
Industrial stream data coding and decoding system and method Download PDFInfo
- Publication number
- CN108551490B CN108551490B CN201810457697.3A CN201810457697A CN108551490B CN 108551490 B CN108551490 B CN 108551490B CN 201810457697 A CN201810457697 A CN 201810457697A CN 108551490 B CN108551490 B CN 108551490B
- Authority
- CN
- China
- Prior art keywords
- data
- level
- cache module
- level cache
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
Abstract
The invention discloses an industrial stream data coding and decoding system and a method, wherein an a-th bottom layer data module is used for acquiring bottom layer data of equipment and sending the bottom layer data to a b-level cache module, wherein a is a positive integer and is more than or equal to 1; the system comprises a b-level cache module, a 1-level cache module, a superior cache module and a server, wherein b is 1 … … L, L is more than or equal to 2, the 1-level cache module is used for receiving bottom-layer data sent by an a-th bottom-layer data module, the superior cache module is respectively used for receiving data sent by adjacent inferior cache modules, the data are stored in storage spaces of the corresponding b-level cache modules, and the data received by the L-level cache module are packaged into data packets and sent to the server; the data channel module is used for transmitting the bottom layer data to the b-level cache module; and the server is used for receiving and decoding the data packet sent by the L-level cache module. The system and the method can acquire a large amount of data in a limited cache space and ensure the integrity and the correctness of the acquired or transmitted data.
Description
Technical Field
The invention belongs to the technical field of industrial big data, and particularly relates to an industrial stream data coding and decoding system and method.
Background
Intelligent manufacturing based on technologies such as cloud computing, big data, artificial intelligence, etc. has become an important direction for the development of industry at present and even in the future. In smart manufacturing, industrial big data technology is a key technology. The method is a basic idea for the technical development of industrial big data by efficiently collecting mass bottom data, constructing an industrial data model and assisting enterprise decision according to the model.
In the implementation process of the industrial big data technology, the most important thing is to collect and arrange the bottom layer industrial data which has the characteristics of high generation frequency, small single data volume, strict time sequence, large data total amount and the like.
At present, few methods are available in the world for collecting and organizing large data of the underlying industry, and most of the methods are specific to a specific device, and a method with certain universality is not proposed.
Disclosure of Invention
The invention aims to provide an industrial stream data coding and decoding system and method, which solve the problem of acquisition and arrangement of bottom industrial big data, can acquire, encode and decode the industrial big data, are convenient for arrangement of the big data and have universality.
In order to achieve the above object, the present invention provides an industrial stream data encoding and decoding system, comprising: the first bottom layer data module a is a positive integer, a is more than or equal to 1 and is used for acquiring bottom layer data of equipment and sending the bottom layer data to the level-b cache module; the b-level cache module, b is 1 … … L, L is greater than or equal to 2, wherein the 1-level cache module is used for receiving the bottom data sent by the a-th bottom data module, the upper-level cache module is respectively used for receiving the data sent by the adjacent lower-level cache module, the data are all stored in the storage space of the corresponding b-level cache module, and the data received by the L-level cache module are packaged into data packets and sent to the server; the data channel module is used for transmitting the bottom layer data to the b-level cache module; and the server is used for receiving the data packet sent by the L-level cache module and decoding the data packet.
Wherein, the b-level cache module comprises: and more than two buffer units, wherein the buffer units store the bottom layer data in a cyclic alternating mode according to the writing condition of the bottom layer data.
Preferably, the underlying data includes: any one of sensing data, current data, instruction data, and program data.
Preferably, in the level 1 cache module, the bottom layer data is converted into basic data and stored in the storage space of the level 1 cache module.
The basic data are sequentially stored in the storage space of the b-level cache module according to the acquisition time; the storage space of the b-level cache module is the sum of the storage spaces of all the cache units contained in the b-level cache module.
The storage space of each cache unit in the b-level cache module is equal and equal to the integral multiple of the minimum unit of file storage of the server, and the threshold value of each cache unit is smaller than the respective storage space.
Preferably, the shortest time written by the b-level cache module is greater than or equal to the sum of the program execution time of the b-level cache module and the longest time transmitted from the b-level cache module to the b + 1-level cache module.
The transmission rate of the bottom layer data to the level 1 cache module is matched with the transmission rate of the level b cache module to the level b +1 cache module.
The invention also provides an industrial stream data coding and decoding method, which adopts the industrial stream data coding and decoding system and comprises the following steps:
step 1: the bottom layer data is connected with the level b cache module through a corresponding data channel, and the data channel is responsible for collecting bottom layer data samples according to a program set time point and transmitting the bottom layer data samples to the level 1 cache module;
step 2: the data channel sampling is controlled by a program: for the same device, the data channel keeps the sampling clocks of all the bottom layer data consistent, and ensures that all the bottom layer data are simultaneously acquired at any sampling time point;
and step 3: the level 1 cache module receives the bottom data acquired by the data channel, converts the bottom data into basic data and writes the basic data into a cache, and each item of basic data is allocated with a corresponding cache space;
and 4, step 4: when sampling starts, the data channel sequentially collects all bottom layer data corresponding to the same equipment according to a program set moment, converts each bottom layer data into basic data, writes the basic data into a cache space corresponding to each item of basic data in a cache unit of the level 1 cache module, judges whether the data is continuously written according to a threshold value of the cache space, stops writing if the data is written into the cache space to be larger than the threshold value, packs and transmits the basic data to the level b +1 cache module, clears the cache unit, switches the position to switch to another cache unit to continuously write, and the cache unit enters a waiting mode; repeating the step 4 until the sampling is finished;
and 5: the b +1 level cache module receives and writes basic data transmitted by the b level cache module into a cache unit of the b +1 level cache module, judges whether data are continuously written according to a threshold value of a cache space, if the data are written to be larger than the threshold value, the writing is stopped, the transmitted data are packaged and transmitted to the b +2 level cache module or a server, the cache unit is emptied and the position is switched to be switched to another cache unit for continuous writing, and the cache unit enters a waiting mode; repeating the step 5 until the L-level cache module transmits the data to the server;
step 6: the server allocates storage space for the data of each device, establishes a corresponding data packet for each device, and is used for storing all the data of the device in a certain time;
and 7: the server runs a data decoding method to obtain analysis data.
The buffer memory space of the buffer memory module adopts a time sequence storage structure.
Preferably, in the steps 4 and 5, the buffer module records sampling time and output time respectively during sampling and transmission, and sorts the basic data according to the sampling time and the output time during time-series storage.
The data sequentially transmitted from the b-level cache module to the b + 1-level cache module comprises: sampling time data, output time data, task number data and basic data; and the basic data are transmitted according to the sequencing order.
Preferably, for the same device, the data channel keeps the sampling frequency of all the bottom layer data consistent, and the sampling frequency range is 1-109Hz。
Preferably, when L is 2, the level 1 cache module stores the basic data in the form of a data line and sends the basic data to the level 2 cache module, the level 2 cache module receives the data line and stores the data line in the form of a data block and sends the data line to the server, and the server receives the data line and stores the data line in the form of a data packet.
In step 6, the data packet sequences and stores the data blocks in the data packet according to the sampling time of the first data line of each data block.
Preferably, in step 7, the server performs a decoding operation according to a set program to convert data from a data structure convenient for transmission into a data structure applied to an analysis decision, and the data decoding method includes:
and 8: after the data decoding starts, the server reads in a data packet of the equipment, establishes a related folder, and declares a variable with a null value as represented by a request;
and step 9: sequentially reading the data in the data packet according to the storage sequence, and judging whether the task number of the data is empty, wherein the judging process is as follows:
when the task number of the data is empty, the subsequent operation is not carried out, the next data is skipped to, and the judgment is repeated;
when the task number of the data is not empty, the position of the data in the data packet is judged, and the judging process is as follows:
(1) when the data is the first data, making the variable QUEST equal to the task number of the data, establishing a corresponding task folder in the folder, establishing a basic data folder corresponding to the basic data in the task folder, and writing the corresponding basic data;
(2) when the data is not the first data, judging whether the variable QUEST is equal to the task number of the data;
when the variable QUEST is not equal to the task number of the data, establishing a corresponding task folder in the folder, establishing a basic data folder corresponding to the basic data in the task folder, and writing the corresponding basic data;
when the variable QUEST is equal to the task number of the data, the corresponding basic data is written in the corresponding basic data folder.
Step 10: when the writing operation of any process to the basic data is completed, judging whether all data in the current data packet is completely read; and when the data is not read completely, repeating the steps 8 and 9 until all the data are read completely, and finishing the operation of the server.
Preferably, the basic data includes: a data sequence, and a data tag corresponding thereto; the data label is a characteristic value calculated by sampling time of each basic data; the data sequence is bottom layer data arranged in sequence according to sampling time; when data transmission is carried out, the data tags are transmitted first, and then basic data are transmitted.
Preferably, the method further comprises:
when sampling interruption occurs in the b-level cache module, setting a cache unit which is written before interruption to terminate writing when sampling is restarted, transmitting the data to an adjacent upper-level cache module, and simultaneously performing position switching to write the data into another cache unit; calculating interruption time according to the acquired data volume and the acquisition frequency, recording the interruption time as output time, writing the output time into data before interruption, writing sampling time of sampling restart into the data after interruption, and adding marks on task numbers of the data before and after the interruption;
when sampling is terminated, terminating the 1-level cache module which is being written in, transmitting the cache data upwards step by step until the L-level cache module, recording termination time as output time, and adding a mark on a task number for terminating the corresponding data;
when the task switching occurs in the sampling, stopping the written cache unit, transmitting the data to the upper-level cache module, and simultaneously switching the cache unit and writing the data;
when the b-level cache module is interrupted when transmitting data to the b + 1-level cache module or the server, firstly searching the last received data tag in the b + 1-level cache module or the server when restarting transmission, then comparing the data tag with the data in the b-level cache module, searching the data position corresponding to the data tag, and performing the following operations:
deleting the data of the transmission label in the b +1 level cache module or the server, starting transmission again, and continuously transmitting the data label and the subsequent data to the b +1 level cache module or the server; or
Finding out an interruption point of the basic data corresponding to the data tag in the b-level cache module or the server, then starting transmission again, and continuously transmitting the basic data after the interruption point to the b + 1-level cache module or the server; when the basic data is not transmitted during interruption, the data of the whole b-level cache module or the server is retransmitted, and the interrupted processing ensures that the b + 1-level cache module or the server receives correct and complete data;
and when the continuous transmission cannot be continued due to overlong interruption time, stopping transmitting the data to the b +1 level cache module or the server by the b level cache module until the data storage quantity of the data unit of the b level cache module which is currently written in exceeds a threshold value, starting transmitting complete data to the b +1 level cache module or the server by the data unit of the b level cache module, and adding marks on task numbers of the data transmitted before and after the interruption.
The industrial stream data coding and decoding system and method of the invention solve the problem of acquisition and arrangement of bottom industrial big data, and have the following advantages:
(1) the system and the method can acquire, encode and decode the industrial big data, are convenient for sorting the big data and have universality;
(2) according to the system and the method, the bottom layer data are stored in a cyclic alternation mode among the cache units, the characteristic of large data volume of big data is met, and a large amount of data can be acquired in a limited cache space;
(3) according to the system and the method, the cache space of the cache module adopts a time sequence storage structure, so that the accuracy of data is ensured, and the data cannot be confused;
(4) the system and the method of the invention have specific processing modes for the interruption, termination and the like of data acquisition or transmission, and ensure the integrity and the correctness of the acquired or transmitted data.
Drawings
Fig. 1 is a schematic structural diagram of an industrial big data acquisition and encoding system of the present invention.
Fig. 2 is a schematic diagram of data transmission of bottom data-first level cache-second level cache according to the present invention.
Fig. 3 is a schematic diagram of data transmission of the first-level cache, the second-level cache and the server according to the present invention.
Fig. 4 is a flow chart of server data decoding according to the present invention.
Detailed Description
The technical solution of the present invention is further described below with reference to the accompanying drawings and examples.
An industrial big data acquisition coding system, as shown in fig. 1, is a schematic structural diagram of the industrial big data acquisition coding system of the present invention, and the system includes: the first bottom layer data module a is a positive integer, a is more than or equal to 1 and is used for acquiring bottom layer data of the equipment and sending the bottom layer data to the level-b cache module; the system comprises a b-level cache module, a 1-level cache module, a superior cache module and a server, wherein b is 1 … … L, L is more than or equal to 2, the 1-level cache module is used for receiving bottom-layer data sent by an a-th bottom-layer data module, the superior cache module is respectively used for receiving data sent by adjacent inferior cache modules, the data are stored in storage spaces of the corresponding b-level cache modules, and the data received by the L-level cache module are packaged into data packets and sent to the server; the data channel module is used for transmitting the bottom layer data to the b-level cache module; and the server is used for receiving and decoding the data packet sent by the L-level cache module.
Wherein, the b-level cache module comprises: and more than two buffer units which store the bottom layer data in a cyclic alternating mode according to the writing condition of the bottom layer data.
Wherein the underlying data comprises: any one of sensing data, current data, command data and program data, which are obtained from the device and converted to digital signals or analog signals for collection.
An industrial big data acquisition and encoding method implemented by the data acquisition and encoding system is shown in fig. 2, which is a schematic data transmission diagram of bottom data-first level cache-second level cache of the invention, shown in fig. 3, which is a schematic data transmission diagram of first level cache-second level cache-server of the invention, shown in fig. 4, which is a server data decoding flow diagram of the invention, and the method comprises the following steps:
step 1': in the connection structure established by the bottom data and the first-level cache through the data channel, each bottom data corresponds to a specific data channel in a one-to-one manner, and the data channel is responsible for acquiring bottom data samples according to a program set time point and transmitting the bottom data samples to the first-level cache;
step 2': data channel sampling is program controlled: for the same device, the data channel keeps the sampling clocks of all the bottom layer data consistent, and ensures that all the bottom layer data are simultaneously acquired at any sampling time point; for the same equipment, the data channel keeps the sampling frequency of all the bottom layer data consistent, and the sampling frequency range can be 1-109Hz；
Step 3': level 1 cache (level 1 cache module) receives the bottom data that data channel gathered to convert the bottom data into basic data and write in the cache, level 1 cache divide into two parts, level 1 cache A and level 1 cache B (buffer unit), and each part all distributes a one-to-one specific buffer memory space for every basic data, and every buffer memory space all adopts the chronogenesis storage structure, and basic data an (an is positive integer, and a is greater than or equal to 4) stores in the buffer memory space according to sampling time in proper order:
step 4': when sampling starts, firstly, the level 1 cache A records sampling time N, then a data channel sequentially collects all bottom layer data corresponding to the same equipment at one time according to a program set moment, converts all the bottom layer data into basic data and simultaneously writes the basic data into a cache space corresponding to all the basic data in the level 1 cache A; once the written data amount of the level 1 cache A is equal to or larger than the set threshold value, immediately implementing data writing position switching according to program setting, enabling the data not to be written into the level 1 cache A any more, but writing each item of basic data into a cache space corresponding to each item of basic data in the level 1 cache B, and recording sampling time M; meanwhile, packing the data in the level 1 cache A into [ data line ] and transmitting the [ data line ] to the level 2 cache, and recording the output time N'; after the data in the level 1 cache A is transmitted to the level 2 cache, clearing the level 1 cache A, and setting the level 1 cache A to enter a waiting mode; once the written data amount of the level 1 cache B is equal to or larger than the set threshold value, immediately implementing data writing position switching according to program setting, enabling the data not to be written into the level 1 cache B, writing each item of basic data into a cache space corresponding to each item of basic data in the level 1 cache A, and recording sampling time K; meanwhile, packing the data in the level 1 buffer B into [ data line ] and transmitting the [ data line ] to the level 2 buffer, and recording the output time M'; after the data in the level 1 cache A is transmitted to the level 2 cache, emptying the level 1 cache B, and setting the level 1 cache B to enter a waiting mode; repeating the steps until the sampling is terminated;
step 5': the level 2 cache receives the transmission data of the level 1 cache, the level 1 cache data is written into the level 2 cache in a form of data line, the level 2 cache is divided into two parts like the level 1 cache, namely a level two cache A and a level two cache B;
specifically, when the level-2 cache receives the level-1 cache transmission data, the data is firstly written into the level-2 cache a, once the written data amount of the level-1 cache a is equal to or larger than the set threshold value, the data writing position switching is immediately implemented according to the program setting, so that the data is not written into the level-2 cache a any more, but the level-1 cache data is written into the level-2 cache B, and meanwhile, the data in the level-2 cache a is packed into [ data blocks ] and transmitted to the server; after the data in the level 2 cache A is transmitted to the server, emptying the level 2 cache A, and setting the level 2 cache A to enter a waiting mode; once the written data amount of the level 2 cache B is equal to or larger than the set threshold value, immediately implementing data writing position switching according to program setting, so that the data is not written into the level 2 cache A any more, but the level 1 cache data is written into the level 2 cache A, and meanwhile, the data in the level 2 cache B is packaged into [ data blocks ] and transmitted to a server; after the data in the level 2 cache A is completely transmitted to the level 2 cache, emptying the level 2 cache B, and setting the level 2 cache B to enter a waiting mode; so reciprocating until the transmission is terminated;
step 6': and if the data transmission from the level 1 cache to the level 2 cache is terminated, terminating the written level 2 cache A/B, and packaging the level 2 cache data for transmission to the server.
Step 7': the server allocates storage space for the data of each device, and establishes a corresponding data packet for each device, which is used for storing all the data of the device within a certain time; the data packets are named according to the related information of the corresponding devices, the second-level cache corresponding to each device transmits the data blocks into the data packets of the device, and the data packets are sorted and stored before and after the first data line sampling time of each data block.
Step 8': the data decoding method is carried out in the server, and the server acquires the analysis data by adopting the data decoding method according to the collected data of the plurality of devices, and the method comprises the following steps (see figure 4):
(1) after receiving a certain amount of data, the server performs decoding operation according to program setting, and converts the data from a data structure convenient for transmission into a data structure applicable to analysis and decision;
(2) after data decoding starts, a server firstly reads in a data packet of a device and records the data packet as a data packet Q, establishes a folder named by related information of the device and records the folder DIR-Q, declares a variable with a value being empty as QUEST, reads one data block in the data packet Q as a data block W in sequence, reads one data line in the data block W as a data line E in sequence, and judges whether a task number in the data line E is empty or not;
if the task number of the data row E is empty, the subsequent operation is not carried out, and the data row E is directly skipped over and the next data row E in the data blocks W is read in sequence;
if the task number of the data row E is not empty, judging whether the data row E is the first row in the first data block in the data packet Q, if the data row E is the first row in the first data block in the data packet Q, making a variable QUEST equal to the task number of the data row E, establishing a folder named by the task number of the data row E in the folder DIR-Q, and then establishing a file named by each basic data label in the folder DIR-Q, wherein the file is named as a basic data 1 file, a basic data 2 file … …, and then writing the sequence of the basic data in the corresponding file; if the [ data line E ] is not the first line in the first [ data block ] in the [ data packet Q ], judging whether the variable QUEST is equal to the [ data line E ] task number, if the variable QUEST is not equal to the [ data line E ] task number, establishing a folder named by the [ data line E ] current task number in the [ folder DIR-Q ] as the [ folder DIR-Q-task number, establishing a file named by each basic data label in the [ folder DIR-Q-task number as the [ basic data 1 file ], a [ basic data 2 file ] … … [ basic data n file ], and writing the data sequence of each basic data into the corresponding file; if the variable QUEST is equal to the task number of the data row E, adding the data sequence of each item of basic data in the data row E to the tail of the file corresponding to each item of basic data label in the folder DIR-Q-task number;
(3) when the basic data file is written in any process, whether all data lines under the current data blocks are read completely is judged;
if all the data lines under the current data blocks under the current data packet are not read completely, the next data line under the current data packet is read again;
if all the data lines under the data blocks under the data packets are read, judging whether all the data blocks under the data packets are read or not; if all the data blocks under the current data packet are not read completely, the next data block under the current data packet is read again; if all the data blocks under the current data packet are read, judging whether all the data packets are read or not; if all the data packets are not read completely, reading the data packets of one device again, establishing a folder named by the related information of the device, and declaring a variable QUEST with one value being empty; if all [ data packets ] have been read, the server operation ends.
Further, the method of the present invention further comprises: if three marks of < first-level cache sampling interrupt mark >, < first-level cache transmitting interrupt mark to second-level cache > and < second-level cache transmitting interrupt mark to server > are met during server data decoding, the front and rear tasks of the middle section are automatically separated, and the method is favorable for processing data in the later period.
The sampling time N and the corresponding output time N 'are grouped together, the sampling time M and the corresponding output time M' are grouped together, and so on.
The data in the above [ data row ] are arranged in the order of sampling time, output time, task number, basic data 1, and basic data 2 … …, basic data n (as shown in fig. 1); each item of basic data can be divided into a data label and a data sequence, and the data sequence of each item of basic data is data sequenced according to the sequence of sampling time.
The total storage space of the level 1 cache is the sum of the respective storage spaces of the level 1 cache A and the level 1 cache B, and the storage spaces of the level 1 cache A and the level 1 cache B are equal and equal to the integral multiple of the minimum unit for storing the server file; the threshold values of the level 1 cache a and the level 1 cache B are set to be slightly smaller than the respective storage spaces to ensure that the data is completely received without overflowing.
The software and hardware collocation should ensure that the shortest time for writing in the level 1 cache is more than or equal to the sum of the execution time of a level 1 cache program and the longest time for transmitting the level 1 cache data to the level 2 cache, the transmission rate of the bottom layer data to the level 1 cache through a data channel should be matched with the transmission rate of the level 1 cache data to the level 2 cache, and the system overall performance should be ensured to be optimal as far as possible.
In step 4', further, the method of the present invention further comprises: if sampling interruption occurs, setting the cache A/B which is being written before interruption to terminate writing when sampling is restarted, transmitting the data to the level 2 cache, and simultaneously switching to write the data into the cache B/A; calculating interruption time according to the acquired data volume and the acquisition frequency, recording the time as output time to be written in the data lines before interruption, writing sampling restart time into the sampling time after the interruption, and adding a level 1 cache sampling interruption mark to the task number of the data lines before and after the interruption.
In step 4', further, the method of the present invention further comprises: if sampling is terminated, terminating the written first-level cache A/B, transmitting the first-level cache data to a second-level cache, recording termination time as output time, and adding a < 1-level cache sampling termination mark > to a task number corresponding to the termination [ data line ].
In step 4', further, the method of the present invention further comprises: if the task switching occurs during sampling, stopping the written cache A/B, transmitting the data to the level 2 cache, and simultaneously switching to write the data into the cache B/A, wherein the task number changes [ data line ].
In step 4', further, the method of the present invention further comprises: if no task is executed during sampling, [ data line ] the task number is written to be null.
In step 4', as described above, when the level 1 buffer transfers data to the level 2 buffer, the basic unit is [ data line ]. Further, the method of the present invention further comprises: when in transmission, the sampling time data is transmitted first, then the output time data is transmitted, then the task number data is transmitted, and finally the acquired basic data are transmitted one by one according to the sequence of the basic data 1 and the basic data 2 …; when basic data are transmitted, the labels are transmitted first, and then the data of the data sequence are transmitted according to the sequence, wherein the data in the data sequence are arranged according to the time sequence.
In step 4', further, the method of the present invention further comprises: if the level 1 cache interrupts the level 2 cache when transmitting data, firstly searching the last received basic data label in the level 2 cache when restarting transmission, then comparing the label with the data in the level 1 cache, searching the data position corresponding to the label, and then implementing the following two schemes:
(1) deleting the data for the tag in the level 2 cache, starting transmission again, and continuously transmitting the tag and the following data to the level 2 cache;
(2) continuously finding out an interruption point of the data under the data label in the level 1 cache, then starting transmission again, and continuously transmitting the data after the interruption point to the level 2 cache; if the data transmitted during the interruption is non-basic data, the data of the whole level 1 cache is retransmitted, and the interrupted processing should ensure that the [ data line ] received by the level 2 cache is correct and complete.
In step 4', further, the method of the present invention further comprises: if the continuous transmission from the level 1 cache to the level 2 cache can not be continued any more due to the overlong interruption time, the system throws that the level 1 cache transmits interruption exception to the level 2 cache, and simultaneously stops the transmission of data from the level 1 cache to the level 2 cache until the storage amount of the A/B data of the level 1 cache currently being written exceeds a threshold value, the transmission of the A/B data of the level 1 cache to the level 2 cache is started again, and < the level 1 cache transmits interruption marks to the level 2 cache is added to the task number of the data line transmitted before and after interruption.
The total storage space of the level 2 cache is the sum of the respective storage spaces of the level 2 cache A and the level 2 cache B, and the storage spaces of the level 2 cache A and the level 2 cache B are equal and equal to the integral multiple of the minimum unit for storing the server file; the threshold values of the level 2 cache a and the level 2 cache B are set to be slightly smaller than the respective storage spaces to ensure that the data is completely received without overflowing.
By means of software and hardware matching, it is guaranteed that the shortest time for writing into the level 2 cache is greater than or equal to the sum of the execution time of the level 2 cache program and the longest time for transmitting the level 2 cache data to the level 2 cache, the transmission rate of the level 1 cache data to the level 2 cache is matched with the transmission rate of the level 2 cache data to the server, and the system overall performance is guaranteed to be optimal as far as possible.
In step 5', further, the method of the present invention further comprises: the data in the data block is the data lines sequenced according to the sampling time, when the secondary cache is ready to be transmitted to the server, a program is operated to judge the sampling time and the output time of the adjacent data lines, if the output time of the previous data line is not equal to the sampling time of the next data line, the discontinuous exception of the data line is thrown out, and the exception position is recorded; in order to facilitate the numbering of the data blocks, each data block is provided with a label corresponding to the data block, and the label is a characteristic value calculated by the sampling time of the first data line in the data blocks.
In step 5', when the second-level cache transmits data to the server, the basic unit is [ data block ]. In step 5', further, the method of the present invention further comprises: when the data blocks are transmitted, the data blocks are sequentially transmitted one by one according to the sampling time sequence of the data lines in the data blocks; the transmission of the data row firstly transmits sampling time data, then transmits output time data, then transmits task number data, and finally transmits the acquired basic data one by one according to the sequence of the basic data 1 and the basic data 2 …; when basic data are transmitted, the labels are transmitted first, and then the data of the data sequence are transmitted according to the sequence, wherein the data in the data sequence are arranged according to the time sequence.
In step 5', further, the method of the present invention further comprises: if the data transmission from the level 2 cache to the server is interrupted, when the transmission is restarted, firstly, the server searches for the sampling time of the last received [ data line ] and the last basic data label of the [ data line ], then compares the label with the data in the level 2 cache, finds out the [ data line ] corresponding to the sampling time and the basic data of the last transmission of the data line, and then implements the following three schemes (the last received [ data line ] is denoted as [ data line X ], and the last transmitted basic data of [ data line ] is denoted as [ basic data Y "):
(1) deleting the data line X in the server, starting transmission again, and continuously transmitting the data line X and the subsequent data lines to the server in sequence;
(2) deleting the basic data Y in the data row X in the server, starting transmission again, and continuously transmitting the basic data Y in the data row X and the subsequent basic data in the data row X and the data row X in the subsequent data row to the server in sequence;
(3) continuously finding out the break point of the data below the basic data Y in the data line X in the level 2 cache, then starting transmission again, and continuously transmitting the data after the break point of the basic data Y, the basic data after the basic data Y in the data line X and the data line after the data line X to the server according to the sequence; the handling of the interrupt should ensure that the [ data blocks ] received by the server are correct and complete.
In step 5', further, the method of the present invention further comprises: if the level 2 cache cannot continuously transmit to the server any more due to the overlong interruption time, the system throws that the level 2 cache transmits interruption exception to the server, simultaneously stops the transmission of data from the level 2 cache to the server, starts the transmission of the level 2 cache A/B to the server completely [ data blocks ] again until the storage amount of the level 2 cache A/B data currently being written exceeds a threshold value, and adds a < level 2 cache transmits an interruption mark to the server on a [ data block ] label transmitted before and after interruption.
In summary, the system and the method for encoding and decoding industrial stream data of the present invention can acquire and process industrial big data, have universality, can satisfy the characteristic that the data volume of big data is large, can acquire a large amount of data in a limited buffer space, and ensure the integrity and correctness of the acquired or transmitted data.
While the present invention has been described in detail with reference to the preferred embodiments, it should be understood that the above description should not be taken as limiting the invention. Various modifications and alterations to this invention will become apparent to those skilled in the art upon reading the foregoing description. Accordingly, the scope of the invention should be determined from the following claims.
Claims (9)
1. An industrial stream data encoding and decoding system, the system comprising:
the first bottom layer data module a is a positive integer, a is more than or equal to 1 and is used for acquiring bottom layer data of equipment and sending the bottom layer data to the level-b cache module;
the b-level cache module, b is 1 … … L, L is greater than or equal to 2, wherein the 1-level cache module is used for receiving the bottom data sent by the a-th bottom data module, the upper-level cache module is respectively used for receiving the data sent by the adjacent lower-level cache module, the data are all stored in the storage space of the corresponding b-level cache module, and the data received by the L-level cache module are packaged into data packets and sent to the server;
the data channel module is used for transmitting the bottom layer data to the b-level cache module; and
the server is used for receiving and decoding the data packet sent by the L-level cache module;
the b-level cache module comprises: more than two buffer units, according to the write-in condition of the bottom layer data, the buffer units store the bottom layer data in a cyclic alternating mode; in the level 1 cache module, converting the bottom layer data into basic data and storing the basic data in a storage space of the level 1 cache module; the basic data are sequentially stored in the storage space of the b-level cache module according to the acquisition time; the storage space of the b-level cache module is the sum of the storage spaces of all the cache units contained in the b-level cache module;
the basic data comprises: a data sequence, and a data tag corresponding thereto; the data label is a characteristic value calculated by sampling time of each basic data; the data sequence is bottom layer data arranged in sequence according to sampling time; when data transmission is carried out, the data tags are transmitted first, and then basic data are transmitted;
the coding and decoding method adopted by the system comprises the following steps: when sampling interruption occurs in the b-level cache module, setting a cache unit which is written before interruption to terminate writing when sampling is restarted, transmitting the data to an adjacent upper-level cache module, and simultaneously performing position switching to write the data into another cache unit; calculating interruption time according to the acquired data volume and the acquisition frequency, recording the interruption time as output time, writing the output time into data before interruption, writing sampling time of sampling restart into the data after interruption, and adding marks on task numbers of the data before and after the interruption; when sampling is terminated, terminating the 1-level cache module which is being written in, transmitting the cache data upwards step by step until the L-level cache module, recording termination time as output time, and adding a mark on a task number for terminating the corresponding data; when the task switching occurs in the sampling, stopping the written cache unit, transmitting the data to the upper-level cache module, and simultaneously switching the cache unit and writing the data;
when the b-level cache module is interrupted when transmitting data to the b + 1-level cache module or the server, firstly searching the last received data tag in the b + 1-level cache module or the server when restarting transmission, then comparing the data tag with the data in the b-level cache module, searching the data position corresponding to the data tag, and performing the following operations:
(1) deleting the data of the transmission label in the b +1 level cache module or the server, starting transmission again, and continuously transmitting the data label and the subsequent data to the b +1 level cache module or the server; or
(2) Finding out an interruption point of the basic data corresponding to the data tag in the b-level cache module or the server, then starting transmission again, and continuously transmitting the basic data after the interruption point to the b + 1-level cache module or the server; when the basic data is not transmitted during interruption, the data of the whole b-level cache module or the server is retransmitted, and the interrupted processing ensures that the b + 1-level cache module or the server receives correct and complete data;
and when the continuous transmission cannot be continued due to overlong interruption time, stopping transmitting the data to the b +1 level cache module or the server by the b level cache module until the data storage quantity of the data unit of the b level cache module which is currently written in exceeds a threshold value, starting transmitting complete data to the b +1 level cache module or the server by the data unit of the b level cache module, and adding marks on task numbers of the data transmitted before and after the interruption.
2. The system for encoding and decoding industrial stream data according to claim 1, wherein the underlying data comprises: any one of sensing data, current data, instruction data, and program data.
3. The system of claim 1, wherein the storage space of each buffer unit in the b-level buffer module is equal to an integer multiple of the minimum unit of file storage of the server, and the threshold of each buffer unit is smaller than its respective storage space.
4. The system for encoding and decoding industrial stream data according to any one of claims 1 to 3, wherein the shortest time written by the b-level cache module is equal to or longer than the sum of the program execution time of the b-level cache module and the longest time transmitted from the b-level cache module to the b + 1-level cache module;
the transmission rate of the bottom layer data to the level 1 cache module is matched with the transmission rate of the level b cache module to the level b +1 cache module.
5. An industrial stream data encoding and decoding method, which uses the industrial stream data encoding and decoding system according to any one of claims 1 to 4, the method comprising:
step 1: the bottom layer data is connected with the level b cache module through a corresponding data channel, and the data channel is responsible for collecting bottom layer data samples according to a program set time point and transmitting the bottom layer data samples to the level 1 cache module;
step 2: the data channel sampling is controlled by a program: for the same device, the data channel keeps the sampling clocks of all the bottom layer data consistent, and ensures that all the bottom layer data are simultaneously acquired at any sampling time point;
and step 3: the level 1 cache module receives the bottom data acquired by the data channel, converts the bottom data into basic data and writes the basic data into a cache, and each item of basic data is allocated with a corresponding cache space;
and 4, step 4: when sampling starts, the data channel sequentially collects all bottom layer data corresponding to the same equipment according to a program set moment, converts each bottom layer data into basic data, writes the basic data into a cache space corresponding to each item of basic data in a cache unit of the level 1 cache module, judges whether the data is continuously written according to a threshold value of the cache space, stops writing if the data is written into the cache space to be larger than the threshold value, packs and transmits the basic data to the level b +1 cache module, clears the cache unit, switches the position to switch to another cache unit to continuously write, and the cache unit enters a waiting mode; repeating the step 4 until the sampling is finished;
and 5: the b +1 level cache module receives and writes basic data transmitted by the b level cache module into a cache unit of the b +1 level cache module, judges whether data are continuously written according to a threshold value of a cache space, if the data are written to be larger than the threshold value, the writing is stopped, the transmitted data are packaged and transmitted to the b +2 level cache module or a server, the cache unit is emptied and the position is switched to be switched to another cache unit for continuous writing, and the cache unit enters a waiting mode; repeating the step 5 until the L-level cache module transmits the data to the server;
step 6: the server allocates storage space for the data of each device, establishes a corresponding data packet for each device, and is used for storing all the data of the device in a certain time;
and 7: the server runs a data decoding method to obtain analysis data;
the buffer memory space of the buffer memory module adopts a time sequence storage structure.
6. The method as claimed in claim 5, wherein in the steps 4 and 5, the buffer module records sampling and output time respectively during sampling and transmission, and sorts the basic data according to the sampling and output time during time-series storage; the data sequentially transmitted from the b-level cache module to the b + 1-level cache module comprises: sampling time data, output time data, task number data and basic data; and the basic data are transmitted according to the sequencing order.
7. The method as claimed in claim 6, wherein the data channel is for all the underlying data in the same deviceThe sampling frequency is kept consistent, and the range of the sampling frequency is 1-109Hz。
8. The industrial stream data encoding and decoding method according to claim 7, wherein when L is 2, the level 1 cache module stores and sends the basic data to the level 2 cache module in the form of data lines, the level 2 cache module receives and stores the data lines in the form of data blocks and sends the data lines to the server, and the server receives and stores the data lines in the form of data packets;
in step 6, the data packet sequences and stores the data blocks in the data packet according to the sampling time of the first data line of each data block.
9. The method as claimed in any one of claims 5 to 8, wherein in step 7, the server performs a decoding operation according to a set program to convert data from a data structure convenient for transmission to a data structure applied to analysis decision, the data decoding method comprising:
and 8: after the data decoding starts, the server reads in a data packet of the equipment, establishes a related folder, and declares a variable with a null value as represented by a request;
and step 9: sequentially reading the data in the data packet according to the storage sequence, and judging whether the task number of the data is empty, wherein the judging process is as follows:
(1) when the task number of the data is empty, the subsequent operation is not carried out, the next data is skipped to, and the judgment is repeated;
(2) when the task number of the data is not empty, the position of the data in the data packet is judged, and the judging process is as follows:
(3) when the data is the first data, making the variable QUEST equal to the task number of the data, establishing a corresponding task folder in the folder, establishing a basic data folder corresponding to the basic data in the task folder, and writing the corresponding basic data;
(4) when the data is not the first data, judging whether the variable QUEST is equal to the task number of the data;
when the variable QUEST is not equal to the task number of the data, establishing a corresponding task folder in the folder, establishing a basic data folder corresponding to the basic data in the task folder, and writing the corresponding basic data;
when the variable QUEST is equal to the task number of the data, writing corresponding basic data into the corresponding basic data folder;
step 10: when the writing operation of any process to the basic data is completed, judging whether all data in the current data packet is completely read; and when the data is not read completely, repeating the steps 8 and 9 until all the data are read completely, and finishing the operation of the server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810457697.3A CN108551490B (en) | 2018-05-14 | 2018-05-14 | Industrial stream data coding and decoding system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810457697.3A CN108551490B (en) | 2018-05-14 | 2018-05-14 | Industrial stream data coding and decoding system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108551490A CN108551490A (en) | 2018-09-18 |
| CN108551490B true CN108551490B (en) | 2021-06-18 |
Family
ID=63494770
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810457697.3A Active CN108551490B (en) | 2018-05-14 | 2018-05-14 | Industrial stream data coding and decoding system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108551490B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109379437B (en) * | 2018-11-21 | 2022-02-01 | 苏州金螳螂文化发展股份有限公司 | Speed processing method and system based on data caching |
| CN109522140B (en) * | 2018-11-23 | 2023-10-20 | 江西洪都航空工业集团有限责任公司 | Method for receiving data by communication board card |
| CN111385332A (en) * | 2018-12-29 | 2020-07-07 | 顺丰科技有限公司 | Internet of things equipment, Internet of things platform access method and equipment |
| CN109889456B (en) * | 2018-12-29 | 2023-09-05 | 厦门海为科技有限公司 | Data transmission method, device, equipment, system and storage medium |
| CN112002115B (en) * | 2020-08-05 | 2021-04-23 | 中车工业研究院有限公司 | Data acquisition method and data processor |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1852318A (en) * | 2006-04-19 | 2006-10-25 | 华中科技大学 | Distributed multi-stage buffer storage system suitable for object network storage |
| CN101937475A (en) * | 2010-08-27 | 2011-01-05 | 中国电子科技集团公司第四十一研究所 | Multi-rate multi-channel data acquisition method |
| CN102497450A (en) * | 2011-12-28 | 2012-06-13 | 北京华电天仁电力控制技术有限公司 | Two-stage-system-based distributed data compression processing method |
| CN103034538A (en) * | 2012-12-24 | 2013-04-10 | 珠海拓普智能电气股份有限公司 | Multi-level cache processing method of drive program in embedded type operation system |
| CN104699626A (en) * | 2013-12-10 | 2015-06-10 | 中兴通讯股份有限公司 | Terminal internal memory processing method, device and terminal |
| CN104935680A (en) * | 2015-06-18 | 2015-09-23 | 中国互联网络信息中心 | Recursive domain name service system and method of multi-level shared cache |
| CN105979218A (en) * | 2016-06-13 | 2016-09-28 | 中南大学 | High-definition real-time monitoring video efficient transmission method |
| CN106155936A (en) * | 2015-04-01 | 2016-11-23 | 华为技术有限公司 | A kind of buffer replacing method and relevant apparatus |
| CN106200642A (en) * | 2016-08-05 | 2016-12-07 | 上海电力学院 | The two-wheeled balance car of a kind of realizing route automatic Memory and reproduction independently goes on patrol method |
| CN107273522A (en) * | 2015-06-01 | 2017-10-20 | 明算科技（北京）股份有限公司 | Towards the data-storage system and data calling method applied more |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101218828B1 (en) * | 2009-07-02 | 2013-01-04 | (주)에임투지 | Cooperative cache method and contents providing method and using request proportion apparatus |
-
2018
- 2018-05-14 CN CN201810457697.3A patent/CN108551490B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1852318A (en) * | 2006-04-19 | 2006-10-25 | 华中科技大学 | Distributed multi-stage buffer storage system suitable for object network storage |
| CN101937475A (en) * | 2010-08-27 | 2011-01-05 | 中国电子科技集团公司第四十一研究所 | Multi-rate multi-channel data acquisition method |
| CN102497450A (en) * | 2011-12-28 | 2012-06-13 | 北京华电天仁电力控制技术有限公司 | Two-stage-system-based distributed data compression processing method |
| CN103034538A (en) * | 2012-12-24 | 2013-04-10 | 珠海拓普智能电气股份有限公司 | Multi-level cache processing method of drive program in embedded type operation system |
| CN104699626A (en) * | 2013-12-10 | 2015-06-10 | 中兴通讯股份有限公司 | Terminal internal memory processing method, device and terminal |
| CN106155936A (en) * | 2015-04-01 | 2016-11-23 | 华为技术有限公司 | A kind of buffer replacing method and relevant apparatus |
| CN107273522A (en) * | 2015-06-01 | 2017-10-20 | 明算科技（北京）股份有限公司 | Towards the data-storage system and data calling method applied more |
| CN104935680A (en) * | 2015-06-18 | 2015-09-23 | 中国互联网络信息中心 | Recursive domain name service system and method of multi-level shared cache |
| CN105979218A (en) * | 2016-06-13 | 2016-09-28 | 中南大学 | High-definition real-time monitoring video efficient transmission method |
| CN106200642A (en) * | 2016-08-05 | 2016-12-07 | 上海电力学院 | The two-wheeled balance car of a kind of realizing route automatic Memory and reproduction independently goes on patrol method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108551490A (en) | 2018-09-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108551490B (en) | Industrial stream data coding and decoding system and method | |
| CN102541640B (en) | Cluster GPU (graphic processing unit) resource scheduling system and method | |
| CN101291546A (en) | Switching structure coprocessor of core router | |
| KR101867286B1 (en) | Distributed processing apparatus and method for big data using hardware acceleration based on work load | |
| CN1316074A (en) | Interrupt/software-controlled thread processing | |
| CN103049240A (en) | Peripheral component interconnect express (PCI-E) device and data receiving reordering method thereof | |
| CN111459052B (en) | Multifunctional vehicle bus simulation detection system and method | |
| CN114330722B (en) | Inference realization method, network, electronic device and storage medium | |
| CN102622323A (en) | Data transmission management method based on switch matrix in dynamic configurable serial bus | |
| US7840725B2 (en) | Capture of data in a computer network | |
| CN110929885A (en) | Smart campus-oriented distributed machine learning model parameter aggregation method | |
| CN101594201A (en) | The method of integrally filtering error data in linked queue management structure | |
| CN116614492B (en) | Laboratory unified data acquisition platform based on various data sources of internet of things | |
| CN114124643B (en) | PaaS-based network equipment flow collection method and device | |
| CN111628779A (en) | Parallel compression and decompression method and system for FASTQ file | |
| CN100356309C (en) | Method and system for efficient command performance of multiple units sharing the same data interface port | |
| CN113553279B (en) | RDMA communication acceleration set communication method and system | |
| CN114970428A (en) | Verification system and method for Flexray bus controller in SoC | |
| CN113992609A (en) | Method and system for processing multilink service data disorder | |
| CN107040601A (en) | A kind of collecting method and system | |
| CN114390117A (en) | High-speed continuous data stream storage processing device and method based on FPGA | |
| CN101282235B (en) | Method, apparatus and system for processing data | |
| CN109189746A (en) | Implementation method, device, equipment and the storage medium of general streaming Shuffle engine | |
| CN110247666A (en) | A kind of system and method for hardware concurrent compression | |
| CN104581772B (en) | A kind of fault-tolerance approach and network node of the message transmissions of network node |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |