CN104735142B - A kind of processing in real time of big data and storage system and its processing method - Google Patents
A kind of processing in real time of big data and storage system and its processing method Download PDFInfo
- Publication number
- CN104735142B CN104735142B CN201510116906.4A CN201510116906A CN104735142B CN 104735142 B CN104735142 B CN 104735142B CN 201510116906 A CN201510116906 A CN 201510116906A CN 104735142 B CN104735142 B CN 104735142B
- Authority
- CN
- China
- Prior art keywords
- data
- frame
- module
- sent
- central control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Arrangements For Transmission Of Measured Signals (AREA)
Abstract
The present invention relates to a kind of processing in real time of big data and storage system and its processing methods, the Central Control Module of the system realizes data dispatch, thread control and Process flowchart, it can coordinate process flow between each module, it is achieved thereby that the quick of test data, reliable treatments and storage, effectively shorten the testing time, improve testing efficiency；And the present invention is worked together using real-time data base and relational database, realizes high speed and mass data storage；In the present invention, data processed result is packaged and Web Publishing by data publication module according to the communication protocol of setting so that each site can show corresponding data result by monitoring platform, realize real-time monitoring of each site to data processed result；Therefore, which can effectively improve the utilization rate of data, the test effect for promoting model, solve the technical barrier of instantaneous big data processing and storage.
Description
Technical field
Measuring signal acquisition and processing technology field the present invention relates to aerospace craft, more particularly to a kind of big data are real
When processing with storage system and its processing method.
Background technology
With space militarization and commercialized fast development, outer space has become current and has safeguarded national security and country
The strategic high ground that interests must be paid close attention to and occupy is mainly manifested in and is disposed with fast railway, is new for the space defence of representative
It challenges and is fitted with two generations navigation the second stage of the project, moon exploration project, international business's transmitting etc. for the carrier rocket task of representative
Ying Xing, the transmitting new demand of more stars.
The successful application of a new generation of China space vehicles device needs to break through to exist including remote high symbol measure systems technology
Interior multinomial breakthrough space transoportation and Space Attack weapon key technology.In order to break through remote high symbol measure systems technology, type
Number require measuring system that can complete real-time processing and storage to whole rocket data in real time, to improve the utilization rate of data, be promoted
The testing efficiency of model.The demand causes measuring system to face following problem：
Whole rocket data type and data volume are significantly increased compared with traditional model, traditional model need to only complete it is gradual, refer to
It makes, the processing of computer word class parameter, is not yet related to for the real-time processing of the data such as 1553B, CAN, RS485, RS422.Cause
How this, design one and can adapt to whole rocket generating date, improve data user rate, meet model to data processing knot
The system that fruit quickly provides demand is a great problem that measuring system faces.
In order to meet the needs of model rapid data interpretation, compare, the testing efficiency of model is improved, measuring system need to will be located
It manages data result and carries out real-time storage, it is ensured that model experiment completes that data interpretation can be carried out.Due to the limitation of computer hardware
(disk read-write speed, system I/O expenses, CPU execution efficiencys etc.) is so that the real-time storage of instantaneous big data becomes measuring system
The huge difficult problem faced.
The content of the invention
It is an object of the invention to overcome the deficiencies of the prior art and provide a kind of big data in real time processing and storage system,
The system realizes the scheduling of data frame between modules using Central Control Module, according to the flow of setting respectively to data frame
Correctness, continuity, frame structure are judged, are extracted departure time, and are sent at data in satisfactory data frame
Reason module is handled, and finally preserves obtained handling result in the database, it is achieved thereby that the real-time processing of data and
Subsequent storage.
The above-mentioned purpose of the present invention is achieved by following technical solution：
A kind of big data handles in real time and storage system, including data providing module, Central Control Module, telemetry frame parsing
Module becomes frame judgment module, the judgment module that takes off, data processing module and database, wherein：
Data providing module：The telemetering true form data frame of external network multicast is received, then deposits the true form data frame
Enter in circular buffer area, and the data frame in the circular buffer area is sent to Central Control Module；
Central Control Module：The data frame that data providing module is sent is received, and the data frame is sent to telemetry frame
Parsing module；The data frame judging result that telemetry frame parsing module is sent is received, if it is judged that frame of display data is correct, then
The data frame is sent to change frame judgment module；Miscue or the frame structure mark for becoming that frame judgment module is sent are received, if
It receives and becomes the frame structure mark that frame judgment module is sent, then send the data frame and frame structure mark judges mould to taking off
Block；After the departure time value for judgment module transmission of taking off is received, the data frame and departure time value are sent at data
Manage module；
Telemetry frame parsing module：The data frame that Central Control Module is sent is received, the size of the data frame and frame is same
Code is walked compared with the frame sign value and the frame swynchronization code of setting that set, that is, judges the correctness of the data frame, and passes through
The count value of the data frame judges the continuity of data frame, then sends judging result to Central Control Module；It is if described
Judging result is incorrect or discontinuous for data frame, then the judging result is recorded in syslog file；
Become frame judgment module：The data frame that Central Control Module is sent is received, data frame mark is judged, such as
Data frame mark described in fruit in the frame mark value range of setting, does not then send miscue to Central Control Module, and will
The error logging is in syslog file；If the data frame mark is correct, according to data frame sign and data frame
Mark determines the frame structure used when handling the data frame, and the mark for sending the frame structure controls mould to center
Block；
Take off judgment module：Data frame and the frame structure mark that Central Control Module is sent are received, according to the frame structure
It indicates corresponding frame structure to parse data frame, the departure time of record in a data frame is then searched in analysis result
Value, and will be worth departure time and be sent to Central Control Module；
Data processing module：Data frame and the departure time value that Central Control Module is sent are received, then according to data frame
Data type handled, extract the source code in the data frame, and the source code and the systematic parameter of setting carried out
It is corresponding, source code is handled using the systematic parameter corresponding processing calculation formula, obtains the physics of the systematic parameter
Magnitude；The systematic parameter, processing calculation formula and handling result are sent to database and preserved；
Database：Receive the handling result that data processing module is sent and preservation.
In above-mentioned big data in real time processing and storage system, database includes real-time data base and relational database,
Wherein, the handling result that real-time data base real-time reception data processing module is sent, and the handling result is forwarded to relation
Database is preserved.
In above-mentioned big data in real time processing and storage system, data processing module is carrying out data processing and was storing
Data processing progress and data degree of being stored in are sent in journey to Central Control Module, Central Control Module receives the progress and goes forward side by side
Row display.
Above-mentioned big data handles in real time and storage system, further includes data publication module, wherein：Data processing module root
According to the data publication content of setting, corresponding data processed result is sent to Central Control Module, Central Control Module forwarding institute
Data processed result is stated to data publication module, data publication module is according to the communication protocol of setting by the data processed result
It is packaged and Web Publishing.
Above-mentioned big data processing and storage system in real time, further include data center, wherein, it preserves in the database and is
System setting value, the default value include frame sign value, frame swynchronization code, frame mark value range, frame structure and systematic parameter
Calculation formula；Data center reads the default value from database and is sent to Central Control Module；Center controls
Module sends the frame sign value set and frame swynchronization code to telemetry frame parsing module, sends frame mark value range and judges to frame is become
Module sends frame structure to the judgment module that takes off, the calculation formula of system parameters to data processing module.
In above-mentioned big data in real time processing and storage system, data processing module is divided into according to processing data type
8 processing units, including slow varying parameter processing unit, order parameter processing unit, computer word parameter processing unit, 1553B
Bus parameter unit, condition code multiplexing parameters processing unit, CAN bus parameter processing unit, secondary calculating parameter processing unit
With serial ports parameter processing unit, each processing unit receives the data frame that Central Control Module is sent, respectively to 8 types
Data are handled.
In above-mentioned big data in real time processing and storage system, Central Control Module is to 8 in data processing module
Processing unit carries out thread control, is the thread of each processing unit distribution setting quantity, each unit is in the distribution line
Data processing is carried out in journey.
In above-mentioned big data in real time processing and storage system, Central Control Module controls system process, i.e.,
In system operation, if processing occurs being again started up asking, reported an error and the request is forbidden.
Based on the above-mentioned big data data processing method that processing is realized with storage system in real time, include the following steps：
(1), data providing module receives the telemetering true form data frame of external network multicast, then by the true form data frame
It is stored in circular buffer area, and the data frame in the circular buffer area is sent to Central Control Module；
(2), the data frame received is sent to telemetry frame parsing module by Central Control Module；Telemetry frame parsing module will
The size and frame swynchronization code of the data frame are compared with the frame sign value and the frame swynchronization code of setting that set, i.e., described in judgement
The correctness of data frame, and judge by the count value of the data frame continuity of data frame, then send judging result to
Central Control Module；If the judging result is incorrect or discontinuous for data frame, the judging result is recorded in it is
In system journal file；
(3), Central Control Module receives the judging result that telemetry frame parsing module is sent；If the judging result is number
It is incorrect or discontinuous according to frame, then it enters step (8)；If the judging result is correct and continuous for data frame, into step
Suddenly (4)；
(4), Central Control Module sends data frame to frame judgment module is become, and becomes frame judgment module and the data frame is indicated
Judged, if data frame mark sends miscue and controlled to center not in the frame mark value range of setting
Molding block, and by the error logging in syslog file；If the data frame mark is correct, big according to data frame
Small and data frame mark determines the frame structure used when handling the data frame, and send the mark of the frame structure to
Central Control Module；
(5), Central Control Module receives miscue or the frame structure mark for becoming that frame judgment module is sent；If in receiving
Hold for miscue, then enter step (8)；If reception content indicates for frame structure, enter step (6)；
(6), Central Control Module sends data frame and frame structure mark to the judgment module that takes off, take off judgment module according to
The frame structure indicates corresponding frame structure and data frame is parsed, and searches and recorded in a data frame in analysis result
Departure time is worth, and will be worth departure time and be sent to data processing module with data frame；
(7), data processing module is handled according to the data type of data frame, extracts the source in the data frame
Code, and by the source code with setting systematic parameter carry out it is corresponding, using the systematic parameter it is corresponding handle calculation formula pair
Source code is handled, and obtains the magnitude of physical quantity of the systematic parameter；By the systematic parameter, processing calculation formula and handling result
Database is sent to be preserved.
(8), terminate.
The present invention has the advantages that compared with prior art：
(1), the present invention realizes data dispatch, thread control and Process flowchart by Central Control Module, can coordinate
Process flow between each module, it is achieved thereby that the quick of test data, reliable treatments and storage, effectively shorten the testing time,
Improve testing efficiency；
(2), the present invention is worked together using real-time data base and relational database, wherein, the processing speed of real-time data base
Degree is fast but memory capacity is small, and the processing speed of relational database is slow but memory capacity is big, can be real using this database scheme
Now high speed and mass data storage；
(3), in the present invention, data publication module data processed result be packaged according to the communication protocol of setting and
Web Publishing so that each site can show corresponding data result by monitoring platform, realize each site to data processing knot
The real-time monitoring of fruit；
(4), parallel processing in the present invention, is carried out to the data of each data type using 8 data processing units, it can be with
Effectively improve the efficiency and reliability of system data processing.
Description of the drawings
Fig. 1 is the big data processing in real time of the present invention and the composition frame chart of storage system；
Fig. 2 is the big data processing in real time of the present invention and the flow chart of storage method.
Specific embodiment
The present invention is described in further detail in the following with reference to the drawings and specific embodiments：
The present invention provides a kind of real-time processing of big data and storage systems, can effectively improve the utilization rate of data, carry
The test effect of model is risen, solves the technical barrier of instantaneous big data processing and storage.The block diagram of system as shown in Figure 1, this
The processing in real time of the big data of invention includes data providing module, Central Control Module, telemetry frame parsing module, change with storage system
Frame judgment module, the judgment module that takes off, data processing module, data publication module, data center and database.
(1), data providing module
Data providing module receives the telemetering true form data frame of external network multicast, is then stored in the true form data frame
In circular buffer area, and the data frame in the circular buffer area is sent to Central Control Module.
(2), Central Control Module
Central Control Module mainly completes thread control, Memory control, data dispatch and management of process in system operation.
During online process control, Central Control Module is that each processing unit in data processing module distributes thread, is allowed
Each unit realizes parallel processing in the thread of distribution.
In Memory control, Central Control Module sets read pointer and is read in the circular buffer area of data providing module
Corresponding data realize the data sending of each intermodule.
During data dispatch, Central Control Module receives the data frame that data providing module is sent, and by the number
Telemetry frame parsing module is sent to according to frame；The data frame judging result that telemetry frame parsing module is sent is received, if it is judged that
Frame of display data is correct, then sends the data frame to change frame judgment module；It receives and becomes the miscue that frame judgment module is sent
Or frame structure mark, become the frame structure mark that frame judgment module is sent if received, send the data frame and frame structure
Indicate the judgment module that takes off；After the departure time value for judgment module transmission of taking off is received, send the data frame and rise
Fly moment value to data processing module.Central Control Module in data processing, receives the number that data processing module is sent
It is shown according to processing progress and data degree of being stored in, and to the progress, working condition of the staff to system can be facilitated
It is monitored in real time..
In management of process, Central Control Module controls system process, i.e., in system operation, if place
Reason occurs being again started up asking, then is reported an error and the request is forbidden, system is avoided repeatedly to start, so as to avoid
The error in data occurred due to the multi-pass operation stored to data.
And central control center can coordinate data publication module to realize data publication, wherein, Central Control Module connects
The data processed result that data processing module is sent is received, and is transmitted to data publication module.
(3), telemetry frame parsing module
Telemetry frame parsing module receives the data frame that Central Control Module is sent, by the size and frame synchronization of the data frame
Code with setting frame sign value and set frame swynchronization code compared with, if comparative result be it is identical if judge the data frame
It is correct, and judge by the count value of the data frame continuity of data frame, i.e., if the count value continuously if show to count
It is continuous according to frame；Then judging result is sent to Central Control Module；If the judging result is incorrect or do not connect for data frame
It is continuous, then the judging result is recorded in syslog file.
(4), frame judgment module is become
Become frame judgment module and receive the data frame that Central Control Module is sent, data frame mark is judged, such as
Data frame mark described in fruit in the frame mark value range of setting, does not then send miscue to Central Control Module, and will
The error logging is in syslog file；If the data frame mark is correct, according to data frame sign and data frame
Mark determines the frame structure used when handling the data frame, and the mark for sending the frame structure controls mould to center
Block.
(5), take off judgment module
The judgment module that takes off receives data frame and the frame structure mark that Central Control Module is sent, according to the frame structure mark
Show that corresponding frame structure parses data frame, and the departure time value of record in a data frame searched in analysis result,
And it will be worth departure time and be sent to Central Control Module.
(6), data processing module
Data processing module receives data frame and the departure time value that Central Control Module is sent, then according to data frame
Data type is handled, and extracts the source code in the data frame, and the systematic parameter of the source code and setting is carried out pair
Should, source code is handled using the systematic parameter corresponding processing calculation formula, obtains the physical quantity of the systematic parameter
Value；Handling result is sent to database to preserve.
In the present invention, data processing module is divided into 8 processing units according to processing data type, including slow varying parameter
Processing unit, order parameter processing unit, computer word parameter processing unit, 1553B bus parameters unit, condition code multiplexing ginseng
Number processing unit, CAN bus parameter processing unit, secondary calculating parameter processing unit and serial ports parameter processing unit, it is described each
Processing unit receives the data frame that Central Control Module is sent, and the data of 8 types are handled respectively.
(7), database
Database includes real-time data base and relational database in the present invention, wherein, real-time data base real-time reception number
According to the handling result that processing module is sent, and the handling result is forwarded to relational database and is preserved.Wherein, count in real time
According to the processing speed in storehouse, fast but memory capacity is small, and the processing speed of relational database is slow but memory capacity is big, using this data
Storehouse scheme can realize high speed and mass data storage.
(8), data publication module
In order to realize the real-time release of data, system of the invention further includes data publication module.In data issuing process
In：Data processing module sends corresponding data processed result to Central Control Module according to the data publication content of setting, in
Heart control module forwards the data processed result, and to data publication module, data publication module will according to the communication protocol of setting
The data processed result is packaged and Web Publishing.Each site can show corresponding data result by monitoring platform,
Realize real-time monitoring of each site to data processed result.
(9), data center
The present invention preserves default value in the database, including frame sign value, frame swynchronization code, frame mark value model
It encloses, the calculation formula of frame structure and systematic parameter；During data center reads the default value and is sent to from database
Heart control module；Central Control Module sends the frame sign value set and frame swynchronization code to telemetry frame parsing module, sends frame mark
Show that value range to frame judgment module is become, sends frame structure to the judgment module that takes off, the calculation formula of system parameters is to counting
According to processing module.
According to method process chart shown in Fig. 2, big data using the present invention processing in real time is carried out with storage system
Data processing method includes the following steps：
(1), data providing module receives the telemetering true form data frame of external network multicast, then by the true form data frame
It is stored in circular buffer area, and the data frame in the circular buffer area is sent to Central Control Module；
(2), the data frame received is sent to telemetry frame parsing module by Central Control Module；Telemetry frame parsing module will
The size and frame swynchronization code of the data frame are compared with the frame sign value and the frame swynchronization code of setting that set, i.e., described in judgement
The correctness of data frame, and judge by the count value of the data frame continuity of data frame, then send judging result to
Central Control Module；If the judging result is incorrect or discontinuous for data frame, the judging result is recorded in it is
In system journal file；
(3), Central Control Module receives the judging result that telemetry frame parsing module is sent；If the judging result is number
It is incorrect or discontinuous according to frame, then it enters step (8)；If the judging result is correct and continuous for data frame, into step
Suddenly (4)；
(4), Central Control Module sends data frame to frame judgment module is become, and becomes frame judgment module and the data frame is indicated
Judged, if data frame mark sends miscue and controlled to center not in the frame mark value range of setting
Molding block, and by the error logging in syslog file；If the data frame mark is correct, big according to data frame
Small and data frame mark determines the frame structure used when handling the data frame, and send the mark of the frame structure to
Central Control Module；
(5), Central Control Module receives miscue or the frame structure mark for becoming that frame judgment module is sent；If in receiving
Hold for miscue, then enter step (8)；If reception content indicates for frame structure, enter step (6)；
(6), Central Control Module sends data frame and frame structure mark to the judgment module that takes off, take off judgment module according to
The frame structure indicates corresponding frame structure and data frame is parsed, and searches and recorded in a data frame in analysis result
Departure time is worth, and will be worth departure time and be sent to data processing module with data frame；
(7), data processing module is handled according to the data type of data frame, extracts the source in the data frame
Code, and by the source code with setting systematic parameter carry out it is corresponding, using the systematic parameter it is corresponding handle calculation formula pair
Source code is handled, and obtains the magnitude of physical quantity of the systematic parameter；By the systematic parameter, processing calculation formula and handling result
Database is sent to be preserved.
(8), terminate.
Handle the test data with storage system to application space vehicles device in real time for big data using the present invention below
It is handled and is stored.
Wherein, table 1 is that test data data volume per second, wherein aperiodic parameters wouldn't count because data volume is smaller.
Table 2 gives the physical quantity result (double types) for preserving all parameter whole process.For above-mentioned test data, system is when 72 is small
The interior processing and storage for completing 103GB or so data.The system of the present invention can synchronously complete the real-time place of all data of model
Reason and real-time storage disclosure satisfy that model handles the demand with storage in real time, effectively shorten the model testing time, accelerate number
According to storage efficiency so that the processing in real time of instantaneous big data and storage are achieved.
1 test data of table data volume statistics per second
The whole data volume statistics of table 2
The above is only the optimal specific embodiment of the present invention, but protection scope of the present invention is not limited thereto,
Any one skilled in the art in the technical scope disclosed by the present invention, the change or replacement that can be readily occurred in,
It should be covered by the protection scope of the present invention.
The content not being described in detail in description of the invention belongs to the known technology of professional and technical personnel in the field.
Claims (8)
1. a kind of big data handles in real time and storage system, it is characterised in that：Including data providing module, Central Control Module,
Telemetry frame parsing module becomes frame judgment module, the judgment module that takes off, data processing module and database, wherein：
Data providing module：The telemetering true form data frame of external network multicast is received, then follows true form data frame deposit
In ring buffer area, and the data frame in the circular buffer area is sent to Central Control Module；
Central Control Module：The data frame that data providing module is sent is received, and the data frame is sent to telemetry frame parsing
Module；The data frame judging result that telemetry frame parsing module is sent is received, if it is judged that frame of display data is correct, is then sent
The data frame to become frame judgment module；Miscue or the frame structure mark for becoming that frame judgment module is sent are received, if received
To the frame structure mark for becoming the transmission of frame judgment module, then the data frame and frame structure mark are sent to the judgment module that takes off；
After the departure time value for receiving judgment module transmission of taking off, the data frame and departure time value are sent to data processing mould
Block；
Telemetry frame parsing module：The data frame that Central Control Module is sent is received, by the size and frame swynchronization code of the data frame
Compared with the frame sign value of setting and the frame swynchronization code of setting, that is, judge the correctness of the data frame, and pass through described
The count value of data frame judges the continuity of data frame, then sends judging result to Central Control Module；If the judgement
As a result it is incorrect or discontinuous for data frame, then the judging result is recorded in syslog file；
Become frame judgment module：The data frame that Central Control Module is sent is received, the frame structure mark of the data frame is sentenced
It is disconnected, if the frame structure mark of the data frame in the frame structure mark value range of setting, send miscue to
Central Control Module, and by the error logging in syslog file；If the frame structure mark of the data frame is correct,
The frame structure used when then determining and handle the data frame according to data frame sign and frame structure mark, and described in transmission
Frame structure is indicated to Central Control Module；
Take off judgment module：Data frame and the frame structure mark that Central Control Module is sent are received, is indicated according to the frame structure
Corresponding frame structure parses data frame, and the departure time value of record in a data frame is then searched in analysis result,
And it will be worth departure time and be sent to Central Control Module；
Data processing module：Data frame and the departure time value that Central Control Module is sent are received, then according to the number of data frame
Handled according to type, extract the source code in the data frame, and the source code carried out with the systematic parameter set it is corresponding,
Source code is handled using the systematic parameter corresponding processing calculation formula, obtains the magnitude of physical quantity of the systematic parameter；
The systematic parameter, processing calculation formula and handling result are sent to database and preserved；
Database：Receive the handling result that data processing module is sent and preservation；
A kind of big data processing in real time and the data processing method of storage system, include the following steps：
(1), data providing module receives the telemetering true form data frame of external network multicast, is then stored in the true form data frame
In circular buffer area, and the data frame in the circular buffer area is sent to Central Control Module；
(2), the data frame received is sent to telemetry frame parsing module by Central Control Module；Telemetry frame parsing module is by described in
The size and frame swynchronization code of data frame judge the data compared with the frame sign value and the frame swynchronization code of setting that set
The correctness of frame, and judge by the count value of the data frame continuity of data frame, judging result is then sent to center
Control module；If the judging result is incorrect or discontinuous for data frame, the judging result is recorded in system day
In will file；
(3), Central Control Module receives the judging result that telemetry frame parsing module is sent；If the judging result is data frame
It is incorrect or discontinuous, then it enters step (8)；If the judging result is correct and continuous for data frame, enter step
(4)；
(4), Central Control Module sends data frame to frame judgment module is become, and becomes frame structure of the frame judgment module to the data frame
Mark is judged, if the frame structure mark of the data frame is sent not in the frame structure mark value range of setting
Miscue is to Central Control Module, and by the error logging in syslog file；If the frame knot of the data frame
Structure mark is correct, then the frame knot used when determining and handle the data frame according to data frame sign and frame structure mark
Structure, and the frame structure mark is sent to Central Control Module；
(5), Central Control Module receives miscue or the frame structure mark for becoming that frame judgment module is sent；If reception content is
Miscue then enters step (8)；If reception content indicates for frame structure, enter step (6)；
(6), Central Control Module sends data frame and frame structure mark to the judgment module that takes off, and the judgment module that takes off is according to described
Frame structure indicates corresponding frame structure and data frame is parsed, and searches in analysis result record taking off in a data frame
Moment value, and will be worth departure time and be sent to data processing module with data frame；
(7), data processing module is handled according to the data type of data frame, extracts the source code in the data frame, and
By the source code with set systematic parameter carry out it is corresponding, using the systematic parameter it is corresponding processing calculation formula to source code into
Row processing, obtains the magnitude of physical quantity of the systematic parameter；The systematic parameter, processing calculation formula and handling result are sent to
Database is preserved；
(8), terminate.
2. a kind of big data according to claim 1 handles in real time and storage system, it is characterised in that：Database includes real
When database and relational database, wherein, the handling result that real-time data base real-time reception data processing module is sent, and by institute
It states handling result and is forwarded to relational database and preserved.
3. a kind of big data according to claim 1 handles in real time and storage system, it is characterised in that：Data processing module
Data processing progress and data degree of being stored in are sent in data processing and storing process is carried out to Central Control Module, center control
Molding block receives the progress and is shown.
4. a kind of big data according to one of claims 1 to 3 handles in real time and storage system, it is characterised in that：Also wrap
Data publication module is included, wherein：Data processing module sends corresponding data processed result according to the data publication content of setting
To Central Control Module, Central Control Module forwards the data processed result to data publication module, data publication module root
The data processed result is packaged and Web Publishing according to the communication protocol of setting.
5. a kind of big data according to one of claims 1 to 3 handles in real time and storage system, it is characterised in that：Also wrap
Data center is included, wherein, default value is preserved in the database, and the default value includes frame sign value, frame synchronization
Code, frame structure indicate the calculation formula of value range, frame structure and systematic parameter；Data center reads the system from database
System setting value is simultaneously sent to Central Control Module；Central Control Module sends the frame sign value set and frame swynchronization code to telemetry frame
Parsing module sends frame structure and indicates value range to frame judgment module is become, sends frame structure mark to the judgment module that takes off, hair
The calculation formula of systematic parameter is sent to data processing module.
6. a kind of big data according to one of claims 1 to 3 handles in real time and storage system, it is characterised in that：Data
Processing module is divided into 8 processing units according to processing data type, single including slow varying parameter processing unit, order parameter processing
Member, computer word parameter processing unit, 1553B bus parameters unit, condition code multiplexing parameters processing unit, CAN bus parameter
Processing unit, secondary calculating parameter processing unit and serial ports parameter processing unit, each processing unit receive center control mould
The data frame that block is sent, is respectively handled the data of 8 types.
7. a kind of big data according to claim 6 handles in real time and storage system, it is characterised in that：Central Control Module
Thread control is carried out to 8 processing units in data processing module, is the thread of each processing unit distribution setting quantity,
Each unit carries out data processing in the distribution thread.
8. a kind of big data according to one of claims 1 to 3 handles in real time and storage system, it is characterised in that：Center
Control module controls system process, i.e., in system operation, if there is being again started up asking, then reports an error
And the request is forbidden.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510116906.4A CN104735142B (en) | 2015-03-17 | 2015-03-17 | A kind of processing in real time of big data and storage system and its processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510116906.4A CN104735142B (en) | 2015-03-17 | 2015-03-17 | A kind of processing in real time of big data and storage system and its processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104735142A CN104735142A (en) | 2015-06-24 |
| CN104735142B true CN104735142B (en) | 2018-05-22 |
Family
ID=53458568
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510116906.4A Active CN104735142B (en) | 2015-03-17 | 2015-03-17 | A kind of processing in real time of big data and storage system and its processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104735142B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105681205B (en) * | 2016-02-04 | 2018-12-21 | 北京宇航系统工程研究所 | A kind of data multicast transmission method of carrier rocket ground launch vehicle measurement master control net |
| CN105752077B (en) * | 2016-02-25 | 2018-03-06 | 上海科梁信息工程股份有限公司 | The moment of torsion distribution method and system of motor vehicle driven by mixed power |
| CN106407072A (en) * | 2016-09-09 | 2017-02-15 | 北京高地信息技术有限公司 | Monitoring system of big data platform |
| WO2018195972A1 (en) * | 2017-04-28 | 2018-11-01 | 深圳市大疆创新科技有限公司 | Method, apparatus and system for processing data |
| CN111091697B (en) * | 2019-11-29 | 2021-08-13 | 湖北航天飞行器研究所 | Telemetry data processing system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6775802B2 (en) * | 2000-08-08 | 2004-08-10 | Qualcomm Incorporated | Method, apparatus, and system for signal prediction |
| CN102724006B (en) * | 2012-06-18 | 2014-07-09 | 上海卫星工程研究所 | Rapid self-checking system and method of satellite comprehensive testing system |
| CN103347025B (en) * | 2013-07-10 | 2015-12-23 | 北京航天自动控制研究所 | A kind of telemetry data reduction processing method of spacecraft |
| CN103957134B (en) * | 2014-04-22 | 2017-05-10 | 航天恒星科技有限公司 | Modular configurable telemetry parameter analyzing and processing system |
-
2015
- 2015-03-17 CN CN201510116906.4A patent/CN104735142B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN104735142A (en) | 2015-06-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104735142B (en) | A kind of processing in real time of big data and storage system and its processing method | |
| CN107038162A (en) | Real time data querying method and system based on database journal | |
| CN104217127B (en) | PMU dynamic data simulation generation method | |
| RU2007101287A (en) | ADVANCED NETWORK STATISTICS DEVICE | |
| CN106374993A (en) | Full-automatic unattended satellite telemetering monitoring platform and data processing method | |
| CN107241232B (en) | Fault positioning system and method suitable for dispatching automation network analysis | |
| CN109525803A (en) | Video structural processing unit and method based on FPGA and artificial intelligence | |
| CN102945198B (en) | A kind of method characterizing high-performance calculation application characteristic | |
| CN109361576A (en) | A kind of PIM monitoring data processing method and system | |
| CN102970082A (en) | Method for testing local communication performance of low-voltage centralized meter reading terminal | |
| CN105162649A (en) | Testing method and system for nuclear power station safety level network performance index | |
| CN103117878A (en) | Design method of Nagios-based distribution monitoring system | |
| CN104702354A (en) | FPGA-based (field programmable gate array-based) PTN (packet transport network) equipment time delay monitoring device and method | |
| CN103678522A (en) | Method for acquiring and converting data of metering system of intelligent transformer substation | |
| CN102645928A (en) | Test platform for baseband equipment of test control system | |
| CN118037188A (en) | Collaborative management system based on multi-platform warehouse information | |
| CN105069029A (en) | Real-time ETL (extraction-transformation-loading) system and method | |
| CN112116790B (en) | CORS early warning monitoring system based on flow frame | |
| CN103095595B (en) | A kind of network data management method and system based on unidirectional parallel multilinked list | |
| CN103399914A (en) | Phasor data processing system based on intelligent telecontrol | |
| CN103368790B (en) | A kind of performance delays monitoring method for electronic trading system and system thereof | |
| CN206149326U (en) | Data acquisition analysis monitoring system | |
| CN111190790A (en) | Cloud computing cluster monitoring method and system based on peak prediction | |
| CN106970976A (en) | A kind of real-time dynamic passenger flow volume statistical method in scenic spot based on visitor's mobile signaling protocol data | |
| CN107390549A (en) | CRH5 types car control logic analogue system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |