CN114257646B - Telemetry data processing method, device, equipment and storage medium - Google Patents
Telemetry data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114257646B CN114257646B CN202111560291.6A CN202111560291A CN114257646B CN 114257646 B CN114257646 B CN 114257646B CN 202111560291 A CN202111560291 A CN 202111560291A CN 114257646 B CN114257646 B CN 114257646B
- Authority
- CN
- China
- Prior art keywords
- data
- telemetry
- stream data
- processing
- telemetry stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 49
- 238000003672 processing method Methods 0.000 title claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 137
- 230000005856 abnormality Effects 0.000 claims description 30
- 238000000034 method Methods 0.000 claims description 26
- 238000004364 calculation method Methods 0.000 claims description 20
- 238000005259 measurement Methods 0.000 claims description 13
- 230000001960 triggered effect Effects 0.000 claims description 13
- 230000002159 abnormal effect Effects 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 11
- 238000001514 detection method Methods 0.000 claims description 9
- 238000004140 cleaning Methods 0.000 claims description 7
- 238000013500 data storage Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 5
- 230000002045 lasting effect Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 6
- 238000013480 data collection Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000002688 persistence Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 230000003139 buffering effect Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000003862 health status Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000010992 reflux Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q9/00—Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q2209/00—Arrangements in telecontrol or telemetry systems
- H04Q2209/20—Arrangements in telecontrol or telemetry systems using a distributed architecture
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application provides a telemetry data processing method, a device, equipment and a storage medium. The telemetry data processing method comprises the following steps: determining a configuration file corresponding to telemetry data according to the type of the telemetry data; based on the configuration file corresponding to the telemetry data, controlling a data acquisition tool to acquire the telemetry data and generating telemetry stream data; serializing the telemetry stream data into message middleware to store the telemetry stream data; pulling telemetry stream data stored in the message middleware based on a distributed real-time computing engine, when the telemetry stream data triggers a preset event, performing data processing on the telemetry stream data based on a processing rule corresponding to the preset event, and returning a processing result to the message middleware or sending the processing result to a target end; the real-time storage of the multi-type data and the telemetering data processing based on various service requirements are realized, and the universality and the efficiency of the telemetering data processing are improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a telemetry data processing method, device, equipment and storage medium.
Background
Telemetry integrates the multidisciplinary technologies such as electronic technology, measurement technology, wireless communication technology and the like, and obtains telemetry data such as working state data and environmental data of an aircraft and a spacecraft through the processes such as data acquisition, coding, transmission, receiving, storage, processing and the like. The telemetry data not only provides basis for performance assessment and design improvement of aviation and aerospace craft, but also provides data support for fault analysis.
Traditional telemetry data processing platforms or systems are generally task-oriented, and need to develop new data processing software based on different business requirements or tasks, such as developing a new calculation engine to process telemetry data, and when the calculation engine is changed, the calculation engine often needs to synchronously modify codes of upstream and downstream devices of a tool after conversion so as to adapt to the new conversion tool, which results in low telemetry data processing efficiency.
Disclosure of Invention
The application provides a telemetry data processing method, a device, equipment and a storage medium, which are used for solving the problem of low telemetry data processing efficiency.
In a first aspect, the present application provides a telemetry data processing method, the method comprising:
determining a configuration file corresponding to telemetry data according to the type of the telemetry data; based on the configuration file corresponding to the telemetry data, controlling a data acquisition tool to acquire the telemetry data and generating telemetry stream data; serializing the telemetry stream data into message middleware to store the telemetry stream data; and pulling telemetry stream data stored in the message middleware based on the distributed real-time computing engine, when the telemetry stream data triggers a preset event, carrying out data processing on the telemetry stream data based on a processing rule corresponding to the preset event, and returning a processing result to the message middleware or sending the processing result to a target end.
Optionally, performing data processing on the telemetry stream data based on a processing rule corresponding to the preset event, and returning a processing result to a message middleware or sending the processing result to a target end, where the processing method includes:
acquiring an abnormality judgment predefined file of a preset event triggered by the telemetry stream data; the complex event processing unit based on the distributed real-time computing engine judges a predefined file according to the abnormality and detects the abnormality of the telemetry stream data; and generating satellite state warning information based on the abnormal detection result, and sending the satellite state warning information to a target end.
Optionally, the telemetry data includes one or more of copying test data, power measurement accuracy data, positioning accuracy data, out-of-lock recapture data, and channel delay consistency data.
Optionally, when the telemetry data is positioning accuracy data, triggering a positioning accuracy calculation event; and carrying out data processing on the positioning precision data based on the processing rule corresponding to the positioning precision calculation, and returning the processing result to the message middleware or sending the processing result to the target end, wherein the method comprises the following steps:
performing deserialization on the positioning accuracy stream data to obtain a first coordinate positioning error, a second coordinate positioning error and a third coordinate positioning error of each signal branch; based on a conversion operator of the distributed real-time calculation engine, determining the maximum comprehensive error of each signal branch according to the first coordinate positioning error, the second coordinate positioning error and the third coordinate positioning error of each signal branch; when the maximum integrated error of any signal branch is greater than a preset error, positioning precision warning information is generated; and sending the positioning accuracy alarm information to a target end.
Optionally, after pulling telemetry stream data stored in the information middleware based on the distributed real-time computing engine, the method further comprises:
data cleaning is carried out on the pulled telemetry stream data based on a distributed real-time computing engine; and when the remote stream data does not trigger a preset event, the cleaned remote stream data is returned to the message middleware and written into the columnar storage engine for backup.
Optionally, after writing the purged telemetry stream data to the columnar storage engine, the method further comprises:
reading the cleaned telemetry stream data stored in the columnar storage engine database in a Java database connection mode; and pushing the processed telemetry data to a target end in real time by adopting a Websocket protocol for display.
Optionally, the method further comprises:
and storing the processed telemetry stream data and the processing result into a distributed file storage system in a lasting way.
In a second aspect, the present application provides a telemetry data processing apparatus, the apparatus comprising:
the configuration file determining module is used for determining a configuration file corresponding to the telemetry data according to the type of the telemetry data; the data acquisition module is used for controlling the data acquisition tool to acquire the telemetry data based on the configuration file corresponding to the telemetry data and generating telemetry stream data; a data storage module for serializing the telemetry stream data into message middleware to store the telemetry stream data; the data processing module is used for pulling telemetry stream data stored in the information middleware based on the distributed real-time computing engine, when a preset event is triggered by the telemetry stream data, performing data processing on the telemetry stream data based on a processing rule corresponding to the preset event, and returning a processing result to the information middleware or sending the processing result to a target end.
Optionally, the data processing module is specifically configured to:
acquiring an abnormality judgment predefined file of a preset event triggered by the telemetry stream data; acquiring an abnormality judgment predefined file of a preset event corresponding to the telemetry stream data; the complex event processing unit based on the distributed real-time computing engine judges a predefined file according to the abnormality and detects the abnormality of the telemetry stream data; and generating satellite state warning information based on the abnormal detection result, and sending the satellite state warning information to a target end.
Optionally, when the telemetry data is positioning accuracy data, the data processing module is specifically configured to:
triggering a positioning precision calculation event based on positioning precision data stored in a pull information middleware of a distributed real-time calculation engine, and performing deserialization on the positioning precision stream data to obtain a first coordinate positioning error, a second coordinate positioning error and a third coordinate positioning error of each signal branch; based on a conversion operator of the distributed real-time calculation engine, determining the maximum comprehensive error of each signal branch according to the first coordinate positioning error, the second coordinate positioning error and the third coordinate positioning error of each signal branch; when the maximum integrated error of any signal branch is greater than a preset error, positioning precision warning information is generated; and sending the positioning accuracy alarm information to a target end.
Optionally, the apparatus further includes:
the data cleaning module is used for carrying out data cleaning on the pulled telemetry stream data based on the distributed real-time computing engine after the telemetry stream data stored in the information middleware is pulled based on the distributed real-time computing engine; and the data backup module is used for reflowing the cleaned telemetry stream data to the message middleware and writing the cleaned telemetry stream data into the columnar storage engine for backup when the telemetry stream data does not trigger a preset event.
Optionally, the apparatus further includes:
the data display module is used for reading the cleaned telemetry stream data stored in the columnar storage engine database in a Java database connection mode after the cleaned telemetry stream data are written into the columnar storage engine; and pushing the processed telemetry data to a target end in real time by adopting a Websocket protocol for display.
Optionally, the apparatus further includes:
and the persistence storage module is used for persistence storing the processed telemetry stream data and the processed result into the distributed file storage system.
In a third aspect, the present application also provides a telemetry data processing apparatus comprising: a processor, and a memory communicatively coupled to the processor; the memory stores computer-executable instructions; the processor executes computer-executable instructions stored in the memory to implement the telemetry data processing method provided in the first aspect of the application.
In a fourth aspect, the application also provides a computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out the telemetry data processing method provided in the first aspect of the application.
In a fifth aspect, the application also provides a computer program product comprising a computer program which when executed by a processor implements the telemetry data processing method of the first aspect of the application.
According to the telemetry data processing method, device, equipment and storage medium, for telemetry data with various types and large data volume, the matched configuration file is determined based on the types of the telemetry data, the real-time acquisition of the telemetry data is realized through the configuration file and the data acquisition tool, telemetry stream data is generated, and unified acquisition of multiple telemetry data is realized; further, based on the message middleware, remote stream data are stored, data decoupling of a data acquisition end and a data processing end is achieved, and reusability of remote data is improved; the distributed real-time computing engine is used for processing the telemetry data based on various service requirements in an event triggering mode, and processing results are returned and sent, so that the universality and the flexibility of the telemetry data are improved, and the processing efficiency of the telemetry data is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic diagram of an application scenario of a telemetry data processing method according to an embodiment of the present application;
FIG. 2 is a flow chart of a telemetry data processing method provided in one embodiment of the application;
FIG. 3 is a flow chart of one implementation of step S204 in the embodiment of FIG. 2 according to the present application;
FIG. 4 is a flowchart of another implementation of step S204 in the embodiment of FIG. 2 according to the present application;
FIG. 5 is a flow chart of a telemetry data processing method according to another embodiment of the present application;
FIG. 6 is a schematic diagram of a telemetry data processing framework provided in accordance with one embodiment of the present application;
FIG. 7 is a schematic diagram of a telemetry data processing apparatus according to one embodiment of the present application;
fig. 8 is a schematic diagram of a telemetry data processing apparatus according to an embodiment of the present application.
Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.
Each satellite needs to transmit up to several thousand telemetry parameters to reflect the operational status and health status of all equipment onboard the satellite. With the development of earth observation technology, various types of satellites and sensors emit and lift off, the volume of telemetry data is increased in a explosive manner, the telemetry data stock of different data centers reaches PB level, and higher requirements are put on a telemetry data processing platform.
Fig. 1 is a schematic diagram of an application scenario of a telemetry data processing method according to an embodiment of the present application, where as shown in fig. 1, a data center, such as a data center of a satellite data production facility, a data center of a university or scientific research facility, a data center of a commercial facility, etc., generally includes a receiver and a telemetry data processing platform, the receiver is configured to receive telemetry data returned by a corresponding satellite system, and the telemetry data processing platform is responsible for cataloging, storing and managing telemetry data returned by the corresponding satellite system. Based on the service requirement of the data center, a corresponding telemetry data processing platform is built, and the telemetry data corresponding to the data center is stored and processed.
Because the telemetry data processing platform of the existing data center is built based on tasks, when tasks or business demands change, the built telemetry data processing platform is often required to be changed, such as a calculation engine of the telemetry data processing platform is replaced or reconfigured, and codes of upstream and downstream devices of the calculation engine are adaptively adjusted, thereby realizing telemetry data processing based on new business demands, and the defects of repeated development, long development period and high development cost exist, so that telemetry data processing efficiency is low.
The application provides a telemetry data processing method, which aims to solve the technical problems in the prior art. The main conception is as follows: when the telemetry data is acquired, based on the type of the telemetry data, different configuration files are adopted to control a data acquisition tool to acquire in real time and convert the telemetry data into telemetry stream data, and then a message middleware stores the telemetry stream data, so that decoupling between a data acquisition end and a data processing end is realized, and the reusability of the telemetry data is improved; the remote measurement stream data is processed based on the mode of event triggering by the distributed real-time computing engine, and the remote measurement data processing of various service demands is realized through processing rules or conversion operators corresponding to various events, so that the universality of a remote measurement data processing platform is improved, and the remote measurement data processing efficiency is improved.
The following describes the technical scheme of the present application and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
FIG. 2 is a flow chart of a telemetry data processing method provided by an embodiment of the present application, which may be performed by a telemetry data processing device, which may be a computer, server, or the like, provided with a telemetry data processing framework, as shown in FIG. 2, the telemetry data processing method comprising the steps of:
step S201, according to the type of the telemetry data, determining a configuration file corresponding to the telemetry data.
The telemetry data are data for monitoring the operation conditions of the satellite, the spacecraft, the aircraft and other equipment and various sensors arranged on the equipment. The remote measurement data corresponding to the instruction can be received in an interactive instruction mode, and the sensor can actively report the acquired remote measurement data in a set report mode.
Specifically, the original data collected by the remote sensor can be received by the receiving device and analyzed to obtain newly added telemetry data, and then the newly added telemetry data in the receiving device is synchronized by the data acquisition tool in a mode of monitoring the port of the receiving device.
In some embodiments, the telemetry data output by the receiving device is in JSON format.
In some embodiments, the ports of the receiving devices of the various data centers may be monitored to enable real-time acquisition of multi-source telemetry data.
Specifically, when new telemetry data exists in the receiving device, determining a configuration file corresponding to the new telemetry data according to the type of the new telemetry data, so as to collect the new telemetry data based on the configuration file.
In some embodiments, the type of telemetry data may be partitioned according to the purpose of the telemetry data, and may also be partitioned according to the source channel of the telemetry data, the corresponding port, the data type, etc. Different types of telemetry data may be collected using different profiles.
Optionally, the telemetry data includes one or more of copying test data, power measurement accuracy data, positioning accuracy data, out-of-lock recapture data, and channel delay consistency data.
The copying test data are data generated when a copying test is carried out on the new equipment, and the compatibility and stability of software and hardware are tested through operation without shutdown. The power measurement accuracy data is data obtained when the power of the motor is measured with high accuracy, such as power value, accuracy and the like. The positioning accuracy data is used for describing the accuracy of the position information of the satellite, the spacecraft, the aircraft and other equipment analyzed by the receiving device on each coordinate axis. The loss-of-lock weight capturing data is data received by adopting a loss-of-lock weight capturing strategy when the receiving device is out of lock, namely, when the receiving device cannot capture signals in an environment where satellite signals are interfered. The channel delay consistency data comprises delay errors of all channels of a telemetry system and is used for carrying out time synchronization.
Step S202, based on the configuration file corresponding to the telemetry data, controlling a data acquisition tool to acquire the telemetry data and generating telemetry stream data.
Specifically, after determining the configuration files corresponding to each newly added telemetry data, the data acquisition tool is controlled to read each configuration file so as to realize real-time acquisition of the corresponding newly added telemetry data and obtain telemetry stream data.
In some embodiments, the data collection tool may be a large data collection tool such as Flume, fluentd.
In some embodiments, one or more Agent components of the jume big data collection tool may be started based on a configuration file corresponding to the telemetry data, and collection of the telemetry data may be performed to obtain telemetry stream data.
The speed of collecting and writing data by the Flume big data collecting tool is high, so that the real-time property of telemetering data collection is improved; at the same time, the jume provides a contextual routing feature that further ensures consistency of data when transmitted and received.
Step S203, serializing the telemetry stream data into a message middleware to store the telemetry stream data.
Wherein the message middleware may be Kafka, roketMQ or the like.
Specifically, one or more telemetry stream data stored in the Channel can be pulled through a Sink component of the flash via the message middleware and serialized into a corresponding Topic for storage, so that decoupling and buffering of data at the data acquisition end and the data processing end are realized.
Step S204, based on the distributed real-time computing engine, pulling telemetry stream data stored in the message middleware, when the telemetry stream data triggers a preset event, performing data processing on the telemetry stream data based on a processing rule corresponding to the preset event, and returning a processing result to the message middleware or sending the processing result to a target end.
The distributed real-time computing engine may be a link, spark, or the like. The preset event may be any event used for performing data processing, such as a complex event in a link, and trigger conditions of various preset events may be predefined, so as to determine whether the telemetry stream data triggers the preset event based on the trigger conditions. The target end is a downstream device of the distributed real-time computing engine and can be a display screen, other servers and the like. Each preset event may correspond to one or more processing rules to perform the processing of the data triggering the preset event.
Specifically, the distributed real-time computing engine is used as a data processing end of telemetry data, can consume telemetry stream data stored in the message middleware in real time, analyze and calculate based on various service requirements to obtain a processing result, and send the processing result to the target end.
Specifically, after the telemetry stream data stored in the information middleware is pulled, the distributed real-time computing engine can judge whether the telemetry stream data triggers a preset event according to the data type and the content of the telemetry stream data, if so, the telemetry stream data is processed based on a processing rule corresponding to the preset event triggered by the telemetry stream data, and a processing result is obtained.
Further, if the telemetry stream data does not trigger any event, a corresponding conversion operator can be determined based on the type of the telemetry stream data, the telemetry stream data is converted to obtain the satellite on-line state, and the satellite on-line state is sent to a front-end application system for visual display.
Optionally, fig. 3 is a flowchart of an implementation of step S204 in the embodiment of fig. 2, where, as shown in fig. 3, step S204 includes the following steps:
step S301 pulls telemetry stream data stored in the information middleware based on the distributed real-time computing engine.
Specifically, after the telemetry stream data is pulled, the telemetry stream data may also be cleaned and converted.
Specifically, the cleaned and converted telemetry stream data can be sent to a display device for visual display of satellite states and ad hoc inquiry.
Step S302, when the telemetry stream data triggers a preset event, acquiring an abnormality judgment predefined file of the preset event triggered by the telemetry stream data.
Specifically, when the telemetry stream data triggers a preset event, an abnormality judgment predefined file of the preset time triggered by the telemetry stream data is obtained, so that the abnormality judgment of the telemetry stream data is performed through the operation of the predefined file.
Step S303, the complex event processing unit based on the distributed real-time computing engine performs anomaly detection on the telemetry stream data according to the anomaly judgment predefined file.
Specifically, the complex event processing unit CEP (Complex Event Processing) of the link may run the abnormality determination predefined file, and detect an abnormality of the telemetry stream data piece based on a processing rule corresponding to the abnormality determination predefined file, so as to determine whether an abnormality exists in the satellite state.
Specifically, the abnormality judgment predefined file may include abnormality judgment conditions of various telemetry stream data, and when any one of the abnormality judgment conditions is satisfied by telemetry stream data, it is determined that there is an abnormality in the satellite state, and an abnormality detection result of the telemetry stream data is generated based on the judgment results of the abnormality judgment conditions.
Step S304, based on the abnormal detection result, generating satellite state warning information, and sending the satellite state warning information to a target end.
If the satellite state is abnormal, generating satellite state warning information based on an abnormal detection result, namely an abnormal judgment condition met by the remote stream data, and sending the satellite state warning information to a target end to prompt the satellite state to be abnormal in time.
Specifically, the satellite state warning information may further include detailed data generated during the telemetry data processing process.
According to the telemetry data processing method provided by the embodiment, aiming at telemetry data with various types and large data volume, a matched configuration file is determined based on the types of the telemetry data, real-time acquisition of the telemetry data is realized through the configuration file and a data acquisition tool, telemetry stream data is generated, and unified acquisition of multiple remote telemetry data is realized; further, based on the message middleware, remote stream data are stored, data decoupling of a data acquisition end and a data processing end is achieved, and reusability of remote data is improved; the distributed real-time computing engine is used for processing the telemetry data based on various service requirements in an event triggering mode, and processing results are returned and sent, so that the universality and the flexibility of the telemetry data are improved, and the processing efficiency of the telemetry data is improved.
Optionally, fig. 4 is a flowchart of another implementation of step S204 in the embodiment of fig. 2, where in this embodiment, step S204 is further explained by taking telemetry, which is positioning accuracy data, as an example, as shown in fig. 4, step S204 includes the following steps:
step S401, pulling positioning accuracy data stored in the information middleware based on the distributed real-time computing engine.
The positioning accuracy data is a parameter for describing satellite positioning accuracy, and may include positioning errors of three mutually perpendicular coordinate axes of X, Y, Z.
Step S402, a positioning accuracy calculation event is triggered.
Step S403, performing deserialization on the positioning accuracy stream data to obtain a first coordinate positioning error, a second coordinate positioning error and a third coordinate positioning error of each signal branch.
Specifically, a message of Topic corresponding to positioning accuracy data in the Kafka cluster, such as a positionPrecise, is pulled based on the Flink, and is deserialized to generate DataStream data including a first coordinate (X) positioning error, a second coordinate (Y) positioning error and a third coordinate (Z) positioning error of each signal branch.
Step S404, determining the maximum integrated error of each signal branch according to the first coordinate positioning error, the second coordinate positioning error and the third coordinate positioning error of each signal branch based on the conversion operator of the distributed real-time calculation engine.
The conversion operator is a precompiled operator for processing the positioning precision data. Taking the example of a distributed real-time computing engine as a link, the conversion operator may include an operator such as keyBy, map, reduce.
Specifically, after generating data of the DataStream including the first coordinate positioning error, the second coordinate positioning error and the third coordinate positioning error of the signal branches, a map operator may be applied to convert the DataStream into a mapStream in a binary group Tuple form, where the binary group Tuple includes the positioning error and its frequency point number. And converting the mapStream according to the frequency point number through a keyBy operator to obtain keydstream, so that the positioning accuracy stream data is divided into data of different frequency points. And further processing keydstream based on a map operator, extracting a time value and a residual data value in the data to form time-stamp stream data of a binary group type multiple 2, obtaining watermark watermarks of each positioning error based on the time value in the time-stamp stream, generating an operator assignadstampsAnd watermarks by calling the watermark of the flash, setting maximum allowable disorder time maxofOrderness, such as 3s, 5s or other values, calculating the current maximum watermark watermarks and the maximum allowable disorder time difference value to obtain watermark stream data watermark stream, calculating the comprehensive error value of each signal branch according to the first coordinate positioning error, the second coordinate positioning error and the third coordinate positioning error, and finally obtaining the maximum value of each comprehensive error value corresponding to each signal branch, namely the maximum comprehensive error.
Step S405, when the maximum integrated error of any signal branch is greater than the preset error, generating positioning precision alarm information, and sending the positioning precision alarm information to a target end.
The preset error may be a preset error threshold.
Specifically, when the maximum integrated error corresponding to any signal branch is greater than the preset error, positioning accuracy warning information is generated based on the watermark time corresponding to the signal branch and the maximum integrated error, positioning accuracy deviation is known in time, a corresponding adjustment strategy is adopted, and positioning accuracy is improved.
Further, the acquired telemetry data of the signal branch (the branch with the maximum integrated error greater than the preset error) in the time period corresponding to the positioning accuracy data stored in the receiving device may be deleted.
In some embodiments, the distributed real-time computing engine may further determine the accuracy of the measurement time of the telemetry data or the telemetry stream data, and in particular, the distributed real-time computing engine may compare the frequency point number and the satellite number of the telemetry stream data pulled from the message middleware with those of the telemetry data in the analog source data, and if the frequency point number and the satellite number are inconsistent with each other, it indicates that a frame loss phenomenon occurs, and may generate a frame loss prompt message, and send the frame loss prompt message to the target end.
Further, when frame loss occurs, telemetry data, which is inconsistent with telemetry data in analog source data, of the frequency point number and the satellite number received in the receiver can be removed.
Fig. 5 is a flowchart of a telemetry data processing method according to another embodiment of the present application, where steps for cleaning telemetry stream data and processing telemetry stream data when a preset event is not triggered by the telemetry stream data are added on the basis of the embodiment shown in fig. 2, and as shown in fig. 5, the telemetry data processing method according to the present embodiment may include the following steps:
step S501, determining a configuration file corresponding to the telemetry data according to the type of the telemetry data.
Step S502, based on the configuration file corresponding to the telemetry data, controlling a data acquisition tool to acquire the telemetry data and generating telemetry stream data.
Step S503, serializing the telemetry stream data into a message middleware to store the telemetry stream data.
Step S504, pulling telemetry stream data stored in the information middleware based on the distributed real-time computing engine.
In some embodiments, telemetry stream data of the non-real-time class may also be persisted into a distributed file storage system (Hadoop Distributed File System, HDFS) for the telemetry stream data of the non-real-time class.
Step S505, performing data cleansing on the pulled telemetry stream data based on the distributed real-time computing engine.
Taking the Flink as an example, the telemetry stream data may be data cleaned based on the Integrator of the Flink.
Step S506, when the cleaned telemetry stream data triggers a preset event, data processing is carried out on the telemetry stream data based on a processing rule corresponding to the preset event, and a processing result is returned to a message middleware or sent to a target end.
Optionally, after performing data processing on the telemetry stream data based on a processing rule corresponding to the preset event, the method further includes:
and storing the processed telemetry stream data and the processing result into a distributed file storage system in a lasting way.
Step S507, when the cleaned telemetry stream data does not trigger a preset event, the cleaned telemetry stream data is reflowed to the message middleware and written into the columnar storage engine for backup.
The columnar storage engine may be ClickHouse, hadoop or the like.
Specifically, after the cleaned telemetry stream data is written into the columnar storage engine, such as ClickHouse, the visual impromptu inquiry and monitoring display of satellite state data can be realized based on a service interface of the columnar storage engine and a corresponding front-end application system.
Further, the telemetry stream data stored in the columnar storage engine may be used by RESTful API (Application Programming Interface, application program interface) packages.
Step S508, the cleaned telemetry stream data stored in the columnar storage engine database is read in a Java database connection mode, and the processed telemetry data is pushed to a target end for display in real time by adopting a Websocket protocol.
Specifically, the method realizes the reading of the cleaned telemetry stream data stored in the columnar storage engine in a JDBC (Java Database Connectivity, java database connection) mode, and pushes the read telemetry stream data to a target end for display by adopting a Websocket protocol.
And in the case that the WebSocket link fracture cannot be reconnected, judging the WebSocket link condition through the target terminal, and then adopting a consumption degradation strategy to enable the target terminal to obtain the remote measurement stream data from the RESTful API, so that the corresponding RESTful API is called, and the stability is improved.
In the embodiment, aiming at telemetry data with various types and large data volume, a matched configuration file is determined based on the types of the telemetry data, the real-time acquisition of the telemetry data is realized through the configuration file and a data acquisition tool, telemetry stream data is generated, and unified acquisition of multiple remote telemetry data is realized; further, based on the message middleware, remote stream data are stored, data decoupling of a data acquisition end and a data processing end is achieved, and reusability of remote data is improved; processing telemetry data based on various service requirements by adopting an event triggering mode based on a distributed real-time computing engine, and refluxing and sending processing results; for telemetry stream data of an un-triggered event, data display is performed through a columnar storage engine and a JDBC mode, and persistent storage is performed through an HDFS, so that the method is high in expandability, and is suitable for the storage requirement of a large amount of telemetry data, and meanwhile, the universality and the flexibility of a telemetry data processing platform are improved, and the processing efficiency of the telemetry data is improved.
FIG. 6 is a schematic diagram of a telemetry data processing framework provided in accordance with one embodiment of the present application, as shown in FIG. 6, comprising: the system comprises a Flume real-time data stream acquisition tool, kafka message middleware, a Flink calculation engine, an HDFS distributed file storage system, a ClickHouse columnar storage engine and a target end.
The method comprises the steps that a receiving device receives telemetry data, a Source module and a Sink module, wherein the Flume is used for synchronizing the telemetry data in real time from the receiving device, collecting the telemetry data to the Source module and then sending the telemetry data to one or more channels, the Sink module is used for self-defining pulling telemetry stream data of any Channel, the pulled telemetry stream data are serialized into the Topic through a Producer of Kafka, the distributed storage tool based on the Kafka and a release subscription mode is used for centralized storage, and decoupling and buffering of original data (telemetry data output by the receiving device) are achieved. And (3) consuming telemetry stream data in Kfka in real time by the Flink, analyzing and calculating the data according to different service requirements, and outputting the result to a target end. And for real-time data of satellite online state and near-line related data, the data is cleaned and converted through the Flink, then the real-time data is loaded and written into the ClickHouse, a data service interface is provided by the ClickHouse and integrated with a front-end application system, and the data stored in the ClickHouse is read based on a JDBC mode, so that the target end performs visual on-site query and monitoring display on the satellite state data. And the Flink CEP (complex event processing) is used for predefining the judgment basis of abnormal data and carrying out real-time early warning on satellite states. Finally, the processed data can be stored in an HDFS distributed file storage system, possibly in an unstructured file form or in a structured data storage mode, so that the later data backtracking and the extraction processing of corresponding service data are facilitated.
FIG. 7 is a schematic diagram of a telemetry data processing apparatus according to one embodiment of the present application, as shown in FIG. 7, the apparatus includes: a profile determination module 710, a data acquisition module 720, a data storage module 730, and a data processing module 740.
The configuration file determining module 710 is configured to determine a configuration file corresponding to telemetry data according to a type of the telemetry data; the data acquisition module 720 is configured to control the data acquisition tool to acquire the telemetry data based on the configuration file corresponding to the telemetry data, and generate telemetry stream data; a data storage module 730 for serializing the telemetry stream data into message middleware for storing the telemetry stream data; the data processing module 740 is configured to pull telemetry stream data stored in the message middleware based on the distributed real-time computing engine, perform data processing on the telemetry stream data based on a processing rule corresponding to a preset event when the telemetry stream data triggers the preset event, and return a processing result to the message middleware or send the processing result to a target end.
Optionally, the data processing module 740 is specifically configured to:
acquiring an abnormality judgment predefined file of a preset event triggered by the telemetry stream data; acquiring an abnormality judgment predefined file of a preset event corresponding to the telemetry stream data; the complex event processing unit based on the distributed real-time computing engine judges a predefined file according to the abnormality and detects the abnormality of the telemetry stream data; and generating satellite state warning information based on the abnormal detection result, and sending the satellite state warning information to a target end.
Optionally, when the telemetry data is positioning accuracy data, the data processing module 740 is specifically configured to:
triggering a positioning precision calculation event based on positioning precision data stored in a pull information middleware of a distributed real-time calculation engine, and performing deserialization on the positioning precision stream data to obtain a first coordinate positioning error, a second coordinate positioning error and a third coordinate positioning error of each signal branch; based on a conversion operator of the distributed real-time calculation engine, determining the maximum comprehensive error of each signal branch according to the first coordinate positioning error, the second coordinate positioning error and the third coordinate positioning error of each signal branch; when the maximum integrated error of any signal branch is greater than a preset error, positioning precision warning information is generated; and sending the positioning accuracy alarm information to a target end.
Optionally, the apparatus further includes:
the data cleaning module is used for carrying out data cleaning on the pulled telemetry stream data based on the distributed real-time computing engine after the telemetry stream data stored in the information middleware is pulled based on the distributed real-time computing engine; and the data backup module is used for reflowing the cleaned telemetry stream data to the message middleware and writing the cleaned telemetry stream data into the columnar storage engine for backup when the telemetry stream data does not trigger a preset event.
Optionally, the apparatus further includes:
the data display module is used for reading the cleaned telemetry stream data stored in the columnar storage engine database in a Java database connection mode after the cleaned telemetry stream data are written into the columnar storage engine; and pushing the processed telemetry data to a target end in real time by adopting a Websocket protocol for display.
Optionally, the apparatus further includes:
and the persistence storage module is used for persistence storing the processed telemetry stream data and the processed result into the distributed file storage system.
The telemetry data processing device provided by the embodiment of the application can execute the telemetry data processing method provided by any embodiment of the application, and has the corresponding functional modules and beneficial effects of the execution method.
FIG. 8 is a schematic diagram of a telemetry data processing apparatus according to one embodiment of the present application, as shown in FIG. 8, comprising: memory 810, processor 820, and computer programs.
Wherein a computer program is stored in memory 810 and configured to be executed by processor 820 to implement a telemetry data processing method provided by any of the embodiments of the application corresponding to fig. 2-5.
Wherein memory 810 and processor 820 are coupled via bus 830.
The relevant descriptions may be understood correspondingly with reference to the relevant descriptions and effects corresponding to the steps of fig. 2 to 5, and are not repeated here.
A non-transitory computer readable storage medium, which when executed by a processor of a telemetry data processing apparatus, causes the telemetry data processing apparatus to perform the telemetry data processing method described above.
For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
Embodiments of the present application also provide a computer program product comprising an executable computer program stored in a readable storage medium. The computer program may be readable by at least one processor of a telemetry data processing apparatus, the computer program being executable by the at least one processor to cause the telemetry data processing apparatus to implement the telemetry data processing method provided by the various embodiments described above.
Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (9)
1. A method of telemetry data processing, the method comprising:
determining a configuration file corresponding to telemetry data according to the type of the telemetry data;
based on the configuration file corresponding to the telemetry data, controlling a data acquisition tool to acquire the telemetry data and generate telemetry stream data, wherein the data acquisition tool is a flight big data acquisition tool;
serializing the telemetry stream data into message middleware to store the telemetry stream data;
pulling telemetry stream data stored in the message middleware based on a distributed real-time computing engine, when the telemetry stream data triggers a preset event, performing data processing on the telemetry stream data based on a processing rule corresponding to the preset event, and returning a processing result to the message middleware or sending the processing result to a target end;
and carrying out data processing on the telemetry stream data based on the processing rule corresponding to the preset event, and returning the processing result to the message middleware or sending the processing result to the target end, wherein the method comprises the following steps:
Acquiring an abnormality judgment predefined file of a preset event triggered by the telemetry stream data;
the complex event processing unit based on the distributed real-time computing engine judges a predefined file according to the abnormality and detects the abnormality of the telemetry stream data;
and generating satellite state warning information based on the abnormal detection result, and sending the satellite state warning information to a target end.
2. The method of claim 1, wherein the telemetry data comprises one or more of copy test data, power measurement accuracy data, positioning accuracy data, out-of-lock weight acquisition data, channel delay consistency data.
3. The method of claim 2, wherein a positioning accuracy calculation event is triggered when the telemetry data is positioning accuracy data; and carrying out data processing on the positioning precision data based on the processing rule corresponding to the positioning precision calculation, and returning the processing result to the message middleware or sending the processing result to the target end, wherein the method comprises the following steps:
performing deserialization on the positioning accuracy stream data to obtain a first coordinate positioning error, a second coordinate positioning error and a third coordinate positioning error of each signal branch;
Based on a conversion operator of the distributed real-time calculation engine, determining the maximum comprehensive error of each signal branch according to the first coordinate positioning error, the second coordinate positioning error and the third coordinate positioning error of each signal branch;
when the maximum integrated error of any signal branch is greater than a preset error, positioning precision warning information is generated;
and sending the positioning accuracy alarm information to a target end.
4. The method of claim 1, wherein after pulling telemetry stream data stored in the information middleware based on the distributed real-time computing engine, the method further comprises:
data cleaning is carried out on the pulled telemetry stream data based on a distributed real-time computing engine;
and when the remote stream data does not trigger a preset event, the cleaned remote stream data is returned to the message middleware and written into the columnar storage engine for backup.
5. The method of claim 4, wherein after writing the purged telemetry stream data to the inline storage engine, the method further comprises:
reading the cleaned telemetry stream data stored in the columnar storage engine database in a Java database connection mode;
And pushing the processed telemetry data to a target end in real time by adopting a Websocket protocol for display.
6. The method according to any one of claims 1-5, further comprising:
and storing the processed telemetry stream data and the processing result into a distributed file storage system in a lasting way.
7. A telemetry data processing apparatus, the apparatus comprising:
the configuration file determining module is used for determining a configuration file corresponding to the telemetry data according to the type of the telemetry data;
the data acquisition module is used for controlling a data acquisition tool to acquire the telemetry data based on the configuration file corresponding to the telemetry data to generate telemetry stream data, and the data acquisition tool is a flight big data acquisition tool;
a data storage module for serializing the telemetry stream data into message middleware to store the telemetry stream data;
the data processing module is used for pulling telemetry stream data stored in the information middleware based on the distributed real-time computing engine, when a preset event is triggered by the telemetry stream data, performing data processing on the telemetry stream data based on a processing rule corresponding to the preset event, and returning a processing result to the information middleware or sending the processing result to a target end;
The data processing module is specifically configured to:
acquiring an abnormality judgment predefined file of a preset event corresponding to the telemetry stream data; the complex event processing unit based on the distributed real-time computing engine judges a predefined file according to the abnormality and detects the abnormality of the telemetry stream data; and generating satellite state warning information based on the abnormal detection result, and sending the satellite state warning information to a target end.
8. A telemetry data processing apparatus, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the telemetry data processing method of any of claims 1-6.
9. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are for implementing the telemetry data processing method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111560291.6A CN114257646B (en) | 2021-12-20 | 2021-12-20 | Telemetry data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111560291.6A CN114257646B (en) | 2021-12-20 | 2021-12-20 | Telemetry data processing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114257646A CN114257646A (en) | 2022-03-29 |
| CN114257646B true CN114257646B (en) | 2023-11-14 |
Family
ID=80793081
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111560291.6A Active CN114257646B (en) | 2021-12-20 | 2021-12-20 | Telemetry data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114257646B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115422208A (en) * | 2022-11-07 | 2022-12-02 | 江苏智云天工科技有限公司 | Defect detection processing method, system and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101140322A (en) * | 2007-09-26 | 2008-03-12 | 北京航空航天大学 | Device and method used for layout satellite navigation foundation region reinforced system |
| CN112859115A (en) * | 2021-02-07 | 2021-05-28 | 北京微纳星空科技有限公司 | Alarm method, device and system for satellite test and storage medium |
| CN113268530A (en) * | 2020-02-17 | 2021-08-17 | 浙江大搜车软件技术有限公司 | Mass heterogeneous data acquisition method and system, computer equipment and storage medium |
| CN113489527A (en) * | 2021-06-07 | 2021-10-08 | 北京邮电大学 | Architecture system of lightweight network middleware and satellite communication method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11159658B2 (en) * | 2018-07-23 | 2021-10-26 | Moj.Io, Inc. | Homogenization of telematics data through unified messaging protocol |
-
2021
- 2021-12-20 CN CN202111560291.6A patent/CN114257646B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101140322A (en) * | 2007-09-26 | 2008-03-12 | 北京航空航天大学 | Device and method used for layout satellite navigation foundation region reinforced system |
| CN113268530A (en) * | 2020-02-17 | 2021-08-17 | 浙江大搜车软件技术有限公司 | Mass heterogeneous data acquisition method and system, computer equipment and storage medium |
| CN112859115A (en) * | 2021-02-07 | 2021-05-28 | 北京微纳星空科技有限公司 | Alarm method, device and system for satellite test and storage medium |
| CN113489527A (en) * | 2021-06-07 | 2021-10-08 | 北京邮电大学 | Architecture system of lightweight network middleware and satellite communication method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114257646A (en) | 2022-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2835446C (en) | Data analysis system | |
| US11209811B2 (en) | Diagnostic device, diagnostic method, and program | |
| CN107133110B (en) | GNSS navigation signal mass data rapid processing method based on cluster parallel operation | |
| CN107223243B (en) | Distributed data management system for embedded controller | |
| CN110325979B (en) | Measurement solution service providing system | |
| US20140288766A1 (en) | Real-Time Monitoring of Vehicle | |
| CN114257646B (en) | Telemetry data processing method, device, equipment and storage medium | |
| CN114189430A (en) | Three-dimensional log full-link monitoring system, method, medium and equipment | |
| CN107797902B (en) | Method and device for monitoring message transmission frequency of robot operating system | |
| KR20120098390A (en) | Inspection system, control server, inspection apparatus and control method of inspection data | |
| CN108154230A (en) | The monitoring method and monitoring device of deep learning processor | |
| CN115437339A (en) | Remote diagnosis method and device, electronic equipment and storage medium | |
| CN112186901A (en) | Panoramic sensing monitoring method and system for transformer substation | |
| CN111464637B (en) | Unmanned vehicle data processing method, device, equipment and medium | |
| CN114090395A (en) | Sensor data processing method and device | |
| CN105117347A (en) | Test data simulation method and system and automation test method and system | |
| CN110875853A (en) | Automatic quality monitoring method and terminal based on metadata | |
| CN110245120B (en) | Stream type computing system and log data processing method thereof | |
| EP2667021B1 (en) | Timestamping in wind turbines | |
| CN111427749A (en) | Monitoring tool and method for ironic service in openstack environment | |
| CN111130882A (en) | Monitoring system and method of network equipment | |
| US11714721B2 (en) | Machine learning systems for ETL data streams | |
| CN116027365A (en) | Distributed GNSS deformation monitoring real-time resolving system, device and method | |
| Christos et al. | Development of an IoT early warning platform for augmented decision support in oil & gas | |
| CN110633182B (en) | System, method and device for monitoring server stability |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |