CN116155362A - Multi-source data processing system and method - Google Patents
Multi-source data processing system and method Download PDFInfo
- Publication number
- CN116155362A CN116155362A CN202310161458.4A CN202310161458A CN116155362A CN 116155362 A CN116155362 A CN 116155362A CN 202310161458 A CN202310161458 A CN 202310161458A CN 116155362 A CN116155362 A CN 116155362A
- Authority
- CN
- China
- Prior art keywords
- data
- data processing
- source
- processing
- state information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 155
- 238000000034 method Methods 0.000 title claims abstract description 34
- 238000004458 analytical method Methods 0.000 claims abstract description 24
- 238000012544 monitoring process Methods 0.000 claims abstract description 22
- 238000010191 image analysis Methods 0.000 claims description 19
- 230000005540 biological transmission Effects 0.000 claims description 18
- 238000007781 pre-processing Methods 0.000 claims description 13
- 238000013500 data storage Methods 0.000 claims description 7
- 230000001360 synchronised effect Effects 0.000 claims description 5
- 238000003745 diagnosis Methods 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 claims description 4
- 238000003672 processing method Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 4
- 239000002699 waste material Substances 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 239000000243 solution Substances 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B7/00—Radio transmission systems, i.e. using radiation field
- H04B7/14—Relay systems
- H04B7/15—Active relay systems
- H04B7/185—Space-based or airborne stations; Stations for satellite systems
- H04B7/1851—Systems using a satellite or space-based relay
- H04B7/18519—Operations control, administration or maintenance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
- H04L47/2425—Traffic characterised by specific attributes, e.g. priority or QoS for supporting services specification, e.g. SLA
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/25—Flow control; Congestion control with rate being modified by the source upon detecting a change of network conditions
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Astronomy & Astrophysics (AREA)
- Aviation & Aerospace Engineering (AREA)
- Radio Relay Systems (AREA)
Abstract
The disclosure provides a multi-source data processing system and method, which are applied to the technical field of satellite data processing and comprise the following steps: the system comprises a data receiving module, a state monitoring module, a flow scheduling module, a data processing module and a data processing module, wherein the data receiving module is used for receiving binary data frames sent by a plurality of data sources in real time in parallel, distributing and sending the data frames to the corresponding data processing module according to the data sources and/or data types, the state monitoring module is used for monitoring data state information and link state information of the data frames sent by the data sources in all directions, the flow scheduling module is used for dynamically scheduling all the data processing modules according to at least one of the link state information, the data state information and the computing node resource conditions of the data sources in all directions, the data processing modules are all used for receiving and processing the binary data frames sent by the data sources in a single direction, calling an analysis plug-in unit according to the data frames, generating an analysis result and realizing satellite load data parallel processing of the data sources.
Description
Technical Field
The present disclosure relates to the field of satellite data processing technologies, and in particular, to a system and a method for processing multi-source data.
Background
With the continuous exploration of human beings in the space field, satellite communication technology is vigorously developed, the types and data volumes of satellite load data are increased, and higher requirements are also put forward on a satellite downlink data processing end. In the satellite on-orbit operation stage, the original data is converted into binary data stream through operations such as compression, encryption, scrambling and the like, then the binary data stream is downloaded to the ground through steps such as packaging, satellite-ground link transmission, ground station receiving and the like, and finally, the binary data stream is restored into real data collected by satellite loads through steps such as station transmission, data descrambling, decryption, decompression and the like.
In order to download the load original data to the ground, the satellite needs to establish a satellite-ground transmission link with each ground station, so that reliable transmission of the data is ensured. Low orbit satellites have limited ground station visibility and satellite-to-ground data link transmission times due to limited orbital altitude and other conditions. In order to ensure the continuity and reliability of the downloading of the low-orbit satellite data and ensure the stable downloading of the data in an important monitoring stage, monitoring means such as double station relay, multi-station tracking and the like are usually adopted, so that the phenomena of repeated receiving of multi-data source data, parallel receiving of multi-frequency band data and the like can occur.
When the above phenomenon occurs, the existing treatment method is as follows: firstly, selecting a binary data stream of a single data source for processing; and secondly, fusing the multi-data source data according to a certain strategy, and then processing the fused data. The method not only can cause a large amount of data waste, but also can reduce the quality of the fused data when a certain data source link is unstable, thereby affecting the whole downlink data processing flow.
Disclosure of Invention
The main purpose of the present disclosure is to provide a multi-source data processing system and method, which aims to solve the problem of parallel processing of satellite load data of multiple data sources.
To achieve the above object, a first aspect of embodiments of the present disclosure provides a multi-source data processing system, including:
the data receiving module is used for receiving binary data frames sent by the data sources in multiple directions in real time in parallel, and sending the data frames to the corresponding data processing module in a shunting way according to the data sources and/or the data types;
the state monitoring module is used for monitoring the data state information and the link state information of the data frames sent by the data sources in all directions;
the flow scheduling module is used for dynamically scheduling all the data processing modules according to at least one of the link state information of the data sources in all the directions, the data state information and the computing node resource conditions;
and the data processing modules are used for receiving and processing binary data frames sent by the unidirectional data source, calling the analysis plug-in according to the data frames and generating analysis results.
Optionally, the data processing module includes:
the data decryption component is used for receiving the original ciphertext data, analyzing the original ciphertext data according to the pre-annotating key, completing synchronous discrimination of the data, decrypting the effective area, converting the original ciphertext data into plaintext data and distributing the plaintext data;
the data preprocessing component is used for extracting an effective data area of a data frame and distributing the data frame to the data processing component according to a data indicating bit, wherein the data indicating bit is used for identifying the effective data area as image processing data or telemetry parameter coding data;
the data processing component is used for processing the data frame according to the data indicating bit.
Optionally, the data processing component includes an image processing plug-in;
the image processing plug-in is used for extracting the image type identification bit of the data frame, pushing the data frame to a corresponding decoder according to the image type identification bit to perform image analysis, and obtaining an image analysis result.
Optionally, the data processing component includes a telemetry parameter resolution plug-in;
the telemetry parameter analysis plug-in is used for converting the original telemetry code contained in the data frame into telemetry parameters with the preset parameter format according to the default parameter format.
Optionally, the data processing module further includes:
the data application component is used for receiving the image analysis result and the telemetry parameter which are pushed by the data processing component, and carrying out visual display and/or secondary analysis and/or utility evaluation and/or fault diagnosis on the image analysis result and the telemetry parameter.
Optionally, the data processing module further includes:
the data storage component is used for receiving the data frame sent by the data preprocessing component, the image analysis result and the telemetry parameter sent by the data processing component, storing the data frame, the image analysis result and the telemetry parameter, and storing in an associated mode according to type and/or time and/or source.
Optionally, the data status information includes at least one of a data source, a data type, a data receiving rate, a data continuity, a data frame loss rate, a receiving start-stop time, and a link connection status.
Optionally, the link state information includes at least one of a link direction, a frequency band, a data type, a data receiving rate, a data transmission quality, and a link connectivity state.
A second aspect of an embodiment of the present disclosure provides a multi-source data processing method, the method including:
receiving binary data frames sent by a plurality of data sources in real time in parallel, and monitoring data state information and link state information of the data frames sent by the data sources in the plurality of directions;
dynamically scheduling all data processing modules according to at least one of the link state information, the data state information and the computing node resource conditions of the plurality of directional data sources;
and sending the data frames to the corresponding data processing modules in a shunting manner according to the data source and/or the data type, wherein each data processing module is used for receiving binary data frames sent by the data source in a single direction and generating an analysis result according to the data frames.
Optionally, the data processing module includes a data preprocessing component and a data processing component, and the method further includes:
extracting an effective data area of a data frame by using the data preprocessing component, and distributing the data frame to the data processing component according to a data indicating bit, wherein the data indicating bit is used for identifying the effective data area as image processing data or telemetry parameter coding data;
and processing the data frame by utilizing the data processing component according to the data indicating bit.
Compared with the prior art, the multi-source data processing system and method provided by the disclosure have at least the following advantages and beneficial effects:
(1) The method adopts a data-driven data processing flow scheduling mode, and dynamically acquires and releases computing resources through a flow scheduling module. The method can reasonably allocate and use the computing nodes, flexibly schedule the available computing resources in the system and solve the problem of hardware resource waste.
(2) The multi-thread data receiving component is adopted in the method, the multi-directional binary data stream is received, and high-throughput data access and low-time-delay multi-source data parallel processing are effectively achieved. Meanwhile, the multi-directional data distribution parallel processing is friendly to a data hierarchical storage mode based on station information, is beneficial to guaranteeing the integrity of data storage, is convenient for analyzing and evaluating the data receiving quality, frame loss and frame error conditions and the like of each ground station in the same time period, and provides data support for follow-up data optimization and space-earth link transmission optimization.
(3) The method adopts a multidirectional data parallel processing mode to completely receive and process all the downloaded data, and ensures the analysis integrity of the satellite downloaded data to the greatest extent. And for subsequent problem investigation and tracing, multi-dimensional data support is provided.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present disclosure, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a multi-source data processing system according to one embodiment of the present disclosure;
FIG. 2 is a flow chart of a method for multi-source data processing according to an embodiment of the disclosure;
FIG. 3 schematically illustrates three format schematics of formatted data for satellite transmission provided by embodiments of the present disclosure;
FIG. 4 schematically illustrates a block diagram of a data processing flow for a single data source in accordance with an embodiment of the present disclosure;
fig. 5 schematically illustrates a detailed flow chart of a data-driven multi-source data processing flow provided by an embodiment of the present disclosure.
Detailed Description
In order to make the disclosure objects, features and advantages of the disclosure more comprehensible, the technical solutions in the embodiments of the disclosure will be clearly and completely described with reference to the accompanying drawings in the embodiments of the disclosure, and it is apparent that the described embodiments are only some embodiments of the disclosure, but not all embodiments. Based on the embodiments in this disclosure, all other embodiments that a person skilled in the art would obtain without making any inventive effort are within the scope of protection of this disclosure.
In order to save the storage space and reduce the transmission pressure of the satellite-ground link, the original data acquired by satellite load are converted into binary data frame streams through operations such as compression coding, encryption, scrambling and the like, and the binary data frame streams are connected with all ground stations through the satellite-ground link to carry out data downloading. After receiving the transmitted satellite original data, the multi-source data processing system provided by the disclosure extracts and analyzes the effective data step by step to generate a data analysis result.
FIG. 1 is a schematic diagram of a multi-source data processing system according to an embodiment of the present disclosure.
As shown in FIG. 1, the multi-source data processing system includes a data receiving module, a status monitoring module, a flow scheduling module, and a plurality of data processing modules.
And the data receiving module is used for receiving binary data frames sent by the data sources in multiple directions in real time in parallel, and sending the data frames to the corresponding data processing module in a shunting way according to the data sources and/or the data types.
And the state monitoring module is used for monitoring the data state information and the link state information of the data frames sent by the data sources in all directions.
And the flow scheduling module is used for dynamically scheduling all the data processing modules according to at least one of the link state information of the data sources in all the directions, the data state information and the computing node resource conditions.
And the plurality of data processing modules are used for receiving and processing binary data frames sent by the data source in a single direction, calling the analysis plug-in according to the data frames and generating analysis results.
In one possible implementation, the data processing module includes: and the data decryption component is used for receiving the original ciphertext data, analyzing the original ciphertext data according to the pre-injection key, completing the processing of synchronous discrimination, effective area decryption and the like of the data, converting the original ciphertext data into plaintext data and distributing the plaintext data.
In one possible implementation, the data processing module includes: a data preprocessing component for extracting an effective data area of a data frame and distributing the data frame to a data processing component according to a data indication bit, wherein the data indication bit is used for identifying the effective data area as image processing data or telemetry parameter coding data; the data processing component is used for processing the data frame according to the data indication bit in a targeted manner. Wherein the image processing data is a video/image compressed code stream.
In a possible implementation, the data processing component comprises an image processing plug-in. The image processing plug-in is used for extracting the data type identification bit of the data frame, pushing the data frame to a corresponding decoder according to the data type identification bit to perform image analysis, and obtaining an image analysis result.
In one possible implementation, the data processing component includes a telemetry parameter resolution plug-in. The telemetry parameter analysis plug-in is used for converting original telemetry codes contained in the data frames into telemetry parameters with the preset parameter format according to the default parameter format.
In a possible implementation manner, the data processing module further comprises a data application component for receiving the image analysis result and the telemetry parameter pushed by the data processing component, and performing visual display and/or secondary analysis and/or utility evaluation and/or fault diagnosis on the image analysis result and the telemetry parameter.
In a possible implementation manner, the data processing module further comprises a data storage component, configured to receive the data frame sent by the data preprocessing component, the image analysis result sent by the data processing component, and the telemetry parameter, store the data frame, the image analysis result, and the telemetry parameter, and store the data frame, the image analysis result, and the telemetry parameter in association according to type and/or time and/or source.
In one possible embodiment, the data status information includes at least one of a data source, a data type, a data reception rate, a data continuity, a data frame loss rate, a reception start-stop time, and a link-up status.
In a possible embodiment, the link state information includes at least one of a link direction, a frequency band, a data type, a data reception rate, a data transmission quality, and a link connectivity status.
FIG. 2 is a flow chart of a method for multi-source data processing according to an embodiment of the disclosure.
As shown in fig. 2, the method mainly includes steps S101-S103:
s101, receiving binary data frames sent by a plurality of data sources in real time in parallel, and monitoring data state information and link state information of the data frames sent by the data sources in the plurality of directions;
s102, dynamically scheduling all data processing modules according to at least one of the link state information, the data state information and the computing node resource conditions of the plurality of directional data sources;
s103, the data frames are sent to the corresponding data processing modules in a shunting mode according to the data source and/or the data type, and each data processing module is used for receiving binary data frames sent by the data source in a single direction and generating analysis results according to the data frames.
Fig. 3 schematically illustrates three formats of formatted data for satellite transmission provided by embodiments of the present disclosure.
As shown in fig. 3, assume a data format T 1 The method is characterized in that original binary data received by a multi-source data processing system consists of a frame synchronization code, an information source address, a data mark, a frame sequence number, a time code and an effective data area of M bytes; data format T 2 For data format T 1 M bytes in total, is formed by frame synchronization code, satellite information, frame number, data length, encryption area, effective data area of N bytes; data format T 3 To extract the data format T 2 The data frame format spliced after the effective data area of the data frame comprises a frame synchronization code, a frame sequence number, a data length, a time code, a data indication bit, an effective data area and a check code.
Fig. 4 schematically illustrates a block diagram of a data processing flow for a single data source according to an embodiment of the present disclosure.
As shown in fig. 4, a satellite data processing flow for a single data source provided by an embodiment of the present disclosure includes a data decryption component, a data preprocessing component, a data processing component, a data application, and a data storage component.
One possible embodiment according to the present disclosure includes:
the method comprises the steps that firstly, a data decryption component receives an original encrypted data stream of a satellite from a certain ground station, verifies the correctness of a data encryption area identification, decrypts data in an effective data area, and obtains plaintext data and issues the plaintext data;
step two, a data preprocessing component receives plaintext data in a data format T 2 According to the method, the correctness of the data synchronous code is verified, and abnormal frames with wrong synchronous codes are removed;
third, the data preprocessing component, according to the data format T 2 Counting frame loss and frame error conditions by utilizing information such as frame numbers, data lengths and the like, and extracting an N-byte effective data area;
fourth step, in frame format T 3 Based on this, T is found in the effective data area extracted in the third step 3 According to the frame synchronization code, effective information such as frame number, data length and the like is searched in sequence, and T is spliced 3 A format data frame;
fifth step, at T 3 Extracting data indicating bits from the format data frames, dividing the data frames into image related data frames or telemetry parameter related data frames, and pushing the data frames to an image processing plug-in unit and a telemetry parameter analysis plug-in unit of a data processing assembly respectively according to the data frame types;
sixth step, image processing plug-in, receive T 3 Image-related data frames in frame format according to T 3 The method comprises the steps of extracting a K byte effective data area, dividing data according to a data type identification bit, enabling multiple threads to process each image coding data stream in parallel, sending the effective data stream to a video or image decoder according to a data domain coding mode by each thread, analyzing and generating image data, and pushing the image data to a downstream data application component and a data storage component;
seventh step, the telemetering parameter analysis plug-in receives T 3 Telemetry parameter related data frames in frame format according to T 3 Frame format, extracting effective data field, dividing telemetry parameter type according to parameter packet identification bit, splicing telemetry parameter packet, resolving binary data stream into telemetry parameter with actual physical meaning according to agreed parameter format, and pushing to downstream data application component and data storageA storage assembly;
and eighth step, the data application component receives analysis results such as images, telemetry parameters and the like, performs secondary processing, and is used for supporting functions such as dynamic scene visual display, data utility evaluation, satellite fault diagnosis and the like.
And ninth, the data storage component is used for classifying and storing the original data, the intermediate data, the image, the telemetry parameter analysis result and the like in the data processing process and supporting query and downloading in a plurality of modes such as serial numbers, data sources, data types, on-board time and the like.
The multi-source data processing system provided by the disclosure is based on the single satellite data source data processing flow, and aims at the practical scene that a plurality of data sources send data streams in parallel, and the multi-source data processing flow is closed by itself through state monitoring and centralized flow scheduling.
Set the ground station set for receiving data of a certain task of a certain satellite as { ST } 1 ，ST 2 ，ST 3 ,. the set of frequency bands supported by each ground station device is { CH } l ，CH 2 ，CH 3 ，...}。
The satellite downloading data rate is known to be related to the frequency band information of each ground station data receiving device, the single frequency band supports to receive the downlink data with multiple rates, and the frequency band information and the downloading rate are in one-to-many relation, so that the frequency band CH can be set i Is provided with a receiving device
And the like, and the capability of various data downlink rates. In addition, for any ground station ST j The start time of the data receiving arc of the mth turn is recorded as +.>
The end time is recorded as->
In the present disclosure, a data receiving module enables multiple threads for data reception and distribution for binary data streams sent by data sources in various directions. Wherein each data source, namely each frequency band of each ground station, corresponds to an independent receiving and distributing thread for receiving and distributing the data stream in the direction. Each receiving and distributing thread establishes a one-to-one mapping relation with the subsequent processing flow through the ground station information, and ensures that the data streams in all directions are processed independently and are not interfered with each other.
Suppose that the data receiving plan of the mth round of a certain task of a certain satellite is
I.e. three ground stations are planned for local data reception in total, with ground station ST 1 For example, ground station ST 1 Reception is taken in->
CH for rate download 1 Frequency band data, receiving arc section is +>
According to the actual receiving situation, assuming that three ground stations normally work in the current circle, the data receiving module expects to establish three independent threads { rth } 1 ，rth 2 ，rth 3 And data receiving and distributing in three directions.
In the disclosure, a state monitoring module starts a plurality of threads for counting data quality of received data and transmission link state information, including information such as data source, frequency band information, data type, receiving rate, frame loss rate, receiving time, link connection state and the like, aiming at binary data streams sent by data sources in all directions, monitors the data receiving condition in all directions, and provides data state information basis for a flow scheduling module.
Taking the data receiving plan of the mth turn of a certain satellite as an example, it is assumed that the three directions of the turn are respectively transmitted by the ground station ST 1 ，ST 2 ，ST 3 The state monitoring module expects to build three independent threads { sth }, issue 1 ，sth 2 ，sth 3 For monitoring the data reception quality and link state in various directionsInformation.
In the disclosure, a flow scheduling module acquires data receiving quality and transmission link state information of each direction, synthesizes a resource scheduling table and a residual resource table, comprehensively allocates computing and storage resources, and provides reasonable operation computing power and storage space for data processing flows of each direction. By sending the scheduling instruction, a corresponding data processing flow, namely, the data processing flow for a single data source as described in fig. 4, is started.
As shown in table 1, the resource scheduling table presets reasonable computing resources and storage space for each software in the data processing flow shown in fig. 4 for different conditions, that is, different downloading rates of different frequency band data, and stores each mapping relation. For example, the data processing flow is composed of a plurality of software, which can be expressed as { soft } 1 ，soft 2 ，soft 3 ,..}, treatment is carried out using
CH for rate download 1 Software soft when frequency band data 1 Need to occupy m 1 CPU and n 1 GB memory to ensure optimal software performance.
TABLE 1
The remaining resources table, as described in table 2, is used to dynamically record the idle cpu and memory resource conditions of all the computing nodes. Assuming that there are multiple compute nodes available for data processing for the task of the star, this can be denoted as { s } 1 ，s 2 ，s 3 … }, with node s 1 For example, the currently remaining available computing resources are: l (L) 1 CPU, r 1 GB memory. When the flow scheduling module performs soft processing on the software 1 After being allocated to the computing node and successfully started, the remaining available computing resources of the computing node are updated as follows: (l) 1 -m 1 ) CPU, (r) 1 -n 1 ) GB memory.
TABLE 2
Data reception plan for the mth round of a certain task of the certain star
If the data transmission of each ground station is normal, the data processing system expects to initiate three processing flows +.>
Corresponding to this. From ground station ST 1 Is +.>
The status monitoring module is->
Immediately sending a notification message to a flow scheduling module after the direction data is monitored at any time, wherein the flow scheduling module acquires the state monitoring information of the direction and the state monitoring information of the direction according to a data frequency band CH 1 And download Rate->
Inquiring a resource scheduling table to obtain calculation resources required by each software under the working condition, inquiring a residual resource table to obtain current available resources, and starting a corresponding data processing flow P according to the ground station information 1 And reasonably distributing the software to each computing node. If instruction scheduling and software start time are ignored, the process flow P 1 Is +.>
In order to avoid the waste of computing resources, the process scheduling module dynamically adjusts the management of the process, and after the data stops being received, the process is closed and the computing resources are released. Assuming that the direction data is stopped from being received T max If no data reenters within the time, the direction data is considered to be trackedAnd finally, the flow scheduling module sends an instruction to stop the data processing flow in the direction, and corresponding computing resources are released, so that the flow is +.>
Wherein T is max The specific numerical values are flexible and configurable, and can be reasonably set according to actual tasks, so that smooth receiving and processing of data are ensured.
In the disclosure, the data processing module, that is, the single data source satellite data processing flow shown in fig. 2, is configured to receive and process binary data streams of a single data source, and each software in the flow cooperates with each other to analyze data step by step, so as to generate an analysis result and store and display the analysis result.
Fig. 5 schematically illustrates a detailed flow chart of a data-driven multi-source data processing flow provided by an embodiment of the present disclosure.
As shown in fig. 5, the multi-source data processing flow described in one embodiment is as follows:
the first step is to obtain the link state information of each direction, wherein the link information state information includes the source of the link data (i.e. the ground station information), the frequency band, the data type, the data receiving rate, the data transmission quality, the link connection state, etc.
And step two, judging whether a data stream is accessed to the system currently according to the link state information of each direction.
And thirdly, if the data stream is accessed to the system currently, the system acquires data source information, namely ground station information, according to the directional link state information.
And fourthly, if no data stream access system exists currently, repeating the first step to the second step, namely continuously waiting for the data transmission data access systems in all directions until a data stream in a certain direction is received.
And fifthly, after receiving the data transmission data stream in a certain direction, acquiring the link state information in the direction, and extracting the ground station information from the link state information.
And sixthly, judging whether a data processing flow related to the ground station exists in the system at the current moment.
Seventh, if the data processing flow related to the ground station in the fifth step does not exist in the current system, starting the data processing flow for the ground station according to the ground station information, and generating a unique flow number according to the time information.
And eighth, if the data processing flow related to the ground station in the fifth step exists in the current system, directly entering the ninth step.
And ninth, accessing a data stream, wherein all software in the system cooperate with each other, and the data is processed step by step.
And tenth, judging whether the directional data flow is interrupted in the fifth step.
And eleventh, if the directional data flow in the fifth step is continuously and stably accessed into the system, data processing is normally performed.
Twelfth, if the direction data stream is interrupted in the fifth step, but the interruption time does not exceed the maximum waiting time Tmax set by the system, repeating the second to tenth steps.
Thirteenth, if the data stream is interrupted in the direction in the fifth step and the data interruption time exceeds the maximum waiting time Tmax set by the system, the data processing flow related to the direction is stopped.
It should be noted that, for the sake of simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the present disclosure is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present disclosure. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all necessary for the present disclosure.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.
The foregoing is illustrative of a multi-source data processing system and method provided by the present disclosure, and is not to be construed as limiting the present disclosure in view of the many variations in detailed implementation and application scope that may be apparent to those of ordinary skill in the art in light of the teachings of the embodiments of the present disclosure.
Claims (10)
1. A multi-source data processing system, the multi-source data processing system comprising:
the data receiving module is used for receiving binary data frames sent by the data sources in multiple directions in real time in parallel, and sending the data frames to the corresponding data processing module in a shunting way according to the data sources and/or the data types;
the state monitoring module is used for monitoring the data state information and the link state information of the data frames sent by the data sources in all directions;
the flow scheduling module is used for dynamically scheduling all the data processing modules according to at least one of the link state information of the data sources in all the directions, the data state information and the computing node resource conditions;
and the data processing modules are used for receiving and processing binary data frames sent by the unidirectional data source, calling the analysis plug-in according to the data frames and generating analysis results.
2. The multi-source data processing system of claim 1, wherein the data processing module comprises:
the data decryption component is used for receiving the original ciphertext data, analyzing the original ciphertext data according to the pre-annotating key, completing synchronous discrimination of the data, decrypting the effective area, converting the original ciphertext data into plaintext data and distributing the plaintext data;
the data preprocessing component is used for extracting an effective data area of a data frame and distributing the data frame to the data processing component according to a data indicating bit, wherein the data indicating bit is used for identifying the effective data area as image processing data or telemetry parameter coding data;
the data processing component is used for processing the data frame according to the data indicating bit.
3. The multi-source data processing system of claim 2, wherein the data processing component comprises an image processing plug-in;
the image processing plug-in is used for extracting the image type identification bit of the data frame, pushing the data frame to a corresponding decoder according to the image type identification bit to perform image analysis, and obtaining an image analysis result.
4. The multi-source data processing system of claim 2, wherein the data processing component comprises a telemetry parameter resolution plug-in;
the telemetry parameter analysis plug-in is used for converting the original telemetry code contained in the data frame into telemetry parameters with the preset parameter format according to the default parameter format.
5. The multi-source data processing system of claim 1, wherein the data processing module further comprises:
the data application component is used for receiving the image analysis result and the telemetry parameter which are pushed by the data processing component, and carrying out visual display and/or secondary analysis and/or utility evaluation and/or fault diagnosis on the image analysis result and the telemetry parameter.
6. The multi-source data processing system of claim 1, wherein the data processing module further comprises:
the data storage component is used for receiving the data frame sent by the data preprocessing component, the image analysis result and the telemetry parameter sent by the data processing component, storing the data frame, the image analysis result and the telemetry parameter, and storing in an associated mode according to type and/or time and/or source.
7. The multi-source data processing system of claim 1, wherein the data status information includes at least one of a data source, a data type, a data reception rate, a data continuity, a data frame loss rate, a reception start-stop time, a link-up status.
8. The multi-source data processing system of claim 1, wherein the link state information includes at least one of a link direction, a frequency band, a data type, a data reception rate, a data transmission quality, a link connectivity status.
9. A method of multi-source data processing, the method comprising:
receiving binary data frames sent by a plurality of data sources in real time in parallel, and monitoring data state information and link state information of the data frames sent by the data sources in the plurality of directions;
dynamically scheduling all data processing modules according to at least one of the link state information, the data state information and the computing node resource conditions of the plurality of directional data sources;
and sending the data frames to the corresponding data processing modules in a shunting manner according to the data source and/or the data type, wherein each data processing module is used for receiving binary data frames sent by the data source in a single direction and generating an analysis result according to the data frames.
10. The multi-source data processing method of claim 9, wherein the data processing module comprises a data preprocessing component and a data processing component, the method further comprising:
extracting an effective data area of a data frame by using the data preprocessing component, and distributing the data frame to the data processing component according to a data indicating bit, wherein the data indicating bit is used for identifying the effective data area as image processing data or telemetry parameter coding data;
and processing the data frame by utilizing the data processing component according to the data indicating bit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310161458.4A CN116155362B (en) | 2023-02-23 | 2023-02-23 | Multi-source data processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310161458.4A CN116155362B (en) | 2023-02-23 | 2023-02-23 | Multi-source data processing system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116155362A true CN116155362A (en) | 2023-05-23 |
| CN116155362B CN116155362B (en) | 2024-07-16 |
Family
ID=86350456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310161458.4A Active CN116155362B (en) | 2023-02-23 | 2023-02-23 | Multi-source data processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116155362B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116827411A (en) * | 2023-06-27 | 2023-09-29 | 北京和德宇航技术有限公司 | Load data analysis method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6836658B1 (en) * | 2000-03-03 | 2004-12-28 | Ems Technologies, Inc. | High data rate satellite communications system and method |
| CN102904834A (en) * | 2012-09-10 | 2013-01-30 | 中国航天科技集团公司第五研究院第五一三研究所 | Satellite-bone data processing system based on advanced orbiting system (AOS) |
| WO2015074584A1 (en) * | 2013-11-20 | 2015-05-28 | 中国石油天然气集团公司 | Automatic inspection and monitoring method based on time domain slotting control |
| CN110049020A (en) * | 2019-03-27 | 2019-07-23 | 中国科学院国家空间科学中心 | A kind of CCSDS AOS protocol data real-time processing method and system |
| CN115633057A (en) * | 2022-10-09 | 2023-01-20 | 陕西航天技术应用研究院有限公司 | Cloud platform-based measurement, operation and control data processing system |
-
2023
- 2023-02-23 CN CN202310161458.4A patent/CN116155362B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6836658B1 (en) * | 2000-03-03 | 2004-12-28 | Ems Technologies, Inc. | High data rate satellite communications system and method |
| CN102904834A (en) * | 2012-09-10 | 2013-01-30 | 中国航天科技集团公司第五研究院第五一三研究所 | Satellite-bone data processing system based on advanced orbiting system (AOS) |
| WO2015074584A1 (en) * | 2013-11-20 | 2015-05-28 | 中国石油天然气集团公司 | Automatic inspection and monitoring method based on time domain slotting control |
| CN110049020A (en) * | 2019-03-27 | 2019-07-23 | 中国科学院国家空间科学中心 | A kind of CCSDS AOS protocol data real-time processing method and system |
| CN115633057A (en) * | 2022-10-09 | 2023-01-20 | 陕西航天技术应用研究院有限公司 | Cloud platform-based measurement, operation and control data processing system |
Non-Patent Citations (5)
| Title |
|---|
| XINGYU ZHOU: "Active Terminal Identification, Channel Estimation, and Signal Detection for Grant-Free NOMA-OTFS in LEO Satellite Internet-of-Things", 《IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS》, 25 October 2022 (2022-10-25) * |
| 冯阳;张国强;: "基于工作流技术的遥感卫星数据接收调度系统的设计与实现", 无线电工程, no. 11, 19 October 2018 (2018-10-19) * |
| 张文彬;王春梅;王静;陈托;智佳;: "基于Spark的有效载荷参数解析处理方法", 计算机工程与设计, no. 02, 16 February 2020 (2020-02-16) * |
| 李犟: "高速卫星遥测数据处理系统的设计方案", 《计算机工程与设计》, 16 January 2012 (2012-01-16) * |
| 杨胜哲;于俊清;唐九飞;: "数据流程序动态调度与优化", 计算机工程与科学, no. 07, 15 July 2017 (2017-07-15) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116827411A (en) * | 2023-06-27 | 2023-09-29 | 北京和德宇航技术有限公司 | Load data analysis method and device, electronic equipment and storage medium |
| CN116827411B (en) * | 2023-06-27 | 2024-03-01 | 北京和德宇航技术有限公司 | Load data analysis method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116155362B (en) | 2024-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10951340B2 (en) | Method and device for transmitting bit error rate information in FlexE overhead frame, and computer storage medium | |
| CN116155362B (en) | Multi-source data processing system and method | |
| CN102480335B (en) | A kind of sending method of business datum and system | |
| CN110287041B (en) | Service data sending method, device, equipment and storage medium | |
| CN104935499A (en) | Message format conversion method and equipment | |
| US10003669B2 (en) | Methods and apparatus to create and transmit a condensed logging data file | |
| CN104219298A (en) | Cluster system and data backup method thereof | |
| CN112953757A (en) | Data distribution method, system and computer equipment | |
| CN103795980A (en) | Cascading video device and data processing method thereof | |
| US7072348B2 (en) | System and method for in-service reconfiguration of a synchronous optical communications network | |
| CN103780325B (en) | A kind of satellite transmission purpose monitoring system | |
| CN113381889B (en) | Network slice determination method and device, electronic equipment and storage medium | |
| US20140068048A1 (en) | Managing remote devices | |
| CN112615909B (en) | Method for storing data in cascade storage server cluster and related equipment | |
| JP2009260652A (en) | Radio communication system | |
| CN104850020A (en) | Data acquisition method and apparatus of converter | |
| CN114629917B (en) | Data processing method and device for cross-system communication and electronic equipment | |
| CN108289165B (en) | Method and device for realizing camera control based on mobile phone and terminal equipment | |
| CN113824649B (en) | Data flow control device for fixed frame length | |
| CN113364869B (en) | Block chain message transmission method, equipment and storage medium | |
| CN114785854A (en) | Service request processing method, device, equipment, storage medium and product | |
| CN111147083B (en) | Method for compressing and transmitting time information | |
| CN109617985B (en) | Method and system for processing data by concurrent MQTT gateway | |
| KR101505139B1 (en) | Telemetry frame generating method and satelite monitoring method by the same | |
| KR101764652B1 (en) | Remote contact monitoring apparatus and method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |