CN115833913A - Multi-source heterogeneous space-based data acquisition method and system based on message driving mode - Google Patents
Multi-source heterogeneous space-based data acquisition method and system based on message driving mode Download PDFInfo
- Publication number
- CN115833913A CN115833913A CN202211469130.0A CN202211469130A CN115833913A CN 115833913 A CN115833913 A CN 115833913A CN 202211469130 A CN202211469130 A CN 202211469130A CN 115833913 A CN115833913 A CN 115833913A
- Authority
- CN
- China
- Prior art keywords
- data
- space
- module
- processing module
- acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention relates to a multi-source heterogeneous space-based data acquisition method and system based on a message driving mode, wherein the method comprises the following steps: planning an acquisition plan according to the requirement on the space-based data by using a data acquisition message processing module, and issuing a data requirement plan; receiving the data demand plan by using a data monitoring and sending module, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, multithread monitoring of newly added space-based data according to a monitoring mechanism which is found to be accessed, and sending monitored space-based data information to a data scheduling and processing module in an information-driven mode; and uniformly scheduling the acquired various types of space-based data by using the data scheduling processing module, and feeding back the processed normalized multi-source data response to the data acquisition message processing module. The invention can realize the automatic access of the space-based data of various types of satellites with different data types, formats, precisions and time delays and various network segments of various loads.
Description
Technical Field
The invention relates to the technical field of space-based data acquisition, in particular to a multi-source heterogeneous space-based data acquisition method and system based on a message driving mode.
Background
The space-based data is used as an important component of the integrated information network, and has the characteristics of high updating speed and large data volume. The data sources are complex and various, and the network systems are different, so that great difficulty is brought to the collection and aggregation of the data. Different space-based data are complex and diverse in source, different in timeliness and stability, and cannot be elastically acquired according to the characteristics of each type of data.
The data-driven acquisition mode is usually realized by shared storage, is greatly influenced by network fluctuation, and has the phenomenon of slow response of storage mounting after running for a certain time, so that data cannot be acquired and accessed in time, or the situation of instantaneous concurrent access of backlog data occurs, and huge impact is caused to a rear-end data processing platform.
In engineering practice, the construction of space-based data acquisition capability has several prominent problems:
(1) In the prior art, the physical isolation for data acquisition among different network systems is not high, data mixing and penetration are easy to cause, and the security of a data source is threatened;
(2) The stability of the data monitoring driving mechanism is poor, the response to a large amount of instantly acquired data is slow, and even response congestion paralysis can be caused, so that data are lost;
(3) The robustness of information interaction in the data acquisition process is low, and information exchange cannot be presented in a closed-loop mode;
(4) The data acquisition system lacks expansibility and compatibility, and the flexible upgrading capability of the data acquisition system is weaker along with the increase of the types of the accessible space-based information.
Disclosure of Invention
In order to solve the technical problems, the invention provides a multi-source heterogeneous space-based data acquisition method and system based on a message driving mode, so as to improve the acquisition capacity of space-based data with complex and diverse sources and different types in the same network segment or different network segments.
In order to achieve the purpose, the technical scheme of the invention is as follows:
in a first aspect, the present invention provides a multi-source heterogeneous space-based data acquisition method based on a message-driven mode, including:
planning an acquisition plan according to the requirement on the space-based data by using a data acquisition message processing module, and issuing a data requirement plan;
receiving the data demand plan by using a data monitoring and sending module, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, multithread monitoring of newly added space-based data according to a monitoring mechanism which is found to be accessed, and sending monitored space-based data information to a data scheduling and processing module in an information-driven mode;
and uniformly scheduling the acquired various types of space-based data by using the data scheduling processing module, and feeding back the processed normalized multi-source data response to the data acquisition message processing module.
According to one aspect of the invention, the data requirements include: data sources and data requirements thereof in the same network segment and data sources and data requirements thereof in different network segments.
According to one aspect of the invention, in the process of planning a data acquisition plan according to the requirement of space-based data and issuing a data requirement planning by using a data acquisition message processing module, the method comprises the following steps:
calling a first network switching device to transmit the data sources of different network segments and the data requirements thereof sent by the data acquisition message processing module to the data monitoring and sending module;
and directly transmitting the data sources and the data requirements thereof in the same network segment to the data monitoring and transmitting module by using the data acquisition message processing module.
According to an aspect of the present invention, in a process of receiving the data demand plan by using a data monitoring and sending module, acquiring a corresponding data source according to a collection type, executing a data collection task, multithread-monitoring newly added space-based data according to a monitoring mechanism that is found to be accessed, and sending a monitored space-based data message to a data scheduling and processing module in a message-driven mode, the method includes:
analyzing the message protocol and the file name of the monitored space-based data;
comparing the monitored file information with the monitored file information by using historical acquisition data of a system cache and a database, if the file information is the same, not acquiring the monitored file to the data scheduling processing module, and otherwise, monitoring the file and acquiring the monitored file to the data scheduling processing module;
and automatically identifying whether cross-network segment transmission is required or not according to the planning and distinguishing data service requirement.
According to one aspect of the invention, the unified scheduling processing of the acquired various types of space-based data by using the data scheduling processing module comprises:
setting a data acquisition strategy by using a mode selection module according to the space-based data acquisition service requirement;
receiving a data driving message by using a data configuration module, and carrying out hierarchical monitoring according to multi-source space-based data of different levels;
the breakpoint continuous transmission module is utilized to pause or continue to receive the space-based data at any time in the transmission process of the space-based data;
identifying and removing the redundant data generated in the data acquisition process by using a data removing module;
setting a clearing strategy according to different acquisition service requirements by using a timing clearing module, and clearing redundant data at regular time;
the quality inspection module is used for inspecting the quality and the normalization of the acquired data and rejecting invalid data;
performing time screening and space screening on the effective data by using a time-space screening module, and selectively accessing the data meeting the preset time-space conditions;
setting the type of the access data by using a type screening module and selecting the type;
the format conversion module is used for automatically matching the data types and uniformly converting the formats and the specifications of the data;
and the converted standard data is subjected to filing storage by utilizing a filing storage module according to a template corresponding to data type matching, stored data information is recorded, and the redundant data is subjected to timing cleaning or one-time cleaning of historical redundant data to obtain the normalized multi-source space-based data.
According to one aspect of the present invention, in the process of uniformly scheduling and processing the acquired various types of space-based data by using the data scheduling and processing module and feeding back the processed normalized multi-source data response to the data acquisition message processing module, the method includes:
after the data monitoring and sending module receives the data demand planning, calling a second network switching device and feeding back the normalized multi-source data responses of different network segments processed by the data scheduling processing module to the data acquisition message processing module through a channel of the second network switching device;
and after the data monitoring and sending module receives the data demand planning, the data monitoring and sending module directly feeds back the normalized multi-source data response of the same network segment to the data acquisition message processing module through the data scheduling processing module.
According to one aspect of the invention, the normalized multi-source data response comprises: and (4) normalized multi-source space-based data and scheduling processing process information.
In a second aspect, the present invention further provides a multi-source heterogeneous sky-based data acquisition system for implementing the multi-source heterogeneous sky-based data acquisition method based on the message-driven mode, including:
the data acquisition message processing module is used for planning an acquisition plan according to the requirement on the space-based data and issuing a data requirement plan;
the data monitoring and sending module is used for receiving the data demand plan, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, multithread monitoring of newly added space-based data according to a monitoring mechanism which is found to be accessed, and sending monitored space-based data information to the data scheduling and processing module in an information-driven mode; and
and the data scheduling processing module is used for uniformly scheduling and processing the acquired various types of space-based data and feeding back the processed normalized multi-source data response to the data acquisition message processing module.
According to another aspect of the present invention, the data scheduling processing module includes:
the mode selection module is used for setting a data acquisition strategy according to the space-based data acquisition service requirement;
the data configuration module is used for receiving the data driving message and carrying out hierarchical monitoring according to the multi-source space-based data of different levels;
the breakpoint continuous transmission module is used for pausing, terminating or continuously receiving the space-based data at any time in the transmission process of the space-based data;
the data deduplication module is used for identifying and deduplicating redundant data generated in the data acquisition process;
the timing cleaning module is used for setting a cleaning strategy according to different acquisition service requirements and cleaning redundant data at regular time;
the quality inspection module is used for inspecting the quality and the normalization of the acquired data and eliminating invalid data;
the space-time screening module is used for carrying out time screening and space screening on the effective data and selectively accessing the data meeting the preset space-time condition;
the type screening module is used for setting and selecting the type of the access data;
the format conversion module is used for automatically matching data types and uniformly converting the format and the specification of the data; and
and the filing storage module is used for filing and storing the converted standard data according to a template corresponding to the data type matching, recording the stored data information, and carrying out timing cleaning or one-time cleaning on the redundant data to obtain the normalized multi-source space-based data.
According to another aspect of the present invention, further comprising: the first network switching equipment is used for transmitting the data sources of different network segments sent by the data acquisition message processing module and the data requirements thereof to the data monitoring and sending module; and
and the second network switching equipment is used for feeding back the normalized multi-source data responses of different network segments processed by the data scheduling processing module to the data acquisition message processing module through a channel of the second network switching equipment after the data monitoring and sending module receives the data demand project.
Compared with the prior art, the invention has the following beneficial effects:
according to the scheme of the invention, the multi-source heterogeneous space-based data acquisition method and system under the message driving mode are provided, and the space-based data of different types and complicated and various sources are stably acquired by the message driving and closed-loop processing method, so that the automatic access of the space-based data of various satellites and various loads with different data types, formats, accuracies and time delays is realized.
According to one scheme of the invention, automatic cross-network access can be realized for space-based data of each network segment, and the data receiving, processing, storing and displaying can be realized in a message driving mode, thereby solving the problem of the stability influence on shared storage caused by network fluctuation.
According to one scheme of the invention, multithread monitoring, data synchronous transmission and processing including data format identification, screening, conversion, duplication elimination and the like are carried out on space-based data according to data sources and categories, so that the operation efficiency of the system is improved, and the continuous and uninterrupted stable operation of the system can be realized.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 schematically shows a flowchart of a multi-source heterogeneous space-based data acquisition method based on a message-driven mode according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating a network architecture of a multi-source heterogeneous space-based data acquisition system based on a message-driven mode according to an embodiment of the present invention;
fig. 3 is a flowchart schematically illustrating a process of a data scheduling processing module according to an embodiment of the present invention.
Detailed Description
The description of the embodiments of this specification should be taken in conjunction with the accompanying drawings, which are to be considered part of the entire written description. In the drawings, the shape or thickness of the embodiments may be exaggerated and simplified or conveniently indicated. Further, the components of the structures in the drawings are described separately, and it should be noted that the components not shown or described in the drawings are well known to those skilled in the art.
Any reference to directions and orientations to the description of the embodiments herein is merely for convenience of description and should not be construed as limiting the scope of the invention in any way. The following description of the preferred embodiments refers to combinations of features which may be present independently or in combination, and the present invention is not particularly limited to the preferred embodiments. The scope of the invention is defined by the claims.
As shown in fig. 1, the multi-source heterogeneous space-based data acquisition method based on the message-driven mode disclosed in the embodiment of the present invention includes the following steps:
and step 100, planning an acquisition plan according to the requirement on the space-based data by using a data acquisition message processing module, and issuing a data requirement plan.
Further, in this embodiment, the data source is divided into the data source in the same network segment and the data source in different network segments according to the location of the data source. Thus, the data requirements include: data sources and data requirements thereof in the same network segment and data sources and data requirements thereof in different network segments. Aiming at multi-source space-based data in different network segments, the data source and the data application service system are not in the same network segment, and the data are various in types and high in timeliness requirement. Compared with a data source network, the data application service system network has different access authority requirements, automatic data circulation is realized by additionally arranging the unidirectional network switching equipment, and the unicity of data flow direction and the integrity of message interaction are kept. Aiming at multi-source space-based data in the same network segment, the data acquired and monitored in the network segment is various in types, inconsistent in naming rule and data format, and high in requirements for real-time performance and safety of data acquisition. For different data source file paths in the same network segment, the embodiment of the invention adopts a multi-thread task mode, ensures the peak processing capacity, finds data, namely point-to-point access, drives the acquisition response through messages and feeds back the acquisition state. The embodiment of the invention adopts a configurable design, supports the increase and change of data types, does not depend on surface layer information such as file names, file suffixes and the like to automatically identify the data types, cleans historical redundant data at regular time, and prevents excessive long-term running data.
Then, in the step 100, a data acquisition plan is planned by using the data acquisition message processing module according to the requirement for space-based data, and a specific implementation process of data requirement planning is issued, and for the situation of data sources of different network segments, a first network switching device is invoked to transmit the data source and the data requirement thereof issued by the data acquisition message processing module to the data monitoring and sending module. And aiming at the condition of the data sources in the same network segment, the data acquisition message processing module is utilized to directly transmit the data sources in the same network segment and the data requirements thereof to the data monitoring and transmitting module.
And 200, receiving the data demand plan by using a data monitoring and sending module, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, performing multi-thread monitoring on newly added space-based data according to a monitoring mechanism which is found to be accessed, and sending monitored space-based data information to a data scheduling and processing module in an information-driven mode.
In this embodiment, the data monitoring and sending module is used to receive the data demand plan, respond to events that change the original state, such as new increase of files of data and messages, change of file content or size, etc., monitor new space-based data in a multithread manner according to a monitoring mechanism that is discovered, that is, accessed, and send the monitored space-based data messages to the data scheduling and processing module in a message-driven mode, so that the emergency situations of network fluctuation and data surge can be solved.
In some embodiments, in the step 200, the data monitoring and sending module is used to receive the data requirement plan, obtain a corresponding data source according to the collection type, execute a data collection task, monitor newly added space-based data in a multithread manner according to a monitoring mechanism that is found to be accessed, send the monitored space-based data message to the data scheduling and processing module in a message-driven mode, analyze a message protocol and a file name of the monitored space-based data, and issue the analyzed collection type and information to the corresponding data source to execute the data collection task. And then comparing the monitored file information with the monitored file information by using historical data collected by a system cache and a database, if the file information is the same, not collecting the monitored file to the data scheduling processing module, and otherwise, monitoring the file and collecting the monitored file to the data scheduling processing module. And automatically identifying whether cross-network segment transmission is needed or not according to the project distinguishing data service requirement.
And 300, uniformly scheduling the acquired various types of space-based data by using the data scheduling processing module, and feeding back the processed normalized multi-source data response to the data acquisition message processing module.
In some embodiments, as shown in fig. 3, the specific implementation process of performing unified scheduling processing on the various acquired space-based data by using the data scheduling processing module in step 300 includes:
the logic of the scheduling processing is mainly planned according to data requirements, receives various types of space-based data in various network segments sent by the data monitoring and sending module, and sets a data acquisition strategy by using a mode selection module according to the service requirements of space-based data acquisition; receiving a data driving message by using a data configuration module, and carrying out hierarchical monitoring according to multi-source space-based data of different levels; in the transmission process of the space-based data, the breakpoint continuous transmission module is used for pausing or continuously receiving the space-based data at any time; in order to save network and storage resources, when repeated data are generated in the acquisition process, the repeated data and historical data are collectively called as redundant data, and the redundant data are identified and deduplicated by using a data deduplication module according to a certain strategy; setting a clearing strategy according to different acquisition service requirements by using a timing clearing module, and clearing redundant data at regular time; then, a quality inspection module is used for inspecting the quality and normalization of the acquired data and eliminating invalid data; performing time screening and space screening on the effective data by using a time-space screening module, and selectively accessing data meeting preset time-space conditions; setting the type of the access data by using a type screening module and selecting the type; the format conversion module is used for automatically matching the data types and uniformly converting the formats and the specifications of the data; and then, the converted standard data is subjected to filing storage by using a filing storage module according to a template corresponding to data type matching, stored data information is recorded, and the redundant data is subjected to timing cleaning or one-time cleaning of historical redundant data to obtain normalized multi-source space-based data.
In this embodiment, in the step 300, the data scheduling processing module is used to perform unified scheduling processing on the acquired various types of space-based data, and in a specific implementation process of feeding back the processed normalized multi-source data response to the data acquisition message processing module, for the conditions of data sources in different network segments, the data monitoring and sending module receives the data demand planning, and then calls the second network switching device and feeds back the processed normalized multi-source data response to the data acquisition message processing module through a channel of the second network switching device. And aiming at the condition of data sources of the same network segment, after the data monitoring and sending module receives the data demand planning, the data monitoring and sending module directly feeds back a standardized multi-source data response to the data acquisition message processing module through the data scheduling processing module.
Further, the normalized multi-source data response comprises: normalized multi-source space-based data and scheduling process messages. In this embodiment, after the acquired space-based data is cleaned according to the scheduling processing logic, the normalized multi-source space-based data and the scheduling processing procedure message are sent to the data acquisition message processing module, and the data acquisition message processing module performs closed-loop processing.
The design process of the multi-source heterogeneous space-based data acquisition method of the embodiment implements universalization, systematization and modularization ideas, a universal platform is constructed, the consistency and standardization of interfaces are ensured, and the acquisition capacity of multi-source space-based data is improved to the maximum extent. Based on a highly automatic principle, all the acquisition modules are uniformly dispatched to work cooperatively through a task message driving mode of centralized monitoring, and closed-loop processing of message interaction can be realized even among different networks containing physical isolation, so that the effect of stable and efficient acquisition of multi-source space-based data is achieved.
The embodiment of the invention also discloses a multi-source heterogeneous sky-based data acquisition system for executing the multi-source heterogeneous sky-based data acquisition method based on the message driving mode, which adopts a method of source control and distributed acquisition and deployment around the acquisition process from 'data demand planning and sending' to 'standardized multi-source data response', as shown in figure 2, and the system comprises a data acquisition message processing module, a data monitoring and sending module and a data scheduling processing module.
The data acquisition message processing module is used for planning an acquisition plan according to the demand for space-based data and issuing a data demand plan; the data monitoring and sending module is used for receiving the data demand plan, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, monitoring newly-added space-based data in a multithread mode according to a monitoring mechanism which is found to be accessed, and sending the monitored space-based data information to the data scheduling and processing module in an information-driven mode; and the data scheduling processing module is used for uniformly scheduling the acquired various types of space-based data and feeding back the processed normalized multi-source data response to the data acquisition message processing module. Meanwhile, the data monitoring and sending module is also used for task management, real-time monitoring of task execution state and tracking closed-loop processing, and collecting and monitoring the state of software and hardware equipment and the state of a data source of the system.
Corresponding to the specific process of the data scheduling processing module shown in fig. 3 for scheduling and processing the collected multi-source space-based data of each network segment and each type, the data scheduling processing module includes the following 10 functional modules: the device comprises a mode selection module, a data configuration module, a breakpoint continuous transmission module, a data duplication removal module, a timing cleaning module, a quality inspection module, a type screening module, a format conversion module and a filing storage module. Wherein the content of the first and second substances,
the mode selection module is used for setting a data acquisition strategy according to different space-based data acquisition service requirements; the coordination problem of factors such as data multisource, timeliness, source control, cooperative scheduling and the like can be solved.
The data configuration module is used for receiving the data driving message and carrying out hierarchical monitoring according to the multi-source space-based data of different levels; and monitoring in different folders according to different types of day-based data. According to the types of the added space-based information, the flexible expansion of the data acquisition capacity can be realized.
The breakpoint continuous transmission module is used for pausing, terminating or continuously receiving the space-based data at any time in the transmission process of the space-based data; the process of data transmission archiving can be flexibly controlled.
The data deduplication module is used for identifying and deduplicating redundant data generated in the data acquisition process; specifically, the module judges deduplication according to a file name, a file size, a file MD5 value and the like, automatically identifies monitored repeated data, and judges the repeated data as redundant data. After the file name and the content are analyzed, the file name and the content are compared with a system cache and a historical database table, and the monitored file are selectively accessed.
The timing cleaning module is used for setting a cleaning strategy according to different acquisition service requirements and cleaning redundant data at regular time; wherein elements of the cleanup policy relate to time, data class, flow type, schema type, and the like.
The quality inspection module is used for inspecting the quality and the normalization of the acquired data and eliminating invalid data; specifically, the module checks naming rules and formats of normalized space-based data, checks integrity of the space-based data, recursively decompresses files in a compressed format, distinguishes compression types by reading byte headers when actual formats of some compressed data are inconsistent with suffix names, and archives the files according to categories.
The time-space screening module is used for carrying out time screening and space screening on the effective data and selectively accessing the data meeting the preset time-space conditions; specifically, data meeting preset space-time conditions are selectively accessed by setting a time window and defining an area range.
The type screening module is used for setting and selecting the type of the access data; further, other types of data which are not in the set range are temporarily not accessed, and the robust link resources can be kept. And after the type setting is cancelled, accessing.
The format conversion module is used for automatically matching data types and uniformly converting the format and the specification of the data; specifically, in order to facilitate the organization and management of the data by the subsequent application system, the module automatically matches the data types, and uniformly converts the naming formats, the data header files, the index files and the like of the multi-source sky-based data so as to improve the operating efficiency of the data application system.
And the filing storage module is used for filing and storing the converted standard data according to a template corresponding to data type matching, recording stored data information, and carrying out timing cleaning or one-time cleaning on the redundant data to obtain the normalized multi-source space-based data. Specifically, the module matches corresponding templates for archiving according to different types of IDs of multi-source data.
In the process of executing tasks by the data scheduling processing module, the task information of each node is recorded into a respective log file, and an alarm can be given in time aiming at abnormal conditions.
As shown in fig. 2, the network architecture of the multi-source heterogeneous space-based data acquisition system based on the message-driven mode disclosed in this embodiment further includes: a first network device and a second network device. The first network switching equipment is used for transmitting the data sources of different network segments sent by the data acquisition message processing module and the data requirements thereof to the data monitoring and sending module; and the second network switching equipment is used for feeding back the normalized multi-source data responses of different network segments processed by the data scheduling processing module to the data acquisition message processing module through a channel of the second network switching equipment after the data monitoring and sending module receives the data demand project.
In this embodiment, the system can collect data from a multi-source space-based data information platform, and realize collection of various space-based data information such as image data, signal data, measurement and feature data. According to the data category, the method can be divided into electronic data and imaging data, wherein the electronic data has more receiving types and quantity, a single file is smaller, and the source is wide; the imaging data receiving formats are different, the single data volume is large, and the real-time requirement is different. Through the methods of message driving and closed-loop processing, the space-based data of different types and sources are stably acquired, so that the automatic access of the space-based data of various types of satellites and various loads with different data types, formats, accuracies and time delays is realized. Meanwhile, multithreading monitoring, data synchronous transmission and processing are carried out according to the data source and the type, and continuous and uninterrupted stable operation of the system can be realized.
The sequence numbers of the above steps related to the method of the present invention do not mean the sequence of the execution of the method, and the execution sequence of each step should be determined by its function and inherent logic, and should not limit the implementation process of the embodiment of the present invention at all.
The above description is only exemplary of the present invention and should not be taken as limiting the invention, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A multi-source heterogeneous space-based data acquisition method based on a message-driven mode comprises the following steps:
planning an acquisition plan according to the requirement on the space-based data by using a data acquisition message processing module, and issuing a data requirement plan;
receiving the data demand plan by using a data monitoring and sending module, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, multithread monitoring of newly added space-based data according to a monitoring mechanism which is found to be accessed, and sending monitored space-based data information to a data scheduling and processing module in an information-driven mode;
and uniformly scheduling the acquired various types of space-based data by using the data scheduling processing module, and feeding back the processed normalized multi-source data response to the data acquisition message processing module.
2. The method of claim 1, wherein the data requirements comprise: data sources and data requirements thereof in the same network segment and data sources and data requirements thereof in different network segments.
3. The method of claim 2, wherein in staging a data collection plan based on a need for space-based data and issuing a data need staging using a data collection message processing module, comprising:
calling a first network switching device to transmit the data sources of different network segments and the data requirements thereof sent by the data acquisition message processing module to the data monitoring and sending module;
and the data acquisition message processing module is utilized to directly transmit the data sources in the same network segment and the data requirements thereof to the data monitoring and transmitting module.
4. The method of claim 2, wherein in the process of receiving the data demand plan by using a data monitoring and sending module, acquiring a corresponding data source according to a collection type, executing a data collection task, multithreading and monitoring newly added space-based data according to a monitoring mechanism that is found to be accessed, and sending the monitored space-based data message to a data scheduling processing module in a message-driven mode, the method comprises:
analyzing the message protocol and the file name of the monitored space-based data;
comparing the monitored file information with the monitored file information by using historical acquisition data of a system cache and a database, if the file information is the same, not acquiring the monitored file to the data scheduling processing module, and otherwise, monitoring the file and acquiring the monitored file to the data scheduling processing module;
and automatically identifying whether cross-network segment transmission is required or not according to the planning and distinguishing data service requirement.
5. The method according to claim 2, wherein the unified scheduling processing of the acquired various types of space-based data by using the data scheduling processing module comprises:
setting a data acquisition strategy by using a mode selection module according to the space-based data acquisition service requirement;
receiving a data driving message by using a data configuration module, and carrying out hierarchical monitoring according to multi-source space-based data of different levels;
the breakpoint continuous transmission module is utilized to pause or continue to receive the space-based data at any time in the transmission process of the space-based data;
identifying and removing the redundant data generated in the data acquisition process by using a data removing module;
setting a clearing strategy according to different acquisition service requirements by using a timing clearing module, and clearing redundant data at regular time;
the quality inspection module is used for inspecting the quality and the normalization of the acquired data and rejecting invalid data;
performing time screening and space screening on the effective data by using a time-space screening module, and selectively accessing the data meeting the preset time-space conditions;
setting the type of the access data by using a type screening module and selecting the type;
the format conversion module is used for automatically matching the data types and uniformly converting the formats and the specifications of the data;
and the converted standard data is subjected to filing storage by utilizing a filing storage module according to a template corresponding to data type matching, stored data information is recorded, and the redundant data is subjected to timing cleaning or one-time cleaning of historical redundant data to obtain the normalized multi-source space-based data.
6. The method according to claim 2, wherein in the process of uniformly scheduling the acquired various types of space-based data by using the data scheduling processing module and feeding back the processed normalized multi-source data response to the data acquisition message processing module, the method comprises:
after the data monitoring and sending module receives the data demand planning, calling a second network switching device and feeding back the normalized multi-source data responses of different network segments processed by the data scheduling processing module to the data acquisition message processing module through a channel of the second network switching device;
and after the data monitoring and sending module receives the data demand planning, the data monitoring and sending module directly feeds back the normalized multi-source data response of the same network segment to the data acquisition message processing module through the data scheduling processing module.
7. The method of any of claims 1-6, wherein the normalizing the multi-source data response comprises: normalized multi-source space-based data and scheduling process messages.
8. A multi-source heterogeneous sky-based data collection system for implementing the multi-source heterogeneous sky-based data collection method based on the message-driven mode according to any one of claims 1 to 7, comprising:
the data acquisition message processing module is used for planning an acquisition plan according to the requirement on the space-based data and issuing a data requirement plan;
the data monitoring and sending module is used for receiving the data demand plan, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, multithread monitoring of newly added space-based data according to a monitoring mechanism which is found to be accessed, and sending monitored space-based data information to the data scheduling and processing module in an information-driven mode; and
and the data scheduling processing module is used for uniformly scheduling and processing various acquired space-based data and feeding back the processed standardized multi-source data response to the data acquisition message processing module.
9. The system of claim 7, wherein the data scheduling processing module comprises:
the mode selection module is used for setting a data acquisition strategy according to the space-based data acquisition service requirement;
the data configuration module is used for receiving the data driving message and carrying out hierarchical monitoring according to the multi-source space-based data of different levels;
the breakpoint continuous transmission module is used for pausing, terminating or continuously receiving the space-based data at any time in the transmission process of the space-based data;
the data deduplication module is used for identifying and deduplicating redundant data generated in the data acquisition process;
the timing cleaning module is used for setting a cleaning strategy according to different acquisition service requirements and cleaning redundant data at regular time;
the quality inspection module is used for inspecting the quality and the normalization of the acquired data and eliminating invalid data;
the space-time screening module is used for carrying out time screening and space screening on the effective data and selectively accessing the data meeting the preset space-time condition;
the type screening module is used for setting and selecting the type of the access data;
the format conversion module is used for automatically matching data types and uniformly converting the format and the specification of the data; and
and the filing storage module is used for filing and storing the converted standard data according to a template corresponding to the data type matching, recording the stored data information, and carrying out timing cleaning or one-time cleaning on the redundant data to obtain the normalized multi-source space-based data.
10. The system of claim 7, further comprising: the first network switching equipment is used for transmitting the data sources of different network segments sent by the data acquisition message processing module and the data requirements thereof to the data monitoring and sending module; and
and the second network switching equipment is used for feeding back the normalized multi-source data responses of different network segments processed by the data scheduling processing module to the data acquisition message processing module through a channel of the second network switching equipment after the data monitoring and sending module receives the data demand project.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211469130.0A CN115833913A (en) | 2022-11-22 | 2022-11-22 | Multi-source heterogeneous space-based data acquisition method and system based on message driving mode |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211469130.0A CN115833913A (en) | 2022-11-22 | 2022-11-22 | Multi-source heterogeneous space-based data acquisition method and system based on message driving mode |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115833913A true CN115833913A (en) | 2023-03-21 |
Family
ID=85530366
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211469130.0A Pending CN115833913A (en) | 2022-11-22 | 2022-11-22 | Multi-source heterogeneous space-based data acquisition method and system based on message driving mode |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115833913A (en) |
-
2022
- 2022-11-22 CN CN202211469130.0A patent/CN115833913A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107943668B (en) | Computer server cluster log monitoring method and monitor supervision platform | |
| CN108667725A (en) | A kind of industrial AnyRouter and implementation method based on a variety of accesses and edge calculations | |
| CA2835446A1 (en) | Data analysis system | |
| CN106649071B (en) | Method and device for reporting log in process | |
| CN101997925A (en) | Server monitoring method with early warning function and system thereof | |
| CN112865311B (en) | Method and device for monitoring message bus of power system | |
| CN113179173B (en) | Operation and maintenance monitoring system for expressway system | |
| CN108052358B (en) | Distributed deployment system and method | |
| CN111245672A (en) | Monitoring method and system for general extensible tracking service full link | |
| CN111400288A (en) | Data quality inspection method and system | |
| Diotalevi et al. | Collection and harmonization of system logs and prototypal Analytics services with the Elastic (ELK) suite at the INFN-CNAF computing centre | |
| CN111127250B (en) | Power data monitoring event analysis system and method | |
| CN110955551B (en) | Intelligent fault diagnosis device based on tomcat middleware | |
| CN115833913A (en) | Multi-source heterogeneous space-based data acquisition method and system based on message driving mode | |
| CN116594840A (en) | Log fault acquisition and analysis method, system, equipment and medium based on ELK | |
| CN109522349B (en) | Cross-type data calculation and sharing method, system and equipment | |
| JP4928480B2 (en) | Job processing system and job management method | |
| CN116016117A (en) | Network equipment operation and maintenance data acquisition method and system, electronic equipment and storage medium | |
| CN115757045A (en) | Transaction log analysis method, system and device | |
| CN114116252A (en) | System and method for storing operation measurement data of regulation and control system | |
| CN115766768A (en) | Method and device for designing sensing center in computational power network operating system | |
| CN115022402A (en) | Agent acquisition method and system based on one-stack integration technology | |
| CN115237989A (en) | Mine data acquisition system | |
| CN110519303B (en) | Communication method and system across isolation devices | |
| CN113434473A (en) | Distributed log processing method, device, medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |