CN114357039A - Satellite constellation cloud processing analysis platform based on big data and use method thereof - Google Patents
Satellite constellation cloud processing analysis platform based on big data and use method thereof Download PDFInfo
- Publication number
- CN114357039A CN114357039A CN202011096282.1A CN202011096282A CN114357039A CN 114357039 A CN114357039 A CN 114357039A CN 202011096282 A CN202011096282 A CN 202011096282A CN 114357039 A CN114357039 A CN 114357039A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- satellite
- task
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a satellite constellation cloud processing analysis platform based on big data and a using method thereof. The operation management layer provides self-task distribution, self-priority confirmation, airborne processing and distribution capabilities through a big data platform and an artificial intelligence enhanced intelligent control and communication network, the support layer is used for ground command control setting and a user terminal and quick response transmitting service, the navigation layer provides backup positioning, navigation and time service in Beidou and GPS rejection environments, the sensing layer provides space situation sensing, detection and tracking of space targets to help satellite collision, the tracking layer provides tracking, aiming and advanced early warning of missile threats, the monitoring layer provides all-weather trusteeship for all determined time key targets, and the space transmission layer provides a global mesh network of data and communication in all weather. The satellite cluster utilization rate is increased, the reliability and disaster tolerance capability of the inter-satellite big data cloud platform are improved, and the system deployment cost and the personnel operation and maintenance cost are reduced.
Description
Technical Field
The invention belongs to the technical field of satellite constellation cloud; in particular to a satellite constellation cloud processing analysis platform based on big data and a using method thereof.
Background
Massive information obtained based on the multimodal heterogeneous sensor is difficult to process efficiently. At present, the types of sensors in the aerospace field are more and more, the code rate is greatly improved, and if the existing information processing and storing technology is only used, part of valuable information is inevitably wasted; data diversity makes heterogeneous data difficult to fuse. The type and format of data obtained by the heterogeneous sensor are diversified, unstructured data (such as semi-structured data of pictures, audio, video and the like) can be generated besides structured data, and the complexity and the dynamic property of the semi-structured data make the description and the fusion of the semi-structured data difficult to be realized by utilizing the existing data framework; and rapid information and data fusion between the satellite platforms of the same type is difficult to carry out. The link interface designs of different types of satellite platforms are not unified, data fusion is difficult to carry out before each other, and small clusters or formation is difficult to form quickly so as to realize quick earth observation and expand the task target of the tracking range.
Disclosure of Invention
The invention provides a satellite constellation cloud processing and analyzing platform based on big data and a using method thereof, which improve the reliability and disaster tolerance capability of an inter-satellite big data cloud platform, and reduce the system deployment cost and the personnel operation and maintenance cost. The platform constructs a database and a data lake with a unified framework by processing and integrating data collected by various sensors, and can flexibly deploy a machine learning model according to different task requirements, so that the data analysis capability is improved, the ground monitoring can be finally carried out independently or under the support of a ground base station, information support is provided for users at all levels on the ground, and (semi) autonomous task planning and satellite cluster (constellation) maintenance are carried out according to data analysis results.
The invention is realized by the following technical scheme:
a satellite constellation cloud processing and analyzing platform based on big data comprises an application layer, a core layer, a hardware layer, a combat management layer, a support layer, a navigation layer, a perception layer, a tracking layer, a monitoring layer and a space transmission layer, wherein the application layer selects an execution task, the core layer stores and processes data collected by the hardware layer, and the hardware layer collects data and provides computing power;
the combat management layer provides the capabilities of self-task allocation, self-priority confirmation, airborne processing and distribution through a big data platform and an artificial intelligence enhanced intelligent control and communication network,
the support layer is used for ground command control setting and user terminals, and quick response transmitting service,
the navigation layer provides backup positioning, navigation and time service in the Beidou and GPS refusing environment,
the sensing layer provides space situation sensing, detection and tracking of space targets to assist satellite collision,
the tracking layer provides missile threat tracking, targeting and advanced early warning,
the monitoring layer provides all-weather hosting for all determined time critical targets,
the space transport layer provides a global mesh network for data and communications around the clock.
Furthermore, the application layer comprises predictive maintenance, data fusion, situation perception, information analysis and ground monitoring, wherein the predictive maintenance is to collect all collapse time and maintenance information of a past module, to summarize problems through big data processing, and to automatically process similar problems once the similar problems appear again;
the data fusion is to observe key targets by cooperating the optical image, the electromagnetic wave and the sar;
the situational awareness is an ability to learn about security risks;
the intelligence analysis is to process the collected information and analyze the information by combining with other information;
the ground monitoring is to monitor and track the ground key targets.
Further, the core layer comprises a navigation layer, a tracking layer, a monitoring layer, a perception layer, a rapid analysis module, an intelligent decision module, a data analysis and decision layer, StreamSQL, a mining algorithm analysis layer, online monitoring, edge calculation, an image marking algorithm, Kafka message queue, an output manager, a data source manager, a streaming task management engine, a distributed execution engine, a storage management engine, an intelligent rapid stream processor, a combat management layer, a space transmission layer, a support layer, a data warehouse, a data lake, a containerized cluster storage center, a data analysis and decision layer and an intelligent rapid stream processing engine,
the rapid analysis module can process data in second level;
the intelligent decision module automatically performs task planning by processing the collected data;
the data analysis and decision layer is a combination of a rapid analysis and intelligent decision module;
the StreamSQL engine mixes event driving and micro batch processing, wherein the event driving refers to a strategy for making a decision in the process of continuous task distribution, namely, the decision is made at any time according to the state of a current task, resources are transferred, and problems in a task process are continuously processed, so that task accumulation is prevented;
the mining algorithm analysis layer is a mining algorithm analysis layer and is subjected to quantitative processing through an R language, so that preprocessing is performed on the data analysis and processing layer;
the online monitoring prevents data processing overload;
the edge calculation processes data at each satellite subsatellite or each data collection end;
the image annotation algorithm is used for data to be annotated in deep learning
The Kafka message queue: these data are usually recorded in the form of logs and then statistically processed at intervals;
the output manager unifies output formats;
the data source manager manages data acquisition;
the stream type task management engine is used for processing continuous data streams and can quickly detect abnormal conditions within a short time after receiving data, and the detection time is different from several milliseconds to several minutes;
the intelligent fast stream processor is used for processing continuous data streams and can quickly detect abnormal conditions within a short time after receiving data, and the detection time is different from several milliseconds to several minutes;
the data warehouse needs to accommodate more data and a larger data set;
the data lake is a large warehouse for storing various original data of an enterprise, wherein the data can be accessed, processed, analyzed and transmitted;
the containerized cluster storage center stores the data;
the data analysis and decision layer carries out second-level processing on the data, and automatically carries out task planning by processing the collected data;
the intelligent fast stream processing engine completes construction of a data calculation layer and extracts data from a data storage layer for processing.
A use method of a big data-based satellite constellation cloud processing analysis platform comprises the following steps:
step 1: forming a unified structure by the structured data generated in the satellite-borne task through an ETL tool and sending the unified structure to a data lake;
step 2: unstructured data generated in the satellite-borne task are provided, a unified structure is formed through an ETL tool, a machine learning platform is provided, and the unified structure is sent to a data lake;
and step 3: forming a uniform structure by using semi-structured data generated in the satellite-borne task through an ETL tool, providing a machine learning platform, and sending the uniform structure to a data lake;
and 4, step 4: the data lake collects the data in the step 1-3 and then sends the data to a ground observation information fusion unit of a big satellite data cloud platform;
and 5: the data lake collects the data in the step 1-3 and then sends the data to an inter-satellite predictive maintenance unit of a big data cloud platform on the satellite;
step 6: analyzing and deciding the result obtained by the earth observation information fusion unit or the inter-satellite predictive maintenance unit;
and 7: sending task instructions to each application terminal after autonomous task planning;
and 8: data generated by each application terminal enters a data lake through a ground observation information fusion unit or an inter-satellite predictive maintenance unit;
and step 9: the data lake transmits the data generated by the application terminal back to the satellite-borne task through the sensing information.
Further, the satellite-borne task of step 1 specifically includes a ground instruction, monitoring of modules in the satellite, observation information to the ground and task planning, and the satellite-borne task includes the steps of
Further, the step 3 of providing a machine learning platform comprises video and image target extraction, text conversion and OCR;
extracting the target in the video or the image to extract the target in the video or the image;
the OCR is to extract numbers and letters in a photographed image.
Furthermore, the earth observation information fusion unit in the step 4 comprises a distributed NewSQL database and a large-scale distributed computation and search engine, the distributed NewSQL database comprises metadata classified storage, real-time query and high-concurrency query, the metadata classified storage stores metadata after classification, the real-time query searches and queries the metadata, and the high-concurrency query processes a large-scale query task in parallel;
the large-scale distributed computation and search engine comprises metadata classified storage, key ground data or information query, full-table retrieval and aggregate analysis, wherein the metadata classified storage is used for classifying and then storing the metadata, and the aggregate analysis is used for classifying data with similar properties into one class.
Further, the inter-satellite predictive maintenance unit of step 5 comprises the following steps;
step 5.1: transmitting data to the prediction model by the real-time characteristic engineering of the online service;
step 5.2: the real-time characteristic engineering of the online service transmits data to all historical data modules;
step 5.3: the prediction model in the step 5.1 carries out predictive edge calculation;
step 5.4: all historical data of the step 5.2 are sent to a historical characteristic project;
step 5.5: modeling after extracting the characteristics by historical characteristic engineering;
step 5.6: verifying the model built in the step 5.5, and jumping to the step 5.7 if the modeling is correct, or jumping to the step 5.5 if the modeling is incorrect;
step 5.7: and deploying the online service through the verification of the built model and cloud computing.
The invention has the beneficial effects that:
1. the software framework provided by the invention improves the utilization rate of a satellite cluster (constellation), improves the reliability and disaster tolerance capability of an inter-satellite big data cloud platform, and reduces the system deployment cost and the personnel operation and maintenance cost.
2. The invention flexibly deploys the machine learning model aiming at different task requirements, thereby improving the data analysis capability, finally carrying out ground monitoring independently or under the support of a ground base station, providing information support for users at all levels on the ground, and carrying out (semi-) autonomous task planning and satellite cluster (constellation) maintenance according to the data analysis result.
Drawings
FIG. 1 is a schematic structural view of the present invention.
FIG. 2 is a flow chart of the method of the present invention.
Fig. 3 is a schematic diagram of satellite cluster data flow in the core star mode according to the present invention.
Fig. 4 is a schematic diagram of a satellite cluster data stream in a cloud constellation mode according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1
The processing and analyzing platform comprises an application layer, a core layer, a hardware layer, a combat management layer, a support layer, a navigation layer, a perception layer, a tracking layer, a monitoring layer and a space transmission layer, wherein the application layer selects an execution task, the core layer stores and processes data collected by the hardware layer, and the hardware layer collects data and provides computing power;
the combat management layer provides the capabilities of self-task allocation, self-priority confirmation, airborne processing and distribution through a big data platform and an artificial intelligence enhanced intelligent control and communication network,
the support layer is used for ground command control setting and user terminals, and quick response transmitting service,
the navigation layer provides backup positioning, navigation and time service in the Beidou and GPS refusing environment,
the sensing layer provides space situation sensing, detection and tracking of space targets to assist satellite collision,
the tracking layer provides missile threat tracking, targeting and advanced early warning,
the monitoring layer provides all-weather hosting for all determined time critical targets,
the space transport layer provides a global mesh network for data and communications around the clock.
Furthermore, the application layer comprises predictive maintenance, data fusion, situation perception, information analysis and ground monitoring, wherein the predictive maintenance is to collect all collapse time and maintenance information of a past module, to summarize problems through big data processing, and to automatically process similar problems once the similar problems appear again;
the data fusion is to observe key targets by cooperating the optical image, the electromagnetic wave and the sar;
the situational awareness is an ability to learn about security risks;
the intelligence analysis is to process the collected information and analyze the information by combining with other information;
the ground monitoring is to monitor and track the ground key targets.
Further, the core layer comprises a navigation layer, a tracking layer, a monitoring layer, a perception layer, a rapid analysis module, an intelligent decision module, a data analysis and decision layer, StreamSQL, a mining algorithm analysis layer, online monitoring, edge calculation, an image marking algorithm, Kafka message queue, an output manager, a data source manager, a streaming task management engine, a distributed execution engine, a storage management engine, an intelligent rapid stream processor, a combat management layer, a space transmission layer, a support layer, a data warehouse, a data lake, a containerized cluster storage center, a data analysis and decision layer and an intelligent rapid stream processing engine,
the rapid analysis module can process data in second level;
the intelligent decision module automatically performs task planning by processing the collected data;
the data analysis and decision layer is a combination of a rapid analysis and intelligent decision module;
the StreamSQL engine mixes event driving and micro batch processing, wherein the event driving refers to a strategy for making a decision in the process of continuous task distribution, namely, the decision is made at any time according to the state of a current task, resources are transferred, and problems in a task process are continuously processed, so that task accumulation is prevented;
the mining algorithm analysis layer is a mining algorithm analysis layer and is subjected to quantitative processing through an R language, so that preprocessing is performed on the data analysis and processing layer;
the online monitoring prevents data processing overload;
the edge calculation processes data at each satellite subsatellite or each data collection end;
the image annotation algorithm is used for data to be annotated in deep learning
The Kafka message queue: kafka is a messaging system that Linkedin was open source in 12 months 2010, primarily for handling active streaming data. Active streaming data is very common in web site applications, including pv of the web site, what content the user has visited, what content has been searched, etc. These data are usually recorded in the form of logs and then statistically processed at intervals;
the output manager unifies output formats;
the data source manager manages data acquisition;
the stream type task management engine is used for processing continuous data streams and can quickly detect abnormal conditions within a short time after receiving data, and the detection time is different from several milliseconds to several minutes; for example, you can be alerted when the temperature reaches freezing point by querying the data stream from the temperature sensor through a streaming process. There are many other names for stream processing: real-time analysis, flow analysis, complex event processing, real-time flow analysis, and event processing.
The distributed execution engine
The storage management engine is used for managing a database;
the intelligent fast stream processor is used for processing continuous data streams and can quickly detect abnormal conditions within a short time after receiving data, and the detection time is different from several milliseconds to several minutes;
the data warehouse (dataware house) is abbreviated as DW or DWH, is a conceptual upgrade of a database, and can be a new database designed to meet new requirements, and the database needs to contain more data and a larger data set; the data warehouse and the database are indistinguishable;
the data lake is a large warehouse for storing various original data of an enterprise, wherein the data can be accessed, processed, analyzed and transmitted;
the containerized cluster storage center stores the data; the advantages are light weight, quick deployment, easy transplantation and elastic expansion;
the data analysis and decision layer carries out second-level processing on the data, and automatically carries out task planning by processing the collected data;
the intelligent fast stream processing engine completes construction of a data calculation layer and extracts data from a data storage layer for processing. Thereby achieving fast, even real-time data processing.
Mainly divided into IaaS (Infrastructure-as-a-Service), PaaS (Platform-as-a-Service) and SaaS (Software-as-a-Service). In the aerospace field, IaaS can be understood as hardware (infrastructure) providing services to user space through inter-satellite links; PaaS is primarily directed to developers by providing a complementary environment on top of the hardware as a service to users. SaaS is a method for providing configured software to users, and users can access the software using client interfaces through various devices.
IaaS is mainly composed of "end" representing various types of sensors and "cloud" which is a satellite cluster (constellation). PaaS is mainly divided into three main parts: the system comprises a containerization cluster operating system, an intelligent fast stream processing engine and data analysis and decision-making. The containerization cluster operating system mainly establishes three databases through distributed storage: NoSQL databases, analytical databases, and highly available databases. NoSQL, which refers to a non-relational database, has dual purposes of large data volume and high performance, and is superior in performance even in the case of large data volume due to its simple structure and no relationship. Analytical databases are generally divided into OLTP (On-line transactional processing) and OLAP (On-line analytical processing), and when the amount of task data reaches a certain level, which involves cross-domain data processing and analysis, the databases need to complete the conversion from OLTP to OLAP, and the two are in a coexistence relationship. In addition to the design of the NoSQL database and the analytic database, the establishment of a highly available database needs to be considered, the establishment of the database aims to reduce the time that the system cannot provide services through design, the core criterion is redundancy so as to prevent a certain server (a subsatellite formed in a satellite constellation) from being inaccessible when the server is down, and the redundancy can realize high availability of the services by deploying at least two servers to form a cluster.
The software analysis processing layer is composed of a rapid analysis module and an intelligent decision module. The rapid analysis module is mainly used for processing data through methods of model training, feature engineering and the like in machine learning, a decision system aims at the defect that a traditional machine learning model usually adopts T +1 offline batch prediction, provides model prediction service for the trained machine learning model and responds to a prediction result in real time, and meanwhile, due to the particularity of the aerospace field, the machine learning model can be embedded into an expert decision flow, so that the expert decision level is improved, and the defect that the machine learning model is weak in interpretability is overcome.
A use method of a big data-based satellite constellation cloud processing analysis platform comprises the following steps:
step 1: forming a unified structure by the structured data generated in the satellite-borne task through an ETL tool and sending the unified structure to a data lake;
step 2: unstructured data generated in the satellite-borne task are provided, a unified structure is formed through an ETL tool, a machine learning platform is provided, and the unified structure is sent to a data lake;
and step 3: forming a uniform structure by using semi-structured data generated in the satellite-borne task through an ETL tool, providing a machine learning platform, and sending the uniform structure to a data lake;
and 4, step 4: the data lake collects the data in the step 1-3 and then sends the data to a ground observation information fusion unit of a big satellite data cloud platform;
and 5: the data lake collects the data in the step 1-3 and then sends the data to an inter-satellite predictive maintenance unit of a big data cloud platform on the satellite;
step 6: analyzing and deciding the result obtained by the earth observation information fusion unit or the inter-satellite predictive maintenance unit;
and 7: sending task instructions to each application terminal after autonomous task planning;
and 8: data generated by each application terminal enters a data lake through a ground observation information fusion unit or an inter-satellite predictive maintenance unit;
and step 9: the data lake transmits the data generated by the application terminal back to the satellite-borne task through the sensing information.
Further, the satellite-borne task in the step 1 specifically includes a ground instruction, monitoring of modules in the satellite, observation information of the ground and task planning.
Further, the step 3 of providing a machine learning platform comprises video and image target extraction, text conversion and OCR;
extracting the target in the video or the image to extract the target in the video or the image;
the OCR is to extract numbers and letters in a photographed image.
Furthermore, the earth observation information fusion unit in the step 4 comprises a distributed NewSQL database and a large-scale distributed computation and search engine, the distributed NewSQL database comprises metadata classified storage, real-time query and high-concurrency query, the metadata classified storage stores metadata after classification, the real-time query searches and queries the metadata, and the high-concurrency query processes a large-scale query task in parallel;
the large-scale distributed computation and search engine comprises metadata classified storage, key ground data or information query, full-table retrieval and aggregate analysis, wherein the metadata classified storage is used for classifying and then storing the metadata, and the aggregate analysis is used for classifying data with similar properties into one class.
8. The use method of the big data based satellite constellation cloud processing analysis platform as claimed in claim 4, wherein the inter-satellite predictive maintenance unit of step 5 comprises the following steps;
step 5.1: transmitting data to the prediction model by the real-time characteristic engineering of the online service;
step 5.2: the real-time characteristic engineering of the online service transmits data to all historical data modules;
step 5.3: the prediction model in the step 5.1 carries out predictive edge calculation;
step 5.4: all historical data of the step 5.2 are sent to a historical characteristic project;
step 5.5: modeling after extracting the characteristics by historical characteristic engineering;
step 5.6: verifying the model built in the step 5.5, and jumping to the step 5.7 if the modeling is correct, or jumping to the step 5.5 if the modeling is incorrect;
step 5.7: and deploying the online service through the verification of the built model and cloud computing.
After data is acquired from a data lake, the data is simultaneously transmitted to two modules, namely an edge computing module and a cloud computing module, wherein the edge computing module is mainly used for carrying out stream processing and a quasi-real-time computing platform for carrying out minipatch processing on the data in a stream form, and the cloud computing module is used for collecting and storing all data and summarizing the data with historical data to carry out feature extraction and model training.
The edge calculation module mainly analyzes the occurrence frequency of a certain problem in a task in the time window, wherein the quasi-real-time characteristic engineering module can store processed data into the characteristic database at a speed of a minute level or even a second level for a prediction system model to use, and the delay of the minute level basically can meet the requirement of a system for making a decision quickly and independently.
Data collected by each sensor in a satellite cluster (constellation) is all put into a database with a unified architecture, wherein the data can be input into the database only through conversion of a machine learning platform due to the particularity of semi/unstructured data. Deploying a cloud computing model under the support of a distributed unified architecture database by taking a satellite cluster (constellation) as a hardware platform, establishing a prediction model by utilizing real-time characteristic data (parameters) and historical characteristic data and a machine learning method, and thus realizing the targeted predictive maintenance of each satellite; in addition, the platform can establish a cloud large-scale distributed search engine through a distributed database, so that various infrared, hyperspectral, SAR and other data observed on the ground can be subjected to fusion analysis, and a focus target can be identified and extracted through the search engine; and performing semi-autonomous satellite cluster (constellation) task planning through data results obtained by earth observation information fusion and inter-satellite predictive maintenance.
According to the figure 3, the edge computing of the core star mode occurs in the subsystem, and the cloud computing occurs in the main star information fusion system.
According to the cloud constellation mode of fig. 4, when one subsystem is taken as a main satellite for cloud computing, the rest subsystems without tasks help the subsystem taken as the main satellite for computing, and the subsystems taken as the main satellite are different under different conditions.
Claims (8)
1. A satellite constellation cloud processing and analyzing platform based on big data is characterized by comprising an application layer, a core layer, a hardware layer, a combat management layer, a support layer, a navigation layer, a perception layer, a tracking layer, a monitoring layer and a space transmission layer, wherein the application layer selects an execution task, the core layer stores and processes data collected by the hardware layer, and the hardware layer performs data acquisition and calculation power supply;
the combat management layer provides the capabilities of self-task allocation, self-priority confirmation, airborne processing and distribution through a big data platform and an artificial intelligence enhanced intelligent control and communication network,
the support layer is used for ground command control setting and user terminals, and quick response transmitting service,
the navigation layer provides backup positioning, navigation and time service in the Beidou and GPS refusing environment,
the sensing layer provides space situation sensing, detection and tracking of space targets to assist satellite collision,
the tracking layer provides missile threat tracking, targeting and advanced early warning,
the monitoring layer provides all-weather hosting for all determined time critical targets,
the space transport layer provides a global mesh network for data and communications around the clock.
2. The big-data-based satellite constellation cloud processing analysis platform as claimed in claim 1, wherein the application layer comprises predictive maintenance, data fusion, situational awareness, intelligence analysis and ground monitoring, the predictive maintenance is to collect and summarize all collapse time and maintenance method information of past modules, and after big data processing, problems are summarized, and once similar problems reappear, the problems can be automatically processed;
the data fusion is to observe key targets by cooperating the optical image, the electromagnetic wave and the sar;
the situational awareness is an ability to learn about security risks;
the intelligence analysis is to process the collected information and analyze the information by combining with other information;
the ground monitoring is to monitor and track the ground key targets.
3. The big-data based satellite constellation cloud processing analysis platform as in claim 1, wherein the core layer comprises a navigation layer, a tracking layer, a monitoring layer, a sensing layer, a fast analysis module, an intelligent decision module, a data analysis and decision layer, a Stream SQL, a mining algorithm parsing layer, an on-line monitoring, an edge calculation, an image annotation algorithm, a Kafka message queue, an output manager, a data source manager, a streaming task management engine, a distributed execution engine, a storage management engine, an intelligent fast Stream processor, a combat management layer, a space transport layer, a support layer, a data warehouse, a data lake, a containerized cluster storage center, a data analysis and decision layer and an intelligent fast Stream processing engine,
the rapid analysis module can process data in second level;
the intelligent decision module automatically performs task planning by processing the collected data;
the data analysis and decision layer is a combination of a rapid analysis and intelligent decision module;
the Stream SQL engine mixes event driving and micro batch processing, the event driving refers to a strategy for making a decision in the process of continuous task distribution, namely, the decision is made at any time according to the current task state, resources are transferred, and problems in the task process are continuously processed, so that task accumulation is prevented;
the mining algorithm analysis layer is a mining algorithm analysis layer and is subjected to quantitative processing through an R language, so that preprocessing is performed on the data analysis and processing layer;
the online monitoring prevents data processing overload;
the edge calculation processes data at each satellite subsatellite or each data collection end;
the image annotation algorithm is used for data to be annotated in deep learning
The Kafka message queue: these data are usually recorded in the form of logs and then statistically processed at intervals;
the output manager unifies output formats;
the data source manager manages data acquisition;
the stream type task management engine is used for processing continuous data streams and can quickly detect abnormal conditions within a short time after receiving data, and the detection time is different from several milliseconds to several minutes;
the intelligent fast stream processor is used for processing continuous data streams and can quickly detect abnormal conditions within a short time after receiving data, and the detection time is different from several milliseconds to several minutes;
the data warehouse needs to accommodate more types and more huge amounts of data;
the data lake is a large warehouse for storing various original data of an enterprise, wherein the data can be accessed, processed, analyzed and transmitted;
the containerized cluster storage center stores the data;
the data analysis and decision layer carries out second-level processing on the data, and automatically carries out task planning by processing the collected data;
the intelligent fast stream processing engine completes construction of a data calculation layer and extracts data from a data storage layer for processing.
4. The use method of the big data based satellite constellation cloud processing analysis platform according to claim 1, wherein the use method comprises the following steps:
step 1: forming a unified structure by the structured data generated in the satellite-borne task through an ETL tool and sending the unified structure to a data lake;
step 2: unstructured data generated in the satellite-borne task are provided, a unified structure is formed through an ETL tool, a machine learning platform is provided, and the unified structure is sent to a data lake;
and step 3: forming a uniform structure by using semi-structured data generated in the satellite-borne task through an ETL tool, providing a machine learning platform, and sending the uniform structure to a data lake;
and 4, step 4: the data lake collects the data in the step 1-3 and then sends the data to a ground observation information fusion unit of a big satellite data cloud platform;
and 5: the data lake collects the data in the step 1-3 and then sends the data to an inter-satellite predictive maintenance unit of a big data cloud platform on the satellite;
step 6: analyzing and deciding the result obtained by the earth observation information fusion unit or the inter-satellite predictive maintenance unit;
and 7: sending task instructions to each application terminal after autonomous task planning;
and 8: data generated by each application terminal enters a data lake through a ground observation information fusion unit or an inter-satellite predictive maintenance unit;
and step 9: the data lake transmits the data generated by the application terminal back to the satellite-borne task through the sensing information.
5. The use method of the big data based satellite constellation cloud processing and analysis platform according to claim 4, wherein the satellite-borne task in the step 1 specifically comprises a ground instruction, intra-satellite module monitoring, earth observation information and task planning.
6. The use method of the big data based satellite constellation cloud processing analysis platform as claimed in claim 4, wherein the providing machine learning platform of step 3 comprises target extraction in video and image, text conversion and OCR;
extracting the target in the video or the image to extract the target in the video or the image;
the OCR is to extract numbers and letters in a photographed image.
7. The use method of the big data based satellite constellation cloud processing analysis platform according to claim 4, wherein the earth observation information fusion unit in step 4 comprises a distributed NewSQL database and a large-scale distributed computation and search engine, the distributed NewSQL database comprises metadata classified storage, real-time query and high-concurrency query, the metadata classified storage stores the metadata after classification, the real-time query searches and queries the metadata, and the high-concurrency query processes a large-scale query task in parallel;
the large-scale distributed computation and search engine comprises metadata classified storage, key ground data or information query, full-table retrieval and aggregate analysis, wherein the metadata classified storage is used for classifying and then storing the metadata, and the aggregate analysis is used for classifying data with similar properties into one class.
8. The use method of the big data based satellite constellation cloud processing analysis platform as claimed in claim 4, wherein the inter-satellite predictive maintenance unit of step 5 comprises the following steps;
step 5.1: transmitting data to the prediction model by the real-time characteristic engineering of the online service;
step 5.2: the real-time characteristic engineering of the online service transmits data to all historical data modules;
step 5.3: the prediction model in the step 5.1 carries out predictive edge calculation;
step 5.4: all historical data of the step 5.2 are sent to a historical characteristic project;
step 5.5: modeling after extracting the characteristics by historical characteristic engineering;
step 5.6: verifying the model built in the step 5.5, and jumping to the step 5.7 if the modeling is correct, or jumping to the step 5.5 if the modeling is incorrect;
step 5.7: and deploying the online service through the verification of the built model and cloud computing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011096282.1A CN114357039A (en) | 2020-10-14 | 2020-10-14 | Satellite constellation cloud processing analysis platform based on big data and use method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011096282.1A CN114357039A (en) | 2020-10-14 | 2020-10-14 | Satellite constellation cloud processing analysis platform based on big data and use method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114357039A true CN114357039A (en) | 2022-04-15 |
Family
ID=81090006
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011096282.1A Pending CN114357039A (en) | 2020-10-14 | 2020-10-14 | Satellite constellation cloud processing analysis platform based on big data and use method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114357039A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114531197A (en) * | 2022-04-24 | 2022-05-24 | 中国科学院空天信息创新研究院 | On-orbit distributed information resource application service system |
| CN117978262B (en) * | 2024-04-02 | 2024-05-31 | 华信正能集团有限公司 | Internet of things data and communication transmission device for space-based satellite constellation |
-
2020
- 2020-10-14 CN CN202011096282.1A patent/CN114357039A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114531197A (en) * | 2022-04-24 | 2022-05-24 | 中国科学院空天信息创新研究院 | On-orbit distributed information resource application service system |
| CN114531197B (en) * | 2022-04-24 | 2022-07-22 | 中国科学院空天信息创新研究院 | On-orbit distributed information resource application service system |
| CN117978262B (en) * | 2024-04-02 | 2024-05-31 | 华信正能集团有限公司 | Internet of things data and communication transmission device for space-based satellite constellation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111371830B (en) | Intelligent cooperative cloud architecture based on data driving under fusion scene of ten thousand networks | |
| CN116307757B (en) | Intelligent data interaction method, interaction system, computer equipment and application | |
| CN109902297B (en) | Threat information generation method and device | |
| US11392733B2 (en) | Multi-dimensional event model generation | |
| US20200301760A1 (en) | Methods and systems for generating and recommending api mashups | |
| CN112668841A (en) | Comprehensive traffic monitoring system and method based on data fusion | |
| CN112559634A (en) | Big data management system based on computer cloud computing | |
| CN114357039A (en) | Satellite constellation cloud processing analysis platform based on big data and use method thereof | |
| CN110661999A (en) | Video monitoring system based on big data | |
| Badidi et al. | Towards a Platform for Urban Data Management, Integration and Processing. | |
| Sha et al. | Smart city public safety intelligent early warning and detection | |
| CN107579944A (en) | Based on artificial intelligence and MapReduce security attack Forecasting Methodologies | |
| CN109784525A (en) | Method for early warning and device based on day vacant lot integration data | |
| CN116560887B (en) | Low orbit satellite fault analysis method based on knowledge cooperation | |
| Xhafa et al. | A software chain approach to big data stream processing and analytics | |
| Hou et al. | Failure prediction of elevator running system based on knowledge graph | |
| Tampakis et al. | Sea area monitoring and analysis of fishing vessels activity: The i4sea big data platform | |
| Yu et al. | A police big data analytics platform: Framework and implications | |
| Costa et al. | Big Data Architectures for Logging and Monitoring Large Scale Telescope Arrays | |
| CN112383435B (en) | Fault processing method and device | |
| CN115242438A (en) | Potential victim group positioning method based on heterogeneous information network | |
| CN115033715A (en) | Interception feedback processing method based on big data analysis interception and information interception system | |
| Wang et al. | Construction of a multi-source heterogeneous hybrid platform for big data | |
| CN112564984A (en) | Distributed safe operation and maintenance method of Internet of things based on big data | |
| Airlangga et al. | A novel architectural design for solving lost-link problems in UAV collaboration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |