CN115134421B - Multi-source heterogeneous data cross-system collaborative management system and method - Google Patents
Multi-source heterogeneous data cross-system collaborative management system and method Download PDFInfo
- Publication number
- CN115134421B CN115134421B CN202210507966.9A CN202210507966A CN115134421B CN 115134421 B CN115134421 B CN 115134421B CN 202210507966 A CN202210507966 A CN 202210507966A CN 115134421 B CN115134421 B CN 115134421B
- Authority
- CN
- China
- Prior art keywords
- data
- source
- type
- task
- demand
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 17
- 238000007726 management method Methods 0.000 claims abstract description 69
- 238000004458 analytical method Methods 0.000 claims abstract description 34
- 230000005540 biological transmission Effects 0.000 claims description 28
- 238000012545 processing Methods 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 6
- 238000013523 data management Methods 0.000 claims description 5
- 238000004806 packaging method and process Methods 0.000 claims description 2
- 238000013480 data collection Methods 0.000 abstract description 4
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 3
- 238000002955 isolation Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000035484 reaction time Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 231100000279 safety data Toxicity 0.000 description 1
- 239000000725 suspension Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
In the multi-source heterogeneous data cross-system collaborative management system, the multi-source heterogeneous data cross-system collaborative management method and the storage medium, the multi-source heterogeneous data cross-system collaborative management system comprises a data access layer, a data bus layer and a data subscription layer. The data subscription layer is used for acquiring data demand tasks and types of the data demand tasks, analyzing the acquired data demand tasks and subscribing the data of the data bus layer according to the analysis result of the data request and the types of the data demand tasks; the data bus layer is used for sending the analysis result of the data subscription layer, the data demand task and the type of the data demand task to the data access layer; the data access layer is used for sending the data demand task and the type of the data demand task to the corresponding data source system according to the analysis result of the data bus layer. The method and the device effectively shorten data collection and update time, realize multi-main-body coordination management of the multi-center heterogeneous online system, and solve the problem of low collaborative management and convergence service efficiency.
Description
Technical Field
The application relates to the technical field of computer application, in particular to a multi-source heterogeneous data cross-system collaborative management system, a multi-source heterogeneous data cross-system collaborative management method and a storage medium.
Background
With the rapid development of remote sensing and network technology, the multi-source heterogeneous data such as satellite and aerial image, ground space-time video, network data, electromagnetic information and geographic information are integrated and utilized, space-time, multi-scale and multi-granularity convergence of sky-ground multi-source heterogeneous data is realized, reliable and expandable data support can be provided for intelligent perception and understanding of public safety events, and the method has important significance for improving the level of large data sharing service facing public safety events, expanding sharing service channels and enhancing sharing service capacity.
However, in the related art, the provided information interaction mode is single, only several types of data in the service system are collected and managed, and the multi-main coordination management of the multi-center heterogeneous online system cannot be realized by safely collecting the multi-source heterogeneous data of the multi-online system through a whole set of system, so that the problem of low collaborative management and centralized service efficiency exists.
Disclosure of Invention
The application provides a multi-source heterogeneous data cross-system collaborative management system and a multi-source heterogeneous data cross-system collaborative management method, which are used for managing interaction of multi-source heterogeneous data cross-security levels.
An embodiment of a first aspect of the present application provides a multi-source heterogeneous data cross-system collaborative management system, including:
the data subscription layer is used for acquiring a data demand task and the type of the data demand task, analyzing the acquired data demand task and subscribing the data of the data bus layer according to the analysis result of the data request and the type of the data demand task;
the data bus layer is used for sending the analysis result of the data subscription layer, the data demand task and the type of the data demand task to the data access layer;
and the data access layer is used for sending the data demand task and the type of the data demand task to the corresponding data source system according to the analysis result of the data bus layer.
An embodiment of a second aspect of the present application provides a multi-source heterogeneous data cross-system collaborative management method, including:
acquiring a data demand task and the type of the data demand task;
analyzing the data demand task, and transmitting the data demand to a corresponding data source system based on an analysis result and the type of the data demand task;
receiving data corresponding to the data demand task sent by the data source system;
and determining and outputting a corresponding transmission mode based on the type of the received data.
Embodiments of the third aspect of the present application provide a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores a computer program; the computer program, when executed by a processor, implements the method as described in the second aspect above.
The technical scheme provided by the embodiment of the application at least brings the following beneficial effects:
in the multi-source heterogeneous data cross-system collaborative management system, the multi-source heterogeneous data cross-system collaborative management method and the storage medium, the multi-source heterogeneous data cross-system collaborative management system comprises a data access layer, a data bus layer and a data subscription layer. The data subscription layer is used for acquiring data demand tasks and types of the data demand tasks, analyzing the acquired data demand tasks and subscribing the data of the data bus layer according to the analysis result of the data request and the types of the data demand tasks; the data bus layer is used for sending the analysis result of the data subscription layer, the data demand task and the type of the data demand task to the data access layer; the data access layer is used for sending the data demand task and the type of the data demand task to the corresponding data source system according to the analysis result of the data bus layer. The method and the device provide standardized and intelligent targeted multi-source heterogeneous data scheduling and updating strategies, effectively shorten data collection and updating time, realize multi-main-body coordination management of the multi-center heterogeneous online system, meet the demand of upper-layer application on demand collection of multi-source heterogeneous data, enable limited transmission channel resources to be used for transmitting data products which are most urgently needed by collection users all the time, and solve the problem of low efficiency of collaborative management and collection service.
Additional aspects and advantages of the application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a schematic diagram of a multi-source heterogeneous data cross-system collaborative management system according to one embodiment of the present application;
FIG. 2 is a flow chart of data transmission of entity relationships in a data service bus according to one embodiment of the present application;
fig. 3 is a flow chart of a multi-source heterogeneous data cross-system collaborative management method according to an embodiment of the present application.
Detailed Description
Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and intended for the purpose of explaining the present application and are not to be construed as limiting the present application.
The following describes a multi-source heterogeneous data cross-system collaborative management system and a method according to an embodiment of the present application with reference to the accompanying drawings.
Example 1
Fig. 1 is a schematic structural diagram of a multi-source heterogeneous data cross-system collaborative management system according to an embodiment of the present application, as shown in fig. 1, may include:
the data subscription layer is used for acquiring data demand tasks and types of the data demand tasks, analyzing the acquired data demand tasks and subscribing the data of the data bus layer according to the analysis result of the data request and the types of the data demand tasks;
the data bus layer is used for sending the analysis result of the data subscription layer, the data demand task and the type of the data demand task to the data access layer;
the data access layer is used for sending the data demand task and the type of the data demand task to the corresponding data source system according to the analysis result of the data bus layer;
the data access layer is also used for safely accessing multi-source heterogeneous data from different data source systems;
the data bus layer is also used for receiving the multi-source heterogeneous data collected by the data access layer, defining different data domains according to different data sources of the multi-source heterogeneous data, defining different topics under different data domains according to different data types under different data domains, and publishing the data in the different data domains;
the data subscription layer is also used for receiving the published data in the data bus layer, determining the type of the received data and determining a corresponding transmission mode for transmission based on the type of the received data.
In an embodiment of the present invention, the data subscription layer may obtain the data requirement task and the type of the data requirement task from the data management platform. And the data management platform can manage and analyze the multi-source heterogeneous data.
And, in an embodiment of the present invention, the types of data demand tasks acquired in the data subscription layer include regular service mode requirements, emergency service mode requirements, and special service mode requirements. The data call and data update requirements vary from one type of requirement to another.
Specifically, in embodiments of the present invention, the regular service mode requirements are general user requirements under daily work or in response to emergency; the emergency service mode requirements are user requirements for data to be quickly acquired or for all available and effective resources to be mobilized in a minimum time; the special service mode requirement is a user requirement of tracking a certain region or a certain time continuously under the condition of taking special task input as a core or through task driving.
And, in embodiments of the present invention, the regular service mode requirements may include periodic or 7 x 24 hour real-time data update requirements; emergency service mode requirements may include data requirements for specific resource types, product levels, time, area, etc.; the special service mode requirements can include time period, frequency, area, image data load requirements, video, service data and intelligent analysis result data with definite tasks.
Further, in an embodiment of the present invention, the parsing of the acquired data request by the data subscription layer includes parsing the data demand task to obtain a data source system number related to the data demand task.
In the embodiment of the invention, the data access layer can send the data demand task to the corresponding data source system according to the data source system number related to the data demand task analyzed by the data subscription layer.
It should be noted that, in the embodiment of the present invention, the multi-source heterogeneous data may include satellite remote sensing data, aerial remote sensing data, terrestrial video data, network information data, geographic information data, and service information data.
And in the embodiment of the invention, when the data in the multi-source heterogeneous data are different, the system where the corresponding data are located is also different. In one embodiment of the present invention, table 1 is a correspondence between a data type and a system/network where the data is located in the multi-source heterogeneous data.
TABLE 1
Further, in the embodiment of the present invention, after the data source system receives the data demand task, the data corresponding to the data demand task may be sent to the multi-source heterogeneous data cross-system collaborative management system according to the type of the data demand task.
Specifically, in the embodiment of the invention, when the type of the data demand task is the conventional service mode demand, if the data type is file data, the data source system periodically and actively pushes corresponding regional catalogue information, and the data transmission proxy service in the data access layer monitors the change of the catalogue information and periodically sends the change to the multi-source heterogeneous data cross-system collaborative management system; if the data type is video/service data, the data source system can acquire, update and aggregate the multi-source heterogeneous data cross-system collaborative management system in real time; if the data type is data source data under different network system environments, cross-network data transmission and information interaction are required to be carried out between the data access layer and the data bus layer.
In the embodiment of the invention, when the type of the data demand task is the emergency service mode demand, if the data type is space remote sensing data, the data source system can improve the priority through system configuration on one hand and complete conflict resolution with other tasks on the other hand after receiving the clear entity data demand urgent task, timely adjust the satellite orbit or gesture to arrange shooting tasks, timely acquire data/information service and simplify corresponding processing flow. Meanwhile, according to specific resource types, product grades, time and regional requirements, the background configuration of the data source system realizes that the automatic distribution service is sent to the multi-source heterogeneous data cross-system collaborative management system; if the data type is unmanned plane data, the data source system submits an air flight application after receiving an emergency task, completes the downloading processing of the flight task and the data, and transmits the data to the data management platform through the multi-source heterogeneous data cross-system collaborative management system; if the data types are video and business data, after receiving an emergency task, the data source system needs to mobilize all video acquisition equipment in the place where the emergency is located, adjust the angle and focal length of video acquisition according to the requirements, monitor the event in all directions, mobilize intelligent analysis resources at the same time, analyze related people, vehicles and objects in real time according to the event properties, trace, analyze and predict the event, and transmit analysis results to the multi-source heterogeneous data cross-system collaborative management system.
Further, in the embodiment of the invention, when the type of the data demand task is a special service mode demand, if the data type is remote sensing data, the data source system acquires catalogue information or entity data of corresponding data according to the definite time period, frequency, region and load requirements of the task and pushes the catalogue information or entity data to the multi-source heterogeneous data cross-system collaborative management system; if the data types are video and service data, the data source system acquires the video and service data in a specific time period and the intelligent analysis result thereof and pushes the video and service data to the multi-source heterogeneous data cross-system collaborative management system.
And, in an embodiment of the present invention, the data source class sent by the data source system to the multi-source heterogeneous data cross-system collaborative management system may include file data, streaming media data, and short message data. In the embodiment of the invention, the file data comprises catalogue information data and entity data, and the numbers of the catalogue information data and the entity data are consistent. And, entity data capacity is large, transmission time is long, the middle may have abnormal situations such as network interruption or service suspension, etc., in the transmission and exchange service design, protection mechanisms such as breakpoint continuous transmission are to be supported; streaming media data can comprise video and sound media data, and the key performances of delay, jitter and error rate must be strictly controlled in the transmission process; the short message data can comprise a target message, a flight message, a WiFi message and the like, has small capacity and high frequency, and can be directly transmitted.
Further, in the embodiment of the present invention, the data access layer may perform secure access on multi-source heterogeneous data from different system sources through the access gateway, respectively. The access gateway may include an external network convergence unit, an external network proxy service unit, an isolated communication unit, an internal network convergence unit, and an internal network proxy service unit.
Specifically, in the embodiment of the invention, for satellite remote sensing image data, unmanned aerial vehicle data and ground monitoring video data, the data are collected into corresponding collecting units through an external network collecting unit, then the data are transmitted through different networks and sent to corresponding external network proxy service units, the external network service units feed back to different levels of isolation communication units through data types to carry out differential isolation processing, the data are sent to an internal network proxy service unit after the isolation unit finishes processing, data spanning from the external network to the internal network is completed, and the internal network proxy service unit can send the isolated safety data to the internal network collecting unit. Based on the method, the multi-source heterogeneous data are safely sent from the external network to the internal network for further data processing through the cooperative processing of a plurality of units.
And in an embodiment of the invention, the communication between the data bus layer and the data subscription layer is completed by using a distributed data service bus. Wherein the domains represent logically isolated communication networks. In particular, multiple interactions running on the same set of hosts or on the same set of hosts are isolated from each other in different domains, and data writers and data readers of different domains do not exchange data.
Specifically, in embodiments of the present invention, the data bus layer and the data subscription layer communicate through data-centric publish/subscribe to transfer data from publishers to interested subscribers. And, prior to communication, an establishment entity is required to establish a publish/subscribe communication with each other. The distributed data service bus may include domains, data writers, publishers, data readers, subscribers, and master entities.
In the embodiment of the invention, the data bus layer can be used as a publisher, and the data subscription layer can be used as a subscriber. And in the embodiment of the invention, the data bus layer can define data from different sources as different domains, so that the data is effectively isolated. Within a data field, different topics may be defined according to data type, so that data may be identified by the topics. And, in embodiments of the present invention, the theme is a specific domain segment, the theme having a unique theme name, data type, and a set of QOS policies. Multiple topics may be published for the same data type, with each topic being associated with only one data type.
And in the embodiment of the invention, a theme can be defined for the subtype data under each big data type, and the data type and QOS strategy of the theme can be defined according to the information required to be communicated, so that preparation is made for subsequent data communication.
Further, in embodiments of the present invention, the distributed data service bus is made up of multiple data domains, each domain containing a set of publishers and subscribers, and when a certain data is published in one domain, it will not be received by subscribers of any other domain.
Fig. 2 is a schematic flow chart of data transmission according to entity relationship in a data service bus according to an embodiment of the present invention.
As shown in fig. 2, in an embodiment of the present invention, a data writer may be used to send data, the data writer being associated with a single topic, a particular topic may have multiple corresponding data writers. The data writers are responsible for queuing the data and transmitting it to the publisher for transmission.
Further, in embodiments of the present invention, publishers are responsible for obtaining published data and publishing it to all relevant subscribers in the domain, which publishers own and manage data writers. One data writer may be owned by only a single publisher, and one publisher may correspond to multiple data writers, based on which the same publisher may send data for multiple different topics of different data types.
Further, in embodiments of the present invention where data readers access data received on a data service bus, a single topic may be associated with one data reader, and a particular topic may correspond to more than one data reader.
Further, in the embodiment of the present invention, the subscriber is responsible for publishing the data actually received, the subscriber owns and manages the data reader, the association between the data reader and the subscriber is subscription, one data reader can only have one single subscriber, and one subscriber can correspond to a plurality of data readers. Based on this, the same subscriber can receive data for many different topics of different data types.
Further, in embodiments of the present invention, multi-source heterogeneous data is transmitted and received within corresponding data domains, where publishers and subscribers are used to manage single or multiple data writers and data readers, respectively, one data reader and one data writer must be associated with the same topic, based on which data published by the data writers can only be received by subscribed data readers.
In embodiments of the present invention, the publish/subscribe mode reduces the consumption of sending data over the network compared to a client/server architecture, with occasional subscription requests at low bandwidth replacing a large number of high bandwidth client requests. Also, for regular data exchanges, publish/subscribe is more efficient in reaction time and bandwidth than client/server side.
And, in embodiments of the present disclosure, the data subscription layer corresponds to subscribers, and subscribers in the data subscription layer receive data from publishers, transmit it to any data readers connected thereto, and then the data readers obtain data from subscribers and reverse list it as the topic to the appropriate type.
In the embodiment of the invention, the message subscription/release interface only needs to subscribe concerned data, and automatically sends the data to the subscriber when the data arrives, and the data transmission is packaged in a JSON format.
Further, in an embodiment of the present invention, the data types in the data subscription layer may include file data, video and service data, and streaming media data.
In an embodiment of the present invention, the determining, by the data subscription layer, the corresponding transmission manner based on the type of the received data may include:
if the type of the received data is file data, packaging according to the file type, the file size and the file address of the file data and then transmitting;
if the type of the received data is streaming media, transmitting the streaming media data in real time;
and if the type of the received data is video and service data, the video and service data are periodically transmitted.
In the multi-source heterogeneous data cross-system collaborative management system, the system comprises a data access layer, a data bus layer and a data subscription layer. The data subscription layer is used for acquiring data demand tasks and types of the data demand tasks, analyzing the acquired data demand tasks and subscribing the data of the data bus layer according to the analysis result of the data request and the types of the data demand tasks; the data bus layer is used for sending the analysis result of the data subscription layer, the data demand task and the type of the data demand task to the data access layer; the data access layer is used for sending the data demand task and the type of the data demand task to the corresponding data source system according to the analysis result of the data bus layer. The method and the device provide standardized and intelligent targeted multi-source heterogeneous data scheduling and updating strategies, effectively shorten data collection and updating time, realize multi-main-body coordination management of the multi-center heterogeneous online system, meet the demand of upper-layer application on demand collection of multi-source heterogeneous data, enable limited transmission channel resources to be used for transmitting data products which are most urgently needed by collection users all the time, and solve the problem of low efficiency of collaborative management and collection service.
Fig. 3 is a flow chart of a multi-source heterogeneous data cross-system collaborative management method according to an embodiment of the present application, and as shown in fig. 3, may include:
step 301, obtaining a data demand task and a type of the data demand task.
And step 302, analyzing the data demand task, and transmitting the data demand to the corresponding data source system based on the analysis result and the type of the data demand task.
Step 303, receiving data corresponding to the data demand task sent by the data source system.
And 304, determining and outputting a corresponding transmission mode based on the type of the received data.
To achieve the above embodiments, the present disclosure also proposes a non-transitory computer-readable storage medium.
The non-transitory computer readable storage medium provided by the embodiments of the present disclosure stores a computer program; the computer program, when executed by a processor, enables a multi-source heterogeneous data cross-system collaborative management system as shown in fig. 3.
In the multi-source heterogeneous data cross-system collaborative management system, the multi-source heterogeneous data cross-system collaborative management method and the storage medium, the multi-source heterogeneous data cross-system collaborative management system comprises a data access layer, a data bus layer and a data subscription layer. The data subscription layer is used for acquiring data demand tasks and types of the data demand tasks, analyzing the acquired data demand tasks and subscribing the data of the data bus layer according to the analysis result of the data request and the types of the data demand tasks; the data bus layer is used for sending the analysis result of the data subscription layer, the data demand task and the type of the data demand task to the data access layer; the data access layer is used for sending the data demand task and the type of the data demand task to the corresponding data source system according to the analysis result of the data bus layer. The method and the device provide standardized and intelligent targeted multi-source heterogeneous data scheduling and updating strategies, effectively shorten data collection and updating time, realize multi-main-body coordination management of the multi-center heterogeneous online system, meet the demand of upper-layer application on demand collection of multi-source heterogeneous data, enable limited transmission channel resources to be used for transmitting data products which are most urgently needed by collection users all the time, and solve the problem of low efficiency of collaborative management and collection service.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and additional implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present application.
Although embodiments of the present application have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the application, and that variations, modifications, alternatives, and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the application.
Claims (8)
1. A multi-source heterogeneous data cross-system collaborative management system, the system comprising:
the data subscription layer is used for acquiring a data demand task and the type of the data demand task, analyzing the acquired data demand task and subscribing the data of the data bus layer according to the analysis result of the data request and the type of the data demand task;
the data bus layer is used for sending the analysis result of the data subscription layer, the data demand task and the type of the data demand task to the data access layer;
the data access layer is used for sending the data demand task and the type of the data demand task to a corresponding data source system according to the analysis result of the data bus layer;
the data bus layer is further used for receiving the multi-source heterogeneous data collected by the data access layer, defining different data domains according to different data sources of the multi-source heterogeneous data, defining different topics under different data domains according to different data types under different data domains, and publishing the data in the different data domains;
after the data source system receives the data demand task, sending data corresponding to the data demand task to the multi-source heterogeneous data cross-system collaborative management system according to the type of the data demand task; the multi-source heterogeneous data comprise satellite remote sensing data, aerial remote sensing data, ground video data, network information data, geographic information data and business information data, and the types of data demand tasks acquired in the data subscription layer comprise conventional service mode demands, emergency service mode demands and special service mode demands;
after the data source system receives the data demand task, according to the type of the data demand task, sending data corresponding to the data demand task to the multi-source heterogeneous data cross-system collaborative management system, wherein the multi-source heterogeneous data cross-system collaborative management system comprises:
when the type of the data demand task is the conventional service mode demand, if the data type is file data, the data source system periodically and actively pushes corresponding regional catalogue information, and a data transmission proxy service in the data access layer monitors the change of the catalogue information and periodically sends the change to the multi-source heterogeneous data cross-system collaborative management system; if the data type is video/service data, the data source system collects, updates and gathers the multi-source heterogeneous data cross-system collaborative management system in real time; if the data type is the data source data under different network system environments, performing cross-network data transmission and information interaction with a data bus layer through a data access layer;
when the type of the data demand task is emergency service mode demand, if the data type is space remote sensing data, after the data source system receives an explicit entity data demand urgent task, on one hand, the priority authority is improved through system configuration, on the other hand, conflict resolution with other tasks is completed, satellite orbit or attitude arrangement shooting tasks are timely adjusted, data/information service is timely obtained, corresponding processing flow is simplified, and meanwhile, according to specific resource types, product grades, time and area demands, automatic distribution service is realized through data source system background configuration and is sent to a multi-source heterogeneous data cross-system collaborative management system; if the data type is unmanned plane data, the data source system submits an air flight application after receiving an emergency task, completes the downloading processing of the flight task and the data, and transmits the data to the data management platform through the multi-source heterogeneous data cross-system collaborative management system; if the data types are video and business data, the data source system mobilizes video acquisition equipment in the place where the emergency is located after receiving the emergency task, adjusts the angle and focal length of video acquisition according to the requirement, monitors the event in all directions, mobilizes intelligent analysis resources, analyzes related people, vehicles and objects in real time according to the event property, traces back, analyzes and predicts the event, and transmits analysis results to the multi-source heterogeneous data cross-system collaborative management system;
when the type of the data demand task is special service mode demand, if the data type is remote sensing data, the data source system acquires catalogue information or entity data of corresponding data according to the definite time period, frequency, region and load requirements of the task and pushes the catalogue information or entity data to the multi-source heterogeneous data cross-system collaborative management system; if the data types are video and service data, the data source system acquires the video and service data in a specific time period and an intelligent analysis result thereof and pushes the video and service data to the multi-source heterogeneous data cross-system collaborative management system;
the data subscription layer is further used for analyzing the data demand task to obtain a data source system number related to the data demand task;
the data access layer is further configured to send the data demand task to a corresponding data source system according to the data source system number related to the data demand task parsed by the data subscription layer.
2. The multi-source heterogeneous data cross-system collaborative management system of claim 1,
the data access layer is also used for safely accessing multi-source heterogeneous data from different data source systems;
the data subscription layer is further configured to receive the published data in the data bus layer, determine a type of the received data, and determine a corresponding transmission mode for transmission based on the type of the received data.
3. The multi-source heterogeneous data cross-system collaborative management system of claim 2, wherein the data bus layer configures a topic name, a data type, and a set of QOS policies for each topic.
4. The multi-source heterogeneous data cross-system collaborative management system according to claim 2, wherein the data access layer is further configured to securely access multi-source heterogeneous data from different system sources through an access gateway, respectively.
5. The multi-source heterogeneous data cross-system collaborative management system of claim 2,
the data types in the data subscription layer comprise: file data, video and service data, streaming media data.
6. The multi-source heterogeneous data cross-system collaborative management system of claim 5, wherein the data subscription layer determines a corresponding transmission manner for transmission based on a type of the received data, comprising:
if the type of the received data is file data, packaging according to the file type, the file size and the file address of the file data and then transmitting;
if the type of the received data is streaming media, transmitting the streaming media data in real time;
and if the type of the received data is video and service data, the video and service data are periodically transmitted.
7. A method for application to a multi-source heterogeneous data cross-system collaborative management system of any of claims 1-6, the method comprising:
acquiring a data demand task and the type of the data demand task;
analyzing the data demand task, and transmitting the data demand to a corresponding data source system based on an analysis result and the type of the data demand task;
receiving data corresponding to the data demand task sent by the data source system;
determining a corresponding transmission mode based on the type of the received data and outputting the transmission mode;
the method further comprises the steps of:
receiving multi-source heterogeneous data collected by the data access layer, defining different data domains according to different data sources of the multi-source heterogeneous data, defining different topics under different data domains according to different data types under different data domains, and publishing the data in the different data domains;
after the data source system receives the data demand task, sending data corresponding to the data demand task to the multi-source heterogeneous data cross-system collaborative management system according to the type of the data demand task;
the multi-source heterogeneous data comprise satellite remote sensing data, aerial remote sensing data, ground video data, network information data, geographic information data and business information data, and the types of data demand tasks acquired in the data subscription layer comprise conventional service mode demands, emergency service mode demands and special service mode demands;
after the data source system receives the data demand task, according to the type of the data demand task, sending data corresponding to the data demand task to the multi-source heterogeneous data cross-system collaborative management system, wherein the multi-source heterogeneous data cross-system collaborative management system comprises:
when the type of the data demand task is the conventional service mode demand, if the data type is file data, the data source system periodically and actively pushes corresponding regional catalogue information, and a data transmission proxy service in the data access layer monitors the change of the catalogue information and periodically sends the change to the multi-source heterogeneous data cross-system collaborative management system; if the data type is video/service data, the data source system collects, updates and gathers the multi-source heterogeneous data cross-system collaborative management system in real time; if the data type is the data source data under different network system environments, performing cross-network data transmission and information interaction with a data bus layer through a data access layer;
when the type of the data demand task is emergency service mode demand, if the data type is space remote sensing data, after the data source system receives an explicit entity data demand urgent task, on one hand, the priority authority is improved through system configuration, on the other hand, conflict resolution with other tasks is completed, satellite orbit or attitude arrangement shooting tasks are timely adjusted, data/information service is timely obtained, corresponding processing flow is simplified, and meanwhile, according to specific resource types, product grades, time and area demands, automatic distribution service is realized through data source system background configuration and is sent to a multi-source heterogeneous data cross-system collaborative management system; if the data type is unmanned plane data, the data source system submits an air flight application after receiving an emergency task, completes the downloading processing of the flight task and the data, and transmits the data to the data management platform through the multi-source heterogeneous data cross-system collaborative management system; if the data types are video and business data, the data source system mobilizes video acquisition equipment in the place where the emergency is located after receiving the emergency task, adjusts the angle and focal length of video acquisition according to the requirement, monitors the event in all directions, mobilizes intelligent analysis resources, analyzes related people, vehicles and objects in real time according to the event property, traces back, analyzes and predicts the event, and transmits analysis results to the multi-source heterogeneous data cross-system collaborative management system;
when the type of the data demand task is special service mode demand, if the data type is remote sensing data, the data source system acquires catalogue information or entity data of corresponding data according to the definite time period, frequency, region and load requirements of the task and pushes the catalogue information or entity data to the multi-source heterogeneous data cross-system collaborative management system; if the data types are video and service data, the data source system acquires the video and service data in a specific time period and an intelligent analysis result thereof and pushes the video and service data to the multi-source heterogeneous data cross-system collaborative management system;
the parsing the data demand task, and sending the data demand to a corresponding data source system based on the parsing result and the type of the data demand task, including:
analyzing the data demand task to obtain a data source system number related to the data demand task;
and sending the data demand task to the corresponding data source system according to the number of the data source system related to the analyzed data demand task.
8. A non-transitory computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the method as claimed in claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210507966.9A CN115134421B (en) | 2022-05-10 | 2022-05-10 | Multi-source heterogeneous data cross-system collaborative management system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210507966.9A CN115134421B (en) | 2022-05-10 | 2022-05-10 | Multi-source heterogeneous data cross-system collaborative management system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115134421A CN115134421A (en) | 2022-09-30 |
| CN115134421B true CN115134421B (en) | 2024-02-20 |
Family
ID=83376529
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210507966.9A Active CN115134421B (en) | 2022-05-10 | 2022-05-10 | Multi-source heterogeneous data cross-system collaborative management system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115134421B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116980449B (en) * | 2023-09-14 | 2023-12-19 | 哈尔滨工程大学三亚南海创新发展基地 | Collaborative management method, system, computer equipment and medium for data resource main body |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103944955A (en) * | 2014-01-21 | 2014-07-23 | 中国科学院遥感与数字地球研究所 | Handset space information service method based on wireless network |
| CN104639496A (en) * | 2013-11-06 | 2015-05-20 | 北京航天长峰科技工业集团有限公司 | Internet of things perception information acquisition method |
| CN106570081A (en) * | 2016-10-18 | 2017-04-19 | 同济大学 | Semantic net based large scale offline data analysis framework |
| CN107819824A (en) * | 2017-10-09 | 2018-03-20 | 中国电子科技集团公司第二十八研究所 | A kind of Urban Data opens and information service system and method for servicing |
| CN108737161A (en) * | 2018-04-20 | 2018-11-02 | 中国人民解放军国防科技大学 | Real-time data bus management method and system for intelligent agent cooperation system |
| CN109460952A (en) * | 2018-10-25 | 2019-03-12 | 北京卫星信息工程研究所 | Heterogeneous network converged communication device and communication means for Emergency Logistics |
| CN109992672A (en) * | 2019-04-11 | 2019-07-09 | 华北科技学院 | Knowledge mapping construction method based on disaster scene |
| CN110532493A (en) * | 2019-08-29 | 2019-12-03 | 北京明略软件系统有限公司 | Processing method and processing device, storage medium and the electronic device of data |
| WO2021222384A1 (en) * | 2020-04-28 | 2021-11-04 | Strong Force Intellectual Capital, Llc | Digital twin systems and methods for transportation systems |
| CN114153920A (en) * | 2021-11-19 | 2022-03-08 | 国家电投集团新疆能源化工有限责任公司 | Big data edge platform and method |
| CN114398442A (en) * | 2022-01-25 | 2022-04-26 | 中国电子科技集团公司第十研究所 | Data-driven information processing system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11921672B2 (en) * | 2017-07-31 | 2024-03-05 | Splunk Inc. | Query execution at a remote heterogeneous data store of a data fabric service |
| US20220027417A1 (en) * | 2020-07-17 | 2022-01-27 | Rapidsos, Inc. | Modular application programming interface system |
-
2022
- 2022-05-10 CN CN202210507966.9A patent/CN115134421B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104639496A (en) * | 2013-11-06 | 2015-05-20 | 北京航天长峰科技工业集团有限公司 | Internet of things perception information acquisition method |
| CN103944955A (en) * | 2014-01-21 | 2014-07-23 | 中国科学院遥感与数字地球研究所 | Handset space information service method based on wireless network |
| CN106570081A (en) * | 2016-10-18 | 2017-04-19 | 同济大学 | Semantic net based large scale offline data analysis framework |
| CN107819824A (en) * | 2017-10-09 | 2018-03-20 | 中国电子科技集团公司第二十八研究所 | A kind of Urban Data opens and information service system and method for servicing |
| CN108737161A (en) * | 2018-04-20 | 2018-11-02 | 中国人民解放军国防科技大学 | Real-time data bus management method and system for intelligent agent cooperation system |
| CN109460952A (en) * | 2018-10-25 | 2019-03-12 | 北京卫星信息工程研究所 | Heterogeneous network converged communication device and communication means for Emergency Logistics |
| CN109992672A (en) * | 2019-04-11 | 2019-07-09 | 华北科技学院 | Knowledge mapping construction method based on disaster scene |
| CN110532493A (en) * | 2019-08-29 | 2019-12-03 | 北京明略软件系统有限公司 | Processing method and processing device, storage medium and the electronic device of data |
| WO2021222384A1 (en) * | 2020-04-28 | 2021-11-04 | Strong Force Intellectual Capital, Llc | Digital twin systems and methods for transportation systems |
| CN114153920A (en) * | 2021-11-19 | 2022-03-08 | 国家电投集团新疆能源化工有限责任公司 | Big data edge platform and method |
| CN114398442A (en) * | 2022-01-25 | 2022-04-26 | 中国电子科技集团公司第十研究所 | Data-driven information processing system |
Non-Patent Citations (2)
| Title |
|---|
| esearch on multi-data sources data center architecture at UOIT;Xiaoli Meng;《IEEE Xplore》;全文 * |
| 面向自然灾害应急任务的时空数据智能聚合方法;仇林遥;《CNKI中国知网》;正文第2.1-2.5节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115134421A (en) | 2022-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9503308B2 (en) | Method, device and system for processing content | |
| KR101179554B1 (en) | Mobile device adopting mobile cloud platform | |
| JP6240318B2 (en) | System and method for managing a wireless network | |
| CN107493191B (en) | Cluster node and self-scheduling container cluster system | |
| WO2017040997A1 (en) | Multiplexed demand signaled distributed messaging | |
| US20180248709A1 (en) | Managing channels in an open data ecosystem | |
| US11070639B2 (en) | Network infrastructure system and method for data processing and data sharing using the same | |
| CN108737527B (en) | Large-scale user access method and system suitable for platform | |
| CN111787058A (en) | Lightweight information subscription and push method in cross-domain virtual data space | |
| CN114615096B (en) | Event-driven architecture-based telecommunication charging method, system and related equipment | |
| CN109586973B (en) | Data stream hopping transmission method based on universal identification network system | |
| CN115134421B (en) | Multi-source heterogeneous data cross-system collaborative management system and method | |
| KR20160092672A (en) | Image processing system and device of meteorological and environmental satellite, and method using the same | |
| EP2996298A1 (en) | Open m2m system and method | |
| US9198029B2 (en) | Service processing method and device for ubiquitous UE | |
| EP2126718A1 (en) | Method and system for a context manager for a converged services framework | |
| CN105205735A (en) | Power dispatching data cloud service system and implementation method | |
| WO2023001083A1 (en) | Service providing method and system, gateway, device, and storage medium | |
| CN110502342A (en) | Motor-driven marginal information service network under a kind of interval network environment | |
| KR20190037095A (en) | Method and architecture of Network Infrastructure for Optimal Application Service Processing and Data Sharing among Application domains | |
| US11716382B2 (en) | System and method for replication of storage presentation in application orchestration | |
| Bano et al. | A novel approach to distributed model aggregation using apache kafka | |
| US20060179342A1 (en) | Service aggregation in cluster monitoring system with content-based event routing | |
| CN116112560B (en) | Data uplink and proxy method and system based on Reactive mechanism | |
| CN114024976A (en) | Big data service architecture based on 5G and method for constructing big data service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |