CN109408484A - A kind of assembly type planetary detection data archive system and method based on data-driven - Google Patents
A kind of assembly type planetary detection data archive system and method based on data-driven Download PDFInfo
- Publication number
- CN109408484A CN109408484A CN201811186417.6A CN201811186417A CN109408484A CN 109408484 A CN109408484 A CN 109408484A CN 201811186417 A CN201811186417 A CN 201811186417A CN 109408484 A CN109408484 A CN 109408484A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- archives
- message
- archiving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Abstract
Present disclose provides a kind of assembly type planetary detection data archive system based on data-driven, comprising: archives data interface module, data transmit-receive service module, message queue module, data-archiving-service cluster module, load balancing service module, cluster management service module.By by the archive modularization of the planetary detection data of different task, separate sources, planetary detection data archive system is constructed with the system architecture of loose coupling, improve the scalability of system, it is that the different archives data component of the real-time dynamically load of driving carries out archives data with data, improves the safety of data；Simultaneously because whole system supports clustering deployment, archival task assignment is rationally carried out by load balancing, improve resource utilization ratio and achieve oncurrent processing ability, solve the problems, such as conventional planet detection data filing system scalability, safety, high efficiency and in terms of exist.
Description
Technical field
This disclosure relates to the moon and planetary detection technical field more particularly to a kind of assembly type planet based on data-driven
Detection data filing system and method.
Background technique
Data are the moon and planetary exploration mission material wealth the most valuable, and the safety of data is achieved is with reliable management
It saves the moon and the planetary science primary basic task of detection achievement and subsequent carries out scientific research, technological applications, section in a deep going way
The basis of general publicity and knowledge dissemination.
Firstly, under the traction of each moon and planetary exploration mission difference scientific goal, the operation of satellite and detector
Period, the payload of carrying, data acquisition modes and data processing algorithm etc. can have larger difference, result in the need for achieving
The detection data type and content information of management constantly change；Along with the detection mode of different task selection is different, make total
It also needs to be changed therewith with interface specification according to filing operation flow.Under traditional mode, filing system is in order to meet new task
Archiving needs need to carry out the improvement upgrading of adaptability, system has that compatibility, stability, improvement cost height etc. are a series of asks
Topic, and the system reform difficulty of a close coupling, scalability difference can be also continuously increased.
Secondly, traditional moon and the archive of planetary detection data are carried out based on survey plan, according to survey plan
It arranges, each archives data requesting party will need the data achieved to be placed in advance in specified shared-file system, archives data
Timing reading from shared-file system needs the data achieved to be achieved according to schedule for side, although lunar exploration data can be completed
Archive, but detection data can not be achieved in real time, while being also unable to satisfy the real time service demand of data.
Furthermore the source of the moon and planetary detection data is more, and archiving needs are frequent during detection mission, when shorter
Between can there is in section a large amount of archives data to request simultaneously, if filing system concurrency performance is lower, will lead to a large amount of archive requests
It overstocks, data cannot be achieved in time, influence the safety of data and the real time service ability of data, therefore this system passes through
Clustering deployment, improves the parallel processing capability of system, guarantees timely, all archives datas received of safe handling and asks
It asks.
Summary of the invention
(1) technical problems to be solved
Present disclose provides a kind of assembly type planetary detection data archive system and method based on data-driven, at least
Part solves technical problem set forth above.
(2) technical solution
According to one aspect of the disclosure, a kind of assembly type planetary detection archives data system based on data-driven is provided
System, comprising: archives data interface module, for the archives data between authority data archive request side and notebook data filing system
The interface protocol of interaction constraint, data transmit-receive service module, for completing the data transmit-receive between system and archives data requesting party；
Message queue module, for storing data transmitting-receiving service module receive archives data request, and to data archive request into
Row is lined up, and is guaranteed the safety for the archives data request that data transmit-receive service receives, is guaranteed each request at least processed one
It is secondary；Data-archiving-service cluster module, including multiple data-archiving-service examples, for executing the business service of archives data；
Load balancing service module calculates load condition in current data archival service cluster by load balance scheduling algorithm in real time
Optimal data-archiving-service example, and data transmit-receive service is passed it to, so that the filing task of whole system is multiple
Balanced distribution is carried out between data-archiving-service；Cluster management service module, it is each in data-archiving-service cluster for monitoring
The operating status of a service processes.
In some embodiments, data-archiving-service cluster module is based on OSGI (Open Service Gateway
Initiative) the dynamic assembly service that specification is realized, by 1~N number of data-archiving-service example forms；Each archives data
Service Instance provides identical archives data ability, and independent mutually, and each data-archiving-service example is according to specifically depositing
Shelves demand, selects the component for needing to load starting, to realize archival service functional class by way of configuring XML file
Freely scale.
In some embodiments, data-archiving-service module is by a series of archives data component and component loading frame group
At, each component provides the archival capabilities of a kind of planetary detection data, and component loading frame provides the dynamic load capability of component,
When data-archiving-service receives archives data task, according to the data category that this is achieved, component loading frame dynamically load
Corresponding component executes specific archive by the component, achieves after completing, and the archives data for generating this archival task is replied
Inventory is described in detail the archive situation of each data, and sends archives data to data transmit-receive service and reply message, is received by data
Archive reply message is transmitted to archives data requesting party by hair service, starts multiple realities on data-archiving-service module physical
Example, provides data-archiving-service in a manner of cluster.
In some embodiments, archives data interface module includes archives data message and archives data inventory, wherein institute
The archives data message stated is divided into transmission label, using three parts of label and application data section, wherein transmission label is used for
Length, message source, the message whereabouts, message generation time of entire message are described；Using label for describing detection mission rank
The length of section, type of message and application data section；Application data section is deposited for describing archives data request inventory or data
Shelves reply the complete trails of inventory；Archives data inventory includes that archives data request inventory and archives data reply inventory, the number
It replys the file of inventory in xml format according to archive request inventory and archives data to be indicated, wherein archives data request is clear
The data file information for needing to achieve in description is applied alone, main includes essential information, history file number and the archive of archival task
Listed files；Archives data replys the implementing result that inventory is used to describe each archives data task, including achieving the base replied
This information, history file number, the archival state of each data file and archive failure cause.
In some embodiments, the message queue module is the first in first out obstruction based on Producer-consumer problem mode
Formula queue, joining the team with thread-safe, go out team, delete and can persistence ability；Data transmit-receive service module includes:
Listening thread, the connection request of archive request side, the message for establishing this system and archives data requesting party are logical for receiving data
Road；Message sink thread, the archives data request message that archive request side is sent for receiving data, as message queue
The archives data request message received is stored in message queue module by producer；Message queue scans thread, as message team
The consumer of column module obtains archives data request from message queue module, by load balancing service from data-archiving-service
The data-archiving-service example for executing this subtask is got in cluster, and archives data request is then transmitted to the data and is deposited
Shelves Service Instance executes archives data task；Message sends thread, and the archives data for forwarding data-archiving-service to send returns
Multiple message gives archives data requesting party, reports to this archival task performance.
In some embodiments, load balancing service module each archives data in archival service cluster for receiving data
The real time load information that Service Instance reports, the archive including the occupied memory of data-archiving-service example, CPU, execution are appointed
Business number, is calculated by dynamic, is periodically that foundation is ranked up from small to large with load capacity to archival service example, for data transmit-receive
Service, which provides, loads relatively small data-archiving-service entrance.
In some embodiments, the basic letter registered when cluster management service is started by each service processes in cluster
Breath, including process title, the working space of process and periodically the heartbeat message that reports monitors each running state of a process, works as hair
Some process not sent heartbeat within the period of setting in existing cluster, the process will be judged as the presence of exception, immediately
Restart the abnormal process；After process restarts recovery, not completing when being abnormal for task will continue to execute.
Assembly type planetary detection data based on data-driven are deposited more another aspect of the present disclosure provides
Shelves method, its step are as follows:
Archives data requesting party will need the data achieved and achieve list to copy shared memory to first, so
Archives data request is sent to the data transmit-receive service module of this filing system by archives data interface module afterwards；
After data transmit-receive service module receives archives data request message, the message is stored in message queue module first
And persistence, it is at least processed primary to guarantee the archives data request message received, then successively read from message queue module
Access is according to archive request message；
Data transmit-receive service module is obtained in current data archival service module by load balancing service module and loads phase
To lesser archival service example, by the archives data request message read from message queue module be transmitted to the load compared with
Small archival service example executes this archival task by the archival service example；
After data-archiving-service module receives archives data request message, analytic message generates after completing data filing
Archives data replys inventory, records the archive performance of each data, and archives data reply inventory is copied to shared deposit
Storage area generates archives data reply message and is sent to data transmit-receive service module；
After data transmit-receive service module receives archives data reply message, it is sent to archives data requesting party, completes this
Secondary archives data.
In some embodiments, data-archiving-service module analytic message includes: data-archiving-service module from message
The storage location for achieving list in shared memory is obtained, the archive list is then read and is parsed, from
And this all data file for needing to achieve is obtained, and successively right based on particular row sing data format used by such data
Data are checked, for checking that underproof data are copied to recycle bin memory block, and abandon the archive of such data,
Database is stored in for its metadata information of the reading data of passed examination, and by data file copies to persistent storage, it is complete
At such archives data.
In some embodiments, after message is stored in message queue by data transmit-receive service module, for entering the number of queue
According to archive request, armed state is set as by message queue module first, while being persisted to disk, it is ensured that this system receives
The archives data request arrived can be processed, and the archives data for going out team is requested, and will be set as by message queue module
Processing status, at this time archives data request physically still in message queue, finish, and receive until the request is processed
To after removal request, can just be removed from queue and disk by message queue module.
Preferably, the data transmit-receive service module, load balancing service module and data-archiving-service module timing
Heartbeat message is sent to cluster management service module, reports its operating status, cluster management service module is sent out by each service
The heartbeat situation sent judges each service whether in normal service state, to restore to exception service.
Preferably, each data-archiving-service module timing reports respective loading condition to load balancing service module,
Wherein loading condition includes memory, the archival task number that this service occupies, and load balancing service passes through each data-archiving-service
The loading condition of example calculates the lesser data-archiving-service of load in real time, in case receiving new archival task.
(3) beneficial effect
It can be seen from the above technical proposal that assembly type planetary detection data archive system of the disclosure based on data-driven
Structure and method at least have the advantages that one of them:
(1) by by the archive modularization of the planetary detection data of different task, separate sources, with the system tray of loose coupling
Structure constructs planetary detection data archive system, improves the scalability of system, is to drive real-time dynamically load different with data
Archives data component carry out archives data, improve the safety of data；
(2) since whole system supports clustering deployment, archival task assignment is rationally carried out by load balancing, is improved
Resource utilization ratio and archive oncurrent processing ability solve conventional planet detection data filing system in scalability, peace
Quan Xing, high efficiency and concurrency etc. there are the problem of.
Detailed description of the invention
Fig. 1 is to implement the assembly type multitask planetary detection data archive system structure based on data-driven according to the disclosure
Schematic diagram.
Fig. 2 is archives data message composite structural diagram in the archives data interface according to disclosure embodiment.
Fig. 3 is the assembly type multitask planetary detection archives data side based on data-driven according to disclosure embodiment
Method flow chart.
Fig. 4 is the assembly type multitask planetary detection archives data system based on data-driven according to disclosure embodiment
System workflow schematic diagram.
Specific embodiment
Present disclose provides a kind of assembly type multitask planetary detection data archive system and method based on data-driven,
Archive request can be automatically parsed during the moon and planetary detection archives data, detection number is met in a manner of data-driven
The demand with service is achieved when factually；By can dynamically load data-archiving-service assembly type clustering design, realize data
Freely scaling for functional class is achieved, meets the different demands of more detection mission archives datas, efficiently solves filing system
The problems such as concurrency low with the close coupling of detection mission, poor expandability and system.
In order to solve the technical issues of proposing in present disclosure, entire technical solution by
For the purposes, technical schemes and advantages of the disclosure are more clearly understood, below in conjunction with specific embodiment, and reference
The disclosure is further described in attached drawing.
Disclosure some embodiments will be done referring to appended attached drawing in rear and more comprehensively describe to property, some of but not complete
The embodiment in portion will be shown.In fact, the various embodiments of the disclosure can be realized by many different forms, without that should be solved
It is interpreted as being limited to embodiments set forth herein；Relatively, these embodiments are provided so that the disclosure meets applicable legal requirement.
In an exemplary embodiment of the disclosure, a kind of assembly type multitask planet based on data-driven is provided
Detection data filing system.Fig. 1 is assembly type multitask planetary detection archives data of the embodiment of the present disclosure based on data-driven
The structural schematic diagram of system.As shown in Figure 1, assembly type multitask planetary detection archives data system of the disclosure based on data-driven
System includes: archives data interface module, message queue module, data transmit-receive service module, data-archiving-service cluster module, bears
Carry balancing service module, cluster management service module.
The angle implemented below from specific deployment visits the present embodiment based on the assembly type multitask planet of data-driven
The various pieces of measured data filing system are described in detail.
System uses the driving with data for the operating of entire data archive system, as long as each archives data requesting party need to
The data preparation to be achieved finishes, and by archives data interface notification notebook data filing system, that is, can trigger notebook data and achieves system
System carry out archives data, as shown in Figure 1, solve with archive plan be drive data archive system present in data can not
In time achieve, caused by problem of data safety, support notebook data filing system complete archives data task all modules
It is as follows:
Archives data interface module is for the archives data between authority data archive request side and notebook data filing system
The interface protocol of interaction constraint is the foundation that this filing system and each archives data requesting party carry out information exchange, including number
According to message (as shown in Figure 2) and two, archives data inventory are achieved, wherein archives data inventory includes archives data request inventory
Inventory is replied with archives data.
The archives data message is divided into transmission label, using three parts of label and application data section, wherein passing
Defeated label is used to describe the length of entire message, message source, message whereabouts, message generate the time (including year, month, day, when,
Minute, second)；Using label for describing detection mission stage, type of message (archive request and archive are replied) and using data
The length of section；Application data section is used to describe archives data request inventory or archives data replys the complete trails of inventory.
Archives data request inventory and archives data described in it are replied the file of inventory in xml format and are indicated,
The wherein data file information that archives data request inventory is used to describe to need to achieve, main includes the essential information of archival task
(satellite mark, task type), history file number and history file list；Archives data replys inventory for describing each data
The implementing result of archival task, main includes achieving the essential information (satellite mark, task type) replied, history file number
(total number, success number and failure number), the archival state of each data file and archive failure cause (are achieving failure shape
Under state).
Message queue module is the first in first out block type queue based on Producer-consumer problem mode, is pacified with thread
It is complete join the team, go out team, deleting and can abilities, the main function such as persistence on the one hand receive and dispatch service module for storing data
The archives data request received, and data archive request is lined up, the processing for solving the request of high-volume archives data is asked
Topic；On the other hand the safety requested for guaranteeing the archives data that data transmit-receive service receives, guarantees each request at least
It is processed primary.In the present embodiment, it is requested for entering the archives data of queue, is set as to be processed by message queue first
State, while being persisted to disk, it is ensured that the archives data request that this system receives can be processed, for going out the data of team
Archive request will be set as just in processing status by message queue, and archives data request at this time is physically still in message team
It, just can be by message queue from queue and disk until the request is processed finishes, and after receiving removal request in column
It removes, can be handled by the response of safety with the archives data request for ensuring all；
Data transmit-receive service module is the communication bridge between this system and archives data requesting party, plays and forms a connecting link
Effect mainly sends thread by listening thread, message sink thread, message and message queue scanning thread forms, wherein
The connection request of listening thread archive request side for receiving data, the message for establishing this system and archives data requesting party are logical
Road；The message sink thread archives data request message that archive request side is sent for receiving data, as message queue
The archives data request message received is stored in message queue module by producer；Message queue scans thread as message queue
The consumer of module obtains archives data request from message queue module, by load balancing service from data-archiving-service collection
The data-archiving-service example for executing this subtask is got in group, and archives data request is then transmitted to the archives data
Service Instance executes archives data task；Message sends thread and is mainly used for the archives data for forwarding data-archiving-service to send time
Multiple message gives archives data requesting party, reports to this archival task performance；
Data-archiving-service cluster module includes multiple data-archiving-service examples, is the industry that this system executes archives data
Business service, is the core of system, the dynamic group realized based on OSGI (Open Service Gateway Initiative) specification
Part service, by 1~N number of data-archiving-service example forms.Each data-archiving-service example provides identical archives data
Ability, mutual independence between them, and there is same status.When there are the data of new type to need to achieve, need to only be based on
New demand develops corresponding archive component, and the component Configuration completely will not to each data-archiving-service example
Influence existing archives data.Meanwhile each data-archiving-service example can pass through and configure according to specific archiving needs
The mode of XML file selects that the component for loading starting is needed to improve to realize freely scaling for archival service functional class
The scalability of data-archiving-service.
Data-archiving-service module is made of a series of archives data component and component loading frame, and each component provides
The archival capabilities of a kind of planetary detection data, component loading frame provide the dynamic load capability of component, work as data-archiving-service
Archives data task is received, according to the data category that this is achieved, the corresponding component of component loading frame dynamically load, by this
Component executes specific archive, achieves after completing, and the archives data for generating this archival task replys inventory, and detailed description is often
The archive situation of a data, and send archives data to data transmit-receive service and reply message, by data transmit-receive service by the archive
It replys message and is transmitted to archives data requesting party, can star multiple examples on data-archiving-service module physical, with cluster
Mode provides data-archiving-service；
Each data-archiving-service example reports load balancing service module in archival service cluster for receiving data
Real time load information (such as: the occupied memory of data-archiving-service example, CPU, execution archival task number), pass through dynamic
It calculates, is periodically to provide load according to being ranked up from small to large for data transmit-receive service with load capacity to archival service example
Relatively small data-archiving-service entrance calculates current data archival service collection by load balance scheduling algorithm in real time
The optimal data-archiving-service example of load condition in group, and data transmit-receive service is passed it to, so that whole system is returned
Shelves task carries out balanced distribution between multiple data-archiving-services, and load is avoided excessively to concentrate and system performance is caused to generate
Bottleneck efficiently dispatches the execution of archives data business.
Cluster management service module is used to monitor the operating status of each service processes in data-archiving-service cluster, is one
Finger daemon in a memory-resident, to ensure the high availability of entire cluster.Cluster management service passes through each in cluster
What a service processes were registered when starting essential information (such as: the working space of process title, process) and the heartbeat that periodically reports
The each running state of a process of information monitoring, when discovery cluster in the not sent heartbeat within the period of setting of some process,
The process will be judged as the presence of exception, restart the abnormal process immediately.After process restarts recovery, when being abnormal not
The task of completion will continue to execute, and thus the continuity of task is guaranteed.
In second exemplary embodiment of the disclosure, a kind of assembly type multitask row based on data-driven is provided
The archiving method of talent scout's measured data, Fig. 3 are the assembly type multitask planets based on data-driven according to disclosure embodiment
Detection data archiving method flow chart.Fig. 4 is the assembly type multitask row based on data-driven according to disclosure embodiment
Talent scout's measured data filing system workflow schematic diagram.As shown in Figure 3,4, the assembly type multitask planet based on data-driven is visited
Steps are as follows for the archiving method of measured data:
S1, archives data requesting party will need the data achieved and archive list first, and (achieving list is
One XML file, description need the data file name, store path and the file size that achieve) copy shared memory to,
Then it is sent by archives data interface module (archives data request message) to the data transmit-receive service module of this filing system
Archives data requests (message format is as shown in Figure 2)；
The message after data transmit-receive service module receives archives data request message, is stored in message queue first by S2
Module and persistence guarantee that the archives data request message received is at least processed primary；, data transmit-receive service module is from disappearing
Archives data request message is successively read in breath Queue module；
S3, data transmit-receive service module are relatively small by loading in the acquisition current archival service of load balancing service module
Archival service example；And the archives data request message read from message queue module is transmitted to the lesser archive of load
Service Instance executes this archival task by the data-archiving-service example；
S4, after data-archiving-service module receives archives data request message, analytic message is obtained from message and is achieved
Then storage location of the list in shared memory reads the archive list and is parsed, to obtain this
The secondary all data files for needing to achieve, and based on particular row sing data format (such as PDS3, PDS4) used by such data
Successively data are checked, for checking that underproof data are copied to recycle bin memory block, and abandon such data
Archive, database is stored in for its metadata information of the reading data of passed examination, and by data file copies to permanently depositing
Storage area completes this kind of archives data, ultimately produces archives data and replys inventory, records the archive performance of each data, and
Archives data reply inventory is copied to shared memory, archives data reply message is generated and is sent to data transmit-receive service mould
Block；
S5 is sent to archives data requesting party, completes after data transmit-receive service module receives archives data reply message
This archives data.
Wherein, after message is stored in message queue by data transmit-receive service module, the archives data for entering queue is requested,
Armed state is set as by message queue first, while being persisted to disk, it is ensured that the archives data that this system receives is asked
Asking can be processed, and the archives data for going out team is requested, and will be set as by message queue just in processing status, at this time the data
Archive request is finished until the request is processed physically still in message queue, and after receiving removal request, just meeting
It is removed from queue and disk by message queue.
Data transmit-receive service module, load balancing service module and data-archiving-service module are fixed in this filing system
When to cluster management service module send heartbeat message, report its operating status, cluster management service module passes through each service
The heartbeat situation of transmission judges that each service whether in normal service state, to restore to exception service, improves whole
The high availability of a cluster.
Each data-archiving-service module needs timing to report respective loading condition to load balancing service module, wherein
Loading condition includes memory, the archival task number that this service occupies, and load balancing service passes through each data-archiving-service module
Loading condition calculate the lesser data-archiving-service example of load in real time, in case receiving new archival task.
In order to achieve the purpose that brief description, any technical characteristic for making same application is described all in above-described embodiment 1
And in this, without repeating identical narration.
So far, attached drawing is had been combined the embodiment of the present disclosure is described in detail.It should be noted that in attached drawing or saying
In bright book text, the implementation for not being painted or describing is form known to a person of ordinary skill in the art in technical field, and
It is not described in detail.In addition, the above-mentioned definition to each element and method be not limited in mentioning in embodiment it is various specific
Structure, shape or mode, those of ordinary skill in the art simply can be changed or be replaced to it.
Furthermore word "comprising" does not exclude the presence of element or step not listed in the claims.It is located in front of the element
Word "a" or "an" does not exclude the presence of multiple such elements.
In addition, unless specifically described or the step of must sequentially occur, there is no restriction in the above institute for the sequence of above-mentioned steps
Column, and can change or rearrange according to required design.And above-described embodiment can be based on the considerations of design and reliability, that
This mix and match is used using or with other embodiments mix and match, i.e., the technical characteristic in different embodiments can be freely combined
Form more embodiments.
Algorithm and display are not inherently related to any particular computer, virtual system, or other device provided herein.
Various general-purpose systems can also be used together with teachings based herein.As described above, it constructs required by this kind of system
Structure be obvious.In addition, the disclosure is also not for any particular programming language.It should be understood that can use various
Programming language realizes content of this disclosure described herein, and the description done above to language-specific is to disclose this public affairs
The preferred forms opened.
The disclosure can by means of include several different elements hardware and by means of properly programmed computer come
It realizes.The various component embodiments of the disclosure can be implemented in hardware, or to run on one or more processors
Software module is realized, or is implemented in a combination thereof.It will be understood by those of skill in the art that can be used in practice micro-
Processor or digital signal processor (DSP) are some or all in the relevant device according to the embodiment of the present disclosure to realize
The some or all functions of component.The disclosure be also implemented as a part for executing method as described herein or
Whole device or device programs (for example, computer program and computer program product).Such journey for realizing the disclosure
Sequence can store on a computer-readable medium, or may be in the form of one or more signals.Such signal can
To download from internet website, perhaps it is provided on the carrier signal or is provided in any other form.
Those skilled in the art will understand that can be carried out adaptively to the module in the equipment in embodiment
Change and they are arranged in one or more devices different from this embodiment.It can be the module or list in embodiment
Member or component are combined into a module or unit or component, and furthermore they can be divided into multiple submodule or subelement or
Sub-component.Other than such feature and/or at least some of process or unit exclude each other, it can use any
Combination is to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so disclosed
All process or units of what method or apparatus are combined.Unless expressly stated otherwise, this specification is (including adjoint power
Benefit require, abstract and attached drawing) disclosed in each feature can carry out generation with an alternative feature that provides the same, equivalent, or similar purpose
It replaces.Also, in the unit claims listing several devices, several in these devices can be by same hard
Part item embodies.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of each open aspect,
Above in the description of the exemplary embodiment of the disclosure, each feature of the disclosure is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the disclosed method should not be interpreted as reflecting the following intention: i.e. required to protect
The disclosure of shield requires features more more than feature expressly recited in each claim.More precisely, as following
Claims reflect as, open aspect is all features less than single embodiment disclosed above.Therefore,
Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, wherein each claim itself
All as the separate embodiments of the disclosure.
Particular embodiments described above has carried out further in detail the purpose of the disclosure, technical scheme and beneficial effects
Describe in detail it is bright, it is all it should be understood that be not limited to the disclosure the foregoing is merely the specific embodiment of the disclosure
Within the spirit and principle of the disclosure, any modification, equivalent substitution, improvement and etc. done should be included in the guarantor of the disclosure
Within the scope of shield.
Claims (10)
1. a kind of assembly type planetary detection data archive system based on data-driven, comprising:
Archives data interface module is interacted for the archives data between authority data archive request side and notebook data filing system
The interface protocol of constraint；
Data transmit-receive service module, for completing the data transmit-receive between system and archives data requesting party；
Message queue module receives and dispatches the archives data request that service module receives for storing data；And archives data is asked
It asks and is lined up, guarantee the safety for the archives data request that data transmit-receive service receives, guarantee that each request is at least located
Reason is primary；
Data-archiving-service cluster module, including multiple data-archiving-service examples, for executing the business service of archives data；
Load balancing service module is calculated in real time by load balance scheduling algorithm and is loaded in current data archival service cluster
The data-archiving-service example of state optimization, and data transmit-receive service is passed it to, so that the filing task of whole system exists
Balanced distribution is carried out between multiple data-archiving-services；
Cluster management service module, for monitoring the operating status of each service processes in data-archiving-service cluster.
2. assembly type planetary detection data archive system according to claim 1, wherein
Data-archiving-service cluster module is standardized based on OSGI (Open Service Gateway Initiative) and is realized
Dynamic assembly service, by 1~N number of data-archiving-service example forms；Each data-archiving-service example provides identical number
Independent according to archival capabilities, and mutually, each data-archiving-service example is according to specific archiving needs, by configuring XML text
The mode of part selects the component for needing to load starting, to realize freely scaling for archival service functional class.
3. assembly type planetary detection data archive system according to claim 1, wherein
Data-archiving-service module is made of a series of archives data component and component loading frame, and each component provides a kind of
The archival capabilities of planetary detection data, component loading frame provide the dynamic load capability of component, when data-archiving-service receives
To archives data task, according to the data category that this is achieved, the corresponding component of component loading frame dynamically load, by the component
Specific archive is executed, is achieved after completing, the archives data for generating this archival task replys inventory, and every number is described in detail
According to archive situation, and to data transmit-receive service send archives data reply message, the archive is replied by data transmit-receive service
Message is transmitted to archives data requesting party, starts multiple examples on data-archiving-service module physical, is provided in a manner of cluster
Data-archiving-service.
4. assembly type planetary detection data archive system according to claim 1, archives data interface module include data
Message and archives data inventory are achieved, wherein
The archives data message is divided into transmission label, using three parts of label and application data section, wherein transmission mark
Sign length, message source, the message whereabouts, message generation time for describing entire message；Using label for describing detection
The length of task phase, type of message and application data section；Application data section for describe archives data request inventory or
The complete trails of archives data reply inventory；
Archives data inventory includes archives data request inventory and archives data replys inventory, archives data request inventory and
Archives data is replied the file of inventory in xml format and is indicated, and wherein archives data request inventory needs to deposit for describing
The data file information of shelves, main includes essential information, history file number and the history file list of archival task；Archives data
Inventory is replied to be used to describe the implementing result of each archives data task, including achieve the essential information replied, history file number,
The archival state and archive failure cause of each data file.
5. assembly type planetary detection data archive system according to claim 1, wherein
The message queue module is the first in first out block type queue based on Producer-consumer problem mode；
Data transmit-receive service module includes:
Listening thread, for receiving data connection request of archive request side establish disappearing for this system and archives data requesting party
Cease channel；
Message sink thread, the archives data request message that archive request side is sent for receiving data are used as message queue
Producer by the archives data request message received be stored in message queue module；
Message queue scans thread, and the consumer as message queue module obtains archives data from message queue module and asks
It asks, gets the data-archiving-service example for executing this subtask from data-archiving-service cluster by load balancing service,
Then archives data request is transmitted to the data-archiving-service example and executes archives data task；
Message sends thread, and the archives data for forwarding data-archiving-service to send replys message and gives archives data requesting party,
Report to this archival task performance.
6. assembly type planetary detection data archive system according to claim 1, wherein
Load balancing service module for receiving data in archival service cluster each data-archiving-service example report it is real-time
Load information, including the occupied memory of data-archiving-service example, CPU, execution archival task number, by dynamic calculate,
Timing is to provide load relatively for data transmit-receive service according to being ranked up from small to large with load capacity to archival service example
Small data-archiving-service entrance.
7. assembly type planetary detection data archive system according to claim 1, wherein
The essential information that cluster management service is registered when being started by each service processes in cluster, including process title, into
The working space of journey and the heartbeat message periodically reported monitor each running state of a process, when some process in discovery cluster
The not sent heartbeat within the period of setting, the process will be judged as the presence of exception, restart the abnormal process immediately；
After process restarts recovery, not completing when being abnormal for task will continue to execute.
8. a kind of archiving method of the assembly type planetary detection data based on data-driven, comprising:
Archives data requesting party will need the data achieved and achieve list to copy shared memory to first, then lead to
It crosses archives data interface module and sends archives data request to the data transmit-receive service module of this filing system；
After data transmit-receive service module receives archives data request message, the message is stored in message queue module first and is held
It is at least processed primary to guarantee the archives data request message received, then is successively read number from message queue module for longization
According to archive request message；
Data transmit-receive service module is obtained in current data archival service module by load balancing service module and is loaded relatively
It is lesser to be transmitted to the load by small archival service example for the archives data request message read from message queue module
Archival service example executes this archival task by the archival service example；
After data-archiving-service module receives archives data request message, analytic message generates data after completing data filing
It achieves and replys inventory, record the archive performance of each data, and archives data reply inventory is copied to shared memory,
It generates archives data reply message and is sent to data transmit-receive service module；
After data transmit-receive service module receives archives data reply message, it is sent to archives data requesting party, completes this number
According to archive.
9. archiving method according to claim 8, wherein data-archiving-service module analytic message includes: archives data
Then it is clear to read the archive data from the storage location for achieving list in shared memory is obtained in message for service module
List is simultaneously parsed, to obtain this all data file for needing to achieve, and based on particular row used by such data
Sing data format successively checks data, for checking that underproof data are copied to recycle bin memory block, and puts
The archive for abandoning such data is stored in database for its metadata information of the reading data of passed examination, and data file is copied
Shellfish completes such archives data to persistent storage.
10. archiving method according to claim 1, wherein after message is stored in message queue by data transmit-receive service module,
Archives data for entering queue is requested, and is set as armed state by message queue module first, while being persisted to magnetic
Disk, it is ensured that the archives data request that this system receives can be processed, and the archives data for going out team is requested, will be by message team
Column module is set as just in processing status, and the archives data is requested physically still in message queue at this time, until the request
It is processed to finish, and after receiving removal request, it can just be removed from queue and disk by message queue module；
Preferably, the data transmit-receive service module, load balancing service module and data-archiving-service module are periodically to collection
Group's management services module sends heartbeat message, reports its operating status, and cluster management service module is sent by each service
Heartbeat situation judges each service whether in normal service state, to restore to exception service；
Preferably, each data-archiving-service module timing reports respective loading condition to load balancing service module, wherein
Loading condition includes memory, the archival task number that this service occupies, and load balancing service passes through each data-archiving-service example
Loading condition calculate the lesser data-archiving-service of load in real time, in case receiving new archival task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811186417.6A CN109408484B (en) | 2018-10-11 | 2018-10-11 | Component type planetary detection data archiving system and method based on data driving |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811186417.6A CN109408484B (en) | 2018-10-11 | 2018-10-11 | Component type planetary detection data archiving system and method based on data driving |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109408484A true CN109408484A (en) | 2019-03-01 |
| CN109408484B CN109408484B (en) | 2022-05-06 |
Family
ID=65467658
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811186417.6A Active CN109408484B (en) | 2018-10-11 | 2018-10-11 | Component type planetary detection data archiving system and method based on data driving |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109408484B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111491017A (en) * | 2020-04-03 | 2020-08-04 | 广州市中海达测绘仪器有限公司 | Data distribution system, method and device for CORS reference station |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1959717A (en) * | 2006-10-09 | 2007-05-09 | 北京道达天际软件技术有限公司 | System and method for preprocessing mass remote sensing data collection driven by order form |
| US20110154355A1 (en) * | 2009-12-22 | 2011-06-23 | Siemens Aktiengesellschaft | Method and system for resource allocation for the electronic preprocessing of digital medical image data |
| CN103036956A (en) * | 2012-11-30 | 2013-04-10 | 航天恒星科技有限公司 | Filing system and implement method of distributed configured massive data |
| CN104239143A (en) * | 2014-09-17 | 2014-12-24 | 中国科学院遥感与数字地球研究所 | Satellite data receiving task scheduling system based on OSGI (Open Service Gateway Initiative) and workflow and satellite data receiving task scheduling method based on OSGI and workflow |
| CN105049493A (en) * | 2015-06-29 | 2015-11-11 | 中国科学院遥感与数字地球研究所 | Virtual networking distributed satellite data service system |
-
2018
- 2018-10-11 CN CN201811186417.6A patent/CN109408484B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1959717A (en) * | 2006-10-09 | 2007-05-09 | 北京道达天际软件技术有限公司 | System and method for preprocessing mass remote sensing data collection driven by order form |
| US20110154355A1 (en) * | 2009-12-22 | 2011-06-23 | Siemens Aktiengesellschaft | Method and system for resource allocation for the electronic preprocessing of digital medical image data |
| CN103036956A (en) * | 2012-11-30 | 2013-04-10 | 航天恒星科技有限公司 | Filing system and implement method of distributed configured massive data |
| CN104239143A (en) * | 2014-09-17 | 2014-12-24 | 中国科学院遥感与数字地球研究所 | Satellite data receiving task scheduling system based on OSGI (Open Service Gateway Initiative) and workflow and satellite data receiving task scheduling method based on OSGI and workflow |
| CN105049493A (en) * | 2015-06-29 | 2015-11-11 | 中国科学院遥感与数字地球研究所 | Virtual networking distributed satellite data service system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111491017A (en) * | 2020-04-03 | 2020-08-04 | 广州市中海达测绘仪器有限公司 | Data distribution system, method and device for CORS reference station |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109408484B (en) | 2022-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP2031507B1 (en) | Systems and/or methods for location transparent routing and execution of processes | |
| Paolucci et al. | Delivering semantic web services | |
| US7788319B2 (en) | Business process management for a message-based exchange infrastructure | |
| Juric et al. | SOA approach to integration | |
| Sheng et al. | Configurable composition and adaptive provisioning of web services | |
| US8549535B2 (en) | Distributed taskflow architecture | |
| US20080300928A1 (en) | Publishing and deploying business processes | |
| WO2007123930A2 (en) | Method and architecture for goal oriented applications, configurations and workflow solutions on-the-fly | |
| US9052972B2 (en) | Determining the processing order of a plurality of events | |
| Czarnul | Modeling, run-time optimization and execution of distributed workflow applications in the JEE-based BeesyCluster environment | |
| Youssef et al. | EOS: enterprise operating systems | |
| Davami et al. | Fog-based architecture for scheduling multiple workflows with high availability requirement | |
| EP3069272B1 (en) | Managing job status | |
| Choi et al. | DEXSim: an experimental environment for distributed execution of replicated simulators using a concept of single simulation multiple scenarios | |
| CN109408484A (en) | A kind of assembly type planetary detection data archive system and method based on data-driven | |
| US20110289041A1 (en) | Systems and methods for managing assignment templates | |
| Pryss et al. | Integrating mobile tasks with business processes: a self-healing approach | |
| Caeiro-Rodrıguez et al. | Dynamicity in scientific workflows | |
| McGough et al. | A standards based approach to enabling legacy applications on the Grid | |
| Ruttenberg | Finding the tool that fits best: cloud based task management for electronic resources | |
| Pryss et al. | Extending business processes with mobile task support: A self-healing solution architecture | |
| Breeding | Introduction and Concepts | |
| Borky et al. | Designing in a Logical/Functional Viewpoint | |
| Shamsa et al. | A decentralized prediction-based workflow load balancing architecture for cloud/fog/IoT environments | |
| Bahsi | Dynamic workflow management for large scale scientific applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |