The content of the invention
It is an object of the invention to provide a kind of efficient storage method of mass data, for the applied characteristic of IT operation and maintenance tools
A kind of highly effective gathering dispatching method of mass data storage is provided, improves efficiency and the accuracy of data loading.
The purpose of the present invention is realized using following scheme:The efficient storage method of mass data, comprises the following steps:
1) configuration rule of storage method, is configured;
2), log-on data storage control and initialized;
3), data storage controller detection storage information caching in whether the information of storage in need, according to configuration rule
The information be put in storage to needs carries out the encapsulation of storage task and management and running;
4), storage operational terminal manager is allocated scheduling according to configuration rule to storage task;
5), be put in storage operational terminal manager and take out task from storage task buffer, and give the idle storage terminal
Perform;
6), be put in storage terminal and perform in-stockroom operation;
7) data storage controller scanning storage information unloading file will be notified by, being put in storage operational terminal manager, by unloading
Storage information re-start scheduling;
8), when in controller closing process, in unloading storage information caching, storage task buffer treat storage information and
Storage information is treated by what storage operational terminal was refused.
More specifically scheme of the invention is:The efficient storage method of mass data, includes following key step:
1) configuration rule of storage method, is configured, the configuration to configuration rule includes setting:The optimal storage letter of every batch of
Breath number, wait optimal information number maximum duration, maximum to carry number of tasks, storage mission failure number of retries, storage information turn
Deposit strategy, storage information unloading file, error message unloading file;
2), log-on data storage control, in start-up course, data storage controller initializes storage information and delays first
Deposit, then scan storage information unloading file whether have the information be not put in storage, if so, then by these information be re-loaded into
In the information cache of storehouse, in case scheduling;
3), after data storage controller starts, storage information is treated in reception, in controller detection storage information caching whether
The information of storage in need, and judge whether the quantity of information has reached the optimal storage information number of default every batch of, if reached
Optimal number has been arrived, then has been divided into batch of data according to optimal number, is packaged into a storage task, submit to storage operational terminal
Manager;If being not reaching to optimal number, waited, waited in optimal information number maximum duration according to rule, if slow
Deposit middle number of data and reach the optimal storage information number of every batch of, then distribute a storage task by optimal data bar number, otherwise wait for
After by total data be encapsulated as a storage task, submit to storage operational terminal manager;
4), when storage operational terminal manager receives storage task, according to configuration rule, judge to be put in storage task buffer
Whether the task quantity in queue, which has had reached default maximum, can carry number of tasks, if it is not, the task is put into
Into storage task buffer;If number of tasks can be carried beyond maximum, turned according to the storage information set in configuration rule
Strategy is deposited, directly dumps to the task in storage information unloading file, and/or, task buffer is put in storage in random unloading in proportion
In partial task into storage information unloading file;
5), storage operational terminal manager is allocated storage task, first determines whether have in storage task buffer
Unappropriated task, if so, then judging whether the storage terminal belonging to the storage operational terminal manager is all within operation
In, if available free storage terminal, a task is taken out from storage task buffer, and give free time storage terminal and hold
OK;If without idle storage terminal, wait until that storage terminal is released, reallocation storage task;
6), after idle storage terminal receives storage task, in-stockroom operation is immediately performed, if in implementation procedure
In, abnormal caused by storage information mistake, storage terminal can then filter the exception information, and continue to execute what is be not carried out
Storage information simultaneously dumps to the storage information of mistake in error message unloading file;If as network, data base administration system
It is abnormal caused by the reasons such as system, disk I/O, then be put in storage terminal and certain amount or time are retried according to default configuration rule
In-stockroom operation, if being still unable to normal storage, all operations of rollback executed, and by the storage in the storage task
Information dumps to storage information unloading file;
7), when the storage terminal for being put in storage operational terminal manager administration is in low load condition, operational terminal pipe is put in storage
Data storage controller scanning storage information unloading file will be notified by managing device, check whether the storage information of unloading, if so,
Then these information are re-loaded in storage information caching, re-start scheduling;
8), when being put in storage in operational terminal manager closing process, storage operational terminal manager stops whole to storage work
Hold manager submit task, storage information caching in storage information, due to concurrent reason, refused by storage operational terminal manager
Exhausted task is dumped in storage information unloading file;Storage operational terminal manager stops receiving new task, stops to entering
Storehouse terminal distribution task, all unallocated tasks in task buffer are dumped in storage information unloading file;Each storage
Terminal stops receiving an assignment, but continues to execute unfinished task, pending to finish, and exits working status.
Management and running are added to mass data storage process compared with the prior art in this way, can significantly improve and carry
The scalability of high system, is changed with adapting to different monitored system and its scale;It can realize flexibly matching somebody with somebody for putaway rule
Put, to adapt to the specific business characteristic of different monitored systems, easily according to the actual conditions adjustment storage of monitored system
Management;And the accuracy and integrality, the reliability for improving data loading work of data loading are improved, pass through unloading machine
System avoids delayed impact of the excess load to system;Also improve the service efficiency of system, can preferably with monitoring system, stream
The concurrent efforts such as journey system, diagnostic system.
Can be using a kind of efficient collection scheduling device of the efficient storage method of mass data of the present invention:It is a kind of efficient
Collection scheduling device, include data storage controller, storage information caching, storage operational terminal manager, storage task
Caching, storage terminal, storage information unloading file, configuration rule, error message unloading file.
Data storage controller:It is responsible for receiving, distributes and treat storage information, in start-up course, scans and load storage letter
Cease the storage information of unloading in unloading file;After data storage controller starts, the data in being cached according to storage information
Bar number, the storage information during storage information is cached according to configuration rule converts in an optimal manner is packaged into storage task, and carries
Give storage operational terminal manager;In system shutdown procedures, data storage controller storage information is cached in, can also
Dumped to comprising the storage task for being rejected submission in storage information unloading file.
Be put in storage operational terminal manager:After system start-up, storage operational terminal manager receives data storage control
The storage task that device is submitted, is then checked for the affiliated whether available free terminal of operational terminal, if it is not, according to configuration rule
Whether the task quantity checked in task buffer has reached maximum carrying quantity, when having reached task maximum carrying quantity
Wait, task is put in storage further according to the unloading strategy unloading in configuration rule;When the affiliated storage operational terminal of the controller is in
During low-load state, which should notify data storage controller, and whether scan in storage information unloading file has unloading
The storage information be not put in storage;In the system shutdown procedures, which stops receiving new task and distribution
Task preserves in task buffer unappropriated storage task to storage information unloading file to storage operational terminal.
Be put in storage terminal:The task that terminal receives manager distribution is put in storage, performs in-stockroom operation, in the process of implementation, if
It was found that the storage information of mistake, then filter the wrong storage information, and error message unloading file is dumped to, then proceeded to
The storage information for being not carried out finishing is performed, it is abnormal in the event of storing, such as IO, database service exception etc., then retract all
Database manipulation, and the storage task that the operational terminal is carrying out is dumped in storage information unloading file.
Configuration rule:The configuration information of storage configuration rule, including the optimal storage information number of every batch of, wait optimal information number
Maximum duration, maximum can carry number of tasks, storage mission failure number of retries, storage information unloading strategy, storage information unloading
File, error message unloading file, criterion is provided for system operation, and corresponding reaction is made under specific circumstances for all parts
Foundation is provided.
The collection scheduling device of this scheme is used, it can be achieved that adding scheduling pipe for mass data stock management process
Reason, scalability, the customization of system are high, are suitable for different monitored system and its scale changes;Putaway rule configuration spirit
It is living, the specific business characteristic of different monitored systems is suitable for, easily according to the actual conditions adjustment storage of monitored system
Management;And accuracy and integrality, the reliability of data loading work of data loading are high, employ unloading mechanism to avoid
Delayed impact of the excess load to system;It is strong with the ability of the concurrent efforts such as monitoring system, flow system, diagnostic system.
Embodiment
All features disclosed in this specification, or disclosed all methods or during the step of, except mutually exclusive
Feature and/or step beyond, can combine in any way.
As shown in Figure 1, in IT O&Ms field, to preserve the magnanimity performance indicator number collected by different acquisition terminal
According to.Configuration rule first, the optimal storage information quantity of definition every batch of is 1000, wait optimal information number maximum duration is 3
Second, maximum can carry number of tasks 500, when the task amount in the storage task buffer that storage operational terminal manager is managed surpasses
When going out maximum number of tasks, the task 30% in random unloading task buffer is into storage information unloading file, storage mission failure
Number of retries 3 times, storage information unloading file are message.dump, error message unloading file error_
message.dump。
In system starting process, data storage controller scans message.dump files first, and whether check wherein has
By the storage information of unloading, if then these information are loaded into the storage information caching of controller.
After system starts, data storage controller, which will be checked in caching, whether there is storage information, if without if
Continue waiting for, if so, then according in configuration rule, judging whether the cache information number has reached 1000, if reached,
1000 storage informations are then extracted, and this 1000 storage informations are encapsulated as a task, submit to storage operational terminal
Manager, and if not up to 1000 datas, wait 3 seconds, in this 3 seconds waiting process, if before 3 seconds cache in
Number of data has reached 1000, then controller is no longer waiting for, and immediately extracts this 1000 data, is encapsulated as one and enters
Storehouse task, submits to storage operational terminal manager, on the contrary, after 3 second stand-by period, is still not reaching to 1000 numbers
According to requiring, then controller all extracts storage information all in caching, is encapsulated as a task and submits storage work
TERMINATION MANAGER.
Storage TERMINATION MANAGER receive task after, first determine whether belonging to operational terminal whether oepration at full load, if
No, then directly the task is put into task buffer, wait it is to be allocated, if operational terminal is all in working status, basis
Configuration rule, judges whether the data in task buffer have reached 500, if having reached 500, at random appoints 500
30% in business dumps in storage information unloading file, and then newly submitting for task is put into task buffer, waits and dividing
Match somebody with somebody.
When the operational terminal belonging to storage operational terminal manager is available free, then operational terminal manager is put in storage, will be appointed
A task submitting earliest is distributed to the idle operational terminal and is performed in business caching, if without idle operational terminal,
Then wait resource release.
When operational terminal receives storage task, storage task is then carried out, in task process is put in storage in execution, hair
Existing storage information mistake, for example, data type not to, the sql error that performs when exception, operational terminal filters out these mistakes
Information, and these information are dumped in file error_message.dump, and continue to execute entering of being not carried out finishing
Storehouse information, preserves the other information be not put in storage.
In the process of implementation, if run into, IO is abnormal, startup, Network Abnormal etc. be not abnormal for database service for operational terminal
When, operational terminal will retry 3 subtasks according to configuration rule, if still there is identical exception, retract all operations,
And all storage informations in the task are dumped in message.dump files.
When the storage operational terminal belonging to storage operational terminal manager is in underrun, it is whole that work is put in storage at this time
Hold manager notification controller, scan in message.dump files whether a part of storage information of unloading, if so, then weigh
Newly it is loaded into the caching of controller, Reseals task, and re-executes these storage tasks.
When in this method system shutdown procedures, controller stops receiving new storage information, and stops submission task and arrive
Operational terminal manager is put in storage, preserves all storage informations in storage information cache, and by storage operational terminal manager
Into file message.dump, storage operational terminal manager stops receiving new task the storage task of refusal, stops dividing
With the task in caching to storage operational terminal, and all tasks in task buffer are dumped into message.dump files
In, storage operational terminal continues to execute being not carried out finishing of the task, according to original logic unloading file when occurring abnormal.