CN1801923A - hierarchical storage management system - Google Patents
hierarchical storage management system Download PDFInfo
- Publication number
- CN1801923A CN1801923A CN 200510000139 CN200510000139A CN1801923A CN 1801923 A CN1801923 A CN 1801923A CN 200510000139 CN200510000139 CN 200510000139 CN 200510000139 A CN200510000139 A CN 200510000139A CN 1801923 A CN1801923 A CN 1801923A
- Authority
- CN
- China
- Prior art keywords
- storage
- module
- server
- data
- tape
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003860 storage Methods 0.000 title claims abstract description 182
- 238000013508 migration Methods 0.000 claims abstract description 46
- 230000005012 migration Effects 0.000 claims abstract description 46
- 238000000034 method Methods 0.000 claims abstract description 7
- 238000005516 engineering process Methods 0.000 claims abstract description 6
- 230000015654 memory Effects 0.000 claims description 51
- 238000012545 processing Methods 0.000 claims description 17
- 238000012544 monitoring process Methods 0.000 claims description 15
- 238000004891 communication Methods 0.000 claims description 11
- 238000009826 distribution Methods 0.000 claims description 8
- 239000000463 material Substances 0.000 claims description 6
- 230000011218 segmentation Effects 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 4
- 238000003491 array Methods 0.000 claims description 3
- OKLGKGPAZUNROU-YUMQZZPRSA-N 2-amino-2-deoxyisochorismic acid Chemical compound N[C@@H]1[C@@H](OC(=C)C(O)=O)C=CC=C1C(O)=O OKLGKGPAZUNROU-YUMQZZPRSA-N 0.000 claims description 2
- 230000014759 maintenance of location Effects 0.000 claims description 2
- 230000007246 mechanism Effects 0.000 claims description 2
- 230000009977 dual effect Effects 0.000 abstract description 2
- 238000007726 management method Methods 0.000 description 25
- 238000010586 diagram Methods 0.000 description 9
- 230000008901 benefit Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 4
- 238000012217 deletion Methods 0.000 description 4
- 230000037430 deletion Effects 0.000 description 4
- 230000003139 buffering effect Effects 0.000 description 3
- 238000002360 preparation method Methods 0.000 description 3
- 238000013468 resource allocation Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 101000969688 Homo sapiens Macrophage-expressed gene 1 protein Proteins 0.000 description 1
- 102100021285 Macrophage-expressed gene 1 protein Human genes 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a hierarchical storage management system for digital media data of a television station. The system consists of a storage scheduling server, a data migration server cluster and management controller, a near-line storage library controller, an FC disk array, NAS storage equipment, near-line storage, offline storage and a backup tape library, wherein the storage scheduling server is provided with hierarchical storage management software; the distributed system structure of server + multiple migrators is adopted, and the migrators are constructed in the SAN environment and can be arbitrarily expanded according to the requirements; a dual network structure of FC + LAN is adopted; the method adopts an object-oriented hierarchical storage management technology and realizes three storage data modes of online, near-line and offline and storage data exchange through hierarchical storage management software. The invention has excellent disaster tolerance performance, and provides basic guarantee for the storage, management, query and reuse of full digital video and audio media, the storage migration of system unstructured media data objects, the effective storage of audio and video data and the scientific management.
Description
Technical field
The invention belongs to broadcasters asset management field, particularly a kind of TV station digital media data graded memory management system.
Technical background
Along with the explosive increase of information age data with at the continuous appearance of TV station's digital media data management specific demand, new challenge has been proposed for the management of storage system.Especially for the management of video/audio materials a large amount of in the field of video applications, need powerful storage management system provide file/move back, back up/function such as reduce, and need carry out good combination as aspects such as nonlinear editing system, news production networks with storage management system with other Video Applications systems.
The data volume that field of broadcast televisions relates to is big, technical sophistication, a lot of projects just solve the storage problem of broadcasters' data in the mode of simple numericalization+storage, because key technology and means are immature, therefore also there is the corresponding techniques problem, for example: the original basic traditional tape storage way to manage that adopts in Chinese Shanghai audio and video files shop, the mode of traditional libraries is also adopted in the inquiry of data and application, after through digital improvement, high code stream adopts the DV form, low code stream adopts the MPEG1 form, has realized the digitlization storage of data substantially, but owing to technology is at that time limit, the whole framework of its system is based upon on the basis of Fast Ethernet network, storage system also adopts traditional classification storing software Petasite, can't realize the data migration based on LAN Free, makes that the whole data throughout of system is lower.
Summary of the invention
The object of the present invention is to provide a kind ofly, be mainly used in and solve a large amount of storage administrations of looking the audio frequency media data at broadcasters' data staging storage management system.
The invention is characterized in that graded memory management system is made up of the memory scheduling server that hierarchical storage management software is housed, data migration server cluster and Management Controller, nearline storage storehouse controller and FC dish battle array, NAS memory device, nearline storage, offline storage and backup tape storehouse; Adopt the distributed architecture of server+many transference apparatus, transference apparatus is structured under the SAN environment, can expand arbitrarily as required; Adopt two web frames of FC+LAN; Adopt OO hierarchical storage management technology, realize online, near line, off-line three kinds of storages data mode and storage exchanges data by hierarchical storage management software.
High speed memory devices such as the general special finger hard disk of online storage are mainly used in the various data objects that storage is often used, simultaneously also as the memory transactions buffering area of whole storage system.
Nearline storage then refers generally to tape library, the full-automatic memory device of vast capacities such as CD server, and its characteristics are that cost is low, and capacity is big, and the more online storage of speed is slow.Be mainly used in that all need the data object of permanent storage and filing in the storage system.Near line is connected by LAN mode or FC mode with online storage, and finish each other data migration by native system, nearline storage generally adopts fully-automatic equipment to finish the loading of storage medium and read, though speed is more online slow, does not also need manual operation.
Offline storage then refers generally to the storage medium of near line storage system is preserved by the mode off-line of labor management, for example be kept at framed bent, drawer etc., way to manage adopts manual management, when storage system need be called certain data, the notifications keeper is loaded into certain medium in the near line system, realize the readjustment of data, the characteristics of offline storage maximum are that memory capacity can not be subjected to device-restrictive fully, limited by the physics place, cost is very low, but needs manual operation can finish the readjustment of data.
The chief component of hierarchical storage management software comprises: memory scheduling server module (Manager), storage data mover module (DataMover), thesaurus Control Server module (LibController), storage monitoring module (Monitor), storage policy server (Policy Server) and system's external interface module (API).
Described memory scheduling server module is divided into: store tasks scheduler module, two parts of data migration strategy module.
1. store tasks scheduler module: be encapsulated as a service routine, realize scheduling and distribution, and realize unified management to all tasks of system and resource to finishing the employed resource of store tasks.Resource described herein mainly refers to finish the hardware device that store tasks relates to, and comprising: transference apparatus, streaming tape, DVD CD, magnetic tape station, DVD driver, manipulator or the like.
2. data migration strategy module: be encapsulated as a dynamic link library (DLL), the strategy of formulating according to the user generates corresponding migration task automatically, and by system's external interface task is submitted to task scheduling modules.Policy module is mainly used in provides automatic storage administration pattern, and has certain flexible configuration function, and the user also can develop corresponding strategy voluntarily according to external interface.
Described storage data mover module is divided into: data migration task control module, driver control module and tape formating module for reading and writing.
1. data are moved task control module: be encapsulated as a service routine, carry out tcp/ip communication with the store tasks scheduler module, to store tasks scheduler module report oneself state, carry out the data migration order that the store tasks scheduler module is sent, and return execution result.Task is obtained in this module groundwork exactly, executes the task and return results.
2. driver control module: be encapsulated as a dynamic base, realize control to equipment such as several data stream magnetic tape station, DVD drivers.
3. tape formating module for reading and writing: be encapsulated as a dynamic base, realize read-write operation to the data flow magnetic tape format.
Described thesaurus Control Server module: be mainly used in of the control of realization system to various automation mass memories storehouse.Mainly comprise: nearline storage storehouse control dispatch service is touched piece, nearline storage Equipment Control driver module.
1. the dispatch service module is controlled in the nearline storage storehouse: by carrying out tcp/ip communication with the store tasks scheduler module, the report oneself state is carried out the nearline storage Equipment Control order that the store tasks scheduler module is sent, and is returned execution result.
2 nearline storage Equipment Control driver modules: be encapsulated as a dynamic base, realize control, comprise various automaticdata stream tape libraries, DVD CD server etc. to multiple nearline storage equipment.
Described storage monitoring module provides the system monitoring interface, contains: store tasks scheduler module information, nearline storage facility information, transference apparatus information, storage medium information module.
1. store tasks scheduler module information module shows store tasks scheduler module relevant information.Comprise Task Distribution, resource distribution, the running status of each module etc.
2 nearline storage facility information modules are used for the relevant information of the equipment of nearline storage in the display system, comprise the data flow tape pool, nearline storage equipment such as DVD CD server.
3. the transference apparatus information module shows the transference apparatus relevant information, comprises the task of current execution, progress and transference apparatus state etc.
4. storage medium information module, the relevant information of nearline storage medium in the display system.
Described storage policy server module is used to realize various storage policies, and this module is mainly used in user's actual storage need carry out interface, and finishes the generation of store tasks by API.
Described system external interface module: the API that storage system externally is provided, realize the external interface of storage system, storage system realizes the storage policy customization by a series of API are provided, and the memory scheduling server carries out the TCP communication by control API and this server.
Hierarchical storage management software of the present invention, its feature also comprises:
1. for guaranteeing the opening of system, employing two-stage API is connected with other system, and one-level is memory scheduling API, adopts standard C ++ interface card.The user can control and dispatch storage system by calling this plug-in unit; Another level is that system provides streaming tape read-write API, and the user can develop the maintenance of data flow magnetic tape file and the exploitation of other associative operations voluntarily by this API.By this two-stage API, system can carry out interface with third party software very easily, or develops some application software voluntarily by the user.
2. the storage administration for the video and audio material large data objects requires the storage organization of system to adopt the SAN structure fully, and data are carried out the storage exchange by the FC passage.
3. the migration of data is finished by data mover independently, and according to bandwidth, every 3-4 streamer tape device can dispose a data mover, has formed the migration server cluster.All memory devices use by sharing by all transference apparatus.And can expand at any time as required.
4. storage system and storage hardware are irrelevant, and system can adjust each storage hardware equipment at any time, as magnetic tape station, and tape pool, memory devices such as CD server use the plug-in unit mode on the software, or other modes are guaranteed the independence of storage system and equipment.
5. data file is filed storage format, adopt the MTF reference format, MTF is the streaming tape data memory format that Microsoft openly issues, though the form complexity causes efficient to reduce, but it is open that its advantage is form, and the user can support the software of MTF to read and write data by any one.
6. adopt the independent database system, database storage software uses SQL Server, ORACLE, and three kinds of normal data library softwares of DB2, and database server both can merge with the system management software, also can separate.The advantage of this mode is that the user can be transplanted to the database of storage system on the Content Management Platform, and like this, the user just can safeguard that a sets of data storehouse platform gets final product, and has reduced maintenance cost.
7. tape group and store tasks can parallel processings, processing for the big object of big capacity, the pattern that system adopts a plurality of tasks in parallel to handle, promptly when certain tape group need be stored several tasks, system can according to circumstances distribute different drivers to file simultaneously on several tapes on the same group.
8. support multiple driver and multiple tape library, the streaming tape driver comprises: LTO, DTF2, AIT, AHDT, VXA, STK; The streaming tape storehouse comprises: SONY, IBM, ADIC, STK.
9. can carry out the segmentation readjustment of Large Volume Data: by in archiving process, creating frame index and second index, system can carry out the segmentation readjustment to Large Volume Data, promptly, automatically big file is cut, and generate a new data file according to going into a little and length of asking.
10. magnetic tape station adopts dynamically shared and static binding dual mode in the tape library: for the use of driver in the storage system, and two kinds of system's employing sharing mode (the SAN environment is shared, and magnetic tape station is connected on the FC by bridger) and exclusive modes.The advantage of sharing mode is resource-sharing, and utilization rate of equipment and installations is higher, and can realize Long-distance Control; The advantage of exclusive mode is reliable and stable, and the performance performance of magnetic tape station better.
11. adopt the arrangement to data flow tape storage space, the streaming tape data are in a single day deleted, if without data preparation, the occupied tape storage space of then deleted data can't utilize again.This storage system adopts manually the arrangement and the system that carry out the tape storage space by storage administration user monitoring interface to put two kinds of mechanism in order according to the tape storage dead band automatically, guarantees effective utilization of streaming tape memory space.
The invention has the beneficial effects as follows:
1. You Yi disaster tolerance performance: because this system adopts distributed frame and task transmission to support transaction, therefore for storage system, the damage of arbitrary node all can not cause catastrophic results in the course of work.As: in the data filing process, damage if carry out the transference apparatus of archive operation, then scheduler can monitor automatically, and distributes other resource to re-execute the work of interrupting just now; If this moment, scheduler damaged, then transference apparatus can continue migration filing action and finishes up to task.Scheduler no longer continues Processing tasks during damaging, in case scheduler recovers, then the migration scheduling also recovers thereupon, and the work of accumulation can not lost.If medium is damaged, then scheduler can be changed medium automatically, provides prompting simultaneously and bad medium is stamped the damage mark.
2. owing to adopt OO hierarchical storage management technology, realized online, near line, three kinds of storage modes of off-line and storage exchanges data, realize data migration based on LAN Free, improved the whole data throughout of system.Look the storage, management, inquiry of audio frequency media and utilize again for total digitalization, be the storage migration of system's unstructured data object (media data object) and effective preservation of audio and video files, the scientific management assurance that provides the foundation.
Below in conjunction with drawings and Examples the present invention is done a detailed explanation.
Description of drawings
Fig. 1 is the system configuration schematic diagram;
Fig. 2 concerns schematic diagram for system software module;
Fig. 3 is an entire system operation flow schematic diagram;
Fig. 4-1 is a storage server operation flow main thread schematic diagram;
Fig. 4-2 is a storage server operation flow Message Processing thread schematic diagram;
Fig. 4-3 is a storage server operation flow task test thread schematic diagram;
Fig. 4-4 is a storage server operation flow task processing threads schematic diagram;
Fig. 5 is a migration server operation flow schematic diagram;
Fig. 6 is a strategic server operation flow schematic diagram.
Embodiment
The graded memory management system structure is as shown in Figure 1: online memory bank comprises two FC disk arrays and a NAS equipment, one of them FC disk array is used for preserving the storage code check and looks audio material, the shared dish when another is used for stored data base and full-text search data and is used as cluster.The NAS memory device is mainly used in the memory scan code check and looks audio material and deposit key frame.1.1 online FC dish battle arrays link to each other with the FC bus among the figure, 1.2 online FC dish battle array links to each other with the FC bus, 1.3 online NAS storage links to each other with the LAN bus with the FC bus respectively, 1.4 the memory scheduling server links to each other with the LAN bus with the FC bus respectively, 1.5 data migration server cluster and Management Controller link to each other with the LAN bus with the FC bus respectively, 1.6 nearline storage links to each other with the LAN bus with the FC bus respectively, 1.7 offline storage links to each other with the LAN bus with the FC bus respectively, 1.8 backup links to each other with the FC bus with tape pool, 1.9 nearline storage storehouse controllers link to each other with the LAN bus with the FC bus respectively.
Graded memory management system software module relation is 2.1 storage monitoring modules (Monitor) as shown in Figure 2,2.2 storage policy server module (Policy Server) and system's external interface module (API), 2.3 be memory scheduling server module (Manager), 2.4 storage data mover module (DataMover), 2.5 thesaurus Control Server modules (LibController).
2.1 storage monitoring module (Monitor): the system monitoring interface is provided, main realize the configuration of system resource and to system resource state, Task Progress monitoring, comprise memory scheduling server info, nearline storage facility information, transference apparatus information, storage medium information module, realize the state of storage administration, information, control safeguards that configuration waits the man-machine interface of all associative operations.Can be configured management, monitoring, control, statistics or the like user's operational administrative to system.This software module adopts the Window standard interface, and organize content comprises:
Resource distribution: configuration emigration device, tape pool, driver, tape and tape grouping;
System parameters configuration: configuration store server ip, port, concurrent number of tasks, disk buffering, tape buffering etc.;
Memory area configuration: configuration store district and strategy thereof, strategy comprise filing strategy, backup policy etc.;
Error reporting: show by this Monitor from operational report, error reporting that each module produced;
Monitoring: the operation conditions of all resources of surveillance, comprise the running status of tape pool, driver, transference apparatus etc., and the operator can come each resource control to system by carrying out the associative operation order, as adding filing task, downloaded object etc.
2.2 storage policy server module (Policy Server) and system's external interface module (API): the API that storage system externally is provided, realize the external interface of storage system, storage system realizes the storage policy customization by a series of API are provided, and the memory scheduling server carries out the TCP communication by control API and this server.
2.3 memory scheduling server module (Manager): contain store tasks scheduler module (migration server and tape pool control program), data migration strategy module, it is the maincenter of whole software, internally be responsible for the allocation schedule of resources such as magnetic tape station, tape pool manipulator, transference apparatus in the storage system, externally be responsible for receiving and responding various types of requests and task, it is made of being:
Resource management and scheduling:, carry out unified allotment and management by task scheduling and resource management server by storage administration user monitoring interface configurations resource.In case memory requirement (external control or internal trigger) takes place, and server carries out resource allocation according to the requirement of all state of resources of system and store instruction.Resource comprises: driver, tape pool object, transference apparatus etc.;
Exterior storage instruction: the memory scheduling server carries out interface by mode and the applications of API, and application software is by calling API and the memory scheduling server carries out the TCP communication, and sends corresponding store instruction.For example: filing data, migration data, near line restore data, off-line restore data etc.Storage managing server is accepted exterior storage control back at every turn and is directly generated a store tasks, and carries out the processed of task by the task processing threads;
Notice thesaurus Control Server carries out automatic equipment control: according to the requirement of resource allocation and store instruction, and for some automation equipments, as tape pool, CD server, manipulator, automatic loading device etc. send the control requirement.Because this kind equipment may carry software, so the memory scheduling server need carry out the control of this kind equipment by the thesaurus Control Server.If storage object belongs to an off-line storehouse, then send application to the keeper and add the dress streaming tape by the thesaurus Control Server;
The notification data migration server is carried out the data migration: resource allocation is finished, and after Equipment Control was finished, the memory scheduling server carried out the migration operation of data by TCP communication notice storage data migration server.The storage data migration server returns corresponding information after carrying out the data migration as requested, and the memory scheduling server is accepted return results and carried out corresponding record.
2.4 storage data mover module (DataMover): contain the migration control module, driver control module and tape formating module for reading and writing, be responsible for specifically moving of data, it receives the task that the memory scheduling server sends, and specifically carry out this task, control by the memory scheduling server, being responsible for depositing scheduling reason server data designated moves in the memory scheduling server appointed equipment, and return corresponding results, the storage data migration server carries out communication by TCP and memory scheduling server, and the instruction of receiving, storing and managing server, carry out data-moving and copy by SAN simultaneously.
2.5 thesaurus Control Server module (LibController): contain nearline storage storehouse control dispatch service and touch piece, nearline storage Equipment Control driver module, be used for the control of automation equipment, as the data flow tape library, manipulator, CD server etc., control mode can be RS422, LAN, FC etc.The memory scheduling server carries out communication by TCP and thesaurus Control Server, and sends control command to the thesaurus Control Server.The thesaurus Control Server is carried out and the translation control signal, carries out concrete control action.After being finished, returning thesaurus Control Server control result or return corresponding control information.
Fig. 3 is the entire system operation flow, 3.1 enter user's configuration-system, 3.2 start-up system configurator GUI, 3.3 carry out the resource distribution of the parameter configuration and 3.4 systems of system, 3.5 startup storage server, 3.6 judge whether startup is successful, the unsuccessful 3.2 start-up system configurator GUI that turn back to, start migration server if successfully carry out 3.7 respectively, 3.10 judge whether startup is successful, the unsuccessful 3.2 start-up system configurator GUI that turn back to, if successfully enter into 3.13 system start-ups, 3.8 start the tape pool control program, 3.11 judge whether startup is successful, the unsuccessful 3.2 start-up system configurator GUI that turn back to, if successfully enter into 3.13 system start-ups, 3.9 startup strategic server, 3.12 judge whether startup is successful, the unsuccessful 3.2 start-up system configurator GUI that turn back to, if successfully enter into 3.13 system start-ups, 3.14 judge whether new task? be that new task is carried out the processing of 3.15 tasks, result is delivered to 3.16 user interfaces submits to new task to stop to 3.13 system start-ups or to 3.17 systems, 3.14 judge whether new task, be not new task, 3.16 user interfaces submit to new task to stop user interface submission new task to 3.13 system start-ups to 3.13 system start-ups or system.
Fig. 4-1 forms the storage server business process map to four threads of 4-4,
4-1.1 main thread normally starts, 4-1.2 the initialization resource status, 4-1.3 initialization task tabulation, is provided with message processing function, beginning listening port at the 4-1.4 middleware, 4-1.5 the beginning task detects thread, 4-1.6 stops, the system of sending withdraws from signal, treat that all threads finish the back, withdraw from.
4-2.1 enter message-oriented middleware message call processing threads, the 4-2.2 informed source is from the message of API, 4-2.3 the increase task, 4-2.4 revises task, and 4-2.5 searches task, 4-2.6 the cancellation task, from the message of tape pool controller, 4-2.7 notifies preparation, 4-2.8 update notifications Media state, from the message of data mover, 4-2.9 notifies preparation, 4-2.10 notice task, result, from the message of system, 4-2.11 turn-off data transference apparatus, 4-2.12 disconnects the tape pool controller.
Start 4-3.1 task detects thread, the 4-3.2 judgement has or not system to withdraw from signal, has, 4-3.7 withdraw from, do not withdraw from signal, 4-3.3 has judged whether the task of being untreated, have, 4-3.4 judge whether to possess the Processing tasks condition, 4-3.5 locking task, task processing threads of unlatching are arranged, going back to the 4-3.2 judgement then has or not system to withdraw from signal, 4-3.3 and the judgement of 4-3.4 all be otherwise, 4-3.6 waits for three seconds, forwards 4-3.2 then to and judges and have or not system to withdraw from signal.
4-4.1 the task processing threads starts, 4-4.2 judge and have or not idle driver, have, 4-4.3 judge to have or not idle transference apparatus that have, 4-4.4 obtains corresponding tape, get and be with successfully, 4-4.5 choose suitable driver, choose success, 4-4.6 chooses idle transference apparatus, choose success, 4-4.7 informing removal device Processing tasks, 4-4.8 ends task, if 4-4.2 and 4-4.3 are judged as not, 4-4.4,4-4.5,4-4.6 be for unsuccessful, then all enter 4-4.9 and restore the state of Resources allocation, 4-4.10 restores task status, enters 4-4.8 and ends task.
Fig. 5 is the migration server operation flow, and this flow process is made up of to two threads of 5-2 Fig. 5-1,
5-1.1 the main thread transference apparatus starts, 5-1.2 initialization data storehouse, success, 5-1.3 initialization ASAPI, success, 5-1.4 search magnetic tape station equipment, success, 5-1.5 ready for sending message is given server, success, the 5-1.6 transference apparatus starts successfully, begins to receive message, and the 5-1.7 program normally withdraws from, if 5-1.2,5-1.3,5-1.4,5-1.5, unsuccessful, enter that the 5-1.8 program is improper to be withdrawed from, 5-1.9 transference apparatus user shutdown command, the 5-1.7 program normally withdraws from.
5-2.1 Message Processing enters type of message, enter the scanning of magnetic tape station scanning 5-2.2 magnetic tape station, 5-2.3 ID returns to server with magnetic tape station, the task that enters is handled the 5-2.4 Processing tasks, 5-2.5 search magnetic tape station equipment, do not find, 5-2.13 return Device Errors, discovering device, 5-2.6 task type, upload, 5-2.7 handle the task of uploading, 5-2.9 the judgement task is finished dealing with not? finish, 5-2.10 the transmission task runs succeeded to server, does not finish, 5-2.11 profiling error reason also sends to server, 5-2.6 task type, to download, handle downloading task, is 5-2.9 judgement task finished dealing with not? finish, 5-2.10 the transmission task runs succeeded to server, do not finish, 5-2.11 profiling error reason also sends to server, the 5-2.6 task type, the segmentation readjustment, 5-2.8 handle segmentation readjustment task, is 5-2.9 judgement task finished dealing with not? finish, 5-2.10 transmission task runs succeeded to server, do not finish, 5-2.11 the profiling error reason also sends to server, the 5-2.1 type of message, and tape ejects, 5-2.14 ejection tape, 5-2.15 return results, the 5-2.1 type of message, task enters inquiry, 5-2.16 return Task Progress, 5-2.17 returns to server with Task Progress.
Fig. 6 is the strategic server operation flow, 6.1 strategic server starts, 6.2 connection database, successful connection, 6.3 Connection Service device, successful connection, 6.5 start successfully, 6.6 the task status that inquiry is being handled, 6.7 refresh task status according to Query Result, enter three tasks then, 6.8 start the deletion mission thread, 6.9 judge whether new task? be that 6.10 deletion corresponding documents return results to 6.8 and start the deletion mission thread, 6.9 judge whether new task? be not, turn back to 6.8 and start the deletion mission thread, 6.11 start the filing mission thread, do 6.12 judge whether new task? be, 6.13 the submission task is given storage server, the task result is turned back to 6.7 refresh task status according to Query Result, do 6.12 judge whether new task? not to turn back to 6.11 and start the filing mission thread, 6.14 start the migration mission thread, 6.15 judge whether new task? be, 6.13 submission tasks are given storage server, the task result is turned back to 6.7 refresh task status according to Query Result, 6.15 judge whether new task? be not, turn back to 6.14 and start the migration mission thread, 6.2 and 6.3 connection failures, 6.4 withdraw from.
Claims (9)
1. graded memory management system is characterized in that: this management system is made up of the memory scheduling server that hierarchical storage management software is housed, data migration server cluster and Management Controller, nearline storage storehouse controller and FC dish battle array, NAS memory device, nearline storage, offline storage and backup tape storehouse; Adopt the distributed architecture of server+many transference apparatus, transference apparatus is structured under the SAN environment, can expand arbitrarily as required; Adopt two web frames of FC+LAN; Adopt OO hierarchical storage management technology, realize online, near line, off-line three kinds of storages data mode and storage exchanges data by hierarchical storage management software;
Online memory bank comprises two FC disk arrays and a NAS equipment, one of them FC disk array is used for preserving the storage code check and looks audio material, shared dish when another is used for stored data base and full-text search data and is used as cluster, the NAS memory device is mainly used in the memory scan code check and looks audio material and deposit key frame; Online FC dish battle array (1.1), (1.2) link to each other with the FC bus, online NAS storage (1.3) links to each other with the LAN bus with the FC bus respectively, memory scheduling server (1.4) links to each other with the LAN bus with the FC bus respectively, data migration server cluster and Management Controller (1.5) link to each other with the LAN bus with the FC bus respectively, nearline storage (1.6) links to each other with the LAN bus with the FC bus respectively, offline storage (1.7) links to each other with the LAN bus with the FC bus respectively, backup links to each other with the FC bus with tape pool (1.8), and nearline storage storehouse controller (1.9) links to each other with the LAN bus with the FC bus respectively.
2. graded memory management system according to claim 1, it is characterized in that described hierarchical storage management software is made up of memory scheduling server module (Manager), storage data mover module (DataMover), thesaurus Control Server module (LibController), storage monitoring module (Monitor), storage policy server module (Policy Server) module and system external interface module (API).
3. graded memory management system according to claim 2, it is characterized in that, described memory scheduling server module is divided into store tasks scheduler module, data migration strategy module, (1) the store tasks scheduler module is encapsulated as a service routine, scheduling and the distribution of realization to finishing the employed resource of store tasks, and realization is to the unified management of all tasks of system and resource, resource described herein mainly refers to finish the hardware device that store tasks relates to, comprise: transference apparatus, streaming tape, DVD CD, magnetic tape station, the DVD driver, manipulator or the like; (2) data migration strategy module package is a dynamic link library (DLL), and the strategy of formulating according to the user generates corresponding migration task automatically, and by system's external interface task is submitted to task scheduling modules.Policy module is mainly used in provides automatic storage administration pattern, and has certain flexible configuration function, and the user also can develop corresponding strategy voluntarily according to external interface.
4. graded memory management system according to claim 2, it is characterized in that, described storage data mover module is divided into: data migration task control module, driver control module and tape formating module for reading and writing, (1) data migration task control module is encapsulated as a service routine, carry out tcp/ip communication with the store tasks scheduler module, to store tasks scheduler module report oneself state, carry out the data migration order that the store tasks scheduler module is sent, and return execution result; (2) the driver control module package is a dynamic base, realizes the control to equipment such as several data stream magnetic tape station, DVD drivers; (3) the tape formating module for reading and writing is encapsulated as a dynamic base, realizes the read-write operation to multiple magnetic tape format.
5. graded memory management system according to claim 2, it is characterized in that, described thesaurus Control Server module comprises nearline storage storehouse control dispatch service and touches piece, nearline storage Equipment Control driver module, (1) nearline storage storehouse control dispatch service module: by carrying out tcp/ip communication with the store tasks scheduler module, the report oneself state, carry out the nearline storage Equipment Control order that the store tasks scheduler module is sent, and return execution result; (2) nearline storage Equipment Control driver module is encapsulated as a dynamic base, realizes the control to multiple nearline storage equipment, comprises various automaticdata stream tape libraries, DVD CD server etc.
6. graded memory management system according to claim 2, it is characterized in that, described storage monitoring module provides the system monitoring interface to contain: store tasks scheduler module information, nearline storage facility information, transference apparatus information, storage medium information module, (1) store tasks scheduler module information module, show store tasks scheduler module relevant information, comprise Task Distribution, resource distribution, the running status of each module etc.; (2) nearline storage facility information module is used for the relevant information of the equipment of nearline storage in the display system, comprise the data flow tape pool, nearline storage equipment such as DVD CD server; (3) transference apparatus information module shows the transference apparatus relevant information, comprises the task of current execution, progress and transference apparatus state etc.; (4) storage medium information module, the relevant information of nearline storage medium in the display system.
7. graded memory management system according to claim 2, it is characterized in that, described storage policy server module is used to realize various storage policies, and this module is mainly used in user's actual storage need carry out interface, and finishes the generation of store tasks by API.
8. graded memory management system according to claim 2, it is characterized in that, described system external interface module: the API that storage system externally is provided, realize the external interface of storage system, storage system realizes the storage policy customization by a series of API are provided, and the memory scheduling server carries out the TCP communication by control API and this server.
9. according to claim 1,2 described graded memory management systems, it is characterized in that, described hierarchical storage management software: (1) employing two-stage API is connected with other system, one-level is memory scheduling API, adopting standard C ++ interface card, another level are that system provides streaming tape read-write API; (2) migration of data is finished by data migration server independently, according to bandwidth, every 3-4 streamer tape device can dispose a data migration server, formed data migration service server cluster, all memory devices use by sharing by all data migration servers, and can expand at any time as required; (3) data file filing storage format adopts the MTF reference format; (4) adopt the independent database system, database storage software uses SQL Server, ORACLE, and three kinds of normal data library softwares of DB2, and database server both can merge with the system management software, also can separate; (5) tape group and store tasks can parallel processings, processing for the big object of big capacity, the pattern that system adopts a plurality of tasks in parallel to handle, promptly when certain tape group need be stored several tasks, system can according to circumstances distribute different drivers to file simultaneously on several tapes on the same group; (6) support multiple driver and multiple tape library, the streaming tape driver comprises: LTO, DTF2, AIT, AHDT, VXA, STK; The streaming tape storehouse comprises: SONY, IBM, ADIC, STK; (7) segmentation that can carry out Large Volume Data is adjusted back: by create a frame index and a second index in archiving process, can carry out the segmentation readjustment to Large Volume Data, promptly, automatically big file is cut, and generate a new data file according to going into a little and length of asking; (8) adopt manually the arrangement and the system that carry out the tape storage space by storage administration user monitoring interface to put two kinds of mechanism in order according to the tape storage dead band automatically.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2005100001397A CN100452861C (en) | 2005-01-05 | 2005-01-05 | hierarchical storage management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2005100001397A CN100452861C (en) | 2005-01-05 | 2005-01-05 | hierarchical storage management system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN1801923A true CN1801923A (en) | 2006-07-12 |
| CN100452861C CN100452861C (en) | 2009-01-14 |
Family
ID=36811639
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2005100001397A Expired - Fee Related CN100452861C (en) | 2005-01-05 | 2005-01-05 | hierarchical storage management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100452861C (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102163232A (en) * | 2011-04-18 | 2011-08-24 | 国电南瑞科技股份有限公司 | SQL (Structured Query Language) interface implementing method supporting IEC61850 object query |
| CN102624911A (en) * | 2012-03-14 | 2012-08-01 | 中山大学 | Cluster-based visible media storage system |
| CN101311912B (en) * | 2007-05-24 | 2012-10-03 | 株式会社日立制作所 | Computing system, method of controlling the same, and system management unit |
| CN103067744A (en) * | 2012-12-28 | 2013-04-24 | 中央电视台 | Media file preparing method capable of dynamically adjusting priority in real time |
| CN103294794A (en) * | 2013-05-23 | 2013-09-11 | 上海爱数软件有限公司 | On-line archiving and file access system |
| CN103810059A (en) * | 2012-11-15 | 2014-05-21 | 国家电网公司 | Backup method and device |
| CN104081374A (en) * | 2011-06-14 | 2014-10-01 | 内茨恩公司 | Hierarchical display-server system and method |
| CN104199926A (en) * | 2014-09-01 | 2014-12-10 | 北京联创信安科技有限公司 | File archiving implementation method and device and file access method and device |
| CN105404474A (en) * | 2015-12-07 | 2016-03-16 | 上海爱数信息技术股份有限公司 | Data migration method of heterogeneous distributed memory system |
| CN107132991A (en) * | 2016-02-26 | 2017-09-05 | 深圳市深信服电子科技有限公司 | The Bedding storage method and virtual storage system of virtual storage system |
| CN107993681A (en) * | 2017-11-22 | 2018-05-04 | 苏州互盟信息存储技术有限公司 | A kind of near line disk array storehouse |
| CN108022607A (en) * | 2017-11-22 | 2018-05-11 | 苏州互盟信息存储技术有限公司 | A kind of near line disk array storehouse with contact point |
| CN108053843A (en) * | 2017-11-22 | 2018-05-18 | 苏州互盟信息存储技术有限公司 | A kind of near line disk array storehouse with motion in one dimension manipulator |
| CN111866038A (en) * | 2019-04-25 | 2020-10-30 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage dynamic defense system and method based on heterogeneous multiple copies |
| CN112328692A (en) * | 2020-11-12 | 2021-02-05 | 桂林旅游学院 | Travel big data information storage method |
| CN116627349A (en) * | 2023-06-14 | 2023-08-22 | 富士胶片（中国）投资有限公司 | Medical image data processing method, medical image data processing device, medical image data processing equipment, medical image data processing medium and computer program product |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6804719B1 (en) * | 2000-08-24 | 2004-10-12 | Microsoft Corporation | Method and system for relocating files that are partially stored in remote storage |
| DE60128200T2 (en) * | 2000-12-15 | 2008-01-24 | International Business Machines Corp. | Method and system for scalable, high performance hierarchical storage management |
| CN1167024C (en) * | 2002-05-09 | 2004-09-15 | 邓祖道 | Digitalized audio-video information resource system for remote education and its implementing method |
-
2005
- 2005-01-05 CN CNB2005100001397A patent/CN100452861C/en not_active Expired - Fee Related
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101311912B (en) * | 2007-05-24 | 2012-10-03 | 株式会社日立制作所 | Computing system, method of controlling the same, and system management unit |
| US8762995B2 (en) | 2007-05-24 | 2014-06-24 | Hitachi, Ltd. | Computing system, method of controlling the same, and system management unit which plan a data migration according to a computation job execution schedule |
| CN102163232A (en) * | 2011-04-18 | 2011-08-24 | 国电南瑞科技股份有限公司 | SQL (Structured Query Language) interface implementing method supporting IEC61850 object query |
| CN102163232B (en) * | 2011-04-18 | 2012-12-05 | 国电南瑞科技股份有限公司 | SQL (Structured Query Language) interface implementing method supporting IEC61850 object query |
| CN104081374B (en) * | 2011-06-14 | 2017-12-08 | 内茨恩公司 | Classification display device server system and method |
| CN104081374A (en) * | 2011-06-14 | 2014-10-01 | 内茨恩公司 | Hierarchical display-server system and method |
| CN102624911A (en) * | 2012-03-14 | 2012-08-01 | 中山大学 | Cluster-based visible media storage system |
| CN103810059A (en) * | 2012-11-15 | 2014-05-21 | 国家电网公司 | Backup method and device |
| CN103067744A (en) * | 2012-12-28 | 2013-04-24 | 中央电视台 | Media file preparing method capable of dynamically adjusting priority in real time |
| CN103067744B (en) * | 2012-12-28 | 2014-03-12 | 中央电视台 | Media file preparing method capable of dynamically adjusting priority in real time |
| CN103294794B (en) * | 2013-05-23 | 2017-07-28 | 上海爱数信息技术股份有限公司 | A kind of online elite archiving and the system for accessing file |
| CN103294794A (en) * | 2013-05-23 | 2013-09-11 | 上海爱数软件有限公司 | On-line archiving and file access system |
| CN104199926A (en) * | 2014-09-01 | 2014-12-10 | 北京联创信安科技有限公司 | File archiving implementation method and device and file access method and device |
| CN105404474A (en) * | 2015-12-07 | 2016-03-16 | 上海爱数信息技术股份有限公司 | Data migration method of heterogeneous distributed memory system |
| CN107132991A (en) * | 2016-02-26 | 2017-09-05 | 深圳市深信服电子科技有限公司 | The Bedding storage method and virtual storage system of virtual storage system |
| CN107132991B (en) * | 2016-02-26 | 2020-11-20 | 深信服科技股份有限公司 | Layered storage method of virtual storage system and virtual storage system |
| CN108053843A (en) * | 2017-11-22 | 2018-05-18 | 苏州互盟信息存储技术有限公司 | A kind of near line disk array storehouse with motion in one dimension manipulator |
| CN108022607A (en) * | 2017-11-22 | 2018-05-11 | 苏州互盟信息存储技术有限公司 | A kind of near line disk array storehouse with contact point |
| CN108053843B (en) * | 2017-11-22 | 2019-08-27 | 苏州互盟信息存储技术有限公司 | A kind of near line disk array library with motion in one dimension manipulator |
| CN108022607B (en) * | 2017-11-22 | 2019-08-27 | 苏州互盟信息存储技术有限公司 | A kind of near line disk array library with contact point |
| CN107993681B (en) * | 2017-11-22 | 2019-10-15 | 苏州互盟信息存储技术有限公司 | A kind of near line disk array library |
| CN107993681A (en) * | 2017-11-22 | 2018-05-04 | 苏州互盟信息存储技术有限公司 | A kind of near line disk array storehouse |
| CN111866038A (en) * | 2019-04-25 | 2020-10-30 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage dynamic defense system and method based on heterogeneous multiple copies |
| CN112328692A (en) * | 2020-11-12 | 2021-02-05 | 桂林旅游学院 | Travel big data information storage method |
| CN116627349A (en) * | 2023-06-14 | 2023-08-22 | 富士胶片（中国）投资有限公司 | Medical image data processing method, medical image data processing device, medical image data processing equipment, medical image data processing medium and computer program product |
Also Published As
| Publication number | Publication date |
|---|---|
| CN100452861C (en) | 2009-01-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100452861C (en) | hierarchical storage management system | |
| US20190188190A1 (en) | Scaling stateful clusters while maintaining access | |
| US9223820B2 (en) | Partitioning data for parallel processing | |
| AU2013347807B2 (en) | Scaling computing clusters | |
| EP1569085B1 (en) | Method and apparatus for increasing data storage capacity | |
| JP3117390B2 (en) | Method of distributing a data set among a plurality of disks and related apparatus and method | |
| US20020194340A1 (en) | Enterprise storage resource management system | |
| CN102567138B (en) | Method and system for prioritization of data items for backup in a computing environment | |
| US7941455B2 (en) | Notification for a distributed file system | |
| US9836516B2 (en) | Parallel scanners for log based replication | |
| US20100011364A1 (en) | Data Storage in Distributed Systems | |
| US11442927B1 (en) | Storage performance-based distribution of deduplicated data to nodes within a clustered storage environment | |
| US20100082546A1 (en) | Storage Tiers for Database Server System | |
| CN1300667C (en) | Maintaining information using a plurality of storage attributes | |
| US7069270B1 (en) | Automated method and mechanism for converting a single instance application to a multiple instance application | |
| US20110296422A1 (en) | Switch-Aware Parallel File System | |
| CN111651302A (en) | Distributed database backup method, device and system | |
| US12079193B2 (en) | Distributed storage systems and methods to provide change tracking integrated with scalable databases | |
| CN102360382B (en) | High-speed object-based parallel storage system directory replication method | |
| CN109308310B (en) | Subsystem data interconnection processing method for asset management platform | |
| CN116414624A (en) | Object backup method and device | |
| CN110347654B (en) | Method and device for online cluster characteristics | |
| US12026129B2 (en) | Atomic cache management of file collections | |
| CN112084156A (en) | Hybrid storage system and self-adaptive backup method of file | |
| CN101799780A (en) | Configuration information file processing method, system, server, and virtual tape library equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20090114 Termination date: 20210105 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |