CA1183274A - Virtual storage system and method - Google Patents
Virtual storage system and methodInfo
- Publication number
- CA1183274A CA1183274A CA000402477A CA402477A CA1183274A CA 1183274 A CA1183274 A CA 1183274A CA 000402477 A CA000402477 A CA 000402477A CA 402477 A CA402477 A CA 402477A CA 1183274 A CA1183274 A CA 1183274A
- Authority
- CA
- Canada
- Prior art keywords
- data
- storage
- host
- storage devices
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired
Links
Landscapes
- Memory System Of A Hierarchy Structure (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Abstract of the Disclosure A virtual storage system for use in conjunction with a host computer is disclosed. The system features a memory control processor external to the host computer which divides user sized data files into blocks of a size con-venient for storage on, e.g., magnetic media and individually assigns these blocks to locations determined external to the host. In this way, the extent of a particular data file is not specified by the user nor is empty space allocated to future use. The virtual memory system may additionally comprise a high speed cache memory for receiving data written to the memory system at high speed from the host and for supplying memory data at high speed to the host, thus improving system performance. Data compression and decompression may be incorporated in the storage system. Numerous data back up and automated recovery processing operations may additionally be performed by this system without specific instruction from the host.
Description
S'rC-l'10 ~83~f~
VIl:~TUAL STOR2~GE SYSTEM ~ND MET~IOD
Field of the Invention ___ This invention relates to the field of data processing and storage. More particularly, the invention relates to a data storage system adapted to be connected to a host digital computer for determination of where on associated magnetic rnemory units indlvidual data records are to be stored, and for recall of the addresses determined upon the host's requiring the data. In this way irnproved efficiency of storage is achieved, while the host computer is largely relieved of the memory control unction.
~a~kcJround of t'he Invention In general, the operation of a data processing system involves two b~sic functions: arithmetic and logic operations p~:Ecrmecl ~n dclta contai.ned withill a host computer, and, sta-ted .1.5 br~adly, i.nput and c~ul:put control. That is, the data to be procc~r.ed murs.t ~:irst be :input to t'he cornputer, while the comp~l~al oll~pllk typ;.cs~ l..y ~esults ;.n further storage of the ~lslta to~ether wi.th results of the operation. While it is possible to perform such processing in such a way that the ~0 d.lta input is processed and the output is generated without lon~-terrn storaye of either the data or of the results o the operation, it is far more cornmon, particularly in large ~rocess.irly systerns, that the data fil.es operated upon are ~erm.~rlent f.i.les such as payroll fi.les, employee files, c~tstomer ~rj l.i.st:s arld tJIe l:ike, w'hich are updated periodi.cally as well ~,r,~ h~ing ~sed to ge!lerate actual comp~ter output s~ch as f~
3~
payroll checks and the like. It is therefore important that means be provided for long-term storage of such data.
Various types of memory means for long-term storage of digital data have been provided in the prior art. These comprise magnetic tape memory, magnetic disc memory, magnetic drum memory~ solid state random access memory, magnetic bub~le memory, and charge coupled device (CCD) memory as well as others. The choice of which sort of memory is to be used for a particular operation inevitably involves a cost/speed trade-off, ~hat is, the faster the access time provided with respect to any given bit stored on a given type of magnetic memory, the more expensive it is to store the bit. There has developed in general a heirarchy of memory according ~o which the central processing unit of the computer comprises solid state random access memory (RAM), while an intermediate high speed cache rnemory used in conjunction with the host may comprise a less expensive, less high speed form of solid 9 tate ~AM or C~D memory. The next step in the heirarchy may be a f:Lxed head d:isk rotating magnetic memory external to ~0 the CPU of relatively lesser speed but capable of storing a vastly ~reater ~uantity of data at signi~icantly lower co~t; further down the heirarchy are user-replaceable disks and finally tape drives.
The prior art has been extensively concerned with ~S improvement in methods of utilizing the various forms of memory so as to achieve higher efficiency of use of the various types of media available, to reduce costs, and simul-taneously to devise methods whereby the various time limitations of the less expensive memory means can be overcome, thus also 3~ .lmproving efficiency. However, as yet no ultimately satisfactory solut:Lon has appeared.
STC-l~O
~ ~ ~3~7~
It wi]l be understood by those skilled in the art that control of access to, e.g., a giverl tape or disk file has generally been accomplished by rneans of a command originating in the host central processing unit. In more basic systems, the user of the computer defines on which portions of what storage media the files necessary to complete his job àre stored. Upon in.itiatlon of the job, the host then passes the appropriate instructions on to the appropriate disk or tape controllers. In more advanced systems -the user of the computer may need only speciEy the name of his file, the host is capable of locating ~he file and, e.g., instructing an operator to mount a particular reel of tape, or instructing a disk controller to access a given portion of a disk drive, as necessary. However, in both schemes, it is the host which L5 is responsi.ble for causing the controller to access the appropriate storage medium. The controller itself is passive and merely responds to the host's commands.
An improvement on this practice was disclosed in ~a.d:i~n Pcl~ent No. 1,153,126, whlch issued on
VIl:~TUAL STOR2~GE SYSTEM ~ND MET~IOD
Field of the Invention ___ This invention relates to the field of data processing and storage. More particularly, the invention relates to a data storage system adapted to be connected to a host digital computer for determination of where on associated magnetic rnemory units indlvidual data records are to be stored, and for recall of the addresses determined upon the host's requiring the data. In this way irnproved efficiency of storage is achieved, while the host computer is largely relieved of the memory control unction.
~a~kcJround of t'he Invention In general, the operation of a data processing system involves two b~sic functions: arithmetic and logic operations p~:Ecrmecl ~n dclta contai.ned withill a host computer, and, sta-ted .1.5 br~adly, i.nput and c~ul:put control. That is, the data to be procc~r.ed murs.t ~:irst be :input to t'he cornputer, while the comp~l~al oll~pllk typ;.cs~ l..y ~esults ;.n further storage of the ~lslta to~ether wi.th results of the operation. While it is possible to perform such processing in such a way that the ~0 d.lta input is processed and the output is generated without lon~-terrn storaye of either the data or of the results o the operation, it is far more cornmon, particularly in large ~rocess.irly systerns, that the data fil.es operated upon are ~erm.~rlent f.i.les such as payroll fi.les, employee files, c~tstomer ~rj l.i.st:s arld tJIe l:ike, w'hich are updated periodi.cally as well ~,r,~ h~ing ~sed to ge!lerate actual comp~ter output s~ch as f~
3~
payroll checks and the like. It is therefore important that means be provided for long-term storage of such data.
Various types of memory means for long-term storage of digital data have been provided in the prior art. These comprise magnetic tape memory, magnetic disc memory, magnetic drum memory~ solid state random access memory, magnetic bub~le memory, and charge coupled device (CCD) memory as well as others. The choice of which sort of memory is to be used for a particular operation inevitably involves a cost/speed trade-off, ~hat is, the faster the access time provided with respect to any given bit stored on a given type of magnetic memory, the more expensive it is to store the bit. There has developed in general a heirarchy of memory according ~o which the central processing unit of the computer comprises solid state random access memory (RAM), while an intermediate high speed cache rnemory used in conjunction with the host may comprise a less expensive, less high speed form of solid 9 tate ~AM or C~D memory. The next step in the heirarchy may be a f:Lxed head d:isk rotating magnetic memory external to ~0 the CPU of relatively lesser speed but capable of storing a vastly ~reater ~uantity of data at signi~icantly lower co~t; further down the heirarchy are user-replaceable disks and finally tape drives.
The prior art has been extensively concerned with ~S improvement in methods of utilizing the various forms of memory so as to achieve higher efficiency of use of the various types of media available, to reduce costs, and simul-taneously to devise methods whereby the various time limitations of the less expensive memory means can be overcome, thus also 3~ .lmproving efficiency. However, as yet no ultimately satisfactory solut:Lon has appeared.
STC-l~O
~ ~ ~3~7~
It wi]l be understood by those skilled in the art that control of access to, e.g., a giverl tape or disk file has generally been accomplished by rneans of a command originating in the host central processing unit. In more basic systems, the user of the computer defines on which portions of what storage media the files necessary to complete his job àre stored. Upon in.itiatlon of the job, the host then passes the appropriate instructions on to the appropriate disk or tape controllers. In more advanced systems -the user of the computer may need only speciEy the name of his file, the host is capable of locating ~he file and, e.g., instructing an operator to mount a particular reel of tape, or instructing a disk controller to access a given portion of a disk drive, as necessary. However, in both schemes, it is the host which L5 is responsi.ble for causing the controller to access the appropriate storage medium. The controller itself is passive and merely responds to the host's commands.
An improvement on this practice was disclosed in ~a.d:i~n Pcl~ent No. 1,153,126, whlch issued on
2~ 30 ~UCJU5t, .1~83 in the name O:e Barry B. White and a~:i.cJn~d ~o l he a~:ignee o:E the pres~nt app:Lication.
'~le or.ig.i.nal. invent.ion of White was aimed at achieving better, more ~Eficient use of the storage space available on disk media by functionally mirriicing a tape drive. Inasrnuch as a ~5 tape clrive need only be addressed once at the beginning of each file and thereafter records can be written sequen-tially thereto without being interspersed with address marks uniquely i~leritlf.iahle (unJ.ike d;.s};-stored data), the arnount oE data slored w.ith;.n a given arca of tape expressed as a pe~centage Qf the t:okal. area avai.].able is extrernel.y hi.gh. By compariSon, -S~'C-l~O
'~le or.ig.i.nal. invent.ion of White was aimed at achieving better, more ~Eficient use of the storage space available on disk media by functionally mirriicing a tape drive. Inasrnuch as a ~5 tape clrive need only be addressed once at the beginning of each file and thereafter records can be written sequen-tially thereto without being interspersed with address marks uniquely i~leritlf.iahle (unJ.ike d;.s};-stored data), the arnount oE data slored w.ith;.n a given arca of tape expressed as a pe~centage Qf the t:okal. area avai.].able is extrernel.y hi.gh. By compariSon, -S~'C-l~O
3;27~
on disk, address marks must be provided for each record stored on a given sector of each disk, the address rnar~.s consume a large proportion of the space allotted. Moreover, it had been the prior practice to allot a particular portion of a given disk to a given file and not to use this portion of that disk for any other file thereafter, so that unless by coinciAence the size of the file was in fact equal to the maximum size allotted to it, which is usually not the case, that area of the disk would be underutilized. The net result is that on the average disXs tend to be used to something less than 50~ of their capacity.
The invention disclosed in the ~hite paten-t was of a virtual storage system to be interposed between a host CPU
and disk drives. The virtual storage system comprised an intelligent processor which could itself make decisions as to where on the associated disk drives data could be stored. The virtual storage system responds to commands nominally issued ~o k~pe clrives by the host and converts these to commands us~fu:l ~or control of disk drives. ln this way, the virtual ~,n s-~orage system allows disks to functiorlally mimic tape in ordcr to ach:icve the cEficiency of addressing and formatting considerations mentioned above. The original invention thus includes the concept of a mernory system external to a host ~ CPU having intelligence for determining where on associated disk drives portions of a given data file are to be stored, and further comprises the concept of individually allotting ~c~ce on disks for individual portions of a given user file.
In kh;.s way the prior art practice of allotting all of a ~iv~n por~ion of a given dis~ to a given file is elimina-ted 3~ al~cl the disk C~ll clccordingly be used to far higher efficiency.
on disk, address marks must be provided for each record stored on a given sector of each disk, the address rnar~.s consume a large proportion of the space allotted. Moreover, it had been the prior practice to allot a particular portion of a given disk to a given file and not to use this portion of that disk for any other file thereafter, so that unless by coinciAence the size of the file was in fact equal to the maximum size allotted to it, which is usually not the case, that area of the disk would be underutilized. The net result is that on the average disXs tend to be used to something less than 50~ of their capacity.
The invention disclosed in the ~hite paten-t was of a virtual storage system to be interposed between a host CPU
and disk drives. The virtual storage system comprised an intelligent processor which could itself make decisions as to where on the associated disk drives data could be stored. The virtual storage system responds to commands nominally issued ~o k~pe clrives by the host and converts these to commands us~fu:l ~or control of disk drives. ln this way, the virtual ~,n s-~orage system allows disks to functiorlally mimic tape in ordcr to ach:icve the cEficiency of addressing and formatting considerations mentioned above. The original invention thus includes the concept of a mernory system external to a host ~ CPU having intelligence for determining where on associated disk drives portions of a given data file are to be stored, and further comprises the concept of individually allotting ~c~ce on disks for individual portions of a given user file.
In kh;.s way the prior art practice of allotting all of a ~iv~n por~ion of a given dis~ to a given file is elimina-ted 3~ al~cl the disk C~ll clccordingly be used to far higher efficiency.
- 4 -STC-l~O
" ~L3L~327~
The Whi-te patent also disclo~es that it might be useful to include data ccmpression and decornpression mearls in the vi.rtual storage system of the invent;,on. Data compression is a concept which had been well known for use in host CPIl's but had not previously been done external to the host. This distinction is subsumed under the fact that this was the firs-t system external to a host to provide intelligent processor means for control of data storage functions.
The present invention is an improvement over the original invention discussed above. According to the presen-t invention, the virtual storage system is not simply interposed bet~een a host computer and disk drives. Instead, the virtual storage system of the present invention operates in conjunction with modi.fications to the operating system of the host computer.
T~pically this w.ill involve some slight reprogramming of the host. Furthermore, the virtual storage system according to the present invention is not constrained to cause disk Arives to functionally mimic tape drives or other defined storage ~lev.ic~s, asj .in the parent inventloll, but .instead presents to ~0 khe~,'hos-t the irnac3e of a tabula_rasa; i.e., a blank shee-t upon which the 'host, and thus -thc user, can wri.te wi-thout con~tra:i.nt. a3 to the forrrlat or disposal of the data.
ects of the Invention Accordingly, it is an object of the invention to ~5 provide an improved data processing and storage system.
A further object of t.he invention is to provide a data processinc3 and storage system in which decisions as -to the loca-t:i.on for storage of data are rnade external to the ari.thrnet:ic and logic unit, in a memory storage system having ~(~ inke'lligence.
... .
STC-140 ~3~
A further object of the invention is to provide a memory system having decision-making capacity with respect to storage locations for data, which additionally comprises means for dividing user-defined files into sub-blocks of data for storage in available locations on memory means.
~ further object of the invention is to provide methods and means for more efficient data storage.
Yet a further object of the invention is to provide a memory storage system in which the allocation of storage ].ocations to given data is made external to a host computer, in the memory system itself, whereby data compression and decompression can be performed upon the data in the memory system and not in the host computer.
Still a further object of the invention is to provide a memory system for use in conjunction with a host digital computer in which duplication of data is performed by the memory system in accordance with usage, exclusive of the control of the host computer.
A Einal object oE the invention is to provide methods or more e~Eicient use of data storage apparatus.
Other aspects and objects of the invention will appear to those skil:Led in the art.
~ummary of the Invention I'he invention fulfills the above mentioned needs of the art and objects of the invention by its provision of a virtual storage system which comprises a processor for allocating spaces on data storage means to blocks of data supplied by a host computer. The processor, after having allocated addresses on, ~.g., disk memory means to the data, causes the data to 3~ ~low from a host interface means, which may typically comprise ~ ~3~7~
a data cache for storage of a given portion of data until appropriate locations can be determined, -through disk interface means and onto -the disks. The addresses are then stored by, for example, user file name. When the host later calls for the same file, the virtual storage system's controller refers to its file location memory and determines where on the various disks -the data is stored. The data can then be transmitted back to the host via the cache for further processing. In this way no space on any given disk is explicitly or permanently assigned -to any given user file. Instead, space on disk is taken up only as needed by active files. In this way the address memory of the controller at all times "knows" what areas of the disks are available for further storage. Similarly, it "knows" which areas of the disks have been recently operated upon l~ and therefore can be used to control efficient duplication of data fo~ backup, i.e., for use in case of loss of the main data file on a primary memory means. In addition to the controller, in a preferred embodiment the virtual storage system of the :LnventLon comprises at least one channel adapter for connecting 2~ the host computer to the cache which may additionally comprise data corrlpre,ssion and decompression means so as to achieve further e~:Lc:Lency of storage.
The present invention is directed to apparatus for the storage of digital data organized into records. The apparatus ?~5 includes host interface means for receiving data from a host computer, cache means connected to -the host interface rneans for temporary storage of the data, s-torage device interface means for ~onnecting the cache means to one or more memory storage devices ~3~7~
o~ plural classes of storage devices for comparatively long-term storage of the da~a, and controller means. The controller means comprises address recall means, and (a) determines where the data is to be stored on the storage devices, (b) controls the flow of the data from -the host interface means through the cache means and the storage device interface means onto the storage devices, and (c) stores the addresses a-t which the data has been stored on the storage devices. As a result, a given user-defined data file may be stored on a plurality of storage devices, selec-ted from plural classes of storage devices, by the controller means.
The invention is also directed to a data processing and storage system, including at least one host computer for performing arithmetic and logic operations on digital data comprising an extremely high speed, relatively low capacity memory means, and interface means for transmitting and receiving data ~rom external memory means, and a virtual storage system.
The virtual storage system comprises relatively low speed, high capaclty storage devices of plural classes of storage devices, ho~s-t inter~`ace means for receiv:ing data from and -transmitting 2~ da-ta to th~ hos-t, a high speed, relatively low capacity cache memory means connec-ted -to -the host :i.nterface rneans, and a controller for allocat:ing appropriate storage locations on the relatively low speed, high capaci-ty storage devices for data received from the host and stored in the cache, for accordingly con-trolling the flow of data from the cache -to -the relatively low speed, high capacity storage devices selected from plural classes of ~torage devices by the con-troller, and for storing the s-torage loc~lklo~ls . . ~ .
~ ~3~7~
The invention is further directed to a data storage and recall system, comprising means for receiving data from means for processing data, means for long-term storage of the data including s-torage devices selected from plural classes of storage devices, means for temporary storage of the data, means for maintaining a directory of available storage loca-tions on the long-term storage means, means for subdividing a user da-ta set to subportions sized to fit the available storage locations, and means for causing the subportions to be stored in the available storage ].ocations and for maintaining a directory of the locations at which the subpor-ti.ons ha-ve been stored, means for controlling flow of the data through the system, memory means -for reca].l of the l.ocations for long--term storage of the da-ta, and means :~or outputting the data upon instruction.
1~
Bri,ef Descrip-tion of the Drawings The inven-tion will be bet-ter understood if reference lc1rnad~ to the accompanylng drawings in which:
~ ig. I shows the prior art method of storing data 2~ on magnet.i.c d:Lsks ancl on magne-tic tapes;
~ i.g. ~ shows the comparable method according to the :Lnvention;
- 7b -~3~
Fig. 3 shows an overall view of the virtual storage system of the invention as connected to host computers, plural disk drives and tape drives;
Yig. 4 shows a more detailed view of the virtual storage system of the invention as connected to a host computer:
Fig. 5 shows the hardware comprising the con~rol processor of the virtual storge system of the invention;
Fig. 6 shows a schematic view of the software components of the virtual storage system of the invention;
Fig. 7 shows the host resident components of the virtual storage system of the invention;
Fig. 8 shows a more detailed block diagram of the virtual storage system of the invention and the channel adapter which serves as the host interface unit;
lS Fig. 9 shows how various disk Erame groups can be organized on a plurality of disks according to the user specifications;
Fig. 10 shows the flow of control signals through the vlrtual ~torage system of the invention;
.~ F'ig. 11 shows a comparable view oE the flow of data through the system O e the inverrtion; and Fig. 12 shows a schematic diagram of hardware which might be used for data compression and decompression in the channel adapter of the system according to the invention.
Descri~on of the Preferred Embodiments As discussed above, the prior art practice when storing a particular file of data on, for example, a disk drive w~s to alLocate a portion of this drive permanently to that Eile alld to write into that space as needed. As files Erequerltly grow in size over time, the practice would ordinarily ~ ~ !
STC- 140 `-~3~
be to allocate a large enough file size on disk that it would not be filled up for some time thereafter despite continuing use. The usual result is that the space allocated for the particular file is not filled thereby and accordingly the disk is not used to full efficiency. This practice is shown schematically in Fig. 1 in which a plurality of disk drives are shown schematically at 1, 2 and 3. The three disks 1-3 are shown divided into allocated areas as a-t A, B, C,...H; for example, drive 2 is allotted to a file denoted E. In each, the axea shown cross-hatched is tha~ which might be occupied by data. The remaining region is therefore unused.
File E is shown as comprising two areas E1, E2. This is mearlt to indicate that the file might grow from El to E2 over a period of, say, several months and that in accordance with the prior art practice, the later added material E2 would be added to the same allocated area on the same disk clr:ive 2 as El previously written thereto.
An additional prior art practice was to store more ~requently used data on disk and less Erequently used data on tape, i.nasmuch as the access tirne of tape is typically greater than that of disk while the cost of tape s,torage is less than that of disk. An additional use oE tape is to "backup" the disk drive; that is, to have the same data recorded both on disk and also on tape so that in the event of damage to the disk or the like the data would be preserved on the tape and the information not lost. Accordingly three tape ~le~ are shown at K, L and M, on reels of tape 4, 5, and 6, ~chcmcltically showing this practice. It will be no~ed that ~ sin~le data fi.le is present on each tape. This is a typical prlor art practice, as it is yenerally deemed simpler to g _ fetch a reel of tape from storage when a new file is sought than to unwind a reel of tape through a first file in order to reach a second file. This, of course, leads to substantial underutilization of the total tape available.
Thus, in the prior art, when the host seeks access to one of the previously assigned user files shown in Fig. 1, typically the host will convert the user's request for a named file into a command to a disk controller to write the contents of the file into the host interface comprised in the disk controller for passage onto the host. Thus the host is required to know where and on what type of memory means the particular file sought for is stored. Where, for example, the sought-for f.ile is stored on a demountable disk or on a tape, the host is required to convert the operator's command into a "mount disk" or "mount tape" instruction typically appearing on an operator console screen and the c~erator is then required to fetch the disk or tape called ~or ~rom an archive. Thls practice while workable is tirne-collsuming and cuts down on the effective computing capacity 2~ o;~ the host com~uter.
Fig. 2 shows in comparison the method by which data ~ 9 are stored according to the present invention. Instead o~ allocating specific areas of specific dis'ks to specific user files, space on disk or tape is assigned only as needed, as the data arrives in the virtual storage system of the invention Eor storage. By allocating storage space only as rec~u:ired, additional area is made available for the storage oE data. Moreover, the space assigned to a given file is not necessarily, or even desirably, on the same disk or tape drive~, but instead may be spread over several so that the f ~
3~
average utilization of each is more or less equal. The fact that it is the external memory system and not the host which decides where and on what sort of storage media given data is to be stored provides considerable additional versatility in operation as well as improved host utilization and simplification of user requirements.
Thus in Fig. 2 the virtual storage system of the invention is shown comprising a plurality of disX drives 10, 11 and 12, each of which has given portions of various records disposed thereon. It will be noted that records El and E2 which might be in fact different portions of the same user file are not together on the disk drives 10 through 12. Tape drives 13 and 14 are additionally provided. These may be used for bacXup of data sought to be duplicated in case of loss of the data on the disk, or may simply be data stored in files which have been inactive for longer than a given minimum periocl and have therefore been copied to the slower access but ~ar le~ls expensive tape drive media. Finally, the virtual ~torage system comprises a third memory 15 which is an address memory stor~ for use in recalling the places on disk and/or tape at which a given file is stored. Upon the host's calling ~ox a specific file, the mernory store 15 is accessed and caused by the processor to write the data from the disk or tape drive to the ho~t. The address store 15 may desirably be a RAM of comparatively limited capacity. The virtual storage ~y~tem of the invention may also comprise a high speed cache m~mory (not shown) such as another RAM of relatively larger c~pacity, for use in assembling all or a sub~portion of a given ile prior to transmission to the host, so that the ~0 ho~t n~ed not receive the data file piecemeal from the virtual STC-140 ~3~
storage system according to the invention.
Fig. 3 shows the overall layout of a data processing and storage system utilizing virtual storage according to the invention. The particular system shown in Fig. 3 shows a pair of host computers 20 and 21 each connected to a pair of virtual control processors 22 and 23, which amount to the heart of the virtual storage system of the invention. These are each in turn connected to a data base, which in the con-~iguration shown each comprise a pair o~ disk controller~ units 24 and 25 each operating a pair of disk drives 26, and 27, 28 and 29, respectively, and to a pair of tape controllers 30 and 31 having attached thereto tape drives 32 and 33, respec-tively. The virtual control processors may each also have secondary connections to the other's data base, for backup purposes. Accordingly when a host requests a specific user file or "virtual volume" the request need only specify which of the two virtual control processors 22 or 23 controls the ~at~ base within which is stored that virtual volume. The virtual control processors 22 and 23 are each able, using ~0 th~ir internal address store, to convert the name oE the file in~o the location of the data on the disk or tape unit(s) involved and forward it to the host without ~urther instruction from the host.
It is well known by those skilled in the art that there iB at present available an IBM Corporation computer ~ystem which uses disk drives external to a host computer ~or expansi.on of the CPU memory space. This is operated ~ccordlng ~o a program contained in the host known as MVS, which is an acronym for "Multiple Virtual Storages". The present 3~ Lnvention similarly relates to what might be termed "virtual"
, S~C-l~0 ~ ~ ~3~7~
storage, but is concerned with long-term storage, not expansion of the CPU storage.
Fig. 4 shows a schematic view of how a virtual st.orage system accoxding to the invention could be operated in conjunction with an IBM 370/158 or 30XX CPU having ~he MVS ~eature. The invention in this preferred embodiment would also function correctly with any other host having MVS capability, such as the Amdahl V-8 host CPU. The host computer 30 contains soEtware for various purposes denoted on the drawing such as user address space, means for accepting jobs from punch cards and orders from a console, and the like. Adap-tation of such a host operating system to be useful with the virtual storage system of the invention requires some reprograrnming of the host computer as will be discussed in further detail b~low. In essence, an additional program referred to as VES, for "Virtual Storage System Execution Subsystem`' is added as is additional. VSS host utility software to generate reports rclat.ing to the operation of the virtual storage system of ~hc .invellt:ion so as to keep the user apprisea of its status.
~0 rrhe host 30 is connected by a bus 32 to the virtual control p~cc~or 34 (VcP) of the invent.ion whic'h in turn is connected Vi~ disk controllers (not shown) by additional busses to one or more disk storage devices 38. The bus 32 leading from , the host 30 to the virtual control processor 34 is connected to a chanrlel adapter (CA) unit 40 which connects the host channel to the memory bus so as to transfer data from the host 30 to th~ d.isk drives 3S as well as Eor performance of additional funct:ion~, as will be discussed in further detail below. The nctual. opcration of the virtual co~trol processor is ~Inder 3~ t,he control o a Virtual Storage System Operating System * Trad~ Mark - 13 -3~27~
(VOS) which receives op~rator input via a VCP console 42;
additional software is conta;ned in the virtual control processor for storage management and reporting functions as noted.
Fig. 5 shows the hardware comprising the virtual con-trol processor (VCP) 34 in broad outline. As shown, the console ~2 is connected to the VCP 34 through a conventiona] channel.
Other conventional channels, numbered CHl-CH5 and indicated generally at 48, are used to interface with backend storage such as disk drives and tape drives. The VCP 34 also comprises one or more channel adapters 50 for interfacing with a like number of host processors (not shown). The channel adapters 50 perform all required interfacing activities with a host.
To thc host the channel appears to be a storage device control unit which responds to chanrlel protocol in a conventional manner. Fach channel adapter 50 responds to two addresses in the preferred embodiment, one for data transfer and one for exch~ange of control information. During data transer, the challnel adapter perEorrns cornpression and decornpression on 2~ llser data. D~tail~ of the channel ~dapters are to be Eound in ~o-p~n(lirlcJ Canadiarl appl:iclatioll Serlal No. ~02,465 f:iled on 7 May, .l982 (At~orney's Docket No. SrrC-142).
The VCP 34 additionally comprises a cache buffer memory 52. This may be conventional solid state random access memory (F~M). The cache buffer 52 is a portion of the VCP rnain mcmory which is used to perform storage management functions on user data. Blocks of user formatted data are ~ssembled in~o "pa~3es" o~ convenient size Eor storage on the backend ~lcvlce chc~sen. For eY~arnple, in a typical disk dri~e, the paqe w~uld be equal t~ one disk track. The cache is accordingly ~32'7~
divided into a plurality of "frames" into each of which one page can f.it. When a frame is full with a page of data it is written to the appropriate backend storage device. A
plurality of frames may be provided in the cache so that simultaneously hosts can be reading and writing to the cache buffer and so that there need be no delay in host operations caused by the operation of the backend storage device chosen.
The final component of the VCP 34 is a processor unit 54 which in a presently preferred embodiment may comprise a Magnuson M80 computer under the control of which the above operations are affected. This processor unit 54 comprises memory means for storing the addresses c>f available space on the baclcend storage devices used. Those addresses in which cla-ta is already stored are cross-referenced in this memory '15 toge-ther with means for identiEying the specific records ~tored at the addresses. In this way, when the host calls Eor a 3peci~ic file the VCP processor 54 can consult this memo~y ~nl determine where in backend'storage the data is 3tored. IE the part.icular storage medi.um is mounted on a ~n cli~'k or tape dr$ve th~ ~irst page oE data can be then read direct].y into t'he cache buffer 52. If, for example, the file called for is stored on a reel of tape not presently mounted, t'he processor can cause an operator instruction to appear on the VCP console 42 or on a host console, to e.g., mount the appropriate reel of tape, whereupon the data can be read into the cache 52 a page at a ti.me for forwarding to the host.
It will be appreciated by those skilled in the art th~t t'he concept o~ using the cache 52 to divide user formatted data which may be of any block size and o:E any total length - 15 ~
~/ ) C--14 0 ~ r7 ~
into pages for storage of size convenient for storage on predetermined backend storage devices, thus treating each page separately from the standpoint of allocating storage space is new to the artO The host thus does not enter into the process of data management and control, to the exten-t that the virtual control processor even reformats the data. By comparison, the prior art practice was to permit the user to define the file to be the maximum length envisioned for future use and to select the type and location of the magnetic storage medium on which the file was to be permanently kept.
Not only is this practice inefficient in that almost inevitably the file would be made bigger than necessary, but it requires that the user (or, in more advanced systems, the host) make decisions as to the storage utilization~ By comparison, use of the virtual storage system of the invention means that each and every file can be divided into blocks of a size convenLen~. for ~torage from the storage system's point of view, thus ensuring Ear higher efficiency of storage. More ov~r, the ~act that the storage location is decided upon by 2~ the ~torage ~ystem relieves both user and host of this task.
In certain embodiments of the present invention even the choice o~ the type of magnetic storage medium on which a given block of data is stored is made by the virtual storage system of the invention, based on usage history of the file.
Fig. 6 shows a block diagram of the software comprised by the virtual storage system of the invention. Again, the VCP con.sole 4~ i.s shown as it has, of course, input to the operatiorl of the system/ as cloes ~he channel adapter. The VCP proce~sor (54 in Fig. 5) operates on the VSS Operating ~ 16 -System (VOS), and on functional software which performs storage management and backup/recovery ,unctions. VOS performs typical operating system functions for the VCP 34, includiny I/0 services, main storage allocation and console communication.
Functional software performs the following storage management functions: control of host/VSS data flow, backend device management, disk drive space management; recovery management, and diagnostic/service reporting. Additionally, the functional s`oftware is used to control backup writing of records stored on disk to tape as well as migration of infrequently used files on disk to tape so as to free more disk space. Further-more, the virtual storage system of the present invention comprises means for adaptively changing the class of storage of a ~iven file if it turns out that the file is used more or less frequently than originally thought likely. It will be understood, of course, that these functions may be performed in interaction with the operator by means o~ the console 42.
As discussed above, the ut:ility of the virt~al storage ~ystem o~ the invention i.s maximized if certain judicious mod:ifications are made to the operating system software of the ho~t, largely so as to delete the host's memory control functions to the extent necessary to allow the VSS to perforrn its memory control function to the fullest. ~he modifications required to the IBM 370/158 MVS program are shown schematically in Fig. 7. Those skilled in the art will readily recognize the various components of the MVS program. The changes involve the VSS Execution Sub-system (VES) and the VSS utility ~unc-tion. The VSS execution sub-system, VES, is the VSS address ~pace which functions as a secondary sub~^sys-tem in the host 3() MVS environment. V~S provides an interface between user jobs ~ ~ ~327~
running on the host and VSS itself. Major functions performed by this VES include initialization of jobs, job management, data management, interface with the console and interface with the job control lanauage cards used to detail a particular job to the host. In order to support VES, VSS maintains a number of table entries and queues in the common area of the MVS address space.
The host resident VSS utility functions provide system support and management personnel with the tools requisite to install, manage and maintain VSS. These functions provide reports on VSS status and performance and provide assistance in data and system recovery. Typically the utilities can include a report formatter to produce VSS formatted report output, user exits to interface with existing security and accounting systems and to monitor activity, transaction enable and disable capability, and means to permit control of VSS from the host computer console. Those skilled in the art will readily unc1~r~tand how MVS may be modified to yield these results.
Fig. 8 ~hows a more detailed block diagram oE the virtual storage ~ystem of the invention. As noted, its chief ~mponent~ are the channel adapter 60 indicated by dotted outline, a sy~tem bus 62 and the VSS processor 64 also indicated by a dotted outline. In a preferred embodiment the system processor 64 comprises what might in other circumstances be ~5 considered an independent computer itself; in a presently preferred embodiment, a Magnuson Co. Model M80t40 is used.
~s noted ahove, this processor contains a storage control unlt 66 connected to a memory 68; that is, a cache memory ~r tem~orary storage of data while storage assignment is b~ing completed by the VSS central processor 70, which also STC-1~0 controls the operation of char)nels 72 ~hich write the dat~
to disk or tape storage media as well known in the prior ar~. The system bus 62 connects the processor 70, the cache 68 and the channels 72 together to the channel adapter 60.
The function of the channel adapter 60 is to mate the input/output host interface 74 with the system bus inter-face 76. In the interim between the two, various operations may be performed on the data under the control of a channel adapter processor 7~, which may chiefly comprise a micro-processor, such as a Zilog Co. Z~000 unit. In the presently preferred configuration shown in Fig. 8 the channel adapter processor 78 operates on operation tags and data in parallel.
Perhaps t'he most important single function performed .in t'he channel adapter in a presently preferred embodiment is lS that of data cornpression upon writing and decompression upon readiny of data to or from backend storage, respectively. It will be unc1erstood by those skilled in tlle art that data com-pLa~ion can :in rnany cases signiEicantly reduce the ac-tual amoun~ Oe data which is required to be stored thus ylelding .0 further storage eE:iciency. It will also be appreciated by khose ski'Lled :in the art that data cornpression has been per-formecl in the prior art but only in a host environment and not by an externally operated memory system having intelligence of its own. Details of the preferred ernbodiment of the channel adapter as mentioned above are discussed in a co-pending application referred to above.
~s noted above, in the prior art -typical large cl~ta process:incJ arld stora~Je systems (i.e., ones havincJ the com;~utinc~ power oE at least one IBM 370/15~ c:Lass processor) I' ~3;;~7~
would typically employ a complete heirarchy of storage devices.
Large sequentially organized data sets would be stored on tape.
Smaller and more active sequential data sets would be stored on disk thus allowing on-line availability. Moving the small and/or active data sets to disk offers the user the advantage of fewer tape drives required and improvec~ system through-put.
However, additional requirements for management of these disk resident data sets arise. Data set placement and backup for reliability and migration become critical and time consuming tasks. The present invention addresses and solves these problems by automating routine storage management functions and by providing migration and backup capabilities which are independent of the host processor. The system of the invention thus provi.des data availability comparable to permanently mounted direct access devices such as disk drives, with access times of semiconductor storage by virtue of provision of a cache memory, and ease of use and control to the programmer of a device-lndependent data set.
~his may be termed a "virtual storage system" (VSS) ln ~hat this langua~e suggests a certain independence from p'hysical constraints and limitation. As virtual storage operating systems (such as the IBM MVS system referred to above ) freed programmers'from the constraints imposed by relatively small areas of real CPU storage, VSS frees the ~5 user from the limitatlons imposed by storage device require-ments. By automatically performing space management functions and ~torage allocation, VSS offers the user true storage devicc independence. In a presently preferred embodiment, t11ese goals are achieved by storing the user data in units called vlrtual volumes. Each virtual volume contains one ~327~
user data 82t and can be whatever size is necessary to contain that data set.
Those skilled in the art w~ll recognize that in the prior art conventional storage volumes were stored on tape or di2k drives of a fixed ~torage capacity determined by the ~vice characteristics. A larg~ data ~et might thus be stored on two tape drives. Conversely *ive different data ~et~ migh~ re~ide on a single disk volume. The vir~ual ~torage 8ystem of the invention minimizes storage management problems by introducing the conc~pt of variable siYe volumes.
Once a u~er creates a variable ~olume for a particular data ~et, it may be expanded or contracted to whatever size i8 nece~sary to contain the actual data. ~he aliocation of ~pace on whatever storaye device is chosen is performed ~utomatically. Moreover, the virtual storage system processor, maintain~ a table which Xeeps track of the volume activi~y and of the location of each of the portions of each volume.
Each volume is compri6ed of an integral number o~
"page~". From the u~er's viewpoint, each volume consists of a 2~ number o data block~ of whatever block size is convenient ~o the user. qhe virtual ~torage system of the ~nvention ~sembles the data block~ into pages conveniently ~ized for it8 own pUrpOB~. In the presently preferred embodiment each page i8 equivalent in size to one disk track. Thus a virtual volume ~onsist~ of a number of full track ~ize pages in ~arious locations ~pread throughout disk storage. The track-sized storage areas in which the pagefi ar~ 6tored are re~erred to as "frames" in both ~he cache ~nu in ~he di~k drive . ~le analoyy may be made that ~he frame i~ the bucke~
~nd ~he pag~ i~ the water. For each page of data tha~ is ~ 21 -STC-14~ 3~
transferred to on-line storage during a write operation a frame must be allocated both in cache and in disk storage.
Plural frames are provided in the cache to allow simul~aneous reading and writing.
A further subdivision of the storage system according to the presently preferred embodiment is shown in Fig. 9.
There groups of disk frames are organized into "disk frame groups" for storage management purposes. Certain attributes of control can be assigned to each frame of a given group.
Each virtual volume will be allocated frames from only one disk frame group. Thi.s enables these attributes to be assigned to each virtual volume so as to readily perform the various management functions of the storage system o~ the invention on the appropriate types of files. For example, one might put a maximum size limit on each virtual volume within a gi~en frame group. One might care to employ appropriate rec~rding modes, for example, shadow recording, in which the data i3 simultaneously written to two different disk tracks eor rel.tabllit~ purposes, or da-ta journaling, in wh;ch data wrltt~n ~imultaneously to a disk track and to a tape clrive, again or reliability improvement, on each virtual volume in a given frame group. Another possibility which can be controlled in this manner is data rnigration; that is, writing all the data in a given volume to tape when a certain percenta~e of the disk frarne group has been Eilled. For example, the data migration threshold--that poin-t at which the di~k ~rame group is deemed full--might be so controlled. These ~ k ~rame group attr:ibutes are typically established at time o~ :ln~tallation oE the VSS sys-tem but might be altered by 3~ rneans o~ console commands or overridden by ~ob control lan~ua~e STC-140 ~32 ~
parameters submitted with respect to a particular file. ~s indicated in Fig. 9, the VSS may control the disk frame group assignment to extend over plural disk drlves. This may be done, for example, to allow no two pages of shadow-recorded data to be written to the same disk drive, again for additional reliability.
Having thus defined the system configuration, examples of control and data flow are given in connection with Figs.
10 and 11, respecti~ely. Each Figure comprises on the left 1~ a schematic depiction of the system and on the right shows a flow chart of the steps involved. Numbers showing each of the steps in the flow chart correspond to numbers shown on the block diagram to show the location of performance of the step in the flow chart. The control flow of an "3PEN" message, i.e., an order requiring preparation of a VSS data set for processing, i9 shown in connection with Fig. 10. If the u~er prcgram contains the instruction OPEN, VES, the virtual ~tor~ge sy~tem exscution subsystem, constructs an OPEN
message for transferring over what appears to the host to be a con~ole interfacei that is, the c'hannel adapter appears to the ho~t as iE it were a console device, so that commands which would ordinarily appear on, e.g. a console, are instead used by the channel adapter and by the virtual control processor of the inventivn to take over these data management functions from the host.
If the host's re~uest is for a new data set, the ~CP
wLll assL~n a new virtual volume to this name. ~f the request is ~or an existing data set the virtual control processor will locate the addresses of the various portions of the selected 3~ vLrtual volume. If the open instruction is for a read operation, ~i3;~
the virtual storage system of the invention wlll allocate a number of cache frames to the virtual volume to allow the first few pages of data to be written into the cache for transmission to the host. If the open is instead for a write operation,-one cache frame is allocated to the volume as are a plurality of disk drive frames~ This having been done, a response is generated and transmitted over the virtual control interface to indicate to the host that the ~irtual storage system of the invention is read~ to actually receive the data. The response will comprise a volume serial number, indicating which volume has been identified as associated with the user data set name given, and a data stream identifi-cation, used to identify the channel adapter address to be used for the data transfer. This opPration having been 1~ completned, data flow is possible.
The flow of data in a read operation is shown in F~ . 11. The data stream address which had been passed back to the host i~ then used to identify the channel adapter acldress. This perm;.ts an~ channel adapter which is part of the virtual control proce~sor with which is associated the re~ue3ted virtual volume to be ~sed for the transfer. The virtual control processor then causes the requested data to be read into the cache one page at a time from the disk drive, or from tape if the data has been migra-ted to tape.
The channel adapter decompresses ~he data and writes it .into the host format as required while transferring the data over the interf~ce to the host's buffers.
Fi~. 12 shows a schematic diagram of hardware which could be u~ed to perform the data compression function. Data 3~ i8 shown being brought in at the left side of the diagram and -- 2~ --read into a first register 100. Typically an eight bit byte will be read into the register 100 upon each cycle of operation of the compression circuit. The eight bits are compared with eight bits stored in a second register 102 in a comparator 104. The comparator 104 is shown as an ordinary AND gate; it will be understood that the eight bits ln register 100 are compared with corresponding ones of the eight bits in register 102~ If the comparator 104 outputs a high signal, indicating that the data in register 100 is identical to that in register 102, a flip-flop 106 i5 set and a counter 108 is incremented.
Upon the first setting of the flip~flop 106 the value stored in counter 108 is set to one. At the same time the first setting of the flip-flop 106 causes a so-called "escape character" generator 110 to output a predetermined "escape character" to the output bus of the circuit, indicated generally at 112. If the byte contained in register 100 is then moved to register 102, that in register 100 becomes the repeated byt~, the number of repetitions of which is being counted by the collnter. XE the next byte read .into register 100 is the ~am~ a~ the last, therefore, the flip-flop 106 being already ~et and thc escape character having already been generated means that only the counter 108 will be incremented. The bytes contained in registers 100 and 102 are also compared in a second comparator 114 which is shown as a NAND gate, so that it has a high output whenever the contents of the two registers are not identical. This is passed to the reset input oE flip-flop ]06 which causes the counter contents to be output to the output bus 112 indicative of the number of times the repeated byte was repeated. This n~lmber together 3~ with an example of the repeated byte and the escape character STC-140 ~3~7~
is all that need be provided to perform data compression.
It will be understood, of course, that if the escape character and the number of repeated bytes are equal in size to the byte, it will only be economical to perform such operations upon four or more identical repeated bytes. This object could readily be achieved by using four registers instead of the two shown, 100 and 102; for simplicity they are not shown. Those skilled in the art will also recognize that it is important to provide a method whereby the escape character can be distinguished from the same byte occurring "naturally"
in the data. This may be done by repeating the escape character whereever it appears as natural data and detecting such repeti.tion of this byte during the decompression routine, which is essentially the inverse of the compression methods ju~k descri.bed.
Those skilled in the art will recognize that there has been described a virtual storage system adapted for connection between a host computér and a plurality of magnetic storage media wh:lch provides substanti~l improvements in systems operation characterl~tics. By providing the memory control intelligence qxternal to the host, both the host and the user are relieved .~rom the chore of de~termining where on magnetic storage media inclividual records are to be stored. Moreover, utilization of this external intelligence together with the concept of the vi.rtual volume of undefined size allows data to be stored in block sizes determined by the memory system at the time of stora~e, rather than by the user in advance of actual data wri-t:ing. ~hi.s allows much higher utilization of data ~torage capacit~. Still another advantage provided by the invention 3~ 1~ that it may be operated to store individual pa~es of data _~'C-140 ~3~
in frames within a da~a cache thus providing the advantage of solid state access time is the host, without storing all data in this very expensive form of memory. An additional advantage provided by the utilization of external intelligence in a memory system is great versatility in the recording modes used with respect to given types of data records, particularly as regards duplication for improvement of reLiability. For example, provision of memory for storage of addresses at which data is stored external to the host allows back up or duplicative storage of this da~a on different media if desired without intervention by the host or by the user. Furthermore, only those portions of records which are changed during pro-cessing during a given operating cycle, such as an 8-hour shift, need be backed up, in contravention o~ the prior art practice wherein if any part of a file was altered, the entire file -ha~l to be copied. ~nother advantage is that such backup ma~
he perEormed to tape such that entire reels of tape are wri-tten wLth the active portions of diffqring ~iles; this yields complete utlllzat:ion oE the storage space available on tape.
2~ Provi~ion O.e intelligence in the memory subsystem al~o enables additional versatility in error correction and recovery operations. If an error occurs in reading or writing of a tape, for example, from the long-term storage into ~he cache, the system of the invention can automatically retrieve the bac]c up or shadow recorded copy without specific instruc-tions from the host or operator. Similarly, simultaneous wrLtincJ o one data fl:Le to two locat~ons on disk may be per-~ormec1 :Ln the ~ystem according to the invention with respect to all records written without operator or host intervention, 3~ the disk frame groups having been defined. Finally, it will 'C-140 be appreciated that all the above improvements made in data processing systems by incorporation of the concepts of the invention are achieved at no cost to the host and may indeed be accompanied in some cases by an improvement in overall system performance as well as improvements in the efficiency of storage usage with concomitant increase in either storage ~vailability or reduction in the system size required to perform a specified function.
Therefore, while a preferred embodiment in the invention has been disclosed and described above, it will be understood by those skilled in the art that there are numerous modifications and improvements which can be made to the _-system of the invention ithout departing from its essential ' spiri~ an~ scope which is therefore to be limited only by the followiny claims.
- 2~ -
" ~L3L~327~
The Whi-te patent also disclo~es that it might be useful to include data ccmpression and decornpression mearls in the vi.rtual storage system of the invent;,on. Data compression is a concept which had been well known for use in host CPIl's but had not previously been done external to the host. This distinction is subsumed under the fact that this was the firs-t system external to a host to provide intelligent processor means for control of data storage functions.
The present invention is an improvement over the original invention discussed above. According to the presen-t invention, the virtual storage system is not simply interposed bet~een a host computer and disk drives. Instead, the virtual storage system of the present invention operates in conjunction with modi.fications to the operating system of the host computer.
T~pically this w.ill involve some slight reprogramming of the host. Furthermore, the virtual storage system according to the present invention is not constrained to cause disk Arives to functionally mimic tape drives or other defined storage ~lev.ic~s, asj .in the parent inventloll, but .instead presents to ~0 khe~,'hos-t the irnac3e of a tabula_rasa; i.e., a blank shee-t upon which the 'host, and thus -thc user, can wri.te wi-thout con~tra:i.nt. a3 to the forrrlat or disposal of the data.
ects of the Invention Accordingly, it is an object of the invention to ~5 provide an improved data processing and storage system.
A further object of t.he invention is to provide a data processinc3 and storage system in which decisions as -to the loca-t:i.on for storage of data are rnade external to the ari.thrnet:ic and logic unit, in a memory storage system having ~(~ inke'lligence.
... .
STC-140 ~3~
A further object of the invention is to provide a memory system having decision-making capacity with respect to storage locations for data, which additionally comprises means for dividing user-defined files into sub-blocks of data for storage in available locations on memory means.
~ further object of the invention is to provide methods and means for more efficient data storage.
Yet a further object of the invention is to provide a memory storage system in which the allocation of storage ].ocations to given data is made external to a host computer, in the memory system itself, whereby data compression and decompression can be performed upon the data in the memory system and not in the host computer.
Still a further object of the invention is to provide a memory system for use in conjunction with a host digital computer in which duplication of data is performed by the memory system in accordance with usage, exclusive of the control of the host computer.
A Einal object oE the invention is to provide methods or more e~Eicient use of data storage apparatus.
Other aspects and objects of the invention will appear to those skil:Led in the art.
~ummary of the Invention I'he invention fulfills the above mentioned needs of the art and objects of the invention by its provision of a virtual storage system which comprises a processor for allocating spaces on data storage means to blocks of data supplied by a host computer. The processor, after having allocated addresses on, ~.g., disk memory means to the data, causes the data to 3~ ~low from a host interface means, which may typically comprise ~ ~3~7~
a data cache for storage of a given portion of data until appropriate locations can be determined, -through disk interface means and onto -the disks. The addresses are then stored by, for example, user file name. When the host later calls for the same file, the virtual storage system's controller refers to its file location memory and determines where on the various disks -the data is stored. The data can then be transmitted back to the host via the cache for further processing. In this way no space on any given disk is explicitly or permanently assigned -to any given user file. Instead, space on disk is taken up only as needed by active files. In this way the address memory of the controller at all times "knows" what areas of the disks are available for further storage. Similarly, it "knows" which areas of the disks have been recently operated upon l~ and therefore can be used to control efficient duplication of data fo~ backup, i.e., for use in case of loss of the main data file on a primary memory means. In addition to the controller, in a preferred embodiment the virtual storage system of the :LnventLon comprises at least one channel adapter for connecting 2~ the host computer to the cache which may additionally comprise data corrlpre,ssion and decompression means so as to achieve further e~:Lc:Lency of storage.
The present invention is directed to apparatus for the storage of digital data organized into records. The apparatus ?~5 includes host interface means for receiving data from a host computer, cache means connected to -the host interface rneans for temporary storage of the data, s-torage device interface means for ~onnecting the cache means to one or more memory storage devices ~3~7~
o~ plural classes of storage devices for comparatively long-term storage of the da~a, and controller means. The controller means comprises address recall means, and (a) determines where the data is to be stored on the storage devices, (b) controls the flow of the data from -the host interface means through the cache means and the storage device interface means onto the storage devices, and (c) stores the addresses a-t which the data has been stored on the storage devices. As a result, a given user-defined data file may be stored on a plurality of storage devices, selec-ted from plural classes of storage devices, by the controller means.
The invention is also directed to a data processing and storage system, including at least one host computer for performing arithmetic and logic operations on digital data comprising an extremely high speed, relatively low capacity memory means, and interface means for transmitting and receiving data ~rom external memory means, and a virtual storage system.
The virtual storage system comprises relatively low speed, high capaclty storage devices of plural classes of storage devices, ho~s-t inter~`ace means for receiv:ing data from and -transmitting 2~ da-ta to th~ hos-t, a high speed, relatively low capacity cache memory means connec-ted -to -the host :i.nterface rneans, and a controller for allocat:ing appropriate storage locations on the relatively low speed, high capaci-ty storage devices for data received from the host and stored in the cache, for accordingly con-trolling the flow of data from the cache -to -the relatively low speed, high capacity storage devices selected from plural classes of ~torage devices by the con-troller, and for storing the s-torage loc~lklo~ls . . ~ .
~ ~3~7~
The invention is further directed to a data storage and recall system, comprising means for receiving data from means for processing data, means for long-term storage of the data including s-torage devices selected from plural classes of storage devices, means for temporary storage of the data, means for maintaining a directory of available storage loca-tions on the long-term storage means, means for subdividing a user da-ta set to subportions sized to fit the available storage locations, and means for causing the subportions to be stored in the available storage ].ocations and for maintaining a directory of the locations at which the subpor-ti.ons ha-ve been stored, means for controlling flow of the data through the system, memory means -for reca].l of the l.ocations for long--term storage of the da-ta, and means :~or outputting the data upon instruction.
1~
Bri,ef Descrip-tion of the Drawings The inven-tion will be bet-ter understood if reference lc1rnad~ to the accompanylng drawings in which:
~ ig. I shows the prior art method of storing data 2~ on magnet.i.c d:Lsks ancl on magne-tic tapes;
~ i.g. ~ shows the comparable method according to the :Lnvention;
- 7b -~3~
Fig. 3 shows an overall view of the virtual storage system of the invention as connected to host computers, plural disk drives and tape drives;
Yig. 4 shows a more detailed view of the virtual storage system of the invention as connected to a host computer:
Fig. 5 shows the hardware comprising the con~rol processor of the virtual storge system of the invention;
Fig. 6 shows a schematic view of the software components of the virtual storage system of the invention;
Fig. 7 shows the host resident components of the virtual storage system of the invention;
Fig. 8 shows a more detailed block diagram of the virtual storage system of the invention and the channel adapter which serves as the host interface unit;
lS Fig. 9 shows how various disk Erame groups can be organized on a plurality of disks according to the user specifications;
Fig. 10 shows the flow of control signals through the vlrtual ~torage system of the invention;
.~ F'ig. 11 shows a comparable view oE the flow of data through the system O e the inverrtion; and Fig. 12 shows a schematic diagram of hardware which might be used for data compression and decompression in the channel adapter of the system according to the invention.
Descri~on of the Preferred Embodiments As discussed above, the prior art practice when storing a particular file of data on, for example, a disk drive w~s to alLocate a portion of this drive permanently to that Eile alld to write into that space as needed. As files Erequerltly grow in size over time, the practice would ordinarily ~ ~ !
STC- 140 `-~3~
be to allocate a large enough file size on disk that it would not be filled up for some time thereafter despite continuing use. The usual result is that the space allocated for the particular file is not filled thereby and accordingly the disk is not used to full efficiency. This practice is shown schematically in Fig. 1 in which a plurality of disk drives are shown schematically at 1, 2 and 3. The three disks 1-3 are shown divided into allocated areas as a-t A, B, C,...H; for example, drive 2 is allotted to a file denoted E. In each, the axea shown cross-hatched is tha~ which might be occupied by data. The remaining region is therefore unused.
File E is shown as comprising two areas E1, E2. This is mearlt to indicate that the file might grow from El to E2 over a period of, say, several months and that in accordance with the prior art practice, the later added material E2 would be added to the same allocated area on the same disk clr:ive 2 as El previously written thereto.
An additional prior art practice was to store more ~requently used data on disk and less Erequently used data on tape, i.nasmuch as the access tirne of tape is typically greater than that of disk while the cost of tape s,torage is less than that of disk. An additional use oE tape is to "backup" the disk drive; that is, to have the same data recorded both on disk and also on tape so that in the event of damage to the disk or the like the data would be preserved on the tape and the information not lost. Accordingly three tape ~le~ are shown at K, L and M, on reels of tape 4, 5, and 6, ~chcmcltically showing this practice. It will be no~ed that ~ sin~le data fi.le is present on each tape. This is a typical prlor art practice, as it is yenerally deemed simpler to g _ fetch a reel of tape from storage when a new file is sought than to unwind a reel of tape through a first file in order to reach a second file. This, of course, leads to substantial underutilization of the total tape available.
Thus, in the prior art, when the host seeks access to one of the previously assigned user files shown in Fig. 1, typically the host will convert the user's request for a named file into a command to a disk controller to write the contents of the file into the host interface comprised in the disk controller for passage onto the host. Thus the host is required to know where and on what type of memory means the particular file sought for is stored. Where, for example, the sought-for f.ile is stored on a demountable disk or on a tape, the host is required to convert the operator's command into a "mount disk" or "mount tape" instruction typically appearing on an operator console screen and the c~erator is then required to fetch the disk or tape called ~or ~rom an archive. Thls practice while workable is tirne-collsuming and cuts down on the effective computing capacity 2~ o;~ the host com~uter.
Fig. 2 shows in comparison the method by which data ~ 9 are stored according to the present invention. Instead o~ allocating specific areas of specific dis'ks to specific user files, space on disk or tape is assigned only as needed, as the data arrives in the virtual storage system of the invention Eor storage. By allocating storage space only as rec~u:ired, additional area is made available for the storage oE data. Moreover, the space assigned to a given file is not necessarily, or even desirably, on the same disk or tape drive~, but instead may be spread over several so that the f ~
3~
average utilization of each is more or less equal. The fact that it is the external memory system and not the host which decides where and on what sort of storage media given data is to be stored provides considerable additional versatility in operation as well as improved host utilization and simplification of user requirements.
Thus in Fig. 2 the virtual storage system of the invention is shown comprising a plurality of disX drives 10, 11 and 12, each of which has given portions of various records disposed thereon. It will be noted that records El and E2 which might be in fact different portions of the same user file are not together on the disk drives 10 through 12. Tape drives 13 and 14 are additionally provided. These may be used for bacXup of data sought to be duplicated in case of loss of the data on the disk, or may simply be data stored in files which have been inactive for longer than a given minimum periocl and have therefore been copied to the slower access but ~ar le~ls expensive tape drive media. Finally, the virtual ~torage system comprises a third memory 15 which is an address memory stor~ for use in recalling the places on disk and/or tape at which a given file is stored. Upon the host's calling ~ox a specific file, the mernory store 15 is accessed and caused by the processor to write the data from the disk or tape drive to the ho~t. The address store 15 may desirably be a RAM of comparatively limited capacity. The virtual storage ~y~tem of the invention may also comprise a high speed cache m~mory (not shown) such as another RAM of relatively larger c~pacity, for use in assembling all or a sub~portion of a given ile prior to transmission to the host, so that the ~0 ho~t n~ed not receive the data file piecemeal from the virtual STC-140 ~3~
storage system according to the invention.
Fig. 3 shows the overall layout of a data processing and storage system utilizing virtual storage according to the invention. The particular system shown in Fig. 3 shows a pair of host computers 20 and 21 each connected to a pair of virtual control processors 22 and 23, which amount to the heart of the virtual storage system of the invention. These are each in turn connected to a data base, which in the con-~iguration shown each comprise a pair o~ disk controller~ units 24 and 25 each operating a pair of disk drives 26, and 27, 28 and 29, respectively, and to a pair of tape controllers 30 and 31 having attached thereto tape drives 32 and 33, respec-tively. The virtual control processors may each also have secondary connections to the other's data base, for backup purposes. Accordingly when a host requests a specific user file or "virtual volume" the request need only specify which of the two virtual control processors 22 or 23 controls the ~at~ base within which is stored that virtual volume. The virtual control processors 22 and 23 are each able, using ~0 th~ir internal address store, to convert the name oE the file in~o the location of the data on the disk or tape unit(s) involved and forward it to the host without ~urther instruction from the host.
It is well known by those skilled in the art that there iB at present available an IBM Corporation computer ~ystem which uses disk drives external to a host computer ~or expansi.on of the CPU memory space. This is operated ~ccordlng ~o a program contained in the host known as MVS, which is an acronym for "Multiple Virtual Storages". The present 3~ Lnvention similarly relates to what might be termed "virtual"
, S~C-l~0 ~ ~ ~3~7~
storage, but is concerned with long-term storage, not expansion of the CPU storage.
Fig. 4 shows a schematic view of how a virtual st.orage system accoxding to the invention could be operated in conjunction with an IBM 370/158 or 30XX CPU having ~he MVS ~eature. The invention in this preferred embodiment would also function correctly with any other host having MVS capability, such as the Amdahl V-8 host CPU. The host computer 30 contains soEtware for various purposes denoted on the drawing such as user address space, means for accepting jobs from punch cards and orders from a console, and the like. Adap-tation of such a host operating system to be useful with the virtual storage system of the invention requires some reprograrnming of the host computer as will be discussed in further detail b~low. In essence, an additional program referred to as VES, for "Virtual Storage System Execution Subsystem`' is added as is additional. VSS host utility software to generate reports rclat.ing to the operation of the virtual storage system of ~hc .invellt:ion so as to keep the user apprisea of its status.
~0 rrhe host 30 is connected by a bus 32 to the virtual control p~cc~or 34 (VcP) of the invent.ion whic'h in turn is connected Vi~ disk controllers (not shown) by additional busses to one or more disk storage devices 38. The bus 32 leading from , the host 30 to the virtual control processor 34 is connected to a chanrlel adapter (CA) unit 40 which connects the host channel to the memory bus so as to transfer data from the host 30 to th~ d.isk drives 3S as well as Eor performance of additional funct:ion~, as will be discussed in further detail below. The nctual. opcration of the virtual co~trol processor is ~Inder 3~ t,he control o a Virtual Storage System Operating System * Trad~ Mark - 13 -3~27~
(VOS) which receives op~rator input via a VCP console 42;
additional software is conta;ned in the virtual control processor for storage management and reporting functions as noted.
Fig. 5 shows the hardware comprising the virtual con-trol processor (VCP) 34 in broad outline. As shown, the console ~2 is connected to the VCP 34 through a conventiona] channel.
Other conventional channels, numbered CHl-CH5 and indicated generally at 48, are used to interface with backend storage such as disk drives and tape drives. The VCP 34 also comprises one or more channel adapters 50 for interfacing with a like number of host processors (not shown). The channel adapters 50 perform all required interfacing activities with a host.
To thc host the channel appears to be a storage device control unit which responds to chanrlel protocol in a conventional manner. Fach channel adapter 50 responds to two addresses in the preferred embodiment, one for data transfer and one for exch~ange of control information. During data transer, the challnel adapter perEorrns cornpression and decornpression on 2~ llser data. D~tail~ of the channel ~dapters are to be Eound in ~o-p~n(lirlcJ Canadiarl appl:iclatioll Serlal No. ~02,465 f:iled on 7 May, .l982 (At~orney's Docket No. SrrC-142).
The VCP 34 additionally comprises a cache buffer memory 52. This may be conventional solid state random access memory (F~M). The cache buffer 52 is a portion of the VCP rnain mcmory which is used to perform storage management functions on user data. Blocks of user formatted data are ~ssembled in~o "pa~3es" o~ convenient size Eor storage on the backend ~lcvlce chc~sen. For eY~arnple, in a typical disk dri~e, the paqe w~uld be equal t~ one disk track. The cache is accordingly ~32'7~
divided into a plurality of "frames" into each of which one page can f.it. When a frame is full with a page of data it is written to the appropriate backend storage device. A
plurality of frames may be provided in the cache so that simultaneously hosts can be reading and writing to the cache buffer and so that there need be no delay in host operations caused by the operation of the backend storage device chosen.
The final component of the VCP 34 is a processor unit 54 which in a presently preferred embodiment may comprise a Magnuson M80 computer under the control of which the above operations are affected. This processor unit 54 comprises memory means for storing the addresses c>f available space on the baclcend storage devices used. Those addresses in which cla-ta is already stored are cross-referenced in this memory '15 toge-ther with means for identiEying the specific records ~tored at the addresses. In this way, when the host calls Eor a 3peci~ic file the VCP processor 54 can consult this memo~y ~nl determine where in backend'storage the data is 3tored. IE the part.icular storage medi.um is mounted on a ~n cli~'k or tape dr$ve th~ ~irst page oE data can be then read direct].y into t'he cache buffer 52. If, for example, the file called for is stored on a reel of tape not presently mounted, t'he processor can cause an operator instruction to appear on the VCP console 42 or on a host console, to e.g., mount the appropriate reel of tape, whereupon the data can be read into the cache 52 a page at a ti.me for forwarding to the host.
It will be appreciated by those skilled in the art th~t t'he concept o~ using the cache 52 to divide user formatted data which may be of any block size and o:E any total length - 15 ~
~/ ) C--14 0 ~ r7 ~
into pages for storage of size convenient for storage on predetermined backend storage devices, thus treating each page separately from the standpoint of allocating storage space is new to the artO The host thus does not enter into the process of data management and control, to the exten-t that the virtual control processor even reformats the data. By comparison, the prior art practice was to permit the user to define the file to be the maximum length envisioned for future use and to select the type and location of the magnetic storage medium on which the file was to be permanently kept.
Not only is this practice inefficient in that almost inevitably the file would be made bigger than necessary, but it requires that the user (or, in more advanced systems, the host) make decisions as to the storage utilization~ By comparison, use of the virtual storage system of the invention means that each and every file can be divided into blocks of a size convenLen~. for ~torage from the storage system's point of view, thus ensuring Ear higher efficiency of storage. More ov~r, the ~act that the storage location is decided upon by 2~ the ~torage ~ystem relieves both user and host of this task.
In certain embodiments of the present invention even the choice o~ the type of magnetic storage medium on which a given block of data is stored is made by the virtual storage system of the invention, based on usage history of the file.
Fig. 6 shows a block diagram of the software comprised by the virtual storage system of the invention. Again, the VCP con.sole 4~ i.s shown as it has, of course, input to the operatiorl of the system/ as cloes ~he channel adapter. The VCP proce~sor (54 in Fig. 5) operates on the VSS Operating ~ 16 -System (VOS), and on functional software which performs storage management and backup/recovery ,unctions. VOS performs typical operating system functions for the VCP 34, includiny I/0 services, main storage allocation and console communication.
Functional software performs the following storage management functions: control of host/VSS data flow, backend device management, disk drive space management; recovery management, and diagnostic/service reporting. Additionally, the functional s`oftware is used to control backup writing of records stored on disk to tape as well as migration of infrequently used files on disk to tape so as to free more disk space. Further-more, the virtual storage system of the present invention comprises means for adaptively changing the class of storage of a ~iven file if it turns out that the file is used more or less frequently than originally thought likely. It will be understood, of course, that these functions may be performed in interaction with the operator by means o~ the console 42.
As discussed above, the ut:ility of the virt~al storage ~ystem o~ the invention i.s maximized if certain judicious mod:ifications are made to the operating system software of the ho~t, largely so as to delete the host's memory control functions to the extent necessary to allow the VSS to perforrn its memory control function to the fullest. ~he modifications required to the IBM 370/158 MVS program are shown schematically in Fig. 7. Those skilled in the art will readily recognize the various components of the MVS program. The changes involve the VSS Execution Sub-system (VES) and the VSS utility ~unc-tion. The VSS execution sub-system, VES, is the VSS address ~pace which functions as a secondary sub~^sys-tem in the host 3() MVS environment. V~S provides an interface between user jobs ~ ~ ~327~
running on the host and VSS itself. Major functions performed by this VES include initialization of jobs, job management, data management, interface with the console and interface with the job control lanauage cards used to detail a particular job to the host. In order to support VES, VSS maintains a number of table entries and queues in the common area of the MVS address space.
The host resident VSS utility functions provide system support and management personnel with the tools requisite to install, manage and maintain VSS. These functions provide reports on VSS status and performance and provide assistance in data and system recovery. Typically the utilities can include a report formatter to produce VSS formatted report output, user exits to interface with existing security and accounting systems and to monitor activity, transaction enable and disable capability, and means to permit control of VSS from the host computer console. Those skilled in the art will readily unc1~r~tand how MVS may be modified to yield these results.
Fig. 8 ~hows a more detailed block diagram oE the virtual storage ~ystem of the invention. As noted, its chief ~mponent~ are the channel adapter 60 indicated by dotted outline, a sy~tem bus 62 and the VSS processor 64 also indicated by a dotted outline. In a preferred embodiment the system processor 64 comprises what might in other circumstances be ~5 considered an independent computer itself; in a presently preferred embodiment, a Magnuson Co. Model M80t40 is used.
~s noted ahove, this processor contains a storage control unlt 66 connected to a memory 68; that is, a cache memory ~r tem~orary storage of data while storage assignment is b~ing completed by the VSS central processor 70, which also STC-1~0 controls the operation of char)nels 72 ~hich write the dat~
to disk or tape storage media as well known in the prior ar~. The system bus 62 connects the processor 70, the cache 68 and the channels 72 together to the channel adapter 60.
The function of the channel adapter 60 is to mate the input/output host interface 74 with the system bus inter-face 76. In the interim between the two, various operations may be performed on the data under the control of a channel adapter processor 7~, which may chiefly comprise a micro-processor, such as a Zilog Co. Z~000 unit. In the presently preferred configuration shown in Fig. 8 the channel adapter processor 78 operates on operation tags and data in parallel.
Perhaps t'he most important single function performed .in t'he channel adapter in a presently preferred embodiment is lS that of data cornpression upon writing and decompression upon readiny of data to or from backend storage, respectively. It will be unc1erstood by those skilled in tlle art that data com-pLa~ion can :in rnany cases signiEicantly reduce the ac-tual amoun~ Oe data which is required to be stored thus ylelding .0 further storage eE:iciency. It will also be appreciated by khose ski'Lled :in the art that data cornpression has been per-formecl in the prior art but only in a host environment and not by an externally operated memory system having intelligence of its own. Details of the preferred ernbodiment of the channel adapter as mentioned above are discussed in a co-pending application referred to above.
~s noted above, in the prior art -typical large cl~ta process:incJ arld stora~Je systems (i.e., ones havincJ the com;~utinc~ power oE at least one IBM 370/15~ c:Lass processor) I' ~3;;~7~
would typically employ a complete heirarchy of storage devices.
Large sequentially organized data sets would be stored on tape.
Smaller and more active sequential data sets would be stored on disk thus allowing on-line availability. Moving the small and/or active data sets to disk offers the user the advantage of fewer tape drives required and improvec~ system through-put.
However, additional requirements for management of these disk resident data sets arise. Data set placement and backup for reliability and migration become critical and time consuming tasks. The present invention addresses and solves these problems by automating routine storage management functions and by providing migration and backup capabilities which are independent of the host processor. The system of the invention thus provi.des data availability comparable to permanently mounted direct access devices such as disk drives, with access times of semiconductor storage by virtue of provision of a cache memory, and ease of use and control to the programmer of a device-lndependent data set.
~his may be termed a "virtual storage system" (VSS) ln ~hat this langua~e suggests a certain independence from p'hysical constraints and limitation. As virtual storage operating systems (such as the IBM MVS system referred to above ) freed programmers'from the constraints imposed by relatively small areas of real CPU storage, VSS frees the ~5 user from the limitatlons imposed by storage device require-ments. By automatically performing space management functions and ~torage allocation, VSS offers the user true storage devicc independence. In a presently preferred embodiment, t11ese goals are achieved by storing the user data in units called vlrtual volumes. Each virtual volume contains one ~327~
user data 82t and can be whatever size is necessary to contain that data set.
Those skilled in the art w~ll recognize that in the prior art conventional storage volumes were stored on tape or di2k drives of a fixed ~torage capacity determined by the ~vice characteristics. A larg~ data ~et might thus be stored on two tape drives. Conversely *ive different data ~et~ migh~ re~ide on a single disk volume. The vir~ual ~torage 8ystem of the invention minimizes storage management problems by introducing the conc~pt of variable siYe volumes.
Once a u~er creates a variable ~olume for a particular data ~et, it may be expanded or contracted to whatever size i8 nece~sary to contain the actual data. ~he aliocation of ~pace on whatever storaye device is chosen is performed ~utomatically. Moreover, the virtual storage system processor, maintain~ a table which Xeeps track of the volume activi~y and of the location of each of the portions of each volume.
Each volume is compri6ed of an integral number o~
"page~". From the u~er's viewpoint, each volume consists of a 2~ number o data block~ of whatever block size is convenient ~o the user. qhe virtual ~torage system of the ~nvention ~sembles the data block~ into pages conveniently ~ized for it8 own pUrpOB~. In the presently preferred embodiment each page i8 equivalent in size to one disk track. Thus a virtual volume ~onsist~ of a number of full track ~ize pages in ~arious locations ~pread throughout disk storage. The track-sized storage areas in which the pagefi ar~ 6tored are re~erred to as "frames" in both ~he cache ~nu in ~he di~k drive . ~le analoyy may be made that ~he frame i~ the bucke~
~nd ~he pag~ i~ the water. For each page of data tha~ is ~ 21 -STC-14~ 3~
transferred to on-line storage during a write operation a frame must be allocated both in cache and in disk storage.
Plural frames are provided in the cache to allow simul~aneous reading and writing.
A further subdivision of the storage system according to the presently preferred embodiment is shown in Fig. 9.
There groups of disk frames are organized into "disk frame groups" for storage management purposes. Certain attributes of control can be assigned to each frame of a given group.
Each virtual volume will be allocated frames from only one disk frame group. Thi.s enables these attributes to be assigned to each virtual volume so as to readily perform the various management functions of the storage system o~ the invention on the appropriate types of files. For example, one might put a maximum size limit on each virtual volume within a gi~en frame group. One might care to employ appropriate rec~rding modes, for example, shadow recording, in which the data i3 simultaneously written to two different disk tracks eor rel.tabllit~ purposes, or da-ta journaling, in wh;ch data wrltt~n ~imultaneously to a disk track and to a tape clrive, again or reliability improvement, on each virtual volume in a given frame group. Another possibility which can be controlled in this manner is data rnigration; that is, writing all the data in a given volume to tape when a certain percenta~e of the disk frarne group has been Eilled. For example, the data migration threshold--that poin-t at which the di~k ~rame group is deemed full--might be so controlled. These ~ k ~rame group attr:ibutes are typically established at time o~ :ln~tallation oE the VSS sys-tem but might be altered by 3~ rneans o~ console commands or overridden by ~ob control lan~ua~e STC-140 ~32 ~
parameters submitted with respect to a particular file. ~s indicated in Fig. 9, the VSS may control the disk frame group assignment to extend over plural disk drlves. This may be done, for example, to allow no two pages of shadow-recorded data to be written to the same disk drive, again for additional reliability.
Having thus defined the system configuration, examples of control and data flow are given in connection with Figs.
10 and 11, respecti~ely. Each Figure comprises on the left 1~ a schematic depiction of the system and on the right shows a flow chart of the steps involved. Numbers showing each of the steps in the flow chart correspond to numbers shown on the block diagram to show the location of performance of the step in the flow chart. The control flow of an "3PEN" message, i.e., an order requiring preparation of a VSS data set for processing, i9 shown in connection with Fig. 10. If the u~er prcgram contains the instruction OPEN, VES, the virtual ~tor~ge sy~tem exscution subsystem, constructs an OPEN
message for transferring over what appears to the host to be a con~ole interfacei that is, the c'hannel adapter appears to the ho~t as iE it were a console device, so that commands which would ordinarily appear on, e.g. a console, are instead used by the channel adapter and by the virtual control processor of the inventivn to take over these data management functions from the host.
If the host's re~uest is for a new data set, the ~CP
wLll assL~n a new virtual volume to this name. ~f the request is ~or an existing data set the virtual control processor will locate the addresses of the various portions of the selected 3~ vLrtual volume. If the open instruction is for a read operation, ~i3;~
the virtual storage system of the invention wlll allocate a number of cache frames to the virtual volume to allow the first few pages of data to be written into the cache for transmission to the host. If the open is instead for a write operation,-one cache frame is allocated to the volume as are a plurality of disk drive frames~ This having been done, a response is generated and transmitted over the virtual control interface to indicate to the host that the ~irtual storage system of the invention is read~ to actually receive the data. The response will comprise a volume serial number, indicating which volume has been identified as associated with the user data set name given, and a data stream identifi-cation, used to identify the channel adapter address to be used for the data transfer. This opPration having been 1~ completned, data flow is possible.
The flow of data in a read operation is shown in F~ . 11. The data stream address which had been passed back to the host i~ then used to identify the channel adapter acldress. This perm;.ts an~ channel adapter which is part of the virtual control proce~sor with which is associated the re~ue3ted virtual volume to be ~sed for the transfer. The virtual control processor then causes the requested data to be read into the cache one page at a time from the disk drive, or from tape if the data has been migra-ted to tape.
The channel adapter decompresses ~he data and writes it .into the host format as required while transferring the data over the interf~ce to the host's buffers.
Fi~. 12 shows a schematic diagram of hardware which could be u~ed to perform the data compression function. Data 3~ i8 shown being brought in at the left side of the diagram and -- 2~ --read into a first register 100. Typically an eight bit byte will be read into the register 100 upon each cycle of operation of the compression circuit. The eight bits are compared with eight bits stored in a second register 102 in a comparator 104. The comparator 104 is shown as an ordinary AND gate; it will be understood that the eight bits ln register 100 are compared with corresponding ones of the eight bits in register 102~ If the comparator 104 outputs a high signal, indicating that the data in register 100 is identical to that in register 102, a flip-flop 106 i5 set and a counter 108 is incremented.
Upon the first setting of the flip~flop 106 the value stored in counter 108 is set to one. At the same time the first setting of the flip-flop 106 causes a so-called "escape character" generator 110 to output a predetermined "escape character" to the output bus of the circuit, indicated generally at 112. If the byte contained in register 100 is then moved to register 102, that in register 100 becomes the repeated byt~, the number of repetitions of which is being counted by the collnter. XE the next byte read .into register 100 is the ~am~ a~ the last, therefore, the flip-flop 106 being already ~et and thc escape character having already been generated means that only the counter 108 will be incremented. The bytes contained in registers 100 and 102 are also compared in a second comparator 114 which is shown as a NAND gate, so that it has a high output whenever the contents of the two registers are not identical. This is passed to the reset input oE flip-flop ]06 which causes the counter contents to be output to the output bus 112 indicative of the number of times the repeated byte was repeated. This n~lmber together 3~ with an example of the repeated byte and the escape character STC-140 ~3~7~
is all that need be provided to perform data compression.
It will be understood, of course, that if the escape character and the number of repeated bytes are equal in size to the byte, it will only be economical to perform such operations upon four or more identical repeated bytes. This object could readily be achieved by using four registers instead of the two shown, 100 and 102; for simplicity they are not shown. Those skilled in the art will also recognize that it is important to provide a method whereby the escape character can be distinguished from the same byte occurring "naturally"
in the data. This may be done by repeating the escape character whereever it appears as natural data and detecting such repeti.tion of this byte during the decompression routine, which is essentially the inverse of the compression methods ju~k descri.bed.
Those skilled in the art will recognize that there has been described a virtual storage system adapted for connection between a host computér and a plurality of magnetic storage media wh:lch provides substanti~l improvements in systems operation characterl~tics. By providing the memory control intelligence qxternal to the host, both the host and the user are relieved .~rom the chore of de~termining where on magnetic storage media inclividual records are to be stored. Moreover, utilization of this external intelligence together with the concept of the vi.rtual volume of undefined size allows data to be stored in block sizes determined by the memory system at the time of stora~e, rather than by the user in advance of actual data wri-t:ing. ~hi.s allows much higher utilization of data ~torage capacit~. Still another advantage provided by the invention 3~ 1~ that it may be operated to store individual pa~es of data _~'C-140 ~3~
in frames within a da~a cache thus providing the advantage of solid state access time is the host, without storing all data in this very expensive form of memory. An additional advantage provided by the utilization of external intelligence in a memory system is great versatility in the recording modes used with respect to given types of data records, particularly as regards duplication for improvement of reLiability. For example, provision of memory for storage of addresses at which data is stored external to the host allows back up or duplicative storage of this da~a on different media if desired without intervention by the host or by the user. Furthermore, only those portions of records which are changed during pro-cessing during a given operating cycle, such as an 8-hour shift, need be backed up, in contravention o~ the prior art practice wherein if any part of a file was altered, the entire file -ha~l to be copied. ~nother advantage is that such backup ma~
he perEormed to tape such that entire reels of tape are wri-tten wLth the active portions of diffqring ~iles; this yields complete utlllzat:ion oE the storage space available on tape.
2~ Provi~ion O.e intelligence in the memory subsystem al~o enables additional versatility in error correction and recovery operations. If an error occurs in reading or writing of a tape, for example, from the long-term storage into ~he cache, the system of the invention can automatically retrieve the bac]c up or shadow recorded copy without specific instruc-tions from the host or operator. Similarly, simultaneous wrLtincJ o one data fl:Le to two locat~ons on disk may be per-~ormec1 :Ln the ~ystem according to the invention with respect to all records written without operator or host intervention, 3~ the disk frame groups having been defined. Finally, it will 'C-140 be appreciated that all the above improvements made in data processing systems by incorporation of the concepts of the invention are achieved at no cost to the host and may indeed be accompanied in some cases by an improvement in overall system performance as well as improvements in the efficiency of storage usage with concomitant increase in either storage ~vailability or reduction in the system size required to perform a specified function.
Therefore, while a preferred embodiment in the invention has been disclosed and described above, it will be understood by those skilled in the art that there are numerous modifications and improvements which can be made to the _-system of the invention ithout departing from its essential ' spiri~ an~ scope which is therefore to be limited only by the followiny claims.
- 2~ -
Claims (10)
1. Apparatus for the storage of digital data organized into records comprising:
host interface means for receiving data from a host computer;
cache means connected to said host interface means for temporary storage of said data;
storage device interface means for connecting said cache means to one or more memory storage devices of plu-ral classes of storage devices for comparatively long-term storage of said data; and controller means, comprising address recall means, for:
(a) determining where said data is to be stored on said storage devices;
(b) controlling flow of said data from said host interface means through said cache means and said storage device interface means onto said storage devices; and (c) storing the addresses at which said data has been stored on said storage devices;
whereby a given user-defined data file may be stored on a plurality of storage devices, selected from plural classes of storage devices, by said controller means.
host interface means for receiving data from a host computer;
cache means connected to said host interface means for temporary storage of said data;
storage device interface means for connecting said cache means to one or more memory storage devices of plu-ral classes of storage devices for comparatively long-term storage of said data; and controller means, comprising address recall means, for:
(a) determining where said data is to be stored on said storage devices;
(b) controlling flow of said data from said host interface means through said cache means and said storage device interface means onto said storage devices; and (c) storing the addresses at which said data has been stored on said storage devices;
whereby a given user-defined data file may be stored on a plurality of storage devices, selected from plural classes of storage devices, by said controller means.
2. The apparatus of claim 1 wherein each said data file is divided into records of length convenient for storage on devices of said classes by said controller, which operates individually on each of said records to determine where and on what class s of storage devices said record shall be stored.
3. The apparatus of claim 1 wherein said host interface means comprises channel adapter means for compression of said data by replacing successive identical bytes of data in a given record by a compressed data word comprised of one of said identical bytes together with a character indicative of the number of times said identical byte was repeated in the uncompressed record.
4. The apparatus of claim 3 wherein said compressed data word additionally comprises a character for inditification of said compressed data word as such.
5. A data processing and storage system, comprising:
at least one host computer for performing arithmetic and logic operations on digital data, comprising an extremely high speed, relatively low capacity memory means, and interface means for transmitting and receiving data from external memory means; and a virtual storage system, comprising:
relatively low speed, high capacity storage devices of plural classes of storage devices;
host interface means for receiving data from and transmitting data to said host;
a high speed, relatively low capacity cache memory means connected to said host interface means; and a controller for allocating appropriate storage locations on said relatively low speed, high capacity storage devices for data received from said host and stored in said cache, for accordingly controlling the flow of data from said cache to said relatively low speed, high capacity storage devices selected from plural classes of storage devices by the controller, and for storing said storage locations.
at least one host computer for performing arithmetic and logic operations on digital data, comprising an extremely high speed, relatively low capacity memory means, and interface means for transmitting and receiving data from external memory means; and a virtual storage system, comprising:
relatively low speed, high capacity storage devices of plural classes of storage devices;
host interface means for receiving data from and transmitting data to said host;
a high speed, relatively low capacity cache memory means connected to said host interface means; and a controller for allocating appropriate storage locations on said relatively low speed, high capacity storage devices for data received from said host and stored in said cache, for accordingly controlling the flow of data from said cache to said relatively low speed, high capacity storage devices selected from plural classes of storage devices by the controller, and for storing said storage locations.
6. The system of claim 5 in which said host computer does not control the location of storage of said data on said relatively low speed, high capacity storage devices, nor the choice of class of device on which said data is stored.
7. The system of claim 5 wherein said data is divided by said controller into blocks of convenient length for storage on said relatively low speed, high capacity storage devices and the location for storage of each said block is determined individually by said controller.
8. The system of claim 5 wherein said controller operates to store a single user-defined data file over plural storage devices as required, whereby a user of said system is not required to define the extent of files for the containment of data operated on by said host.
9. The system of claim 5 wherein a single user identified data file may be divided into blocks by said controller and stored on differing ones of said relatively low speed storage devices.
10. A data storage and recall system, comprising means for receiving data from means for processing data, means for long-term storage of said data including storage devices selected from plural classes of storage devices, means for temporary storage of said data, means for maintaining a directory of available storage locations on said long-term storage means, means for subdividing a user data set to sub-portions sized to fit said available storage locations, and means for causing said subportions to be stored in said available storage locations and for maintaining a directory of the locations at which said subportions have been stored, means for controlling flow of said data through said system, memory means for recal of said locations for long-term storage of said data, and means for outputting said data upon instruc-tion.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US26195181A | 1981-05-08 | 1981-05-08 | |
| US261,951 | 1981-05-08 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1183274A true CA1183274A (en) | 1985-02-26 |
Family
ID=22995577
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000402477A Expired CA1183274A (en) | 1981-05-08 | 1982-05-07 | Virtual storage system and method |
Country Status (2)
| Country | Link |
|---|---|
| JP (1) | JPS57203280A (en) |
| CA (1) | CA1183274A (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS61221842A (en) * | 1985-03-25 | 1986-10-02 | Sharp Corp | Computer system |
| JP2010113727A (en) * | 2009-12-21 | 2010-05-20 | Hitachi Ltd | Control method and storage system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4084234A (en) * | 1977-02-17 | 1978-04-11 | Honeywell Information Systems Inc. | Cache write capacity |
| JPS5918747B2 (en) * | 1979-04-24 | 1984-04-28 | 富士通株式会社 | Sector file control method |
| US4276927A (en) * | 1979-06-04 | 1981-07-07 | The Trane Company | Plate type heat exchanger |
-
1982
- 1982-05-07 CA CA000402477A patent/CA1183274A/en not_active Expired
- 1982-05-08 JP JP57077390A patent/JPS57203280A/en active Granted
Also Published As
| Publication number | Publication date |
|---|---|
| JPH0430056B2 (en) | 1992-05-20 |
| JPS57203280A (en) | 1982-12-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US4467421A (en) | Virtual storage system and method | |
| USRE36989E (en) | Virtual storage system and method | |
| US5491810A (en) | Method and system for automated data storage system space allocation utilizing prioritized data set parameters | |
| US4476526A (en) | Cache buffered memory subsystem | |
| EP0654736B1 (en) | Dynamically expandable storage unit array system | |
| US5805864A (en) | Virtual integrated cartridge loader for virtual tape storage system | |
| US5239647A (en) | Data storage hierarchy with shared storage level | |
| US6185661B1 (en) | Worm magnetic storage device | |
| JP3597945B2 (en) | Method of holding embedded directory information for data compression in direct access storage device and system including directory record | |
| US6000009A (en) | Method and apparatus for allocation of disk memory space for compressed data records | |
| EP0476841A2 (en) | Storage management across file systems | |
| US5214781A (en) | Method of managing storage medium | |
| US4974197A (en) | Batching data objects for recording on optical disks with maximum object count | |
| JP2005501317A (en) | External data storage management system and method | |
| JPS58161060A (en) | Suitable region dividing method for cash memory | |
| US5337197A (en) | Method and system for maintaining directory consistency in magneto-optic media | |
| CA1183274A (en) | Virtual storage system and method | |
| JP3214444B2 (en) | Control method and control device for magnetic disk drive | |
| CN101295228B (en) | Virtual tape server and method for export of at least one logical volume of source volume pool | |
| US20100131469A1 (en) | Storage management device and file deletion control method | |
| JP3725375B2 (en) | Data reorganization method in storage media library | |
| US20020141296A1 (en) | Method for reorganizing additional writing format storage media volume | |
| US6725350B2 (en) | Storage management system, apparatus, and method for supporting a multiplicity of logical block identification | |
| McDonald et al. | Improving file system performance by dynamically restructuring disk space | |
| EP0341036B1 (en) | A method of operating an information handling system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKEC | Expiry (correction) | ||
| MKEX | Expiry |