CA1231460A - Real time data transformation and transmission overlapping device - Google Patents
Real time data transformation and transmission overlapping deviceInfo
- Publication number
- CA1231460A CA1231460A CA000488367A CA488367A CA1231460A CA 1231460 A CA1231460 A CA 1231460A CA 000488367 A CA000488367 A CA 000488367A CA 488367 A CA488367 A CA 488367A CA 1231460 A CA1231460 A CA 1231460A
- Authority
- CA
- Canada
- Prior art keywords
- data
- medium
- buffer
- transformation
- data medium
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired
Links
Landscapes
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
REAL TIME DATA TRANSFORMATION AND TRANSMISSION OVERLEAP DEVICE
ABSTRACT
A real time data transformation and transmission apparatus transforms data from a first data device and transfers the transformed data to a second data device which need not have a data transfer rate consistent with the first data device. Data from the first data device is divided into blocks and is compressed by a compression device and written into a buffer.
A controller controls the buffer to transmit compressed data to the second data device as a function of the data receiving rate of the second data medium provided that the buffer contains a predetermined amount of data.
While the buffer is transmitting data, the compressor is compressing further blocks of data which are being written to the buffer such that the predetermined amount of data is stored in the buffer upon completion of the buffer transmitting a block of data. This ensures that complete blocks of data are transmitted to the second data medium at the data receiving rate of the second data medium.
ABSTRACT
A real time data transformation and transmission apparatus transforms data from a first data device and transfers the transformed data to a second data device which need not have a data transfer rate consistent with the first data device. Data from the first data device is divided into blocks and is compressed by a compression device and written into a buffer.
A controller controls the buffer to transmit compressed data to the second data device as a function of the data receiving rate of the second data medium provided that the buffer contains a predetermined amount of data.
While the buffer is transmitting data, the compressor is compressing further blocks of data which are being written to the buffer such that the predetermined amount of data is stored in the buffer upon completion of the buffer transmitting a block of data. This ensures that complete blocks of data are transmitted to the second data medium at the data receiving rate of the second data medium.
Description
:~L2~3~0 REAL TIME DATA TRAXSFO~TION AND TR~`:S~1ISSIO~ OVERLAPPING DEVICE
BACKGP~O"~D OF THE IDEATION
Field of the Invention This invention relates to data transformation and transmission, and in particular to a device for compressing data from a fast access storage devils and for transmitting compressed data to a slower medium.
Description of the Prior Art Compression or transformation of data usually results in an irregular flow of compressed data which is not conducive to efficient recording on magnetic tape or transmitting over communication channels with reasonable efficiency. Computers with large, volatile direct access main storage devices usually require that the data contained therein be saved on nonvol-anile, removable media such as tape for archival and backup purposes. The amount OX data to be store in conjunction with the relatively slow sequential access speed o' tape storage devices compared to the fast access speed of direct access storage devices has led to significant efforts to compress data and to increase the speed of tape storage devices. However, there is still a disparity in their relative speeds which is usually dealt with by first storing blocks of data from the direct access storage device Jo 20 unto a buffer which is dedicated to providing data to tape during save operations. I, ~984-017 ;~Z3~
In U. S. Patent Jo. 4,360,840 to i~olfrum et at, data is compressed in real tile as it us produced by a facsimile raster canter and stored in a buffer for transmission. Data is not transmitted until a full pave of text has been compressed and stored. While this procedure reduces the amount of buffer space required to store â pave of data, it does not address the problem of transmitting the data from the buffer while it is being come pressed. This can result in loss of valuable transmission time and possibly wrier larger storage devices to store a full pave of text.
U. S. Patent Jo. 3,490,690 to Apple et at relates to compression of an instruction trace and recording the compressed data to tape. Compressed data is written to tape and sections of data are dropped and not recorded when the tape gets behind or "no data" characters are recorder on the tape when the tape jets ahead. This results in lost data and does not optimize the data sty capability of the tape.
Seymour of the Invention A real time gala transformation and transformed data transmission device is provided which compresses data provided from a first data medium arid provides the compressed data to a second data medium which accepts data at a rate stower than thy rate at which tile first data medium provides swept. The compressed date is provided to eke second data medium as required ho top second data medium to operate in an efficient manner. A
tran3fcrm.ltion means rccciveC 2 first Blake of data from the first data medium and compresses the data into a corresponding block of compressed data which is preferably smaller than the uncompressed block of data. A
R~8~-nl7 -~23~l4~(~
buffer means receives the compressed data and stores blocks of compressed data for provision to the second data medium. Jo control jeans us ccu?iec to the transformation means and to the buffer neons for controlling the buffer means to provide a continuous flow of compressed data to the second data medium as a function of the data acceptance rate of the second data medium. Overlapped with the provision of compressed data to the seco.,G
data medium, the control means also causes the transformation means to compress a second bloc or data into the buffer jeans as a function of the amount of usable free space in the buffer means.
In the preferred embodiment, the first data medium is a direct access storage device of a computer itch periodically has its data save onto the second medium which is a tape storage device. The tape storage device receives data at a rate compatible iota constant operation of the tare storage device so that the tape need not be stopped and restarted which considerably slows the save operation. Data which is stored Jo the tape is compressed if. accordance with a desired compression scheme to increase data density on the tape which reduces the number of tapes needed for the save operation and reduces the tire required to store data on tape.
The first block of data prom tile direct access storage device is compressed and stored in the buffer means. Once the entire firs bloc'.; of data is stored in the buffer means, write to tape operation begins.
Twill the first block of data is being written to tape, the control means coordinates the writing to the buffer means of the second block of data.
Since the data transfer rate of the direct access storage device is much greater than the data receiving rate of the tape device, the second block Ripley I
-L-of data is preferably written into the buffer means before the first block of data is completely written to the tape. The second hock no data is then written to tape, while a third block of data is compressed to the buffer means in whatever free space exists. The control means assures what kowtow in the buffer is not lost while blocks are being stored and WriteNow at the same Tao; the control means also allocates buffer cycles between storing and writing data, with writing data to the tape having priority over the storing of compressed data from the direct access storage device.
The Sterno and writing of compressed blocks of data continues until lo all the desired data is saved on tape. The tape is run in a continuous or streaming mode unless there is not a complete block of compressed data available from the buffer means to prevent under running of the tape.
In the event that there is not a complete block of data available from the buffer means, the tape will stop in an interlock gap until a complete block is available. The storage capacity of the buffer means is chosen to be large enough to hold the largest possible block of data which has been compressed In some instances this may be larger than the block of date to be compressed if the block of data does not lend itself to compression.
with that size buffer means, and consider no the data transfer rate of the direct access storage device and the tape device, and the transror~atioll or compression characteristics of the data to be saved, the availability of a complete block of compressed data from the suffer means when required by the tare device is virtually assured.
ROY
~23~6'~
I, In en. alternate embodiment, data is compressed or transformed arc stoned in the buffer means. Coordination of the overlc.?pin~ o' ~ransforning and storing, data in the buffer means with writing to and from tape is based on the amount of compressed data stored in the buffer means.
In this embodiment, the size of the buffer means is determine as a function of the data transfer rates of the direct access storage device and the tape device, and the transformation characteristics of tile data to be saved such that transformed data is always available when requested by the tape device.
The present invention has the advantage of compressing data in real time as defined by the tape device requirements for data. This permits the data to be compressed to its limit in accordance with the selected compression technique and be written to tape US fast as the tape accepts the data. Fewer tapes are required for a save operation because the data is compressed to its limit. Because the compression and writing to tape are overlapped a desired amount as a function of the predetermined size of the blocks of data, the tape operates in a continuous or streaming rode thus reduce no the lime required for the save operation.
B of Description or the Drawing_ JIG. I is a si~plifled block diagram of a real time data transformation and transmission o~erlappin~. device in accordance with the present involution; and ~9~4-01, 3~l~60 Figs PA and 2B, the left side ox I muting with the right side of us, are a schematic 'lock diagram of the data transformation and transmission overlapping device of Fig. 1.
Detailed Description of the Preferred Embodiments .,, ., . . .,,, _ real time data transformation and transmission device is indicated generally at 10 in Fig. 1. A first data medium 12 such as a direct access storage device of a computer is coupled by a line 14 to a data transformation means 16 (also referred to as data compressor 16~ which transforms data as by compression, encryption or other forms of data transformation. The lines referred to herein comprise busses, coaxial cable, optical fibers or other appropriate communicative means. In the preferred embodiment, data compressor 16 receives a block of data from first data medium I and compresses the block of data by removing redundant data bytes or characters itch are typically randomly scattered throughout the block of data.
i ; sty compressor 16 provides the compressed data to n buffer lo 210ng a line I Buffer I comprises a dynamic random access memory and provides compressed data to a second data medium 24 on a line 26. Second data medium US has a data transfer rate which is usually slower than the data reinsurer rate of first data medium 12.
In one functional embodiment, second data medium 24 is a magnetic tape unit such as an I'M model 3430, having known start and stop times and first data medium 12 is a random access storage device such as an IBM model 3370.
* P~gistered trade Mark ?~09~ 17 ., ....
I
-7- Jo ennui a section of rlata contains a series of redundant boles, a zap during itch rho data is provided to buffer 18 occur. If data err written row the buffer 18 directly to the second data medium 24 as the data is being compressed, the gap old cause second data edema 24 to stop and roared the tape to the correct porn. to start receiving data again.
To solve this problem, a control means or controller 78 is coupled between compressor 16 by a line 30 and buffer lo by a line 3 Controller I determines Lyon a Blake of data has been compressed ho compressor 16 and 7~rlttelt to buffer 18. Controller 28 then ir.-tiates the writing of that block of data to the second data medium 24 at a rate determined by second data medium 24 requests for bytes of data. Lyle the dote it being transmitted to the sPconfi data tedium 24, a Nat bloc of data so compressed by compressor 16 as controlled by controller I. The next block of data is compressed into buffer 18, which is a first in, first out data huller, before the first block of coF~pre~seci data is co~letely transr;titted to the second data moderate 24 such that the sickness data medium 24 operates in a streaming mode and it not recaptured to stop arid start during a bloc or bitterly data blocks.
In rigs I and ZB, a hardware implementation of a Syste7~ts ~etvork.
Architecture (Sty) compression algorithm used, and the apparatus of the present inventlolt t S shown. The block of data is received from a storage device 34 one byte at a time on a line 36 and is rotten into a first resister 38 and then into a second resister 40 via a line 62 while the knockout byte ox data is written into first register 38 such that the registers keynoter. sequen;lal bites ox the block of data.
~09l~ 7 ~23~60 Jo first comparator 44 receives the first byte of data in second l-~gister 40 on a line 46, the second byte of data in first resister ~-~, and a prime or preselected byte on a line 48 from a memory device 52. Frost comparator 44 compares the first and second bytes of data with the prize byte to determine if the data contains a series of at least two prime bytes. A second comparator 54 receives the first data byte on line 46, the second data hype on line 42 and a third data byte on line 36 to determine if there is a series of at least three identical bytes. A series of at least three identical bytes of data which are not the same as the pyre byte, detected in this manner are referred to as a nor prime series. If no prime or non prime series are detected, the condition is referred to as mixed data.
Ed data bytes are written into a buffer 53 which is coupled by a line 60 through a selector 62 to line 46. Selector 62 is controller by a kippers sequencer 64 which receives information identifying the type of data series prom first comparator 44 and second comparator 54 on lines 68 and 70 respectively. From this information, compress sequencer 64 controls formation of a string control byte (SOB) ho a SOB coder 72. SOB coder 72 forms the SOB as a function of information provided from cypress sequencer 64 011 a line Al The information and hence the SKYE is representable o.
the type of series ox bytes and indicates the number of bytes in the particular series o' bytes it represents.
my controlling selector 62 via a line US, compress sequencer 64 controls the content and order in which data provided to selector 62 by sucker wrester 40 via line by and Subs provided to selector 62 Lo SOB
I nl7 9,.;~3~
q ; coder 72 on a line 79, are written into buffer 58. in the case of a Roy series, the SKYE is not fulgid by data b-t-- because the prize Tao is redefined in memory device 52. An SKYE indicating a norpri~e series to followed by a data byte of the repeated character. on SKYE ind'ca~i~~ mixed dawn is fulgid by all bytes of the mixed gala.
Using the above SUE compression scheme, data is usual LO compressed o'er than 50 percent. on the preferred e~Dodi~ent, an SKYE s a bite of dât2 'I
having 2 bits defining the type of series it represents, and 6 bits itch represent the number of bytes of data in the series up to 'ox. suffer f~8 is capable of storing 32~768 bytes. I. a bloc is 211 let data, the date ill expand by one byte, the SOB, for each 63 b toes of data. Wherefore, tile size o. a Blake of data to be cc~pressed ~'25 predeter~-'ned to go 3-,2;6 bytes 90 that it hill always fit if. borer 58 when compressed.
Use of other compression or encryption routines is within the scope 07' the present invention. The particular routine described above is implemented ; in hardware to obtain a desired high steed of compression, Compress sequencer 64 is coupled to first register 38 and second register 40 by a line I Jo shift data from line TV into first east. US
and to shift data from first register 33 to second register 40. Compress sequencer 64 also indicates to a compress ~edcless resister lo in I' h. -8 by a line if., the correct buffer 58 address for data to be written irrupt.
Compress sequence 64 is coupled to 2 SOB address register 114 by a lyre 116 and to a previous address resister lo by a line lo SKYE address ~0984-017 ill Z 3 Al Lo I
register 114 contains the buffer 58 address to the SOB indicating the type of data run it precedes.
Subs are written to buffer I at a time after a run of mixed data occurs because the SOB indicates the length of the run which is not Nemo until the run of mixed data is finished. However, the address of the SOB
precedes the data whether it is mixed or ncnprime. Previous address resister 118 contains the address of the last byte of the last pre~iiouslv compressed block in buffer 58. An address selector 124 which is coupled to compress address register 110 and SOB address register 114 by lines 120 and 10 128 respectively provides buffer a with the appropriate address for data and Subs on a line 129 as they are written into buffer 58. SOB address register 114 receives an address from congress address register !10 on fire 126 when initiated by compress sequencer 64 on line 116.
k buffer eon oiler 132 lo coupled to address swallowtail 124 by a line 134 15 arid initiates selection and provision of the address to buffer 58 by address selector 124. Buffer controller 132 is coupled to compress sequencer I by line 136 and 138. Buffer controller 13~ rants a butter cycle to compress sequencer fix on line 136, and compress sequencer 64 indicates on line 138 that a byte of data or a SOB is available to be 20 written eon buffer 58 and the buffer address has been updated. In this manse , a bloclc of data is co~preFIsed and written to buffer 58.
Once a complete block of compressed data is avai~eb e Roy buffer C8, the block is transmitted to a set of latches 150 over a line 152. the avail2bi1ity of a complete block of compressed data is indicated by a Reunion 3l23~ I
finished block line 139, which is set responsive to a complete block of data having beer. transferred from date storage device 34 to first resister 38. Finished block line 139 is coupled to compress sequencer 64 which nitrates transmission of the compressed block of data. Compress sequencer 64 is also coupled to storage medium 34 ho a line 141 to initiate transfer of data from storage medium 34 to first fog suer 38.
Ye second data Tedium indicated at 154 receives the compressed data from latches 150 at the rate required b-- second data medium 154. A device sequencer 158 is coupled to second data tedium 154 by lines 160 and 162.
10. Line 160 provides requests for bytes of data from second data medium 154.
Line 162 provides an indication to second data medium 154 that a compressed block of data has been provided to second data medium 154.
- Device sequencer 158 is also coupled to buffer control 132 by line 164 which provides requests for a buffer 58 cycle from second data medium 154.
Device sequencer 158 increments a device address resister 170 over a live 172 such that the device address register 170 contains the address of the byte to be written to latches 150 from buffer 58. The device address indicated by device address register 170 is provided by a line 174 to selector 124 for provision to buffer I
The device address is also provided to A previous address comparator 176 and a compress address comparator 1/8 by line 174. The previous ; address coQrarator 176 compares the device address to the previous address indicating the address of the last byte of the latest bloc of data completely written to buffer 58. The previous address is provided to the '~0984-Q L ?
.1~3~ pa!
--I " I
previous address comparator 176 by the previous address register 118 on a line 1~0. Ike previous address is not inserted into previous address register 118 until a complete block is WriteNow to buffer 58.
Inn a comparison is indicated on a line 182 winch is coupled to device sequencer 158, device sequencer 158 stops the transmission of compressed data from buffer 58 to second data medium 154 because a complete block of data has been transmitted. The previous address is not changed before a block is completely transmitted to second data medium !54. A
desired inter bloc zap is then established on second data medium 154 Chile the previous address is changed and the next block of compressed data is transmitted without interruption of the operation of second data medium 154. Thus, second data medium 154 can operate efficiently in a streaming or a start stop mode.
Compress address comparator 178 is coupled to line 126 to receive the address contained in compress address register 110 which indicates the present address of the buffer 58 that data is being written into. An address in device address register 170 equal to an address in compress address register 110 indicates that data is available to be written into buffer US, but that previously compressed data from that address has not vet been transmitted to second data medium 154. Compress address comparator 178 is coupled by a line 182 to buffer control I co Prevent the Front of a huller cycle to compress sequencer fix and thus ensure that data is rot retaken to buffer 58 until data having the tame address is transmitted to second data medium 154.
R0984-0l7 I 1 ~60 In a further proofer embodiment, previous address resister lo is loaded iota.. 2 buffer 58 address at which compressed data is stored rich is a desired number of bytes from the byte currently being written to second data medium 154. The number of bytes is predetermined as a function of S the relative data transfer rates of storage device 34 and second data medium 154 together with the predicted transformation characteristics OX
the data in storage device 34. This permits data to be arranged in other than compressed blokes of a size defined by the compressibility of the data. Since the address n previous address register 118 is changer as data it being written to second data tedium 15~, second data medium 15~
arranges the data to best suit its characteristics. Inter bloc zaps are inserted by second data medium 154 where and if desired.
In yet a further preferred embodiment, the second data tedium 154 comprises an interface to a communication system such as a packet sicken system. In this embodiment, busier 53 provides compressed packets to suckered data medium 154 as a function of the transmission bandwidth of the second data medium. Buffer 53 serves as a buffer eon both the compression characteristics of tile packet and access irregularities to the second data medium, thus ensuring the availability of a compressed packet for transmission.
rougher controller 13~ prioritizes access to buffer 58. ~ufrer 38 access is requested by device sequencer 15~ on line 164, compress sequencer Go on line 13 end a refresh controlle~(r.ot shown) when buffer 58 is a dyllamic memory. Highest prlorltv is given to device sequencer 158 followed by compress sequencer 64~ Lowest priority is given to refresh.
~09~4-017 - :~.;23.1l~
if all three request access simultaneously, buffer controller 132 grants p irrupt as described above. In doing this, it places priority an trays-milting data to second data medium 154 to keep second data medium 154 operating in a continuous manner. Thus, data is saved in a minimal remount of time with the use of a minimum amount of second data medium 154, whether it be magnetic tape or transmission bandwidth.
jowl the invention has been shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various charges in form and details may be made therein without departing from the spirit and scope of the invention.
~84-017
BACKGP~O"~D OF THE IDEATION
Field of the Invention This invention relates to data transformation and transmission, and in particular to a device for compressing data from a fast access storage devils and for transmitting compressed data to a slower medium.
Description of the Prior Art Compression or transformation of data usually results in an irregular flow of compressed data which is not conducive to efficient recording on magnetic tape or transmitting over communication channels with reasonable efficiency. Computers with large, volatile direct access main storage devices usually require that the data contained therein be saved on nonvol-anile, removable media such as tape for archival and backup purposes. The amount OX data to be store in conjunction with the relatively slow sequential access speed o' tape storage devices compared to the fast access speed of direct access storage devices has led to significant efforts to compress data and to increase the speed of tape storage devices. However, there is still a disparity in their relative speeds which is usually dealt with by first storing blocks of data from the direct access storage device Jo 20 unto a buffer which is dedicated to providing data to tape during save operations. I, ~984-017 ;~Z3~
In U. S. Patent Jo. 4,360,840 to i~olfrum et at, data is compressed in real tile as it us produced by a facsimile raster canter and stored in a buffer for transmission. Data is not transmitted until a full pave of text has been compressed and stored. While this procedure reduces the amount of buffer space required to store â pave of data, it does not address the problem of transmitting the data from the buffer while it is being come pressed. This can result in loss of valuable transmission time and possibly wrier larger storage devices to store a full pave of text.
U. S. Patent Jo. 3,490,690 to Apple et at relates to compression of an instruction trace and recording the compressed data to tape. Compressed data is written to tape and sections of data are dropped and not recorded when the tape gets behind or "no data" characters are recorder on the tape when the tape jets ahead. This results in lost data and does not optimize the data sty capability of the tape.
Seymour of the Invention A real time gala transformation and transformed data transmission device is provided which compresses data provided from a first data medium arid provides the compressed data to a second data medium which accepts data at a rate stower than thy rate at which tile first data medium provides swept. The compressed date is provided to eke second data medium as required ho top second data medium to operate in an efficient manner. A
tran3fcrm.ltion means rccciveC 2 first Blake of data from the first data medium and compresses the data into a corresponding block of compressed data which is preferably smaller than the uncompressed block of data. A
R~8~-nl7 -~23~l4~(~
buffer means receives the compressed data and stores blocks of compressed data for provision to the second data medium. Jo control jeans us ccu?iec to the transformation means and to the buffer neons for controlling the buffer means to provide a continuous flow of compressed data to the second data medium as a function of the data acceptance rate of the second data medium. Overlapped with the provision of compressed data to the seco.,G
data medium, the control means also causes the transformation means to compress a second bloc or data into the buffer jeans as a function of the amount of usable free space in the buffer means.
In the preferred embodiment, the first data medium is a direct access storage device of a computer itch periodically has its data save onto the second medium which is a tape storage device. The tape storage device receives data at a rate compatible iota constant operation of the tare storage device so that the tape need not be stopped and restarted which considerably slows the save operation. Data which is stored Jo the tape is compressed if. accordance with a desired compression scheme to increase data density on the tape which reduces the number of tapes needed for the save operation and reduces the tire required to store data on tape.
The first block of data prom tile direct access storage device is compressed and stored in the buffer means. Once the entire firs bloc'.; of data is stored in the buffer means, write to tape operation begins.
Twill the first block of data is being written to tape, the control means coordinates the writing to the buffer means of the second block of data.
Since the data transfer rate of the direct access storage device is much greater than the data receiving rate of the tape device, the second block Ripley I
-L-of data is preferably written into the buffer means before the first block of data is completely written to the tape. The second hock no data is then written to tape, while a third block of data is compressed to the buffer means in whatever free space exists. The control means assures what kowtow in the buffer is not lost while blocks are being stored and WriteNow at the same Tao; the control means also allocates buffer cycles between storing and writing data, with writing data to the tape having priority over the storing of compressed data from the direct access storage device.
The Sterno and writing of compressed blocks of data continues until lo all the desired data is saved on tape. The tape is run in a continuous or streaming mode unless there is not a complete block of compressed data available from the buffer means to prevent under running of the tape.
In the event that there is not a complete block of data available from the buffer means, the tape will stop in an interlock gap until a complete block is available. The storage capacity of the buffer means is chosen to be large enough to hold the largest possible block of data which has been compressed In some instances this may be larger than the block of date to be compressed if the block of data does not lend itself to compression.
with that size buffer means, and consider no the data transfer rate of the direct access storage device and the tape device, and the transror~atioll or compression characteristics of the data to be saved, the availability of a complete block of compressed data from the suffer means when required by the tare device is virtually assured.
ROY
~23~6'~
I, In en. alternate embodiment, data is compressed or transformed arc stoned in the buffer means. Coordination of the overlc.?pin~ o' ~ransforning and storing, data in the buffer means with writing to and from tape is based on the amount of compressed data stored in the buffer means.
In this embodiment, the size of the buffer means is determine as a function of the data transfer rates of the direct access storage device and the tape device, and the transformation characteristics of tile data to be saved such that transformed data is always available when requested by the tape device.
The present invention has the advantage of compressing data in real time as defined by the tape device requirements for data. This permits the data to be compressed to its limit in accordance with the selected compression technique and be written to tape US fast as the tape accepts the data. Fewer tapes are required for a save operation because the data is compressed to its limit. Because the compression and writing to tape are overlapped a desired amount as a function of the predetermined size of the blocks of data, the tape operates in a continuous or streaming rode thus reduce no the lime required for the save operation.
B of Description or the Drawing_ JIG. I is a si~plifled block diagram of a real time data transformation and transmission o~erlappin~. device in accordance with the present involution; and ~9~4-01, 3~l~60 Figs PA and 2B, the left side ox I muting with the right side of us, are a schematic 'lock diagram of the data transformation and transmission overlapping device of Fig. 1.
Detailed Description of the Preferred Embodiments .,, ., . . .,,, _ real time data transformation and transmission device is indicated generally at 10 in Fig. 1. A first data medium 12 such as a direct access storage device of a computer is coupled by a line 14 to a data transformation means 16 (also referred to as data compressor 16~ which transforms data as by compression, encryption or other forms of data transformation. The lines referred to herein comprise busses, coaxial cable, optical fibers or other appropriate communicative means. In the preferred embodiment, data compressor 16 receives a block of data from first data medium I and compresses the block of data by removing redundant data bytes or characters itch are typically randomly scattered throughout the block of data.
i ; sty compressor 16 provides the compressed data to n buffer lo 210ng a line I Buffer I comprises a dynamic random access memory and provides compressed data to a second data medium 24 on a line 26. Second data medium US has a data transfer rate which is usually slower than the data reinsurer rate of first data medium 12.
In one functional embodiment, second data medium 24 is a magnetic tape unit such as an I'M model 3430, having known start and stop times and first data medium 12 is a random access storage device such as an IBM model 3370.
* P~gistered trade Mark ?~09~ 17 ., ....
I
-7- Jo ennui a section of rlata contains a series of redundant boles, a zap during itch rho data is provided to buffer 18 occur. If data err written row the buffer 18 directly to the second data medium 24 as the data is being compressed, the gap old cause second data edema 24 to stop and roared the tape to the correct porn. to start receiving data again.
To solve this problem, a control means or controller 78 is coupled between compressor 16 by a line 30 and buffer lo by a line 3 Controller I determines Lyon a Blake of data has been compressed ho compressor 16 and 7~rlttelt to buffer 18. Controller 28 then ir.-tiates the writing of that block of data to the second data medium 24 at a rate determined by second data medium 24 requests for bytes of data. Lyle the dote it being transmitted to the sPconfi data tedium 24, a Nat bloc of data so compressed by compressor 16 as controlled by controller I. The next block of data is compressed into buffer 18, which is a first in, first out data huller, before the first block of coF~pre~seci data is co~letely transr;titted to the second data moderate 24 such that the sickness data medium 24 operates in a streaming mode and it not recaptured to stop arid start during a bloc or bitterly data blocks.
In rigs I and ZB, a hardware implementation of a Syste7~ts ~etvork.
Architecture (Sty) compression algorithm used, and the apparatus of the present inventlolt t S shown. The block of data is received from a storage device 34 one byte at a time on a line 36 and is rotten into a first resister 38 and then into a second resister 40 via a line 62 while the knockout byte ox data is written into first register 38 such that the registers keynoter. sequen;lal bites ox the block of data.
~09l~ 7 ~23~60 Jo first comparator 44 receives the first byte of data in second l-~gister 40 on a line 46, the second byte of data in first resister ~-~, and a prime or preselected byte on a line 48 from a memory device 52. Frost comparator 44 compares the first and second bytes of data with the prize byte to determine if the data contains a series of at least two prime bytes. A second comparator 54 receives the first data byte on line 46, the second data hype on line 42 and a third data byte on line 36 to determine if there is a series of at least three identical bytes. A series of at least three identical bytes of data which are not the same as the pyre byte, detected in this manner are referred to as a nor prime series. If no prime or non prime series are detected, the condition is referred to as mixed data.
Ed data bytes are written into a buffer 53 which is coupled by a line 60 through a selector 62 to line 46. Selector 62 is controller by a kippers sequencer 64 which receives information identifying the type of data series prom first comparator 44 and second comparator 54 on lines 68 and 70 respectively. From this information, compress sequencer 64 controls formation of a string control byte (SOB) ho a SOB coder 72. SOB coder 72 forms the SOB as a function of information provided from cypress sequencer 64 011 a line Al The information and hence the SKYE is representable o.
the type of series ox bytes and indicates the number of bytes in the particular series o' bytes it represents.
my controlling selector 62 via a line US, compress sequencer 64 controls the content and order in which data provided to selector 62 by sucker wrester 40 via line by and Subs provided to selector 62 Lo SOB
I nl7 9,.;~3~
q ; coder 72 on a line 79, are written into buffer 58. in the case of a Roy series, the SKYE is not fulgid by data b-t-- because the prize Tao is redefined in memory device 52. An SKYE indicating a norpri~e series to followed by a data byte of the repeated character. on SKYE ind'ca~i~~ mixed dawn is fulgid by all bytes of the mixed gala.
Using the above SUE compression scheme, data is usual LO compressed o'er than 50 percent. on the preferred e~Dodi~ent, an SKYE s a bite of dât2 'I
having 2 bits defining the type of series it represents, and 6 bits itch represent the number of bytes of data in the series up to 'ox. suffer f~8 is capable of storing 32~768 bytes. I. a bloc is 211 let data, the date ill expand by one byte, the SOB, for each 63 b toes of data. Wherefore, tile size o. a Blake of data to be cc~pressed ~'25 predeter~-'ned to go 3-,2;6 bytes 90 that it hill always fit if. borer 58 when compressed.
Use of other compression or encryption routines is within the scope 07' the present invention. The particular routine described above is implemented ; in hardware to obtain a desired high steed of compression, Compress sequencer 64 is coupled to first register 38 and second register 40 by a line I Jo shift data from line TV into first east. US
and to shift data from first register 33 to second register 40. Compress sequencer 64 also indicates to a compress ~edcless resister lo in I' h. -8 by a line if., the correct buffer 58 address for data to be written irrupt.
Compress sequence 64 is coupled to 2 SOB address register 114 by a lyre 116 and to a previous address resister lo by a line lo SKYE address ~0984-017 ill Z 3 Al Lo I
register 114 contains the buffer 58 address to the SOB indicating the type of data run it precedes.
Subs are written to buffer I at a time after a run of mixed data occurs because the SOB indicates the length of the run which is not Nemo until the run of mixed data is finished. However, the address of the SOB
precedes the data whether it is mixed or ncnprime. Previous address resister 118 contains the address of the last byte of the last pre~iiouslv compressed block in buffer 58. An address selector 124 which is coupled to compress address register 110 and SOB address register 114 by lines 120 and 10 128 respectively provides buffer a with the appropriate address for data and Subs on a line 129 as they are written into buffer 58. SOB address register 114 receives an address from congress address register !10 on fire 126 when initiated by compress sequencer 64 on line 116.
k buffer eon oiler 132 lo coupled to address swallowtail 124 by a line 134 15 arid initiates selection and provision of the address to buffer 58 by address selector 124. Buffer controller 132 is coupled to compress sequencer I by line 136 and 138. Buffer controller 13~ rants a butter cycle to compress sequencer fix on line 136, and compress sequencer 64 indicates on line 138 that a byte of data or a SOB is available to be 20 written eon buffer 58 and the buffer address has been updated. In this manse , a bloclc of data is co~preFIsed and written to buffer 58.
Once a complete block of compressed data is avai~eb e Roy buffer C8, the block is transmitted to a set of latches 150 over a line 152. the avail2bi1ity of a complete block of compressed data is indicated by a Reunion 3l23~ I
finished block line 139, which is set responsive to a complete block of data having beer. transferred from date storage device 34 to first resister 38. Finished block line 139 is coupled to compress sequencer 64 which nitrates transmission of the compressed block of data. Compress sequencer 64 is also coupled to storage medium 34 ho a line 141 to initiate transfer of data from storage medium 34 to first fog suer 38.
Ye second data Tedium indicated at 154 receives the compressed data from latches 150 at the rate required b-- second data medium 154. A device sequencer 158 is coupled to second data tedium 154 by lines 160 and 162.
10. Line 160 provides requests for bytes of data from second data medium 154.
Line 162 provides an indication to second data medium 154 that a compressed block of data has been provided to second data medium 154.
- Device sequencer 158 is also coupled to buffer control 132 by line 164 which provides requests for a buffer 58 cycle from second data medium 154.
Device sequencer 158 increments a device address resister 170 over a live 172 such that the device address register 170 contains the address of the byte to be written to latches 150 from buffer 58. The device address indicated by device address register 170 is provided by a line 174 to selector 124 for provision to buffer I
The device address is also provided to A previous address comparator 176 and a compress address comparator 1/8 by line 174. The previous ; address coQrarator 176 compares the device address to the previous address indicating the address of the last byte of the latest bloc of data completely written to buffer 58. The previous address is provided to the '~0984-Q L ?
.1~3~ pa!
--I " I
previous address comparator 176 by the previous address register 118 on a line 1~0. Ike previous address is not inserted into previous address register 118 until a complete block is WriteNow to buffer 58.
Inn a comparison is indicated on a line 182 winch is coupled to device sequencer 158, device sequencer 158 stops the transmission of compressed data from buffer 58 to second data medium 154 because a complete block of data has been transmitted. The previous address is not changed before a block is completely transmitted to second data medium !54. A
desired inter bloc zap is then established on second data medium 154 Chile the previous address is changed and the next block of compressed data is transmitted without interruption of the operation of second data medium 154. Thus, second data medium 154 can operate efficiently in a streaming or a start stop mode.
Compress address comparator 178 is coupled to line 126 to receive the address contained in compress address register 110 which indicates the present address of the buffer 58 that data is being written into. An address in device address register 170 equal to an address in compress address register 110 indicates that data is available to be written into buffer US, but that previously compressed data from that address has not vet been transmitted to second data medium 154. Compress address comparator 178 is coupled by a line 182 to buffer control I co Prevent the Front of a huller cycle to compress sequencer fix and thus ensure that data is rot retaken to buffer 58 until data having the tame address is transmitted to second data medium 154.
R0984-0l7 I 1 ~60 In a further proofer embodiment, previous address resister lo is loaded iota.. 2 buffer 58 address at which compressed data is stored rich is a desired number of bytes from the byte currently being written to second data medium 154. The number of bytes is predetermined as a function of S the relative data transfer rates of storage device 34 and second data medium 154 together with the predicted transformation characteristics OX
the data in storage device 34. This permits data to be arranged in other than compressed blokes of a size defined by the compressibility of the data. Since the address n previous address register 118 is changer as data it being written to second data tedium 15~, second data medium 15~
arranges the data to best suit its characteristics. Inter bloc zaps are inserted by second data medium 154 where and if desired.
In yet a further preferred embodiment, the second data tedium 154 comprises an interface to a communication system such as a packet sicken system. In this embodiment, busier 53 provides compressed packets to suckered data medium 154 as a function of the transmission bandwidth of the second data medium. Buffer 53 serves as a buffer eon both the compression characteristics of tile packet and access irregularities to the second data medium, thus ensuring the availability of a compressed packet for transmission.
rougher controller 13~ prioritizes access to buffer 58. ~ufrer 38 access is requested by device sequencer 15~ on line 164, compress sequencer Go on line 13 end a refresh controlle~(r.ot shown) when buffer 58 is a dyllamic memory. Highest prlorltv is given to device sequencer 158 followed by compress sequencer 64~ Lowest priority is given to refresh.
~09~4-017 - :~.;23.1l~
if all three request access simultaneously, buffer controller 132 grants p irrupt as described above. In doing this, it places priority an trays-milting data to second data medium 154 to keep second data medium 154 operating in a continuous manner. Thus, data is saved in a minimal remount of time with the use of a minimum amount of second data medium 154, whether it be magnetic tape or transmission bandwidth.
jowl the invention has been shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various charges in form and details may be made therein without departing from the spirit and scope of the invention.
~84-017
Claims (14)
1. Real time data transformation and transmission apparatus for overlapping transformation of data provided from a first data medium which has a selected data transfer rate and transmission of the transformed data to a second data medium which has a selected data receiving rate, said apparatus comprising:
transformation means coupled to the first data medium for transforming data received from the first data medium;
buffer means coupled to the transformation means and to the second data medium for storing the transformed data and transmitting the transformed data to the second data medium; and control means coupled to the transformation means and to the buffer means for controlling the transformation means to provide transformed data to the buffer means as a function of the amount of data in the buffer means and controlling the buffer means to transmit the transformed data to the second data medium as a function of the data receiving rate of the second data medium provided that the buffer means contains an amount of data sufficient to ensure that data is transmitted to the second data medium at the data receiving rate of the second data medium.
transformation means coupled to the first data medium for transforming data received from the first data medium;
buffer means coupled to the transformation means and to the second data medium for storing the transformed data and transmitting the transformed data to the second data medium; and control means coupled to the transformation means and to the buffer means for controlling the transformation means to provide transformed data to the buffer means as a function of the amount of data in the buffer means and controlling the buffer means to transmit the transformed data to the second data medium as a function of the data receiving rate of the second data medium provided that the buffer means contains an amount of data sufficient to ensure that data is transmitted to the second data medium at the data receiving rate of the second data medium.
2. The apparatus of Claim 1 wherein said control means initiates the transmission of data from the buffer means to the second data medium when a predetermined amount of data is completely stored in the buffer means.
3. The apparatus of Claim 2 wherein the predetermined amount of data comprises a block containing a selected number of bytes of transformed data.
4. The apparatus of Claim 2 wherein the predetermined amount of data comprises a block containing a selected number of bytes of data from the first data medium prior to transformation, and wherein the control means controls the buffer means such that a complete block of data is compressed into the buffer means before transmission of the transformed data to the second data medium is initiated by the control means.
5. The apparatus of Claim 2 wherein the predetermined amount of data is equal to blocks of data of a selected size and the control means controls the buffer means such that a complete block of data is transformed and stored into the buffer means by the transformation means before transmission of the transformed data to the second data medium is initiated by the control means.
6. The apparatus of Claim 1 wherein the control means controls the amount of data stored in the buffer means to be at least equal to a
6. The apparatus of Claim 1 wherein the control means controls the amount of data stored in the buffer means to be at least equal to a
Claim 6 Continued predetermined amount of data such that the second data medium receives transformed data at a substantially continuous rate.
7. The apparatus of Claim 1 wherein the second data medium comprises a magnetic tape device operating in a streaming mode.
8. The apparatus of Claim 7 wherein the data transfer rate of the first data medium is substantially greater than the data transfer rate of the tape device.
9. The apparatus of Claim 1 wherein the second data medium comprises a magnetic tape device operating in a start/stop mode.
10. The apparatus of Claim 1 wherein the transformation means comprises a data compression means for compressing the data from the first data medium.
11. The apparatus of Claim 10 wherein the data comprises blocks of characters, said data compression means compressing the blocks of characters as a function of repetitive series of the characters.
12. The apparatus of Claim 1 wherein the transformation means comprises a data encryption means for encrypting the data from the first data medium.
13. The apparatus of Claim 12 wherein the transformation means compresses data at 2 rate which is greater than the data receiving rate of the second data medium.
14 . The apparatus of Claim 1 wherein the buffer means comprises a first in, first out buffer which receives transformed data from the transformation means and provides the data to the second data medium as requested by the second data medium for substantially continuous operation of the second data medium.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US68954585A | 1985-01-07 | 1985-01-07 | |
| US689,545 | 1985-01-07 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1231460A true CA1231460A (en) | 1988-01-12 |
Family
ID=24768937
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000488367A Expired CA1231460A (en) | 1985-01-07 | 1985-08-08 | Real time data transformation and transmission overlapping device |
Country Status (1)
| Country | Link |
|---|---|
| CA (1) | CA1231460A (en) |
-
1985
- 1985-08-08 CA CA000488367A patent/CA1231460A/en not_active Expired
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US4956808A (en) | Real time data transformation and transmission overlapping device | |
| US6321237B1 (en) | Recording and reproducing apparatus and method for accessing data stored on a randomly accessible recording medium, and for managing data thereon | |
| EP0935394A2 (en) | Transmission of time-shifted data to a plurality of channels | |
| US6212681B1 (en) | Information processing apparatus and method therefor in a data transfer network | |
| AU674064B2 (en) | Video server memory management | |
| DE69525556T2 (en) | Device and method executed on a computer for real-time multimedia data transmission in a distributed computer arrangement | |
| CA2178376A1 (en) | Video data retrieval method for use in video server environments that use striped disks | |
| EP0744868A2 (en) | Method and apparatus for guaranteeing average case disk transfer bandwith and seek time for a video server | |
| RU2328779C2 (en) | Method and device for data reproduction, recorded in interactive record medium, in conjunction with associated auxiliary data | |
| CA2142380A1 (en) | Buffer management policy for an on-demand video server | |
| JPH11119923A (en) | Data access controller and record medium recorded with data access control program | |
| KR960025209A (en) | Method and apparatus for retrieving a requested data unit among data units divided into stripes stored on a plurality of disks | |
| US7660837B2 (en) | Method for automatically managing disk fragmentation | |
| US20060059190A1 (en) | File recording method and file recording apparatus | |
| AU3723395A (en) | Method of pre-caching data utilizing thread lists and multimedia editing system using such pre-caching | |
| EP0369689A3 (en) | Vector quantizer for images | |
| KR100244386B1 (en) | Memory device and a facsimile apparatus using the same | |
| US7096327B2 (en) | System for and method of accessing blocks on a storage medium | |
| US9247193B2 (en) | Transferring multiple records including video data | |
| CN101425310B (en) | Methods for reserving index memory space in AVI recording apparatus | |
| CA1231460A (en) | Real time data transformation and transmission overlapping device | |
| US6516134B1 (en) | Data recording/reproducing method and apparatus | |
| US7092192B1 (en) | Videotape recording and playback device, and videotape player | |
| US20090067818A1 (en) | Video recording and reproducing system and reading method of video data | |
| US8200067B2 (en) | Processing of scalable compressed video data formats for nonlinear video editing systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKEX | Expiry |