CA1105141A - Markov processor for context encoding from given characters and for character decoding from given contexts - Google Patents

Abstract

A MARKOV PROCESSOR FOR CONTEXT ENCODING
FROM GIVEN CHARACTERS AND FOR CHARACTER
DECODING FROM GIVEN CONTEXTS
Abstract In an apparatus for generating variable length codewords c(aj) and c(aj) responsive to corresponding fixed length codewords b(ai) and b(aj), where ai and aj are source alphabet characters, ai.epsilon.Al and aj.epsilon.A2, ambiguity arises whenever any fixed length charac-ter to be encoded can instantaneously represent source characters aj and aj drawn from two or more dissimilar alphabets i.e., A1 or A2. This is resolved by the inclusion of a Markov processor in com-bination with the apparatus. The processor establishes the point in the sequence when transitions occur between fixed length char-acters in one alphabet to fixed length characters in another alpha-bet by the message context. The processor includes a map of state and transition paths. This map models certain statistical regulari-ties of runs of fixed code elements and the relative likelihood that an ambiguous fixed code character appearing in a first run belongs to a given alphabet. The processor, starting from an arbitrary initial position, tracks any given run of fixed code characters applied to the encoder in terms of a succession of states and paths.

Description

5~

FROM GIVEN C~RACTERS AND FOR CHARACTER
DECODING FROM GIVEN CONTEXTS
Background of the Invention This invention relates to fixed to variable length word encod ing and to variable to fixed length word decoding. More particularly, the invention relates to mechanisms for resolving ambiguities when transitions between substrings of different alphabets occur. This enables the assignment of an optimum variable length word in encod-ing. It further enables the parsing of comma free bit streams of variable length code words and for resolving ambiguities between said variable length words and their fixed length representations in decoding.
Fixed to variable length encoding consists of assigning a vari-able length code word c(a) to the appearance of a fixed length code-word b(a). Each fixed length code word represents a character "d"
in a source alphabet A.

:
, ..
. ' ' ' :: : :

~ ~ SA9-75-024 -2-, 1 Since b(a) can represent both a character a, in Al and A2 in A2, ambiguity arises in both the encoding and decoding processes.
The encoder must determine when a ~ransition from bne alphabet to another has occurred to make an optimum variable length code assignment, c(a). The decoder must detPrmine which alphabet was selected by the encoder to parse the comma free bit stream and to properly reassign the fixed length code word b(a) to the received variable length code word c(a~.
There exists cost and performance advantages in the machine storage, transfer and manipulation of character alphabets using equal fixed length codewords. Among the advantages are uniformity and standardization of storage cell/register sizes, the number of conductors for data busing, and the reduced or non-existant informational overhead to tract character boundaries. ~owever, variable length representation is attractive for transmission and storage, where the average compressed codeword length may be less than the fixed code word length.
English text may be machine described by fixed length code words from several different alphabets. For example, there can exist an alphabet of upper case characters, A, ~, ... , Z; an alphabet of lower case characters a, b, ..., z; or an alphabet of numbers and symbols 1, 2, 3,...,%, +, etc. It is possible to construct a fixed length code of length L whose capacity are =2L~ T = the number of upper case characters plus the number of lower case characters, etc. ~lowever, where R < T, then some fixed length codewards b(a) will be ambiguous. Indeed, the cost trade- ~ -offs may be such that the increase in length L of b(a) may be far more expensive t:han the use of mechanisms for resolving the ambiguities.
As previously mentioned, fixed to variable length encoding becomes attractive where the variable length codeword representation .... . . .

r~

1 is a compressed version of the former. Compression is achieved by employing certain statistical regularities connected with the source alphabet. The most often used regularity is the ordering of characters on a relative frequency of occurrence and assigning the shortest length codewords to the most frequPntly occurring characters. This can lead to the rarest occurring characters having very long codewords. The upper limit of code-word length is that of the register size. To avoid the necessity of long registers, those infrequent characters whose variable codewords require more than a fixed register's length would be transmitted with a specific variable length prefix followed by the character in the clear, i.e., not encoded. This means that the encoder output consists of frequent variable length codeword sequences and infrequent fixed length words with the special prefix.
The encoder output can be viewed as a serial, comma-free bit stream insofar as the variable length words are concerned.
Placing commas, separations, between the variable words would sharply reduce compression advantage.
The prior art is replete with many examples of fixed and variable length encoders. For example, Blasbalg, USP 3,237~170, describes an adaptive compactor that in effect varies the variable length code word assignments as the statistics of the relativ~ frequency of occurrence of the source alphabet change.
Wernikoff, USP 3,394,352~ applies each fixed length word in parallel to differently structured encoders. He uses the short-est codeword from among the plurality with a tag to per~it de-coding. ?
Although Blasbalg changes his output codewords, he still preserves a unique one-to-one relationship between each input and output character. The same can be said of Wernikoff. In the SA9-75-024 4_ , .:

1 latter case, the output tag identifies the encoder/decoder to be used. In contrast, the problem addressed by this invention is that of resolving ambiguous terms, first at the encoder and then at the decoder.
Summary of the Invention The invention is based on the observation that a Markov processor possesses a "remembrance" property and that its state-to-state transitions can be modified to reflect emperica11y derived statistics concerning runs of f`ixed and variable length code words.
This is used to establish context and identify the substring alphabet. For instance, the liklihood of occurrence that a present fixed length code word is an uppercase character given a run of K
upper case characters is small. Another instance might be that the expectancy of b(a) is assignable to an upper case character after a run of any length of lower case characters might also be quite negligible. These observations yield a series of decision rules for treating ambiguous characters as belonging to the same or diFferent alphabet as the run of preceding terms. The exact rules will vary as the imperically derived statistics and source alphabets vary.
The invention is embodied in an apparatus for generating one of several variable length codewords c(ak) and c(aj) responsive to each fixed length binary representation bk(a), b(a) being indicative of source alphabet characters, aj being a member of alphabet Al and a; being a member of alphabet A2. The apparatus comprises an r state register, state transmission means for changing a given present state Gp of the register to one of a preselected number of next states according to a selected one of external s;gnals s; and sj, 1 <p <r; a memory containing c(a),S
as a function of location b(a),G; and means responsive to each fixed length word b(a) and internal~state representa~ion G for .

"~,""_ _ .
,~ , . . . . . . .

r~ ,, s~

1 accessing the memory, and for applying the extracted variabl~
length word c(a) to the output and for also applying the extracted signal S to the state transition means.
The state transition means incl~des a map of the Markov processor states and transition paths~ This map models certain statistical regularities of runs of fixed co~e elements and the relative liklihood that one or more consecutive ambiguous terms appearing in a certain type of run belongs to ~ given alphabet and should be encoded/decoded accordingly. The processor starts from an arbitary ini-tial state and tracks any given run of b(a)'s applied thereto in terms of a succession of states and paths. Thus, for any given b(a) and present machine state G, then the alphabet A, c(a), and path S can be obtained by table lookup. c(a) C(A) represents the instantaneous encoder output. Significantly, the state transition means causes the register to change from its present state to a next state according to the map and the table extracted value 5 applied to said means.
This invention takes advantage of the fact that the number of actual machine states can be mapped into a smaller number of sta~es for representational purposes. This is due to the fact that a run of characters in the same alphabet will place a machine in one or another of the subset of machine states. This reduces the amount of memory and processing required to perform encoding/decoding.
In receiving a series of variable length codewords as a com~a-free bit stream" the decoder includes encoding logic and a Markov processor for both parsing the characters in the stream and for resolving whether a given c(a) represents b(aj) or b(aj). In this regard, the Markov processor traverses the same sequence of states as the encoder Markov processor when encoding the fixed leng~h word stream.
~: SA9-7~-024 ~5-.

1 Brief Description of the Drawing FIG lA dep;cts runs of alphabetically designated fixed length code words with and without ambiguous terms together with rules of action.
FIGS lB and lC show a finite state machine implernenting the rules of lA in both tabular (lB) and graphic (lC) forms.
FIG 2 sets forth a register for storing the identity of the present machine state and a state-to-state tracking mechanism.
FIGS 3A and 3B, respectively, represent the table contents of an encoding and decoding read only store (ROS).
FIG 4 sets forth the fixed to variable length word encoder and decoder in system relation.
FIG 5A is a logical depiction of the encoder emphasizing the register assignments and cycle definition of FIG 5B.
FIG 6A is a logical depiction of the decoder emphasi2ing the register and selector circuit assignments together with the cycle definition of F~G 6B.
Description of the Preferred Embodiment It is well known that variable length codewords C may be assigned, one to one, to each member of a given alphabet. The length L of the codeword is inversely related to the relative frequency of occurrence P of the term in the alphabet. Indeed, algorithms for minimizing the average length of the variable length codewords assigned to the terms of an alphabet are well represented in the art. See Raviv, USP 3,67 5,211; USP 3,6 75,212;
Cocke et al, USP 3,701,111. These are based in whole or in part on the work of Huffman9 "A Method for the Construction of Minimum Redundancy Codes", Proceedings of the IRE, September 1952, pages~l098-1011.
Referring now to Table 1, there are shown two arbitrary alphabets Al and A2. Each member aj o-f Al, symbolized ajEAl .
. ~ .
.

: .. ., .. . . . : , , , . . . ~ : . , . ~ . . .. . ..

1 consists of a number 1, 2, 3,... ,7. Likew-ise, aj~A2 consists of a lower case Greek letter. The relative frequency P(aj) is followed by an optimum length Huffman encoded variable length word C(aj). The length L(a;) of the codeword C(aj) is listed in the next column for convenience. Finally, the expected lenyth for each entry is formed from the product P(aj)xL(aj). Each Huffman code was independently calculated for each alphabet. Note, the minimum average Huffman codeword length L(aj)=0.34 bits per symbol while L(aj)=0.25 bits per symbol. It; follows that codes optimized in respect of their source alphabets are not optimized when strings are concatenated. Also to be noted is the length of the in-frequently occurring members.
aj~Al P(aj) C(a;) L(aj) R(aj)xL(aj) aj~ A2 P(a;) C(aj~ L(aj) R(aj)xL(aj) 1 0.20 10 2 0.40 t .10 000 3 .30

2 0.18 000 3 0.54 u .09 1010 4 .36

3 0.10 011 3 0.30 v .09 1011 4 .36

4 0.10 110 3 0.30 w .07 1001 4 .28 0.10 111 3 0.30 x .06 1000 4 .24 6 0.06 0101 4 0.24 y .03 11110 5 .15 7 0.06 00100 5 0.30 z .01 11111110 8 .08 Avg.=2 38=0.34 bits/symbol Avg.=l 77=0.25 TABLE I
Referring now to FIG lA, there is shown runs of alphabetically des;gned fixed length code words. For purposes of describing this invention, the runs, their statistics, and the source alphabets are wholly arbitrary. In the figure, each character aj is a member of alphabet Al an~d each character aj is a member of alphabet A2 - ' In the figure, runs of fixed length codewords representing characters in the same alphabet b(a;~, b(aj),...,b(a;) imply that , : - .. . . .: . .
.. .

- 3~ 5 ~

1 a machine should stay in the same alphabet for encoding/decoding the next character. Where a run is interrupted by a single "ambiguous character ? ", the machine stays in the same alphabet. In this regard, an "ambiguous character" may be indicative of a change of alphabet. For runs in alphabet Al interrupted by two ambiguous terms ? ? , then the machine changes to the other alphabet A2 for encoding/decoding the next character.
For runs in A2 interrupted by four successive ambiguous terms ? ? ? ? terms, then transfer must be made to the other alphabet Al.
Referr;ng now to FIGS lB and lC, there is shown a state trans-ition table and map of a finite machine suitable for recognizing the runs of interest. The machine must have a sufficient number of states to "remember" to a depth of four. This is to take into account the transition rule governing runs of characters ;n alphabet A2 interrupted by four ambiguous terms. The state transition table is formulated by assuming the apparatus exists in some initial internal state and then for each combination of runs, a new state is defined to which the apparatus should go from the initial state. Now, for each of the states into which the apparatus could have gone from the first state, this process of defining new states is repeated. A graphical version of the table is that of a map depicted along side thereof. The table and map may be read as follows. 3 Assume the apparatus is in initial state 1 and a character b(aj) is received for encoding, the machine would then move to state 2 over the path designated "i". I-f, ;nstead, b(aj~ had been received for encoding, the apparatus would stay in the same state , :~ ; 1.
30 ~ Starting~in state 1, i-f the apparatus received a succession of terms b(ai), b(aj).,,,.b(aj), then it would move through successive internal states 1, 2, 3, ~ to and remain in state 5.
SA9-75-0~4 -9-~' : ' r~ -1 Likewise, if the apparatus were in any other internal state and subject to a run of b(aj),...,b(aj), the apparatus would migrate back to state 1.
The paths connecting successive machine states are designated sj or sj. Because a run o~ terms in the same alphabet will cause the apparatus to migrate to either state 1 or state 5, the internal machine states are connectecl. This aspect will be dis-cussed in connection with the description of FIG 2.
Referring now to FIG 2, there is shown a register 7 ~or storing the identity of the present state of the machine and a state-to-state track;ng mechanism. In this regard, the contents of register 7 designate the internal state of the encoding/decoding apparatus and also serve to access the contents of a read only store 9. The read only store contents include the state transition table of FIG lB toyether with a binary indication of the next state the apparatus should go to given that the character to be encoded is b(aj3 or b(aj). Thus, if the machine were in state four and the character to be encoded were b(aj), then the next state would be state two. The difference between this and the state table in FIG lB lies in the use of the parameter "G". This parameter is indicative o~ the internal state of the machine and represents the state to which the apparatus would migrate to if presented by a succession of characters in one alphabet or the other as pointed out in the discussion with regard to FIG lB.
ROS 9 is accessed by the contents of register 7. The data stored at the address is transferred to data register 11. The next state inFormation is sent over respective paths 13 and l5, while the binary mapped representation of the internal state "G"
is conducted over path 17 to state register 19 and out o~ the register and state controller 23 over path 210 The binary indication ' .

1 on path 1 which represents path selection controls which next state address will be transmitted from register 11 into register 7. If there exists a zero on line 1, then the appropriate left hand column entry is selected. If a one appears on line 1, then the appropriate right hand column entry is selected.
Referring now to FIG. 4, there is shown an overall block diagram of a fixed to variable length encoder ~1 coupled to a variable to fixed length decoder 53 over path 49. The encoder broadly comprises an encoding ROS 33 and associate accessing means, a state register and controller 23 to provide the state to state tracking and an output shift register ~7 for placing the variable length codewords c(a~ on line 49. Decoder 53 includes a decoding ROS 65, shifting and accessing circuits, together with se1ection networks 68 and encoder logic and state control element 51'. Each fixed length decoded word b(a) is gated through the selection network 68 and applied simultaneously to output path 25' and element 51'.
The contents of encoding ROS 33 and decoding ROS 65 are representively set forth in FIGS 3A and 3B. The details of register and state controller 23 are found in FIG 2, while the selection networks 68 and 51' may be seen in FIG 6A.
Referring now to FI~ 3A, there is shown a set of concord-ance tables in which for each b(aj), there is to be found a corr-esponding c(ajO~ a length L and path S. The same holds for each : b(aj) and c(aj), L and S. Ambiguity may be found, for example9 between b(ai)=7 and b(aj) = 3 in that both c(a) = 110. Addition-ally, certain of the fixed code words such as b(aj) = 4, 5, 6 and b(aj) = O, 1, 2, 4, 7 have C(A) = 111. The prefixed 111 does not indicate ambiguit.y, but is a prefix denoting that the fixed code word b(a) immediately follows and has not been encoded.
A reciprocal situation is represented in the decoder ROS

,.' ,:

1 contents of FIG 3B. Here the sequence 111 is immediately denoted as being in fixed code. An additional column designates whether the c(a), b(a) of interest represents clear or compressed text.
Referring now to FIG 5A taken together with FIG 5B, there is shown the detail for the fixed to variable length encoder 51 with register and cycle definition. In this embodiment, the bits representing each fixed length word b(a) are assumed to be applied with parallel and word series. Upon detectiny the pulse leading edges on path 25, a signal is provided by leading edge detector 26 to initiate the clocking actions of clocking and distribution network 28. The major cycles provided by network 28 are shown in FIG 5B. In this regard, cycle A causes data to be gated into address register 31 and a preselected portion of output register 47 over paths 29 through AND gate 27. The time between energizing gate 27 defines the encoding cycle. As soon as the data is loaded in register 31, the read cycle of the encoding ROS 33 begins. That is, the data stored at location defined by b(a) is accessed and transmitted to data register 35 during load ROS data cycle B . Register 35 is partitioned so as to reflect one-to-one the variable length field c(aj), the length attribute L, the preferred path s; or Sj and the correspond ing fields with regard to c~aj).
Encoder 51 includes selection registers 37, 39, and 41.
Selection register 37 responsive to the G signal on line 21 selects which one of the two variable length words will be placed in output register 47 over path 45. If G = O, then c(al) is ; insented. If G = 1, then c(aj) in register 35 are transferred to register 47.
Selection tegister 39 selects that length attribute for controll1ng shift~counter 43 corresponding to the selected C(A) in register 35. The same can be said for selection register 41 :
~ SA9-75-024: ~ -12-` ` -, : . .. . : - : , ..

1 with respect to path S. After data register 35 has been loaded during cycle B and the appropriate variable length word and associated parameters have been selected, the output register ~7 is loaded during the load cycle C . The contents of register 47 are serially applied to line 49 by shifting a predetermined number of stages out by decrementing the contents of shift counter ~3 and providing a shifting signal over path 44 to register 47.
Referring now to FIG 6A and 6B, there is shown a logic diagram of the decoder emphasizing register and selected circuit assignments together with the cycle definition in FIG
6B. The variable length words c(a), or less frequently, the fixed length b(a) prefixed by 111, are serially received on path 49 and shifted into register 55. The contents of the predetermined number of positions in register 55 are loaded into address register 63 over path 61, while the remaining contents terminate in selector 77 in selection network 68 over path 59. Parenthetically, : the amount of shifts during each decoding cycle is determined by the contents in register 39' located in encoder logic in state :20 control 51'. The register passes a count which resides in shift ;: control 57 as communicated over path 40'. The count is decremented .
to 0, each decrement resulting in a shlft in the contents of register 55 from left to right by 1 position. As a consequence, ~: the contents of register 63 are governed by those predetermined:: ~ bit positions of register 55 that have been shi~ted under control ~ of L. The contents of register 63 then access the decoding ~:
,~ ROS 65. The contents~at the location defined by the address register include b(aj), C/C, b(aj) and C/C. As may be recalled, there is a one bit iDdication accompanying each b(a) indicating whether it represents a character transmitted in the clear or :~ compressed, i.e.~,~ C/C. ~:

:: ;, , 1 Selection networks 68 consist of registers and selectors for insuring that a preselected one of the two b(a)'s contained in register 67 is placed on path 25'. This, in turn, is dependent upon the binary value of state signal "G" obtained from controller 51' over path 21'.
Each derived fixed length word b(a~ is applied to an encod-ing ROS 33' in logic and state controller 51'. This permits the decoder 53 to be in the same state as that of the encoder when a given variable length c(a) or transmitted fixed length character was applied to path 49. Consequently, the state G on path 21' should be identical to that of the encoder at the point in time when c(a) was selected to represent a correspondong fixed length code word b(a).
ENCODING
Initial State=l b(a) S 6 0 1 0 Next State 1 1 2 3 4 5 5 5 Alphabet A2 A2 A2 A2 A2 A2 Al Al c(aj O 10 111000 111001 111000 111001 01 01 DECODING
Initial State=l c(a) 0 10 111000 111001 111000 111001 01 01 Alphabet A2 A2 ~2 A2 A2 A2 Al Al Next State 1 1 2 3 4 5 5 5 b(a) 5 6 0 1 0 : TABLE 2 The above table 2 illustrates the operation of encoder 51 and : decoder 53.
Re~erring now to all of the drawings, let us ~irst examine encoding action and assume that state controller 23 is in initial ~ .
~: :

1 state 1 with b(a) = 5. From FIG 3A, the correspondiny path is j. This, in turn, means th.at the next state as shown in FIG lB
is state 1 with the character being encoded in alphabet A2 where c(aj) = O. For the new character b(a~ = 6 and the present state being 1 with G = 1, then the path from FIG 3A is sj which from FIGS lB and 2 means that the next state is 1: Also, from FIG
3A, the alphabet is A2 with C(Aj) = 10 with a lenyth of L = 2.
The third fixed word b(a) = O with G = 1 means that from encoding table in FIG 3A, the path to the next state is Sj and that the text will be transmitted in the clear with a prefix 111 followed by 000. From FIGS lB and 2 given a present state of 1 with Sj, then the next state is 2. The remainder of the encoding table can be verified by the same procedure.
Referring now to the decoding table in table 2, assuming that the decoder 53 has received on path 49, c(a) = O. From the decoder ROS contents shown in FIG 3B, it may be observed that three bits are used to address the ROS. Thus, for 000 n, then b(a) = O or 5. This ambiguity can be resolved by recourse : to the fact that the output from controller 23' is state G, which we will assume = 1. This yields b(aj) = 5 and the path to the .
next state is j. Referring now to FIGS lB and 2, the next state is 1, the state ~alue G = 1, and the alphabet is A2. The : remaining portions of the decode table can be confirmed by repeat-; ing the above procedure.
. While the invention has been particularly shown and described ; with reference to a preferred embodiment thereof, it will be under-stood by those skilled in the art that changes in form and details may be made therein without departing from the spirit and scope of the invention.
. ~ .

~ SAg-75-024 -15-. . : .

Claims (2)

The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:

1. In an apparatus for generating fixed length codewords b(ai) and b(aj) responsive to corresponding variable length codewords c(ai) and c(aj), where ai and aj are source alphabet characters ai.epsilon.A1 and aj.epsilon.A2; the combination comprising:
means responsive to each variable length codeword for generat-ing an ordered m-tuple comprising at least two fixed length codewords;
a Markov processor responsive to each fixed length codeword from the apparatus for emulating the internal state of a fixed to variable length encoder and for generating an alphabet selector contro sig-nal; and a selection and distribution network responsive to the control signal for selecting the fixed length word from the m-tuple and apply-ing said selected Fixed length word to the apparatus output and also for applying said selected fixed length word to the input of the Markov processor so as to enable said processor to track the state-to-state transitions of the fixed to variable word length encoding apparatus.

2. In an apparatus according to claim 1, wherein the means for generating an ordered m-tuple of fixed length codewords includes a shift register; said Markov processor in addition to generating a control signal, also generates a length attribute and means for shift-ing the contents of said shift register by an amount proportional to :
the length attribute.