JPS61500345A - Data compression method and device - Google Patents

Abstract

(57) [Summary] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

[Detailed description of the invention] Data compression method and device Background of the invention The present invention is necessary for encoding alphanumeric data to store or transmit alphanumeric data. The present invention relates to a method and apparatus for reducing the number of signals. in particular when storing it in a computer device or transmitting it by a data communication device. Beneficial.

The prior art for encoding alphanumeric text is 8 bits, commonly called bytes. It relies on a method of substituting a binary code for each character of the text. This kind of One of the codes defines its characters based on the American Standard Code for Information Interchange (ASCII). as a parity bit (odd-even check bit) or set to zero. Consists of 8 bits determined by Here is a list of these codes.

For example, Ralston et al., “Computer Science and Engineering” 2nd edition of "Alling's Dictionary" (Van Nostrand Re1nhold, 198 3) Listed on pages 125 and 126.

However, 8 bits are used to represent each character in large volumes of alphanumeric text. This puts a severe strain on the limits of modern microcomputers and communication equipment. For example, the New Testament contains over 170,000 words and 1,03 Contains 6,000 separate characters. Therefore, remember the New Testament This requires data storage of 1 megabyte or more. With modern memory technology Even so, this type of requirement makes it relatively expensive to memorize entire texts and other writings. It takes. Similarly, it is also possible to transmit amounts of encoded text comparable to books. It is relatively expensive and time consuming.

Two-character combinations are used more frequently to reduce data storage and transmission requirements. The standard code has previously been modified to use a constant 8-bit code to represent came. In this way, the double letter "th" has one "t".

The other one is one 8-bit code instead of two 8-bit codes representing “h”. It can be expressed as

However, this technique is relatively limited in the data compression it can achieve. in general , the length of the binary code required to represent alphanumeric text can be reduced by approximately 40%. Wear. Summary of the invention based on the frequency with which paired characters appear in a particular text The amount of data compression that can be achieved when storing alphanumeric data using digital code formation. We have devised a technique that significantly improves this. According to our invention, alphanumeric text A dictionary is created that assigns each different class of and its associated punctuation marks a unique code.

Each class in the alphanumeric text is then replaced by the symbol that points to that word in the dictionary. 0 For example, each code is a series of binary digits whose 1i! 16 to identify or address contains up to 2 bits, so the dictionary can store up to 2" = 65,536 This is more than enough for memorizing words related to most books. is the amount of It takes only 2 bytes to identify any one of these 65,536 words Replace each type of text with 2 bytes of information, since only the following information is needed: If you do this, you can reduce the average number of numbers required to memorize text by about a third. Ru. If the dictionary contains more than 65.536 words, at least some The number of bits required in the code would have to be greater than or equal to 16; If the number of words in the dictionary is a power of 2 less than or equal to 16, then the number of bits in each code is greater than or equal to 16. Can be on top. The advantage is that dictionaries can be stored on conventional microcomputer devices. The text can be created very quickly using the can be reconstructed into a human-readable format by computer equipment.

The number of bytes required to memorize a dictionary is the number of bytes required to memorize words in alphabets and This can be substantially reduced by exploiting the redundancy of characters. thus, If the second of two mnemonic words contains the same five letters as the first word, then You can give meaning to a word by memorizing the one letter that represents the five letters of the word. ・Dictionary by using the same plural words, possessives, and cognates except for punctuation marks Due to the redundancy of a large number of words in the dictionary, this technique reduces the size of the dictionary by about a third. can be reduced.

To further reduce the length of compressed text, you can often use the most frequently used This is usually achieved by representing the word to be expressed in a code shorter than 2 bytes. Since more than half of all the words in the list are made up of the few most frequently used words, For example, 1-byte codes instead of 2-byte codes for the most frequently used words. By using In this case, it can be reduced by more than 50%.

The techniques described above achieve compression of critical data while preserving boundaries between words. new When we tested the New Testament with the King James Version of the Book, we found that these techniques could be used to convert the New Testament into Book 1. ．． 036,000 characters into 220,000 bytes using some compression method. can be stored in 183,000 bytes using other compression methods. there were. In addition, in a test conducted with approximately 900,000 characters in training materials for lawyers, was able to compress text to less than 150,000 bytes.

Dictionaries contain each class in alphanumeric text, so a particular word or several words can be You can check whether it is used in Furthermore, the words in the text Confirm that the location in each category in the dictionary points to each segment of text in which that word appears. It can be specified by adding a child. This feature allows text to To find words that occur in the same segment.

It is possible to compare confirmators that bind to different words.

Brief description of the drawing These and other objects, features and advantages of our invention relate to its modes of implementation. It will become clearer from the detailed description below.

FIG. 1 is a flowchart illustrating the general concept of one preferred embodiment of our invention. to.

FIG. 2 is a flowchart illustrating the embodiment shown in FIG. 1 in more detail; FIG. 3 is a flowchart illustrating details of FIG. 2, and FIG. 4 is a flowchart illustrating the details of FIG. a flowchart illustrating a second feature of the embodiment, and FIG. 5 is a diagram illustrating an exemplary apparatus used in a preferred embodiment of our invention. Ru.

Description of preferred embodiments of the invention As shown in Figure 1, the alphanumeric text in our invention first consists of By creating a dictionary that combines each class into a unique code of up to 16 bits (2 bytes). As is well known, using a pattern of 1 or O in 16 bits, It can represent any number from zero to 65,536. compressed text To create a list, each class is replaced by the symbol that points to that word in the dictionary. dictionary The size of is determined by memorizing the words in the dictionary at any time in alphabetical order, and the resulting It can be reduced by exploiting character redundancy.

Conveniently, the length of the compressed text is 2 times the most frequently used words. It can be further shortened by representing it with a code whose length is less than a byte.

Preferably, these means are performed by a conventional microcomputer.

In a microcomputer, special means for implementing the technique shown in Figure 1 are shown in Figure 2. is shown. Firstly.

The text of a book or other material to be compressed is converted into a linear list of words.

In practice, this means that a carrier return/line advance follows each type of text. request to be inserted into. For this purpose and for convenience, each category refers to the series in the text. All alphanumeric symbols are considered including one punctuation mark between consecutive spaces. this , a career return/line will occur every time a space appears in the text. The infeed is only inserted, and is part of the one space immediately before the alphanumeric text. If there are many spaces between possible words, select the previous space in the text. A pace is treated as a single word consisting of space characters.

Once the linear list is created, it can be classified using traditional classification methods, resulting in All words of the text are placed in alphabetical order.

The alphabetized list avoids memorization duplication and allows you to measure the frequency of use of each memorized word. Processed by a microcomputer to perform calculations. In this way, the alphabet The entire list of alphabetized words is added to each class from the first alphabetized list. and the first occurrence of that word in the alphabetized list. Replaced by a new compressed list specifying the current count.

For example, this procedure is performed as shown in FIG.

Each type of alphabetized list is then processed by a microcomputer. is removed. in this case.

Compare the word with previously retrieved words to see if it is a new word.

If the two words are the same, then the word in question is an old word and has a frequency counter of 1. 1ti11, and the next word is extracted from the list.

If the two words are different, the word in question is a new word and the old word and the content of the frequency counter are written to a new list.

The frequency counter is reset to 1 and stored for comparison of new words.

To create a dictionary, each class of the compressed alphabet is divided into its respective codes. Can be guessed. However, to reduce storage requirements, the length should be less than or equal to 2 bytes. code with more frequently used words using one of several techniques. For example, a 1-byte code should be assigned to the most frequently used word. can be assigned to. To do this, we first need to download the compressed alphabet. Make a copy of the strike and memorize it. The list of words and frequency calculations are then Create a new list in which words are sorted by a frequency counter and placed in order of decreasing frequency of use. obtain. In some technologies, one of the 8 bits of a byte is encoded with a 2-byte code. It can be used as a 1-byte code to identify that byte. like this , the other 7 bits of the byte are used to create 128 different codes. I can be there. If this byte is not identified as a 1-byte code, then 2 The remaining 17 bits in the byte code represent 32,768 different words in the text. It can be used for identification.

Therefore, in this technique, each of the 128 most frequently used words is 128 are assigned one of different 1-byte codes, and the remaining words have different 2-byte codes. Assigned.

Alternatively, the number of one-byte codes can be calculated as the number of different words used in the text. can be changed depending on. In particular, The maximum number of different words that can be used is X + 256 (256 - (indicates the number of 1-byte codes used). Obviously, X must be less than or equal to 256. or an equivalent positive integer. From this, the maximum number of 1-byte codes that can be used is The large number is x<(256”-Y)/255 (1) (where Y is a different word in the text ).

For example, if there are 12,000 different words in the text, then X=209.

Thus, the 209 most frequently used words are represented by 209 one-byte codes. The remaining 11,791 words are represented by 2-byte codes.

Therefore, when using this technique, equation (1) is the maximum number of one-byte codes that can be used. It is used to calculate . The number of this most frequently used word is then.

Can be assigned a byte code. The remaining words in the text are then assigned 2-byte codes. Can be used.

Whichever method is used to determine the number of one-byte codes, the through a computer by assigning a code to each class in a consecutive numerical order. A dictionary is created. The numerical order of these signs is from low to high, or from high to low. , but in the implementation described here it must be monotonically increasing or decreasing. , in the following description, the numerical order is upwards. The good thing is that 1 bar The words represented by the code are assigned to the first dictionary, and the remaining words are assigned to the second dictionary. Can be guessed. To minimize storage requirements, each class and 2 bytes, as detailed below, Since the second dictionary that combines the codes has at most 256 memory words, it is usually There is no need for this dictionary to be alphabetical, however, Since the words to be memorized are used very frequently in the text, it is important to minimize their search time. For this purpose, it is preferable to use text with the most frequently used words first. Store the words in order of frequency of use.

It is desirable that the dictionary to be stored contains only the words contained in it and does not contain any codes. For example.

A format of symbols encoded by ASCII, with one byte representing each symbol. Then, the words are memorized.

Since there are only 96 ASCII codes, one bit of each byte is used for other purposes. used. This bit is used to identify the beginning of each class. In particular, each category At the beginning of , set the 8 bits of its first ASCII character to “1”, while Identified by setting 8 bits of every other ASCII character in the word to “0” be done. As a result, the code associated with a particular word in the dictionary is Count the number of words up to the word and add that number to the number of the sign combined with the first word in the list. It's decided just by looking at it. This calculation is done by masking the 8 bits of each byte and scans each byte from the first word in the list to the word in question. This is simply done by counting the occurrences of each "1" bit in .

For example, if the initial dictionary contains 209 words, oooo ooo.

Codes with binary values from 11010001 to 11010001 are assigned to these words.

To determine the code assigned to a particular word, the computer checks the first byte or a dictionary starting with the byte immediately preceding the particular word whose sign value is to be computed It simply counts the occurrences of each 1 bit in the 8 bit positions of each byte in the 8 bit position. this Since the numerical value of the sign assigned to the first word in the dictionary is zero, the calculated value is the sign For example, the code assigned to a word in the second dictionary is the binary value 11010. It will start with 01000000000. Therefore, the value of its sign is the first Teach the word in the same way as the dictionary and combine the binary value 1101 with the first word of the second dictionary It is determined by adding 001000000000 to the calculated value.

To speed up calculation procedures, it is useful to have a study table that identifies the symbols that combine certain words. 0 For example, the code combined with the first word starting with each of the 26 letters of the alphabet can be memorized. and have the same first letter as the word whose sign is being calculated The calculation procedure can be started with the first word.

After the dictionary was created, a microcomputer could select each category from the first linear list. , look up the word in the first or second dictionary, and use the code obtained from that dictionary to form a linear Compress alphanumeric text by replacing words in a list. to this process , each category of the first dictionary is first examined and the ASCII codes of those words are is tested to see if it matches the word to be replaced by the sign, Calculate the number of each test that could not be confirmed. 1 If both match, disconfirmation The number of tests will indicate the value of that sign. However, the code combined with the first word The value is zero. If said match is not found in the first dictionary, the compiler The computer moves to the second dictionary. Here, use the survey table to find the starting point for the dictionary's survey. Add 0 For example, change the first letter of the word for which the code is specified to the first letter starting with that letter. can be used to find the word in the questionnaire.

The questionnaire will give you a numerical value for the sign of the word. Then, start with the above characters. Examine the different words in the class and see if the ASCII code for each class matches the class. confirm. The counter is incremented by one for each spoken word that could not be tested. And that If a word is found, combine the code of the word with the first word starting with the same first letter. Calculate by adding the calculated value of the counter to the value obtained from the search table of the matched code. This person In the method, the entire linear list of words is placed in a code list to create encoded text. It is possible to compress using phase detection. That is because the words in this dictionary are alphabetical. The words are arranged in blocks, and almost every word is the same as the word that precedes it in the dictionary order. Caused by containing at least one initial letter or common initial letter do. The first word preceded by the second word in sequence must also have at least the same letter as the initial letter. also contains two words, in order to express the second word, (1) the first word and (2) the number of the same letters compared to the first letter of the second word. A convenient method is to use a string of characters to indicate the remaining characters that are different from the first word. However, Each word in the dictionary is a number specifying the number of initial letters that are the same as the letters of the preceding entry word. and.

It is stored using ASCII codes representing the different remaining characters. Prompt processing In order to speed up the search, the number can be easily used when searching for the first letter of a single word. is stored in binary format. "storage" that appears consecutively in the dictionary as one column , "5tore", and "5tored". in this case , "5tore" is in "storage" preceded by the first four characters. &e” are different, so the binary number equivalent to 14″ and the ASCII equivalent to “e” expressed using letters. Also, “5tored”.

The first five characters are the same as the preceding “5tore” but the “d” is different, so it is compatible with 5#. It is expressed using the corresponding binary number and the ASCII character corresponding to "d".

After this are encoded texts, dictionaries, and W4 examination forms.

and a computer program that reads the encoded text on the tape, It is stored either on disk or on a suitable medium such as ROM. Also this the transmission of the same information from one location to another through a data communication system You can also do it. When data is compressed using our invention, it is possible to compress data using a full-scale volume. ゛The complete text of a book can be printed on one or two 5 1/4 inch (13 mm) blocks. It is possible to store it on a DVD disk. Generally, the length of the text is one word. By replacing it with a code, it can be reduced to about 60-70%. . It also assigns 1-byte codes to words that are used very frequently in the text. By doing so, it is possible to achieve a further reduction of 25%, and in some cases up to 50%. I can do it. Therefore, by actually implementing the present invention, the length of the text can be reduced completely. The body can easily be reduced to about 75%. Dictionary is the length of the text Although it is an obvious fact that the length of the second dictionary increases By representing the same initial letters of successive words using numerical codes, can be done. This reduces the length of the dictionary by using about 3 factors1 The amount of compression that can be achieved by practicing the invention is illustrated in Example 1 below. Also our The invention can be used to improve the transmission capacity of the channel required to transmit this type of text. A reduction in quantity can also be achieved in practice.

The flow chart in Figure 4 can also be used to generate data from encoded text using a computer. This diagram illustrates how to restore alphanumeric text with . shown in this As shown in the figure, a computer sequentially extracts each individual code and calculates the information associated with that code. To find words.

Search one of the dictionaries. In the case of a 1-byte code.

The computer only needs to load the binary value of that sign into a counter; read the words in the first dictionary in order, starting with the most frequently used word, and Decrement the count by 1 for each byte whose second bit position is “1” bit. and continue until the value indicated by the counter becomes zero. At this point, the second word to be read is , corresponds to the word represented by the code initially loaded into the counter. second dictionary When searching, the computer searches for codes starting with individual letters of the alphabet. Make effective use of the questionnaire relating to the first word. In other words, the computer Run the survey table in reverse order, subtracting the sign values in the table from the sign values to be converted to text. The moment the difference between the two values changes from a negative value to a positive value, the The computer searches for the first word that starts with the same letter as the word represented by the code. reach. This allows the computer to convert this sign value to text. The same thing as subtracting the sign value and reading the bytes of different words starting with that letter. Repeat the process 1 for each byte where the 8th bit position is a “1” bit. Decrease the count by 1.

Continue until the count reaches zero. The word examined in the first order at the time of zero is determined by the code. It corresponds to the word identified as The search is performed from either the first or second dictionary. However, after this, the word 1 displays, printers, or other types of computers. data is sent to the data output device.

The computer then advances to the next code.

Margin below Our invention can be applied to any system executed by a computer. It can be applied to the following systems. Performs text encoding and encoded Appropriate equipment is suitable for the process of recovering original alphanumeric text from text. Any computer can be used as long as it runs on a suitable program (as shown in Figure 5). Generally, such a computer includes a processor (10), a first and a second Memory (20 and 30), keyboard (40) and cathode ray tube CRT (50) It is composed of Such devices also include a printer (6) as an optional feature. 0) and a communication interface (70). These devices are 1 The microprocessors are interconnected by a data bus (90) as shown in the figure. It is controlled from the sensor (10) through a signal line (90). Furthermore, the memory can be addressed by the line (100). The configuration shown in Figure 5 is This is generally accepted as the organization of a regular microcomputer. dictionary A program that creates and encodes alphanumeric text is stored in read-only memory for convenience. can be stored in a first memory. The same device can also be used to If you also use the program to restore alphanumeric text from The encoded text created by 0 can also be stored in memory (20) , along with a dictionary and a lookup table, typically stored in memory (30). Also the restoration program RAM should be stored in memory (30) if memory (20) is not available. Can be done. These encoded texts, dictionaries, questionnaires, and restoration programs The RAM communicates with another microcomputer at a remote location through a communication interface (70). It is also possible to transmit data to a computer.

Memory (30) can be programmable read-only memory (FROM) or magnetic tape. , or a floppy disk device; this is because these devices If the storage has a large enough capacity to store the entire text of a book in one reasonable size. This is because it can be stored on a small number of floppy disks or a small number of floppy disks. Ru. Also, when using FROM, the text encoded in FROM, the dictionary A suitable device (not shown in the figure) for recording such devices are generally well known. . The present invention can also be used when it is desirable to store a large number of books in one record. In order to do this, a fixed disk device with extremely large capacity is required.

8 The device in Figure 5, which can use large ROM boards, is stored on disk. When used to recover the original alphanumeric text from It is convenient to transfer the entire contents of the disk to a semiconductor memory for processing. Semiconductor metal Mori accelerates dictionary word lookups and reduces processing time in order to perform processing at very high speeds. can be shortened. Also for this purpose, a normal microcomputer It is convenient to compress the dictionary to a size that fits the memory storage capacity of be. We did this when we had 64 kilobytes of semiconductor memory available. I implemented it and confirmed that it was effective.

Our invention can be applied in a wide range of fields. mentioned before The present invention can be used to compress alphanumeric text for data storage and transmission. can be done. It also allows for quick restoration of the original text. , compressed data can be used in various applications that were previously done using the original text. It can be erected. Also, compressed data is completely useless without a dictionary. generate encoded text and dictionaries and use them for storage and transmission purposes. This separation allows for reliable storage and/or transmission of alphanumeric data. It can be carried out.

Dictionaries contain individual words of alphanumeric text, but they are relatively short. This dictionary can also be used as an effective tool when searching for information. Ru. Among other things, you can simply scan the dictionary to see if a particular word is used in alphanumeric text. You can easily check whether the Also, for individual words in the dictionary, Add identifiers to specify each segment of text in which the word appears, and then For example, if this identifier is 1 byte long and that byte is Assume that each of the eight bit positions in the text corresponds to one of the eight segments of the text. Let's try to set it. In this example, a 1 bit in any of the 8 bit positions of the byte. , it indicates that there is a related word in the corresponding segment of the text. Ru. By using identifiers in this way, the alphanumeric characters surrounding the word in question This can speed up searching for text. It searches for categories in which no words appear. This is because there is no need to search.

Additionally, by comparing each bit of the identifier related to different words, to see if the words are used in the same part of the text.

Obviously, identifiers are necessary to discover word usage more precisely, so 2. You can change the size.

In the practice of our invention, many variations are also possible. Until now, we Invents alphanumeric text, binary digit tokens and ASCII codes in words However, this invention can be practiced using all methods of symbols, and Symbols can be tokenized and encoded in various ways.

For example, foreign languages, mathematical symbols, graph symbols, punctuation marks, etc. can all be practiced with this invention. and these symbols can be ASCII, Extended ASCII or It can be represented by any code chosen.

Although the use of binary tokens is preferred in our invention practice, we Although it may be preferable to represent it in other bases, such as hexadecimal, this invention also can be practiced using tokens with digits of any radix.

We use codes of less than 2 bytes to remember frequently used words. Here are two examples of how to reduce the size of tokenized text. It's here. However, many other techniques can be used. For example, most books The number of terms used in this is clearly smaller than the 65,536 words that can be expressed in 16 bits. Inode.

Each of the words in the alphabetized text.

In most cases, it is possible to express with 16 bits or less.

For example, 32,768 words can be represented in 15 bits, and 16,384 words can be represented in 14 bits. It can be expressed as So other ways to assign tokens to bits represents each different word by a different token with its lowest number of bits. Calculate the lowest number of bits that can be ignored, then have that lowest number of bits. The idea is to assign a different token to each word. If used If the number of words is 65,536 or more, the same principle applies.

17. Tokens of 18 or more bits for each different token in the text can be assigned to words.

An alternative approach is to reduce the token with two fields. Ru. This first field is a fixed-length field that specifies the length of the second field. It is old. In this technique, the tokens are strictly used to calculate the frequency of each word. Therefore, the shortest tokens that are assigned to words are the ones that occur most often in the text. The 5th shortest token assigned to the next most used word is It will be assigned. In this method, the dictionary, in order of frequency calculation, The most frequent words are memorized first in the dictionary.

It will be remembered.

With this technique, one token can have a length of 12 bits. but , when the frequency distribution of words follows a very steep curve, as is often the case, the text The average number of bits required to represent each word in the text is can be significantly reduced, as in the case of The tokenized text is divided into two When stored using tokens with fields, the tokens are stored in a parallel list. one of the lists is a list of only the first field, and the other is a list of only the first field. The first list is the second field list. It is convenient to do as follows. Ru. The data is stored in the same order in the two lists. Therefore, tokenization To convert the text into the original alphanumeric text, the calculator uses the first file Read 4 bits from the field and read the second field list from these 4 bits. Determine the number of bits to read from Count the words from the beginning of the memorized dictionary and find the alphanumeric value associated with that bit. Discover the letters. In this way, the most used words are represented by 0000 in the first list and zero bits in the second list; The two most used words are 0001 in the first list.

Also in the next list they are represented by 1 bit; the next four words are represented by 1 bit in the first list. is 0010 and is represented by 2 bits in the second list. That's how it goes.

If the computer reads 0000 in the first list, these bits will be read in the second list. indicates that there are no entries in the dictionary, so the calculator It searches for the most frequent word, which is the first word of the word.

When the calculator reads 0001 in the first list, it reads the next bit in the second list. Read, depending on whether the bit in the second bit is zero or one, the second or or a third word.

mentioned above. The technology of memorizing each word in the form of a token also It can be extended to the storage of loops (i.e. phrases). Ordinary phrases are recognized by all. be known. "rof the", rand theJ and rto theJ.

It seems to be used fairly frequently in most English alphanumeric texts. So The phrase ``u'' is automatically assigned one place in the dictionary, but has only one tone. A kun is reserved for one occurrence of one such phrase.

Conversely, one phrase scans alphanumeric text and identifies a subset of the most frequently used words. You can compare words and easily discover 0 for example, the most frequently used 1 00 words may constitute this sub-set. This step describes the most frequently used A phrase with a high easy to assemble by testing each word of the text. I can do it. If it is not the most frequently used word, the next word will be extracted. It will be done. If so, then the word is on the list of most frequently used words. It is used with the previous word in . Finally, the most frequent If you come to a place that is not on the leaf list, the memorized word will not be added to the list of 4 phrases. It will be done. Once the entire text is scanned, the memorized list of phrases is sorted into alphabets. The usage frequency of one phrase is calculated by removing duplicates and calculating the usage frequency of one phrase. Depending on the number of tokens that can be used to represent Starting from things, these phrases are assigned; these phrases are assigned to other It replaces the phrase in the text before the phrase is assigned. dictionary and toe Does the token represent a single word from the point of view of the converted text? It makes no difference whether it represents a group or not. Therefore, 1 original alphanumeric text is 1 figure It can be easily reconfigured by following the process in step 4.

Example 1 In our inventive exercise, we created a dictionary that related tokens to each word. the New Testament by replacing each word in the book with that token. I memorized the entire book. To reduce the space required to memorize dictionaries, memorize almost all of them alphabetically, starting with the first letter of the preceding word in the dictionary. compressed using a numeric code to represent the same first letter.

In our initial effort to memorize text in tokenized form, we One-byte tokens were used to represent high-level words. In the book of the New Testament, there are about 1 There are 4,000 different words, so we have selected about 200 of the most frequently used words. The remaining 13,800 words are represented by 2-byte tokens. I did. With this method, approximately 65% of the New Testament's 170,000 words are written in one-byte chunks. It is expressed in . Using this one-byte token, we create the New Testament The entire book's 1,036,000 characters can be stored with approximately 220,000 bytes of storage capacity. I remembered it.

To further reduce memory requirements, two-field talk of the type described above It has been found to be advantageous to use In particular, the New Testament is the most The five most frequently used words, the number of times they are used, and the number of times each word is used. It is clear from Table 1 that the tokens used to represent leaves, etc. The frequency curve of words is very steep. 0Table 1 ■Token ■Words ■ Number of uses 2. By using field tokens, you can read the text of the New Testament. This reduces the number of bytes required to store the entire file to approximately 183,000 bytes. I was able to do it.

Example 2 The operation of the general technique of Figure 1 is explained in Matthew 2 from Chapter II. -Can be explained using poem 3: (omitted) According to the invention, we create one dictionary in which each word has one token.

■Table■ Here, a list is created to organize all the words in the text, as shown in Table 1 ■. It is classified as alphabetical.

■Table■ The list of alphabets is then processed to remove duplicate entries and compiled into one table. As shown in ■, the frequency calculation for each entry is calculated.

■Table■ In the preferred form of the invention, there is a list of words and frequency calculations.

So to get a new list where the words are organized in order of decreasing frequency of use, , classified by frequency calculation. The text in Example 2 is very short, so it is Categorize the list by using smaller tokens to represent frequently used words. However, as emphasized above, such a classification This is useful if the strike size is fairly long.

Individual words are then sequential entries in an alphabetized list of words. tokens with increasing numbers assigned to It is. In this way, the assignment of tokens to words in Example 1 and 2 is shown in Table 1■ It becomes like this.

■Table V In this example, it takes only 6 bits to uniquely identify each different word. It is clear that it is necessary. Obviously, the number of bits is tokenized varies depending on the number of different words used.

Finally, the calculator uses Replace each word in the linear list in table ■ with the corresponding token shown in table ■ It's possible.

■Table■ In Tora 2, there is not much point in compressing the word dictionary.

However, in large texts where many words have the same first letter, the dictionary When all the first letters of one word are the same as the first letters of the preceding word.

It can be compressed by replacing those characters with a single number.

The reconstruction of the original text consists of 1 each token at a time, as shown in Figure 4. until it is read, finds the equivalent word, searches for it, and prepares it as the appropriate output. This is achieved by being used for counting through the dictionary.

As mentioned above, dictionaries can also be used to determine where one word is used in alphanumeric text. It can also be used in information retrieval to indicate that In this application, the word Using an identifier to indicate a portion of text that is Speed up your search for that word.

In the case of the New Testament, there are four Gospels, Act of the Apoc. ttes, Apocalypse, Pauline Epistles and nov-Pauling Each of the Epistles / J (1) Identification 1 This can be done with byte identifiers.

As will be apparent to those skilled in the art, there are many variations to the invention described above. shape is possible.

Figure 1 End J t2 diagram Sai 3rXJ E 4 Figure Sai 5 diagram Procedural amendment document December 6, 1985 Mr. Michibe Uga, Commissioner of the Patent Office 1. Indication of incident PCT/US841016672, title of invention Data compression method Law and Apparatus 3, Persons Making Amendments Relationship to the case: Patent applicant Name Text: Sciences Corporation 4, Agent Address (100) Marunouchi-5-1-3, Chiyoda-ku, Tokyo; Target of amendment: Special Documents and power of attorney as well as specifications and claims pursuant to the provisions of Article 184-5, Paragraph 1 of the Permission Act. Engraving a range of translations F. Contents of amendment: Patent applicant's document pursuant to the provisions of Article 184-5, Paragraph 1 of the Patent Law Fill in the name of the representative in the column.

international search report

Claims (27)

[Claims] 1. In a machine-based system for storing or transmitting text, said text is Associating one different token with each different word or group of words in the text , the average number of digits needed to represent the token is Create a dictionary with fewer digits than the average number of digits needed to represent said word in step, and each word or group of words is identified by the dictionary as the word or group of words. Replace the tokens associated with a group of words, thereby representing the said text. The number of digits required to A method for compressing files. 2. The method of claim 1, wherein the text includes alphanumeric symbols and phrases. Consists of comma words. 3. The method of claim 1, wherein each word is a continuous space in the text. A string of alphanumeric characters and a single symbol, such as a punctuation mark, located between spaces. Ru. 4. In the method according to claim 1, the step of creating a dictionary includes A step that instructs the text words in block notation to form an alphabetical arrangement. and remove all duplicate words in the alphabetical list and simplify the the step of creating a simplified alphabetical list; Each step consists of assigning a different token in the list. 5. In the method of claim 4, each different token is one different token. have numeric values and represent different tokens in the reduced alphabetical list. The step of assigning words is to alphabetize different tokens in consecutive numerical order. It consists of assigning words in different orders. 6. In the method according to claim 4, the step of creating a dictionary further includes the following: It consists of two steps: the first is to identify the most frequently occurring words in the text. The other is to determine which words appear the most, and which words appear less frequently. This means assigning tokens that are shorter than the tokens assigned to the given word. 7. In the method according to claim 6, the step of allocating the token comprises the step of allocating 1-byte tokens to the first 128 most used words; and assigning tokens longer than one byte to the remaining words. 8. In the method according to claim 7, the tones assigned to each word are The first byte of the token indicates whether the token is 1 byte long or longer than 1 byte. It has one bit position containing the bit. 9. In the method set forth in claim 6, the step of allocating tokens includes the following: It consists of steps: first, the remaining words are represented by two-byte tokens. 1-byte talk that can be used to express the words most often used when being spoken to The second is to calculate the maximum number of words that are used in large quantities. The third is to allocate a 1-byte token and the third is to assign a 2-byte token to the remaining words. The first step is to allocate a token. 10. In the method according to claim 4, the step of creating a dictionary further includes the following: Consists of steps: in an alphabetized list to create a frequency number. Counting the duplicate entries of words, then following the frequency calculation for each word. Sorting the compressed alphabet list, the last, most appearing Tokens that are shorter than the tokens assigned to words that occur less frequently in words It is to allocate. 11. In the method of claim 10, the step of allocating the tokens comprises: It consists of the following steps: one token with two fields for each word. but its first field is of fixed length and the length of the second field is and talk according to the frequency calculation for each word as mentioned above. tokens can be assigned to words, so the shortest tokens appear most often in the text. The next shortest token is assigned to the next most occurring word. , and so on. 12. 12. The method of claim 11, wherein the first field contains four binary Numbers have equal length. 13. In the method according to claim 4, the step of creating a dictionary further includes the following: Consisting of steps: first, to a different token with its lowest number of bits. Therefore, we need to calculate the minimum number of bits required to represent each different word. and the second is to represent different tokens with the lowest number of bits in different words. It is to assign to leaves. 14. The method according to claim 1 further comprises the following steps: How many times in both words are the first letters of the word the same as the first letter of the preceding word? Compresses the dictionary by replacing it with a single number indicating whether the first character of That's true. 15. In the method according to claim 1, the text is divided into multiple segments. However, the means of creating a dictionary is based on the way the word appears in each different word. means for providing an indicator specifying the portion of the text to be displayed. 16. A dictionary produced by the method according to claim 16. 17. A dictionary produced by the method according to claim 1. 18. One dictionary for each different word or group of words in the text. In systems using machines that attach signals such as these, text can be reconstructed from these signals. The method to create consists of the following steps: extracting the next token from the above signal. , find a word with the above token in the dictionary, and put the above word into the machine The use is to prepare the output of the system. 19. Machine-based systems for storing or transmitting text compress or transmit text. The method consists of the following steps: A different token for a word or group of words to represent that word in that system. The average number of digits required to represent that token is less than the minimum average number of digits required to represent that token. Create a dictionary with a low number of digits, then find out how many digits are needed to represent the text. of the word or words in the dictionary to create a compressed text that can be reduced. Place each word or group of words with a token given to the group. and extracting the next token from the compressed text, or Find the word with that token in the dictionary, and then use the machine is to prepare that word for output. 20. The method of claim 19, wherein the text includes alphanumeric symbols and punctuation. Consists of dot words. 21. In the method according to claim 19, the step of creating a dictionary includes the following step. consisting of: in alphabetical order to create an alphabetized list. ordering the words of the text, then the compressed alphabet list Remove all duplicate words in the alphabet list to make Assigning different tokens to different words in the shortened alphabet list And so it is. 22. The tool for compressing text consists of the following means: Different tokens for different words or groups of words, that word in that system the average number of digits needed to represent that token less than the average number of digits needed to represent it One way is to create a dictionary that gives the number of digits, etc., and then A token with a word or group of words that reduces the number of digits required for The idea is to replace each word or group of words in that dictionary. 23. The device of claim 22, wherein the text includes alphanumeric symbols and punctuation. Consists of dot words. 24. In the device according to claim 22, each word may be an alphanumeric character or a phrase. A string of symbols between consecutive spaces in text, such as commas. 25. In the device according to claim 22, the means for creating a dictionary comprises: Sort words in text alphabetically to create an alphabet list Alphabet list to create a condensed alphabet list, a means of ordering means to remove all duplicate words in the A means of assigning different tokens to different words. 26. In the tool according to claim 22, the means for creating a dictionary further comprises: consists of: a means of determining which words are used most often in a text, and Tokens assigned to words that occur more often are shorter than those assigned to words that occur less frequently. A means of allocating new tokens. 27. In the device according to claim 22, the text is divided into the popularity of the parts. The means of creating a dictionary is to identify the parts in which the word appears for each different word. It consists of means for giving an indicator to specify where.