Classifications

• • H—ELECTRICITY
  • H03—ELECTRONIC CIRCUITRY
  • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
  • H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
  • H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/901—Indexing; Data structures therefor; Storage structures
  • G06F16/9014—Indexing; Data structures therefor; Storage structures hash tables

Definitions

• the invention relates generally to computer system storage, and more particularly to an improved method and system for storing data associated with a table.
• a large table of entries may contain a relatively smaller subset of non-contiguous entries that is supported by a particular application.
• Unicode is a world-wide character encoding standard based on a 16-bit unit of encoding. With Unicode-configured products supporting only the code points used in a specific language, a smaller, non-contiguous subset of that language's code points are used.
• JIS Japanese Industrial Standard
• a hash table may be used to store the attributes, indexed by a hash value computed from the code point.
• One type of hash table is a hash table with collision resolution. This type of hash table must store information to handle collisions, usually accomplished by storing the key with each entry. Extra time is required to test for collisions, and even more time is required to resolve a collision when a collision is detected. Lastly, this type of hash table typically frequently winds up with a number of extra, unused entries, i.e., the table is not always densely packed. Lastly, perfect hash algorithms (with no collisions) provide desirable performance in many applications, as they can provide a good tradeoff between memory usage and speed.
• Yet another object is to provide the method and mechanism as characterized above that densely packs together the hashed values.
• Another object is to provide a method and mechanism of the above kind that provides for a very fast computation of the hash value.
• a related object is to provide the method and mechanism that operate in an automated manner.
• Yet another object is to provide the above method and mechanism that is fast, reliable, cost-efficient, flexible and extensible to various applications.
• the present invention provides a method and mechanism for converting a non-contiguous subset of values in a large range into a contiguous or mostly contiguous smaller range with a perfect hash.
• a subset of values is selected from among the values in the range, and the range is organized into a two-dimensional matrix.
• One dimension of the matrix represents pages, and the other dimension represents offsets into the pages.
• the matrix includes mapping information such as a bit indicative of whether the value represented by that intersection is a selected value in the subset, e.g., the bit equals one if selected in the subset or zero if not.
• the matrix is then converted into a one-dimensional array (e.g., bitmap) by selecting each page, testing the mapping information (e.g., one-bits) on each page for conflicts with mapping information in the one-dimensional array, and shifting each page an amount until no conflicts are detected.
• the page mapping information is overlaid onto the one- dimensional array at the shifted position, and the shift amount is recorded in a page offset table.
• the value is separated into a page number and its offset into the page. The value is thereby hashed into the value of the subset range by looking up the shift amount for the page and adding the shift amount to the offset into the page.
• FIGURE 1 is a block diagram representing a computer system for creating a perfect hash in accordance with present invention
• FIG. 2 is a block diagram representing an exemplary table of possible entries
• FIGS. 3 - 4 are block diagrams representing alternative ways of viewing selected entries of the table of FIG. 2;
• FIGS. 5 - 6 are block diagrams representing bitmap matrixes for storing information about the selected entries
• FIGS. 7 - 10 comprise block diagrams sequentially illustrating the overlaying of pages from a bitmap matrix into a one-dimensional resultant bitmap in accordance with one aspect of the invention
• FIGS. 11 - 14 represent offset values in a page offset table as the pages of the bitmap matrix are overlaid one another in accordance with one aspect of the invention
• FIGS. 15 - 18 are representations of a hash table being filled in with entry information
• FIG. 19 is a flow diagram generally representing the steps for creating the perfect hash in accordance with the present invention
• FIG. 20 is a flow diagram generally representing the steps for overlaying pages of the bitmap matrix to create the perfect hash in accordance with the present invention
• FIG. 21 is a block diagram representing a computer system for computing a hash value from a key in accordance with another aspect of the present invention.
• FIG. 22 is a flow diagram generally representing the steps for computing the hash value from a key.
• FIG. 1 there is shown a computer system generally designated 20 for re-mapping non-contiguous values using an offset table in accordance with the present invention.
• the computer system 20 is shown a computer system generally designated 20 for re-mapping non-contiguous values using an offset table in accordance with the present invention.
• RAM random access memory
• non-volatile storage such as a hard disk-drive, optical drive or the like.
• the non-volatile storage 28 can be used in conjunction with the RAM to provide a relatively large amount of virtual memory via well-known swapping techniques.
• ROM read-only-memory
• embedded systems can benefit from the present invention .
• the computer system 20 also includes at least one input device 30, typically a keyboard and/or a user pointing device such as a mouse, connected through input- output circuitry (I/O) 32 for communicating user commands to the processor 22.
• I/O input- output circuitry
• at least one local output device 34 such as a computer display monitor and speakers are connected to the I/O 32 for communicating information, such as via a graphical user interface, from the processor 22 to the user of the system 20.
• a re-map process 36 converts the subset into a contiguous, or mostly contiguous, zero-based range.
• the large range of values may correspond to indexes ranging from OOOOh (hexadecimal) to Offffh, for indexing a large table 38 of character attributes, e.g., the indexes are Unicode code points.
• OOOOh hexadecimal
• Offffh Offffh
• the re-map process 36 essentially takes the entries which are of interest and re-packs them into a mostly dense hash table 40, in a manner that allows the original table key to be received and converted to a hash value index to the mostly-dense hash table 40.
• a table of offsets 42 is generated which, as described below, is used in conjunction with the original index value to compute the hash value of the mostly-dense hash table 40.
• a first aspect involves the conversion of the subset of non-contiguous values, such as the keys of the large table 38, into the offset table 42.
• a second aspect of the invention involves the subsequent use of the offset table 42 by an operating system, driver, application program or the like, to hash a key of the large table 38 into a hash value, e.g., a hash value index to the mostly-dense hash table 40.
• the mostly-dense hash table 40 may be loaded into virtually any type of computer- based device, to provide a table of Unicode character attributes or the like.
• the hash value to the hash table 40 is computed and the attributes found therefrom.
• FIG. 2 provides a table 38 of sixteen entries, of which only certain entries are to be remapped into a mostly dense hash table 40. More particularly, the entries indexed by 0000b, 0001b, 0010b, 0110b, 1000b and 1111b are of interest to an application program. For example, as conceptually represented in FIG.
• steps 800 - 804 the indexes of the larger table 38 are selected and laid out in a bitmap matrix of rows (pages) and columns.
• FIG. 5 shows a four row-by-four column matrix 44 for the sixteen entry table.
• Those indexes of the entries which are of interest, corresponding to the entries in FIG. 3 have a corresponding bit in the bitmap set equal to one, indicated by an "X" in the bitmap matrix 44 of FIG. 5 (for purposes of visibility and to avoid confusion with the ones and zeroes of the indexes), while all other bits are zero (or blank) .
• an initial matrix may, for example, consist of 256 rows (8 bits) and 256 columns (8 bits) .
• row zero is set with bits equal to one for any desired Unicode characters between OOOOh to OOFFh, row one for desired characters in the range 0100 to Olffh, and so on.
• each row may be considered a page, numbered from 00b to lib, whereby the column number is an offset into the page from 00b to lib.
• the columns may be used as pages, with the row numbers representing the offset into the page.
• the original four-by-four matrix 44 of FIG. 5 is conceptually restructured into a new matrix 45, such that the columns of FIG. 5 become the pages, (shown horizontally in FIG. 6), and the rows become the offsets into the page (shown vertically in FIG. 6) .
• the pages are overlaid atop one another, one at a time, into a dense or mostly dense resultant one- dimensional bitmap 46. Since the selected indexes (as represented by the one-bits in the bitmap) are noncontiguous and are typically widely scattered, the bits on the several pages tend to interlace with no conflicts, or at least very few conflicts. Whenever a bit representing an index does conflict with another bit, the current page being overlaid onto the resultant bitmap 46 is shifted one character to the right. When the page interlaces perfectly with no conflicts, the shift amount is recorded as an offset for that page in the page offset table 42.
• a less expensive way to obtain a reasonable order for overlaying the pages is to rank the pages such that pages having more one-bits thereon are overlaid into the resultant bitmap 46 before those having less one-bits (i.e., a most to least ranking) .
• a ranking process is called at step 806 to count the one-bits on the various pages and rank the pages according to the count.
• Such a one-bit weighted ranking process has been found to provide an extremely dense packing. Indeed, when other optimizations (described below) are combined with this ranking process, packings of one-hundred percent (or very nearly one-hundred percent) are achieved in a rapid manner.
• the first page which starts the hash table is preferably selected to be the first page in the bitmap matrix 45, regardless of the number of characters thereon.
• selecting zero as a desired index and starting with this page as zero ensures that a zero index in the larger table 38 always maps to zero, which is useful when dealing with null terminated strings and the like.
• FIG. 7 shows the overlay process, corresponding to the flow diagram of FIG. 20, starting with the first page of FIG. 6 at step 900, pageoo, since pageoo includes the zero index. As shown in FIG.
• step 900 directly continues to step 908 since there can be no conflicts at this time and, assuming zero is selected, there are no leading zeroes on the first page.
• step 900 directly continues to step 908 since there can be no conflicts at this time and, assuming zero is selected, there are no leading zeroes on the first page.
• any table data corresponding to each one-bit on the pageoo will be written to the hash table 40 (such as at a step 814) .
• the "A" attribute data will be written to the position indexed by a zero
• the "I" attribute data to the position indexed by a two, as shown in FIG. 15.
• Step 914 tests for another page, and since one is available, returns to step 900 where the next page, pageio, is selected.
• FIG. 8 represents the overlaying of the next page, pageio, onto the previous resultant bitmap 46 a , which produces a new resultant bitmap 46 b .
• pageio is the next to be overlaid because the ranking process of step 806 determined that this row had the most bits set to one thereon (two) compared to the remaining rows.
• step 902 the pageio is first shifted left an amount equal to any leading zero bits. In this position, a test for conflicts is performed at step 904.
• One such test for conflicts is accomplished by AND-ing the bits of the current page (pageio) to the previous resultant bitmap 46 a , and testing for a zero result. If zero, then there are no conflicts. If nonzero, a conflict is detected, whereby at step 906 the page is shifted right one bit and the conflict test repeated. The shifting continues as necessary until no conflicts are detected.
• the conflict detection and resolution processes of steps 904 - 906 shifts pageio three positions to the right of zero, (as best represented in FIG. 8), before the page is combined with the previous bitmap 46 a .
• the offset of three for this page is recorded in the page offset table 42, as best shown in FIG. 12.
• the array data e.g., the "C" and "G" attributes, will be added to hash table 40 at the appropriate locations, as shown in FIG. 16.
• the resultant bitmap 4 ⁇ b is now five bits in length, and the hash table 40 will have to have space allocated for at least five entries.
• the resultant bitmap 46 grow as necessary to accommodate the shifted bits, as will the hash table 40, however trailing zero bits (blanks) are not considered part of the resultant bitmap 46 since only ones represent (indexes to) entries of interest. Note that the amount that the resultant bitmap 46 grows, which determines the ultimate size of the hash table 40, is dependent on how dense the packing is for the subset of values .
• step 914 The next page (steps 914 and 900) to be overlaid onto the resultant bitmap 46 is the pagen.
• pagen and pageoi each have one bit set therein, but for purposes of this example it is assumed that the ranking process of step 806 has ranked pagen for placement before pageoi.
• pagen has three leading zero bits (blanks) .
• step 902 the pagen is first shifted left three times before testing for conflicts.
• step 904 determines that a conflict exists at this (negative three) offset, and thus at step 906 the page is shifted right one place (the offset is incremented) and the conflict test repeated.
• step 904 determines that there are no conflicts, and thus at step 908, the pagen, with an offset of negative two, is combined with the previous resultant bitmap 46b into a new resultant bitmap 46 c (FIG. 9) .
• the offset of negative two for this page is recorded in the appropriate location (at location three) in the page offset table 42, as best shown in FIG. 13. Note that later, such as at step 814, the data corresponding to this index, e.g., the attributes of "P,” will be saved in the hash table 40 in the position corresponding to the index location in the resultant bitmap 46 c as shown in FIG. 17.
• the last remaining page to be overlaid onto the resultant bitmap 46 is the pageoi.
• pageoi has no leading zero bits (blanks). Accordingly, the pageoi is not shifted left by step 902 before testing for conflicts.
• step 904 a conflict exists at the zero offset, and thus the page is shifted right one place at step 906, the offset is incremented and the conflict test of step 904 repeated.
• conflicts continue to be detected at step 904 and the page shifted by step 906 until the pageoi is offset by positive five positions.
• the pageoi with an offset of plus five, is combined with the previous resultant bitmap 46 C into a new, and final resultant bitmap 46 (FIG. 10) .
• the pageoi was shifted the entire size of the previous resultant bitmap. More particularly, since the previous bitmap 46 c had one- hundred percent packing, pageoi was overlaid beginning at the end of the previous resultant bitmap. Note that it is feasible to test for one-hundred percent packing, and, if detected, immediately overlay the next page at the end rather than start at the beginning and repeatedly shift to the end.
• the offset of five for this page is recorded in the page offset table 42, as best shown in FIG. 14. The process now returns to FIG. 19.
• the attribute data corresponding to this index e.g., the attributes of "B” will be saved in the hash table 40 in the position corresponding to the index location in the resultant bitmap 46, as shown in FIG. 18.
• the six scattered indexes and corresponding data of the initial sixteen-entry table 38 can be remapped into a densely-packed hash table 40 indexed by values ranging from zero to five, along with the page offset table 42 of four entries (e.g., bytes). Note that even with such a small example, the memory savings may be significant. For example, if the initial sixteen-entry table 38 allocated one (1) kilobyte per index, the table required sixteen (16) kilobytes of memory. However, the same data may be stored in a six (6) kilobyte hash table, plus four storage units (e.g., words or bytes) for the page offset table.
• the page offset table 42 is needed to convert an key to the initial large table 38 into a hash value index of the dense hash table 40.
• Larger subsets of even larger arrays provide similar savings, and particularly benefit from the automated remapping process.
• various bitmap matrixes may be tried, including those which do not have the same number of rows and columns. For example, with sixteen bit indexes, splits other than 8 bits by 8 bits may be used to re-map the values in the above-described manner.
• the size of the resultant hash table 40 and the page offset table 42 are preserved as statistics associated with the hash table 40.
• step 802 the process loops back to step 802 to construct other matrixes, each time varying the way in which the values are split into rows and columns.
• 16-bit indexes can be divided into 128 pages of 512 columns (7 by 9 bits), 512 pages of 128 columns (9 by 7 bits), 64 pages of 1024 columns (6 by 10 bits) and so on. If a particular loop iteration improves on the formerly best split, the page offset table and accompanying statistics are substituted for the former ones, whereby the split that uses the least amount of space will ultimately remain after all (reasonable) splits have been tried (step 812) . Note that the above- described mapping operation is sufficiently fast such that all sensible combinations may be tried until the optimal mapping available via this method is recognized.
• the offset table 42 therefor may be used to write the original table entries into a dense or mostly-dense hash table 40 at step 814.
• FIG. 21 another computer system is shown in FIG. 21.
• like components to those in FIG. 1 have been renumbered one- thousand numbers higher than their counterparts in FIG. 1.
• the storage 1024 includes the completed hash table 1040 and the page offset table 1042, such as for the purpose of displaying selected characters such as Unicode characters.
• a second aspect of the invention deals with the conversion of a key of the original, large table 38, to a hash value 1044, which serves as an index to the dense hash table 1040.
• the tables 1040 and 1042 are typically stored in non-volatile memory, such as burned into a ROM or the like.
• an application program 1048 or the like first loads (or if already loaded, locates) the page offset table 1042 and hash table 1040 into the storage 1024 (at step 1100 of FIG. 22) .
• An key 1046 (input value) is received by the application program 1048 or the like and is provided to a hash computation process 1050, whereby at step 1104 (FIG. 22) the value is first split into the same bit groupings that were used to construct the page offset table 1042.
• the bit split is apparent from the size of the page offset table 1042, i.e., the number of page offset table 1042 entries equals the number of pages and thus indicates the number of bits used to store the page' s offset information, e.g., 128 entries corresponds to seven bits.
• the split and other information e.g., a filename of the corresponding hash table
• the high portion of the value is masked off by the computation process 1048 as part of step 1104, leaving the appropriate number of valid low bits representing the page number.
• the high portion equals two bits of the four possible bits, and thus the upper two bits are masked off.
• the remaining two bits are used as a zero- based index into the offset table 1042. For example, if a 0110b is received, such as for the purpose of displaying the "G" character, the upper bits are masked off leaving a 10b.
• the upper bits which represent the offset into the page, are obtained by shifting the non-masked upper bits the same number of bits in the page number to the right.
• the page number is used as in index into the page offset table 1042, whereby a look-up operation is performed to determine the offset value for the corresponding page.
• a 10b provides an offset value of three.
• this value is added to the offset into the page (equal to 01b as determined from the masking and shifting of the upper bits at step 1104) .
• the result (three plus one equals four) is the computed hash value index to the hash table 1040, and is returned to the application program 1048 at step 1110, which then obtains the appropriate data from the hash table 1040.
• the four (100b) entry in the hash table 40 of FIG. 18 properly indexes the "G" character.
• the table below sets forth the results of all valid computations, which are otherwise not described in detail herein:
• the computation is extremely fast, consisting only of an AND, (for masking), a shift, indexing an array (the page offset table 1042) and an add.
• the hash computation requires a valid input value, i.e., one that is supported by the application employing the hash table.
• a validity check (such as represented by step 1102) may be performed on the value to ensure that it corresponds to a valid input before performing the hash computation. If the input value is valid, the hash is perfect, as any valid key provides a unique result without collisions.
• the offset table need be maintained to compute the hash value. Accordingly, any other information accumulated during the construction of the offset table such as the resultant bitmaps and the statistics may be discarded. Of course, some metadata and/or other information may be maintained with the offset table and/or hash table if desired, e.g., so that an application can load the proper hash table and corresponding offset table if more than one are available in a system.
• a method and mechanism for creating a perfect hash on a given subset of noncontiguous values of a larger range densely packs together the hashed values while providing for a very fast computation of the hash value.
• the method and mechanism operate in an automated manner and functions particularly well with subsets of noncontiguous Unicode points.
• the method and mechanism are fast, reliable, cost-efficient, flexible and extensible.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Databases & Information Systems (AREA)
• Software Systems (AREA)
• Data Mining & Analysis (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Publications (1)

Family

ID=25354449

Family Applications (1)

Country Status (5)

Cited By (2)

Families Citing this family (36)

Citations (5)

Family Cites Families (5)

• 1997
  • 1997-06-05 US US08/869,913 patent/US6014733A/en not_active Expired - Lifetime
• 1998
  • 1998-06-04 WO PCT/US1998/011680 patent/WO1998055929A1/en not_active Ceased
  • 1998-06-04 CN CNB988011085A patent/CN1206604C/zh not_active Expired - Fee Related
  • 1998-06-04 AU AU78211/98A patent/AU7821198A/en not_active Abandoned
  • 1998-06-04 JP JP50295599A patent/JP4965009B2/ja not_active Expired - Fee Related
• 2009
  • 2009-05-20 JP JP2009122449A patent/JP4717130B2/ja not_active Expired - Fee Related
• 2010
  • 2010-09-21 JP JP2010211217A patent/JP4717154B2/ja not_active Expired - Fee Related

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events