Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/10—File systems; File servers
  • G06F16/13—File access structures, e.g. distributed indices
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/10—File systems; File servers
  • G06F16/17—Details of further file system functions
  • G06F16/174—Redundancy elimination performed by the file system
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
  • G06F16/22—Indexing; Data structures therefor; Storage structures
• • H—ELECTRICITY
  • H03—ELECTRONIC CIRCUITRY
  • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
  • H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
  • H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
  • H03M7/3084—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction using adaptive string matching, e.g. the Lempel-Ziv method
  • H03M7/3088—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction using adaptive string matching, e.g. the Lempel-Ziv method employing the use of a dictionary, e.g. LZ78
• • H—ELECTRICITY
  • H03—ELECTRONIC CIRCUITRY
  • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
  • H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
  • H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
  • H03M7/60—General implementation details not specific to a particular type of compression
  • H03M7/6052—Synchronisation of encoder and decoder

Definitions

• This disclosure relates to a method for content agnostic file referencing.
• the method may further relate to a method for content agnostic data compression.
• File referencing techniques generally require knowledge about the kind of data being stored in order to efficiently index the data in a file referencing system. Similarly, knowledge about the data at issue is also generally used in creating improved compression approaches to reduce data size for transmission, storage, and the like.
• this disclosure provides a method for improving computing technology with an enhanced content-agnostic file referencing system. The method improves the operation of the computer itself.
• the disclosed method has several important advantages. For example, the disclosed method permits file referencing of any content type.
• the disclosed method additionally permits a significant reduction in the amount of information or data that must be persisted or transmitted, as data may be generated at access time as opposed to persisted.
• Various embodiments of the present disclosure may have none, some, or all of these advantages. Other technical advantages of the present disclosure may also be readily apparent to one skilled in the art.
• FIG. 1 is a flowchart outlining the steps of one embodiment of the present disclosure.
• FIG. 2 is another flowchart outlining the steps of another embodiment of the present disclosure.
• FIG. 3 is a flowchart outlining the steps of an alternate embodiment of the present disclosure.
• the present disclosure relates to a method for content-agnostic indexing of data.
• the method may be used for a variety of computer-specific needs, including for example as a file referencing system or a compression system.
• One embodiment of the present invention comprises a method as described in the flow chart depicted in FIG. 1.
• Binary data (//,) for instance, a data file
• the method uses this information, at step 106, the method calculates all permutations of data of the identified length. For example, if the input data is:
• the method determines the index ( «/) of the input binary data file in the generated permutations. Using the example above, the index ( «/) returned would be “1”. Finally, rather than storing or transmitting the input binary data (i.e.“01”), the system instead stores the length (2) and the index (1).
• the method needs only a length (/( «,)) and an index (// / ) as input.
• the input provided would be the length (2) and the index (1).
• the system calculates all permutations of the inputted length. As above, that would generate the following permutations:
• n length in appropriate n-ary units respective to the order of the system
• steps generate) oin.to s step Input.kmp_search("# ⁇ steps ⁇ " ,”# ⁇ input ⁇ ”) p input
• an input byte string is converted into a bit string corresponding to a representation of the input byte string. This bit string is what is then processed through the method described herein.
• a table may be pregenerated with all permuations of data of a particular length. This pregenerated table may be persisted in memory, either non-volatile or volatile memory. Using the above example, if the predetermined length is 2-bits, the pregenerated table will include all permutations of 2-bit data, such as
• this table may be stored in an array with corresponding indices as follows:
• This pregenerated table may be stored on disk, in RAM, or otherwise.
• this pregenerated table is stored with the computing system that reduces file size (or squeezes a file) as well as the computing system that expands a reduced file (or unsqueezes the data).
• the method“chunks” the data into smaller subsets of data.
• “chunk” means to take a data string and create smaller data strings comprising subsets of the larger data string. All chunks together would form the original data string. For example, if the input data is:
• each individual chunk will then be compared to the pregenerated table to see if there is a match.
• each chunk will not be found in the table as the table has permutations for all 2-bit chunks.
• each chunk will be chunked again, resulting in the following:
• the method will continue for each chunk until a point where the particular chunk is located in the pregenerated table. At that point, the chunk will be associated with its respective index, and preferably a series of tuples will be generated indicating the chunk level and the corresponding index.
• the system chunked twice, so the index association will be as follows:
• each chunk is represented with a chunk level (2) and corresponding index into the pregenerated table.
• the data may be chunked in any number of ways. For instance, the data may be chunked based on a pre-determined size as in the above example (where the predetermined size was 4-bits for purposes of example). Alternatively, the input data may be recursively chunked into 2 separate data chunks, until each data chunk may be found in the pregenerated table. Using the same input data as above, a method of chunking the data by splitting it would result in the following first level chunk:
• segments“1”,“1”,“0”, and“1” are chunked into data smaller than the pregenerated table size (i.e. segments“1”,“1”,“0”, and“1”). These segments may be padded in order to compare them to the pregenerated table.
• the numbers may be stored either using big endian or little endian byte order, so long as consistency is maintained. Using big endian byte order, for example, the chunked data above would be represented as:
• the data may be originally chunked like above, by breaking it into 4 bit sequences:
• Pregenerated Table comprising all permutations of data of a particular length is created at step 302. As indicated above, preferably that table is persisted in some fashion.
• the system receives input data to be squeezed at step 304.
• the process then chunks the data into smaller segments until the data length is of a length that would be located in the Pregenerated Table at steps 306 and 308.
• the process maintains the chunk level so that the system knows how many times an input data set has been chunked. Each chunk is then located in the Pregenerated Table at step 310.
• the chunk, its chunk level, and the respective index in the Pregerated Table is associated, resulting in the squeezed data at step 312.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Data Mining & Analysis (AREA)
• Databases & Information Systems (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Software Systems (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (1)

Family

ID=71520909

Family Applications (1)

Country Status (6)

Cited By (1)

Citations (6)

Family Cites Families (2)

• 2020
  • 2020-01-08 KR KR1020217025238A patent/KR20210110875A/ko unknown
  • 2020-01-08 WO PCT/US2020/012661 patent/WO2020146448A1/en unknown
  • 2020-01-08 CA CA3126012A patent/CA3126012A1/en active Pending
  • 2020-01-08 EP EP20737931.4A patent/EP3908937A4/en active Pending
  • 2020-01-08 JP JP2021540318A patent/JP2022518194A/ja active Pending
  • 2020-01-08 AU AU2020205970A patent/AU2020205970A1/en active Pending

Patent Citations (6)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events