JP2022518194A - Content agnostic file indexing method and system - Google Patents

Abstract

A computer-implemented method for unknowing references to the contents of a binary data file, which pre-generates a table of all sequences of data of a particular length, determines the length of the binary data file, and that length is binary. A step that contains the number of bits in the data file, a step that chunks binary data into chunks of smaller length data, and for each chunk, determine if that chunk is in a pregenerated table, and if so. Uses that chunk index of the pregenerated table, otherwise chunks the data again until the subchunk is placed in the pregenerated table, and binary data using the number of chunks and the associated index. How to include a step to show the file.

Description

(Mutual reference of related applications)
This application is a part of patent application No. 15 / 730,043 entitled "Method and System for Subject Organic File Indexing" filed on October 11, 2017. It is a continuation application and the content of the application is incorporated herein by reference in its entirety for any purpose.

(Computer program list-Sequence list)
The following computer program list is submitted in attachment to this specification and is incorporated by reference. Each file is referenced by reference. The following computer program list is in the following format. <Size in bytes><Createddate><Filename>.

3864 May 16 2018 squeeze-master-README-md.txt *

83675 May 16 2018 squeeze-master-SqueezeReport-ipynb.txt *

4293 May 16 2018 squeeze-master-demo_app-py.txt *

98 May 16 2018 squeeze-master-gitignore.txt *

1383 May 16 2018 squeeze-master-requirements.txt *

2490 May 16 2018 squeeze-master-rpc_server-py.txt *

239 May 16 2018 squeeze-master-scripts-buildprotos.txt *

942 May 16 2018 squeeze-master-scripts-file_test-py.txt *

1391 May 16 2018 squeeze-master-scripts-generate_key-py.txt *

711 May 16 2018 squeeze-master-scripts-generate_keyset-py.txt *

629 May 16 2018 squeeze-master-scripts-keys_from_folder-py.txt *

377 May 16 2018 squeeze-master-scripts-lzw_test-py.txt *

107 May 16 2018 squeeze-master-scripts-runserver.txt *

3928 May 16 2018 squeeze-master-scripts-squeeze-bytes-report-py.txt *

63 May 16 2018 squeeze-master-scripts-squeeze_file-py.txt *

1060 May 16 2018 squeeze-master-scripts-squeeze_test-py.txt *

947 May 16 2018 squeeze-master-scripts-string_test-py.txt *

222 May 16 2018 squeeze-master-scripts-test_binary-py.txt *

1799 May 16 2018 squeeze-master-scripts-test_rpc-py.txt *

2736 May 16 2018 squeeze-master-scripts-time-squeeze-string-py.txt *

211 May 16 2018 squeeze-master-scripts-time_keygen-py.txt *

65 May 16 2018 squeeze-master-scripts-unsqueeze_file-py.txt *

80 May 16 2018 squeeze-master-setup-py.txt *

10657 May 16 2018 squeeze-master-squeeze- _ init _ -py.txt *

2783 May 16 2018 squeeze-master-squeeze-bitstring-py.txt *

9191 May 16 2018 squeeze-master-squeeze-keys-py.txt *

613 May 16 2018 squeeze-master-squeeze-performance-csv.txt *

22445 May 16 2018 squeeze-master-squeeze-squeeze_pb2-py.txt *

2232 May 16 2018 squeeze-master-squeeze-squeeze_pb2_grpc-py.txt *

3366 May 16 2018 squeeze-master-squeeze-proto.txt *

875 May 16 2018 squeeze-master-templates-layout-html.txt *

816 May 16 2018 squeeze-master-templates-upload_form-html.txt *

1513 May 16 2018 squeeze-master-templates-uploaded_file-html.txt *

200 May 16 2018 squeezerpc-master-Makefile.txt *

1131 May 16 2018 squeezerpc-master-README-md.txt *

7 May 16 2018 squeezerpc-master-gitignore.txt *

8995 May 16 2018 squeezerpc-master-main-go.txt *

21292 May 16 2018 squeezerpc-master-squeeze-squeeze-pb-go.txt *

3366 May 16 2018 squeezerpc-master-squeeze-proto.txt *

The present disclosure relates to a method for referencing a content unknown file. The present method further relates to a method for compressing content unknown data.

File reference techniques generally require knowledge of the types of data stored in order to efficiently index data in a file reference system. Similarly, knowledge of the data in question is also commonly used in creating improved compression approaches to reduce the data size for transmission, storage, etc.

There is a need in the industry to improve file reference and data compression techniques to reduce the amount of data that must be stored and / or transmitted.

According to one embodiment, the present disclosure provides a method of improving computing technology with an enhanced content agnostic file reference system. This method improves the operation of the computer itself.

The disclosed method has several important advantages. For example, the disclosed method allows file references of any content type.

The disclosed method can further reduce the amount of information or data that must be persisted or transmitted, as it can be generated at access, as opposed to the data being persisted.

The various embodiments of the present disclosure may not have any of these advantages, or may have some or all of them. Other technical advantages of the present disclosure may be readily apparent to those of skill in the art.

For a more complete understanding of the present disclosure and its advantages, reference is made herein below in conjunction with the accompanying drawings.
The same reference code refers to the same part or step through several figures in the drawing.
It is a flowchart which outlines the step of one Embodiment of this disclosure. It is another flowchart which outlines the step of another embodiment of this disclosure. It is a flowchart which outlines the step of the alternative embodiment of this disclosure.

The present disclosure relates to methods for agnostic indexing of data content. This method can be used for various computer-specific needs, such as, for example, as a file reference system or a compression system.

Although the following disclosure illustrates the invention in connection with the compression of binary data by way of example, the teachings work equally well with any type of data that is more appropriate to be referred to as "n-ary" data. For example, the methods and systems also work with qubits and bits.

One embodiment of the present invention includes a method as described in the flowchart depicted in FIG. Binary data (n _i ) to be persisted or transmitted (eg, a data file) is analyzed to determine the bit length (l ( _ni )). Using this information, in step 106, the method calculates all sequences of data of the identified length. For example, if the input data is as follows:
01

The input data is 2 bits long. In step 106, all 2-bit permutations are generated, i.e.
{00} {01} {10} {11}

At step 108, the method determines the index (n _f ) of the input binary data file in the generated sequence. In the above example, the returned index (n _f ) is "1". Finally, instead of storing or transmitting the input binary data (ie "01"), the system instead stores the length (2) and index (1).

If the original input data needs to be decoded (eg, a request to retrieve the original binary data from a disk or receive data transmitted over a network), this method is lengthy as input (l (l). Only ni)) and index ( _{n f} ) are needed. In the above example, the length (2) and the index (1) are the inputs. As shown in FIG. 2, the system calculates all the sequences of the input length. In the above example, the following sequence is generated.
{00} {01} {10} {11}

The system moves to the specified index (1 in the above example) and returns a sequence. Using the above example again, the original binary data "01" is returned.

The above method is described for illustrative purposes in terms of a binary system (ie, the input data is binary data). The method and system operate similarly for n-ary systems. The above binary system basically operates in the Euclidean plane, but in n-ary data, Hilbert space provides conceptually the same advantage. This method and process can be generalized for n-ary data as follows.
d ^ n = p (i)
(D ^ n) n = p (f)
d = system order n = appropriate length in n-ary units according to the system order p (i) = initial index p (f) = final index

Given two alternative ordered systems in the same input file, a system with a higher order has a higher n-ary density compared to an alternative system with a lower order. Should be noted.

An example of this method is disclosed in the Ruby Code Snippet below. The following snippet shows a method as disclosed in FIG.

The following snippet shows the method disclosed in FIG. 2 when the input length (l (ni)) is 16 and the index ( _{n f} ) is 72,629.

In a preferred embodiment, the input byte string is converted into a bit string corresponding to the representation of the input byte string. This bitstring is then processed by the method described herein.

In an alternative embodiment, instead of generating the table based on the length of the data, the table may be pregenerated with all the sequences of data of a particular length. This pre-generated table may be persisted in either non-volatile or volatile memory. In the above example, if the predetermined length is 2 bits, the pre-generated table contains all the forward sequences of the 2-bit data as follows.
{00} {01} {10} {11}

In one embodiment, the table may be stored in a corresponding indexed array as follows:

This pre-generated table may be stored on a disk, RAM, or the like. Preferably, this pre-generated table is stored on a computer system that reduces the file size (or squeezes the file) and expands the reduced file (or squeezes the data).

Upon receiving the input data, the method "chunks" the data into smaller subsets. As used herein, "chunk" means taking a data string to create a small data string that constitutes a subset of the large data string. All chunks are combined to form the original data string. For example, the input data is 011001110001
If,

It will be chunked into the following 4-bit chunks.
0010 0111 0001

The individual chunks are then compared against the pre-generated table to see if there is a match. In the above example, if the chunk size is 4 bits, the table has a sequence of all 2-bit chunks, so each chunk cannot be found in the table. Therefore, each chunk is chunked again and becomes as follows.
01 10 01 11 00 01

This method continues for each chunk until a particular chunk is placed in a pre-generated table. At that point, the chunks are associated with each index, preferably a set of tuples indicating the chunk level and the corresponding index is generated. In the above example, the system has chunked twice, so the index associations are as follows:
{2,1} {2,2} {2,1} {2,3} {2,0} {2,1}

In this example, the original input data "011001110001" was finally divided into 6 chunks of 2 bit length. As shown, each chunk is represented by a chunk level (2) and a corresponding index to the pregenerated table.

The data can be chunked in any number of ways. For example, the data may be chunked based on a pre-determined size, as in the example above (where the pre-determined size was 4 bits for the purposes of the example). .. Alternatively, the input data may be recursively chunked into two separate data chunks until each data chunk is found in a pregenerated table. The method of dividing and chunking data using the same input data as above results in the following first-level chunks.
011001 110001

Now the dataset is not found in the pre-generated table and is chunked again.
011 001 110 001

Again, the chunked data is not found in the pregenerated table and must be chunked again.
00 1 00 1 11 0 00 1

Notably, some segments are chunked into data that is smaller than the size of the pre-generated table (ie, segments "1", "1", "0", "1"). .. These segments may be padded for comparison with pre-generated tables. If consistent, the numbers can be stored in big endian or little endian byte order. For example, when using big endian byte order, the above chunk data is expressed as follows.
00 10 00 10 11 00 00 10

After that, this method continues in the same manner as above.

It is not necessary for all chunks of data to be found in a pregenerated table at the same data chunk level. For example, using the pre-generated table above for a 2-bit combination, if the input data is:
0110011100

Originally, this data can be divided into 4-bit sequences and chunked as described above.
0110 0111 00

Similar to the above, the first two 4-bit sequences (ie "0110" and "0111") must be re-chunked into smaller chunks in order to be placed in the pre-generated table, resulting in , Becomes the following chunks.
01 10 01 1 1 00

Also, as mentioned above, chunks are associated with their chunk level and their corresponding index as follows:
{2,1} {2,2} {2,1} {2,3} {1,0}

Note that the last tuple above indicates that the chunk level is 1 as the chunk did not require a second chunk.

When the input data is reduced to a set of chunk levels and indexes, the set of chunk levels and indexes is used to identify the original data. Associations can be stored as a set of tuples, as separate bitstrings, and in other ways.

To recreate (or unsqueeze) data based on a set of chunk levels and indexes, the process works in reverse. Again, the system needs the same pre-generated table. For each chunk level and index tuple, the system references a pre-generated table to unpack the squeezed chunks and revert to the original data.

This alternative embodiment is shown in the flowchart of FIG. First, a pre-generated table containing all the sequences of data of a particular length is created in step 302, such as: As mentioned above, preferably the table is persisted in some way. Next, the system receives the squeeze target input data in step 304. The process then chunks the data into smaller segments in steps 306 and 308 until the length of the data is long enough to be placed in the "pre-generated table". As mentioned above, this process maintains the chunk level so that the system knows how many times the input dataset has been chunked. Each chunk is then placed in the pregenerated table in step 310. Finally, the chunks, their chunk levels, and their respective indexes in the pregenerated table are associated, resulting in squeezed data in step 312.

Although the present disclosure has been described in terms of specific embodiments and generally relevant methods, changes and sequence of these embodiments and methods will be apparent to those of skill in the art. Accordingly, the above description of exemplary embodiments does not limit the present disclosure. Other modifications, substitutions, and modifications may be made without departing from the spirit and scope of this disclosure.

Claims (19)

A computer implementation method for agnostic references to the contents of binary data files.
A step that pregenerates a table using an input seed, in which the table contains all the sequences of bits of a given length, and
A step of determining the length of the binary data file, wherein the length includes a step including the number of bits of the binary data file and a step.
A step of chunking the binary data file into partial strings, each substring having a length smaller than the length of the binary data file.
For each chunk of the binary data file, it is determined if the chunk is in the pregenerated table, and if the chunk is in the pregenerated table, the chunk is pregenerated. A step of associating an index of the position of chunks in a table and splitting the chunked binary data into smaller chunks if the chunk is not in the pregenerated table.
A method comprising a step of showing the binary data file using the number of chunks and the associated index of all chunks. A step showing the binary data file using the number of chunks and the associated index of all chunks
The method of claim 1, comprising the step of persisting the number of chunks and all relevant indexes to storage instead of the binary data file. A step showing the binary data file using the number of chunks and the associated index of all chunks
The method of claim 1, comprising the step of transmitting the number of chunks and the associated index of all chunks on behalf of the data file. The method of claim 3, wherein the transmitting step transmits the number of the chunks and the associated index of all chunks over the network. The method of claim 3, wherein the transmitting step transmits the number of chunks and the associated index of all chunks on the bus. The step of showing the binary data file using the number of chunks and the associated index of all chunks is
The method of claim 1, wherein each ordered pair indicates a chunk level and an associated index, comprising the step of creating a tuple of ordered pairs. The step of showing the binary data file using the number of chunks and the associated index of all chunks comprises persisting the number of chunks and the associated index of all chunks on the storage device. The method described in. The method of claim 7, wherein the storage device is an optical disc. The method of claim 1, wherein the pre-generated table is a hash table. The method of claim 1, wherein the pre-generated table is a matrix. The method of claim 1, wherein the pre-generated table is persisted in volatile memory. The method of claim 1, wherein the pre-generated table is persisted in non-volatile memory. Chunking the binary data file into partial strings further
The method of claim 1, comprising chunking the binary data into chunks of a predetermined length. 13. The method of claim 13, wherein the predetermined length is 2 megabytes. 13. The method of claim 13, wherein the predetermined length is less than 2 megabytes. 13. The method of claim 13, wherein the predetermined length is greater than 2 megabytes. Chunking the binary data file into partial strings further
The method of claim 1, wherein the binary data file is recursively divided into two chunks of the same size. A method of retrieving data based on the number of chunks and the associated index of all chunks.
A step of pre-generating a table using an input seed, wherein the table contains all the sequences of bits of a given length, and the number of chunks and related using the pre-generated table. Steps to generate an index and
For each chunk, a method that includes placing data in the table at the index associated with that chunk and returning the data associated with each chunk. The step that returns the data associated with each chunk is
18. The method of claim 18, comprising concatenating the data associated with each chunk into a single bitstream.

Images