Classifications

• • H—ELECTRICITY
  • H03—ELECTRONIC CIRCUITRY
  • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
  • H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
  • H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
  • H03M7/60—General implementation details not specific to a particular type of compression
  • H03M7/6064—Selection of Compressor
  • H03M7/607—Selection between different types of compressors
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/10—File systems; File servers
  • G06F16/17—Details of further file system functions
  • G06F16/174—Redundancy elimination performed by the file system
  • G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16B—BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
  • G16B50/00—ICT programming tools or database systems specially adapted for bioinformatics
  • G16B50/50—Compression of genetic data
• • H—ELECTRICITY
  • H03—ELECTRONIC CIRCUITRY
  • H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
  • H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
  • H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
  • H03M7/60—General implementation details not specific to a particular type of compression
  • H03M7/6064—Selection of Compressor
  • H03M7/6082—Selection strategies
  • H03M7/6088—Selection strategies according to the data type

Definitions

• the present disclosure is directed generally to methods and systems for compressing a data file using a file compression/decompression system.
• a single genomic data file can comprise sequencing results for millions or billions of locations across the genomes of numerous samples, with annotations and other data attributes.
• a Variant Call Format (VCF) file for example, can comprise gene sequence variations with millions of rows and numerous columns comprising data attributes.
• Compression of a data file typically comprises one or more data compressors. While there are numerous generic compressors that can be applied directly to a data file, their compression performances are usually suboptimal since they do not fully exploit the similarity or predictability of the data of individual fields within the dataset for the effective removal of data redundancy.
• One approach for improving compression ratio is therefore first dividing a dataset into fields with distinct characteristics, and then compressing each of the fields individually with a compressor that can take the most advantage of the specific data model of the field.
• this generally requires that the data fields and their associated compressors be pre-determined for each and every file type, and the encoding and decoding processes to be hardcoded accordingly in the software used for compression and decompression.
• the rigidity of such an approach makes it difficult to accommodate emerging data types or adopt new algorithms.
• a file compression/decompression system receives a data file for compression comprising a plurality of different attributes for each of a plurality of different samples.
• the system identifies a first attribute of the plurality of different attributes, and selects a plurality of different compression types and/or compression configurations from a predetermined plurality of compression types and configurations based on the first attribute.
• the system compresses data from the received genomic data file for the identified first attribute using each of the selected plurality of compression types and/or compression configurations, individually, and determines which one of the selected plurality of compression types and/or compression configurations is most suitable for compression. Then the system generates a compression parameter data structure comprising an identification of the one of the selected plurality of compression types and/or compression configurations most suitable for compression of the identified first attribute, and compresses the data from the received genomic data file for the first attribute to generate a compressed genomic data file using the identified compression type and/or compression configuration. The compression parameter data structure and the compressed genomic data file are then stored and can be used for decompression.
• a method for compressing/decompressing a genomic data file using a file compression/decompression system includes: (i) receiving a data file for compression, the data file comprising a plurality of different attributes; (ii) identifying a first attribute of the plurality of different attributes, and storing a specification of the identified first attribute in an attribute parameter data structure; (iii) selecting, based on the identified first attribute, a plurality of compression types and/or compression configurations from a predetermined plurality of compression types and/or compression configurations; (iv) compressing at least some of the data from the received data file for the identified first attribute using each of the selected plurality of compression types and/or compression configurations, individually, wherein one or more metrics are measured for each individual compression; (v) determining, based on the one or more metrics of each of the individual compressions, which one of the selected plurality of compression types and/or compression configurations is most suitable for compression; (vi) generating a compression
• the method further includes repeating the identifying, selecting, compression, and determining steps for at least a second attribute of the plurality of different attributes, wherein the compression parameter data structure comprises a specification of a one of the selected plurality of compression types and/or compression configurations most suitable for compression of the identified second attribute.
• the method further includes decompressing the stored compressed data file, wherein decompression comprises consulting the compression parameter data structure to identify a decompression type for the first attribute.
• each of the selected plurality of compression types and/or compression configurations is selected because it is capable of compressing the data from the received data file for the identified first attribute.
• the data file is a genomic data file.
• the one or more metrics comprises one or more of compression ratio and processing time for said compression.
• a compression type is most suitable when the corresponding compression comprises a higher compression ratio and/or a faster processing time relative to the other of the plurality of selected compression types and/or compression configurations.
• the method further includes providing, via a user interface, information about the generated compression parameter data structure and/or the compressed data file.
• the method further includes providing to a user, via a user interface, information about one or more of the identified compression types and/or compression configurations such that the user can select a compressor type and/or configuration to be applied to compress an attribute.
• a system for compressing/decompression a data file includes: a data file for compression, the data file comprising a plurality of different attributes; and a processor configured to: (i) identify a first attribute of the plurality of different attributes and store a specification of the identified first attribute in an attribute parameter data structure; (ii) select, based on the identified first attribute, a plurality of compression types and/or compression configurations from a predetermined plurality of compression types and/or compression configurations; (iii) compress at least some of the data from the received data file for the identified first attribute using each of the selected plurality of compression types and/or compression configurations, individually, wherein one or more metrics are measured for each individual compression; (iv) determine, based on the one or more metrics of each of the individual compressions, which one of the selected plurality of compression types and/or compression configurations is most suitable for compression; (v) generate a compression parameter data structure comprising an identification of the one of the selected plurality of compression
• the system comprises a user interface configured to provide information about the generated compression parameter data structure and/or the compressed data file.
• a processor or controller may be associated with one or more storage media (generically referred to herein as “memory,” e.g., volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM, floppy disks, compact disks, optical disks, magnetic tape, etc.).
• the storage media may be encoded with one or more programs that, when executed on one or more processors and/or controllers, perform at least some of the functions discussed herein.
• Various storage media may be fixed within a processor or controller or may be transportable, such that the one or more programs stored thereon can be loaded into a processor or controller so as to implement various aspects as discussed herein.
• program or “computer program” are used herein in a generic sense to refer to any type of computer code (e.g., software or microcode) that can be employed to program one or more processors or controllers.
• FIG. 1 is a flowchart of a method for compressing a genomic data file using a file compression/decompression system, in accordance with an embodiment.
• FIG. 2 is a schematic representation of a file compression/decompression system, in accordance with an embodiment.
• FIG. 3 is a schematic representation of a data file, in accordance with an embodiment. Detailed Description of Embodiments
• a data file compression/decompression system receives a genomic data file for compression, the genomic data file comprising a plurality of different attributes (or data fields) for each of a plurality of different genomic samples.
• the system identifies a first attribute of the plurality of different attributes and selects, based on that attribute, a plurality of compression types and/or compression configurations from a predetermined plurality of compression types and compression configurations, wherein each of the selected plurality of compression types and/or compression configurations is capable of compressing the data from the received genomic data file for the identified first attribute.
• the data file compression system compresses at least some of the data from the received genomic data file for the identified first attribute using each of the selected plurality of compression types and/or compression configurations, individually, wherein one or more metrics are measured for each individual compression, the one or more metrics comprising one or more of compression ratio and processing time for said compression.
• the system determines which one of the selected plurality of compression types and/or compression configurations is most suitable for compression.
• a compression type is most suitable when the corresponding compression comprises a higher compression ratio and/or a faster processing time relative to the other of the plurality of selected compression types and/or compression configurations.
• the system then generates a compression parameter data structure comprising an identification of and/or the processing instructions for the one of the selected plurality of compression types and/or compression configurations most suitable for compression of the identified first attribute, and compresses the from the received genomic data file for the first attribute to generate a compressed genomic data file.
• the compression parameter data structure and the compressed genomic data file are stored, where the stored compression parameter data structure is configured to be retrieved and used for decompression of the compressed genomic data file.
• the novel methods and systems described or otherwise envisioned herein provide a framework for the automatic configuration of compressors for data fields based on their performances, and enable compressors to be specified explicitly and stored along with compressed data.
• a decoder can execute decompression and inverse transform algorithms based on the processing steps and parameters specified in the compressor configuration associated with the attribute.
• the novel methods and systems described or otherwise envisioned herein provide numerous advantages. For example, the methods and systems provide flexibility for customizing compressor configurations for a file, and ease the adoption of new transform and compression algorithms.
• the methods and systems accommodate any new file types by identifying and providing compressor configurations. Since processing instructions are contained in the compressor configurations and not hardcoded in the software, a decoder can readily process a file without the need for modifications of codes. Third, auto-configuration of compressor settings helps to improve compression performance without requiring user knowledge or experience. The methods and systems provide many other advantages as well.
• the compressor configuration framework system comprises one or more of the following components: (i) a uniform data interface for specifying a compressor, such as a compression process that consists of a sequence of data transform and compression steps and their default parameters; (ii) a uniform data interface for specifying a data attribute and associating it to a compressor with customized parameters; (iii) an algorithm for encoding attribute data based on the processing steps specified in the associated compressor configuration; (iv) an algorithm for decoding attribute data based on the processing steps specified in the associated compressor configuration; and (v) a mechanism for the automatic determination of compressor configurations based on performance.
• a uniform data interface for specifying a compressor, such as a compression process that consists of a sequence of data transform and compression steps and their default parameters
• a uniform data interface for specifying a data attribute and associating it to a compressor with customized parameters
• an algorithm for encoding attribute data based on the processing steps specified in the associated compressor configuration an algorithm for decoding attribute data based on the processing steps specified in the associated
• FIG. 1 in one embodiment, is a flowchart of a method 100 for compressing a genomic data file using a file compression/decompression system.
• the methods described in connection with the figures are provided as examples only, and shall be understood not to limit the scope of the disclosure.
• the file compression/decompression system can be any of the systems described or otherwise envisioned herein.
• the file compression/decompression system can be a single system or multiple different systems.
• a file compression/decompression system 200 is provided. Referring to an embodiment of a file compression/decompression system 200 as depicted in FIG.
• the system comprises one or more of a processor 220, memory 230, user interface 240, communications interface 250, and storage 260, interconnected via one or more system buses 212.
• FIG. 2 constitutes, in some respects, an abstraction and that the actual organization of the components of the system 200 may be different and more complex than illustrated.
• file compression/decompression system 200 can be any of the systems described or otherwise envisioned herein. Other elements and components of the file compression/decompression system 200 are disclosed and/or envisioned elsewhere herein.
• file compression/decompression system 200 comprises or is in direct or indirect communication with a file database 270.
• the file database comprises, among other possible data, one or more files that can be compressed, or have been compressed, using the file compression/decompression system 200.
• the one or more files comprise genomic files, with a genomic file comprising gene sequence variations with millions of rows and numerous columns comprising data attributes, potentially for many different genomic samples.
• the file database can be any such database, including but not limited to the databases and systems described or otherwise envisioned herein.
• file compression/decompression system 200 comprises or is in direct or indirect communication with a compression and decompression algorithm database 280.
• the compression and decompression algorithm database comprises a library of different compression algorithms and decompression algorithms. These compression and decompression algorithms can be utilized to compress and decompress the data files by the data file system. New compression algorithms and decompression algorithms can be added to the database at any time.
• the compression and decompression algorithm database can be any such database, including but not limited to the databases and systems described or otherwise envisioned herein.
• the file compression/decompression system receives an uncompressed data file for compression, the data file comprising a plurality of different attributes for each of a plurality of different samples.
• the genomic data file comprises a plurality of different attributes for each of a plurality of different genomic samples.
• the data file can be received locally or remotely from any source, including but not limited to the file database 270, which itself can be a local or remote database.
• the file data system identifies a first attribute among the plurality of different attributes in the data file.
• An attribute can be any information from the data file. Referring to FIG. 3, in one non-limiting example of a data file, is a portion 300 of a genomic data file comprising a plurality of columns 310 where each column is an attribute.
• the file data system can be configured to recognize known attributes within a data file, or can be configured to predict or estimate or otherwise determine new or unrecognized attributes within a data file. Accordingly, the file data system can be configured to identify a first attribute among a plurality of different attributes in a data file using a wide variety of different methods. According to an embodiment, after identifying an attribute, the system specifies the attribute in an attribute parameter data structure (such as Attribute Parameter Set described herein).
• an attribute parameter data structure such as Attribute Parameter Set described herein.
• the data file compression/decompression system selects a plurality of compression types or algorithms (a “compressor”) and/or compression configurations from a predetermined plurality of compression types, algorithms, and configurations.
• a compressor or compression configuration can comprise, for example, a sequence of compression algorithms and the parameter(s) associated with the compression algorithms.
• the compression types, algorithms, and compression configurations can be selected locally or remotely from any source, including but not limited to the compression and decompression algorithm database 280, which itself can be a local or remote database.
• one or more of the plurality of compression types, algorithms, and compression configurations is selected from the predetermined plurality of compression types, algorithms, and compression configurations based at least in part on the identified attribute.
• a compressor or configuration may be selected based on the ability of the compressor to compress the data from the received data file for the identified first attribute.
• the system may preferentially identify compressors or configurations capable of compressing the first attribute data, and/or may preferentially exclude compressors incapable of compressing the first attribute data.
• the file data system compresses data using the identified compressors.
• the file compression/decompression system individually compresses the first attribute data. This individual compression by the plurality of compressors can be performed sequentially or simultaneously.
• each compressor compresses only a subset of the first attribute data, where the subset is an identical subset for each compressor in order to allow for more objective comparison of compression metrics among the plurality of compressors.
• each compressor compresses all of the first attribute data.
• the file compression/decompression system measures or otherwise obtains or identifies one or more compression metrics for each individual compression by each compressor. This enables objective comparison of compression performance among the plurality of compressors, and the selection of a preferred compressor for the first attribute data.
• compression metrics can be compression ratio, processing time, compression accuracy, and/or any other compression metric.
• the file compression/decompression system determines which of the selected plurality of compression types, algorithms, or compression configurations is most suitable for compression, using the measured or obtained metrics. Accordingly, the file compression/decompression system compares, ranks, or otherwise utilizes the metrics corresponding to each individual compressor and selects a preferred compressor. For example, a compressor or configuration may be most suitable or preferred for compression when the compression or configuration comprises a higher or best compression ratio and/or a faster processing time relative to the other of the plurality of selected compression types or configurations. According to a first embodiment, the file compression/decompression system is configured to select the compressor with the best compression ratio.
• the file compression/decompression system is configured to select the compressor with the fastest compression or processing time.
• the file compression/decompression system is configured to select the compressor with the best combination of compression ratio and compression or processing time.
• the file compression/decompression system repeats these identifying, selecting, compression, and determining steps for one or more additional attributes.
• the system can be configured to identify a preferred compressor for some or all attributes found within the data file.
• the file compression/decompression system generates a compression parameter data structure, such as a compression parameter data file.
• the generated compression parameter data structure comprises an identification of and/or specification(s) for the selected compressor and/or compression configurations for at least the first identified attribute, and optionally an identification of the compressor(s) and/or compression configuration(s) selected for a plurality of attributes in the data file (for example, specifying the use of a compressor for multiple attributes).
• the compression parameter data file can comprise any information necessary to utilize a compressor, including compressor settings and other information.
• the file compression/decompression system compresses the data from the received data file to generate a compressed data file, using the identified compressor(s) and/or compression configuration(s). Accordingly, this can comprise the compression of first attribute data, data for multiple attributes, or data for all attributes in the file.
• the file can be utilized immediately, such as sending or otherwise transmitting or using the compressed file, or the compressed file can be stored in local or remote storage for use in further steps of the method. Many other options are possible.
• the generated compression parameter data structure is stored such that the stored file can be retrieved and used for decompression of the compressed data file.
• the compression parameter data structure can be stored in a remote or local database which can be a component of or otherwise in communication with file compression/decompression system 200.
• the compression parameter data structure can be stored together with the compressed data file, or stored with a reference to or from the compressed data file.
• system 200 comprises a processor 220 capable of executing instructions stored in memory 230 or storage 260 or otherwise processing data to, for example, perform one or more steps of the method.
• processor 220 may be formed of one or multiple modules.
• Processor 220 may take any suitable form, including but not limited to a microprocessor, microcontroller, multiple microcontrollers, circuitry, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), a single processor, or plural processors.
• FPGA field programmable gate array
• ASIC application-specific integrated circuit
• Memory 230 can take any suitable form, including a non-volatile memory and/or RAM.
• the memory 230 may include various memories such as, for example LI, L2, or L3 cache or system memory.
• the memory 230 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices.
• SRAM static random access memory
• DRAM dynamic RAM
• ROM read only memory
• the memory can store, among other things, an operating system.
• the RAM is used by the processor for the temporary storage of data.
• an operating system may contain code which, when executed by the processor, controls operation of one or more components of system 200. It will be apparent that, in embodiments where the processor implements one or more of the functions described herein in hardware, the software described as corresponding to such functionality in other embodiments may be omitted.
• User interface 240 may include one or more devices for enabling communication with a user.
• the user interface can be any device or system that allows information to be conveyed and/or received, and may include a display, a mouse, and/or a keyboard for receiving user commands.
• user interface 240 may include a command line interface or graphical user interface that may be presented to a remote terminal via communication interface 250.
• the user interface may be located with one or more other components of the system, or may located remote from the system and in communication via a wired and/or wireless communications network.
• Communication interface 250 may include one or more devices for enabling communication with other hardware devices.
• communication interface 250 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol.
• NIC network interface card
• communication interface 250 may implement a TCP/IP stack for communication according to the TCP/IP protocols.
• TCP/IP protocols Various alternative or additional hardware or configurations for communication interface 250 will be apparent.
• Storage 260 may include one or more machine-readable storage media such as readonly memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, or similar storage media.
• ROM readonly memory
• RAM random-access memory
• storage 260 may store instructions for execution by processor 220 or data upon which processor 220 may operate.
• storage 260 may store an operating system 261 for controlling various operations of system 200.
• memory 230 may also be considered to constitute a storage device and storage 260 may be considered a memory.
• memory 230 and storage 260 may both be considered to be non-transitory machine-readable media.
• non-transitory will be understood to exclude transitory signals but to include all forms of storage, including both volatile and non-volatile memories.
• processor 220 may include multiple microprocessors that are configured to independently execute the methods described herein or are configured to perform steps or subroutines of the methods described herein such that the multiple processors cooperate to achieve the functionality described herein.
• processor 220 may include a first processor in a first server and a second processor in a second server. Many other variations and configurations are possible.
• file compression/decompression system 200 comprises or is in direct or indirect communication with a file database 270.
• the file database comprises, among other possible data, one or more files that can be compressed, or have been compressed, using the file compression/decompression system 200.
• the one or more files comprise genomic files, with a genomic file comprising gene sequence variations with millions of rows and numerous columns comprising data attributes, potentially for many different genomic samples.
• file compression/decompression system 200 comprises or is in direct or indirect communication with a compression and decompression algorithm database 280.
• the compression and decompression algorithm database comprises a library of different compression algorithms and decompression algorithms. These compression and decompression algorithms can be utilized to compress and decompress the data files by the data file system. New compression algorithms and decompression algorithms can be added to the database at any time.
• storage 260 of system 200 may store one or more algorithms, modules, and/or instructions to carry out one or more functions or steps of the methods described or otherwise envisioned herein.
• storage 260 may comprise, among other instructions or data, attribute identification instructions 262, compressor formulation instructions 263, compressor selection instructions 264, storage instructions 265, and/or reporting instructions 266.
• attribute identification instructions 262 direct the system to identify an attribute among a plurality of different attributes in a data fde received by the system.
• An attribute can be any information from the data file.
• the fde data system can be configured to recognize known attributes within a data file or can be configured to predict or estimate or otherwise determine new or unrecognized attributes within a data file. Accordingly, the file data system can be configured to identify a first attribute among a plurality of different attributes in a data file using a wide variety of different methods.
• compressor formulation instructions 263 direct the system to select a plurality of compressors from a predetermined plurality of compression types, algorithms, and/or compression configuration(s).
• the compression types, algorithms, and/or compression configuration(s) can be selected locally or remotely from any source, including but not limited to the compression and decompression algorithm database 280.
• the compressor formulation instructions 263 also direct the system to compress some or all of the attribute data using the identified compressors and/or compression configuration(s) individually. This individual compression by the plurality of compressors can be performed sequentially or simultaneously.
• each compressor compresses only a subset of the first attribute data, where the subset is an identical subset for each compressor in order to allow for more objective comparison of compression metrics among the plurality of compressors.
• each compressor compresses all of the first attribute data.
• compressor formulation instructions 263 also direct the system to measure or otherwise obtain or identify one or more compression metrics for each individual compression by each compressor. This enables objective comparison of compression performance among the plurality of compressors and/or compression configuration(s), and the selection of a preferred compressor and/or compression configuration for the first attribute data.
• compression metrics can be compression ratio, processing time, compression accuracy, and/or any other compression metric.
• compressor selection instructions 264 direct the system to determine which of a selected plurality of compression types, algorithms, and/or compression configuration(s) is most suitable for compression, using measured or obtained metrics.
• the file compression/decompression system can compare, rank, or otherwise utilize the metrics corresponding to each individual compressor and/or compression configuration and selects a preferred compressor and/or compression configuration.
• a compressor and associated configurations may be most suitable or preferred for compression when the compression comprises a higher or best compression ratio and/or a faster processing time relative to the other of the plurality of selected compression types.
• the different compressor configurations and the corresponding metrics can be presented to a user through a user interface in order to facilitate the manual selection of the compressor to be applied on an attribute.
• storage instructions 265 direct the system to generate a compression parameter data structure, such as a compression parameter data file.
• the generated compression parameter data structure comprises an identification of and/or specifications for the selected compressor and/or compression configuration(s) for at least the first identified attribute.
• the compression parameter data file can comprise any information necessary to utilize a compressor, including compressor settings and other information.
• the storage instructions 265 can also direct the system to store the generated compression parameter data structure such that the stored file can be retrieved and used for decompression of the compressed data file.
• the compression parameter data structure can be stored in a remote or local database which can be a component of or otherwise in communication with file compression/decompression system 200.
• the compression parameter data structure can be stored together with the compressed data file, or stored with a reference to or from the compressed data file.
• reporting instructions 266 direct the system to provide information about the compression, the data file, and/or the compressors via a user interface 240 of the system.
• the provided information can be any information described or otherwise envisioned herein.
• the system may provide the information to a user via any mechanism, including but not limited to a visual display, an audible notification, a text message, an email, a page, or any other method of notification.
• the file compression/decompression system comprises a Compressor Parameter Set.
• a Compressor Parameter Set is a data structure that contains the compressor configuration, including all the instructions and default parameters, required for the encoding and decoding of attribute data, and should be stored along with the compressed data.
• Each compressor configuration carries a unique ID to be used for specifying its association with an attribute, and a sequence of compressor steps, each consisting of one or more of the following elements:
• transform and compression which involve changing the format of data representation and converting fixed-length symbols into compressed bit streams by means of probabilistic modeling.
• the algorithm ID is a predefined code that references a known algorithm of the specified type. In order to process the data, all algorithms being referenced must be installed and made accessible to the encoder or decoder.
• each parameter is composed of a parameter name, a data type, the number of values in the parameter and the parameter value(s).
• Table 1 is example syntax of Compressor Parameter Set, which captures the key elements of a compressor configuration as described above, and explanations on the semantics are provided following the table. Note that the name, ordering and data type of individual fields can be changed without any impact on the functionality.
• u( «) denotes unsigned integer using n bits
• u(v) denotes unsigned integer with the number of bits dependent on the value of other syntax elements
• f( «) denotes fixed-pattern bit string using n bits
• st(v) denotes null-terminated string with a variable length.
• compressor lD is the unique identifier of the compressor within the annotation table. Note that the value 0 is reserved for no compression.
• n compressor steps is the total number of processing steps in this compressor configuration.
• compressor_step_ID[i] is the identifier of the compressor step
• algorithm_ID[i] is the identifier of the algorithm to be applied in this step.
• Each algorithm type should have its own ID map;
• use_default_pars[i] is a flag, if set to one, indicates default parameters of the algorithm are used and therefore no changes to the parameters are needed.
• j] is the predefined ID of an input variable known to the algorithm
• j] prev_out_var_ID[i][j] are respectively the ID of the previous compressor step and the ID of the corresponding output variable that contains the data to be passed to the current algorithm through the input variable specified by in_var_ID[i][j].
• n_completed_out_vars[i] is the number of completed output variables of the algorithm that require no further processing and can go directly to the output buffer of the compressor.
• completed_out_var_ID[i][j] specifies the predefined ID of an output variable known to the algorithm that is completed and requires no further processing.
• nesting zero bit is one bit set to 0 inserted as necessary to ensure byte alignment of the data structure.
• Algorithm Parameters is a data structure for specifying the parameter settings of the algorithm referenced by algorithm ID.
• Table 2 is a table of example syntax for the Algorithm Parameters.
• n_pars is the number of parameters that need to be modified for executing the algorithm referenced by algorithm ID. Its value must not be greater than the number of parameters defined in the algorithm.
• par_ID[i] is the ID of one of the parameters defined in the algorithm
• par_type[i] is the data type ID of the parameter
• par_num_array_dims[i] is the number of dimensions of the parameter, 0 if it is a scalar value
• par_val[i] [][][] contains the parameter value(s). Its number of dimensions is equal to (par array dims + 1) and any extra dimensions can be omitted.
• the file compression/decompression system comprises an Attribute Parameter Set.
• An Attribute Parameter Set is a data structure that contains the configuration of an attribute, including some basic information and the compressor settings for its processing.
• Attribute Parameter Set contains the definitions of additional attributes, including their parameters, grouping, ordering and associated compressors.
• Attribute Parameter Set semantics [0084] According to an embodiment are the following Attribute Parameter Set semantics:
• attribute lD is the identifier of the attribute, unique within an annotation parameter set.
• attribute name len is the length of the attribute name.
• attribute name is the name of the attribute with length attribute name len.
• attribute type specifies the data type of the attribute.
• attribute num array dims specifies the number of array dimensions of an attribute entry, 0 if it is a scalar value.
• attribute default value is the default value of the attribute (i.e., it can be used for sparse encoding when most values equal to the default are excluded).
• attribute missing value is the missing value of the attribute to be used in place of a null value in the output after the decoding process.
• compressor lD is the a compressor lD as defined in the compressor_parameter_set used for the processing of the current attribute.
• n steps with dependencies is the number of steps in the compressor referenced by compressor lD that involve dependency variables.
• compressor_step_ID[i] is the ID of the compressor step
• n_dependencies[i] is the number of dependency variables required by the algorithm of the compressor step.
• dependency_var_ID[i][j] is the predefined ID of the dependency variable known to the algorithm
• dependency_is_attribute[i][j] is a flag, if set to one, indicates the dependency data is stored in an attribute; otherwise, the dependency data is stored as a descriptor
• dependency_ID[i][j] specifies the ID of the attribute or descriptor containing the dependency data. Note that the length of the ID is 16 bits for attribute and 8 bits for descriptors.
• nesting zero bit is one bit set to 0 inserted as necessary to ensure byte alignment of the data structure.
• the encoding of the raw attribute data is done by the sequential execution of the compressor steps defined in the associated compressor configuration.
• the encoding algorithm is outlined as follows:
• the generated outputs consist of a list of data objects and their corresponding variable IDs. e. If an output variable is specified as completed, then its data object should be placed in the output buffer of the compressor. Otherwise, it should be stored in a data map to be looked up through its step and variable IDs by a subsequent step for further processing. Note that each encoded data block/object is stored using the typed data structure, as specified herein, to preserve its data type and array form.
• Table 4 Typed Data Syntax. f. For the last step, return the content of the output buffer, which is a concatenation of the encoded blocks.
• the file compression/decompression system comprises an attribute decoding process.
• the decoding of the compressed attribute data involves the reversal of the compressor steps defined in the associated compressor configuration by performing step-by- step inverse operations of the algorithms starting from the last step to the first step.
• the decoding algorithm is outlined as follows:
• each compressor step i a. Collect all input variable data of the current step either from the encoded blocks in step 2 above or from the output variable(s) of its upstream decoding step. b. If there are any dependency variables defined for the current step, obtain the decoded data of the dependency variables from the corresponding attributes or descriptors specified by dependency _is_attribute[i][] and dependency_ID[i][], c. retrieve from the compressor parameter set any changes to the default parameter settings for the current step. d. Apply an inverse operation of the algorithm specified for the current step on the input variable data using the associated parameter settings and any dependency variable data. e.
• step 1 If it has not reached the first step, store the output variable data to be further processed by subsequent decoding steps using the pair of step ID and variable ID as key for lookup. f. If it has reached the first step, fill in any missing values and return the output variable data of the inverse operation as the attribute data decoded by the compressor.
• compressor configurations can be pre-defined by a group of data compression experts and adopted for use by the community, there are times when compressor configurations need to be specified, for example, when handling new or exclusive file types, or when a file of known type allows the inclusion of unplanned attributes.
• One possible way is to let a user manually specify the compressor configurations, perhaps with the assistance of a graphical tool that offers different options and explanations. This approach, however, requires much user knowledge and experience on data compression, and often results in suboptimal configurations.
• the system can comprise a list of available transform and compression algorithms available for selection, and for each is known the types of data the algorithm can handle and the possible connections between the algorithms.
• the mechanism can work as follows. For each attribute in the file:
• a decoder can apply the aforementioned decoding algorithm on each attribute using the configuration data that comes with the file.
• Table 9 summarizes the various data sizes after applying Compressor Configurations 1 and 2 using the encoding algorithm.
• the compression ratios of Compressor Configurations 1 and 2 are respectively 5.19 and 3.68. Since Compressor Configuration 1 has a better performance in terms of compression ratio, it should be selected for use on the compression of sparse expression data in general.
• the example demonstrates that the method significantly reduces the size of the compressed data file relative to prior art methods (such as direct compression by BSC). This improves the functionality and efficiency of the file compression/decompression system, and revolutionizes storage capabilities of a computer system or database. Further, when dealing with a file comprising millions or billions of data points for each of one or more attributes, compression comprises millions or billions of calculations, something the human mind is not equipped to perform, even with pen and pencil.
• the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements.
• This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified.
• inventive embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed.
• inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and/or method described herein.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Physics & Mathematics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Databases & Information Systems (AREA)
• Bioethics (AREA)
• Medical Informatics (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Bioinformatics & Computational Biology (AREA)
• Biotechnology (AREA)
• Evolutionary Biology (AREA)
• General Health & Medical Sciences (AREA)
• Biophysics (AREA)
• Spectroscopy & Molecular Physics (AREA)
• Genetics & Genomics (AREA)
• Data Mining & Analysis (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Compression, Expansion, Code Conversion, And Decoders (AREA)
• Information Transfer Between Computers (AREA)

Priority Applications (7)

Applications Claiming Priority (4)

Publications (1)

Family

ID=78709427

Family Applications (1)

Country Status (7)

Families Citing this family (1)

Citations (3)

Family Cites Families (3)

• 2021
  • 2021-11-11 AU AU2021382348A patent/AU2021382348A1/en active Pending
  • 2021-11-11 JP JP2023530194A patent/JP2023551154A/ja active Pending
  • 2021-11-11 US US18/035,805 patent/US12380967B2/en active Active
  • 2021-11-11 EP EP21810975.9A patent/EP4248565A1/en active Pending
  • 2021-11-11 WO PCT/EP2021/081320 patent/WO2022106289A1/en not_active Ceased
  • 2021-11-11 KR KR1020237020674A patent/KR102950305B1/ko active Active
  • 2021-11-11 IL IL302962A patent/IL302962A/en unknown

Patent Citations (3)

Non-Patent Citations (3)

Also Published As

Similar Documents

Legal Events