KR20240078422A - Conditional transcoding for encoded data - Google Patents

Abstract

The transcoder is started. The transcoder may include a buffer to store input encoded data. The index mapper can map an input dictionary to an output dictionary. The current encoding buffer may store input encoding data, an input dictionary, and modified current encoding data that may be responsive to a map from the input dictionary to the output dictionary. The previous encoding buffer may store input encoding data, an input dictionary, and modified previous encoding data that may respond to a map from the input dictionary to the output dictionary. The rule evaluator may generate an output stream in response to current encoding data modified in the current encoding buffer, previous encoding data modified in the previous encoding buffer, and transcoding rules.

Description

Conditional transcoding for encoded data {CONDITIONAL TRANSCODING FOR ENCODED DATA}

The present invention relates generally to storage devices, and more specifically to transcoding of data within storage devices.

Storage devices such as solid state drives (SSD) can store relatively large amounts of data. The host processor can request data from the SSD to perform operations on the data. Transferring this data to the host processor may require a relatively significant amount of time, depending on the specific architecture connecting the host processor and the SSD. For example, if a host processor and an SSD are connected using 4-lane, third-generation Peripheral Component Interconnect Express (PCIe), the maximum amount of data that can be transferred between the SSD and the host processor is approximately 4 GB per second.

There is a need to reduce the amount of data transmitted to the host and take advantage of the columnar format.

The purpose of the present invention is to provide an apparatus and method that increases the speed and efficiency of data transfer between a storage device and a host.

A transcoder according to an embodiment of the present invention includes a buffer for storing input encoding data, an index mapper for mapping an input dictionary and an output dictionary, a map between the input encoding data, the input dictionary, and the input dictionary and the output dictionary. a current encoding buffer storing current encoding data modified in response to, previous input encoding data, the input dictionary, and a previous encoding buffer storing modified previous encoding data generated in response to the map, and in the current encoding buffer and a rule evaluator that generates an output stream in response to the stored modified current encoding data, the modified previous encoding data stored in the previous encoding buffer, and a transcoding rule.

A method according to an embodiment of the invention includes, at a transcoder, receiving a first data chunk from input encoded data from a storage device, determining whether the first data chunk is of interest to a host computer, the host computer generating a first encoded data from the first data chunk based at least in part on the first data chunk of interest, receiving, at the transcoder, a second data chunk from the input encoded data from the storage device. determining whether the second data chunk is not of interest to the host computer, performing a second encoding from the second data chunk based at least in part on the second data chunk that is not of interest to the host computer. Generating data and outputting the first encoded data and the second encoded data to the host computer.

An article comprising a non-transitory storage medium storing instructions according to an embodiment of the present invention includes the steps of: receiving, at a transcoder, a first data chunk from input encoded data from a storage device when the instructions are executed by a machine; , determining whether the first data chunk is of interest to a host computer, generating first encoded data from the first data chunk based at least in part on the first data chunk of interest to the host computer, In a transcoder, receiving a second data chunk from the input encoded data from the storage device, determining whether the second data chunk is not of interest to the host computer, Generating second encoded data from the second data chunk based at least in part on the second data chunk, and outputting the first encoded data and the second encoded data to the host computer.

When using the transcoder of the present invention according to the above-described characteristics, the storage device and the host device can improve data transmission speed and performance by minimizing unnecessary transactions.

1 shows a system including a storage device such as a solid state drive (SSD) capable of supporting transcoding of encoded data, according to an embodiment of the present invention.
Figure 2 shows some additional details of the machine of Figure 1.
Figure 3 shows the storage device of Figure 1 and the processor of Figure 1 exchanging the same data using different approaches.
Figure 4 shows the storage device of Figure 1 and the processor of Figure 1 exchanging transcoded data according to an embodiment of the present invention.
Figure 5 shows details of the storage device of Figure 1 according to an embodiment of the present invention.
Figure 6 shows details of the transcoder of Figure 4 according to an embodiment of the present invention.
Figure 7 shows the stream splitter of Figure 6, which separates input encoded data into chunks according to an embodiment of the present invention.
Figure 8 shows the index mapper of Figure 6, which maps an input dictionary to an output dictionary according to an embodiment of the present invention.
Figure 9 shows an example file saved in column format.
FIG. 10 shows the storage device of FIG. 1 implementing transcoding for data stored in a column format, according to an embodiment of the present invention.
FIG. 11 shows the column chunk processor of FIG. 10 that implements transcoding for data stored in column format according to an embodiment of the present invention.
FIGS. 12A to 12C are flowcharts of exemplary operations of the transcoder of FIGS. 4 to 6 for transcoding data according to an embodiment of the present invention.
FIG. 13 is a flow diagram showing an example procedure for the stream splitter of FIG. 6 to separate input encoded data into chunks.
FIGS. 14A to 14B are flowcharts showing exemplary procedures of the column chunk processor of FIG. 10 and/or the transcoder of FIG. 4 for transcoding data stored in a column format according to an embodiment of the present invention.
FIG. 15 is a flowchart showing an example procedure of the index mapper of FIG. 6 for mapping an input dictionary to an output dictionary according to an embodiment of the present invention.
Figures 16A-16B are flowcharts illustrating example procedures for the in-storage computer controller of Figure 10 for managing attributes received from the host computer of Figure 1 and potentially performing acceleration functions on transcoded data.

Reference is made to embodiments of the technical idea of the present invention, examples of which are shown in the attached drawings. In the detailed description that follows, various specific details are provided to facilitate a thorough understanding of the technical spirit of the present invention. However, those skilled in the art can implement the technical idea of the present invention without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks are not described in detail so as not to unnecessarily obscure aspects of the embodiments.

Although terms such as first, second, etc. are used herein to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, the first module may be named the second module without departing from the scope of the technical idea of the present invention. Likewise, the second module may be named the first module.

Terms used in the description of the technical idea of the present invention are used only for the purpose of describing specific embodiments and are not intended to limit the technical idea of the present invention. As used in the description of the technical idea of the present invention and the appended claims, singular expressions are intended to include plural expressions as well, unless clearly indicated from the context. The term “and/or” is intended to include any and all possible combinations of one or more associated items. The terms “comprise” and/or “comprising” when used in the detailed description specify the presence of the recited features, integers, steps, operations, elements, and/or components, and include one or It does not exclude the presence or addition of many other properties, integers, steps, operations, elements, components, and/or groups thereof. The elements and features of the drawings are not necessarily to scale.

For example, using a Field Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), Graphics Processing Unit (GPU), or other processor to bring some processing capabilities closer to an SSD has several advantages. First, the connection between the SSD and the near processor can support higher bandwidth than connecting the SSD with the host processor, allowing for faster data transfer. Second, by eliminating the need for the host processor to process data, the near processor can process the data while the host processor can perform other functions.

However, near-storage processing of data has potential drawbacks when compressing or encoding data. Some near-storage processors that operate on raw data may decompress or decode the data before operating on it. Additionally, the near-storage processor can report results back to the host processor. If the resulting amount of data transferred to the host processor is greater than the amount of raw data, the benefits of using a near-storage processor are lost, or, in the worst case, if the compressed or encoded data were transferred to the host processor in the first place. More data can be transmitted to the host processor.

Additionally, transcoding may be performed on data in general, but if the data is stored in a columnar format, some adaptation may be performed to utilize the columnar format.

Near-data processing of data in compressed formats can negate some of the benefits of offloading. If the connection between the SSD and the host processor supports the transfer of The amount of data may be 'X*Y*Z'. If the product is lower than

In some embodiments of the invention, column storage may use data encoding (e.g., Run Length Encoding (RLE)) and/or compression (snappy) to reduce storage space. Encoding rather than compression can provide major entropy reduction. After encoding, the compression ratio tends to be small (less than about 2).

In some embodiments of the invention, for example, based at least in part on an encoding algorithm, the encoded data mat is near-processed without inflating the results (i.e., when the encoded raw data is transmitted to the host processor). larger results are sent to the host processor). Encoding algorithms that can be used without bloating the results include Dictionary compression, Prefix Encoding, Run Length Encoding (RLE), Cluster Encoding, Sparse Encoding, and Indirect Encoding ( Indirect Encoding) may be included, but is not limited thereto, and other encoding algorithms may be used in conjunction with embodiments of the present invention. Embodiments of the invention described below will focus on RLE and bit packing, but embodiments of the invention can be extended to include other encoding algorithms.

There are also additional questions about how to teach the transcoder what data to filter. This is a problem where the dictionary may be stored somewhere other than where the data is stored, which reduces the size of the data stored. Column stores, an example of this storage format, make it easy to find data of interest. However, because the dictionary may be stored separately from the data, the system must be able to find the dictionary as well as the corresponding data to perform transcoding.

Embodiments of the present invention enable filtering of encoded data without inflating the data. Filtered data can be re-encoded using encoding information embedded in the encoded data using conversion rules. The transcoder in embodiments of the present invention may filter encoded data and modify the encoding transmitted to the host. Therefore, instead of the host having to process plain data (which can be significantly larger than the encoded/compressed data depending on the compression algorithm and/or the efficiency of the encoded data), the host can receive and process the encoded data. Because the bandwidth between the host and the storage device can have limitations that can substantially affect the time it takes to transfer data, sending encoded data requires less processing time compared to sending plain data (filtered or unfiltered). You can save.

A circular buffer can store enough data to be processed at one time. Embodiments of the present invention may replace the circular buffer with a buffer using a different structure.

The index mapper can provide a mapping from the input dictionary map to the reduced dictionary map for use with the output stream.

The current encoding buffer can store data read from the input stream according to the appropriate encoding. A rule evaluator using information from the transcoding rules, the current encoding buffer, and the previous encoding buffer can determine how to process data in the current encoding buffer. Depending on whether the data in the current encoding buffer can be combined with the data in the previous encoding buffer, the rule evaluator updates the previous encoding buffer based on the data in the current encoding buffer, outputs the previous encoding buffer (and then outputs the previous encoding buffer). replace the buffer), or perform other actions. For example, if the transcoder identifies values that are considered "don't care" values in the current encoding buffer (discussed further below), those values will be replaced with existing "don't care" values in the previous encoding buffer. It can be combined with the value "(don't care)".

A stream splitter can be used to identify different parts (streams) of an input stream encoded using different encodings. If a single encoding scheme is used, the encoding scheme can be passed as a parameter (i.e., encoding type). Otherwise, if multiple encoding schemes are used (i.e., no encoding type is used), the encoding scheme for a given stream is determined by examining the input stream itself. For example, the first byte of data stored in column storage format encoding may include encoding type information. In the case of a mixture of RLE (Run Length Encoding) and bit packing, if LSB is '0', 'Encoding Type = RLE', and if LSB is '1', 'Encoding Type = Bit Packing'.

As examples of different encoding operations, consider RLE and bit packing (BP). In RLE, a variable unsigned integer is used to indicate how often a value is repeated, and a fixed length value is given. So, for example, instead of sending '00000011 00000011 00000011 00000011 00000011 00000011 00000011 00000011 00000011 (9 copies of decimal value 3)', the data would be '00001001 (9 copies of decimal value 9) To be encoded as '00000011 (decimal value 3)' This indicates that '00000011' is repeated 9 times.

In bit packing (BP), data determined to take up less space can be combined with other values. For example, if data is normally stored using 8 bits, it would take up a total of 32 bits to store 4 values. However, if the values are known to be no more than 4 bits each, two values can be stored in a single byte, i.e. this is bit packing. The space savings are slightly less than described, since there is some overhead in representing compressed and uncompressed data, but they are still advantageous.

The encoding includes a group number of unsigned bytes followed by a list of group values of one or more bytes. The maximum number of values in a group is 8 and the maximum number of groups can be 63. Therefore, for example, to represent the data '00000000 00000001 00000000 00000001 00000000 00000001 00000000 00000001 (decimal value 01010101)', the group is '00000001 (group 1) (0, 1) 00010000 (0, 1) 00010000 (0, 1) It can be defined as ‘00010000 (0, 1)’.

As mentioned above, RLE (and other encodings) can use variable unsigned integers. Variable unsigned integers can also use encoding. In every 8-bit group, the most significant bit may indicate whether the current byte is the last byte of the value or whether there is at least one subsequent byte. If multiple bytes are used, the least significant byte is displayed first, and the most significant byte is displayed last. Thus, for example, the decimal value '1' may be displayed as '00000001', the decimal value '2' may be displayed as '00000010', and so on until '01111111 (decimal value 127)'. The decimal value '128' can be displayed as '10000000 00000001', the decimal value '129' can be displayed as '10000000 00000010', etc. Basically, binary values are divided into groups of 7 bits, with each group of 7 bits starting with 1, except for the highest group. For example, the decimal value '16,384' can be displayed as '10000000 10000000 00000001'.

When processing encoded data using a transcoder, some data may be considered "don't care" data. This means that there may be some data that is not valuable to the task being performed. Data that is considered "don't care" data may be mapped to different values as a result of transcoder operation.

Let's consider a situation where a database stores citizenship (nationality) information for various people. Citizenship can be stored using a string (such as "China", "Korea", "India", "United States", etc.). However, because the possible values for citizenship are drawn from a finite set, dictionaries can be used to reduce the amount of data stored in the database. Therefore, a representative value rather than the name of a country, for example, a value of "0" represents China, a value of "1" represents India, a value of "2" represents South Korea, and a value of "3" represents the United States. (Indexes) can be stored in the database. Since there are 195 countries (as of July 19, 2019), we can use 1 byte to store the index, which is much less than what would be used to store a string of country names, using 1 byte per character.

However, the accelerated operation being performed (for example, the operation may be calculating the number of U.S. citizens in a database) may be of interest to U.S. citizens. Therefore, citizens of other countries are not relevant to the task: these are “indefinite” values. The transcoder can map dictionaries and indices to reflect the data to which operations are applied.

Column formats can use RLE or bit packing (BP) to encode information. Given a portion of a value string stored in a columnar storage format, 1 bit can be used to indicate whether the data is stored using RLE or bit packing; The remaining data can be understood accordingly.

To understand how a transcoder according to embodiments of the present invention can provide an alternative dictionary for encoded data, consider a situation where the data contains citizenship information for many people. Although the names of the countries of which each individual is a citizen are very long, the number of country names is relatively small (even representing 200 countries would only take up about 8 bits, which would still make each citizen's country name string at 1 byte per character of the country name). However, dictionaries can significantly reduce the amount of data stored. Such encoding may use any desired encoding scheme (e.g., RLE encoding, dictionary compression, prefix encoding, bit packing, cluster encoding, sparse encoding, and indirect encoding).

Now, if the applied predicate (filtering the data) looks for only US citizens, then data related to citizens of other countries is not of interest. For example, a host might want to know how many U.S. citizens are in its database. As a result of the conversion, the dictionary can be reduced to one entry for US citizens (there may be an implicit or explicit entry for the "don't care" entry), and the RLE encoding can be converted to a non-US citizen. It can be compressed to combine adjacent RLE items for citizens of different countries. Therefore, the encoding of data is compressed into a dictionary containing 1 (or 2) rows. Since data related to non-US citizens can be indexed into a single entry in the new dictionary, the actual encoding data can also be reduced. Therefore, by pushing a Predicate into the transcoder, encoded data can be filtered and new encodings can be provided that ultimately reduce the amount of data transmitted to the host. A dictionary map may represent a mapping between an original dictionary and a transcoding dictionary.

Although a Field Programmable Gate Array (FPGA) may be used to implement a transcoder (among other features), embodiments of the present invention may utilize, for example, an Application-Specific Integrated Circuit (ASIC), Graphics Processing Unit (GPU), etc. or other implementation forms including another processor executing software. Additionally, the In-storage Compute (ISC) controller may be separate from the FPGA or may be implemented as part of the FPGA.

Given a particular file on which an acceleration function (such as filtering) is to be performed, the ISC controller can use the File-Block (File2Block) map to identify the blocks that store the file's data, along with their order. The ISC controller can be implemented as a component within the host (separate from the storage device itself), or it can be a controller that is part of the storage device. These blocks can be accessed to provide an input stream that can be input (via an input buffer) to the transcoder.

When a file is stored in columnar format, the data unit may itself be a column chunk containing multiple data pages. That is, the input buffer can receive column chunks from storage modules within the storage device, so that the transcoder can operate on the column chunks. In general, each column chunk may contain its own metadata that can specify the encoding method used for that column chunk and/or the dictionary to be applied to the data in that column chunk. However, not all storage formats use this arrangement: for example, columnar storage formats may store metadata in separate areas of the file (as opposed to within each column chunk): You can specify the encoding and dictionary used together. Therefore, when a file is stored using this columnar storage format, the ISC controller retrieves (retrieves) the encoding and dictionary from the metadata area of the file (located using the File2Block map), and allows the transcoder to retrieve whatever it wants from the columnar chunks. Rather than assuming that the information is being thrown or received (of course, if you are using a columnar storage format, there may be no dictionary pages in the column chunks), you provide that information to the transcoder. The same encoding method can be applied to all column chunks, but the encoding method itself may be a hybrid method that uses two or more clearly distinct encoding methods and switches between them as appropriate. For example, a hybrid encoding method may be a method that combines RLE encoding and bit packing.

In addition to determining the dictionary and encoding method, the ISC controller can also extract attributes to be applied to the encoded data and push those attributes down to the transcoder. A transcoder can use all this information in a variety of ways. For example, information about the encoding used with a file can be used to select transcoding rules to be used with the data, while dictionaries and properties can be used to create transcoding dictionaries and dictionary maps.

The attribute evaluator uses the attributes to determine which items in the dictionary are of interest and which are not, a transcoding dictionary that stores the values of interest (and possibly values representing "undefined" items), and a transcoding dictionary from the original dictionary. Create a dictionary map that maps indices.

If the transcoding dictionary contains an entry for a "don't care" value, this operation technically adds the entry to the dictionary (since the original dictionary does not contain such a value). Adding these items may cause new problems. Adding a "don't care" entry to the transcoding dictionary typically occurs in the first entry of the transcoding dictionary (index 0), and is intended to represent a value that does not match an attribute. However, generating a new value for a "don't care" item can be expensive: that is, the system being launched will have to scan the entire dictionary (since all existing indices are 1 apart) and try again. It can be mapped. Adding a "don't care" entry may result in memory reallocation or a bitwidth overflow: for example, if all possible values for a given number of bits are already in use as dictionary indices. Inserting a "don't care" entry into the dictionary may increase the number of bits used to represent the index by one. If a data page uses part of a dictionary, the bitwidth of the data page may be smaller, and adding a "don't care" entry to the transcoding dictionary can result in a single valid value from the data page. It becomes unusable. For example, with a bitwidth of 1, adding a "don't care" entry may contain more values than can be represented using a single bit, whereas with a bitwidth of 2 , there may be room for space for "don't care" entries without overflowing the bitwidth.

The solution to this problem is to determine whether attribute pushdown will reduce the dictionary size. If the dictionary can be reduced to at least one entry, then space is reserved for "don't care" entries without worrying about bitwidth overflow. If the dictionary is not reduced by at least one entry, the encoded data can be sent directly to the ISC controller/host without performing transcoding, thereby avoiding the possibility that transcoding increases the amount of data.

The output of the transcoder can be returned back to the ISC controller (via an output buffer). This serves two purposes. First, although pushing the properties down into the transcoder can produce transcoded data, there may still be operations that need to be performed on the transcoded data. For example, if a host attempts to count the number of US citizens in a file, the transcoded data identifies those citizens but does not count them: that operation can be performed by an acceleration function in the ISC controller. Second, the transcoded data can be sent back to the host for further operations. Since the ISC controller communicates with the host, it provides a path to send transcoded data to the host.

Figure 1 shows a system including a solid state drive (SSD) capable of supporting transcoding of encoded data according to an embodiment of the present invention. Machine 105, which may be a host computer in FIG. 1, may include a processor 110, memory 115, and storage device 120. Processor 110 may be any of a variety of processors. 1 shows a single processor 110, machine 105 may include any number of processors, each processor may be a single-core or multi-core processor, and can be combined in any desired combination. It can be.

Processor 110 may be connected to memory 115. The memory 115 may be flash memory, Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Persistent Random Access Memory (Persistent Random Access Memory), Ferroelectric Random Access Memory (FRAM), or magnetoresistive random access memory ( It may be any of a variety of memory, such as non-volatile random access memory (NVRAM), such as MRAM), etc. Memory 115 may be any desired combination of different memory types and may be managed by memory controller 125. Memory 115 may be used to store data “short-term,” where the data is not expected to be stored long-term. Examples of short-term data may include temporary files, data used locally by an application (which may have been copied from another storage location), etc.

Processor 110 and memory 115 may support an operating system on which various applications can run. These applications may issue requests to read or write data from memory 115 or storage device 120. Memory 115 may be used to store data referred to as “short-term,” whereas storage device 120 may be used to store data that is “long-term,” i.e., where the data is expected to be stored for a long period of time. can be used Storage device 120 may be accessed using device driver 130. Storage device 120 may be of any desired format, such as a hard disk drive (HDD), solid state drive (SSD), and any other desired format.

Figure 2 shows details of the machine of Figure 1. 2, machine 105 generally may include one or more processors 110, which may include a clock 205 and a memory controller 125 that may be used to coordinate the operation of the machine's components. . Processors 110 may be coupled to memory 115, which may include, for example, random access memory (RAM), read-only memory (ROM), or other stateful media. Processors 110 may also be connected to storage devices 120 and a network connector 210, such as an Ethernet connector or a wireless connector. Processors 110 may also be coupled to buses 215, which may be an input/output interface that may be managed using a user interface 220 and an input/output engine 225, among other components. Can be mounted on a port.

Figure 3 shows the storage device 120 of Figure 1 and the processor 110 of Figure 1 transferring the same data using different approaches. In one approach (the general approach), the data is stored in storage 305 in storage device 120, which may be, for example, platters on a hard disk drive or flash memory chips in a flash memory storage device, such as an SSD. It can be read from and transmitted directly to the processor 110. If the total data stored (encoded and/or compressed) on storage device 120 is X bytes, then this is the amount of data to be transferred to processor 110. This analysis takes into account the amount of storage used to store encoded and/or compressed data, where unencoded and uncompressed data would probably be bytes larger (otherwise, the amount of storage used to encode and/or compress the data would be larger). may not be advantageous). So, for example, if data has approximately 10 GB of storage available unencoded and uncompressed, but has approximately 5 GB of storage available when encoded and/or compressed, then approximately 5 GB of data, rather than approximately 10 GB, will be stored on the storage device. It may be transmitted from 120 to the processor 110.

Data transfer from storage 120 to processor 110 may be considered in terms of the bandwidth provided to transfer the data (and consequently the time used affecting the transfer). When data stored in storage device 120 is encoded and/or compressed, when data stored in storage device 120 is transmitted directly to processor 110 (as shown by arrow 310), storage device 120 ) can be transferred at an effective rate of B bytes per second. Continuing with the example described above, let us consider a situation where the connection between storage device 120 and processor 110 includes a bandwidth of approximately 1 GB per second. Because encoded and/or compressed data can occupy approximately 5 GB of space, encoded and/or compressed data can be transmitted at a transfer rate of approximately 1 GB/sec for a total of 5 seconds. However, since the total data stored (before encoding and/or compression) is about 10 GB, the effective transfer rate of data B is about 2 GB per second (since about 10 GB of unencoded and uncompressed data is transferred in about 5 seconds).

In contrast, if an in-storage processor 315 is used to pre-process data to reduce the amount of data transmitted to processor 110, less raw data will be transmitted. (since the in-storage processor 315 is more selective about the data being transferred). Meanwhile, the in-storage processor 315 decompresses and processes the data. (You can also decode the data if possible). Accordingly, the amount of data to be transferred from in-storage processor 315 to processor 110 may be reduced by the selection of data, but may also be increased by the amount of compression (and possibly encoding): logarithmically, in-storage Data to be transferred from processor 315 to processor 110 (shown via arrow 320) may be expressed in 'X*Y*Z' gigabytes, where 'X' is the encoded and/or compressed 'Y' is the amount of space used to store the data, 'Y' is the compression ratio (how much data is stored reduced using compression (and possibly encoding)), and 'Z' is the selectivity ratio (how much data is selected from the uncompressed data). )am. Similarly, the effective rate at which data can be transferred from the storage processor 315 to the processor 110 is 'B*Y*Z' bytes/second.

A simple comparison of the two formulas is that using the in-storage processor 315 to select data to send to processor 110 when 'X*Y*Z < It shows excellence when Y*Z < 1'. Otherwise, the amount of data to be transmitted after preprocessing by the in-storage processor 315 is greater than the amount of data encoded and/or compressed even if the in-storage processor 315 does not apply selectivity: i.e., the in-storage processor 315 It is more efficient to transmit the original encoded and/or compressed data than to attempt to select data to transmit to processor 110.

FIG. 4 shows the storage device 120 of FIG. 1 and the processor 110 of FIG. 1 exchanging transcoding data according to an embodiment of the present invention. As shown in Figure 4, the encoded and/or compressed data is stored in storage 305 (again, storage 305 may be a platter of a hard disk drive, a flash memory chip of a flash memory storage device such as an SSD, or some physical data storage). This data (compressed data 405) may be passed to decompressor 410, which may decompress the data to produce decompressed data 415. The decompressor 410 (or decompression engine) may perform hardware decompression or use appropriate circuitry (general-purpose processor, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), graphics processing unit (GPU), or general purpose (GPGPU). The decompressed data 415 may be additionally encoded because encoding and compression may be separate processes. Transcoding can be thought of as a process of converting data from one encoding to another encoding.

All of the above-described processes may occur within storage device 120. Once transcoder 420 processes decompressed data 415 and produces transcoded data 425, transcoded data 425 may be provided to host computer 105. The decoder 430 may decode the transcoded data 425 to generate filtered plain data 435. The filtered plain data 435 may be provided to the processor 110 and then a desired operation may be performed on the filtered plain data 435.

It should be noted that decoder 430 decoding transcoded data 425 may include knowing something about the encoding applied to transcoded data 425. This information may include, for example, the specific encoding scheme used in the transcoded data 425, or the dictionary used in the transcoded data 425. 4 does not show this information being transferred from storage device 120 to host computer 105, but this information is transferred to host computer 105 in parallel with (or as part of) transcoded data 425. It can be delivered. Of course, the transcoded data 425 is actually unencoded and uncompressed (if the result of the operation of the transcoder 420 transmits a larger number of actual bytes than would transmit unencoded and uncompressed data). may occur), the transcoded data 425 may omit information about the encoding method or dictionary.

At this point, it is worth discussing the difference between encoding and compression. Although the two concepts of reducing the amount of storage used to store data are related, there are some differences. Encoding usually involves using a dictionary to provide an index to data that is too long to contain directly and has a relatively small number of distinct values. For example, there are about 195 different countries. If your data stores information about the citizenship of a large number of people, including each person's nationality directly will use a large amount of space, using at least a few bytes (assuming 1 byte per character in the name of the country). I do it. On the other hand, all values 1-195 can be expressed using a single byte. By using a dictionary to represent country names and storing an index of the appropriate country names in the data, the amount of data to be stored can be significantly reduced without loss of information. For example, the information "United States, United States, Korea, Korea, Korea, China, India, China, China, China, China, China, United States" could instead be displayed in a dictionary as shown in Table 1, and " It is displayed as "3, 3, 2, 2, 2, 2, 0, 1, 0, 0, 0, 0, 0, 3" and is reduced from 153 characters to 40 characters. Even considering the 52-character dictionary, simply using a dictionary can save a lot of money.

ID nation 0 China One India 2 Korea 3 United States of America

The value of the dictionary can decrease as the number of values in the dictionary increases. For example, if there are 1,000,000 different possible values, each index could be stored using 20 bits. Of course, this may still be less than the number of bits used to store the value directly, but the benefit of encoding (related to storing unencoded data) may be reduced. And, if the value stored for each item of data is unique, or if the amount of space used to store the index is approximately the same size as the amount of space used to store the value, then using an encoding that uses a dictionary will only allow the data to be stored can actually increase the amount. Continuing with the example of data about people, using a dictionary to store ages is no more efficient than storing ages directly.

On the other hand, compression generally uses coding methods such as Huffman code. The data is analyzed to determine the relative frequency of each data, so that more frequent data is assigned shorter codes and less frequent data is assigned longer codes. Morse code, although not the Hoffman code, is a well-known example of a code that uses short sequences for more frequent data and longer sequences for less frequent data. For example, the letter "E" can be represented by the sequence "dot" (followed by a space), while the letter "J" can be represented by the sequence "dot dot dash dash dash" (followed by a space). can be displayed. (Morse code uses spaces to indicate where one symbol ends and another begins, and because a sequence for one symbol can be a prefix of a sequence for another symbol (the "E" is represented by a dot, whereas "J" begins with a dot but contains other symbols), Morse code is not proper Hoffman code, but many people have some familiarity with Morse code and use shorter symbols for more frequent data and less frequent symbols. (This makes it a commonly useful example of code that uses longer codes as symbols for frequent data).

Going back to encoding methods, once the dictionary is set up, there are several different encoding methods that can be used to further encode the data. Examples of such encoding schemes include Run Length Encoding (RLE), bit packing, prefix encoding, cluster encoding, sparse encoding, and indirect encoding: embodiments of the present invention may also use other encoding schemes. It may be possible. Run length encoding (RLE) and bit packing are discussed here as they will be used in various examples later; Information about other encoding methods is easy to find.

Run Length Encoding (RLE) relies on the premise that values often occur in groups. Instead of storing each value separately, a single copy of that value can be stored with a number indicating how often that value occurred in the data. For example, if the value "2" occurs four times in a row, instead of storing the value "2" four times (which would use 4 bytes of storage), we store the number of occurrences of the value "2" ("4") and can be stored together (2 bytes of storage available). So, continuing the example above, the sequence "3, 3, 2, 2, 2, 2, 0, 1, 0, 0, 0, 0, 0, 3" becomes "[2, RLE], 3, [ 4, RLE] 2, [1, RLE] 0, [1, RLE], 1, [5, RLE], 0, [1, RLE], 3". The "[2, RLE], 3" encoding can be understood to mean that there is information encoded using RLE: the value is "3", and this value is repeated twice; Other RLE encodings are similar. (The reason the expression includes an indicator that RLE encoding is used is related to the potential use of hybrid encoding schemes, discussed with reference to Figure 7 below.) This sequence can use a total of 12 bytes: for each encoding 1 byte stores the number of times the next value is repeated, and 1 byte stores the value to be repeated.

Compared to 14 bytes to store the original sequence, 12 bytes does not significantly reduce the amount of space to store data. However, proportionally this encoding can reduce the amount of storage required for this data by approximately 14%. With data taking up about 5GB, even a roughly 14% reduction in used storage is a significant savings, potentially saving around 700MB.

As an alternative to the number of occurrences of each value, the starting position of each group can be stored. If you use the starting position instead of the number of occurrences of each value, the data is "[0, RLE], 3, [2, RLE] 2, [6, RLE] 0, [7, RLE], 1, [8, RLE ], 0, [13, RLE], 3".

The above discussion uses RLE to describe situations where repeated values fit into a single byte. Otherwise, what if, for example, the repeated value is "1000" ("1000" can be stored using 10 bits)? In this case, RLE can serialize values into 7-bit groups. The 8th bit of each byte, which may be the most significant bit in the byte, may indicate whether the byte continues from another byte.

For example, let's consider the value "1000". The value "1000" can be expressed in binary as "11 1110 1000". Because this representation uses 10 bits, the value may be too large to store in a single byte. Therefore, the value can be divided into groups of 7 bits (leading '0's are added so that each group contains 7 bits): "0000111 1101000". Now, you can add a "1" in front of the first byte of a sequence to indicate that the value it represents continues in the next byte, and a "0" in front of the second byte in a sequence to indicate that the value ends at that byte. Therefore, the bit sequence becomes “10000111 01101000”. When the system reads this bit sequence, it looks at the most significant bit in each byte, determines whether the value lasts beyond that byte, or ends with that byte, and removes that bit when reassembling the bit sequence into a value. You can see that Therefore, "10000111 01101000" becomes "0000001111101000" (adding two additional leading zeros brings the representation up to 2 bytes), and the original value "1000" can be recovered.

Of course, if one bit in each byte is used to identify whether the byte value is a continuation or not, that bit may not be used as part of the value. Therefore, even if a value fits a single byte, additional bits may be included to indicate that the value is not continued in other bytes. Additionally, if a value fits in 8 bits but not 7 bits (for example, a value between 128 and 255), 2 bytes are used to store the entire value, with the bit indicating whether to continue using the value in the next byte. can be represented (since the most significant bit of the value is moved to the next 7-bit group in the encoding).

When using RLE, bits and/or bytes can be presented in any desired order. For example, the bits may be given from most significant bit to least significant bit, or from least significant bit to most significant bit; Bytes can be similarly aligned in two ways. Thus, for example, if the bytes are presented from least significant to most significant, but the bits of each byte are presented from most significant to least significant and use consecutive bits, the value "16384" may be encoded as "10000000 10000000 00000001". This bit sequence can be interpreted as follows: The first bit of each byte is a continuation bit ("1" indicates that the next byte continues the value, "0" indicates that the value does not continue in the next byte). indicates). After removing the consecutive bits, what remains is "0000000 0000000 0000001". When the bytes are reordered from most significant to least significant (and reorganized into existing 8-bit groups by discarding leading '0's), the value becomes "01000000 00000000", which corresponds to the binary value "16384".

Bit packing, on the other hand, uses the idea that bits whose value is less than the total byte can be used. For example, if the values to be stored include 0, 1, 2, and 3, 2 bits may be used to represent each value. A full byte can be used to store each value, but using a full byte means that 75% of the storage is actually unused. Bit packing takes advantage of this phenomenon by storing more than one value in a single byte (or byte sequence). Bit packing is especially advantageous when the sequence of values is repeated rather than a single value.

For example, consider the sequence "0, 1, 0, 1, 0, 1, 0, 1", and use approximately 4 bits to uniquely identify each value (i.e., values greater than 15 are not used). Let’s consider (not). Instead of storing each value separately (requiring a total of 8 bytes), you can use the "[4, BP] 0, 1" encoding. This encoding indicates that a single byte stores 4 bits representing the value "0" and 4 bits representing the value "1", and that byte is repeated four times. (Like RLE encoding, bit packing encoding can include an indicator that the data has been encoded using bit packing for use in a hybrid encoding scheme.) The first byte indicates the number of times the data in the group is repeated; The second byte stores the value in the group itself. This encoding can use about 2 bytes to store the data, resulting in about a 75% reduction in the amount of storage used by the sequence.

When using bit packing, data can be packed in any desired way. For example, when packing the sequence "0, 1" where each value uses 4 bits, the sequence would be "00010000" (packing the values from the least significant bit to the most significant bit) or "00000001" (packing the values from the most significant bit to the least significant bit). can be expressed as packing). Some implementations of bit packing may use either strategy, but reverse the order in which the bits are placed in the stream (corresponding to the least significant bit first). In bit packing, other techniques may be used to pack the bits.

Of course, bit packing does not limit the groups that fit into a single byte. Like RLE, values in bit packing can use bits to identify whether the value continues in the next byte.

Because both encoding and compression attempt to reduce the space used to store data representations, their benefits may not be multiplied. Both encoding and compression attempt to reduce the space used to store data. However, if data is compressed in one way (such as encoding), applying a different compression method (such as compression) may not help. Compression can be applied to the data after it has been encoded, and may still slightly reduce the amount of storage space used, but the impact of compression on encoded data may be less than the benefit of compression on unencoded data. (If all methods of compressing data could be applied to the same advantage, regardless of the data being compressed, one would hope to be able to reduce all arbitrary data to an incredibly small size simply by applying repeated compression methods. As is readily apparent after a little thought, such results are not realistic in the real world.)

Figure 5 shows details of storage device 120 of Figure 1. Referring to FIG. 5 , the storage device 120 is shown as an SSD, but embodiments of the present invention may support other types for the storage device 120 with appropriate modifications. As shown in FIG. 5, the storage device 120 includes a host interface layer 505 (HIL), an SSD controller 510, and various flash memory chips 515-1 to 515-8, also referred to as “flash memory storage.” ), which can be connected to various channels (from 520-1 to 520-4). Host interface layer 505 may manage communications between storage device 120 and machine 105 of FIG. 1 . These communications may include requests to read data from and to write data to storage device 120 . The SSD controller 510 may use a flash memory controller (not shown in FIG. 5) to manage read and write operations along with garbage collection and other operations on the flash memory chips 515-1 to 515-8.

SSD controller 510 may include a translation layer 525 (also referred to as a flash translation layer (FTL)). The conversion layer 525 may perform the function of converting the logical block address (LBA) provided by the machine 105 of FIG. 1 into the physical block address (PBA) of the SSD 120 where data is actually stored. In this way, machine 105 of Figure 1 can reference data using its own address space without having to know the physical address on storage device 120 where the data is actually stored. This may be advantageous, for example, when data is updated: because storage device 120 may not have updated the data in its original location, storage device 120 invalidates existing data and Updates can be recorded in the new PBA. Alternatively, if data is stored in a block selected for garbage collection, the data may be written to a new block on the storage device 120 before the block is erased. By updating the translation layer 525, the machine 105 of Figure 1 is decoupled from where the data is actually stored as the data is moved to a different physical block address (PBA).

SSD controller 510 may also include a file-block map 530. The file-block map 530 can specify which blocks are used to store data for the file. File-block map 530 can be used, for example, when data is stored in columnar format. File-block map 530 may be part of translation layer 525 (in which case file-block map 530 may not be considered a separate component of storage device 120), or translation layer 525 (e.g., the transform layer 525 may be used for data using a relatively small number of blocks, while the file-block map 530 may be used for data using a relatively large number of blocks). ), or the conversion layer 525 may be completely replaced (in this case, the conversion layer 525 may not be present in the SSD controller 510).

SSD controller 510 may also include transcoder 420. However, embodiments of the invention may include configurations with transcoder 420 elsewhere within storage device 120 (e.g., transcoder 420 may be located elsewhere within storage device 120, among many possibilities). It may be implemented using a general-purpose processor (running appropriate software), an FPGA, ASIC, GPU, or GPGPU), or even external to the storage device 120.

Storage device 120 may also include an in-storage processor 315 (not shown in FIG. 5) of FIG. 3 that can execute instructions that control how to use data stored in storage device 120. Additionally, the in-storage processor 315 of FIG. 3 may be used for in-storage computing functions to execute operations locally on the storage device 120 instead of the processor 110 of FIG. 1. Like transcoder 420, in-storage processor 315 of FIG. 3 may use a general-purpose processor (running appropriate software), FPGA, ASIC, GPU, or GPGPU located somewhere within storage device 120, or even It may also be implemented outside of the storage device 120.

5 illustrates a storage device 120 including eight flash memory chips 515-1 to 515-8 comprised of four channels 520-1 to 520-4, according to an embodiment of the present invention. Can support any number of flash memory chips configured with any number of channels. Similarly, Figure 5 shows that the SSD controller 510 may include a transcoder 420 and/or the in-storage processor 315 of Figure 3, although embodiments of the present invention, unlike Figure 5, do not include a transcoder. It may be configured as 420 or the in-storage processor 315 of FIG. 3.

Figure 6 shows details of transcoder 420 of Figure 4. Referring to FIG. 6, the transcoder 420 may receive various inputs such as input dictionaries, input streams, and encoding types, and generate various outputs such as output dictionaries and output streams. In short, transcoder 420 can operate to take an input stream that can be encoded using an encoding scheme specified by the encoding type and produce an output stream. (Although the input stream may be encoded, the description below considers the situation where the input stream is uncompressed: if the input stream is compressed, the input stream will be decompressed before further processing.) The output stream will have the same encoding as the input stream. method, or may be encoded using a different encoding method (or both: as discussed below, when a hybrid encoding method is used, some data may be changed from one encoding method to another).

Additionally, the encoding itself can be changed even if the encoding method does not change between the input stream and the output stream. For example, if a particular value is assigned to a different index in the input and output dictionaries, changes in the dictionary should be reflected in the values used in the actual data. To this end, transcoder 420 can also take an input dictionary and map it to an output dictionary.

ID nation 0 Don't Care One United States of America

As an example of these last two points, let us again consider the dictionary shown in Table 1 above. We now consider a situation in which host computer 105 of Figure 1 was interested in data regarding U.S. citizens. Table 1 represents the data received from the input stream and can therefore be viewed as an input dictionary. Meanwhile, Table 2 may be an output dictionary representing data of the output stream. There are at least three caveats about Table 2. First, Table 2 contains two items compared to the four items shown in Table 1. Second, Table 2 includes items marked "Don't Care" (although used by other names, at this point the data expressed by the corresponding value is of interest to the host computer 105 of Figure 1). Because there is none). Third, "United States of America" has 'ID3' in Table 1, but "United States of America" has 'ID1' in Table 2. This final caveat means that references to 'ID3' in the input stream may be changed to 'ID1' in the output stream (otherwise, the data may be meaningless).

To accomplish these operations, transcoder 420 may include various components. The transcoder 420 includes a circular buffer 605, a stream splitter 610, an index mapper 615, a current encoding buffer 620, a previous encoding buffer 625, a transcoding rule 630, and rule evaluation. It may include group (635).

Circular buffer 605 may receive a data stream from storage 305 of FIG. 3 located within storage device 120 of FIG. 1. Because the total data to be processed can be large (for example, several gigabytes or terabytes of data), trying to load all the data at once and process it within some piece of storage may be impractical. Therefore, the input stream must be received as a stream and buffered, allowing data to be processed in smaller units than the entire data set. 6 shows buffer 605 as a circular buffer, embodiments of the invention may use any type of buffer to store data received from the input stream.

Stream splitter 610 can take data from circular buffer 605 and split the data into chunks. The chunk may then be passed to the index mapper 615. A chunk may represent a unit of data to be processed by different components within transcoder 420 and should not be confused with the term “chunk” as the term may be used in other contexts (e.g., the term “chunk”). “Column chunks” can also be used, see Figure 9 below).

FIG. 7 shows the stream splitter 610 of FIG. 6 splitting input encoded data, which may be part (or all) of an input stream, into chunks. In Figure 7, the input data is shown as containing three encoded data: "[1, BP], 3, 3, [4, RLE], 2, [5, RLE], 0" among other data. . As discussed above, these chunks represent data encoded using bit packing and RLE encoding schemes. This encoding represents the (unencoded) sequence of values "3, 3, 2, 2, 2, 2, 0, 0, 0, 0, 0". For each individual encoding, host computer 105 of Figure 1 may or may not be interested in that data (or portions of that data). Whether the host computer 105 of FIG. 1 is interested in each encoding value may depend on the transcoding rules 630: The stream splitter 610 of FIG. 6 determines which host computer 105 of FIG. You may not know if you are interested in the data. Accordingly, stream splitter 610 of FIG. 6 may split the input data stream into chunks where each chunk includes a different piece of encoded data. Accordingly, chunk 705-1 has the encoding “[1, BP], 3, 3”, chunk 705-2 has the encoding “[4, RLE], 2”, and chunk 705-3 has the encoding “[1, BP], 3, 3”. May include the encoding "[5, RLE], 0".

There are at least two additional points to note about Figure 7. First, note also the example input stream shown in Figure 7, where some data is encoded using bit packing and some data is encoded using RLE. If all data is encoded using a single encoding scheme (e.g., RLE), stream splitter 610 of FIG. 6 can in fact determine the encoding type input to transcoder 420 of FIG. 6. However, sometimes hybrid encoding methods are used. In a hybrid encoding scheme, some data may be encoded using one encoding scheme (e.g. RLE), and some data may be encoded using another encoding scheme (e.g. bit packing) (the concept is The encoding method used in can also be generalized to more than one). In the hybrid encoding method, since the information itself does not tell the stream splitter 610 which data is encoded with which encoding method, the transcoder 420 of FIG. 6 may not receive the encoding type as input. Instead, stream splitter 610 in Figure 6 can look at the chunks themselves and determine which encoding scheme is used for each chunk.

One way to determine the encoding scheme used to encode a particular chunk may be to examine the values of specific bits within the chunk. For example, a columnar storage format may use the least significant bit of the first byte to indicate whether a particular chunk of data can be encoded using RLE or bit packing: if the value of that bit is "0", then RLE can be used, and if the value of that bit is "1", bit packing can be used. This bit is then removed from the byte, and the remaining bits are logically shifted right by 1 bit to produce the value used for encoding.

For example, let's consider chunk 705-1. Chunk 705-1 contains the bit sequence “00000011 00110011”. When the stream splitter 610 of FIG. 6 reads the first byte “00000011”, the stream splitter 610 of FIG. 6 may check the least significant bit (the last “1”). Since the least significant bit may be a “1,” stream splitter 610 in Figure 6 may determine that this chunk can be encoded using bit packing. This least significant bit is removed, and the remaining bits of the first byte are logically shifted to the right by one bit, creating the byte “00000001”. The first (most significant) bit of this byte may be "0", so stream splitter 610 of Figure 6 will cause the byte to be simply "00000001" (with a "0" bit indicating that the value may not continue in the next byte). may be removed, and another leading "0" may be added) and may be determined to indicate that the group (to be determined) may be repeated once. Afterwards, the stream splitter 610 in FIG. 6 can read the next byte “00110011”. Because the most significant bit of this byte may be “0,” stream splitter 610 of Figure 6 knows that this value will not continue in the next byte. Consecutive bits are removed and a leading zero is added to produce the value "00110011", which represents the values "3" and "3". Accordingly, stream splitter 610 of FIG. 6 may determine that the encoding uses bit packing to indicate that the value “3” may be repeated twice.

On the other hand, we decide to consider chunk (705-2). Chunk 705-2 will contain the bit sequence “00001000 00000010”. When the stream splitter 610 of FIG. 6 reads the first byte “00001000”, it can check the lowest bit (last “0”). Since the least significant bit may be “0,” stream splitter 610 in Figure 6 may determine that this chunk can be encoded using RLE. This least significant bit is removed and the remaining bits of the first byte are logically shifted to the right by one bit, creating the byte "00000100". The first (most significant) bit of this byte may be a "0", so stream splitter 610 of Figure 6 causes the byte to be simply "00000100" (with a "0" bit indicating that the value may not continue in the next byte). may be removed, and another reading "0" may be added) and that its value (to be determined) may be repeated four times. Afterwards, the stream splitter 610 of FIG. 6 can read the next byte “00000010”. Because the most significant bit of this byte may be “0,” stream splitter 610 of Figure 6 knows that this value will not continue into the next byte. The value "00000010" can be produced by removing consecutive bits and adding leading zeros. Accordingly, stream splitter 610 of Figure 6 may determine that the encoding uses RLE to indicate that the value "2" may be repeated four times.

Of course, stream splitter 610 in Figure 6 may not perform all of this analysis on the bit sequence. All stream splitters 610 in Figure 6 read bytes until a byte is encountered with "0" as the most significant bit (this sequence of bytes indicates the encoding scheme and number of repetitions of the upcoming value), then reads bytes with "0" as the most significant bit. Bytes are read until another byte is encountered (this sequence of bytes represents the value being encoded). Stream splitter 610 in Figure 6 may pass the read bits (representing the entire encoded chunk) to index mapper 615 in Figure 6 (and for further processing by rule evaluator 635 in Figure 6). ): The index mapper 615 of FIG. 6 and/or the rule evaluator 635 of FIG. 6 performs the analysis described to determine which encoding scheme is used for a chunk and which value(s) are so encoded. It can be done. However, when the stream splitter 610 of FIG. 6 (or index mapper 615 of FIG. 6 or any other component of the inventive concept) performs analysis to determine the encoding scheme used to encode a particular data chunk, The stream splitter 610 of FIG. 6 (or the index mapper 615 of FIG. 6 or other components) may pass the encoding type to other components according to policy to avoid repetition of this analysis. This behavior can be particularly important if the bits that identify the encoding scheme are removed from the chunk as it is processed: without the encoding type, components that later process the encoded data may not process the encoded data correctly.

Second, it should be noted that chunks 705-2 and 705-3 both represent consecutive chunks encoded using RLE. Stream splitter 610 in Figure 6 can be expected to consider all consecutive RLE encodings as a single chunk (separating the chunks using different encoding schemes). However, keep in mind that the goal is to transcode the input stream, consolidating all data of no interest into a single "don't care" value. Recall that stream splitter 610 in Figure 6 may not have information about the data of which host computer 105 in Figure 1 is interested. If stream splitter 610 in Figure 6 considers all encodings using the same encoding scheme to be the same chunk, stream splitter 610 in Figure 6 will eventually determine the data of interest to host computer 105 in Figure 1. It is mixed with data that does not exist. Furthermore, if all data within the input stream is encoded using the same encoding scheme, the entire input stream will be considered a single chunk, which precludes the functionality of the stream splitter 610 as part of the transcoder 420 in Figure 6. It is done.

Third, while the above discussion focuses on a hybrid encoding scheme that uses 1 bit to distinguish two different encoding schemes, embodiments of the invention generalize to hybrid encoding schemes that use two or more distinct encoding schemes. It can be. Of course, if more than one encoding scheme is used, one or more bits may be used to distinguish the different encoding schemes. For example, if 3 or 4 encoding schemes are used, 2 bits may be used to differentiate the encoding schemes, and if 5, 6, 7 or 8 different encoding schemes are used, 3 bits may be used to differentiate the encoding schemes. It can be used to distinguish between different encoding schemes, etc.

(It should be noted that the bits used to distinguish encoding schemes may also be used for other purposes. For example, consider a situation where three encoding schemes are used. The least significant bit of the first byte is a specific value. (such as "0"), one encoding scheme, such as RLE, can be used, and then the least significant bit is used to represent the value, but if the least significant bit of the first byte is another specific value ("1"). ), the next least significant bit can be used to distinguish between the remaining two encoding methods (such as bit packing and cluster encoding).

Referring again to FIG. 6, the index mapper 615 may receive a chunk from the stream splitter 610. Index mapper 615 may map encoded values from an input dictionary to encoded values of an output dictionary. For example, consider again the dictionaries shown in Table 1 and Table 2 above. Since the value corresponding to "United States" may be of interest, the value "3" may be replaced with the value "1" when found in the encoded chunk; When found in an encoded chunk, all other values may be replaced with the value "0".

Figure 8 shows the index mapper 615 of Figure 6 mapping an input dictionary to an output dictionary. 8, the index mapper 615 is shown receiving an input dictionary 805 and generating an output dictionary 810. Given information about what data host computer 105 of FIG. 1 is interested in, index mapper 615 can generate output dictionary 810. Index mapper 615 may also create a map from input dictionary 805 to output dictionary 810. Continuing with the example described above, this map may specify the mapping shown in Table 3. As shown, "3" can be mapped to index "1"; All other indices can be mapped to index "0".

input index output index 0 0 One 0 2 0 3 One

There are several points worth noting about the index mapper 615. First, although the index mapper 615 is shown as a separate component from the transcoder 420 in FIG. 6, the index mapper 615 can operate in conjunction with the rule evaluator 635 in FIG. 6 (or a portion thereof). can be implemented as). Second, how index mapper 615 can generate output dictionary 810 (and the map shown in Table 3) may depend on the data of interest to host computer 105 of Figure 1. A method by which the index mapper 615 learns data of interest from the host computer 105 of FIG. 1 will be described with reference to FIG. 11 described later. Third, transcoding data may involve both the index mapper 615 that maps the input dictionary 805 to the output dictionary 810 and the transcoding rules 630 of FIG. 6: Transcoding rules 630 of FIG. 6 ) depends on the map from the input dictionary 805 to the output dictionary 810. The converse is not true: the map from the input dictionary 805 to the output dictionary 810 (and the resulting operation of the index mapper 615) can be generated without reference to the transcoding rules 630 of FIG. 6. there is.

The fourth point about the index mapper 615 is more subtle. Index mapper 615 effectively adds a new entry to the output dictionary 810, a "don't care" value. To simplify implementation, it is understood that the index mapper 615 always uses the same index for “don't care” values. The size of the input dictionary 805 may vary depending on the data set, so index "0" can always be used.

However, what happens if it turns out that all of the data in the data set is of interest to host computer 105 of Figure 1? In this case, index mapper 615 added entries to output dictionary 810, but no entries were removed from output dictionary 810. The combination of these two facts means that the output dictionary 810 can be larger (by one entry) than the input dictionary 805. Let us consider a situation where the input dictionary 805 has exactly 2n entries for some value of n. This fact means that every index into the input dictionary 805 can be expressed using n bits. Adding a "don't care" entry to the output dictionary 810 means that there are '2n+1' entries in the output dictionary 810, which means that now 'n+1 bits are in the data set. This means that it is used to represent all possible values: This problem is called 'bit overflow'. These additional bits may affect the encoded data, requiring the addition of new bits to represent the data correctly. Accordingly, a single small change to the output dictionary 810 can have a huge ripple effect on the data representation, significantly increasing the amount of storage space used to represent the encoded data.

The above example focuses on the situation where the introduction of a "don't care" entry adds a new bit representing all possible indices to the output dictionary 810, but the size of the output dictionary 810 increases with A similar problem can occur if new bits are incremented to where they can be used to represent all possible indices. Consider again the input dictionary shown in Table 1, and consider a situation where host computer 105 of Figure 1 is interested in Chinese and Indian citizens (indexes "0" and "1" in Table 1). These indices can be expressed using a single bit (since 1 bit can be used to represent the values "0" and "1"). By encoding these values using bit packing, eight of these values can be packed into a single byte. However, if the index "0" is assigned to the "don't care" value in the output dictionary 810, the indices for China and India will have different values (e.g. "1" and "2"). is mapped. The value "2" uses 2 bits, so no more 8 values can be packed into a single byte: a bit overflow has occurred.

There are several available solutions to the bit overflow problem. One is to check whether any index into the input dictionary 805 represents data that is not of interest to the host computer 105 of Figure 1. If all the data in the input dictionary 805 is found to be of interest to the host computer 105, then there is absolutely no point in transcoding the input stream, and the input stream can be mapped directly to the output stream without modification.

However, bit overflow issues can still occur in bit packing, so while this solution is useful, it may not be sufficient. To avoid bit overflow in bit packing, a solution would be to ensure that the number of bits used to represent any index in the output dictionary 810 is not greater than the number of bits used to represent any index in the input dictionary 805. You can. Two possible solutions will be described here. One solution is to assign the highest possible index in the output dictionary 810 to the "don't care" value: this first transfers all indices of interest from the input dictionary 805 to the output dictionary 810. After mapping, use the lowest unused index as the "don't care" value. Another solution is to identify an index in the input dictionary 805 that is not of interest to the host computer 105 of Figure 1 and use that index as a "don't care" value. In both solutions, the index into the input dictionary 805 is not replaced by a larger index in the output dictionary 810, which avoids bit overflow problems. A disadvantage of this solution is that it may not be possible to select an index for "don't care" that is independent of the input dictionary 805.

Referring again to Figure 6, the current chunk (which can be processed by index mapper 615) may be stored in current encoding buffer 620. From there, rule evaluator 635 can evaluate the encoding data in the current encoding buffer 620 along with the encoding data in the previous encoding buffer 625 and determine whether the encoding should be changed and what data should be output to the output stream. do. In short, the rule evaluator 635 can determine whether the encoding data in the current encoding buffer 620 can be combined with the encoding data in the previous encoding buffer 625. If so, the encoding data in the current encoding buffer 620 may be added to the encoding data in the previous encoding buffer 625; If not, the encoded data in the previous encoding buffer 625 can be output to the output stream, and the encoded data in the current encoding buffer 620 can be moved to the previous encoding buffer 625. (This analysis takes into account situations where there is data in the previous encoding buffer 625. If the previous encoding buffer 625 does not contain data, as may happen, for example, with the first data chunk, then the current encoding There are no issues with attempting to combine encoded data in buffer 620 with transcoded data in previous encoding buffer 625.)

This leads to the question, “When can encoded data be combined?” The short answer is that chunks of encoded data can be combined when the chunks both represent data that host computer 105 of FIG. 1 is interested in or data that host computer 105 is not interested in. Some examples may help explain how rule evaluator 635 operates. In both examples, the input stream contains the same data ("[1, BP], 3, 3, [4, RLE], 2, [1, BP], 0, 1, [5, RLE], 1, [1, BP], 3"), and the input dictionary is as shown in Table 1. In both examples, a row represents a "snapshot" of what is in the current encoding buffer 620 and the previous encoding buffer 625, and what was output to the output stream at that time.

line current encoding buffer old encoding buffer output stream One [1, BP], 3, 3 2 [4, RLE], 2 [1, BP], 1, 1 3 [1, BP], 0, 1 [4, RLE], 0 [1, BP], 1, 1 4 [5, RLE], 1 [6, RLE], 0 [1, BP], 1, 1 5 [1, BP], 3 [11, RLE], 0 [1, BP], 1, 1 6 [1, BP], 1, 1, [11, RLE], 0, [1, BP], 1

As shown in the first row of Table 4, the first chunk processed by rule evaluator 635 is “[1, BP], 3, 3”. Since this chunk may contain data of interest (value "3"), rule evaluator 635 uses the output dictionary 805 of FIG. 8 to replace value "3" with value "1". A map to (810) can be used. This transcoded chunk may be moved to the previous encoding buffer 625 (as shown in the second row of Table 4).

In row 2 of Table 4, the second chunk to be processed by rule evaluator 635 is “[4, RLE], 2”. This chunk may not contain the data of interest (the value "2"), so to replace the value "2" with the value "0" (this data represents "don't care" data) Rule evaluator 635 may use the map from input dictionary 805 to output dictionary 810 in FIG. 8 . This chunk may contain "don't care" data, but since the previous encoding buffer 625 may contain data of interest, the data in the previous encoding buffer 625 (Table 4 may be output to an output stream (as shown in the third row), and the currently transcoded chunk may be moved to the previous encoding buffer 625 (as shown in the third row of Table 4).

In row 3 of Table 4, the third chunk processed by rule evaluator 635 is “[1, BP], 0, 1”. Since this chunk may not contain the data of interest (the values "0" and "1"), a rule is created to replace the value "0" and value "1" with the value "0" (indicating that it is undefined data). Evaluator 635 may use the map from input dictionary 805 of FIG. 8 to output dictionary 810.

Since this chunk may contain "don't care" data, and the previous encoding buffer 625 may already contain "don't care" data, these two chunks can be combined This chunk uses bit packing, but the chunk in the previous encoding buffer 625 uses RLE, so either encoding method can be replaced by the other. In this example, bit packing-encoded data may be transcoded using RLE. (If two or more values are stored as a single value using bit packing, the entire group can be duplicated, meaning that the number of duplicated values can be a multiple of the number of packed values. On the other hand, RLE stores a single value ) As a result, the previous encoding buffer 625 can now store "[6, RLE] 0" (as shown in the fourth row of Table 4), which is It combines the "don't care" value and the two "don't care" values from the third chunk.

In row 4 of Table 4, the fourth chunk processed by rule evaluator 635 is “[5, RLE], 1”. Since this chunk may not contain the data of interest (the value "1"), the rule evaluator 635 uses the A map from input dictionary 805 to output dictionary 810 can be used. do

Since this chunk may contain "don't care" data and the previous encoding buffer 625 may already contain "don't care" data, these two chunks are combined It can be. Since both chunks use RLE as the encoding method to encode the same "don't care" value, the rule evaluator 635 increments the duplicate value of the chunk in the previous encoding buffer 625 to create two chunks. Chunks can be combined. As a result, the previous encoding buffer 625 can now store "[11, RLE] 0" (as shown in row 5 of Table 4), which is the four "don'ts" from the second chunk. It combines the "'t care" value, the two "don't care" values from the third chunk, and the five "don't care" values from the fourth chunk.

In row 5 of Table 4, the second chunk processed by rule evaluator 635 is “[1, BP], 3”. Since this chunk may contain data of interest (value "3"), rule evaluator 635 uses the output dictionary 805 of FIG. 8 to replace value "3" with value "1". A map to (810) can be used. Since this transcoded chunk may contain data of interest, while the previous encoding buffer 625 may contain "don't care" data, this transcoded chunk may contain data of interest. It may not be combined with the chunk in (625).

At this point, typically the transcoded data in the previous encoding buffer 625 will be output to the output stream and the currently transcoded chunks will be moved to the previous encoding buffer 625. However, since the current transcoded chunk is the last chunk in the input stream, both transcoded chunks can be output (of course, the chunk in the previous encoding buffer 625 is output first). Row 6 of Table 4 shows the final output.

In the second example, host computer 105 of Figure 1 requested data regarding Korean citizens. As can be seen in Table 1, the index of “Korea” is “2”. Therefore, the output dictionary can be expressed as Table 5.

ID nation 0 Don't Care One korea

line current encoding buffer old encoding buffer output stream One [1, BP], 3, 3 2 [4, RLE], 2 [1, BP], 0, 0 3 [1, BP], 0, 1 [4, RLE], 1 [1, BP], 0, 0 4 [5, RLE], 1 [1, BP], 0, 0 [1, BP], 0, 0, [4, RLE], 1 5 [1, BP], 3 [7, RLE], 0 [1, BP], 0, 0, [4, RLE], 1 6 [1, BP], 0, 0, [4, RLE], 1, [8, RLE], 0

As shown in the first row of Table 6, the first chunk processed by rule evaluator 635 is “[1, BP], 3, 3”. Since this chunk may contain data of no interest (the value "3"), a rule is evaluated to replace the value "3" with the value "0" (indicating it is "don't care" data). Group 635 may use the map from input dictionary 805 of FIG. 8 to output dictionary 810. This transcoded chunk may be moved to the previous encoding buffer 625 (as shown in the second row of Table 6).

In row 3 of Table 6, the third chunk processed by rule evaluator 635 is “[1, BP], 0, 1”. Since this chunk may not contain the data of interest (the values "0" and "1"), a rule is created to replace the value "0" and value "1" with the value "0" (indicating that it is undefined data). Evaluator 635 may use the map from input dictionary 805 of FIG. 8 to output dictionary 810. This chunk may contain data of no interest, but the previous encoding buffer 625 may contain data of interest, so the data in the previous encoding buffer 625 will be ) is output to the output stream, the currently transcoded chunk may be moved to the previous encoding buffer 625 (as shown in the fourth row of Table 6).

In row 3 of Table 6, the third chunk processed by rule evaluator 635 is “[1, BP], 0, 1”. Since this chunk may not contain the data of interest (the values "0" and "1"), a rule is created to replace the value "0" and value "1" with the value "0" (indicating that it is undefined data). Evaluator 635 may use the map from input dictionary 805 of FIG. 8 to output dictionary 810. This chunk may contain data of no interest, but the previous encoding buffer 625 may contain data of interest, so the data in the previous encoding buffer 625 will be ) may be output to the output stream and the currently transcoded chunk (as shown in the fourth row of Table 6) may be moved to the previous encoding buffer 625.

In row 4 of Table 6, the fourth chunk processed by rule evaluator 635 is “[5, RLE], 1”. Since this chunk does not contain the data of interest (the value "1"), the rule evaluator 635 uses the input dictionary of FIG. 8 to replace the value "1" with the value "0" (indicating undefined data). The map from 805 to output dictionary 810 can be used.

Since this chunk may contain "don't care" data, and the previous encoding buffer 625 may contain "don't care" data, these two chunks are combined. It can be. This chunk uses RLE, but the chunk in the previous encoding buffer 625 uses bit packing, so either encoding scheme can be replaced by the other. In this example, encoded data in bit packing may be transcoded using RLE. (Again, RLE can be chosen because single values rather than groups of values can be duplicated.) As a result, the old encoding buffer 625 now has "[7, RLE] 0" (shown in row 5 of Table 4). ), which combines the two "don't care" values from the third chunk and the five "don't care" values from the fourth chunk.

In row 5 of Table 6, the second chunk processed by rule evaluator 635 is “[1, BP], 3”. Since this chunk may not contain the data of interest (the value “3”), the rule evaluator 635 uses the A map from input dictionary 805 to output dictionary 810 can be used.

Since this chunk may contain "don't care" data, and the previous encoding buffer 625 may contain "don't care" data, these two chunks are combined. It can be. This chunk uses bit packing, but the chunk in the previous encoding buffer 625 uses RLE, so either encoding method can be replaced by the other. In this example, encoded data in bit packing can be transcoded using RLE. As a result, the previous encoding buffer 625 can now store "[8, RLE] 0", which is two "don't care" values in the third chunk and five "don't care" values in the fourth chunk. It combines the "don't care" value with one "don't care" value from the fifth chunk.

Finally, since the fifth chunk is the last chunk of the input stream, the rule evaluator 635 can output the transcoded data to the previous encoding buffer 625. Row 6 of Table 6 shows the final output.

None of the examples above illustrate a situation where consecutive chunks contain the data of interest. Embodiments of the present invention may handle this situation in different ways. In one embodiment of the invention, any chunk of the previous encoding buffer 625 may be output to the output stream when the current encoding buffer 620 contains data of interest (i.e., the current encoding buffer 620 If it contains data of interest, no attempt is made to combine data in the current encoding buffer 620 with data in the previous encoding buffer 625). In another embodiment of the invention, chunks of the current encoding buffer 620 and chunks of the previous encoding buffer 625 may be combined. However, in this embodiment of the invention, whether such a combination is feasible may depend on whether the values of interest are the same. For example, if one chunk contains data about Chinese citizens and another chunk contains data about Korean citizens, these chunks may or may not be combined according to an embodiment of the present invention. On the other hand, if both chunks contain data about Korean citizens, it may be possible to combine the two chunks.

The rule evaluator 635 uses the transcoding rules 630 to determine data of interest and data of no interest, data stored in the previous encoding buffer 625 vs. data of interest. You can determine what data can be output, and whether chunks can be transcoded from one encoding scheme to another.

As mentioned above, rule evaluator 635 may also include index mapper 615. In an embodiment of the invention in which the rule evaluator 635 includes an index mapper 615, the rule evaluator 635 applies the index mapper to the contents of the current encoding buffer 620 before the transcoding rules 630 are applied. (615) can be applied.

Table 7 shows some rules that can be used when the encoding method used is RLE or bit packing. In embodiments of the invention where different encoding schemes may be used, the rules may be varied accordingly: all such variations are considered embodiments of the invention. Additionally, embodiments of the present invention may include rules for managing transcoding data between two or more different types of encoding methods. For example, a hybrid encoding scheme may use three different encoding schemes: The transcoding rule 630 in Figure 6 allows the current encoding buffer 620 and the previous encoding buffer 625 in Figure 6 to use any other encoding. When containing encoded data, you can use a pair of methods to specify how to transcode the data.

In Table 7, P represents data that may be of interest to the host computer 105 of FIG. 1, and DC represents data that may not be of interest to the host computer 105. (How data is identified as being of interest or not will be further explained with reference to Figure 11 below.) If variables (e.g., x, y, or z) are used, these variables They may represent a count of the number of values that are or are not of interest to the host computer 105 of FIG. 1 . For example, the expression "[g, BP] P (x), DC (y), P (z)" (used in Rule 7 and Rule 9) indicates that the data was encoded using bit packing: Start group The group contains the x values of interest in the group, the y values of interest in the middle of the group, and the z values of interest at the end of the group. x, y, z, g and G are 'g*G = x + y + z, 1 ≤ g ≤ 63, x mod G = 0, y mod G = 0, z mod G, y ≠ 0 and y ≥ 16 Constraints such as 'number of bits per division packed value' must be met. Finally, the PEB (in the output column) may indicate that when a rule is selected for an application, anything previously stored in the encoding buffer 625 may be output to the output stream. Table 7 also considers the situation where any data has already been mapped by the index mapper 615, and therefore contains values corresponding to the output dictionary 810 of FIG. 8.

rule current encoding buffer old encoding buffer new old encoding buffer new current encoding buffer Print One [ r ,RLE]P Anything [ r , RLE], P Next chunk P.E.B. 2 [ r ,RLE]DC [ r ₁ , R L E ] DC [ r ₁ + r , R L E ] DC Next chunk 3 [ r ,RLE]DC Anything else [ r ,RLE]DC Next chunk P.E.B. 4 [ g , BP], P Anything [ g , BP], P Next chunk P.E.B. 5 [ g , BP]DC( y ) [ r ₁ , R L E ] DC [ r ₁ + y , R L E ] DC Next chunk 6 [ g , BP] DC( y ), P( z ) [ r ₁ , R L E ] DC [ r ₁ + y , R L E ] DC [ z / 8, BP] PO( z ) 7 [ g , BP] P( x ), DC( y ), P( z ) [ r ₁ , R L E ] DC [ x / 8, BP] P( x ) [( y + z )/8, BP] DC( y ), P( z ) P.E.B. 8 [ g , BP] DC( y ), P( z ) Anything else [ y ,RLE]DC [ z / 8, BP] P( z ) P.E.B. 9 [ g , BP] P( x ), DC( y ), P( z ) Anything else [ x / 8, BP] P( x ) [( y + z )/8, BP] DC( y ), P( z ) P.E.B.

The above discussion generally describes how transcoding may be performed on data. However, when data is stored in column format, the column format can be utilized to facilitate transcoding. Before explaining this use, it is useful to understand the column format. For purposes of explanation, the column format is described with reference to SSDs, but embodiments of the present invention may include other storage devices that may utilize the column format.

Figure 9 shows an example file saved in column format. Figure 9 shows the file. The file may include file metadata 905 and column chunks 910-1, 910-2, and 910-3. Figure 9 shows three column chunks 910-1 to 910-3, but embodiments of the present invention may include any number of chunks (0 or more) without limitation.

File metadata 905 may include metadata related to files. Figure 9 shows file metadata 905 including a file-block map 915 and a dictionary page 920, although other metadata may also be stored. The dictionary page 920 may be a dictionary used to encode values in data in a file, such as the dictionary shown in Table 1 above. Dictionary page 920 may also store multiple dictionaries that can be used to encode other data in the file, for example, one dictionary storing country names and another dictionary storing surnames.

File-block map 915 may identify the block storing the individual column chunks 910-1, 910-2, and 910-3 as well as their relative order. The file-block map 915 also specifies the order of data pages within each column chunk 910-1, 910-2, and 910-3, or the page order is determined by the column chunks 910-1, 910-3. 2 and 910-3). File-block map 915 may provide information regarding which blocks are used to store all files stored in storage device 120 of FIG. 1. File-block map 530 of FIG. 5 And, the file-block map 915 is similar to the file-block map 530 of FIG. 5, except that it can provide information regarding which blocks are used to store the files shown in FIG. 9. can do. (Of course, both file-block maps can be used together: the file-block map 530 of FIG. 5 is used to find blocks that store file metadata 905 for each file, and the file-block map 530 of FIG. The file-block map 915 can be used to find the column chunks that store the column chunk for the file.)

In general, a single column chunk can span multiple blocks, and a single block can store multiple column chunks. There is little difficulty with more general solutions to data storage, as long as there is some way to identify where the data is stored and what that data represents (e.g. the file that contains that data). However, for the purposes of this discussion, we will consider the situation where a column chunk fits into a single block and blocks do not share column chunks. Accordingly, each of the column chunks 910-1, 910-2, and 910-3 may be stored in a separate block.

Within column chunk 910-1 (column chunks 910-2 and 910-3 are similar) there is a dictionary page 925 and data pages 930-1, 930-2, and 930-3. It can exist. Although Figure 9 shows three data pages, embodiments of the invention may include any number (0 or more) of data pages within a column chunk. Data pages can store the actual data of a file by dividing it into units that fit each page.

The dictionary page 925 may store a dictionary used for data in the column chunk 910-1. As with dictionary page 920, dictionary page 925 can store a number of dictionaries that can be used to encode different data within the file.

At this point, a question may be raised why Figure 9 shows both dictionary page 920 and dictionary page 925. This is because dictionary pages 920 and 925 may be used as implementations of different column formats. For example, a one-column storage format may use a single dictionary for the entire file, which may be stored in dictionary pages 920. However, other column formats may use individual dictionary pages 925 in each column chunk 910-1, 910-2, and 910-3. The advantage of using dictionary pages 925 is that if a particular column chunk does not use a dictionary, or if a particular value is not used in the data within a particular column chunk, then this information is omitted from the dictionary page 925 and the The idea is that it can be reduced in size (or eliminated entirely). However, on the other hand, multiple dictionary pages 925 in different column chunks may result in data duplication: that is, the same dictionary entry may be used in multiple column chunks. This is why dictionary pages 920 and 925 are displayed as dotted lines, and either one may be omitted depending on the column storage format in use. (In fact, it may be the case that the file does not use a dictionary at all, in which case dictionary pages 920 and 925 may be omitted altogether.)

Now that the columnar format has been described, an explanation is available for adaptations for use of transcoder 420 of FIG. 4 in storage devices that use the columnar format. FIG. 10 shows the storage device 120 of FIG. 1 configured to implement transcoding where data is stored in a columnar format. As shown in FIG. 10, the storage device 120 may include a host interface layer 505, a storage device controller 510, and storage 515, the functions of which are described with reference to FIG. 5 described above. (Again, storage device 120 may be an SSD, hard disk drive, or any other storage device that can use a columnar format).

Storage device 120 may also include an in-storage computer controller 1005, a column chunk processor 1010, and an in-storage computer 315. The in-storage computer controller 1005 can manage information transmitted to the in-storage computer 315 and the column chunk processor 1010. For example, host computer 105 of FIG. 1 may request that storage device 120 perform some acceleration function, such as counting the number of citizens of a particular country, and in-storage computer controller 1005 may Identification of country of interest) may be provided to the column chunk processor 1010. In-storage computer controller 1005 may also access data, particularly column chunks, from storage 515 and provide that data to column chunk processor 1010. The in-storage computer controller 1005 may also determine the encoding scheme used for the data (assuming a single encoding scheme is used to use column chunks or entire files rather than a hybrid encoding scheme), and the column chunk processor 1010 Provides an encoding type. Finally, the in-storage computer controller 1005 again receives the transcoded data from the column chunk processor 1010 and, as appropriate, stores the transcoded data in response to a request from the host computer 105 of FIG. 1 (FIG. It may return to the host computer 105 of FIG. 1 or the in-storage computer 315 (via the host interface layer 505 of 1). The structure and operation of the column chunk processor 1010 are explained with reference to FIG. 11 below.

The in-storage computer controller 1005 and column chunk processor 1010 may be implemented using a suitably programmed general-purpose processor, FPGA, ASIC, GPU, or GPGPU, among other possibilities. In-storage computer controller 1005 and column chunk processor 1010 may be implemented using the same hardware or different hardware (e.g., in-storage computer controller 1005 may be implemented as an ASIC, while column chunk processor 1010 may be implemented using Processor 1010 may be implemented with an FPGA), they may be implemented as a single unit or as separate components.

Figure 11 shows the column chunk processor 1010 of Figure 10 configured to transcode data stored in column format. Referring to FIG. 11, the column chunk processor 1010 can receive an input stream, encoding type, and properties as input, and generate an output stream as output. The input stream may be stored in input buffer 1105. The input stream can be a single page of data from a column chunk, or it can be all of the data within the column chunk. Data from input buffer 1105 may be provided as an input stream to transcoder 420 (as described above with reference to FIG. 6): The encoding type may be received from the in-storage computer controller 1005. It should be noted that the transcoder 420 may include the circular buffer 605 of Figure 6, so the input buffer 1105 may be omitted: the data is sent to the stream splitter 610 of Figure 6 for operation. It may be stored in the circular buffer 605 of FIG. 6. However, in some embodiments of the invention, circular buffer 605 in Figure 6 may not be large enough to store an entire data page or column chunk (or data may be removed from circular buffer 605 in Figure 6). (the input stream can provide data faster than the input stream). In this case, input buffer 1105 may act as temporary storage for data that may not immediately fit into circular buffer 605 of Figure 6.

The output of transcoder 420 (the output stream described with reference to FIG. 6) may be stored in output buffer 1110. Again, data may be transmitted directly to its destination as it is produced by transcoder 420, but it may be useful to transmit data in specific units, such as complete data pages or column chunks. In this situation, output buffer 1110 may store the output stream until an appropriate data unit is generated. At that point, column chunk processor 1010 may transmit the output stream to in-storage computer controller 1005 of FIG. 10 or host computer 105 of FIG. 1 as appropriate for the requested transcoding operation.

The index mapper 615 (shown external to the transcoder 420 in FIG. 11, although the index mapper 615 may be part of the transcoder 420 as shown in FIG. 6) includes an attribute evaluator 1115. and “Don Care (No Justice)” may receive information from the evaluator 1120. Attribute evaluator 1115 receives attributes from in-storage computer controller 1005 of Figure 10 and uses the attributes to determine data of interest. The comparison operator uses attributes to identify which values in the input dictionary 805 of FIG. 8 (which may be any of the dictionary pages 920, 925 of FIG. 9) are of interest to the host computer 105 of FIG. 1. Can be used by evaluator 1115. Doncare evaluator 1120 may operate similarly (but in mirror form) to identify data of no interest. It should be noted that the attribute evaluator 1115 and the money evaluator 1120 operate complementary, so it is possible to use either evaluator (data that does not meet the criteria for one evaluator) As a result, it fits the criteria of other evaluators): Therefore, either the attribute evaluator 1115 or the money evaluator 1120 can be omitted. This information may be provided by the attribute evaluator 1115 and the doncare evaluator 1120 to the index mapper 615, which may select the output dictionary of FIG. 8 from the input dictionary 805 of FIG. 8. It is possible to establish a mapping to (810).

As an example, consider again the query from host computer 105 of Figure 6 to count the number of items in a data set containing United States citizens. When this query arrives, attributes can be extracted (e.g.: "Citizenship = United States": the exact format of the attribute may vary depending on the dataset format and the application used to submit the query). An inspection of the input dictionary 805 of Figure 8 (such as shown in Table 1) can be used to replace "United States" with the value "3". Accordingly, the attribute provided to index mapper 615 may specify "citizenship = 3", after which index mapper 615 will output the output dictionary 810 of Figure 8 (as shown in Table 2), and the You can create a map.

It should be noted that the results of attribute evaluator 1115 may also be provided to transcoder 420 for use in constructing transcoding rules 630 of FIG. 6. Because the transcoding rules 630 of FIG. 6 depend on the host computer 105 of FIG. 1 knowing the data of interest, the transcoding rules 630 of FIG. 6 are adapted to use the results of the attribute evaluator 1115. It can be adapted. For example, consider again the rule shown in Table 7. The results of attribute evaluator 1115 (or the map from input dictionary 805 in FIG. 8 to output dictionary 810 (described in Table 3)) may be used to set appropriate values for P and DC in various rules. You can.

It should also be noted that in Figure 11, the attribute (predicate) is applied to all data input to the transcoder 420 as an input stream. Although it is reasonable to conclude that the properties will apply to the entire data set for which host computer 105 of Figure 1 submits a query, transcoder 420 views the input stream as complete, even if it represents only a portion of the data set. . For example, column chunk processor 1010 may process each of the data pages 930-1, 930-2, and 930-3 in FIG. 9 as its own “input stream” using transcoder 420. You can. Since the transcoder 420 has no knowledge of what the input stream represents, this process works without problems.

FIGS. 12A to 12C show flowcharts of exemplary procedures for transcoding data by the transcoder 420 of FIGS. 4 and 6 according to an embodiment of the present invention. At block 1205 of Figure 12A, transcoder 420 of Figure 6 may check whether there is still data to be received from the input stream. Generally, this input stream can come from any source, but as discussed previously in Figures 9-11, this input stream can be pages of data from column chunks when the data is stored in a columnar format. When there is no more data remaining to be received from the input stream, at block 1210 the transcoder 420 of Figure 6 encodes any remaining transcoders in either the previous encoding buffer 625 of Figure 6 or the current encoding buffer 620 of Figure 6. You can check whether there is coded data. If transcoded data remains in either the previous encoding buffer 625 in FIG. 6 or the current encoding buffer 620 in FIG. 6, the transcoded data in the previous encoding buffer 625 in FIG. 6 is sent to the output stream. is output, and then the transcoded data in the current encoding buffer 625 of FIG. 6 is output. In most situations, the current encoding buffer 620 of FIG. 6 should be empty, since the rule evaluator 635 may operate on data in the current encoding buffer 620 of FIG. 6. Even in situations where data may remain in the current encoding buffer 620 of FIG. 6 as a result of applying the transcoding rule 630 of FIG. 6 (e.g., as shown in rules 6-9 of Table 7), The rule evaluator 635 in FIG. 6 is used before the transcoder 420 in FIG. 6 finds new data from the input stream (via the circular buffer 605 in FIG. 6 and the stream splitter 610 in FIG. 6). It will act on that data: transcoder 420 in Figure 6 may wait for the current encoding buffer 620 in Figure 6 to be cleared before attempting to process the next chunk of data in the input stream. However, if transcoded data remains in the current encoding buffer 620 of FIG. 6, the transcoded data may be output to an output stream. Once all data has been output to the output stream at block 1215, processing may be terminated (until transcoder 420 of Figure 6 is expected to process a new input stream).

Assuming there is still data to process from the input stream, at block 1220 the circular buffer 605 of Figure 6 may receive the next encoded data from the input stream, after which the stream splitter 610 of Figure 6 may split the encoded data. The first chunk can be identified and the chunk can be delivered to the index mapper 615 of FIG. 6. (In an embodiment of the invention in which the index mapper 615 of Figure 6 is provided as an effective part of the rule evaluator 635 of Figure 6, the stream splitter 610 of Figure 6 divides the encoded data chunk into the current encoding buffer of Figure 6. 620.) At block 1225, the index mapper 615 of FIG. 6, or the rule evaluator 635 of FIG. 6, may determine whether a chunk of data is of interest, more specifically: , depending on whether the data chunk contains data requested by the host computer 105 of FIG. 1 (e.g., from an attribute).

If the chunk of encoded data contains data of interest to the host computer 105 of FIG. 1, then at block 1230 (FIG. 12B) the index mapper 615 of FIG. 6, or the rule evaluator 635 of FIG. 6. may use the map from the input dictionary 805 in Figure 8 to the output dictionary 810 in Figure 8 to re-encode any data in the chunk. At block 1235, rule evaluator 635 of FIG. 6 may check whether data transcoded in previous encoding buffer 625 of FIG. 6 is of interest to host computer 105 of FIG. 1. If not (and recall that the current chunk is of interest to the host computer 105 of FIG. 1 as determined at block 1225 of FIG. 12A), then at block 1240 the transcoder 420 of FIG. 6 ) may output the transcoded data from the previous encoding buffer 625 of FIG. 6 to an output stream, and at block 1245 the transcoder 420 of FIG. 6 may output the currently transcoded chunk to the previous encoding buffer 625 of FIG. After storing in encoding buffer 625, the procedure returns to block 1205 in Figure 12A.

On the other hand, if, as determined at block 1235, the previous encoding buffer 625 of Figure 6 also stored data of interest to the host computer 105 of Figure 1, then at block 1250 the rule evaluator of Figure 6 635 may determine whether the current chunk and the transcoded chunk of the previous encoding buffer 625 of FIG. 6 use the same encoding scheme. Otherwise, at block 1255 the rule evaluator 635 of Figure 6 uses one of the chunks (either the chunk in the current encoding buffer 620 in Figure 6 or the chunk in the previous encoding buffer 625 in Figure 6). You can change the encoding method. (In situations where more than one encoding method is used, the rule evaluator 635 in Figure 6 determines the encoding method used for the two chunks of the current encoding buffer 620 in Figure 6 and the previous encoding buffer 625 in Figure 6. ) Then, if chunks in both the current encoding buffer 620 of FIG. 6 and the previous encoding buffer 625 of FIG. 6 are known to use the same encoding scheme, the rule of FIG. 6 is evaluated at block 1260. Group 635 may combine the two chunks into a single chunk, which may be stored in the previous encoding buffer 625 of FIG. 6, after which the process may return to block 1205 of FIG. 12A.

Figure 12B shows that the current chunk may be transcoded twice: once at block 1230 (when the value is updated to correspond to output dictionary 810 of Figure 8) and once at block 1255 ( When the encoding scheme of the current chunk changes (when changing from one encoding scheme to another). You can perform these two tasks separately, but you can also combine them. That is, you can change the encoding method and update the value at the same time. Embodiments of the invention include performing these operations separately and as single steps.

Recall again Figure 12B, which illustrates the operations performed when a current chunk is of interest to host computer 105 of Figure 1 (as determined at block 1225 of Figure 12A). If the chunk is not currently of interest to host computer 105 of FIG. 1 (as determined at block 1225 of FIG. 12A), the index mapper 615 of FIG. 6 (or FIG. 6) at block 1265 (FIG. 12C) The rule evaluator 635 in will use the map from the input dictionary 805 in Figure 8 to the output dictionary 810 in Figure 8 to re-encode all data in the chunk (in particular, the "undefined" values). You can. At block 1270, rule evaluator 635 of Figure 6 may check whether data transcoded in previous encoding buffer 625 of Figure 6 is of interest to host computer 105 of Figure 1. If so (recall that the current chunk is not of interest to the host computer 105 of FIG. 1, as determined at block 1225 of FIG. 12A), then at block 1275 the transcoder 420 of FIG. 6 may output the transcoded data in the previous encoding buffer 625 of FIG. 6 to an output stream, and at block 1280, the transcoder 420 of FIG. 6 may transmit the current transcode to the previous encoding buffer 625 of FIG. The coded chunk can be stored, and then the process returns to block 1205 in FIG. 12A.

On the other hand, if, as determined at block 1270, the previous encoding buffer 625 of Figure 6 also stores data of no interest to the host computer 105 of Figure 1, then at block 1285 the rule evaluator of Figure 6 635 may determine whether the current chunk and the transcoded chunk of the previous encoding buffer 625 of FIG. 6 use the same encoding scheme. If not the same encoding scheme, at block 1290 the rule evaluator 635 of FIG. 6 evaluates one of the chunks (either the chunk in the current encoding buffer 620 in FIG. 6 or the chunks in the previous encoding buffer 625 in FIG. 6 You can change the encoding method used by one of the following: (In situations where more than one encoding method is used, the rule evaluator 635 in Figure 6 determines the encoding method used for both chunks of the current encoding buffer 620 in Figure 6 and the previous encoding buffer 625 in Figure 6. can be changed.) Then, if chunks in both the current encoding buffer 620 of FIG. 6 and the previous encoding buffer 625 of FIG. 6 are known to use the same encoding scheme, the rule of FIG. 6 is used at block 1295. The evaluator 635 may combine the two chunks into a single chunk, which may be stored in the previous encoding buffer 625 of FIG. 6, after which the process may return to block 1205 of FIG. 12A.

Figure 12C shows that the current chunk may be transcoded twice: once at block 1265 (when the value is updated to correspond to the output dictionary 810 of Figure 8) and once at block 1290 ( When the encoding scheme of the current chunk changes (when changing from one encoding scheme to another). You can perform these two tasks separately, but you can also combine them. That is, you can change the encoding method and update the value at the same time. Embodiments of the invention include performing these operations separately and as single steps.

Throughout Figures 12A-12C, there is an implicit assumption that some data is present in the previous encoding buffer 625 of Figure 6. For example, blocks 1235 and 1270 describe a situation where there is some data in the previous encoding buffer 625 of Figure 6. This is generally a reasonable assumption since the transcoded data may be buffered in the previous encoding buffer 625 of Figure 6 to support the combining of data chunks to be combined (chunk combining if the data has already been output to the output stream) It may be too late to try). However, there may be situations where there is no data stored in the previous encoding buffer 625 of FIG. 6. As an example, when the first chunk of the input stream is processed, there is no data in the previous encoding buffer 625 (since nothing has been previously processed in that input stream).

As a second example, there may be an encoding scheme that does not support data chunk combining, in which case there is absolutely no value in storing the previous chunk in the previous encoding buffer 625 of Figure 6. If there is no data in the previous encoding buffer 625 of FIG. 6, a comparison of the current chunk with the (non-existent) chunk from the previous encoding buffer 625 of FIG. 6 or the (non-existent) chunk from the previous encoding buffer 625 of FIG. There is no point in outputting chunks that do not work. A simple solution is that if there is no data in the previous encoding buffer 625 of Figure 6, nothing that depends on the presence of data in the previous encoding buffer 625 is performed. Thus, for example, in Figure 12B, if there is no data in previous encoding buffer 625, processing may jump directly from block 1230 to block 1245 (and 12C, processing may jump directly from block 1265 to block 1280 (to buffer the currently transcoded chunk in the previous encoding buffer 625 of FIG. 6). ).

Close examination of Figures 12b and 12c shows that the difference between them is relatively small. Some notable differences are in blocks 1230 and 1265, and the different branches leaving blocks 1235 and 1270. In practice, even this difference is relatively minor: blocks 1230 and 1265 both relate to re-encoding based on the output dictionary 810 of Figure 8 (block 1265 specifically names the use of an “undefined” value ). And although the branches leaving blocks 1235 and 1270 are labeled differently, the reason is that blocks 1235 and 1270 are about determining whether the current chunk can be combined with the previous chunk. Accordingly, FIGS. 12B and 12C can theoretically be combined at the expense of some loss of clarity with respect to the operational sequences.

FIG. 13 shows a flow diagram of an example procedure for the stream splitter 610 of FIG. 6 for splitting input encoded data into chunks. Referring to Figure 13, at block 1305, stream splitter 610 of Figure 6 may receive input encoded data (originating from storage 305 of Figure 3 within storage device 120 of Figure 1). , which may be buffered in a buffer such as the input buffer 1105 in FIG. 11 or the circular buffer 605 in FIG. 6. At block 1310, stream splitter 610 of FIG. 6 may divide the input encoded data into chunks. At block 1315, stream splitter 610 of Figure 6 may transmit the chunk to transcoder 420 of Figure 6 (or to index mapper 615 of Figure 6 or current encoding buffer 620 of Figure 6). ).

14A to 14B illustrate example procedures for the column chunk processor 1010 of FIG. 10 and/or the transcoder 420 of FIGS. 4 and 6 for transcoding data into a column format according to an embodiment of the present invention. Shows a flow chart. Figures 14A-14B also show an expanded example of how the stream splitter 610 of Figure 6 receives input encoded data as described at block 1305 of Figure 13 in at least one embodiment.

At block 1405 of Figure 14A, the column chunk processor 1010 of Figure 10 processes the file-block map 915 of Figure 9 for a file, or alternatively or cumulatively the file-block map 530 of Figure 5. ) can be accessed. At block 1410, the column chunk processor 1010 of FIG. 10 uses the file-block map 915 of FIG. 9 to find the file metadata 905 of FIG. 9 followed by the input dictionary 920 of FIG. 9. You can use it. If each of the column chunks 910-1, 910-2, and 910-3 in Figure 9 includes its own dictionary page 925 in Figure 9, the dictionary page 925 in Figure 9 is the file meta in Figure 9. Data 905 may be omitted, in which case block 1410 may be omitted as shown by dashed line 1415. Then, at block 1420 using the file-block map 915 of FIG. 9, the column chunk processor 1010 of FIG. 10 may identify the column chunks for the file (which may be stored in the storage device of FIG. 1 ( may be a block of data stored in 120).

At block 1425 (FIG. 14B), column chunk processor 1010 of FIG. 10 may determine whether there are more column chunks (blocks) to access. Otherwise (column chunks no longer exist), the processor terminates. On the other hand (if column chunks exist), at block 1430, the column chunk processor 1010 of Figure 10 processes the dictionary of Figure 9 from the column chunks 910-1, 910-2, or 910-3 of Figure 9. Page 925 can be accessed. If the file metadata 905 of FIG. 9 stores the dictionary page 920 of FIG. 9, the column chunks 910-1, 910-2, and 910-3 of FIG. 9 store the dictionary page 920 of FIG. 9. 925 may be omitted, as shown by dashed line 1435, and block 1430 may be omitted. At block 1440, column chunk processor 1010 of FIG. 10 processes data pages 930-1, 930- of FIG. 9 from column chunks 910-1, 910-2, and 910-3 of FIG. 9. 2 and 930-3) can be accessed. At block 1445, column chunk processor 1010 of Figure 10 processes input dictionary 805 of Figure 8 and data pages 930-1, 930-2, and 930-3 of Figure 9 for the column chunk. It can be delivered (in that order) to the transcoder 420 in Figure 6, the stream splitter 610 in Figure 6, or the index mapper 615 in Figure 6.

FIG. 15 shows a flow diagram of an example procedure of the index mapper 615 of FIG. 6 for mapping the input dictionary 805 of FIG. 8 to the output dictionary 810 of FIG. 8 according to an embodiment of the present invention. Referring to Figure 15, at block 1505, the index mapper 615 of Figure 6 may receive the input dictionary 805 of Figure 8 (e.g., from the column chunk processor 1010 of Figure 10). . At block 1510, index mapper 615 of Figure 6 may determine which data in input dictionary 805 of Figure 8 is of interest. The index mapper 615 of FIG. 6 makes this determination using, for example, attributes provided from the host computer 105 of FIG. 1, perhaps via the in-storage computer controller 1005 of FIG. 10. At block 1515, index mapper 615 of FIG. 6 may generate output dictionary 810 of FIG. 8. Output dictionary 810 may contain all entries of interest to host computer 105 of FIG. 1, but may contain all entries of no interest to host computer 105 of FIG. 1 into one "don't care." "It can be integrated by value. At block 1520, index mapper 615 of FIG. 6 may map values from input dictionary 805 of FIG. 8 to dictionary 810 of FIG. 8. Finally, at block 1525, index mapper 615 of FIG. 8 may output output dictionary 810 of FIG. 8.

16A-16B illustrate the in-storage computer controller of FIG. 10 for managing attributes received from the host computer 105 of FIG. 1 and potentially performing acceleration functions for transcoded data according to an embodiment of the present invention. A flow diagram of an example procedure for (1005) is shown. At block 1605 of FIG. 16A, the in-storage computer controller 1005 of FIG. 10 may receive attributes from the host computer 105 of FIG. 1. At block 1610, the in-storage computer controller 1005 of Figure 10 may access the input dictionary 805 of Figure 8 to obtain encoded data covered by the query. At block 1615, the in-storage computer controller 1005 of Figure 10 may identify an item in the input dictionary 805 of Figure 8 that is covered by the attribute (i.e., of interest to the host computer 105 of Figure 1). 8 entry in the input dictionary 805). At block 1620, in-storage computer controller 1005 of Figure 10 may generate output dictionary 810 of Figure 8 containing the items covered by the attribute. At block 1625, the in-storage computer controller 1005 of Figure 10 maps the items of the input dictionary 805 of Figure 8 covered by the attribute to the items of the output dictionary 810 of Figure 8.

At block 1630, the in-storage computer controller 1005 of FIG. 10 may identify items in the input dictionary 805 of FIG. 8 that are not covered by the attribute (i.e., the in-storage computer controller 1005 of FIG. 1 entries in the input dictionary 805 of Figure 8 that are not of interest). At block 1635, in-storage computer controller 1005 of Figure 10 may add a “don't care” entry to output dictionary 810 of Figure 8. At block 1640 (FIG. 16B), the in-storage computer controller 1005 of FIG. 10 divides items of the input dictionary that are not covered by attributes into "don't care" items of the output dictionary 810 of FIG. 8. can be mapped to

At block 1645, the rule evaluator 635 of FIG. 6 (within the transcoder 420 of FIG. 6) adapts the transcoding rules 630 of FIG. 6 according to a query from the host computer 105 of FIG. 1. You can use properties to do this. At block 1650, index mapper 615 of FIG. 6 and rule evaluator 635 of FIG. 6 (potentially both located within transcoder 420 of FIG. 6) encode data from an input stream to an output stream. To transcode, one may use the map from the input dictionary 805 of Figure 8 to the output dictionary 810 of Figure 8 and the transcoding rules 630 of Figure 6 (as discussed above with reference to Figures 12A-12C). together)

At this point, a variety of options exist. As shown at block 1655, in-storage computer controller 1005 of FIG. 10 receives the output stream from transcoder 420 of FIG. 6 and transmits the transcoded data to host computer 105 of FIG. 1. Transferring, at block 1660, the in-storage computer controller 1005 of FIG. 10 may transmit the output dictionary 810 of FIG. 8 to the host computer 105 of FIG. 1. Alternatively, at block 1665, the in-storage computer controller 1005 of FIG. 10 may apply acceleration functions to data in the output stream, and at block 1670, the in-storage computer controller 1005 of FIG. 10 may apply acceleration functions to the data in the output stream. Can transmit the results of the acceleration function to the host computer 105 of FIG. 1.

12A-16B, some embodiments of the inventive concept are shown. However, those skilled in the art will recognize that other embodiments of the present invention are possible by changing the order of blocks, omitting blocks, or including links not shown in the drawings. All such variations of the flowcharts are considered embodiments of the invention, whether or not explicitly described.

Embodiments of the present invention provide technical advantages over the prior art. In a conventional system, the decoded data is transmitted to host computer 105 of Figure 1. Although the data transmitted to host computer 105 of FIG. 1 is optional (i.e., the data transmitted to host computer 105 of FIG. 1 includes data of interest), the data is still transmitted without compression or encoding, which This means that space savings are achieved through selectivity. In contrast, since most of the reduction in storage is achieved through encoding rather than compression, transmitting encoded data to host computer 105 of FIG. 1 is generally equivalent to transmitting decoded data to host computer 105 of FIG. 1. It involves transmitting less data than Additionally, because data can be transcoded from one encoding scheme to another, using transcoder 420 in Figure 6 may be more efficient than decoding and re-encoding the data in separate operations. .

The following discussion is intended to provide a brief general description of suitable machine(s) on which certain aspects of the invention may be implemented. The machine or machinery may be controlled at least in part by input from conventional input devices such as a keyboard, mouse, etc., as well as instructions received from other machines, interaction with a virtual reality (VR) environment, biometric feedback, or other input signals. You can. As used herein, the term “machine” is intended to broadly encompass a single machine, a virtual machine, or a system of communicatively coupled machines, virtual machines, or devices operating together. Exemplary machines include computing devices such as personal computers, workstations, servers, portable computers, handheld devices, phones, tablets, etc., as well as transportation devices such as personal or public transportation (e.g., automobiles, trains, taxis, etc.).

The machine or machines may include embedded controllers such as programmable or non-programmable logic devices or arrays, Application Specific Integrated Circuit (ASIC), embedded computers, smart cards, etc. The machine or machines may utilize one or more connections to one or more remote machines, such as through a network interface, modem, or other communication combinations. Machines may be connected to each other by means of physical and/or logical networks, such as intranets, the Internet, local area networks (LANs), wide area networks (WANs), etc. Those skilled in this field will understand that network communications utilize a variety of wired and/or wireless short-range and long-range carriers and protocols including radio frequency (RF), satellite, microwave, IEEE 802.11, Bluetooth, optical, infrared, cable, laser, etc. You will understand that

Embodiments of the technical idea of the present invention include functions, procedures, data structures, and application programs that, when accessed by a machine, cause the machine to perform tasks or define abstract data types or low-level hardware contexts. It may be described with reference to or in cooperation with associated data including the like. Associated data may include, for example, volatile and/or non-volatile memory such as RAM, ROM, etc., or other storage devices, and hard drives, floppy disks, optical storage, tapes, flash memory, memory sticks, and digital video disks. , may be stored in associated storage media, including biometric storage, etc. Associated data may be transmitted in the form of packets, serial data, parallel data, transmission signals, etc. across transmission environments including physical and/or logical networks, and may be used in a compressed or encrypted format. Associated data may be used in a distributed environment and may be stored locally and/or remotely for machine access.

Embodiments of the inventive concept are executable by one or more processors and include a tangible, non-transitory machine-readable medium containing instructions to perform elements of the inventive concept as described herein. can do.

The various operations of the above-described method may be performed by any suitable means capable of performing the operations, such as various hardware and/or software component(s), circuit(s) and/or module(s). The software may include an ordered list of executable instructions for implementing logical functions, and may contain any instruction execution system, device, such as single or multi-core, for use in connection with a single or multi-core processor or system containing the processor. It may be implemented as a “processor-readable medium.”

The blocks or steps of methods or algorithms and functions described in connection with the embodiments disclosed herein may be implemented directly as hardware, software modules executed by a processor, or a combination of the two. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a tangible, non-transitory computer-readable medium. The software module may be RAM (Random Access Memory), flash memory, ROM (Read Only Memory), EPROM (Electrically Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), register, hard disk, removable disk, CD ROM, or as known in the art. It may be stored in any other known form of storage medium.

Although the principles of the inventive concept have been described and illustrated with reference to illustrated embodiments, it will be recognized that the illustrated embodiments may be arranged and detailed without departing from the principles thereof, and may be combined in any desired manner. will be. And, although the foregoing discussion focuses on specific embodiments, other configurations are contemplated. In particular, although expressions such as “according to embodiments of the present invention” are used in the present specification, these phrases are generally intended to refer to the possibility of embodiments, and are not intended to limit the concept of the present invention to the configuration of specific embodiments. As used herein, these terms may refer to the same or different embodiments that can be combined with other embodiments.

The above-described exemplary embodiments should not be construed as limiting the inventive concept. Although several embodiments have been described, those skilled in the art will readily appreciate that many modifications may be made to these embodiments without departing substantially from the novel teachings and advantages of the present disclosure. Accordingly, all such modifications are intended to be included within the scope of the inventive concept as defined in the claims.

Embodiments of the inventive concept may be extended without limitation to the following statements:

Statement 1. Embodiments of the present invention include:

a buffer for storing input encoding data;

an index mapper to map input and output dictionaries;

a current encoding buffer storing modified current encoding data in response to the input encoding data, the input dictionary, and a map between the input dictionary and the output dictionary;

a previous encoding buffer storing previous input encoding data, the input dictionary, and modified previous encoding data generated in response to the map; and

and a transcoder including the modified current encoding data stored in the current encoding buffer, the modified previous encoding data stored in the previous encoding buffer, and a rule evaluator that generates an output stream in response to a transcoding rule.

Statement 2. An embodiment of the present invention includes a transcoder according to statement 1, wherein the index mapper is responsive to transcoding rules.

Statement 3. An embodiment of the present invention includes a transcoder according to statement 1, and the transcoding rules are responsive to the index mapper.

Statement 4. An embodiment of the invention includes a transcoder according to statement 1, wherein the index mapper responds to a selected subset of items in the input dictionary.

Statement 5. An embodiment of the present invention includes a transcoder according to statement 1, and the rule evaluator is a processor, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU), or a general-purpose Includes at least one GPU (GPGPU).

Statement 6. An embodiment of the present invention includes a transcoder according to statement 5, and the rule evaluator further includes at least one software for implementing a transcoding rule and storage for storing a table listing the transcoding rules. .

Statement 7. An embodiment of the present invention includes a transcoder according to statement 5, and the rule evaluator further includes circuitry for implementing transcoding rules.

Statement 8. An embodiment of the present invention includes a transcoder according to statement 1, wherein the rule evaluator operates to generate modified current encoded data from input encoded data using transcoding rules.

Statement 9. An embodiment of the invention includes a transcoder according to statement 8, wherein the rule evaluator is operative to add modified previously encoded data to the output stream.

Statement 10. An embodiment of the present invention includes a transcoder according to statement 9, wherein the rule evaluator is further operative to move the modified current encoding data from the current encoding buffer to the modified previous encoding data from the previous encoding buffer. do.

Statement 11. An embodiment of the present invention includes a transcoder according to statement 8, wherein the rule evaluator is operative to modify modified previously encoded data to include modified current encoded data using transcoding rules.

Statement 12. An embodiment of the present invention includes a transcoder according to statement 11, wherein the rule evaluator is further operative to change a first encoding scheme of the input encoding data to a second encoding scheme when generating modified current encoding data. do.

Statement 13. An embodiment of the present invention includes a transcoder according to statement 11, wherein the rule evaluator is further operative to change the first encoding scheme of the input encoding data to a third encoding scheme when generating modified current encoding data. do.

Statement 14. An embodiment of the present invention includes a transcoder according to statement 8, wherein the rule evaluator is operative to determine a first encoding scheme of the input encoding data from the input encoding data, and a first encoding scheme that is one of at least two encoding schemes. 1 Encoding method is used in the input encoding data.

Statement 15. An embodiment of the present invention includes a transcoder according to statement 1, identifying a first chunk in the input encoded data using a first encoding scheme, and identifying a second chunk in the input encoded data using a second encoding. It further includes a stream splitter that identifies.

Statement 16. An embodiment of the present invention includes a transcoder according to statement 1, wherein the index mapper is operative to map at least one item of the input dictionary to a “don't care” value in the output dictionary.

Statement 17. An embodiment of the present invention includes a transcoder according to statement 1, wherein the index mapper is operative to add “don't care” values to the output dictionary.

Statement 18. Embodiments of the present invention include a transcoder according to statement 1,

The input encoding data is compressed input encoding data; and

The transcoder further includes a decompression engine.

Statement 19. An embodiment of the present invention includes a transcoder according to statement 1, wherein the transcoder is operative to generate an output stream from input encoded data without decoding the input encoded data.

Statement 20. An embodiment of the present invention includes a transcoder according to statement 1, wherein the transcoder is included in a solid state drive (SSD) storage device.

Statement 21. An embodiment of the present invention includes a transcoder according to statement 20, wherein input encoded data is received from storage in an SSD storage device.

Statement 22. A method according to an embodiment of the present invention:

At the transcoder, receiving a first data chunk from input encoded data from a storage device;

determining whether the first data chunk is of interest to a host computer;

generating first encoded data from the first data chunk based at least in part on the first data chunk of interest to the host computer;

receiving, at the transcoder, a second data chunk from the input encoded data from the storage device;

determining whether the second data chunk is not of interest to the host computer;

generating second encoded data from the second data chunk based at least in part on the second data chunk that is not of interest to the host computer; and

and outputting the first encoded data and the second encoded data to the host computer.

Statement 23. Embodiments of the invention include a method according to statement 22, wherein generating second encoded data from a second data chunk based at least in part on a second data chunk of no interest to the host computer comprises: 1 It includes the step of changing the value of the encoded data to a “don’t care” value.

Statement 24. Embodiments of the invention include a method according to statement 23, generating second encoded data from a second data chunk based at least in part on a second data chunk that is not of interest to the host computer, comprising: and combining the second encoded data with third encoded data containing a “don’t care” value.

Statement 25. Embodiments of the invention include a method according to statement 24, wherein generating second encoded data from a second data chunk based at least in part on a second data chunk that is not of interest to the host computer comprises: and changing the first encoding method of at least one of the two data chunks and the third encoded data to a second encoding method.

Statement 26. An embodiment of the present invention includes a method according to statement 25, wherein changing the first encoding scheme of at least one of the second data chunk and the third encoded data to a second encoding scheme comprises: changing the first encoding scheme of at least one of the second data chunk and the third encoded data to a second encoding scheme; It includes changing the first encoding method of the second encoding data to the second encoding method of the second encoding data.

Statement 27. An embodiment of the present invention includes a method according to statement 25, wherein changing the first encoding scheme of at least one of the second data chunk and the third encoded data to a second encoding scheme comprises: changing the first encoding scheme of at least one of the second data chunk and the third encoded data to a second encoding scheme; It includes changing the first encoding method to the second encoding method.

Statement 28. An embodiment of the invention includes a method according to statement 22, wherein the host computer generates first encoded data from a first data chunk based at least in part on the first data chunk of interest, comprising: and combining the encoded data with third encoded data.

Statement 29. An embodiment of the invention includes a method according to statement 28, wherein the host computer generates first encoded data from a first data chunk based at least in part on the first data chunk of interest, comprising: It further includes changing the first encoding method of at least one of the first data chunk and the third encoded data to a second encoding method.

Statement 30. An embodiment of the present invention includes a method according to statement 29, wherein changing a first encoding scheme of at least one of the first data chunk and the third encoded data to a second encoding scheme comprises: changing the first encoding scheme of at least one of the first data chunk and the third encoded data to a second encoding scheme; It includes changing the first encoding method of the first encoding data to the second encoding method of the first encoding data.

Statement 31. An embodiment of the present invention includes a method according to statement 29, wherein changing the first encoding scheme of at least one of the first data chunk and the third encoded data to a second encoding scheme comprises: changing the second encoded data It includes changing the first encoding method to the second encoding method.

Statement 32. Embodiments of the invention include a method according to statement 22, wherein the host computer generates first encoded data from a first data chunk based at least in part on the first data chunk of interest, comprising: transcoding generating first encoded data from the first data chunk based at least in part on a rule; and

Generating second encoded data from the second data chunk based at least in part on the second data chunk of which the host computer is not interested includes generating second encoded data from the second data chunk based at least in part on transcoding rules. It includes steps to:

Statement 33. Embodiments of the invention include a method according to statement 22, wherein receiving, at a transcoder, a first chunk of data from input encoded data from a storage device:

Receiving input encoded data from a stream splitter;

Identifying, by the stream splitter, a first data chunk encoded using a first encoding scheme and a second data chunk encoded using a second encoding scheme of the input encoding data:

and receiving a first data chunk of input encoded data from a stream splitter.

Statement 34. Embodiments of the invention include a method according to statement 22, wherein:

Receiving an input dictionary from a storage device;

mapping the input dictionary to an output dictionary based at least in part on the host computer's data of interest and the host computer's non-interest data; and

and outputting the output dictionary to the host computer.

Statement 35. Embodiments of the present invention include a method according to statement 34, wherein mapping an input dictionary to an output dictionary based at least in part on data of interest in the host computer and data of interest in the host computer comprises: transcoding rules; and mapping the input dictionary to the output dictionary based at least in part on .

Statement 36. Embodiments of the present invention include a method according to statement 34, wherein mapping an input dictionary to an output dictionary based at least in part on data of interest in the host computer and data of interest in the host computer comprises: in the input dictionary and mapping the input dictionary to the output dictionary based at least in part on the subset of selected items.

Statement 37. An embodiment of the invention includes a method according to statement 22, wherein the transcoder operates to generate first encoded data and second encoded data from input encoded data without decoding the input encoded data.

Statement 38. Embodiments of the present invention include a method according to statement 22, wherein the transcoder is included in a solid state drive (SSD) storage device.

Statement 39. Embodiments of the invention include methods according to statement 38,

Receiving at the transcoder the first data chunk from the input encoded data from the storage device includes receiving at the transcoder the first data chunk from the input encoded data from storage in the SSD storage device; and

Receiving the second data chunk from the input encoded data from the storage device at the transcoder includes receiving the second data chunk from the input encoded data from storage in the SSD storage device at the transcoder.

Statement 40. Embodiments of the present invention include a product that includes a non-transitory storage medium, wherein the non-transitory storage medium stores instructions that, when executed by a machine, perform the following operations:

At the transcoder, receiving a first data chunk from input encoded data from a storage device;

determining whether the first data chunk is of interest to a host computer;

generating first encoded data from the first data chunk based at least in part on the first data chunk of interest to the host computer;

receiving, at the transcoder, a second data chunk from the input encoded data from the storage device;

determining whether the second data chunk is not of interest to the host computer;

generating second encoded data from the second data chunk based at least in part on the second data chunk that is not of interest to the host computer; and

Outputting the first encoded data and the second encoded data to the host computer is performed.

Statement 41. Embodiments of the invention include an article according to statement 40, wherein generating second encoded data from a second data chunk based at least in part on a second data chunk that is not of interest to the host computer comprises: 1 It includes the step of changing the value of the encoded data to an “undefined” value.

Statement 42. Embodiments of the invention include an article according to statement 41, wherein generating second encoded data from a second data chunk based at least in part on a second data chunk that is not of interest to the host computer comprises: and combining the second encoded data with third encoded data including a “don’t care” value.

Statement 43. Embodiments of the invention include an article according to statement 42, wherein generating second encoded data from a second data chunk based at least in part on a second data chunk that is not of interest to the host computer comprises: and changing a first encoding method of at least one of the second data chunk and the third encoded data to a second encoding method.

Statement 44. Embodiments of the present invention include an article according to statement 43, wherein changing the first encoding scheme of at least one of the second data chunk and the third encoded data to a second encoding scheme comprises: changing the second data chunk and the third encoded data. and changing the first encoding method of the chunk to the second encoding method of the second encoded data.

Statement 45. Embodiments of the present invention include the product according to statement 43, wherein changing the first encoding scheme of at least one of the second data chunk and the third encoded data to a second encoding scheme comprises: changing the third encoding scheme and changing a first encoding method of data to a second encoding method.

Statement 46. Embodiments of the invention include an article according to statement 40, wherein generating first encoded data from a first data chunk based at least in part on the first data chunk of interest to a host computer comprises: and combining the first encoded data with the third encoded data.

Statement 47. Embodiments of the invention include an article according to statement 46, wherein generating first encoded data from a first data chunk based at least in part on the first data chunk of interest to a host computer comprises: It further includes changing a first encoding method of at least one of the first data chunk and the third encoded data to a second encoding method.

Statement 48. Embodiments of the present invention include an article according to statement 47, wherein changing a first encoding scheme of at least one of the first data chunk and the third encoded data to a second encoding scheme comprises: changing the first data chunk and the third encoded data to a second encoding scheme; and changing the first encoding method of the chunk to the second encoding method of the first encoded data.

Statement 49. Embodiments of the present invention include an article according to statement 47, wherein changing the first encoding scheme of at least one of the first data chunk and the third encoded data to a second encoding scheme comprises: changing the third encoding scheme and changing a first encoding method of data to a second encoding method.

Statement 50. Embodiments of the invention include an article according to statement 40, wherein generating first encoded data from a first data chunk based at least in part on the first data chunk of interest to a host computer comprises: generating first encoded data from the first data chunk based at least in part on transcoding rules; and

Generating second encoded data from the second data chunk based at least in part on the second data chunk that is not of interest to the host computer includes generating second encoded data from the second data chunk based at least in part on transcoding rules. Includes creation steps.

Statement 51. Embodiments of the invention include an article according to statement 40, wherein receiving, at a transcoder, a first chunk of data from input encoded data from a storage device comprises:

Receiving input encoded data from a stream splitter;

identifying, by the stream splitter, a first data chunk encoded using a first encoding scheme and a second data chunk encoded using a second encoding scheme of the input encoding data; and

Receiving a first data chunk from input encoded data provided from a stream splitter.

Statement 52. Embodiments of the invention include an article according to statement 40, wherein the non-transitory storage medium stores additional instructions that, when executed by a machine, result in:

Receiving an input dictionary from a storage device;

mapping the input dictionary to an output dictionary based at least in part on the host computer's data of interest and the fraud host computer's non-interest data; and

A step of outputting the output dictionary to the host computer.

Statement 53. Embodiments of the invention include an article according to statement 52, wherein mapping an input dictionary to an output dictionary based at least in part on data of interest to the host computer and data of no interest to the host computer comprises: transcoding and mapping the input dictionary to the output dictionary based at least in part on rules.

Statement 54. Embodiments of the invention include an article according to statement 52, wherein mapping an input dictionary to an output dictionary based at least in part on data of interest to the host computer and data of no interest to the host computer comprises: and mapping the input dictionary to the output dictionary based at least in part on a subset of items selected from the input dictionary.

Statement 55. Embodiments of the invention include an article according to statement 40, wherein the transcoder is operative to generate first encoded data and second encoded data from input encoded data without decoding the input encoded data.

Statement 56. Embodiments of the invention include an article according to statement 40, wherein a transcoder is included in a solid state drive (SSD) storage device.

Statement 57. Embodiments of the present invention include products according to Statement 56,

Receiving at the transcoder the first data chunk from the input encoded data from the storage device includes receiving at the transcoder the first data chunk from the input encoded data from storage in the SSD storage device; and

Receiving the second data chunk from the input encoded data from the storage device at the transcoder includes receiving the second data chunk from the input encoded data from storage in the SSD storage device at the transcoder.

Statement 58. Embodiments of the present invention include a storage device comprising:

Storage for input encoded data;

a controller that processes read and write requests from a host computer on the storage;

an in-storage computer (ISC) controller that receives attributes originating from a host computer applied to input encoded data stored in storage; and

A transcoder comprising an index mapper for mapping from an input dictionary to an output dictionary for input encoded data, wherein the input dictionary includes at least one first item and at least one second item, the at least one first item The first item is mapped to at least one third item, and at least one second item is mapped to an “undefined” item in the output dictionary.

Statement 59. An embodiment of the present invention includes a storage device according to statement 58, wherein the transcoder is a processor, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or a general-purpose Includes at least one of GPUs (GPGPU).

Statement 60. An embodiment of the present invention includes a storage device according to statement 58, wherein the ISC controller is operative to apply acceleration functions to output encoded data from the transcoder.

Statement 61. An embodiment of the present invention includes a storage device according to statement 60, wherein the ISC controller is operative to output the results of the acceleration function on the output encoded data from the transcoder to the host computer.

Statement 62. An embodiment of the present invention includes a storage device according to statement 58, wherein the ISC controller is operative to convey the output encoded data of the transcoder to the host computer.

Statement 63. An embodiment of the present invention includes a storage device according to statement 62, wherein the ISC controller is operative to deliver the output dictionary to the host computer.

Statement 64. An embodiment of the present invention includes a storage device according to statement 58, wherein the transcoder is operative to generate output encoded data based at least in part on input encoded data and a map from an input dictionary to an output dictionary.

Statement 65. Embodiments of the present invention include a storage device according to statement 64, wherein the transcoder:

Buffer to store input encoding data;

index mapper;

a current encoding buffer storing modified current encoding data in response to input encoding data, modified current encoding data, and a map from an input dictionary to an output dictionary;

a previous encoding buffer for storing the previous encoding data modified in response to the previous input encoding data, the modified previous encoding data, and a map from the input dictionary to the output dictionary; and

It includes a rule evaluator for generating an output stream in response to modified current encoding data in the current encoding buffer, modified previous encoding data in the previous encoding buffer, and transcoding rules.

Statement 66. An embodiment of the present invention includes a storage device according to statement 65, wherein the transcoding rules are based at least in part on the attributes.

Statement 67. An embodiment of the present invention includes a storage device according to statement 65, wherein the rule evaluator is configured to: without decoding the input encoding data, modify the current encoding data in the current encoding buffer, the modified previous encoding data in the previous encoding buffer, and Generates an output stream in response to transcoding rules.

Statement 68. Embodiments of the present invention include a storage device according to statement 64,

The input encoded data uses a first encoding method;

The output encoded data uses a second encoding method; and

The second encoding method is different from the first encoding method.

Statement 69. An embodiment of the present invention includes a storage device according to statement 58, wherein input encoded data is stored in the storage in a columnar format.

Statement 70. An embodiment of the present invention includes a storage device according to statement 69, wherein the input encoded data includes an input file stored using the Apache Parquet storage format.

Statement 71. An embodiment of the present invention includes a storage device according to statement 69, and further includes a column chunk processor for processing column chunks containing input encoded data and delivering the input encoded data to the transcoder.

Statement 72. An embodiment of the present invention includes a storage device according to statement 71, wherein the column chunk processor includes a transcoder.

Statement 73. An embodiment of the present invention includes a storage device according to statement 71, wherein the column chunk processor is a processor, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU), or Includes at least one general-purpose GPU (GPGPU).

Statement 74. An embodiment of the present invention includes a storage device according to statement 58, wherein the transcoder is operative to generate transcoding rules to apply to input encoded data based at least in part on the attributes to generate output encoded data. .

Statement 75. An embodiment of the present invention includes a storage device according to statement 74, wherein the transcoder is operative to generate output encoded data without decoding input encoded data.

Statement 76. Embodiments of the present invention include methods comprising:

Receiving attributes to be applied to input encoded data from a transcoder;

accessing an input dictionary for input encoded data;

identifying at least one first item in the input dictionary that is covered by the attribute and at least one second item in the input dictionary that is not covered by the attribute;

generating an output dictionary that excludes at least one second item from the dictionary that is not covered by the attribute, wherein the transcoding dictionary includes at least a third item and an “undefined” item; and

"undefined" of the output dictionary by a transcoder that maps at least one first item of the input dictionary to at least one third item of the output dictionary and maps at least one second item of the input dictionary that is not covered by the attribute It includes a step of mapping to the “(don’t care)” item.

Statement 77. Embodiments of the present invention include a method according to statement 76, wherein input encoded data is stored in a columnar format.

Statement 78. Embodiments of the invention include a method according to statement 77, wherein the input encoded data includes an input file stored using the Apache Parquet storage format.

Statement 79. Embodiments of the invention include a method according to statement 76, wherein the input encoded data includes column chunks stored in a columnar format.

Statement 80. Embodiments of the invention include a method according to statement 76, wherein:

transcoding input encoded data into output encoded data using a dictionary map; and

and outputting output encoded data.

Statement 81. Embodiments of the invention include a method according to statement 80, wherein transcoding input encoded data into output encoded data using a dictionary map includes:

Receiving a first data chunk from input encoded data at a transcoder;

determining that a first data chunk is covered by an attribute;

generating first encoded data from a first data chunk based at least in part on a first data chunk of interest to the host computer using the dictionary map;

At the transcoder, receiving a second data chunk from input encoded data from a storage device;

determining that the second data chunk is not covered by the attribute;

generating second encoded data from a second data chunk based at least in part on a second data chunk that is not of interest to the host computer using the dictionary map; and

and outputting first encoded data and second encoded data.

Statement 82. Embodiments of the invention include a method according to statement 81, wherein receiving a first data chunk from input encoded data at a transcoder comprises:

At the column chunk processor, receiving a list of block identifiers (IDs) from an in-storage computer (ISC) controller;

Accessing, by a column chunk processor, a column chunk containing a block ID in a block ID list;

Retrieving input encoded data from a column chunk from a column chunk processor; and

and passing input encoded data from the column chunk processor to the transcoder.

Statement 83. Embodiments of the invention include a method according to statement 81, further comprising generating transcoding rules for applying to input encoded data based at least in part.

Statement 84. An embodiment of the present invention includes a method according to statement 80, wherein transcoding input encoded data into output encoded data using a dictionary map comprises, without decoding the input encoded data, using the dictionary map: and transcoding the input encoded data into output encoded data.

Statement 85. Embodiments of the present invention include methods according to statement 80,

The input encoded data uses a first encoding method;

The output encoded data uses a second encoding method; and

The second encoding method is different from the first encoding method.

Statement 86. An embodiment of the present invention includes a method according to statement 80, wherein outputting the output encoded data includes outputting the output encoded data to an ISC controller.

Statement 87. An embodiment of the present invention includes a method according to statement 86, wherein outputting the output encoded data to the ISC controller further includes outputting an output dictionary to the ISC controller.

Statement 88. Embodiments of the present invention include a method according to statement 87, further comprising transferring output encoding data and an output dictionary from an ISC controller to a host computer.

Statement 89. Embodiments of the present invention include a method according to statement 87, further comprising performing an acceleration function on the output encoded data by the ISC controller to generate accelerated data.

Statement 90. An embodiment of the present invention includes a method according to statement 89, further comprising outputting accelerated data from the ISC controller to the host computer.

Statement 91. An embodiment of the present invention includes a method according to statement 76, further comprising outputting an output dictionary.

Statement 92. Embodiments of the invention include a method according to statement 76, wherein receiving attributes to apply to input encoded data comprises receiving attributes to apply to input encoded data from an ISC controller. .

Statement 93. An embodiment of the present invention includes a method according to statement 92, further comprising receiving an input dictionary from an ISC controller.

Statement 94. Embodiments of the invention include a method according to Statement 76, wherein:

determining that there are no items in the input dictionary that are not covered by the attribute; and

and outputting the input encoded data without transcoding the input encoded data into output encoded data.

Statement 95. Embodiments of the invention include articles comprising a non-transitory storage medium, wherein the non-transitory storage medium stores instructions that, when executed by a machine, result in:

Receiving attributes to be applied to input encoded data from a transcoder;

accessing an input dictionary for input encoded data;

identifying at least one first item in the input dictionary that is covered by the attribute and at least one second item in the input dictionary that is not covered by the attribute;

generating an output dictionary that excludes at least one second item in the input dictionary that is not covered by the attribute, wherein the transcoding dictionary includes at least a third item and a “don't care” item; and

By the transcoder, at least one first item of the input dictionary is mapped to at least one third item of the output dictionary, and at least one second item of the input dictionary that is not covered by the attribute is mapped to an “undefined” It includes a step of mapping to the “(don’t care)” item.

Statement 96. Embodiments of the invention include an article according to statement 95, wherein input encoded data is stored in a columnar format.

Statement 97. Embodiments of the invention include an article according to statement 96, wherein the input encoded data includes an input file stored using the Apache Parquet storage format.

Statement 98. Embodiments of the present invention include an article according to statement 95, wherein the input encoded data includes column chunks stored in a columnar format.

Statement 99. Embodiments of the invention include products according to statement 95, wherein the non-transitory storage medium stores additional instructions that, when executed by a machine, result in:

transcoding input encoded data into output encoded data using a dictionary map; and

and outputting output encoded data.

Statement 100. Embodiments of the invention include an article according to statement 99, wherein transcoding input encoded data into output encoded data using a dictionary map includes:

Receiving a first data chunk from input encoded data at a transcoder;

determining that a first data chunk is covered by an attribute;

generating first encoded data from a first data chunk based at least in part on a first data chunk of interest to the host computer using the dictionary map;

Receiving, at the transcoder, a second data chunk from input encoded data from a storage device;

determining that the second data chunk is not covered by the attribute;

generating second encoded data from a second data chunk based at least in part on a second data chunk that is not of interest to the host computer using the dictionary map; and

and outputting first encoded data and second encoded data.

Statement 101. Embodiments of the present invention include products according to Zansul 100, wherein receiving a first data chunk from input encoded data at a transcoder comprises:

At the column chunk processor, receiving a list of block identifiers (IDs) from an in-storage computer (ISC) controller;

Accessing, by a column chunk processor, a column chunk containing a block identifier (ID) in a block identifier (ID) list;

Retrieving, by a column chunk processor, input encoded data from a column chunk; and

It includes transferring input encoded data from the column chunk processor to the transcoder.

Statement 102. Embodiments of the invention include an article according to statement 100, wherein the non-transitory storage medium, when executed by a machine, generates transcoding rules based at least in part on the attributes, and applies additional transcoding rules to the input encoded data. Save the command.

Statement 103. Embodiments of the invention include an article according to statement 99, wherein the step of transcoding input encoded data into output encoded data using a dictionary map comprises: transcoding the input encoded data using the dictionary map without decoding the input encoded data; and transcoding the data into output encoded data.

Statement 104. Embodiments of the present invention include products according to Statement 99,

The input encoded data uses a first encoding method;

The output encoded data uses a second encoding method; and

The second encoding method is different from the first encoding method.

Statement 105. Embodiments of the present invention include a product according to statement 99, wherein outputting the output encoded data includes outputting the output encoded data to an ISC controller.

Statement 106. An embodiment of the present invention includes a product according to statement 105, wherein outputting the output encoded data to the ISC controller further includes outputting an output dictionary to the ISC controller.

Statement 107. Embodiments of the present invention include an article according to statement 106, wherein the non-transitory storage medium, when executed by a machine, transfers output encoding data and an output dictionary from an ISC controller to a host computer.

Statement 108. Embodiments of the present invention include an article according to statement 106, wherein the non-transitory storage medium, when executed by a machine, performs an acceleration function on output encoded data by an ISC controller to produce accelerated data. Save additional commands.

Statement 109. Embodiments of the present invention include an article according to statement 108, wherein the non-transitory storage medium stores additional instructions that, when executed by a machine, output accelerated data from the ISC controller to the host computer.

Statement 110. Embodiments of the invention include an article according to statement 95, wherein the non-transitory storage medium stores additional instructions that, when executed by a machine, output an output dictionary.

Statement 111. Embodiments of the invention include an article according to statement 95, wherein receiving attributes to apply to input encoded data includes receiving attributes to apply to input encoded data from an ISC controller.

Statement 112. Embodiments of the invention include an article according to statement 111, wherein the non-transitory storage medium stores additional instructions that, when executed by a machine, receive an input dictionary from an ISC controller.

Statement 113. Embodiments of the invention include products according to statement 95, wherein the non-transitory storage medium stores additional instructions that, when executed by a machine, result in:

determining that there are no entries in the input dictionary that are not covered by the attribute; and

and outputting the input encoded data without transcoding the input encoded data into output encoded data.

Consequently, considering the various permutations of the embodiments described herein, this detailed description and accompanying material are intended to be illustrative only and should not be construed as limiting the scope of the inventive concept. Therefore, what is claimed as an inventive concept is the scope and spirit of the following claims and all modifications that may come within their equivalents.

Claims (8)

a buffer for storing input encoding data;
an index mapper for mapping from an input dictionary to an output dictionary;
a current encoding buffer storing modified current encoding data responsive to the input encoding data, the input dictionary, and a map from the input dictionary to the output dictionary;
a previous encoding buffer storing previous input encoding data, the input dictionary, and modified previous encoding data responsive to the map from the input dictionary to the output dictionary; and
A transcoder comprising a rule evaluator that generates an output stream in response to the modified current encoding data stored in the current encoding buffer, the modified previous encoding data stored in the previous encoding buffer, and transcoding rules. According to claim 1,
The rule evaluator is a transcoder that includes at least one of a processor, a Field Programmable Gate Array (FPGA), an Application-Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU), and a general-purpose GPU (GPGPU). According to claim 1,
A transcoder wherein the rule evaluator is operative to generate the modified current encoded data from the input encoded data using transcoding rules. According to claim 3,
and the rule evaluator is operative to modify the modified previously encoded data to include the modified current encoded data using the transcoding rules. According to claim 4,
The rule evaluator is operable to change a first encoding method of at least one of the input encoding data and the modified previous encoding data to a second encoding method. According to claim 1,
The transcoder further comprising a stream splitter that identifies a first chunk in the input encoded data using a first encoding scheme and identifies a second chunk in the input encoded data using a second encoding scheme. According to claim 1,
A transcoder wherein the index mapper is operative to map at least one item of the input dictionary to a “don't care” value of the output dictionary. According to claim 1,
The transcoder is operative to generate an output stream from the input encoded data without decoding the input encoded data.

Images