KR101253700B1 - High Speed Encoding Apparatus for the Next Generation Sequencing Data and Method therefor - Google Patents

Abstract

A high speed compression apparatus for NGS data and a compression method thereof are disclosed. A high speed compression apparatus for NGS data according to an embodiment of the present invention, the data compression unit for applying different compression methods by separating the DNA sequence data into a sequence identifier line and a base data line for the NGS (Next Generation Sequencer) data; And in order to minimize the interference between the current data block by the data compression unit and the next data block, the base data line is converted into binary data by referring to an encode map. An interference minimizing unit that combines byte data; And a post-processing partition storage unit that divides and stores the compressed data so that the compressed data can be decompressed in parallel.

Description

High Speed Encoding Apparatus for the Next Generation Sequencing Data and Method therefor}

Embodiments of the present invention relate to a high speed compression apparatus and method thereof for NGS data (Next Generation Sequencing Data). More specifically, the present invention relates to a high speed compression apparatus for NGS data and a method for compressing and dividing NGS data at high speed so as to be suitable for transmission of DNA sequence data in the NGS preparation stage in cloud computing.

Cloud Computing is a computing environment in which information technology (IT) related services such as data storage, network, and content use can be used at a time through servers on the Internet. Information is stored permanently on servers on the Internet. It refers to a computer environment that is temporarily stored in clients such as desktop devices, tablet computers, laptops, netbooks, and smart phones such as IT devices. In other words, all user information is stored in a server on the Internet, and this information can be used anytime and anywhere through various IT devices. In other words, cloud computing is a computing service that borrows computing resources such as hardware and software that exist in an intangible form, such as a cloud, and pays for them as needed. Refers to a technology that integrates and provides computing resources in a virtualization technology. Cloud computing, an innovative computing technology that provides IT-related services such as data storage, processing, network, and content usage on a server on the Internet represented as a cloud, is sometimes referred to as an on-demand outsourcing service for IT resources using the Internet.

By adopting cloud computing, companies or individuals can save tremendous costs, time and labor such as the cost of maintaining, maintaining, and managing computer systems, purchasing and installing servers, updating, and purchasing software. Can contribute.

In addition, if the data is stored on a personal computer (PC), data may be lost due to a hard disk failure, but in a cloud computing environment, the data is stored on an external server so that the data can be safely stored and the limitation of storage space can be overcome. You can view and modify the documents you have worked on anytime, anywhere. However, if the server is hacked, personal information may be leaked, and if a server failure occurs, data may not be available.

Recently, various services such as tablet computers or smartphones can be easily used through cloud computing environments built on various portal sites such as Google, Daum, and Naver. Cloud computing has attracted attention as a next-generation Internet service because of its ease of use and industrial ripple effect, and emerged as a new IT integrated management model in the late 2000s.

Meanwhile, the Human Genome Project, which began in 1988, is a grand project meant to read the entire genome sequencer of 3 billion base pairs in humans. Though initially considered reckless, its value was deemed so important that it was called the Apollo Plan of Biology, and the development of various technologies centered around the genome sequencer. As a result, the project was announced in 2003 sooner than expected, and scientists around the world are working on bioinformatics research using the complete sequence of human information. And now, in life sciences, we are really thinking about deciphering the sequences of each person and getting information in them that can improve the health of each individual.

One of the biggest barriers here is the cost and time required to decipher 3 billion base pairs. It is also clear from the fact that starting in 1988, it was an international project that only a single genome was deciphered in 2003. But scientists now expect to be able to decipher one person's genome in the near future for about $ 1000. The reason for this is that new technologies are constantly emerging and the performance of genome sequencers is quantum leaping. Recently, products called the next generation genome sequencers continue to appear.

Recently, researches on using cloud computing to efficiently process a large amount of Next Generation Sequencer (NGS) data have been studied.

However, transferring hundreds of gigabytes of NGS data to cloud storage such as Amazon's EC2 and EBS requires a lot of time and money. An important factor in the coupling between cloud computing environment and NGS is not only the compression rate of DNA (Deoxyribonucleic acid) sequence for long term storage but also the time factors such as transmission speed and compression time. Therefore, processing methods for DNA sequence data need to be searched for as little working time as possible.

Embodiments of the present invention have been devised to meet the above-described needs, and can be used to compress NGS data at high speed and to perform decoding in parallel so as to be suitable for transmission of DNA sequence data in the NGS preparation stage in cloud computing. An object of the present invention is to provide a high speed compression apparatus for NGS data and a method thereof, which can divide data.

In the high-speed compression apparatus of NGS data according to the embodiment of the present invention for achieving the above object, different compression method by separating the DNA sequence data into a sequence identifier line and a base data line for NGS (Next Generation Sequencer) data Data compression unit for applying; And in order to minimize the interference between the current data block by the data compression unit and the next data block, the base data line is converted into binary data by referring to an encode map. And an interference minimizing unit for combining the byte data.

Here, the sequence identifier line may be a portion that is repeated at a frequency equal to or greater than a set value. In this case, the data compression unit may compress the sequence identifier line using a general-purpose compression method such as LZMA (Lempel-Ziv-Markov chain Algorithm), gzip, bzip2, or the like.

In addition, the base data line may be a part composed of Adenine Cytosine Guanine Thymine (ACGT) having random pattern characteristics. In this case, the data compression unit may compress the base data line by using a bit operation level compression.

According to an embodiment of the present invention, it is possible to compress the NGS data at high speed so as to be suitable for transmission of DNA sequence data in the NGS preparation step in cloud computing. Compression results can also be partitioned for decryption in parallel in cloud computing.

1 is a view schematically showing a high speed compression apparatus of NGS data according to an embodiment of the present invention.
2 is a diagram illustrating an example of a compression process using SOLiDzipper.
3 is a diagram illustrating a result of comparing compression efficiency and time when different compression methods are used.
4 is a diagram illustrating a change in Rt according to a data transmission rate and a compression method.
5 is a flowchart illustrating a fast compression method of NGS data according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, a detailed description of known techniques well known to those skilled in the art may be omitted.

In describing the constituent elements of the present invention, the same reference numerals may be given to constituent elements having the same name, and the same reference numerals may be given thereto even though they are different from each other. However, even in such a case, it does not mean that the corresponding component has different functions according to the embodiment, or does not mean that the different components have the same function. It should be judged based on the description of each component in the example.

In the following description of the embodiments of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

In addition, in describing the component of this invention, terms, such as 1st, 2nd, A, B, (a), (b), can be used. These terms are intended to distinguish the constituent elements from other constituent elements, and the terms do not limit the nature, order or order of the constituent elements. When a component is described as being "connected", "coupled", or "connected" to another component, the component may be directly connected or connected to the other component, Quot; may be "connected," "coupled," or "connected. &Quot;

1 is a view schematically showing a high speed compression apparatus of NGS data according to an embodiment of the present invention. The high speed compression apparatus 100 for NGS data according to an embodiment of the present invention includes a data compression unit 110 and an interference minimization unit 120. In addition, due to the storage method of the interference minimizing unit 120, the compression result may be divided to be decoded in parallel in cloud computing.

The data compression unit 110 separates DNA sequence data into sequence identifier lines and base data lines for NGS data, and applies different compression methods. Here, the sequence identifier line may be a portion that is repeated at a frequency equal to or greater than a set value. In this case, the data compression unit 110 may compress the sequence identifier line using a general compression method such as LZMA (Lempel-Ziv-Markov chain Algorithm), gzip or bzip2. In addition, the base data line is a part composed of ACGT (Adenine Cytosine Guanine Thymine) or 0123 (color space) having random pattern characteristics. In this case, the data compression unit 110 may compress the base data line by using a bit operation level compression. However, the compression method is not limited to the described methods, and various compression methods are applicable.

There are two main compression strategies for encoding. The first trick is to separate parts of the DNA sequence data that are easily compressed from those that are not. In general, a region that is easily compressed is a portion that is repeated at a high frequency such as a tag ID or a sequence number, and a portion that is not easily is a DNA sequence portion having random pattern characteristics. Therefore, it is desirable to apply the encoding method differently according to the area of data. At this time, the plain text portion can be compressed using a general-purpose compression algorithm that compresses plain text such as LZMA (Lempel-Ziv-Markov chain Algorithm) or gzip with excellent efficiency, and uses ACGT (Adenine Cytosine). The base part consisting of Guanine Thymine and the quality value part consisting of a fixed number range can use bit operation level encoding.

The second strategy is to minimize the interference between the compression result of the data blocks read from storage and the next block processing. In general, as the size of the data dictionary used for compression increases, the compression efficiency may increase. For example, G-SQZ, a recent DNA compression method, can compress DNA sequences by about 80% using Huffman coding. However, G-SQZ needs to make Huffman tree during encoding process. As the alphabet length of the analysis data increases, the size of Huffman tree also increases. Due to the nature of DNA data, compressing hundreds of GBytes of data requires physical memory space to load a huge Huffman tree. However, the data dictionary increase due to the increase in the compression size has a problem that the data dictionary search time can be increased as much as the data dictionary size for the next data block processing.

To this end, the interference minimizing unit 120 may refer to an encode map of the base data line in order to minimize the interference between the current data block and the next data block by the data compression unit 110. After converting into binary data, combine them into byte data.

Fast compression of NGS data performs character-centric bit operations using characteristics that vary in a limited range of base data or quality values of DNA sequences. For example, csfasta, in CSFASTA / QUAL format, requires 2 bits of space to map the change of 2 ² values into a bit value, such as the color space characters '0,1,2,3'. . It is assumed that one byte character '0' of the color space is mapped to binary data '00', '1' is '01', '2' is '10', and '3' is mapped to '11'. The four-byte data '0113' of the color space is encoded into a one-byte character '00 01 01 11 '= 0x17 by performing a shift operation to one byte space after a bit operation. The quality value of CSFASTA / QUAL has a minimum value of -1 and a maximum of 40 values. Therefore, a 6-bit space is required when encoding a quality value that allows a change in the value of 2 ⁶ . The two bits of space left in the previous operation are divided and stored in different quality values.

2 is a diagram illustrating an example of a compression process using SOLiDzipper. In SOLiDzipper, data is read by a data read block, and a preprocessing block separates DNA sequence data into sequence identifier lines and base data lines for NGS data as described above. The main processing block may compress plain text information such as an ID of a sequence and a plain text number using a general compression method. At this time, the quality value compression such as converting the quality value into 1 byte or assigning 4 quality values to 3 bytes may be performed, and the csfasta compression may be performed to allocate 2 bits to ACGT or 0123 or combine 4 bases into 1 byte. Unlike conventional compression methods, SOLiDzipper does not use a compression dictionary scheme, thus minimizing computing resource requirements and pre seek time.

3 is a diagram illustrating a result of comparing compression efficiency and time when different compression methods are used.

In summary, the implementation of NGS high-speed compression is to obtain the DNA sequence data compression experiment.

a) What is the difference between compressing a format in which field values change over a limited range, such as fasta, in a different way than traditional compression mechanisms?

b) are there any differences in the compression results when mixing coding methods with traditional compression methods?

For this purpose, NGS Fast Compression is a 64-bit Linux machine in the form of a Java 1.6 command-line (Linux: 2.6.29.4-167.fc11.x86_64 Fedora 11 64bit, Intel (R) Core (TM)). 2 Duo CPU E8400 3.00GHz, 4Gbyte memory) and the experiment was conducted in the same environment. The compression target for the experiment was to select data in the CSFASTA / QUAL format generated on the SOLiD platform. The reason for this is that, due to the structural characteristics of CSFASTA / QUAL, the data size is much larger than that of FASTAQ with the same contents, and the relative compression effect is high during compression.

Compression comparison experiments include the fast compression option (mx = 1) of gzip (version 1.3.12) and LZMA (version 4.65), the highest compression efficiency option (mx = 9) of LZMA, and the latest DNA sequence compression method. G-SQZ (version 0.6) was used.

4 is a diagram illustrating a change in Rt according to a data transmission rate and a compression method.

The results show the difference in compression performance when the coding method for DNA data is compared with the conventional DNA compression method or general purpose compression method.

The general purpose of DNA compression is to reduce the long term storage costs associated with bulk data compression. Therefore, DNA compression may be an important factor in compression efficiency. However, in the case of the combination of cloud computing environment and NGS, as important factors as the compression ratio of DNA sequence are time factors such as transmission speed and compression time to cloud computing environment. Ready to job time (Rt) is the sum of compression, transmission, and decompression time. If the time factor is not taken into account in DNA sequence compression, an increase in Rt occurs with increasing compression and decompression time. Thus, if Rt increases, the benefit of reducing NGS work time using cloud computing is lost. Rt of the DNA sequence data to cloud computing can be solved as in Equation 1.

[Equation 1]

In Equation 1, 133 GigaByte data generated by SOLiDTM 3.5 is substituted into the compression result of FIG. 3 and the change in Rt according to the transmission rate is shown in FIG. 4. In FIG. 4, the result of 400 hours or more was cut out due to a problem on the ground.

According to FIG. 4, except for a research institute or a company having a dedicated network capable of delivering a gigabit transmission rate to a cloud computing environment, high-speed compression is considered the best method in consideration of time factors in a general transmission environment. Able to know.

Representative characteristics of the encoding method are as follows.

a) Only supports known text format sequence formats.

b) As the size of the compression target increases linearly from several gigabytes to several terabytes, the resulting time also increases linearly. In addition, even if the size of data changes during compression of the same format, the change in compression rate is small.

c) Since the stored compressed files are stored in fixed block units, data at a specific location can be restored without decompressing the entire file.

Since encoding is byte to multi-bit encoding using a limited value change of DNA sequence data, there is a limitation in general plain text compression application except for the DNA sequence format. However, even if the encoding is extended to have a multi-byte quality value or color space in the future, the effect of the encoding can still be expected if there is space left in the storage method. Encoding is designed to help research NGS with cloud computing. However, cost savings due to long term data retention after transfer to the cloud environment are beyond the subject of the present invention. Embodiments of the present invention have provided a realistic alternative to compressing, transmitting and decompressing massive amounts of DNA sequence data generated remotely to a network with cloud computing at minimal cost. Thus, when moving data to cloud computing, it showed the best possible trade-off of dramatically reducing work latency at the expense of several percent compression efficiency compared to conventional DNA compression methods.

FIG. 5 is a flowchart illustrating a high speed compression method of NGS data by the high speed compression device of the NGS data of FIG. 1.

1 and 5, the data compression unit 110 separates DNA sequence data into sequence identifier lines and base data lines for NGS data (S501). In addition, the data compression unit 110 compresses by applying different compression methods to the separated sequence identifier line and the base data line, respectively (S503). Here, the sequence identifier line may be a portion that is repeated at a frequency equal to or greater than a set value. In this case, the data compression unit 110 may compress the sequence identifier line using a general purpose compression method. In addition, the base data line may be a part composed of Adenine Cytosine Guanine Thymine (ACGT) having random pattern characteristics. In this case, the data compression unit 110 may compress the base data line by using a bit operation level compression. However, the compression method is not limited to the described methods, and various compression methods are applicable.

The interference minimizing unit 120 uses the binary data (reference) to reference the encode map in order to minimize the interference between the current data block by the data compression unit 110 and the next data block. binary data), and then combine into byte data (S505). The post processing storage 140 may separate and store the converted data of the interference minimization unit 120 using the division storage 130.

The present invention is not necessarily limited to these embodiments, as all the constituent elements constituting the embodiment of the present invention are described as being combined or operated in one operation. In other words, within the scope of the present invention, all of the components may be selectively operated in combination with one or more. In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. In addition, such a computer program may be stored in a computer-readable medium such as a USB memory, a CD disk, a flash memory, etc., and read and executed by a computer, thereby implementing embodiments of the present invention. As the storage medium of the computer program, a magnetic recording medium, an optical recording medium, a carrier wave medium, or the like may be included.

Furthermore, all terms including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined in the Detailed Description. Terms used generally, such as terms defined in a dictionary, should be interpreted to coincide with the contextual meaning of the related art, and shall not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present invention.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, the protection scope of the present invention should be interpreted by the claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: high speed compression device of NGS data
110: data compression unit 120: interference minimizer
130: data division storage 140: post-processing storage

Claims (10)

A data compression unit which separates DNA sequence data into sequence identifier lines and base data lines and applies different compression methods to NGS (Next Generation Sequencer) data;
In order to minimize the interference between the current data block by the data compression unit and the next data block, the base data line is converted into binary data by referring to an encode map. An interference minimizing unit for combining byte data; And
Post-processing split storage that splits and stores compressed data so that it can be decompressed in parallel
High speed compression device of the NGS data comprising a.
The method of claim 1,
The sequence identifier line is a fast compression apparatus for NGS data, characterized in that the portion having a characteristic that is repeated at a frequency of more than a predetermined value, such as a tag ID or sequence number.
3. The method according to claim 1 or 2,
The data compression unit,
And compressing the sequence identifier line using a general-purpose compression method.
The method of claim 1,
The base data line is a high-speed compression device for NGS data, characterized in that the part consisting of adenine Cytosine Guanine Thymine (ACGT) or 0123 (color space) having a random pattern characteristics.
The method according to claim 1 or 4,
The data compression unit,
And compressing the base data line using a bit operation level compression.
Separating DNA sequence data into sequence identifier lines and base data lines for NGS (Next Generation Sequencer) data;
Compressing by applying different compression methods to the sequence identifier line and the base data line, respectively; And
In order to minimize the interference between the current data block by the data compression unit and the next data block, the base data line is converted into binary data by referring to an encode map. Combining into byte data
Fast compression method of the NGS data, characterized in that it comprises a.
The method according to claim 6,
The sequence identifier line is a fast compression method of the NGS data, characterized in that the part having a characteristic that is repeated at a frequency of more than a predetermined value, such as a tag ID or sequence number. 8. The method according to claim 6 or 7,
The compression step,
And compressing the sequence identifier line using a general purpose compression method.
The method according to claim 6,
The base data line is a high-speed compression method of NGS data, characterized in that the part consisting of adenine Cytosine Guanine Thymine (ACGT) or 0123 (color space) having a random pattern characteristics.
10. The method according to claim 6 or 9,
The compression step,
And compressing the base data line using a bit operation level compression.

Images