CN111881324B - High-throughput sequencing data general storage format structure, construction method and application thereof - Google Patents

Abstract

The invention provides a general storage format structure of high-throughput sequencing data, a construction method and application thereof. By the invention, different types of high-throughput sequencing data can be stored in one format, so that the defect that the interoperability of the data is influenced due to the diversity of the data formats is overcome. And meanwhile, the universal format is structured, and compared with text unstructured data, the universal format is easier and quicker to filter and extract data.

Description

Common storage format structure of high-throughput sequencing data, its construction method and application

Technical field

The invention belongs to the technical field of biological information processing and relates to a universal storage format structure of high-throughput sequencing data, its construction method and application.

Background technique

With the rapid development of high-throughput sequencing technology, differences in the instruments or suppliers used in sequencing, sequencing principles, and development backgrounds or goals, such as readability, integration, space saving, and other factors, have resulted in more and more problems. coming from more and more types of sequencing data. In order to analyze these data, people have designed a large number of analysis software, but most of these analysis software define their own data storage format (S.Pabinger, A.Dander, M.Fischer, R.Snajder, M.Sperk, M.Efremova, B.Krabichler, M.R.Speicher, J.Zschocke, and Z.Trajanoski, "Asurvey of tools for variant analysis of next-generation genome sequencing data," BriefBioinform, vol.15, no.2, pp.256-78, Mar , 2014). For example, BAM/FASTQ/QSEQ, BAM/HDF5/FASTQ and BAM/SFF/FASTQ are file formats that can be processed by Illumina, PacBio and Ion Torrent sequencers respectively. The above reasons have resulted in the diversity of data formats.

Data interoperability is a key link in big data analysis. Many format conversion tools have been developed one after another. Their main function is to convert high-throughput sequencing data from one format to another (H.Li, B.Handsaker,A.Wysoker,T.Fennell,J.Ruan,N.Homer,G.Marth,G.Abecasis,andR.Durbin, "The Sequence Alignment/Map format and SAMtools," Bioinformatics, vol.25, no .16, pp.2078-9, Aug 15, 2009.; M.R. Breese, and Y. Liu, "NGSUtils: a software suite for analyzing and manipulating next-generation sequencing datasets," Bioinformatics, vol.29, no.4, pp .494-6, Feb15, 2013.).

However, most of them are developed for specific and limited formats, and format conversion not only loses information but also requires a lot of computing resources. If there is no ready-made tool to convert a format into the required format, you have to wait for others to develop it, or write a program to implement it yourself. This is not an easy task for non-professional program developers.

Contents of the invention

In view of the above technical problems, the purpose of the present invention is to provide a universal storage format structure of high-throughput sequencing data, its construction method and application. Different types of high-throughput sequencing data can be stored in one format, thus overcoming the impact of the diversity of data formats on data interoperability. At the same time, the general format is structured, which makes it easier and faster to filter and extract data than text and unstructured data.

In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

The present invention provides a universal storage format structure for high-throughput sequencing data, including four components: a header component, a sequence component, a quality score component, and a sequence information component, wherein:

The header component is used to store the header description information of the file;

The sequence component is used to store sequence information, where the sequence information is a base sequence or a file path for storing the base sequence;

The quality score component is used to store the quality score of the sequence, where the sequence quality score is a quality score string or a file path that stores the quality score;

The sequence information component is used to store records and characteristics of the sequence.

Preferably, the universal storage format structure of high-throughput sequencing data is designed based on XML and XML Schema technology.

Preferably, the header component includes the sub-element meta_info, which contains name and value attributes; the sequence component includes one or more seq sub-elements to represent the sequence, and each seq sub-element has a unique identifier. Used for the sequence information component; the quality score component contains one or more qual sub-elements to represent the sequence quality score. Each qual sub-element has a unique identifier and is used for the sequence information component; in the sequence information component Contains one or more seqinfo sub-elements, one seqinfo sub-element represents a sequence record.

The present invention also provides an editing tool based on the above-mentioned universal storage format structure of high-throughput sequencing data, for creating and editing universal storage format files for high-throughput sequencing data, and performing format conversion between NGS files and NGSGF files.

Preferably, the above editing tool is written in Java through NetBeans IDE 10.0.

Preferably, the above editing tool runs corresponding operations through GUI and command line calls.

Preferably, the formats supported by the above-mentioned editing tool for conversion include five formats: FASTA, FASTQ, SAM, VCF, and CAF.

The present invention also provides a method for constructing the universal storage format structure of the above-mentioned high-throughput sequencing data, which includes the following steps:

1) Collect existing high-throughput data formats and classify them into five types: sequence and quality score formats, alignment formats, assembly formats, mutation formats, annotation and visualization formats;

2) Analyze the specific specifications of each format to find commonalities and characteristics;

3) Design a universal storage format structure based on commonalities and characteristics of content.

Preferably, the sequence and quality score formats include Fasta/CSFASTA, Fastq/CSFASTQ, Qseq, SCARF, QUAL, 2bit/nib, and SFF formats; the alignment formats include: SAM, BAM, bowtie, and maq formats; Assembly formats include ACE, AFG, and CAF formats; mutation formats include GVF, pileup, and VCF formats; and annotation and visualization formats include BED, bigBED, Wig, bigWig, BedGraph, and GFF/GTF formats.

The present invention also provides the application of the above universal storage format structure of high-throughput sequencing data in representing, storing, editing and converting high-throughput sequencing data.

The present invention adopts a format structure based on XML and XML Schema technology, which can store many different types of current high-throughput sequencing data. The format structure specifies the structure of data storage, and the specific storage content is variable according to the sequence information, so that This format can not only store data in existing formats, but also cope with emerging data formats.

The beneficial effects of the present invention are as follows:

First of all, the universal storage format structure of the present invention uses a component structure to divide the sequence and description information into four parts, making the format structure clear and self-descriptive to facilitate future expansion.

Secondly, the universal storage format structure of the present invention introduces reference ideas into biological data formats, which is a technology widely used in computer science. In this common storage format, a link is used, and different sequence information can refer to the same sequence or quality score if the content is the same or similar. It avoids storing duplicate content.

Third, the universal storage format structure of the present invention fully utilizes the advantages of the currently popular NGS data format and can store most biological sequence information. In addition, this universal storage format structure inherits the flexibility and extensibility of XML. Due to the rapid development of NGS technology, new concepts and analysis tools are constantly emerging, and old data formats are difficult to adapt to current needs. The scalability of the universal storage format structure of the present invention overcomes the problems of specific data formats, and its flexibility can adapt to the needs of future development.

Finally, the universal storage format structure of the present invention is very readable, so it can be easily processed by computer programs, and is more readable for humans, making it easier to understand the stored content. This advantage can be attributed to the tree-structured nature of XML.

Description of drawings

Figure 1 shows 26 high-throughput data storage formats commonly used in the prior art.

Figure 2 shows the overall technical architecture of the present invention.

Figure 3 shows the overall format structure of the NGSGF of the present invention.

Figure 4 shows the format structure of the NGSGF head assembly of the present invention.

Figure 5 shows the format structure of the NGSGF sequence component of the present invention.

Figure 6 shows the format structure of the NGSGF quality score component of the present invention.

Figure 7 shows the format structure of the NGSGF sequence information component of the present invention.

Figure 8 shows a screenshot of the user interface of NGSGFEditor in Embodiment 2 of the present invention.

Figure 9 shows screenshots of two NGSGFEditor projects in Embodiment 2 of the present invention.

Figure 10 shows a screenshot of the "Step 1: Create a new NGSGF file" interface in Embodiment 2 of the present invention.

Figure 11 shows a screenshot of the "Step 2: Add Sequence" interface in Embodiment 2 of the present invention.

Figure 12 shows a screenshot of the "Step 3: Add quality score" interface in Embodiment 2 of the present invention.

Figure 13 shows a screenshot of the "Step 4: Add sequence information" interface in Embodiment 2 of the present invention.

Figure 14 shows a screenshot of the "Step 5: Save NGSGF file" interface in Embodiment 2 of the present invention.

Figure 15 shows a screenshot of the interface for converting FASTQ and NGSGF format files through the NGSGFEditor GUI in Embodiment 3 of the present invention.

Figure 16 shows a screenshot of the interface for displaying help by inputting "java-jarNGSGFEditor.jar-h" in Embodiment 3 of the present invention.

Figure 17 shows a screenshot of the interface for converting SAM and NGSGF format files using the NGSGFEditor command line in Embodiment 3 of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention and the accompanying drawings.

In order to solve the current compatibility problem of NGS data, we have developed a new XML-based universal storage format, hereinafter referred to as NGSGF, which can satisfy most NGS data types. NGSGF is based on Extensible Markup Language (XML), which is widely used in data storage on the Internet and in fields such as mathematics and biology. NGSGF is used to describe data generated by NGS technology. Different types of information used by NGS are integrated into NGSGF, such as alignment, assembly and annotation information. Because XML is highly extensible, it is easy to extend NGSGF with new features.

The present invention first studies the data storage format currently used in the field of high-throughput sequencing data. A total of 26 commonly used high-throughput data formats are collected and divided into five types: sequence and quality score format (Sequence orquality score), alignment format (Alignment), assembly format (Assembly), and mutation format (Variant) , Sequence annotation & visualization formats (Sequence annotation&visualization), wherein sequence and quality score formats can include Fasta/CSFASTA, Fastq/CSFASTQ, Qseq, SCARF, QUAL, 2bit/nib, SFF formats; alignment formats can include: SAM, BAM , bowtie, maq formats; assembly formats can include ACE, AFG, CAF formats; mutation formats can include GVF, pileup, VCF formats; annotation and visualization formats can include BED, bigBED, Wig, bigWig, BedGraph, GFF/GTF formats, such as As shown in Figure 1.

Then the specific specifications of each format are analyzed, mainly analyzing the content stored in the format and the organization form of the content. Once you understand the specific specifications of each format, look for commonalities and unique features. Design a common storage format based on commonalities and characteristics of content. As shown in Figure 3, the format NGSGF newly proposed by the present invention includes four components: header component (header_lines), sequence component (list_of_seqs), quality score component (list_of_quals), and sequence information component (list_of_seqinfos). Among them, the header component is to store A component of header description information. Most existing NGS file formats contain header information to describe the stored content. As shown in Figure 4, the sub-element meta_info is included in header_lines. Meta_info contains name and value attributes, which are used to store NGS header description information; the sequence component is a component that stores sequence information, and the sequence information is the base sequence or the file path to store the base sequence. Storing file paths enables NGSGF to store large sequence files. As shown in Figure 5, one or more seq sub-elements are included in the list_of_seqs component to represent the sequence. Each seq sub-element has a unique identifier, which is used in the list_of_seqinfos component; the quality score component is a component that stores the sequence quality score, and the sequence quality score is a quality score string or a file path that stores the quality score. The storage file path enables NGSGF to store large quality score files. As shown in Figure 6, the list_of_quals component contains one or more qual sub-elements to represent the sequence quality score. Each qual sub-element has a unique identifier and is used in the list_of_seqinfos component; the sequence information component stores sequence records and The characteristic component, as shown in Figure 7, contains one or more seqinfo sub-elements in the list_of_seqinfos component, and one seqinfo sub-element represents a sequence record. Normally, NGSGF sequence records are stored in this component. This universal storage format can store content stored in the above 26 formats.

Based on the designed structure, the present invention has also developed corresponding editing and conversion software (shown as NGSGFEditor and NGSGF Format Converter in the figure). The software can not only edit high-throughput data files based on the format structure, but also edit existing text-based files. Interconversion between high-throughput data formats and common formats designed based on XML is shown in Figure 2.

Example 1

Using the NGSGF format designed in the present invention, data in FASTA, FASTQ, SAM, CAF, and VCF formats commonly used for high-throughput sequencing can be stored.

1. Sequence format FASTA data is stored in NGSGF format.

Data from FASTA:

＞KM081703.1 Abbottina rivularis mitochondrion, complete genome↓

GCTAGTGTAGCTTAATCCAAAGCATAACACTGAAGATGTTAAGATGAGCCCTAAGAAGCTCCGCATGCAC↓

＞AF511507.1 Alligator sinensis mitochondrion, complete genome↓

CAATAAAGACTTAGTCCCGGTCTTTCTTATTAACTACCACTTAACCTATACATGCAAGCATCCACGAACCA←

Corresponding NGSGF data:

2. Sequence and quality score format FASTQ data is stored in NGSGF format.

Data from FASTQ:

@EAS54_6_R1_2_1_413_324↓

CCCTTCTTGTCTTCAGCGTTTCTCC↓

+↓

;;3;;;;;;;;;;;7;;;;;;88↓

@EAS54_6_R1_2_1_540_792↓

TTGGCAGGCCAAGGCCGATGGATCA↓

+↓

;;;;;;;;;;7;;;;--;;3;83↓

Corresponding NGSGF data:

3. Sequence alignment format SAM data is stored in NGSGF format.

SAM data:

Corresponding NGSGF data:

4. Sequence assembly format CAF data is stored in NGSGF format.

Data from CAF:

DNA: 22ak93c2.rlt↓

GTCGCnCATAAGATTACGAGATCTCGAGCTCGGTACCCTTCAAGCGATTCTCCTGCCTCA↓

↓

BaseQuality: 22ak93c2.r1t↓

4 4 8 4 4 4 4 4 4 4 4 4 6 8 17 21 14 7 6 6 6 7 7 6 8 14 16 21 15 2020↓

24 26 21 18 18 14 14 19 23 10 8 8 15 20 16 29 26 34 29 39 29 31 29 31↓

|↓

Sequence: 22ak93c2.r1t↓

Is_read↓

Padded↓

Staden_id 11↓

Clipping QUAL 39 331↓

Align_to_SCF 1 43 1 43↓

Align_to_SCF 44 317 45 318↓

Align_to_SCF 319 716 319 716↓

SCF_File 22ak93c2.r1tSCF↓

Primer Universal↓primer↓

Strand Reverse↓

Dye Dve_terminator↓

Template 22ak93c2↓

Clone bK216E10↓

Sequencing_vector″m13mp18″↓

Seq_vec SVEC 1 38″M13mp18″↓

Tag ALUS 43 180↓

Tag DONE 43 43″AUTO-EDIT：deleted C at 43(terminator,isolated,strong)″↓

Tag DONE 254 254″AUTO-EDIT: replaced T by g at 254(terminator, isolated, strong)″↓

Tag ALUS 269 402↓

Tag DONE 283 283″AUTO-EDIT：replaced T by g at 283(terminated, isolated, strong)″↓

Tag AMBG 298 302″AUTOEDIT: Check this edit cluster!″↓

Tag DONE 317 317″AUTO-EDIT：replaced C by a at 317(terminated, compound, strong)″↓

Tag DONE 318 318″AUTO-EDIT：inserted g at 318(terminated, compound, strong)″←

Corresponding NGSGF data:

5. Sequence mutation format VCF data is stored in NGSGF format.

VCF data:

##fileformat＝VCFv4.2↓

##fileDate＝20090805↓

##reference=file:///seq/references/1000GenomesPilot-NCBI36.fasta↓

##INFO＝<ID＝NS, Number＝1, Type＝Integer, Description="Number ofSamples With Data">↓

##FILTER＝<ID＝q10，Description="Quality below 10">↓

##FOREAT＝<ID＝GT, Number＝1, Type＝String, Description="Genotype">↓

#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT NA00001 NA00002 NA00003↓

20 14370 rs6054257 G A 29 PAsS NS＝3; DP＝14; AF＝0.5; DB; H2 GT：GQ：DP：HQ0|0：48：1：51，51 1|0：48：8：51，51 1 /1:43:5:.,.↓

Corresponding NGSGF data:

Example 2 Create NGSGF file

NGSGFEditor is designed for creating and editing NGSGF files. It has a user-friendly GUI and can also be run from the command line. It will be very helpful for users to operate with NGSGF files. Figure 8 shows the user interface of NGSGFEditor, showing the process of running the software (A: Start NGSGFEditor), converting the format (B: Convert SAM file), opening the file (C: Open NGSGF file), and editing the file (D: Edit NGSGF file). interface.

The NGSGFEditor in this embodiment is written in Java through NetBeans IDE 10.0, and contains two projects, as shown in Figure 9.

Here we use NGSGFEditor to create an NGSGF file in FASTQ format. The content of FASTQ is:

@EAS54_6_R1_2_1_413_324↓

CCCTTCTTGTCTTCAGCGTTTCTCC↓

+↓

;;3;;;;;;;;;;;7;;;;;;88↓

@EAS54_6_R1_2_1_540_792↓

TTGGCAGGCCAAGGCCGATGGATCA↓

+↓

;;;;;;;;;;7;;;;--;;3;83←

Step 1: Create a new NGSGF file

Click the new button to create a new NGSGF file. As shown in Figure 10.

Step 2: Add Sequence

(1) Right-click (hereinafter referred to as "right-click") the "ngs" node, and a pop-up menu will appear. Click on the "list_of_seqs" menu to create a new node. As shown in Figure 11(1).

(2) Right-click the "list_of_seqs" node to add the "seq" sub-node. As shown in Figure 11(2).

(3) Right-click the "seq" node to add the "nid" attribute. As shown in Figure 11(3).

(4) Right-click the "nid" node and select the "Edit" menu to edit the node value. As shown in Figure 11(4).

(5) Add an "origin" node similar to the "nid" node. Right-click the "origin" node, select the "Edit" menu, and enter the sequence value. As shown in Figure 11(5).

Step 3: Increase Quality Score

(1) Right-click "ngs" to add the "list_of_quals" node. As shown in Figure 12(1).

(2) Add "nid" and "origin" nodes. Right-click the "origin" node to increase the quality score. As shown in Figure 12(2).

Step 4: Add sequence information

(1) Right-click the "ngs" node to add the "list_of_seqinfos" node. As shown in Figure 13(1).

(2) Right-click "list_of_seqinfos" to the "seqinfo" node. As shown in Figure 13(2).

(3) Right-click the "seqinfo" node to the "seq" node. As shown in Figure 13(3).

(4) Right-click the "seq" node to add the "seqref" attribute. As shown in Figure 13(4).

(5) Right-click the "seqinfo" node to the "qual" node. As shown in Figure 13(5).

(6) Add "seqref" and "qualref" nodes to the "seq" and "qual" nodes. Right-click the "seqref" and "qualref" nodes and enter reference values. In this example, the sequence of the first record is "s1" and the quality score is "q1". As shown in Figure 13(6).

Add the second record of the FASTQ file just like the first record.

Step 5: Save the NGSGF file

Finally, sequences are saved in the "list_of_seqs" node, quality scores are saved in the "list_of_quals" node, and FASTQ records are saved in the "list_of_seqinfos" node. As shown in Figure 14.

Example 3 Converting NGS files and NGSGF files

Users can also use NGSGFEditor to convert between NGS files and NGSGF files.

Currently, NGSGFEditor supports five formats: FASTA, FASTQ, SAM, VCF, and CAF.

NGSGFEditor can be executed under Windows and Linux systems.

Format conversion can be invoked via the GUI and command line.

1. Via NGSGFEditor GUI

1.1 Convert FASTQ to NGSGF

(1) Use the "Add" button to add FASTQ files.

(2) Output directory Use the "Browse" button to select a folder.

(3) Click the "Start" button.

As shown in Figure 15(1).

1.2Convert NGSGF to FASTQ

(1) Use the "Add" button to add NGSGF files.

(2) Select NGSGF for input and FASTQ for output.

(3) Output directory Use the "Browse" button to select a folder.

(4) Click the "Start" button.

As shown in Figure 15(2).

2. Use NGSGFEditor command line

This example is implemented in Linux system.

Enter "java-jar NGSGFEditor.jar -h" to display help. As shown in Figure 16.

2.1 Convert SAM to NGSGF

Enter "java-jar NGSGFEditor.jar -c sam2ngsgf-iinput_path -o output_path" to convert SAM to NGSGF. As shown in Figure 17(1).

2.2 Convert NGSGF to SAM

Enter "java-jar NGSGFEditor.jar -c ngsgf2sam-iinputpath -o outputpath" to convert NGSGF to SAM. As shown in Figure 17(2).

It can be understood that, based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present invention.

Claims (3)

1. A method for constructing a universal storage format structure for high-throughput sequencing data. The universal storage format structure for high-throughput sequencing data is designed based on XML and XML Schema technology and includes the following steps: 1) Collect existing high-throughput data formats and classify them into five types: sequence and quality score formats, alignment formats, assembly formats, mutation formats, annotation and visualization formats; 2) Analyze the specific specifications of each format to find commonalities and characteristics; 3) Design a universal storage format based on common and characteristic content. The universal storage format structure of high-throughput sequencing data includes four components: header component, sequence component, quality score component, and sequence information component, where: The header component is used to store the header description information of the file. The header component contains the sub-element meta_info, which contains name and value attributes; The sequence component is used to store sequence information. The sequence information is a base sequence or a file path for storing the base sequence. The sequence component contains one or more seq sub-elements to represent the sequence. Each seq sub-element Each has a unique identifier used to locate sequence information components; The quality score component is used to store the sequence quality score. The sequence quality score is a quality score string or a file path for storing the quality score. The quality score component contains one or more qual sub-elements to represent the sequence quality score. , each qual sub-element has a unique identifier, which is used to locate the sequence information component; The sequence information component is used to store the records and characteristics of the sequence. The sequence information component contains one or more seqinfo sub-elements, and one seqinfo sub-element represents a sequence record. 2. The method for constructing a universal storage format structure of high-throughput sequencing data according to claim 1, characterized in that the sequence and quality score formats include Fasta/CSFASTA, Fastq/CSFASTQ, Qseq, SCARF, QUAL, 2bit/ nib, SFF format; the alignment format includes: SAM, BAM, bowtie, maq format; the assembly format includes ACE, AFG, CAF format; the mutation format includes GVF, pileup, VCF format; the annotation and visualization Format includes BED, bigBED, Wig, bigWig, BedGraph, GFF/GTF format. 3. Application of the universal storage format structure of high-throughput sequencing data obtained by the construction method of claim 1 or 2 in representing, storing, editing and converting high-throughput sequencing data.