bigNet Track Format

A net describes which parts of one genome line up with which parts of another. It is built from chains, and it sorts them into levels. The top level holds the best chain over each part of the target genome. Inside the gaps of that chain sit the next best chains, and so on. The net is how the Genome Browser shows orthology between two assemblies, because it says which alignment is the real counterpart of a region rather than just one of many possible ones.

The bigNet format holds the same data as a net file, but compressed and indexed as a bigBed. Net files are converted to bigNet files with the program bedToBigBed, run with the -as option to pull in a special autoSql (.as) file that defines the fields of the bigNet.

The bigNet files are in an indexed binary format. The main advantage of this format is that only those portions of the file needed to display a particular region are transferred to the Genome Browser server. The bigNet file stays on your own web-accessible server (http, https or ftp), not on the UCSC server, and only the portion needed for the currently displayed chromosomal position is locally cached as a "sparse file". If you do not have access to a web-accessible server and need hosting space for your bigNet files, please see the Hosting section of the Track Hub Help documentation.

A bigNet track is used in a track hub. It is not available as a custom track.

bigNet format definition

The following autoSql definition is used to specify bigNet files. Save it as bigNet.as. It is pulled in when the bedToBigBed utility is run with the -as=bigNet.as option. The same file is in the UCSC source tree at src/hg/lib/bigNet.as.

    table bigNet
    "bigNet - a net of pairwise alignments in bigBed format"
        (
        string chrom;       "Reference sequence chromosome or scaffold"
        uint   chromStart;  "Start position in chromosome"
        uint   chromEnd;    "End position in chromosome"
        string name;        "Name of the query sequence"
        uint   score;       "Score (0-1000)"
        char[1] strand;     "+ or - for strand of the query sequence"
        uint   level;       "Depth in the net.  Odd levels are fills, even levels are gaps"
        uint   qStart;      "Start of alignment on query sequence"
        uint   qEnd;        "End of alignment on query sequence"
        uint   chainId;     "Id of the chain that fills this gap.  0 for a gap"
        uint   ali;         "Bases in gap-free alignments"
        double chainScore;  "Score from the chain"
        string type;        "Syntenic type: gap, top, syn, nonSyn or inv"
        int    qOver;       "Overlap with parent gap on query side.  -1 for undefined"
        int    qFar;        "Distance from parent gap on query side.  -1 for undefined"
        int    qDup;        "Bases with two or more copies in query.  -1 for undefined"
        int    tN;          "Unsequenced bases on target.  -1 for undefined"
        int    qN;          "Unsequenced bases on query.  -1 for undefined"
        int    tR;          "RepeatMasker bases on target.  -1 for undefined"
        int    qR;          "RepeatMasker bases on query.  -1 for undefined"
        int    tNewR;       "Lineage specific repeat bases on target.  -1 for undefined"
        int    qNewR;       "Lineage specific repeat bases on query.  -1 for undefined"
        int    tOldR;       "Bases of ancient repeats on target.  -1 for undefined"
        int    qOldR;       "Bases of ancient repeats on query.  -1 for undefined"
        int    tTrf;        "Bases of tandem repeats on target.  -1 for undefined"
        int    qTrf;        "Bases of tandem repeats on query.  -1 for undefined"
        )

One row is one fill or one gap of the net. The level field records how deep it sits. Odd levels are fills and even levels are gaps, so a top level chain has level 1, the gaps inside it have level 2, and the chains that fill those gaps have level 3. The Genome Browser rebuilds the nesting from these levels and the target coordinates, so a row must be contained by the row one level above it.

The last thirteen fields are filled in by the netClass and netSyntenic programs. If you have not run them, those fields hold -1 and the Genome Browser leaves them out of the details page.

Note that the bedToBigBed utility uses a substantial amount of memory: approximately 25% more RAM than the uncompressed BED input file.

Creating a bigNet track

A bigNet track shows only the net. To see the alignment behind a piece of the net, the browser needs the chains as well, so a bigNet track is paired with a bigChain track built from the same alignment. Build the bigChain track first, then follow these steps.

Step 1. Download the bedToBigBed and netToBigNet programs from the UCSC binary utilities directory.

Step 2. Use the fetchChromSizes script from the same directory to create a chrom.sizes file for the target assembly (e.g., hg38). Alternatively, download the chrom.sizes file for any assembly hosted at UCSC from our downloads page (click on "Full data set" for any assembly).

Step 3. Use the netToBigNet utility to turn the net file into the input for bedToBigBed:

netToBigNet hg38.mm39.net bigNet.pre

netToBigNet sorts its output, so it is ready for the next step. It stops with an error if the net is missing the fields that netClass and netSyntenic add. Pass -warn to convert such a net anyway.

Step 4. Create the bigNet file using the bedToBigBed utility:

bedToBigBed -type=bed6+20 -as=bigNet.as -tab bigNet.pre hg38.chrom.sizes bigNet.bb

Step 5. Move the newly created bigNet file (bigNet.bb) to a web-accessible http, https or ftp location.

Step 6. Add the track to the trackDb.txt file of a track hub. The type line names the query assembly and the chain track that holds the alignments:

track myChains
shortLabel Mouse Chain
longLabel Mouse (mm39) chained alignments
type bigChain mm39
bigDataUrl bigChain.bb
linkDataUrl bigChain.link.bb
otherDb mm39
visibility pack

track myNet
shortLabel Mouse Net
longLabel Mouse (mm39) alignment net
type bigNet mm39 myChains
bigDataUrl bigNet.bb
otherDb mm39
visibility full

The second word of the type line is the query assembly. The third word is the name of the chain track in the same hub. The browser follows it when you click an item in the net, so that the details page can show the alignment and link to the other assembly.

The bedToBigBed program can be run with several additional options. For a full list of the available options, type bedToBigBed (with no arguments) on the command line to display the usage message.

Sharing your data with others

If you would like to share your bigNet data track with a colleague, learn how to create a URL by looking at Example 6 on this page.

Extracting data from the bigNet format

Because the bigNet format is an extension of bigBed, the bigBedToBed program can be used to extract data from a bigNet file. This program can be downloaded from the binary utilities directory. It can be run to extract all of the data or a portion of it:

bigBedToBed http://myorg.edu/mylab/bigNet.bb -chrom=chr21 -start=0 -end=1000000 out.txt

You can also use the bigBedSummary and bigBedInfo programs from that directory to see a summary of the file or its header and field list.

Troubleshooting

If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row:

bedClip bigNet.pre chrom.sizes bigNet.clipped.pre

If the track draws but every item sits on one row, check that the level field is filled in and that each row is contained by the row one level above it. The browser draws a row at the depth its level names, and a level that has no parent is drawn at the top.