Ingestion of 2k VCFs in parallel on a cluster? about tiledb-vcf HOT 2 CLOSED

Hi @darked89, thanks for the question.

TileDB supports fully parallel writes, so you can easily split-up the ingestion across as many nodes as you like. Just launch each job with its own disjoint subset of VCF files and they’ll all be stored in the same TileDB-VCF dataset, no temporary databases or merging operations needed 🙂 .

I don't have a cluster handy but I can show how we typically run parallel ingestions using TileDB Cloud's serverless compute. You didn't mention which API you're using so I'm defaulting to Python.

First create a new array on S3

import tiledbvcf
from tiledb.cloud.compute import Delayed

array_uri = "s3://my-array"

ds = tiledbvcf.Dataset(array_uri, stats = True, verbose = True, mode = "w")
ds.create_dataset(extra_attrs = ["fmt_GT"])

then create a UDF to handle the ingestion for each batch of samples.

def ingest_vcf_files(array_uri, vcf_uris, cfg):
    print(f"Ingesting {len(vcf_uris)} starting with {vcf_uris[0]}")
    cfg = tiledbvcf.ReadConfig(tiledb_config = cfg)
    ds = tiledbvcf.Dataset(array_uri, mode = "w", verbose = True, cfg = cfg)
    ds.ingest_samples(sample_uris = vcf_uris, threads = 2, memory_budget_mb = 512)
    return vcf_uris

def return_samples(file_list):
    out = []
    [out.extend(i) for i in file_list]
    return out

Then assuming we have a list of 100 VCF files, we'll split them into 5 batches of 20

n = 20
batched_vcfs = [vcf_uris[i * n:(i + 1) * n] for i in range((len(vcf_uris) + n - 1) // n )]

and create an instance of our delayed UDF for each batch.

delayed_writes = [Delayed(ingest_vcf_files)(array_uri, b, cfg) for b in batched_uris]
ingested_samples = Delayed(return_samples, name = "Combine")(delayed_writes)

Hope that helps. Let me know if you have any follow-up questions!

from tiledb-vcf.

Dear Aaron,

That really made my day ;).
I will have to test it out using our environment but I assume it will work as described.
FYI: I have tested tiledbvcf-cli using Singularity: works out of the box.

from tiledb-vcf.