Wednesday, July 8, 2015

Can you make a BitTorrent 'channel' for just some files?

I have a problem I'd like to solve with BitTorrent -- I think. BT is great for two things: moving large files around quickly, and distributing storage capacity. Those are two things which data scientists badly, badly need. The only real alternative is for large data storage to be bankrolled by a large company acting in the public good. That happens sometimes, but there's something satisfying about the concept of a truly public infrastructure.

The downside, for me, with BT, is twofold. One is the 'negative branding' -- the association (merited or otherwise) with various kind of dubious content including piracy. The second is just providing a high quality set of data which isn't swamped by irrelevant content. You don't want to get a whole pile of television shows when you're actually trying to get some engineering-quality data.

There's a third niggle, which is how to handle realtime or streaming data.

So here's my question for the internet: how do I create a channel for scientific data using BitTorrent? I'm happy to get my hands dirty -- I'm a software dev after all.