Take the 2-minute tour ×

I am doing some distributed work with RackSpace cloud servers and I am using bittorrent to distribute my files. It works surprisingly well. However, distributing the torrent files themselves are not so nice. How would you go around doing that? Right now I just scp the torrent files to the servers, and of course I could write a script that copies it to sqrt(n) servers instructing each to again copy to sqrt(n) but that's a pita to work it.

share|improve this question
add comment

4 Answers

up vote 23 down vote accepted

Not knowing what exactly your problem is, I can recommend pscp from parallel-ssh as a tool to upload small files to multiple servers.

You prepare a list of servers to upload to and let it know what to take locally and where to put it remotely. For example:

$ pscp -h list-of-servers file.torrent /tmp/
[1] 02:11:22 [SUCCESS]
[2] 02:11:22 [SUCCESS]
[3] 02:11:22 [SUCCESS]
[4] 02:11:22 [SUCCESS]
[5] 02:11:22 [SUCCESS]
[6] 02:11:22 [SUCCESS]
[7] 02:11:25 [FAILURE] Exited with error code 1
share|improve this answer
There's also code.google.com/p/pdsh –  chx Oct 31 '12 at 20:03
add comment

Given they're very small files typically couldn't you just have a 'dropbox' using HTTP with a cron'ed script doing a curl/wget with wildcard?

share|improve this answer
This is the method I would use, no need to push a file out to each server and maintain a server list, have them periodically check for new .torrent file –  jwbensley Oct 31 '12 at 17:41
you could also use Puppet to do this –  Noah Yetter Nov 1 '12 at 5:20
add comment

You could have the torrent clients check an rss feed for new torrent files or switch to magnet links instead of torrent files.

share|improve this answer
RSS may not be an option if he/she is using a basic command line torrent client. –  Hengjie Oct 31 '12 at 22:44
He could you Flexget with rtorrent without gui –  chewbakka Nov 1 '12 at 11:07
Yep, flexget and rtorrent would work well for this –  Grant Nov 2 '12 at 13:09
+1 "switch to magnet links instead of torrent files." –  michael_n Jan 11 at 12:14
add comment

You might want to look into murder

Murder is a method of using Bittorrent to distribute files to a large amount of servers within a production environment. This allows for scaleable and fast deploys in environments of hundreds to tens of thousands of servers where centralized distribution systems wouldn't otherwise function


Murder was/is developed by the folks at twitter, and they're using it daily to distribute files.

share|improve this answer
add comment

Your Answer


