10

Many times I have seen the note telling users not to update their repositories too often (i.e more than once a day) because that will put a stress too large for the servers. Also I understand that it takes monstrous machines for such repositories.

I am wondering if there is something like BitTorrent for package management? Or if there isn't, is it feasible to have such a system?

(I'm thinking about a system where each user keeps the packages that they have and serve it in the same fashion as BitTorrent.)

phunehehe
  • 20,240
  • 1
    I've never seen such a note. Also, that's the reason there are mirrors. – tshepang Jan 27 '11 at 12:29
  • @Tshepang Here is the one for Gentoo (at the bottom of the page), I also remember seeing it for another distro as well http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=2&chap=1 – phunehehe Jan 27 '11 at 14:22
  • Perhaps that's unique to src-based systems? – tshepang Jan 27 '11 at 16:42
  • man, the correct answer should be to use appropriate cache headers, only fetch if newer... and I'm pretty sure that Arch has it somewhere... it's more for rolling distro's and people like me addicted to updates – xenoterracide Jan 28 '11 at 10:09

4 Answers4

8

Indeed it's possible and of course it exists, but all of the existing projects are still just experiments.

I found two interesting papers:

So yes, it's possible and it's indeed a really interesting subject.

tshepang
  • 65,642
tmow
  • 1,255
  • Look, this is the first answer of the kind that I was expecting :) – phunehehe Jan 27 '11 at 16:38
  • @phunehehe thanks, I fixed a bit the answer. – tmow Jan 28 '11 at 08:24
  • 1
    something worth noting is that this would make international controls on crypto almost impossible...maybe why it hasn't been done - the US Govt for example considers crypto to be a "weaponized" tech. it would kind of suck to get arrested as a weapons trafficker for hosting a linux package. – RobotHumans Jan 28 '11 at 15:09
  • @ aking1012 This is a good point, but the control is feasible, it's enough to download all the packages and verify them one by one (the same with a normal distribution), the important thing is that the GNU/Linux distributor provide a list of packages with a cksum (sha*, md5, whatever), that is what happen, anyway, in normal package distributions. – tmow Jan 28 '11 at 16:34
  • I was able to find the first paper ( http://www2.cs.uh.edu/~paris/MYPAPERS/P2p08.pdf ), but does anybody know where to find the second one eleven years later? – optical Oct 03 '22 at 03:11
4

Also, if this is within a corporate setting, you can setup local mirrors and/or proxies to alleviate this problem. There is no need for a bittorrent that way. If you are using a Debian based distribution, you can already just use apt-cacher/apt-proxy to do it.

sybreon
  • 724
  • Thanks, but this is not really about me or my company. I was just thinking of it as a way to avoid having stress on the server of any distribution. Would be fantastic, wouldn't it? – phunehehe Jan 27 '11 at 14:40
1

For apt: apt-transport-debtorrent. Note that you still need to get the package list (i.e. apt-get update) from one of the mirrors so that it knows what packages are available .

syockit
  • 753
  • 2
  • 6
  • 17
1

You can try setting up a sort of apache based load balancer and specify several different mirrors as the "back" end. Each time you hit your proxy a different real mirror will be hit, thereby spreading your requests among several mirrors.

But, most mirrors only update once a day. A single mirroring job can be lengthy, especially when there's a large update. For mirror operators more than once per day usually is not worth it, and mirroring frequently puts much more stress on the upstream than a simple apt-get update.

Remember, mirrors and distributions are often done on a volunteer basis. Those volunteers have to pay big bucks for bandwidth, storage, electricity, cooling, etc.

I operate a Debian mirror myself. Please, help us out and don't slam us too hard.

bahamat
  • 39,666
  • 4
  • 75
  • 104