Page 1 of 1

Too Many Pieces Error in Large (>1TB) Torrents?

Posted: Sun Apr 01, 2012 5:31 am
by DarkSniper
Hi, I'm attempting to download part of a 1TB+ torrent which has nearly 300k pieces in it. However, Deluge refuses to download it, putting the torrent in the "Error" category, stating "too many pieces in torrent:" as the status.
I'm wondering what the current limit is, and if it is possible to (safely) raise it by editing the something in the source code and recompiling deluge and/or libtorrent.
Thanks!

Environment:
OS: Arch Linux
Deluge Version: 1.3.4
libtorrent Version: 0.15.10
Interface: GTKUI/WebUI connecting to Deluged on headless server
Filesystem: XFS
Installation Method: Package
Plugins: AutoAdd, Batch Renamer, Blocklist, DeletePartials, Label, Notifications, Toggle, TotalTraffic
Link to Problematic Torrent: https://sites.google.com/site/tlmcfiles ... edirects=0

Re: Too Many Pieces Error in Large (>1TB) Torrents?

Posted: Sun Apr 01, 2012 5:06 pm
by johnnyg
Hi,

the problem is related to libtorrent not deluge, so no need to recompile deluge.

In particular, the maximum number of pieces in libtorrent is set to ~262k to reduce memory usage.
I've attached a patch which will increase this limit to roughly 1 million pieces but bare in mind this will increase libtorrent's memory usage (not sure by how much).

A better alternative would be to contact whoever created that torrent and make sure they use piece sizes >= 8MiB for torrents >= 1TiB.

Re: Too Many Pieces Error in Large (>1TB) Torrents?

Posted: Mon Apr 02, 2012 1:56 am
by DarkSniper
Ah, Okay. Thanks for the patch, it did exactly what I needed.
The box I'm running deluge on usually has plenty of free memory, so a little extra usage is fine. So far it seems to be using about the same amount of memory, but slightly more CPU, though this isn't an issue at this point.
I'll suggest it to them, and see if they'd be willing to switch for the next release.

Re: Too Many Pieces Error in Large (>1TB) Torrents?

Posted: Fri Apr 19, 2013 6:47 pm
by felly117
Hi

I don't want to unnecessarily bump old topics but I'm having the exact same issue as OP was.

However I have no idea on how to go about patching libtorrent. Can someone please help me do this.

Any help is much appreciated

Thanks.

Re: Too Many Pieces Error in Large (>1TB) Torrents?

Posted: Fri Apr 19, 2013 9:45 pm
by Cas
If you use libtorrent 0.16 then this should be enabled by default.

Re: Too Many Pieces Error in Large (>1TB) Torrents?

Posted: Sat Apr 20, 2013 5:40 am
by felly117
I'm using Deluge on Raspbian which I believe uses libtorrent 0.14.9.

Is there a way of updating this?

Thanks

Re: Too Many Pieces Error in Large (>1TB) Torrents?

Posted: Sat Apr 20, 2013 11:55 am
by Cas
There is always a way just might not be simple for Raspbian as it will require building from source. Note that if you are upgrading to libtorrent 0.16 you should also upgrade Deluge to 1.3.6.