Deluge ignores file selection

[rez]

I have reached a dead end it seems and in need of help.

Deluge 2.0.3-4 (debian 12 bookworm) and 2.1.2~dev0 (debian 13 trixie) on fresh Debian 12 both download files marked as 'Skip' resulting in inability to get only several files from huge torrent packs which I don't have space for.

I have read about full disk allocation and use it for many years. Screenshots attached.
I have checked configs to make sure it is there.

Code: Select all

    "plugins_location": "/opt/torrents/.config/deluge/plugins",
    "pre_allocate_storage": true,
    "prioritize_first_last_pieces": true,

FS is ext4, permissions are correct, no ACL in use, it downloads fine, except it downloads everything.
On the screenshots total size is 468Gb, only 1 file sized 4.5Gb is selected and progress bars for unselected folders show that they start downloading when they shouldn't.

What I may be missing?
It worked on Debian 11 (2.0.3-3.1) and I really don't want to downgrade on new system, but I see that everything works on previous install, Debian 11, and not on Debian 12.

[ambipro]

I assume that only a very small amount of these unselected folders ever get downloaded. If so, this has to do with the way pieces in torrents are organized. Your client wants a piece of whichever files you have selected, and the other folders you have not selected are contained within the pieces necessary for that file to be downloaded.

There is no way for you to essentially get the full amount of the files without in some circumstances potentially downloading a piece of the files you have not selected. That's just how pieces work. You can go to the details tab on the torrent itself and see the number of pieces and the size of the pieces.

[rez]

I assume that only a very small amount of these unselected folders ever get downloaded. If so, this has to do with the way pieces in torrents are organized. Your client wants a piece of whichever files you have selected, and the other folders you have not selected are contained within the pieces necessary for that file to be downloaded.

If only that was the case!
It downloads many gigabytes of different files and does this until it runs out of space.
So that's not the case. Several hundreds of Kbs are OK due to how bittorrent works, but in my case Deluge ignores 'Skip' altogether.

This is not happening on Debian 11 setup.

Some additional info:
I see that large parts file that grows altogether during download. For the experiment I have continued to download this torrent on both deluge 2.0.3-3.1 (debian 11) and 2.1.2~dev0 (debian 12 with trixie package installed, but I can test on 2.0.3-4 too)

2.0.3-3.1:

Code: Select all

rez@pc00:~/_hdd/_torrents/serial$ ls -lah
total 18M
drwxr-x--- 3 rez rez 4,0K Dec  3 03:39 .
drwxr-xr-x 8 rez rez 4,0K Aug  8  2022 ..
-rw-r--r-- 1 rez rez  21M Dec  2 16:00 .34d546caa663bc19728d0213fd3c726feb9ed368.parts
drwxr-xr-x 5 rez rez 4,0K Dec  1 17:05 Smallville.1080p.AMZN.WEB-DL.Rus.Ukr.Eng_TeamHD

2.1.2~dev0:

Code: Select all

root@xenia:/opt/torrents/files/video/serial# ls -lah
total 2,1G
drwxrwx--- 3 deluge deluge 4,0K Dec  3 01:46 .
drwxrwx--- 6 deluge deluge 4,0K May  8  2023 ..
-rw-rw---- 1 deluge deluge 2,1G Dec  3 03:41 .34d546caa663bc19728d0213fd3c726feb9ed368.parts
drwxrwx--- 3 deluge deluge 4,0K Dec  3 01:46 Smallville.1080p.AMZN.WEB-DL.Rus.Ukr.Eng_TeamHD

I really am at a loss here.
I have used deluge for more than 10 years now and this is a first.

[rez]

I have downgraded to 2.0.3-3.1 in debian 12 to rule out machine/fs/setup issues.
And it works as expected, downloading only 1 requested file (see screenshot). So this is something that was changed afterwards.
I readd torrent file each time I test (just in case).

I hope that helps Cas and team to check what was changed in between 2.0.3-3.1 and 2.0.3-4.

[ambipro]

We are on 2.1.1 as stable now. I'd recommend you upgrade to the latest stable version at least. This issue hasn't been reported on 2.1.1, and I'm not sure I've seen it reported on your versions, but I'd try to install the latest version.

Debian's repo's are out of date.

[rez]

We are on 2.1.1 as stable now. I'd recommend you upgrade to the latest stable version at least. This issue hasn't been reported on 2.1.1, and I'm not sure I've seen it reported on your versions, but I'd try to install the latest version.

That's what I did the first time: upgraded to 2.1.2~dev0 with no luck, alas =(
For now I'll continue testing because that's just strange that file_priorities is ignored.
Maybe the culprit is deluge-web which I use to add torrents, idk.

[rez]

Yes, looks like deluge-web is at fault here.
I have installed 2.1.2~dev0 back just to be on track. Issue is the same. However!
If I select no files in deluge-web at torrent add stage, it still shows full size for all files in 'Size' column and proceeds accordingly. But after setting priorities to 1 file to any other, then back to 'Skip', then again, it selects only that 1 file correctly. All folder bulk actions are ignored, so that won't work. I have to manually set priority for every file I want 1-2 times, only then it is flagged correctly.
I have no idea why, but this way it works on 2.1.2~dev0.
Still, that wasn't an issue on 2.0.3-3.1.

Another issue: after file has been select for downloading there's no turning back. It is not possible to set priority to 'Skip' unless I remove torrent and add it back.
Edit: no, it's possible, but each time I have to force recheck to be able to skip. That is... not convenient.

[mhertz]

I looked at the changelog of the debian packages 2.0.3-3.1 and 2.0.3-4, and nothing that should do such imho, distro-specific patching I mean, as both is deluge 2.0.3 obviously, but anyway, I suspected libtorrent being culprit myself, and these two debian packages are differentiated by deps to libtorrent 2.0.8(2.0.3-4) vs 1.2.9(2.0.3-3.1) in the releases, and the latter 2.1.2~dev0 is libtorrent 2.0.9 btw.

I checked the deluge commit-log for webUI's filestab.js, and only very slight related changes in recent years time roughly, and as said 2.0.3 is same version obviously(for the working vs non-working debian package), so nothing to check for there regarding that, and only debian changelog merely, as I had checked before.

Last, I started by testing this on my used distro, using libtorrent 2.0.9 and deluge 2.1.1, and could reproduce your exact issue(s), _when_ having "prioritize first and last pieces" enabled, as I see you had from your screenshoot. Disabling that, makes the issues you describe go away here. Haven't gone further than that, atleast yet, and haven't neither looked into this option in-depth.

[rez]

Last, I started by testing this on my used distro, using libtorrent 2.0.9 and deluge 2.1.1, and could reproduce your exact issue(s), _when_ having "prioritize first and last pieces" enabled, as I see you had from your screenshoot. Disabling that, makes the issues you describe go away here. Haven't gone further than that, atleast yet, and haven't neither looked into this option in-depth.

Thank you! I will test with this option disabled, it usually helped preallocation in 1.x versions and I use it out of habit.
Meanwhile I have tested deluge-console and it sets priorities without this issue in 2.1.2~dev0.

[mhertz]

Hmm, that sounds like I'm wrong then(as it respects that option too), apologies if that's the case. Console-UI was BTW fixed in v2.0.4 for skip priority actually being skip instead of low. I look into it again latter today/evening and post back. First I couldn't reproduce(yesterday), and then tried your settings and then could, so suspected it was about the full allocation option, but disabling that, still could reproduce, so narrowed it down to that other option before mentioned, in two extra on/off testings each, but your report just sounds like possibly unrelated. BTW, that 2.1.2-dev0 I'm pretty sure just is 2.1.1, like mine, and just an issue happening at times during the building process, with naming screwup. Anyway, will find out in-depth, later.

Edit: You're right, in console-UI that option isn't hindering functionality, whereas it does in GTK-UI and web-UI. I'll investigate this evening, or tomorrow and find out why exactly.

Edit2: The reason for former, I found was something I also actually had reported years ago, but apparently forgot. The rewritten deluge-console in deluge2 doesn't apply/save settings in the curses preferences pane to session correctly(i'll see if can find culprit for this, and add to report, later), so when I tested before, I had the prioritize setting disabled, and it worked fine, and then when enabled it and retried, then again worked fine, though as said, I forgot that the option effectively wasn't even enabled in libtorrent(reentering preferences pane looks like it's set, but if restarting console-UI and then rechecking, then will see it's really not). However that would indicate that when you tested it previously yourself in deluge-console and found working, then would have had the option disabled too, before starting deluge-console, or disabling it through cli/non-interactive mode, and not through the curses pane(it's easy to see if option enabled or not, from checking for existence of hidden dot "parts" file, next to download payload, as can only check core.conf after deamon stopped usually, where settings saved from memory to disc). Anyway, I highly suspect then, that the different libtorrent versions show different behaviour regarding this option, or more specifically libtorrent v1.2.x vs v2.x branches presumably. Hope helps.

Edit3: Last, indeed no issue anymore with that option when changing to libtorrent 1.2.x, where I tested latest v1.2.19 on deluge 2.1.1, without issue. Btw, it's seemingly deluge itself which in 'core.torrent.py' defines upped piece-priorities for first/last pieces and applies said generated defenition list through a generic prioritize_pieces libtorrent option, so either this is an issue with said generic prioritize_pieces libtorrent 2.x option, or instead a deluge issue of generating the arg correctly(list of changed piece priorities for torrent) to that option for libtorrent 2.x.