Libtorrent 2.0.8 update.

Specific support for Deluge on Microsoft Windows OS
mhertz
Moderator
Moderator
Posts: 2195
Joined: Wed Jan 22, 2014 5:05 am
Location: Denmark

Libtorrent 2.0.8 update.

Post by mhertz »

Here's latest libtorrent 2.0.8 for x64 Windows for anybody wanting update manually:

https://paste.c-net.org/WilburTeller

Replace the original file with this, under '‰programfiles%\deluge'.

This isnt on pypi because issues with building the 32 bit version seemingly, hence me posting this. I extracted the working 64 bit version from there automated buildsystem on github(which is used to publish on pypi normally).

I'm not gonna bother build the failed 32 bit version myself honestly(think just needs downgrade boost to build) - It's almost 2023 afterall, meaning we've had x64 supported CPU's available for 20 years by now already ...

Last, good people here, highvoltage and ambipro, ran tests and reported high memory usage from this. I then also myself tested it on a family members win11 PC showing same high memory usage on 2.0.8 and 2.0.7, but not original included 2.0.6, and on my Win10 VM showing happening on 2.0.7, 2.0.8 and 2.0.6. It's expected that libtorrent 2.x uses more memory than 1.2.x but some report windows not freeing the cached memory again fast enough, leading to performance issues(I didn't test if that was an issue for me or not though). Qbittorrent added a workaround where they restrict the working set memory usage outside of libtorrent, hence if comparing with that, then need know this. If affected, then the OS cache can be disabled for libtorrent with ltconfig plugin if needed, or one can revert to 1.2.x instead if preferred.
User avatar
ambipro
Moderator
Moderator
Posts: 418
Joined: Thu May 19, 2022 3:33 am
Contact:

Re: Libtorrent 2.0.8 update.

Post by ambipro »

To clarify mhertz high memory usage that I had reported when testing this, deluge's process itself reports normal usage, but windows memory usage reports a giant spike during downloads of any time period (downloads lasting more than say, a minute) gradually from my normal of around ~30% memory usage about 1% every 10 seconds all the way to the low 90%'s total usage, but no process would show this usage. mhertz and I could only guess/conclude that this was something to do with libtorrent's memory mapping or cache for the files.

After the downloads concluded, the memory would almost immediately be freed and memory usage would decrease from 90%+ usage back to ~30% (normal)...this happened every time.

Side note: I have 16GB memory, so it was taking a solid 8-10GB of memory to download even a single torrent. Currently I'm not having any issues on 2.0.7 like this, despite mhertz reporting that it was also happening to him and my .pyd being from him as well.

I'm sticking with 2.0.7. You can find links to 2.0.7 I'm using as well as 1.2.17 @ my post viewtopic.php?f=12&t=56260&p=235266#p235266

As always thank you mhertz, we love you.
mhertz
Moderator
Moderator
Posts: 2195
Joined: Wed Jan 22, 2014 5:05 am
Location: Denmark

Re: Libtorrent 2.0.8 update.

Post by mhertz »

I'm sincerely touched by your kind words my friend, appreciate you! :) Thanks also for clarifying :)
mhertz
Moderator
Moderator
Posts: 2195
Joined: Wed Jan 22, 2014 5:05 am
Location: Denmark

Re: Libtorrent 2.0.8 update.

Post by mhertz »

Latest libtorrent 1.2.x(v1.2.18) wasen't published to pypi either for same reason, so likewise for that:

https://paste.c-net.org/TaintedMinister

(Btw, sorry for previous long-time "pypa" wrong-spellings, as I just noticed.)
Paralel
Member
Member
Posts: 23
Joined: Mon Nov 04, 2019 6:53 am

Re: Libtorrent 2.0.8 update.

Post by Paralel »

Is this memory behavior known to the people who maintain LibTorrent? Are they planning on releasing a 2.0.9 to correct it, or is this just abnormal behavior in certain circumstances?

Thanks again for all your hard work, mhertz. Without you and a select few other people, Deluge would have surely died a rather ignominious death. Which, given the torrent clients out there now, would have been a true tragedy.
mhertz
Moderator
Moderator
Posts: 2195
Joined: Wed Jan 22, 2014 5:05 am
Location: Denmark

Re: Libtorrent 2.0.8 update.

Post by mhertz »

You're always so appreciative and supportive Paralel, thanks alot my friend for saying that, despite being way to kind it nonetheless made me happy to read, cheers buddy most welcome! :)

This is something that is being worked on or looked into yes, but it's not fully sure to lead-dev Arvid what the issue is I believe. When people state they have all there memory used, then he asks what is the issue specifically with that? The implied question is regarding libtorrent now lets the windows kernel or it's page-cache component or whatever called, do the caching itself, so uses often as much free and unused memory as it can, but people are just not used to seeing such, because libtorent didn't use to do such before(on 1.2.x where libtorrent itself allocated such cache and not OS). Then for the people who state that the memory is not freed again whenever other processes need it, then that of-course is an issue, but I don't think Arvid knows why happens, and haven't reproduced it yet if remember correct. However he have added something like on windows continually releases dirty pages at x amount of time I believe, and a write-through disk_io write-cache option or something which I believe is default now, also helping clearing cache faster than before, and then continues to think about it I think. I though also only saw the cache(memory) clearing happening at download finished, like ambipro reported, though I only tested it in short time honestly. Personally my question is maybe more regarding why some libtorrent 2.x doesn't do this in certain conditions/setups? Or why the OS doesn't rather. I here mean after reading Arvid's explenation of what memory-mapped files entails. Plus of-course why not freed when that happens to people. Sorry cannot give you better answer then that i'm affraid. Also the OS cache can be disabled for libtorrent with ltconfig as said, but agreed it would be cool if having an option for using OS cache but only in conservative amounts, like ambipro reports on his setup with 2.0.7.

More info: https://github.com/arvidn/libtorrent/issues/6667
Paralel
Member
Member
Posts: 23
Joined: Mon Nov 04, 2019 6:53 am

Re: Libtorrent 2.0.8 update.

Post by Paralel »

From reading through the entire thread on GitHub concerning this problem, there seems to be very little progress on fixing this issue, as it has existed since 2.0.5 and no solution appears to be imminent.

In order to handle this problem qbittorrent went ahead and implemented a feature in their software to correct the unusual behavior until the Libtorrent developers can come to an agreement on a fix. You can find the fix qbittorent used here:

https://github.com/qbittorrent/qBittorrent/pull/16485

For Deluge, I suggest that we consider implementing a similar fix otherwise this is apparently going to be a continuing problem.
User avatar
ambipro
Moderator
Moderator
Posts: 418
Joined: Thu May 19, 2022 3:33 am
Contact:

Re: Libtorrent 2.0.8 update.

Post by ambipro »

Paralel wrote: Sat Feb 11, 2023 2:23 am From reading through the entire thread on GitHub concerning this problem, there seems to be very little progress on fixing this issue, as it has existed since 2.0.5 and no solution appears to be imminent.

In order to handle this problem qbittorrent went ahead and implemented a feature in their software to correct the unusual behavior until the Libtorrent developers can come to an agreement on a fix. You can find the fix qbittorent used here:

https://github.com/qbittorrent/qBittorrent/pull/16485

For Deluge, I suggest that we consider implementing a similar fix otherwise this is apparently going to be a continuing problem.
See viewtopic.php?f=9&t=56387 for a plugin that accomplishes this until perhaps Cas implements something similar natively.
mhertz
Moderator
Moderator
Posts: 2195
Joined: Wed Jan 22, 2014 5:05 am
Location: Denmark

Re: Libtorrent 2.0.8 update.

Post by mhertz »

Thanks for the reference ambipro.

@all, I forgot to correct something I said in this thread previously. I said you could disable OS cache with ltconfig if having issues, which isn't possible sorry. Also I said I couldn't see the issue on the officially provided libtorrent 2.0.6 when on win11(but could on win10), but I retested it some time ago and couldn't reproduce(so now happened on win11 also), so the page-caching of windows kernel is somewhat bipolar seemingly, well that or me lol :lol:

Last, I was somewhat abbrasive(here/elsewhere) about this often wasn't a real problem and all memory used only was "scary" for many not understanding would be cleared again upon request of other processes, but it is a big issue in fact. Good quote by Arvid lead-libtorrent dev here:
My understanding of the main issue people are reporting fall into a few different buckets:

The disk read cache balloons and infringes more important pages
The dirty pages aren't flushed in a timely manner and infringe on more important pages.
Checking files, i.e. sequentially reading all files, isn't as fast.
Downloading to a remote mount is slow and inefficient.
I think there may be another bucket that has emerged recently where just having a lot of file mappings in a process might be expensive, so torrents with many small files are inefficient.

It's not so easy to measure and quantify these problems.
That the main deluge version is libtorrent 1.2.x is probably a good idea still imho.

Edit: I forgot to also say that I learned the 32 bit version of libtorrent 2.x isn't using memory mapped files, because too limited in its virtual address space beeing 32 bit only. Instead features a less effecient implementation of the 1.2.x approach, but is single threaded(currently) and no disk-cache. This style(pread/pwrite) can also be enabled if wanted, on the 64 bit libtorrent BTW, but it still has the "memory filling up" issue reportedly, surprisingly, so that issue in and off itself to me sounds like being because of having no own disk-cache and relying on kernel of OS for caching, coupled with high-throughput as often seen with torrenting.
Paralel
Member
Member
Posts: 23
Joined: Mon Nov 04, 2019 6:53 am

Re: Libtorrent 2.0.8 update.

Post by Paralel »

There is also an issue the lead developer missed in his summary that impacts Windows but not Linux. Linux allows for memory overcommitment meaning that a program can request any amount of memory, above and beyond physical memory and the limits of what is possible with the swap file on a given medium for storage. Someone in the discussion thread demonstrated this when Libtorrent 2.0.8 had requested 9TB, and it was allowed. Linux has no problem with this situation unless one actually uses the memory then it will OoM when the page file fills up the medium it is stored on. Windows does not allow for overcommitment. If a process asks for more memory than is present as physical memory and the limit of the swap file you will immediately get an OoM error. So, this behavior in Linux isn't necessarily an issue, but with Windows it is far more likely to trigger an OoM error sooner rather than later.

Also, thanks for the link @ambipro, I was unaware that a plugin solution has been implemented. Also thank you for the plugin @mhertz, this should make all the difference.
Post Reply