Page 54 of 66

Re: [Unofficial] Deluge 2.0.x installer

Posted: Tue Jun 09, 2020 2:53 am
by petersasi
Hi Folks,

I just wanted to give you an update in future directions:
  1. Now we have deluge2 built with the latest Boost, Libtorrent, OpenSSL, Python, GTK there is not much else to do, I just keep watching the projects
  2. Libtorrent 1.2.7 was released 9 days ago, I am not expecting a new minor version for approx a month
  3. Libtorrent 2.0 Release Candidate was released afterwards, but I have doubts if Deluge will work with a new major version out of the box. This one signifies support for BitTorrent V2 which itself is largely a preemptive reaction to SHA1 being broken; large portions of BitTorrent are designed around the 20-byte length of a SHA1 checksum. They've decided to move forward with SHA256 truncated to 20 bytes to avoid incompatibilities with existing infrastructure such as the Mainline DHT. It would offer some other nice benefits, not sure if Deluge wants to exploit those. It is overhauled to use memory mapped files, hopefully using boost iostreams, so it should work on windows too. However: this one will really depend on Deluge compatibility.
  4. Boost 1.73 was released in April and the seem to release minor versions every 4 months so next one probably comes in August
  5. OpenSSL is also mainly working on a new major version 3.0 although a 1.1.1h may come with a new vulnerability exposed
  6. New python minor version is not expected till mid-July
  7. About: GTK: the 3.24 branch is used** where 3.24.20 was released just a month ago
  8. I keep familiarizing myself with the internals of the build process and keep finding stuff I need to ask MHerz about and update*
  9. I do have some more code to prettify and maybe more automation along just keeping python installed if it is the latest version, and automated version checking, and the mentioned size reduction - but these are all not very high priority, so I am not overly motivated (read lazy) :lol:
  10. TL;DR: let me know if you think something is important as I see nothing urgent and may end up not doing much as long as the current latest build works for everyone. ;)
*MHerz: can I ask a question about your patch to the wingtk builder? The build process is so long I might just have missed this css patch not succeeding:
In C:\deluge2\gvsbuild-build\gtk3.patch you try to patch /patches/gtk3-24/win32.patch to add some background color changes, unfortunately this patch is no longer there in the repo: ... es/gtk3-24 only has one gtk_update_icon_cache.patch and nothing else. This win32.patch used to patch /gtk/theme/win32/gtk-win32-base.css from the original GTK repo. Maybe they just upstreamed their win32 patch? Or moved it?
** MHerz: do I assume correctly there is no chance Deluge would work with a newer GTK version (e.g. 3.98.5) and wingtk can only build 3.24.x, right?

Re: [Unofficial] Deluge 2.0.x installer

Posted: Wed Jun 10, 2020 3:55 am
by mhertz
Clearly the right man for the job there :D

petersasi wrote:MHerz: can I ask a question about your patch to the wingtk builder?

You can ask me as many questions you'd like mate :)

The build process is so long I might just have missed this css patch not succeeding:
In C:\deluge2\gvsbuild-build\gtk3.patch you try to patch /patches/gtk3-24/win32.patch to add some background color changes, unfortunately this patch is no longer there in the repo: ... es/gtk3-24 only has one gtk_update_icon_cache.patch and nothing else. This win32.patch used to patch /gtk/theme/win32/gtk-win32-base.css from the original GTK repo. Maybe they just upstreamed their win32 patch? Or moved it?

The patch I made(gtk3.patch), when applied, generates a new patch file(win32.patch, see diff man page's '-N' switch) which gets applied later by gvsbuild itself, as per the addition(s) to

** MHerz: do I assume correctly there is no chance Deluge would work with a newer GTK version (e.g. 3.98.5) and wingtk can only build 3.24.x, right?

Hmm, unstable gtk4 i.e gtk-3.9x... Sorry, I never looked into that, and honestly don't know, as didn't deem relevant, as wanted at max(min?) a RC branch in terms of "stableness" of components used. About gvsbuild on this, then there's an open issue on it still, somewhat old though(2017), and a not-finished work-in-progress gtk4 branch available:

Ask away if elaboration is needed on any of this or something else.

Ohh, unrelated, but about the previously talked about size-reduction, then when I did it, then I just moved something away from install-folder and checked if worked afterwards when started/used, so pure trial and error, and also a script or app showing biggest files under install-folder, so as to get most bang for your buck, but also removed folders with smaller files, as sounded from folder name or content, like something not used by deluge etc(tested it too of-course), and then afterwards get the paths of moved away items for incorporating into build-script as cleanup.

Anyway, a quick thought and probably pretty stupid lol, but what if simply starting deluge and then deleting the deluge install-folder and select "ignore-all" or whatever called on windows, to the locked files(in use) prompts coming :D I mean to get all unallocated files away quickest possible, and then afterwards use an app or script to compare cleaned vs uncleaned install-folders to get all the differences for adding into the cleanup section of buildscript. Sounds little crazy though and maybe issues with it, but just a quick random thought lol :D

There should be about 20% more to be possible to delete, as was mentioned once by doadin in the #3201 ticket, as per when he compared filesize of mine VS his personal build, and his build used the freezing which he had mostly got working but still some issue(s) left he had trouble fixing, and the freezing did make it amongst else possible to remove unused files. BTW, now on closer thought, why not simply PM him and kindly ask for him running 'dir /b PATH >list.txt' on his build and send you for complete folder/file listing of used files to go by. If/when thinking about that, later, of-course :) Back when it was 500+ mb(mostly gtk3 of-course), originally, I cared about reducing it and so did, but also didn't dwell much into it later as too thought "good enough" atleast for now.

Re: [Unofficial] Deluge 2.0.x installer

Posted: Mon Jun 15, 2020 5:49 am
by petersasi
GTK3: oh, so the latest stable release is indeed 3.24.20 - we are good then, thanks.
Patch creating a patch with is then applied by another patch: thanks for the explanation, got it now. :idea:
About the cleanup: yes, the simple delete-all-you-can approach could work as well as long as all necessary files are kept open / locked and not closed after use.
However I just did a quick ProcessMonitor filtered on only FileSystem Operations, only deluge.exe & deluged.exe & python.exe & pythonw.exe:
deluge.exe only touches these paths:
  1. C:\Program Files\Deluge
  2. C:\Program Files\Deluge\deluge.exe
  3. C:\Program Files\Deluge\pythonw.exe
  4. C:\Program Files\SystemResources\deluge.exe.mun
  5. C:\Windows\apppatch\sysmain.sdb
  6. C:\Windows\Globalization\Sorting\SortDefault.nls
  7. C:\Windows\System32\imm32.dll
deluge.exe only touches these paths:
  1. C:\Program Files\Deluge\deluged.exe
  2. C:\Program Files\Deluge\Lib
  3. C:\Program Files\Deluge\pythonw.exe
  4. C:\Program Files\SystemResources\deluged.exe.mun
  5. C:\Windows\apppatch\sysmain.sdb
  6. C:\Windows\Globalization\Sorting\SortDefault.nls
  7. C:\Windows\System32\imm32.dll
pythonw.exe for DelugeD(aemon) operates on 1569 paths, of that 1495 has "C:\Program Files" in it, and 1499 in "c:\program files\Deluge"
pythonw.exe for Deluge (thinclient) operates on 4131 paths, of that 3884 has "C:\Program Files" in it, and 3880 in "c:\program files\Deluge"
There are 13644 files in C:\Program Files\Deluge out of which ever 4068 are ever used by Deluge and DelugeD.

Nota bene: there are lots of files like:
  • C:\Program Files\Deluge\Lib\site-packages\zope\interface\__pycache__\interfaces.cpython-38.pyc <--- both processes have 8 operations on it
  • C:\Program Files\Deluge\Lib\site-packages\zope\interface\__pycache__\interfaces.cpython-38.pyc.1540571079424 <--- only one process one IRP_MJ_CREATE on it.
  • C:\Program Files\Deluge\Lib\site-packages\zope\interface\__pycache__\interfaces.cpython-38.pyc.1938603351760 <--- only one IRP_MJ_CREATE on it by the other process.
These also disappear after the processes have exited.

Excluding these "temp" files 3760 are ever used.

I have an Excel with a pivot about this, will compare to he actual directory contents soon, I should be able to delete close to 10k files after that. :)

EDIT: Example of files never used are
  1. the folders "test", "tests" and and "unittest" - basically any path that has "test" in it - 1437 of these.
  2. 279 paths starting with C:\Program Files\Deluge\Lib\site-packages\win32\Demos
  3. 124 paths on C:\Program Files\Deluge\data\lib\gobject-introspection\giscanner
  4. 950 non-English locale files <---- don't think we can remove these
  5. 5932 paths in C:\Program Files\Deluge\data\share\icons and C:\Program Files\Deluge\Lib\site-packages\deluge\ui\web\icons <---- don't think we can remove these

Re: [Unofficial] Deluge 2.0.x installer

Posted: Mon Jun 15, 2020 8:38 am
by mhertz
Wow, really impressed with that report mate :) No need giving you hints I see, lol :D Good job as usual :)

Re: [Unofficial] Deluge 2.0.x installer

Posted: Mon Jun 15, 2020 3:29 pm
by mystikfox
Thanks for putting in the work to make our lives easier!

I was doing some testing before upgrading and while everything else seems to be working fine, I did notice one issue. I'm sure the deluge-console isn't really used by 99% of the crowd, but I do rely on it quite a bit for automating certain tasks. It seems that executing deluge-console fails due to missing Curses library:

Code: Select all

C:\Program Files\Deluge>deluge-console
Traceback (most recent call last):
  File "C:\Program Files\Deluge\lib\", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Deluge\lib\", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Deluge\deluge-console.exe\", line 7, in <module>
  File "C:\Program Files\Deluge\lib\site-packages\deluge\ui\console\", line 19, in start
  File "C:\Program Files\Deluge\lib\site-packages\deluge\ui\console\", line 154, in start
    from deluge.ui.console.main import ConsoleUI  # import here because (see top)
  File "C:\Program Files\Deluge\lib\site-packages\deluge\ui\console\", line 31, in <module>
    from deluge.ui.console.modes.preferences import Preferences
  File "C:\Program Files\Deluge\lib\site-packages\deluge\ui\console\modes\preferences\", line 3, in <module>
    from deluge.ui.console.modes.preferences.preferences import Preferences
  File "C:\Program Files\Deluge\lib\site-packages\deluge\ui\console\modes\preferences\", line 33, in <module>
    from deluge.ui.console.widgets.sidebar import Sidebar
  File "C:\Program Files\Deluge\lib\site-packages\deluge\ui\console\widgets\", line 12, in <module>
    import curses
  File "C:\Program Files\Deluge\lib\curses\", line 13, in <module>
    from _curses import *
ModuleNotFoundError: No module named '_curses'

From my understanding, a modern Ncurses package is not available on Windows for use with a version of Python that is compatible with Deluge 2, but I'll leave that up to you guys with more experience and knowledge on that than myself.

Re: [Unofficial] Deluge 2.0.x installer

Posted: Mon Jun 15, 2020 9:50 pm
by mhertz
That is indeed specifically the issue unfortunetly.

I researched it quite a bit, little before stopping, but didn't find a solution yet, unfortunetly, at the time.

Sorry cannot remember it fully, but found a curses implementation that also was on pypi, I believe it was some kinda wrapper for pdcurses or something, making many things be compatible to regular *nix curses etc, and when building deluge with that added, and tried deluge-console again, then the missing curses error went away, but another came up, about something related to windows(not MS windows) and when looking that up then indeed that function was missing(in pdcurses), and didn't know what to replace the deluge code lines with, that called said missing curses function. I only remember it was something with hwd that was the issue and missing in said implementation(pdcurses).

BTW, the old deluge-console from deluge1, if I remember correctly, back when testing it previously, didn't either support the curses UI I.e when running deluge-console without any commands after it, but did work with added commands following, and which shouldn't need curses in my thinking. Deluge2 features a new rewritten deluge-console with new curses UI and also added commands(e.g 'move'), but seemingly the commands to run in non curses mode, still depend on curses, faulty or not, I dunno for sure, but sounds "wrong", but if could get curses included/working, then a non-issue regardless, and likely even make curses mode work, also on windows, unlike deluge1.

Edit: @petersasi, if ever having motivation to try this, then I believe you could add an extra 'pip install' line in deluge build scripts like this: 'pip install windows-curses', there is py38 wheels available on pypi, as just checked, so no compilation needed(for pdcurses in C) to possible break in msvc. Also, the error I now remember was related to sigwinch, about window resizing, not available in pdcurses, but maybe even works now all by itself, as see they added a patch 6 months ago or so, which should workaround such cases, hopefully, but honestly don't remember if was included when I tested it previously, but kinda think so, as I don't belive it was that long ago I messed around with it e.g maybe 3 months ago max I'm thinking, but worth a try, if ever feeling like checking it out, though as mystikfox previously stated, then i'd also guess that this feature(deluge-console) on windows isn't used by vast majority anyways, so up to you of-course if even deeming it worthy of your time.

Re: [Unofficial] Deluge 2.0.x installer

Posted: Tue Jun 16, 2020 1:47 am
by petersasi
Thanks mystikfox for reporting!
This is kind of beyond the installer, that Deluge uses a module that is not included in Python proper on Windows, but let's try to find a solution.

Thanks for the comments MHerz,
Any chance you have found the same as me?

How do you add that to the build? (I am still not a python guy unfortunately) do I just pip install windows-curses when adding the other wheels to python installed into C:\Program Files\Deluge in deluge-build?

They say they did add a hack to fix some incompatibility lately: "Starting from windows-curses 2.0, in the name of pragmatism, these wheels (but not Gohlke's) include a hack to make resizing work for applications developed against ncurses without Python code changes:"

Re: [Unofficial] Deluge 2.0.x installer

Posted: Tue Jun 16, 2020 3:33 am
by mhertz
Yes, same one :)

Yeah, just add extra pip line like that(pip install windows-curses), under the other pip install lines in 'C:\deluge2\deluge-build\deluge-{stable,dev}.cmd', as you said. Note, I'm no python guy either lol, though did learn a couple small things here and there from this project, which is nice, and made me able to make some python fixes myself even, here and there.

Yes, that's the patch I talked about, and indeed the issue, but as said, I don't remember if was already included when I tested it, and still spitting out error, but certainly sounds from its description like a proper fix to issue.

Of-course up to you if wanna include/mess-with in first place and just some FYI type thing solely.

BTW, sorry that I answered the question first, but just did solely because had some experience with the issue and about what issue was, and wouldn't have done so otherwise :)

Edit: Sorry, yes indeed, the pip line to add is with the preceding path to deluge2\scripts under programfiles, like the rest, so just copy/paste one of these lines as new extra line and change the module-name after 'install' to instead 'windows-curses'.

Re: [Unofficial] Deluge 2.0.x installer

Posted: Tue Jun 16, 2020 3:59 pm
by mhertz
Sorry for butting in again, but slight thing to add additionally, is that I saw that the PR by djlegolas regarding ignoring untrue libtorrent 1.2.x tracker errors from UIs was updated, so new link should be used in both deluge buildscripts, so if wanted and when feeling like it(don't stress, later I mean, not know) then change curl/patch line using deluge's github as source for patch, using curl and piping to patch, and is the only line like that in deluge-dev.cmd and the top one of the two subsequent ones in deluge-stable.cmd, and so change only the URL part of the lines to the newest commit i.e. into instead:

Code: Select all

(Old line has URL with: */commit/23* I believe.)

PR conversation:
cas-- on May 4 Member
This is comparing strings so the version comparison will not work as you expect it to. Look for other examples of version comparisons in the code.

DjLegolas on May 4 Author Contributor
Fixed :)

cas-- commented on May 10
In the commit message, please use [Core] instead of [TorrentManager], it's too specific. Can you also include a link to the github discussion, thanks

Note, I haven't checked if still builds and doesn't break by this change(I mean if patch doesn't apply anymore, or make subsequent ones fail etc), and if having issues then give me a shout to help you fix, but again, as said, for later and not now, and just something to remember possibly for next time rebuilding.

Re: [Unofficial] Deluge 2.0.x installer

Posted: Wed Jun 17, 2020 12:45 am
by petersasi
Thanks, will replace that too in the next releases. Actually windows-nurses already added, and some more last meaningful variable names, but I want to make comprehensive slimming as well in the next release. Can you please give me the issue URL to trwck for that?
Other plans include boost and libtorrent review to maybe get rid of some of my hacks by correctly telling the build system to build for MS VS 2019 based on the libtorrent repo discussions.