Deluge Windows Builds

[doadin]

I looked through your script just quickly, and please forgive me for not just making PR, but I haven't run your scripts and just some quick comments regarding your gvsbuild section, then you need pycario and pygobject and enable gi presumingly etc. Well, maybe you can get it work from pypi building from src, though failed for me - though pycairo have wheels I see, but regardless. Also adwaita-icon-theme needed. Sorry, I know you know all this, atleast if I remember correct, because I initially got help from your old gvsbuild cmd-script and Cas's yaml config, when first looked into gvsbuild long ago and had some trouble with it because non-default msvc-location etc.

Then, fontconfig (gtk3-dep) fails if not having git and won't work from msys2 git neither(even if added to path and working from cmd alone), but works through e.g mingit(-regular/msys2 or busybox, I used busybox version as smaller and reported faster - git for windows btw) extracted and added to path. Librsvg fail fixed now, so don't need cater to that anymore. If not using --python-dir or --same-python, and should be --same-python here, then by default downloads and builds through another python gotten through nuget, currently 3.10(optionally controlled through --python-ver), which not good when need pygobject and pycairo in py3.7 wheels here.

This line works elsewhere for me btw: (haven't tested if all the deps needed though)

Code: Select all

python build.py -d build --gtk3-ver=3.24 --archives-download-dir=%DOWNLOAD_DIR% --build-dir="%BUILD_DIR%" --msys-dir="%MSYSPATH:~1,-9%" --vs-ver=%VS_VER% --platform=x64 --vs-install-path=%MSVC_DIR% --same-python -k --enable-gi --py-wheel enchant gtk3-full pycairo pygobject lz4 --skip gtksourceview,emeus,clutter --capture-out --print-out

Yes I remember this now, I had some problems before with the VMs I use to test things like this in a clean setup and lost one of the VMs and script was lost. I have made a change to the script now based on this. I took in most of it except the download dir and build dir, could add later and likely will but it won't stop the build from running so for now leaving it out.

Newest python and vc2019 work good also btw, seemingly, in current gvsbuild, openssl, libtorrent and deluge builds/runs. If you ever switch(msvc), then only need these two msvc components btw, saving little space(1gb+ if remember correct):

Code: Select all

if exist "%programfiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" "%programfiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -version [16.0,17.0) -prerelease -products * -nologo -requires Microsoft.Component.MSBuild Microsoft.VisualStudio.Component.VC.Tools.x86.x64 | findstr . && if exist "%programfiles(x86)%\Windows kits\10" set dirty=1
if not defined dirty curl -LO https://aka.ms/vs/16/release/vs_BuildTools.exe && vs_BuildTools.exe --quiet --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows10SDK.19041 --installPath "%~dp0msvc" --wait && del vs_BuildTools.exe

You can see in the script I do use a VS Installer Config to install only certain components, if this like has more then needed I would definitely like to reduce this. Faster builds would be great.

Last, bsdtar, default in msys2, should be able to extract all needed, so 7z.exe addition redundant.

I add 7z because in my own experience its has worked the fastest, also is native and I regularly use it in cmd and UI for other things, kind of a personal preff here and is so small I think I'm going to leave it in it maybe add 30 sec to build time and thats only on the first run if you don't start a new VM/build environment each time it obviously won't add time to secondary runs so not a huge loss here.

Thanks for the help! I am now seeing I have not added installing git nor the wheels grom GTK(because it wasn't build as wheels yet again thanks) This is partly why I posted as I mentioned before there was some very obvious mistakes but now I think it has goten significantly better. I still have not tested or applied the 64-bit changes to 32-bit so it likely has the same issues but I will get there.

[mhertz]

Thanks for reply. I don't know how much you can save from that config you use, but as said atleast on vc2019-buildtools I found through trial & error could get away with only specify the vctools.x86.64 and sdk10, so 2 components, nothing else, and works with gvsbuild, openssl and libtorrent builds.

I see your point on 7z of-course, am just generally a stickler for minimalism and little bad wording on my part

In the gvsbuild command, I had '-k' added just for testing at a time, and should probably be removed, sorry, though don't hurt neither(continues building other non-dependant projects if one fails). Anyway, I tested that the needed stuff is just: gtk3, adwaita-icon-theme, pycairo and pygobject, so can skip the '--skip' and gtk3-full - normally adwaita-icon-theme should depend on hicolor-icon-theme, but doesn't in gvsbuild, seemingly an oversight, so would add that also(just to avoid cosmetic warning in deluge-debug.exe's cmd about missing hicolor-icon-theme - was also in gtk3-full btw).

I add build-dir and download-dir/cache because prefer extract/use under where script run and not hardcode e.g. 'C:\', - You could also remove --gtk-ver as default 3.24, and --msys-dir as default, but not important.

You can save ~200MiB install-size from using mingit-busybox(and reported little faster) instead full-git, also from git-for-windows(under homepage's 'all-releases'/github), meant for scripts etc, and not interactive usage.

I exclude the 'c' dir from extraction through bsdtar of the strawberry perl archive, maybe 7z.exe has such too, haven't checked, or else can delete afterwards if wanted, and 393mb saved out of the full 557mb, so 165mb left - works for these use-cases here atleast.

Petersasi once found and used "lto=on" option for b2/libtorrent(plus optimization=speed instead of space), which I then also borrowed. Arvid states biggest CPU culprit is hashing, so I added also this to openssl, through adding it to the default cflaggs/ldflags as no dedicated switch like b2:

Code: Select all

perl configure VC-WIN64A --prefix=%OUT_DIR% --openssldir=%OUT_DIR% CFLAGS="/W3 /wd4090 /nologo /O2 /GL" LDFLAGS="/nologo /debug /LTCG"

Just adding for just in case interessted, might as well - I again specify extra options because use non-standard dirs under script dir.

Last, for x86 gvsbuild, then they only support building on an x86_64 OS and just building for x86, so no need use 32bit e.g. msys etc, infact failed in building gtk3 x86 stack when not using 64 bit msys with "not valid win32 application" error - msys(Edit: gvsbuild, not msys) internally depends on several tools it downloads/use behind scenes and some of them not even available now in x86, when I tried hacking it long ago - I asked if they could please reconsider this approach, but they said to much work for very little benefit, as pretty much all have x64 now(to atleast build from, to target elsewhere).

Happy hollidays btw and thanks again for scripts

[doadin]

Cool, yea I very much appreciate the input thanks for the advice for GTK I'm going to look at making those changes. I havnt yet but looking to reduce the VS install as well I think I had it installing the IDE as well because at one point I was looking at the libtorrent code for something but obviously for just a CI/CD this is not needed so I think I can reduce that down. For the most part I think I have the script ready I might just remove the 32-bit/x86 bit completely for now. The script should be pretty stable? I just got my VM setup to start fresh on each boot so I can test the script start to finish so hopefully I can make some progress fairly soon and maybe we can get so closer to "official" builds going with proper python freezing and such. Again thanks for your input.

[mhertz]

Thanks I meant btw gvsbuild, not msys, not working for building on x86, needing some x64 build components under hood, and gvsbuild devs, not msys, stating not worth it to change when I opened issue-ticket regarding it.

The script should be pretty stable?

Sorry for being lazy and not testing it for you, but I now finally did, and looks nice, though have some smaller issues apparently. It had two errors and prompted for pressing "next" upon msys2 install - I quit the script then, as missed components would make fail later regardless - I remember that msys2 annoying installer before I changed to the sfx archieve - they use qt installer if remember correct, and silent commands don't work good, and a conf needed and even with that then still wants user input - I googled alot and found solution finally through experimenting, I believe atleast, but honestly don't remember now and don't think I added to git at that point but can try to search back if you need more info on it(maybe rambled on it here before).

I haven't looked into your script for the errors I mentioned, but i'm sure it's just a slight typo/oversight, and don't have time now to check, but i'm sure you don't need my help on it regardless

Anyway, looks very professional and nice, for a change (talking about myself not you here btw).

I'll fix the issues and test further and report back if anything to report, if you don't beat me to it, as you state to have VM setup now anyway. Probably first going to be later though, possible first tomorrow.

Code: Select all

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        12/27/2021  11:59 AM              0 vsconfig.conf
Installing Python...
Installing NASM...
7z : The term '7z' is not recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify that the path is correct
and try again.
At C:\All.ps1:130 char:9
+         7z x nasm-2.15-win64.zip -oc:\
+         ~~
    + CategoryInfo          : ObjectNotFound: (7z:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Installing OpenSSL Source Code...
7z : The term '7z' is not recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify that the path is correct
and try again.
At C:\All.ps1:140 char:9
+         7z x strawberry-perl-5.14.4.1-64bit-portable.zip -oc:\perl\
+         ~~
    + CategoryInfo          : ObjectNotFound: (7z:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Installing msys...

Edit(s):

* Was just 7z missing from PATH, so defined that earlier in script - now runs and just still need that stupid manual interaction with msys2 installer - as said will report back later.

* Strawberry-perl install fails, just because you forgot update the install command when having updated the download one, so filename mismatch - currently running and not finished yet..

* Change PATH(moved to beginning) Perl path from c:\perl64\bin to c:\perl\perl\bin.

* As said, not more time now, but it cannot find nmake(openssl), so add a preceding invoke-batchfile line running vsdevcmd from msvc - Also need upgrade nasm, as nasm error during openssl build of "section allignement changed during code generation". From changelog, I think you need nasm 2.15.03+, as issue there sounding related - I use latest nasm in my own scripts and works fine btw(v2.15.05). - Just noticed you use openssl-master, which is openssl-v3.x, probably reason why also fails at end with error, though strange one of module libssl3.dll is x86 and target x64 - I'm testing again now with latest 1.1.x i.e. 1.1.1m, which I use, update soon - * vcdevcmd needs x64 arg, just copied it from other section in your script - I think will work when just using vcvars64 instead as i'm used too(instead of finding needed args for vcdevcmd)... - Worked.

* Libtorrent fails - You need define msvc-14.1 toolset on b2 line for msvc2017. I changed vcdevcmd to vcvars64 and I setup needed env-vars and added user-config.jam under C:\boost_1_77_0\tools\build, and it didn't work for path with spaces regarding python, also disrecommended in docs - I tried double-quotes and such, and can try again later, but for now I just copied your python37 dir into C: and added that path to user-config.jam - also needed add path to openssl-win64 - it states looks in that path by default, but failed building the openssl related files and so I added that to b2 line, which worked - there where warning about probably wanting libtorrent-link=static instead of link=static, so i removed link=static and kept libtorrent-link-static and then info about about link=shared together with runtimes static option disabled or ignored or something, so I removed the static runtime option just in case, but maybe fine with it too. Here's the added lines btw:

Code: Select all

Invoke-BatchFile "${env:ProgramFiles(x86)}\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"

$env:BOOST_ROOT="c:\boost_1_77_0"
$env:BOOST_BUILD_PATH="c:\boost_1_77_0\tools\build"

Add/change b2 line with: toolset=msvc-14.1 openssl-include=C:\OpenSSL-Win64\include openssl-lib=C:\OpenSSL-Win64\lib

Copy '%programfiles%\Python37' temporarilly to C:\Python37 and generate C:\boost_1_77_0\tools\build\user-config.jam with: using python : 3.7 : C:/Python37 : C:/Python37/include : C:/Python37/libs ;

(Can just use 'OutFile -FilePath' from powershell, or alike, to generate that user-config.jam, though as said, didn't look into how get %programfiles% working yet, sorry, to avoid needing copy python over, or maybe just consider installing/extract python into "better" place)

* Need change msys in PATH from 'C:\msys64\bin' to 'C:\msys64\usr\bin'. Also chokes on the %programfiles% path here in atleast one project of gvsbuild, so will test here also through before copied C:\Python37. * Works better now , though fails in pycairo, as no bdist_wheel command - google states to just install wheels, so i'll include such preceding line. Currently gvsbuild patches pycairo and pygobject with a py38=+ patch, i'm suspecting will break working through py37(as
using non-exisiting os.add_dll_directory call), but i'll check that later when build works(we can e.g use sed to remove those patches, or check if --python-ver 3.7.9 omits them, despite extra redundant step of download/use python 3.7.9 which gvsbuild does with that option). Then harfbuzz fails, and googling the error states it happened for one on vc2017, but not vc2019, which sounds likely as i've not seen it, and only use vc2019 myself. This will take some time debug, and have other things needing testing too myself, so if not beaten to it, then will report back in little later time.

[mhertz]

Thanks for updates(github for others reading). Sorry for propossing the user-config.jam in non-standard location, and see you use the official one, but through trial and error I found a place under build-dir where also accepted, and prefer taint host-machine as little as possible, hence my thought-process on that, so you don't think i'm reading-impaired

I haven't tested your scripts more since last, as messing with something else currently, but will get back to it eventually, if not beaten to it, but I see you still use msvc 15 definitions. I know there are two (version-scheme-)ways to define that, and the one you use i'm pretty sure isn't the right one for here, and if you google that msvc ver to use for bjam/b2/boost, then it's 14.1 for msvc-2017 - I changed your b2 toolset line to that, and so unneeded to also include it in the user-config.jam, but doesn't hurt neither of-course. I don't use the bare b2 line, before the "libtorrent" b2 line - I know it's stated in cmd help output to do, but redundant, but again dosen't hurt - and I see you want to do things in the propper documented way generally, which is respectable and arguably better

Last, i'm sure I did test that user-config.jam way you use, and didn't worked, likewise double-quotes, backslash-escaping the actual spaces, and finally what's recommended in bjam docs i.e. tripple escaped double-quotes - they do there also recommend using env-vars instead(using double double-quotes around when defining them), which I honestly didn't get arround to test - little stupid honestly, but as regardless got those issues with gvsbuild later, then just decided to copy over the python dir. Btw, if this is only for building deluge stack and nothing else, then why not simply download and extract everything(not there already) under a build-dir and so only install one thing i.e. msvc, as not working "portably" I checked(nsis neither need installing btw)? Your project your rules/reasoning obviously and just curious Nice touch with the update script btw.

Thanks.

Edit(s):

* Remember the msys /usr/bin ammend to PATH.

[doadin]

Thanks for updates(github for others reading). Sorry for propossing the user-config.jam in non-standard location, and see you use the official one, but through trial and error I found a place under build-dir where also accepted, and prefer taint host-machine as little as possible, hence my thought-process on that, so you don't think i'm reading-impaired

Not having host "taint" is a good idea, my idea for the script is for it to be run on a vm that starts fresh each time so I didn't think much on this but if it works else where could change that.

I haven't tested your scripts more since last, as messing with something else currently, but will get back to it eventually, if not beaten to it, but I see you still use msvc 15 definitions. I know there are two (version-scheme-)ways to define that, and the one you use i'm pretty sure isn't the right one for here, and if you google that msvc ver to use for bjam/b2/boost, then it's 14.1 for msvc-2017 - I changed your b2 toolset line to that, and so unneeded to also include it in the user-config.jam, but doesn't hurt neither of-course. I don't use the bare b2 line, before the "libtorrent" b2 line - I know it's stated in cmd help output to do, but redundant, but again dosen't hurt - and I see you want to do things in the propper documented way generally, which is respectable and arguably better

I was going to check how the install works not sure if boost cares as far as telling it which toolset to use but I think my script installs through vs the newest version of 14 which is like 14.10 now or something so may need to double check that portion.

Last, i'm sure I did test that user-config.jam way you use, and didn't worked, likewise double-quotes, backslash-escaping the actual spaces, and finally what's recommended in bjam docs i.e. tripple escaped double-quotes - they do there also recommend using env-vars instead(using double double-quotes around when defining them), which I honestly didn't get arround to test - little stupid honestly, but as regardless got those issues with gvsbuild later, then just decided to copy over the python dir.

Btw, if this is only for building deluge stack and nothing else, then why not simply download and extract everything(not there already) under a build-dir and so only install one thing i.e. msvc

This script I intend to be very versatile I checks certain directories for installed deps and source code. If either is there it will not re-download/install, this way the script can be run many times without the need to start a fresh OS. Or can be used on a fresh OS for development, or personal use, as some people may be doing till we get a official build releases going. Also can be easily changed to run on a hosted CI/CD like github actions by changing the checks to point to locations that the CI/CD host has the deps installed at. This way we could use github actions or travisci or appveyor or jenkins etc etc to make official build any time there is a git tag(useful for cas).

, as not working "portably" I checked(nsis neither need installing btw)? Your project your rules/reasoning obviously and just curious Nice touch with the update script btw.

Thanks.

Edit(s):

* Remember the msys /usr/bin ammend to PATH.

I have not added the part to make a nsis installer yet need a working build first to check for needed changes, I had a updated nsis installer before should be fairly easy.

Yea I figured the update script would be nice especially when i start with a clean VM a lot and need to re-download the script. I'm just going to have it on a virtual disk and mount that with the clean VM.

[mhertz]

Thanks for reply and explenations doadin, appreciated. I'm pretty sure using toolset=15 didn't work and first when using 14.1 did, hence my adamentness, but could remember wrong. Anyway, if making more edits later when have time/motivation for that, then will try keep it very short to not spam your thread, not the best at that honestly. Thanks again.

Edit: Just wanted to lastly state toolset msvc-14.1 is correct for libtorrent(boost), but you where correct in previously using --vs-ver 15 in gvsbuild i.e. the two different version-schemes I referred to previously. Anyway, i'm finishing up my own stuff now and then will call it quits again with this windows stuff as spent way too much time on it already(my stuff, not yours) - thanks again.

[doadin]

Thanks for reply and explenations doadin, appreciated. I'm pretty sure using toolset=15 didn't work and first when using 14.1 did, hence my adamentness, but could remember wrong. Anyway, if making more edits later when have time/motivation for that, then will try keep it very short to not spam your thread, not the best at that honestly. Thanks again.

Edit: Just wanted to lastly state toolset msvc-14.1 is correct for libtorrent(boost), but you where correct in previously using --vs-ver 15 in gvsbuild i.e. the two different version-schemes I referred to previously. Anyway, i'm finishing up my own stuff now and then will call it quits again with this windows stuff as spent way too much time on it already(my stuff, not yours) - thanks again.

I seem to have it like 99% complete. I was able to get everything compiled and deluge runs from python however I am having an issue where pyinstaller exe's are giving an error saying it can't find distribution certifi. I even have the nsis installer script working. Just need to find out why pyinstaller exe can't load certifi. There is a certifi colder in the dist folder too, it only has a pem file though.

[mhertz]

Looks like the windows users very soon will get an official installer, from your nice work. Awesome news, thanks again mate

I know nothing about pyinstaller, so have no input. There's some little info, when googling "pyinstaller can't find certifi", e.g. requests can't find the pem, and some code to add to make it work etc, plus another workaround of another pyinstaller/cerifi issue, but as not having it to "play" with myself, then hard to troubleshoot in blind, though anyway no need, as i'm sure you're better at this than me, so you'll figure it out eventually.

You say only a pem file in the certifi folder if understanding right, so does it work if you added the missing certifi files from the deluge unfreezed build's certifi folder under Lib\site-packages\certifi and into the dist\certifi folder of freezed app? Or else if not work, could play with the google-hits info. As said, I know nothing about pyinstaller/freezing so if saying something stupid here, then apologize in advance.

Good luck mate, and thanks again.

Edit: you're using the certifi hook of pyinstaller-hooks-contrib right? (Again, sorry if dumb question lol ) https://github.com/pyinstaller/pyinstal ... certifi.py

Edit2: Sorry, no solution, just wanted to say that above edit is useless so ignore - I made quick test with installing pyinstaller and making test.py with "from certifi import where; print(where())", and ran 'pyinstaller test.py', and it also only included the pem in dist\certifi, but still running test.exe ran fine and outputted to cmd the dir of the pem as per the certifi.where() function, and which is also that function which is the entirety of deluge's cerifi usage. Also, the certifi hook from contrib was auto-loaded I saw in pyinstaller building output. I Also even uninstalled certifi from host, to make sure not somehow loaded, and still worked, though test.py on host failing as certifi missing - I don't really get how that can work in freezed version, because only the pem seems included, but the certifi package when using where(), needs __init_.py and core.py, because there is where() even defined/included, and couldn't see included elsewhere under dist, but whatever, must regardless be somewhere included still and just wanted you to not spend extra time on my clever ideas, so need look elsewhere, just don't understand how your freezed deluge cannot find certifi distribution, when a simple example like mine worked.

[doadin]

Looks like the windows users very soon will get an official installer, from your nice work. Awesome news, thanks again mate

I know nothing about pyinstaller, so have no input. There's some little info, when googling "pyinstaller can't find certifi", e.g. requests can't find the pem, and some code to add to make it work etc, plus another workaround of another pyinstaller/cerifi issue, but as not having it to "play" with myself, then hard to troubleshoot in blind, though anyway no need, as i'm sure you're better at this than me, so you'll figure it out eventually.

You say only a pem file in the certifi folder if understanding right, so does it work if you added the missing certifi files from the deluge unfreezed build's certifi folder under Lib\site-packages\certifi and into the dist\certifi folder of freezed app? Or else if not work, could play with the google-hits info. As said, I know nothing about pyinstaller/freezing so if saying something stupid here, then apologize in advance.

Good luck mate, and thanks again.

Edit: you're using the certifi hook of pyinstaller-hooks-contrib right? (Again, sorry if dumb question lol ) https://github.com/pyinstaller/pyinstal ... certifi.py

Edit2: Sorry, no solution, just wanted to say that above edit is useless so ignore - I made quick test with installing pyinstaller and making test.py with "from certifi import where; print(where())", and ran 'pyinstaller test.py', and it also only included the pem in dist\certifi, but still running test.exe ran fine and outputted to cmd the dir of the pem as per the certifi.where() function, and which is also that function which is the entirety of deluge's cerifi usage. Also, the certifi hook from contrib was auto-loaded I saw in pyinstaller building output. I Also even uninstalled certifi from host, to make sure not somehow loaded, and still worked, though test.py on host failing as certifi missing - I don't really get how that can work in freezed version, because only the pem seems included, but the certifi package when using where(), needs __init_.py and core.py, because there is where() even defined/included, and couldn't see included elsewhere under dist, but whatever, must regardless be somewhere included still and just wanted you to not spend extra time on my clever ideas, so need look elsewhere, just don't understand how your freezed deluge cannot find certifi distribution, when a simple example like mine worked.

Yea honestly I'm not sure why it is not working. Yes it is running the certifi hook I see it in the log, thats why im confused. It knows it needs it its running the hook to get it, yet it still is'nt in the dist? And deluge runs fine pre-freeze so I know it has all of the requirements installed. This is an interesting situation.