Page 51 of 66

GTK build ran, some files not moved

Posted: Thu May 28, 2020 1:24 am
by petersasi
Hi, update for this morning:
  1. added the pip update command to the cmd script
  2. I have added the capture output to the script
  3. As well as moving the log file to the gvsbuild-build folder before deleting c:\gvsbuild - 4.5Mbytes megabytes text file, wow
  4. took a look the VC bat files in hex viewer, they do not have the UTF8 BOM or UTF16 etc. encoding on first sight, just start with 0D 0A etc.
  5. Thanks for initiating the fix, seems to have started the build process at least. The folder name Unicode characters are unfortunately not chosen by me, I log into the Windows 10 on this laptop with an Office365 user in AzureAD and this is the folder name Windows generated automatically. At least we helped some other folks with this testing, I guess. :) (I really hope I will not have to create a new local user with 7-bit ASCII home folder name for the other builds either... :roll: Maybe I can change what home folder msys uses instead? :?)

EDIT: the build completed, these moves did not find the files in overlay:

Code: Select all

C:\deluge2\gvsbuild-build>move C:\deluge2\overlay\data\bin\vcruntime140_1.dll C:\gtk-build\gtk\x64\release\bin
The system cannot find the path specified.
C:\deluge2\gvsbuild-build>move C:\deluge2\overlay\data\bin\msvcp140.dll C:\gtk-build\gtk\x64\release\bin
The system cannot find the path specified.
C:\deluge2\gvsbuild-build>move C:\deluge2\overlay\data\etc\gtk-3.0\settings.ini C:\gtk-build\gtk\x64\release\etc\gtk-3.0
The system cannot find the path specified.
C:\deluge2\gvsbuild-build>move C:\deluge2\overlay\data\share\icons C:\gtk-build\gtk\x64\release\share
The system cannot find the path specified.

EDIT3: all 4 restored from the original build package (deluge2.7z)

EDIT2: do we need to keep C:\gtk-cache or that could also be removed at the end of the build script?

I have also updated the previous post with this build and proceeding to the next. (wanna test and make sure they work for all of them :)

LibTorrent build

Posted: Thu May 28, 2020 2:07 am
by petersasi
(A) Because and not deleting and not re-downloading the boost archives, I had to change the decompression from "%programfiles%\7-Zip\7z.exe" x boost_*.7z -oC: - that did decompress both versions to the below so that the move afterwards works as intented:

Code: Select all

set BOOST_ARCHIVE=boost_1_70_0.7z
rem add -C - so that download is resumed / skipped
curl -C - -LO
"%programfiles%\7-Zip\7z.exe" x %BOOST_ARCHIVE% -oC:
move C:\boost_* C:\boost

(B) I have added this to the scripts (hope we do not mind the outdated boost, I remember these are the versions you have found to successfully build with these libtorrent versions :P):

Code: Select all

rem try to get rid of many warnings to help readability - they are:
rem Info: Boost.Config is older than your compiler version - probably nothing bad will happen - but you may wish to look for an update Boost version.  Define BOOST_CONFIG_SUPPRESS_OUTDATED_MESSAGE to suppress this message.

(C) What seemed more of a problem is many-many lines of vcvarsall.bat not found after the below:

Code: Select all

b2 crypto=openssl openssl-lib=c:/openssl-win64/lib openssl-include=c:/openssl-win64/include libtorrent-link=static boost-link=static release optimization=space stage_module --abbreviate-paths address-model=64 toolset=msvc -j4

warning: Did not find command for MSVC toolset. If you have Visual Studio 2017 installed you will need to specify the full path to the command, set VS150COMNTOOLS for your installation, or build from the 'Visual Studio Command Prompt for VS 2017'.

'"C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.26.28801/bin/Hostx64/vcvarsall.bat"' is not recognized as an internal or external command, operable program or batch file.

I have checked and I only have an x64 and an x86 folder in
C:/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.26.28801/bin/Hostx64/ ; the referenced bat file should be found in C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\. I modified the script to copy the bat file to that location too and then remove it after calling
Fun fact: copy accepts and runs with forward slashes but del fails with them, I had to fix that to be able to delete afterwards...

(D) I have run the libtorrent 1.1 build, the boost build does suggest some invocation changes:

Code: Select all

cl : Command line warning D9035 : option 'GZ' has been deprecated and will be removed in a future release
cl : Command line warning D9036 : use 'RTC1' instead of 'GZ'
cl : Command line warning D9002 : ignoring unknown option '/MLd'

(E) I have noticed that the two bulld scrips differ in 1.1 not having these two lines below, is this on purpose?

Code: Select all

for /f %%i in ('dir /b C:\deluge2\deluge-2* ^| findstr /v dev') do copy /y libtorrent.pyd C:\deluge2\%%i\Lib\site-packages
for /f %%i in ('dir /b C:\deluge2\deluge-2* ^| findstr dev') do copy /y libtorrent.pyd C:\deluge2\%%i\Lib\site-packages

(F) I have noticed this as well in the build options, did you ever think about changing this to optimization=speed?

Code: Select all

b2 crypto=openssl libtorrent-link=static boost-link=static release optimization=space stage_module --abbreviate-paths address-model=64 toolset=msvc -j4

(G) The buld of libtorrent 1.2 now succeded but it only copies libtorrent.pyd to C:\deluge2\overlay\Lib\site-packages but not to C:\deluge2\libtorrent\lt1.2.3 (which is in fact should be lt1.2 or lt1.2.6) is this a mistake to fix, or on purpose, or have I messed up my buld script?

Re: [Unofficial] Deluge 2.0.x installer [Unmaintained/unsupported]

Posted: Thu May 28, 2020 3:22 am
by mhertz
Nice :)

Yeah, msys2 username/dir can be changed, independant of windows, but seemingly is not needed, and was a python parsing issue here, as msys2 did work it seems.

The error'ed move commands, after thinking about it, must be because during the error'ed out initial run, then the data folder under overlay was deleted, but no new copied into place, as the gvsbuild failed. Then at subsequent run, then those files are missing and so cannot be moved over. The missing files will be needed for subsequent runs, as else will always be missing, so if you could copy them into place from e.g. your deluge local install, as also would make deluges fail to run when building those later, because of said missing files/folders. The missing files/folder can be found in data folder in e.g. 'C:\Program Files\Deluge', or whereever having installed into. Maybe/likely I should have added another folder under deluge2, to store these things and copy over from there, each time from gvsbuild.cmd, instead of as now, as if a build fails, like now, then would miss those files, but just thought was no need, since always available from last gvsbuild build under overlay anyway, and didn't think about the issue of a failed build possibly.

Sorry for inconvenience.

About libtorrents, yes they will show lots of warnings of deprecated commands and whatnot of this and that, which is irrelevant and if the slightest real error happens then the build fails, which I messed with I believe not exaggerating 50 times minimum initially when getting these libs to even finish build without error'ing out. It's the build-files that's provided that gives these warnings and not my scripts. One of the warnings is for strictly if using msvc 2017 then will fail when not providing msvc version specifically, but the default value of just msvc, which means default installed msvc version, will work fine for msvc 2019. I initially copied the vcvars file in place to shut it up, but after retesting some things months ago, then I just removed that part, as was just for appearance and didn't change end-result/functionality, but up to you of-course. It looks like it fails, but actually that is a message of it working, oddly, if you added a pause command as last line, to be able to see it. I just check filedate under libtorrent.pyd under the libtorrent folder of lt1.1 or lt1.2 dir(under Lib\site-packages) to make sure - if the output, if adding that pause - states this here, then it's working:

Code: Select all

common.copy libtorrent.lib
        1 file(s) copied.
common.copy libtorrent.pyd
        1 file(s) copied.
...updated 272 targets...
running build
running build_py
package init file 'libtorrent\' not found (or not a regular file)
package init file 'libtorrent\' not found (or not a regular file)
        1 file(s) copied.
The system cannot find the file specified.
The system cannot find the file specified.
The system cannot find the file specified.
The system cannot find the file specified.

You can try remove the double rd lines if wanted, for avoiding all these "cannot find file", but I would'nt recommend it, as will hit you maybe once in 10-20 runs, with that annoying async file-delete timing issue error of windows, several threads about on stackexchange etc, and hit me plenty of times personally, at random times, untill reading different workarounds where this the simplest, as never happens twice in row apparantly. Also, I could just use 'rm -rf' from msys2 instead, which I guess would work, but unsure if an underlying windows issue, which it seems as is described as exisitng both in rd, a powershell delete applet and dotnet delete function, so internal windows delete timing issue when run from script.

Yes, that is on purposse that these lines are omitted. They are used in lt1.2 because they additionally are copied into deluge folders as are default deluge libtorrent version when not changing away from default libtorrent 1.2.x in installer. As you haven't built deluges yet, I guess, then no worries, as gets inherrented from overlay when later building those, hence my mention of order of scripts not matter previously.

No, I honestly never looked into that optimization option, and what even stands for, but sure do if you deem appropriate :)

Good job again, changing things to your liking. I agree it would be faster to keep things around, especially when static versions used all the time, and was just a habbit of mine to clean up after myself and have tidy build-dirs left afterwards etc.

Edit: The extra additional rd lines, can obviously just be shut up by echhoing stderr/out or whatever named on windows to NUL, to get cleaner output, though personally didn't trouble me much, as said did run usually without even being able to see the output often, when not testing it of-course, and just had a command run that checks filedate or runstate afterwards.

Improvements in the libtorrent build scripts

Posted: Thu May 28, 2020 5:08 am
by petersasi
I have done a few more updates to speed up builds and maintainability on many runs:
(A) stash the uncompressed boost versions next to the script in the libtorrent-build dir and:
  • ask 7zip to only extract if the files that are not already there: -aos
  • create a symlink to the right version: mklink /d C:\boost "%~dp0\%BOOST_VERSION%"
  • add -N to the patch command in lt-1.1.cmd patching boost
(B) I decided to stash PATH and restore it at the end of the script otherwise it accumulates so long that sooner or later build fails with: "Command line too long" (previously I have done all trial runs in the same CMD.EXE ran as admin).

(C) Replaced C:\Program Files (x86) with %programfiles(x86)% and C:\Program Files with %programfiles% where possible.

The latter two also done for deluge-dev.cmd, deluge-stable.cmd, openssl.cmd and gvsbuild.cmd.

With all these changes boost and libtorrent build runs practically without any warnings and faster for both 1.1 and 1.2 (sorry, I am doing it on an old and unused T430). :)

deluge stable and dev build

Posted: Thu May 28, 2020 6:29 am
by petersasi
Working on building deluge itself:
(A) Added PATH preservation to the scripts as above.
(B) Added the curl option -C - and commented out deleting the python installer
(C) Added the same pip upgrade to avoid many warnings, for readability

Code: Select all

rem upgrade pip as suggested by itself when the old version is run
python -m pip install --upgrade pip

Seems like I have succeeded in building deluge-stable, however these are not copied and I did not find them in the original download pack either, where should I look for them? Or shall I remove this copy operation?

Code: Select all

C:\deluge2\deluge-build>xcopy /ehq "C:\deluge2\themes\icons" "C:\Program Files\Deluge2\data\share\icons\"
File not found - icons
0 File(s) copied

Other than this, both deluge-stable and deluge-dev seem to be built properly. :D


Posted: Thu May 28, 2020 7:29 am
by petersasi
Both successfully built, please give them a try if you are brave enough. :)
I have tried the 2.0.4.dev38 version with libtorrent 1.2.6 and Awadita-dark in thin client mode and deluged-daemon on anothe box, works for me. :) ... g?e=kYyEkv
(BTW, how did you shorten your OneDrive link? This one actually reveals too much information about my account.)

Re: [Unofficial] Deluge 2.0.x installer [Unmaintained/unsupported]

Posted: Thu May 28, 2020 8:38 am
by mhertz
Hey there, you're making me look bad with all these nice changes! ;) Just kidding of course good job customizing to your liking and omitting warnings and whatnot.

I just tested both your builds and they're spot on, great job mate! :)

Yes, that I see is a left-over from older times, and I changed it recently and completelly forgot that line should be removed. It doesn't hurt of-course, but is useless. Not important at-all, but in install-components.cmd, not that you have use for it anymore, then the last del command, has a file defined to be deleted that I also changed away from, so could be deleted that file-name, though again doesn't matter one bit. It's auto-install.js, as before I found there where tarballs available of msys, then I used there stupid qt-created installer, which only can be installed unattended from downloading special script online(auto-install.js), and in uninstall_components.cmd there needed also other script, not available from download sites anywhere online, so I echo'ed the needed script into file and uninstalled unattended, but ditched all that when found a simple tarball, to extract on install and delete on uninstall.

I didn't do anything else but rightclick the Deluge2 folder I made under %onedrive% and selected share, and then copy link, ohh, also changed the default settings in that dialog to allow download but not edits, which I believe it was set for by default, and then link was like that, so dunno sorry. Maybe try an url shortener online, just google url shortener.

Have you thought about if you will overtake this thread here, or make your own thread i'll link to and let this thread here die off? I ask because If you continue here, then I will remove the unmaintained mention and exchange my link with yours on first post, or link to a subsequent post in thread where you post link, or whatever is best, and mention the takeover, or if you make new thread then I will link to that on first post.

Thanks again for investing your time on this :)

Re: [Unofficial] Deluge 2.0.x installer [Unmaintained/unsupported]

Posted: Thu May 28, 2020 9:12 am
by petersasi
  1. I think I will use a different file sharing instead of my work account, then
  2. once I have that in place you can probably change the link in the first post and indicate the new versions are no longer provided by you
  3. I will remove the two copies and the js, thanks.
  4. Do some more cleanup and the re-publish your build starter pack too :)

Re: [Unofficial] Deluge 2.0.x installer [Unmaintained/unsupported]

Posted: Thu May 28, 2020 10:08 am
by mhertz
Sounds good, thanks mate :)

BTW, just reread one of the threads where people complained about rd /s /q failing(when happens, it usually states folder not empty, or something along those lines, even though the switches used makes that irrelevant, but whatever, and so in addition to my previous thought of simply echoing to null, for second line of each repeated rd command, then in that thread were also posted a simple workaround to use if wanted(obvious solution, but nonetheless):

Code: Select all

rd /s /q c:\deleteme
if exist c:\deleteme rd /s /q c:\deleteme

There's many threads about it, and some state it's because of locked files, permissions or needing run fscheck to fix disc etc, but is in fact also an intermittent issue and where one in the know once posted was because of said async delete timing issue, but many don't know that and logically assume locked file or permission issue etc, I.e the usual things going wrong, as only possible cause.

There's several bugreports about it(async issue), e.g :

rd /s fails to account for this asynchronous behavior, which has two implications:

Problem (a): Trying to delete a nonempty directory (which invariably requires recursive deletion of its content first) can fail - infrequently, but it does happen, and the failure is not reflected in the exit code (although a message is issued to stderr).

Btw, in the gvsbuild.cmd build-script, there is alot of rd cleanup commands just for saving install size space, and I never got to doubling those up, but whenever thinking about adding that, then I put it on the "back-burner" still, because isn't important if fails in that section, because only to save space and cannot break anything if failing there, but would make sence to add regardless, but up to you of-course. Other parts need it though, as had seen breakage of deluge being able to start afterwards, because of such issue and hence why I re-added it back again.

Re: [Unofficial] Deluge 2.0.x installer

Posted: Thu May 28, 2020 12:06 pm
by mhertz
Sorry, double-post, just wanted to say that i've removed the unsupported/unmaintained part from thread-title here and included a notion at top of first post with this:

Note: I myself stopped with this project here, but petersasi stepped up and is now in charge of releasing unofficial deluge 2.0.x installer builds here, and so link now points to his releases and not mine anymore. You can thank him and/or post questions to him in this thread going foreward.

Please take your time of course with sending me the link to exchange, and I only posted this to state if you approve of the wording or if you want something changed or something. I also added a notion next to download-link about it currently still points to my releases and your link coming soon. (somewhat self-contradicting lol, and better to have waited, but whatever, and atleast people know it's not abandoned anymore, by edited thread-title and new notion)

Note, of-course if you change your mind, or feel it's to much later, or what have you, then give me a shout or PM or whatever and i'll remove you from the post again, just so you don't feel preassured or something, e.g. I don't know if problems will come later and you get fed up of the breaking or whatever have you :) Just sayin' :)