Deluge not reporting hash-checks properly

[mhertz]

Hello guys, and nice to see you bengalih interesting discussion you're having, and not gonna interrupt your back and fourths here, except to say thanks for reporting bengalih, and sorry I should have posted a big warning or something in the ltconfig thread, and looked into fixing if needed, and reporting upstream possibly, been little lazy about it, plus later forgot, when I also noticed this discrepancy some years ago during own testing with ltconfig. I will look into it later and post back if making something, or have some input atleast. Agreed would be nice if ltconfig got forked and maintained properly, though just don't feel like i'm good enough at that personally.

I'm waiting to post back in the ltconfig thread until having something atleast somewhat intelligent to say, so not ignorring you there, just adding both replies here. The libtorrent settings and dht and few more smaller stuff is saved and restored between runs and at x mins, like ambipro stated, for easy and fast restoral of such for clients by libtorrent API(so don't need code those settings manually back to session), and for deluge this isn't an issue, but for plugins like ltconfig, then it gets sorta misleading when deleting stuff from config-file's as bengalih found. Thinking about it, then little tricky because when option not in config, then ltconfig have no way of knowing if changed by itself in older times and later deleted, or if deluge or some other plugin did, but atleast it knows for a fact that it didn't itself have settings defined for ammending said option now and hence lists it as unconfigured by itself, as is correct, for current session and not back in time, but very much misleading agreed. I must confess I didn't read 100% close everything in this thread, and have to read back more close about suggested workarounds etc. Also I have to check how exactly ltconfig does it's saving and what default libtorrent preset explicitly is/is-gotten etc(I know what it sounds like obviously, but just making sure, as e.g. like high-performance-seed preset just is a hardcoded set of settings in ltconfig surpricingly to me, and not actually gotten from libtorrents own function for it i.e. using high_performance_seed() function). So, my humble opinion is this isn't deluge's problem, but a damn big problem regardless, from a usability standpoint. Yes you can delete session.state fine, and then you only just loose any options not already hardcoded by deluge or ltconfig. Hmm, have to check if changing an option in deluge itself, and then later deleting said changed option from core.conf and then restarting deluge, if it also shows option as unset in GUI, despite is set, previously, and so restored from session.state, and so if that is the case, which sounds like it actually, then indeed also a deluge issue - well you could say we are not supposed to do that, but still. Requires some thought(maybe some I said above is wrong possibly, or misunderstood, very possible), like you guys are doing nicely

Thanks guys

[bengalih]

Hello guys, and nice to see you bengalih ...
So, my humble opinion is this isn't deluge's problem, but a damn big problem regardless, from a usability standpoint. Yes you can delete session.state fine, and then you only just loose any options not already hardcoded by deluge or ltconfig. Hmm, have to check if changing an option in deluge itself, and then later deleting said changed option from core.conf and then restarting deluge, if it also shows option as unset in GUI, despite is set, previously, and so restored from session.state, and so if that is the case, which sounds like it actually, then indeed also a deluge issue - well you could say we are not supposed to do that, but still. Requires some thought(maybe some I said above is wrong possibly, or misunderstood, very possible), like you guys are doing nicely
Thanks guys

Thanks, long time no talk! Anyway just some thought/questions/comments on the above.

First off, I agree this is definitely more of an ltConfig then a Deluge issue, however it just comes down to semantics.
It isn't like I'm looking to blame any particular piece of software or dev, and I don't know enough about how plugins (and specifically ltConfig) interfaces with Deluge and what possible limitations there are there to state with any authority that some additional API, or way that a plugin can address certain aspects of Deluge would make solving this easier (or avoided it from the beginning). I think it is still a valid point to make though that without the available plugins for Deluge, it would not be anywhere near as popular a torrent client. So it is a little hard to separate what is a Deluge vs. plugin problem. If someone needs to use AutoAdd (e.g.) or ltConfig or any other plugin to get the behavior that they might be able to get OOB with another torrent client then from a "product" perspective the Deluge community needs to recognize that. I won't keep bringing this up, because honestly it isn't important - what's important is that a Deluge user's experience (plugins or not) should be designed to avoid these types of things.

Can you give some clarification on your statement about when deleting session.state "you only just loose any options not already hardcoded by deluge or ltconfig."? What options would you lose? I might be mistaken but, for the most part (barring this issue) I've always assumed that Deluge works logically like pretty much any designed software. By that I mean that if you make a change in the UI interface that options gets saved into one of more config files. In Deluge's case, multiple files - but mostly core.conf for the main program, the supporting authentication and web conf files and then a .conf file for each plugin. So when you say an option hardcoded by deluge or ltconfig I assume you mean an option that the user either set - and therefore it is stored in one of these config files - or didn't set, in which case the default values are used.

IOW, I'm still confused about the purpose of session.state and why it needs to exist at all (especially between launches if it has the potential to cause issues as it has shown here). Your statements seem to infer that the config file(s) are not the sole source of what the GUI will show for settings. Now I learned early on that Deluge doesn't like its config files modified directly when it is running. Usually any changes made while it is running get removed from the files on restart. I also admit I don't fully understand when and how the .bak files are used. That being said modifying a conf file when Deluge is not running one should expect those changes to get loaded and displayed properly in the GUI. If something is interfering like that - like session.state - that's a problem.

So I think it is a good start to understand exactly what session.state is doing and why it needs to exists and why it is being used as a higher source of setting validation then what is stored in the ltconfig file itself. It sounds like we are unsure on all those, but if I'm mistaken then I would love to understand it.

Finally, just so I'm clear - is the original developer of ltConfig totally MIA? Is he just not developing the plugin anymore , or is he totally unreachable?

thanks for your efforts!

[ambipro]

Just giving my opinion on the thread you keep pulling with a few things I've tried to address previously...

The purpose, as I understand it, of the state files is to retain information for Deluge to essentially load into libtorrent (like a profile). These are configuration settings in the case of session.state and obviously the fastresume for the other state files. In almost every case, there is no problem with this approach, and seeing as you both don't want to recheck every torrent on restart and want your configurations to be persistent across restarts in almost every case, is a logical approach. I understand that you may not see the value or comprehend why a specific thing is done the way it is, but I can almost guarantee that there was a reason time was spent to do it this way. It would seem foolish to waste time adding this otherwise. You'd have to ask whoever wrote it why that was, unfortunately, but given that it exists and time was spent to write it, there's almost certainly a point. Not every feature caters to the needs of every user (some are never even used by most users) and given that your problem was caused by a third-party plugin behaving in an unexpected/undesirable way, it just furthers the point.

Another point I'd like to make is that this is open-source free software.You keep returning to putting the responsibility of the plugins that anyone can write to do almost anything within the constraints of Python and permission on the host system and post on the internet, onto Deluge itself in some way. Let's compare it to Linux. This is like saying that Ubuntu, Debian, Linus Torvalds, etc, should be responsible for preventing a user from making all mistakes in software they write and/or use, or software they choose to download that isn't bug-free. It's third party. I understand that mhertz has an "official" role and has taken to maintain some aspects of the plugin, but he did not write the original and seems extremely reluctant to take the project on as "his own" fully from my discussions with him. ltConfig seems more or less abandoned otherwise. This is why OSS is great, but also bad. There is no onus to perpetually maintain software, or further, to never have bugs. That's just not reasonable. People move on, other's may take over in whatever way they want. Speaking personally, as a developer myself, we want the best experience for a user as possible if we're developing software for a community. However, if we expect to meet every single user case, need, or feature-set without any problems we are only setting ourselves up for disappointment and failure.

You had a bug directly affect you, but it was extremely edge-case in regards to the severity of the effect. I understand that it should be addressed. I'm not arguing any of that. What I "disagree" with is the approach you take to describing the problem as needing some sort of huge refactor and core changes to all systems.

I'm just slightly curious why you're so stuck on this track the way you are...bugs exist, when they can be addressed and there is a need for them to be, they mostly are in OSS, either by the developer, a maintainer or a third party such as mhertz who steps up. It's a simple case that there is a bug in the plugin, it fell through some cracks previously, and it can be addressed now. This more or less just shows the system of OSS working. It's third party, the developer is MIA, and despite it being very useful, as others are, there's nothing wrong with Deluge natively in this specific situation from my perspective.

In writing this, along with the other posts, I can see how the posts I've made could be interpreted as less than agreeable. However, I deal with developing a rather large project and doing the majority of support for the user base in addition to supporting Deluge here. I have my own biases and opinions on OSS as a whole and where responsibilities and dealing with bugs lie.

To clarify further, none of my posts in any way are attempting to place blame or shrug off the need for this to be fixed to prevent future issues, large or small, merely to shed another perspective on the system as a whole and the difference in a user's perspective relative to a developmental one.

On a side note, specifically addressing session.state vs ltConfig.conf - you removed a setting entirely from it. It only sets what is present. This is where the disconnect lies with session.state - it retained what was previously set in libtorrent, and ltConfig.conf contained nothing to indicate it needed to be set to any value - so it didn't set it. Thus, session.state was used for that value. It has nothing to do with preferring one over the other. If this was the case, ltConfig would not function at all.

[mhertz]

I thought about this yesterday, didn't feel like messing with it honestly that day, but the more I thought about it, the more I realized it would be hard to fix, I mean make work consistently, in the various edge-cases, and that I began to believe it wasen't an ltconfig issue even, but more as said an annoying confusing usability kinda issue, but not technically wrong I guessed. Now looking at the code, I see it actually has code to revert back to default when unticking an option, but still there's an obvious issue still I see. Anyway, I reproduced that it indeed revert back to default, but still before mentioned issue there. What I mean is that there's two issues, one is that at start of plugin(with deluge), then ltconfig saves the original libtorrent settings, and so when unticking an option then it reverts back to default, and where default means that saved part at plugin startup, so just from today/current-run. So, closing deluge, or deluged specifically when in thinclient mode, and then restarting and unticking an option, then it will still stick because ltconfig thinks is back to default again. Same is the other issue of if deleting a setting from ltconfig.conf, then ltconfig have no way of knowing this, and so will do nothing about that option, and it might be in session.state already, from previous ltconfig run. These issues I don't see anyway to overcome honestly, so we just have to, if feel needed, delete session.state manually upon changing config-files or if wanna make sure everything is right when not sure if such a situation could have occured otherwise, also from just GUI usage etc. It helps seeing it as ltconfig can change options for current running session, and if enabled in GUI or available in config, then means defined for chainging by ltconfig, and if not, then isn't, and nothing other is known about defaults, other than from current startup, and is out of scope imho to make it think about other plugins changing settings, deluge's preferences changed after startup, and old settings still in session-state from itself or other plugins maybe etc etc.

Btw, I see in code that the libtorrent default preset also just is a hardcoded list of settings, like high-performance-seed and min-memory-usage, which I don't like as some settings outdated or missing or changed, but not much I can do about that honestly, except manually keep updated and which i'm to lazy for honestly, and I only updated previously the high-performance-seed preset because thought was most important. Side-point, I also see in code that when selecting a preset, then it only saves to config the settings that are different from currently loaded libtorrent settings, so if re-using configs, then have to keep that in mind, e.g. before mentioned deleting of session.state etc. That is nice for not having config-settings that is irrelevant/redundant, but could make it little less precise. I can change that, but honestly I prefer to leave it, as all these are edge-cases which are nice normally, and just annoying in before said edge-cases. Ohh, forgot to say the pre-ltconfig "preset", is actually gotten from session and not from hardcoded list, though pretty obvious, but just adding for completeness.

Last, session state, as bengalih asked about, is as ambipro stated. When coding a client, then have to setup the libtorrent session and add all the settings you want away from default, and instead of adding a bunch of them every time, then can save and restore the session settings through that, plus it also adds the dht table and ipfilters and such, so a save/restore of current state, as ambipro stated. Also, as changed settings sometimes only saved to config at closing an app, then when a loop defined for saving current settings at x mins(in session.state), then also nice of-course. Honestly i've been to lazy to see how deluge handles that, well other than save and restore the session from libtorrent calls, at startup/closing and at x mins in loop, but I mean in regards to applying it's own settings, but I did check that they are matched i.e. if deleting a setting from core.conf, which is in session.state, then is reverted back to default again - well i'm guessing it just add all the options again regardless, from core.conf, and if any misses, then just uses the default, which is overriding the one in session.state, and this isn't really possible implement for ltconfig, but anyway, not an issue here, so in effect, you don't miss anything really(from deleting it), except in the case of ltconfig settings not in your config and so neither restored from said deleted session.state anymore, but as said pretty niche edge-case, and you shouldn't miss any deluge settings as far I know, sorry for stating before that this was a possibility, I don't see how unless mistaken, and eveything is just reloaded in from deluge and plugins again, so no worries, and as said I before deleted it from the unofficial installer upon every update, as got tired of reports of old libtorrent settings making updated libtorrent stalled and breaking deluge starting, and repeating the same thing over and over, so would say it's fine, but as always keep backups(in general).

If we can think of something, or if I have misunderstood anything, then of-course we can look at it some more, but just currently my humble opinion. Atleast this situation was made aware about, well for people finding this lol, but regardless, so thanks bengalih for reporting and thanks ambipro for your thoughts, appreciate you guys

[bengalih]

Thanks to both of you for continuing to look into this. I don't want to take the time to go through and quote both of your last posts to address each point individually, so let me just sum up some of my thoughts/responses:

For one, I don't think I ever put forth the argument that this was "needing some sort of huge refactor and core changes to all systems." I don't think I even came close to that. I did posit that I would expect the program to work by honoring configuration file as is the norm and also have argued (and will continue to do so) that one cannot separate Deluge from its' plugins holistically because it is the sum of all of these that makes Deluge such an attractive an well used package. I also acknowledge that this is OSS and I understand what goes along with that from good and bad aspects. I'm obviously not here trying to push any specific developer to make changes to deal with my "edge case" (which, I will also dispute that is what it was based on the now realized behavior of the plugin). But, I will also dispute the fact that just because something was done a certain way in the past means it was done the right way, or even the best way to deal with issues that might come up later. Again, none of this is to ask for, or even state that either Deluge of ltConfig needs some grand redesign. I may have suggested that there are limitations in the current design that might prohibit a more flexible approach. But there are always limitations, and this was a generalized "perhaps it can't be fixed because of X", but I don't see how I ever advocated for a huge refactoring of anything.

I also acknowledge that as OSS, I'm free to fix it myself. I'll be the first to admit I likely don't have the ability to do this in either a timely or efficient manner. I don't think that coming to a forum to discuss a genuine problem, explain the situation, and offer some suggestions amounts to complaining or demanding (not that I am accusing either of you of stating as such). It is clear from the conversation that there are unknowns in how this functions and those unknowns will only continue to cause issues. More issues with a product, more people move away from it or bad-mouth it, etc. And, I think we can safely say we are all here to support Deluge's continued exitence.

So, seeing how both of you have much greater insight into how this works under the good because you are both better programmers than I and more familiar with the particular structure of the Deluge code, I ultimately leave the decision in your hands. As I stated from the get-go a "bug is just an undocumented feature" and the sole fix for this may simply be better documentation and education as to how these pieces play together. While I am not a developer by trade I have been involved in product development for software that is used by Fortune 50 companies. I mainly dealt with implementation, but part of my role was to liaison with product management and the developers directly to improve the product. I say this only as a final emphasis on all of my above points to show that as a "user" I also understand all of your concerns and have some past experience in dealing with them. So, please take that into account when choosing what lens to view my comments though.

All of this being said from a user (and product design usability standpoint) I would just say that this is what one would expect when looking at Deluge becoming familiar with the design:
1) It appears to be based on a .conf file structure like many, many, (many) linux based packages. This structure has followed it into the Windows world and I expect in these regards would likely operate the same (i.e. this issue would occur) on both platforms.
1a) Conf file generally work the same way. Put in a config setting to alter the defaults, remove the setting to revert to default.
2) Linux software based on a .conf structure generally allows modification of .conf files (even while the service is running) and these files are usually honored completely at next restart (and sometimes when sending a signal to reload to the process).
3) Caching issues do occur - to reference the product I worked with - it definitely had issues where cached data would skew results even if a configuration file was updated, it might show results from the prior configuration.
4) Cache files usually have a timeout. Again these issues I speak on with my product would have eventually (usually within 1-24 hours) been removed and replaced with valid data/configuration.

So, my main point though most of this is that it is simply not common knowledge how session.state works and why it is there.
Clearly it has the ability to cache configuration settings that have long been removed from conf files.
Additionally, the idea that editing a conf file to remove a setting and expecting that setting to be removed, instead of the concept of having to forcefully/manually reverse a setting is not intuitive in the least.

I think if a .conf file can't be trusted, that's a problem. I would be really surprised than anyone (user or developer) would argue against that case. That being said, I'm not saying that the software needs to be re-written (although I am saying that there probably is a better way to design the software to accomplish all the required goals - but I'm not the one to do it, not am I suggesting that someone else do it. I'm just stating that it is likely possible).

It seems to me that this problem can probably be avoided by doing two things:
1) Simply removing session.state every time Deluge starts and/or stops and create a new one based on the settings in the .conf files.*
2) Adding some clear verbiage - perhaps within the ltConfig's main settings page that states both that you need to manually reverse any settings and/or remove the session.state file (or whatever else is deemed accurate). This would go a very long way and would then only be affecting the users who choose not to read.



* It seems from ambipro's statements there are some other things in the .conf file that we might not want to remove that can't be recreated? I'm still unclear on those things. It is also possible to simply parse the file to remove/reset things. If I were designing the software (and this is by no means a slight at what was done originally, only in restrospect), it would seem that parsing the existing config file and modifying the session.state to be updated with the configuration might be a better way to approach it.

Is the format of session.state defined fully by libtorrent? Or is it defined by ltConfig? I cannot find reference to a generic libtorrent session.state file, so my assumption it is ltConfig's way to store session information and then ltConfig somehow uses this to pass on to libtorrent to set the session parameters?

If there is indeed info in session.state that we would want preserved, and could not be recreated on start-up, it would seem to make sense that was either broken into a separate section in that file or moved to another file altogether to be used in conjunction. This way ltConfig could set all the session.state data that is defined in the ltconfig.conf file (by totally over-writing it with the .conf file settings) while still retaining whatever else cannot be re-created. Just a thought, and not one I expect to see implemented until someone actually chooses to do a larger redesign of the plugin (if that ever happens).

Anyway - I'm here to discuss, but I don't want to keep retreading old ground - especially when it concerns what/who/how to blame. The fact is there is an issue. I don't think I'm the person in the best position to solve it, so I'm bringing it to the "experts" along with my experience, observations, and suggestions as both a longtime user/fan, a fairly technical person, and someone who has been on several sides of the development landscape.

I would at the very least like to see more visibility for the issue because I think this has been a misunderstood topic for a while now.
I will have to read mhertz's post above a couple of more times to try and distill exactly what session.state is doing. It might not be a bad idea to have an explanation of it, why its needed, and when (or when not) you shouldn't remove it as a sticky somewhere.

I appreciate both your efforts at working towards whatever solution you deem fit.

[mhertz]

I completely agree that deleting a conf-setting, should mean it's not configured, and for deluge itself is like that, because it has a big list of all it's settings and it's default values, and so if any is missing from core.conf then it just resets it back to default. This is also how all other plugins work, however obviously isn't possible for ltconfig, which has every libtorrent option inside, and granted has a default preset, but is outdated and hardcoded, and since it doesn't know the default values from previous runs, or changed from deluge itself in updates, or other plugins etc, hence the issue. It gracefully resets to default from current session when unticked, nothing else, and I cannot see how overcome this issue personally, unfortunetly, but open to suggestions

Ltconfig knows nothing about session.state, it's only deluge which controls that, well libtorrent, but deluge signals this to libtorrent, for save and restoring it. Ltconfig can only make changes to the things you tell it i.e. what is in the config, and it can not know what was done in older times in previous runs. It's outside the scope of deluge to check for mismatches between ltconfig specifically, and itself, unfortunetly.

You only have to delete session.state, if deleting lines from your config which previously was in effect, or if unticking options in GUI made from previous runs. If can remember the default, then can just switch it back, before unticking/deleting, and then don't need delete session.state. I agree this is really strange and annoying, but unfortunately is what it is, with stuff like this, and i'm afraid of confusing people needlessly with mentions of this, as for most people will make more confusion then helping them I feel, and so just not really sure how go about it. Normally a stray option is not that big deal, but obviously in your case it was a bigger issue because was about integrity checks. Still would be nice if worked like everything else, but just hard in this specific case, unless making delge stop using session.state, which neither the answer imho.

Libtorrent API reference (session): https://www.libtorrent.org/reference-Se ... sion-state
Libtorrent tutotial (session): https://www.libtorrent.org/tutorial-ref ... sion-state

Deluge loads session.state: https://github.com/deluge-torrent/delug ... re.py#L125
Deluge later loads it's settings(from core.conf): https://github.com/deluge-torrent/delug ... re.py#L135
Deluge has list of it's own defaults if deleted from config: https://github.com/deluge-torrent/delug ... ger.py#L37

[ambipro]

Does the statesave plugin only save the fastresume states? Could this be leveraged either by incorporating code or by using it as well to force a session save?

[mhertz]

I googled it and found it's mine lol Thanks for the suggestion bro, yes but sadly saving state will not get around this i'm affraid(we cannot purge the conflicting already loaded cached setting out of state again). Appreciate the thought. And of course, please somebody correct me if i'm wrong in something here

[ambipro]

I googled it and found it's mine lol :) Thanks for the suggestion bro, yes but sadly saving state will not get around this i'm affraid(we cannot purge the conflicting already loaded cached setting out of state again). Appreciate the thought. And of course, please somebody correct me if i'm wrong in something here :)

lol that was why I suggested it, because you wrote the plugin.

It would seem to me that applying the entire default, followed by the entire ltConfig.conf would be the best and easiest solution. I will look at Deluge's git just to make sure though.

It's not ideal, but given the custom aspect of this plugin and it's behavior, options are somewhat limited.

Hopefully I can find something....finishing up my taxes right now, will PM you or post back here.

[mhertz]

Funny you say that, I just wen't back to post that I remembered a PR I just rechecked, about applying the default preset before a new preset, or alike - I never really understood the need before when read it, but now because of this dicussion, I kinda get it.

Personally I find this an interesting dicsussion, but as said i'm not sure there is anything to fix, as is sadly the way of the beast or how to put, given the design and usage, and i'm not interested in making bigger workarounds for this here honestly, but as said open to discussing it

I'm not sure this is ideal either, the PR I mean(changed to always be done possibly, before reading config, if possible), because the default preset is hardcoded, and what if away from deluge's settings and if away from the real updated libtorrent values etc etc, so not liking it honestly, but just for conversation

Anyway, appreciate your thoughts as always bro.

Edit: Maybe would help come with the link,.. https://github.com/ratanakvlun/deluge-ltconfig/pull/24