Deluge not reporting hash-checks properly

[bengalih]

First off - apologies for the somewhat click-baity title, although based on my experience this is what has happened to me.
I am a long time Deluge user and like so many things about the tool, but this experience has really shaken my trust.

TL:DR: Jump to the bold summary section below.

I have been running Deluge on a Windows Server system in headless mode for years now and access it with thin client on my laptop.
I actually have a couple instances of it running, one for public trackers and another for some private trackers.

My private tracker instance, util recently was only seeding a few torrents for years now. I recently joined another tracker and had begun seeding some new torrents, most of which I had created. However some of the torrents were files I (seemingly) already had which were dead on the private tracker and I wanted to revive them. To do this, I performed the process I have always performed and to my knowledge, the correct one. I download the torrent file and add it to Deluge. I specify the output directory which holds my copy of the files and start the torrent in a paused state. Once the torrent is added, I manually check the hash by right-clicking and choosing "Force Re-check." If the check shows that all files, and the torrent itself is at 100%, then I start the seed.

This is the process I followed for about half a dozen torrents and they all checked 100% ok and appeared to seed. In fact, even the tracker site showed that I was the sole seed for these torrents. However, luckily, someone reached out regarding one of the torrents that they claimed they could not get more than 85% of. This led me down an investigative path. I did multiple tests trying to hash the torrent file against my copy. This included zeroing out some of the files, removing some of the file from the directory, etc. In every instance Deluge would continue to report that the checks on these files were 100%, even when the file didn't exist in the location! The only time I was able to get it to fail is if I removed every single file from the destination folder. If I were then to add 80% of them back and re-check, Deluge would claim 100% of all the files passed the check.

After upgrading and rebuilding and attempting with a totally fresh install this didn't happen. This had me slowly moving pieces from my old configuration over to try and determine the issue. Luckily, I was able to determine that the issue was the session.state file. After I deleted this file, when checking against the current fileset it would fail to hash (as it should have). I then retested all my torrents (close to 200) they all properly hashed with the exception of the half dozen or so which I did not create myself and only attempted to reseed using Deluge to check the hash, which prior to removal of the session.state file all stated they were 100% checked. After removing the session.state file all these files failed, which is probably the proper result since my original source of these files was not this tracker (although the files matched in name and size, and thus why I was dependent upon Deluge to properly tell me if they pass hash check or not).

So, to summarize. Deluge was reporting that files from an added torrent were properly passing hash and allowed them to enter seeding state which reported this to the tracker and thus to peers. In reality though, these files should not have passed hash and seemingly only did so due to a somehow corrupted session.state file. When removed, these torrents no longer pass the hash check. Regardless of almost everything else it seems unthinkable that the client should incorrectly pass hash check on files under any circumstances, especially when forcing that check. If the session.state file somehow allowed it to do so (which it clearly did based on my multiple testings) that would indicate a major issue to me.
As stated, originally I had been running a 2.0.4dev38 version on this machine (probably 3 years...since it was released?) and seemingly had no issues seeding the few torrents I was. During this process I have upgraded to the latest 2.1.1 and, while I did not wipe out my entire config (primarily because I had hundreds of seeding torrents), it too obviously suffered from this issue of incorrectly passing the hash with a corrupted session.state.

I'm hoping that we can understand why this happened, because I'm incredibly fearful of it happening again. I've always preferred Deluge over anything else and really don't want to move off of it, but as I said, I'm really shaken by this (and honestly embarrassed to explain this situation to the tracker site because somehow Deluge allowed me to in essence "spoof" being a seed with bad data!).

Thanks

[bengalih]

After writing the above I went back to do some more digging (I had spent hours on this last night and was a bit burnt, and wanted to get this post up to solicit input from others before moving forward). I believe I have found what the issue was but, while it gives me some sense of relief, I still feel that Deluge should not have allowed this to happen.

I kept a complete copy of the config directory which was exhibiting the problem and went back to take a look at the session.state files. TBH, I'm not really sure the point of the session.state file and why, if it can cause issues and seemingly can be removed without much ill effect, it even exists (or isn't reset when Deluge is restarted).

After comparing the two files I found that the misbehaving one had the following set inside it:
disable_hash_checks

I likely set this option 3 years ago when I was dealing with this issue.
I would have used ltConfig plugin to set it (I don't know of any other way). However, as I have thought - and verified by looking at my ltconfig.conf file, this setting is no longer set in the file. My recollection is I had added it for testing and then removed it, which aligns with the actual data I am finding.

However, seemingly this setting remained cached within the session.state file and apparently was causing none of my files to hash.

There are several considerations here that worry me. The most obvious is that a setting which was removed from all configurations still remained in this session.state file and impacted performance (in a huge way based on this particular setting).

The second is how Deluge responded to this setting. Now, I realize that ltConfig is a plugin and the Deluge interface can't be expected to deal with every possible situation, however the integrity checking of hashes seems like something so important I wish Deluge would behave differently.
Now, according to the libtorrent documentation, the disable_hash_checks is described as:

when set to true, all data downloaded from peers will be assumed to be correct, and not tested to match the hashes in the torrent this is only useful for simulation and testing purposes (typically combined with disabled_storage)

That isn't the best explanation of how it apparently works. Because it seemingly doesn't just deal with "data downloaded from peers". It deals even with local data that you are attempting to hash against the torrent file itself. Additionally, Deluge was giving me no indication in the GUI that hashes were being skipped. Granted, I do believe in retrospect that hashes were finishing more quickly than they should have, but by no means were they eliminated. Even manually forcing the check would result in some progress meter and delay.

Again, I realize asking Deluge to deal with every issue that a user might configure in a plugin is not realistic. However the integrity of the torrent is so essential I believe it would be helpful for Deluge to do some type of check if "disable_hash_checks" is enabled and perhaps write some identifying info into the progress bar when a user does a check (especially a forced one) that would indicate that hashing isn't actually happening.

You could argue and write this off as user error - saying "hey you are the one that put the disable_hash_check setting in", and that is true. However I also disabled it after I was done testing and had no way of knowing that Deluge was keeping this cached in session.state and continuing to skip all hash checks. Perhaps this was the "perfect storm" of issues, but it made for a very dangerous setup and I hope that this makes other people aware. If not too much work, integrating some sort of safety check either of disable_hash_checks specifically, or at least fixing whatever is wrong with session.state would be beneficial.

thanks

[ambipro]

This seems like an incredibly strange series of events, and I understand your concern. This issue boils down, I think, to whether libtorrent settings (what Deluge uses for the actual torrenting) settings should (I guess) be checked on or forced in any way.

On one hand, I see your point with the hash checking being vital to the ecosystem torrenting relies on, but on the other this isn't something that should almost ever be used (disable_hash_check) - except as stated for development/testing purposes. How you came to enable it is a question in and of itself, and leaving it enabled for so long at that.

I could see the argument to add some sort of check/notification for this in ltConfig perhaps, but it seems like such a weird edge-case to worry about at scale. A very quick or near instance hash check on multiple torrents and having previously enabled this setting would seem to be a huge red flag for anyone in the scenario you described, not that I'm trying to place blame here.

You can do pretty much anything with a plugin...and while adding safeguards does seem like an option, there is no way to account for behaviors users configure or misconfigure, or every possible conflicting setting, or anything that accounts for every scenario. At a certain point, you're left with checking logs, trying a clean config, putting back settings incrementally, seeing if it's still behaving this way, and asking for help.

I'm sorry your faith has potentially been shaken in Deluge, but for a comical and taken to the extreme example, this is like saying that an operating system shouldn't make it possible to delete a file because you could delete something important. Unfortunately, there is nothing wrong with the session.state file, it's the state of your configuration for Deluge (viewtopic.php?p=228748#p228748) and you configured it to do this, which in some scenarios could be completely valid. Adding to this, after a long time you forgot and didn't reset it back, you were left confused by the behavior.

I'm glad you've figured it out, but at the same time given the inherent granular configuration of Deluge and further employment of custom libtorrent settings with a third-party plugin, it's concerning to see you believe that this is anything other than an accidental user configuration issue.

I hope you continue to use Deluge, and I hope this post doesn't come across as an attack - as it is not meant as such - merely observations and my perspective on the issue you faced as a whole.

[bengalih]

On one hand, I see your point with the hash checking being vital to the ecosystem torrenting relies on, but on the other this isn't something that should almost ever be used (disable_hash_check) - except as stated for development/testing purposes. How you came to enable it is a question in and of itself, and leaving it enabled for so long at that.

You are focusing on the minor point/issue here which should not be obscuring the bigger point.
I explain above (with link to my old post) specifically why I enabled this to test. I also explain that I disabled this years ago once my testing was done and verified that the setting no longer exists within the ltconfig configuration files. (to be clear, I mean I turned the disable hashing on to test, and then I removed it when I was done testing).

Nonetheless, the setting was still being enforced because it was stuck within the session.state file.

That is really the issue. The fact that this particular setting was so potentially damaging to the integrity of every torrent file that passed through my system after that makes it more severe.

In short, the *main* issue is why would this setting have persisted in session.state even after it had been removed from the ltconfig setup?
It persisted for years...through multiple restarts of Deluge and the system itself. There is not a lot of info on the session.state file and why it even exists (since it seems one is always told they can delete it without issue).

This is the main issue and I don't believe that I can be blamed for setting this option for an hour while I wanted to test why Deluge was constantly rehashing my torrents, when I followed the same steps to remove the setting and verified it was removed from the ltconfig file.

[ambipro]

I'm not sure what to say as to why the setting persisted, I've never had any issue with settings in ltConfig not changing when set. Have you looked in your ltConfig.conf file to see if the setting was there (in your snapshot of the broken configuration, not in the current one, and also checking to see if its set to 0 rather than 1, and not just absent - if that makes sense)

I believe it's possible (although without looking at the actual .py code and knowing whether you did this or not, is just speculation) that instead of disabling the disable_hash_check setting, applying these changes, and then disabling the option altogether, you simply unchecked the main option checkbox without setting the "disabled" setting - which I think just prevents it from being set to whatever is in ltConfig.conf going forward, not resetting it to any value (default or otherwise) specifically. If you disabled it, and didn't apply the settings, and then disabled the option altogether, it's possible that it was never actually turned off. That's just based on my experience with using the plugin and some speculation though, as I said I haven't looked at the plugin's code extensively in this regard, have only used it quite a bit and glanced a good while ago at its code.

I don't think I'm focusing on a minor point, Deluge is widely known to be the most extensible and configurable client out there. What you're describing is something no one else I'm aware of has reported in the ltConfig plugin thread or otherwise as far as I can tell. As I said, your situation, while unfortunate, seems potentially to be just a bad series of events you didn't execute "properly" to disable the setting after your testing.

If you really want to address the latent settings issue, because you think it's actually a bug, unless you can give some sort of steps to reproduce the latent settings being saved as you describe them being, I'm not entirely sure what would be done shy of forcing users to use certain settings or putting constraints on what is configurable natively and via plugins.

As I previously mentioned, you can do almost ANYTHING inside of a plugin. Yes libtorrent settings are critical to the application, but creating a plugin framework that is hamstrung in what it can do isn't a solution I would personally like to see employed.

session.state file is something used by libtorrent's initialization and configuration if my memory serves me, and I believe the data originates from libtorrent as well and is just encoded for Deluge to use when initializing the library. While you can delete it, you will lose information that is stored in it. This doesn't negate its utility just because things don't catastrophically break with it missing. There are defaults, after all.

[bengalih]

I've gone ahead and continued the conversation here in the ltconfig thread:
viewtopic.php?p=237825#p237825

This is due to a several years old conversation that I had with mhertz there about what seems to be related issue.

As was discussed if you follow that conversation, it was recommend by mhertz (and others outside of that thread) to manually edit the ltconfig file. This was due to the GUI having issue setting there (which, years later it still does). There is a workaround which I actually was the one who shared with mhertz in that thread. Anyway, since directly modifying that file was an option, and seemingly one that was blessed by mhertz, it is the option that I used since it was the most straightforward. (and yes I only edited the file when Deluge was shut down).

It is clear, and reproducible that editing that file to remove a setting does not work properly because that setting is apparently also stored within session.state and reapplied at next start.

So, to be clear, lets say you have this snippet in your ltconfig.conf file:

Code: Select all

    "apply_on_start": false,
    "settings": {}

Which is the default setting, meaning no settings were enabled. You then go ahead and add the following:

Code: Select all

 
     "apply_on_start": false,
    "settings": {
        "dht_upload_rate_limit": 20000,
        "disable_hash_checks": true,
        "file_pool_size": 500
        }

Regardless of how it was set there, but lets say you initially set it in the GUI.
Then, after testing you go and modify the file manually to change it to this:

Code: Select all

 
     "apply_on_start": false,
    "settings": {
        "dht_upload_rate_limit": 20000,
        "file_pool_size": 500
        }

Any rationale person used to any type of configuration file dating back decades would rightly assume that the disable_hash_checks setting would now be set back to its default. This is because it is no longer being specifically set in any configuration file and because the default setting is false/disabled.

But, indeed this is not what happens because the file was manually edited it does not seem to be removed from the session.state data and therefore continues to persist despite it being removed from the plugins configuration file. Your second paragraph above speaks a bit to this issue, but by your own admission you don't even know for sure. I haven't interacted with you before (I don't believe), but you seem to be a moderator and despite not being a member for even 2 years at this point I would like to make the assumption that you are fairly knowledgeable on the product. You can see therefore that your own confusion and/or lack of understanding how it works offhand would translate to even more ignorance on the part of your average (and even not so average) user.

The explanation you give in your paragraph has been known as a sort-of workaround since before your time here. Yes, people have always been confused/not-sure of why sometimes you can't seem to simply uncheck the ltconfig option that you previously set. It seemed sometime you had to force it by not unselecting it, but keeping it selected and manually setting the value you wanted it to have. A procedure that should not have to be done when you simply want to revert something to its default settings. But, AFAIK no one has been able to explain or verbalize why this is. Well, I think I just have...it is because of the session.state file.

So, indeed, this has been discussed in the ltconfig thread in the past (just before your time), and apparently it caused confusion even among the experts back then and nothing has been done to address it since. I think that this just needs to be discussed openly and verified by someone other than me (which I can't believe is difficult as there is nothing very strange about my setup). Even if it can't be verified, it is clearly happening (and easily reproducible) on my end. It should be acknowledged and at the very least given more prominence so people are aware this is a possibility. I mention some other possible solutions regarding deleting session.state in my post in the ltconfig thread.

I think it is best that if you would like to respond to my concerns above you continue in the ltconfig thread since I have begun discussion there. On your prerogative you might even wish to merge some of these posts into that thread (although I have linked to this thread from there for sake of clarity).

I'm a stan of Deluge, but I'm not an apologist for a program which misbehaves or exhibits unknown/inconsistent behavior. I realize that the difference between a bug and a "feature" is how well it is known and documented and I don't feel like my concerns can simply be dismissed as user error.

[ambipro]

Briefly, this is not my first account here, not that that's important, and I've been using Deluge for WELL over a decade at this point. I don't know how that's relevant but your repeated mentions of this accounts age and "before my time" almost feels like a jab, I hope it's not intended as such. I, myself, am a developer. My github account - which is also not my first - is on my profile here and you can see many related projects and scripts. I only mentioned that I don't know for sure about the plugins nature, but suspect it is as I described, based on behavior I've seen. I can go and look in the source myself, but I just assume wait for @mhertz to chime in and potentially fix this if he wants. I've got open PR's I'm trying to merge on other projects and am currently dealing with the Flu anyway.

I'm not sure you're taking my posts as I intend, either by the misfortune of being sick and poorly communicating, or just that you disagree or feel attacked or something. I'm not saying you're entirely to blame, the behavior in this case could be more straight-forward, but as I said - I think what you're describing is reproducing exactly what I described with disabling the option entirely in the GTK-UI before setting it back.

I do agree, that if the option is not included in the ltConfig, when it's disabled entirely the defaults from other settings or libtorrents defaults could/should be restored in a general UX sense. I just don't think that's how it operates and knowing after the fact what settings were once in the ltConfig and what are missing NOW (editing the config directly) is probably the main hurdle to address. You could set EVERYTHING back to defaults on every startup, but I'm not entirely sure this is an overall solution either and could possibly affect settings in other plugins or options in Deluge. I just don't know all of the implications without doing some testing and investigating further, it's not my project or code. (For instance, I recall you can change the connection_speed in ltConfig and it will reflect in the main preferences settings - not in ltConfig alone in connections/sec. So if you unset this and it is set to defaults, you could be affecting other settings.)

The thing to really understand, is while mhertz has taken on maintaining some of the aspects of ltConfig now, it is still a third party plugin and blaming it inherently on Deluge as a program seems somewhat misguided. You can do almost anything in Python, so if I wrote a plugin to wipe your C: drive, and you used it, it isn't necessarily Deluge's fault....The plugin needs to be addressed. That's kind of my overall message. It doesn't really track to expect Deluge to manage every aspect of every potential plugin and changes it can make without adding severe limitations that could potentially hamper development. Just my opinion, though...Just like everything else in this post.

edit: Also if apply on startup is not set, your changes wont be applied at startup. Not sure if that's how you had it. Just had to mention.

[bengalih]

So, first off. Yes I was addressing your account's age as a possible factor in your insight into this information. Not as a dig however, simply I wouldn't have expected an account that was only 2 years old to be aware of every topic that was discussed prior. When I had that prior conversation in the ltconfig thread with mhertz, I regarded him as very knowledgeable on the topic and, despite that, he provided incorrect information regarding how ltconfig stored its data (which he later amended after I guess he remembered or found out from elsewhere). Point being is that no-one can know everything and I was just pointing out that there were older threads which related to this and I thought that pointing out they were prior to your time (at least this accounts) was relevant since you might have been unaware. So, while I did mean to imply that the age of your account might indicate some lack of knoweldge on the existence of earlier threads, I did not mean this in a derogatory way. Hope we are squared there.

Next, I wouldn't say I feel attacked, but I did feel a bit dismissed that this result was due to "user error." While clearly I, at some point, made the changes that ultimately resulted in the behavior, the way the program behaves to allow this is simply not good design. To add to this point (and clarify it), realize that when I started this thread I was not sure what the root cause is. The fact that it is now determined to be ltconfig plugin, and your observation that ltconfig is a plugin and not a core part of Deluge is a valid one. We should not blame Deluge for a plugin which is misbehaving. OTOH, Deluge's appeal (and downfall) is that it is built on plug-ins. If it were not for all the plugins available many people would have long left for other clients which incorporate already much of the functionality you can only get from plugins. On top of that ltconfig is perhaps one of the more "important" plugins as Deluge cannot really be considered a serious seeding client without it (due to max torrent limitations and other tuning). So, technically you are correct - this is an ltconfig plugin. But I don't feel we can absolve Deluge completely for enabling the behavior.

Look, the fact is we both seem to agree (based on your last post) that the way this works isn't ideal. We both also agree there is confusion and ignorance about it. I'm not a developer by trade, but am fairly technically inclined and have also been a Deluge user for at least as long as you and it seems difficult for us to even understand without additional reference/testing. As mentioned I've stuck it out with Deluge (which as a Windows user has been tough) and I'd like to continue doing so. Even though I seem to have discovered the root cause and remediated it on my end, the issue is still there. As people who are invested in Deluge to one degree or another my purpose here is to now share with the community and try to enact some type of positive change which might result in this behavior being minimized - be that by actual changes in code, better documentation, or better visibility to the user base.

If all I cared about was fixing *my* issue, I'd be gone already. As someone that has been in the IT industry for close to 30 years, some of that time dealing with product development, I don't think this is something that should just be dismissed as "Not Deluge's fault" or "You did it wrong."
Maybe you, mhertz, and the other people that can actually enact changes around here disagree, in which case only these threads will serve as warning for the intrepid user who is trying to figure out why their configuration is corrupted down the line. I hope though that my experience and validation of this issue is enough for some movement forward so the next user doesn't have to worry about it.

[ambipro]

I appreciate the clarification, and while I hoped as much - I was not entirely sure. We are square.

I never meant to imply that this was all simply boiled down to user error, as I said I'm dealing with the Flu (day 3) and my communications might be slightly less than optimal, but merely a combination of unintended behavior and temporary user configuration and simple "bad luck" if you will.

I think the solution I've come to the conclusion of, which I will reiterate to mhertz in a PM as well as the ltConfig thread, is that upon disabling an option entirely with the "main" checkbox next to the name of the option, the "Default Libtorrent Settings" profile should be read, that setting previously there should be replaced with it in the .conf, and applied to libtorrent, which should then be reflected in the session.state folder. Going forward, it would be irrelevant that the setting persisted (or was removed if you edit it manually) - it might not even be necessary to write it to the conf file, but simply force it in libtorrent. session.state should be updated thereafter, I believe its ever 3 minutes?

I understand the concerns, as I stated, but I think in general this can be addressed in ltConfig fairly easily, either as I described above or similarly, and mitigate any future issues assuming you are on an updated build of that plugin.

Beyond that, as I said, I don't think this is something inherently wrong with the architecture of the plugin system or versatility therein.

[bengalih]

Yes, as I mentioned from the get-go this could have been a "perfect-storm" type of scenario, and I also agree that this doesn't indicate that something is inherently wrong with the core architecture of Deluge or the plugin system. From s development standpoint, I don't know what limitations a plugin has over native functionality. My assumption (quite possibly wrong) is that the session.state file needs to be written with the ltconfig information because libtorrent somehow gets initialized to read this file before it can be read/loaded from ltconfig.conf.
Whether that's true or not, the only real drawback to the plugin system is that you have many pieces maintained by different people who sometimes go off the grid. I found the Github for ltConfig, but it hasn't been updated in 3-4 years and the issues appear to be stale.

I know that there has been some outside development on it, but (at least in the case of ltConfig) it seems that the updated file is usually just buried in a forum post somewhere as opposed to being forked out and easily found on GH along with a README and any other issues that might come along with that.

I'm not sure I 100% understand your proposed solution above, but from my reading on it I would disagree in some respects. Perhaps your solution is based on your more intimate understanding of the limitations of what can and can't be done in the code, but here is where I would disagree:
I think removing the existence of the .conf file would be a bad idea (and that includes not writing the actual configuration to that file as you seem to infer might not be necessary). Based on how all the other plugin architecture seems to work, I believe that manual modification of a .conf file should be a 100% legitimate way to modify a setting. For instance, you should be simply able to copy over a .conf file from one setup to another and expect it to work. Additionally, due to the special case here of the ltconfig GUI being wonky, direct editing (and honoring) of the config file is, I feel, important to preserve.

I think I see your strategy of reading all the libtorrent defaults and applying those directly when they aren't explicitly stated within the .conf. I don't think that is the worst idea, but it also seems like a band-aid to force in default values that should already be, well...default.
Again, I'm sure I don't know the code as intimately as you (as I have never even tried to look into it), but to my layman's eye it would seem that the issue is with the session.state not being updated. When and how does this file get modified? Wouldn't it be better to simply re-write the session.state data whenever any change is applied to the ltConfig plugin? If the session.state is truly transient and can be removed with no ill-effects, shouldn't ltConfig simply delete that file on startup/shutdown so that it can be recreated with the correct settings defined in the .conf file? Again, I'm just throwing stuff out there, but my main point is that I feel whatever is defined in the .conf file should be honored and that means that if a setting is not defined it should default to libtorrent defaults. That follows the architecture of both the plugin system and the purpose of a .conf file and how pretty much anyone (IT savvy or not) would probably expect a configuration file to behave (i.e. only set the settings that are defined, otherwise don't change how they normally work).

Ofc, I don't think that either you or I should probably be the sole voice of decision here, so hoepfully we can have discssion.
However, most important thing is to get over the flu! We can continue discussion once you are on the mend.
I don't consider myself very active here unless I am having an issue with something, but I will try to keep checking in over the next couple of weeks until we have some sort of traction on best way to resolve and can get some more insight on how session.state actually functions and its necessity.

thanks