Blocklist plugin fails to import files

[fmar]

I'm trying to get the blocklist plugin to work, but it fails every time I want to use a local file. I tried gzip, zip and a plain text file, none of them works. If I download a file from via http it works fine.

I tried it in Windows and Linux too, different errors but the same results. http download works, file URIs not.
On both systems I use the stable 2.0.3 version. On Debian I use the official Ubuntu version, on Windows the unofficial one.
I also tried deluge-2.0.4.dev67 on Windows, with the same result.

Has anyone been able to get this plugin to work?

Windows:

Code: Select all

Unhandled error in Deferred:
14:52:19 [CRITICAL][twisted                           :154 ] Unhandled error in
Deferred:
14:52:19 [CRITICAL][twisted                           :154 ] Unhandled error in
Deferred:

Traceback (most recent call last):
  File "C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\init.py", line 27, in enable
    return self.plugin.enable()
  File "c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py", line 115, in enable

  File "C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\twisted\internet\task.py", line 194, in start
    self()
  File "C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\twisted\internet\task.py", line 239, in __call__
    d = defer.maybeDeferred(self.f, *self.a, **self.kw)
--- <exception caught here> ---
  File "C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py", line 161, in check_import

  File "c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py", line 444, in import_list

  File "c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py", line 533, in auto_detect

  File "c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\detect.py", line 31, in detect_compres
sion

builtins.OSError: [Errno 22] Invalid argument: 'file:///C:/Users/deluge/AppData/Roaming/deluge/blocklist/ipfilter.v0152.zip'

14:52:19 [CRITICAL][deluge.log                        :204 ] twisted.internet.de
fer
[Failure instance: Traceback: <class 'OSError'>: [Errno 22] Invalid argument: 'file:///C:/Users/deluge/AppData/Roaming/deluge/blocklist/ipfilter.v0152.zip'
C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\init.py:27:enable
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py:115:enable
C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\twisted\internet\task.py:194:start
C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\twisted\internet\task.py:239:__call__
--- <exception caught here> ---
C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\twisted\internet\defer.py:151:maybeDeferred
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py:161:check_import
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py:444:import_list
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py:533:auto_detect
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\detect.py:31:detect_compression
]
14:52:19 [CRITICAL][deluge.log                        :204 ] twisted.internet.defer
[Failure instance: Traceback: <class 'OSError'>: [Errno 22] Invalid argument: 'file:///C:/Users/deluge/AppData/Roaming/deluge/blocklist/ipfilter.v0152.zip'
C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\init.py:27:enable
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py:115:enable
C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\twisted\internet\task.py:194:start
C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\twisted\internet\task.py:239:__call__
--- <exception caught here> ---
C:\Build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\twisted\internet\defer.py:151:maybeDeferred
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py:161:check_import
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py:444:import_list
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\core.py:533:auto_detect
c:\build\deluge-mhertz2021\deluge-2.0.3\lib\site-packages\deluge\plugins\blocklist-1.4-py3.8.egg\deluge_blocklist\detect.py:31:detect_compression
]

Linux:

Code: Select all

Unhandled error in Deferred:
14:41:13 [CRITICAL][twisted                           :154 ] Unhandled error in Deferred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gi/overrides/Gtk.py", line 1641, in main
    return _Gtk_main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/gtkui.py", line 153, in _on_button_force_download_clicked
    client.blocklist.check_import(force=True)
  File "/usr/lib/python3/dist-packages/deluge/ui/client.py", line 551, in __call__
    return self.daemon.call(self.base, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/deluge/ui/client.py", line 500, in call
    return defer.maybeDeferred(m, *copy.deepcopy(args), **copy.deepcopy(kwargs))
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/core.py", line 161, in check_import
    d = self.import_list(self.config['url'])
  File "/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/core.py", line 444, in import_list
    self.auto_detect(blocklist)
  File "/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/core.py", line 533, in auto_detect
    self.config['list_compression'] = detect_compression(blocklist)
  File "/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/detect.py", line 31, in detect_compression
    with open(filename, 'rb') as _file:
builtins.FileNotFoundError: [Errno 2] No such file or directory: 'file:///home/buzz/Documents/Deluge2/ipfilter.dat.gz'

14:41:13 [CRITICAL][deluge.log                        :93  ] twisted.internet.defer 
[Failure instance: Traceback: <class 'FileNotFoundError'>: [Errno 2] No such file or directory: 'file:///home/buzz/Documents/Deluge2/ipfilter.dat.gz'
/usr/lib/python3/dist-packages/gi/overrides/Gtk.py:1641:main
/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/gtkui.py:153:_on_button_force_download_clicked
/usr/lib/python3/dist-packages/deluge/ui/client.py:551:__call__
/usr/lib/python3/dist-packages/deluge/ui/client.py:500:call
--- <exception caught here> ---
/usr/lib/python3/dist-packages/twisted/internet/defer.py:151:maybeDeferred
/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/core.py:161:check_import
/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/core.py:444:import_list
/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/core.py:533:auto_detect
/usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/deluge_blocklist/detect.py:31:detect_compression
]

[mhertz]

I tested this on my system, on linux with 2.0.4.dev56, and I see your issue. I looked at the debug-output, like yours posted likewise, and checked code and was thinking about what could be the issue, while also running some various tests with deluge and the file-uri, like you, and purely by chance, then despite counter-intuitive I decided to end testing that with just adding regular path, and lo and behold that worked, though would be nice if the heading mentioned this, instead of just 'URL' Don't know why or what changed from deluge1, as there's several references to using file-URIs online(and didn't bother compare v1 and v2 plugins, as found workaround anyway).

Inside preferences then it lists 0 byte for the list, but I have md5sum'ed the generated blocklist.cache and they where identical between one generated from URL and one from local file, so that 0 byte notion just is refering to amount downloaded, and not the actual list size persay and the local file also list same amount of imported entries upon startup in lower right corner.

I checked a gzipped list and an unzipped, and both worked, but the gzipped one worked better imho, as the deluge UI would sometimes stall a couple secs if using uncompressed file, and the code also in detect.py has no section for uncompressed, though still worked, but just better with the compressed imho. I didn't tested other formats, but the code shows bzip and zip also should be supported.

I tested with the naunter list here btw: https://github.com/Naunter/BT_BlockLists

Hope helps

[fmar]

Thanks for the testing and feedback @mhertz

I upgraded my Debian unstable VM, but they're still at version 2.0.3 and the import still fails.
I don't know if I want to start building Deluge on linux though
I'll see what I can find out tomorrow, am too tired already.

Yep, compressed or uncompressed should both work. That's at least how I interpreted the detect_compression function.

[fmar]

I don't know why, but if I download the file from http://list.iblocklist.com/?list=ydxerp ... eformat=gz and save it locally, the import works!

But instead of the "file:" URI I used "C:/Users/deluge/AppData/Roaming/deluge/blocklist/ydxerpxkpcfqjaybcssw.gz". Works in dev and stable on Windows.

[dumlavetra]

Hello!

for me blocklist plugin crashes if I using the naunter list https://github.com/Naunter/BT_BlockLists

Code: Select all

[ERROR   ][deluge.ui.web.json_api        :168 ] Error calling method `web.get_torrent_status`: 'NoneType' object has no attribute 'call'
[ERROR   ][deluge.ui.web.json_api        :1512] 'NoneType' object has no attribute 'call'
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/deluge/ui/web/json_api.py", line 160, in _handle_request
    result = self._exec_local(method, params, request)
  File "/usr/lib/python3.10/site-packages/deluge/ui/web/json_api.py", line 122, in _exec_local
    return meth(*params)
  File "/usr/lib/python3.10/site-packages/deluge/ui/web/json_api.py", line 640, in get_torrent_status
    d = component.get('SessionProxy').get_torrent_status(torrent_id, keys)
  File "/usr/lib/python3.10/site-packages/deluge/ui/sessionproxy.py", line 159, in get_torrent_status
    d = client.core.get_torrent_status(torrent_id, keys, True)
  File "/usr/lib/python3.10/site-packages/deluge/ui/client.py", line 551, in __call__
    return self.daemon.call(self.base, *args, **kwargs)
AttributeError: 'NoneType' object has no attribute 'call'
22:57:09 [ERROR   ][deluge_blocklist.core         :448 ] Failed to read!!!!!!
22:57:09 [ERROR   ][deluge_blocklist.core         :1512] [Failure instance: Traceback: <class 'ValueError'>: not enough values to unpack (expected 2, got 1)
/usr/lib/python3.10/threading.py:1016:_bootstrap_inner
/usr/lib/python3.10/threading.py:953:run
/usr/lib/python3.10/site-packages/twisted/_threads/_threadworker.py:47:work
/usr/lib/python3.10/site-packages/twisted/_threads/_team.py:181:doWork
--- <exception caught here> ---
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:238:inContext
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:254:<lambda>
/usr/lib/python3.10/site-packages/twisted/python/context.py:118:callWithContext
/usr/lib/python3.10/site-packages/twisted/python/context.py:83:callWithContext
/usr/lib/python3.10/site-packages/deluge/plugins/Blocklist-1.4-py3.10.egg/deluge_blocklist/readers.py:43:read
]

My workaround for this was to modify readers.py. So the plugin is not able to import IPv6 addresses and IPv4 with a port number. At this time 86 addresses was skipped but without crash and:

Code: Select all

Added 622613 ranges to ipfilter as blocked

Code: Select all

--- deluge_blocklist/readers.py
+++ deluge_blocklist/readers.py.new
@@ -93,7 +93,7 @@
     """Blocklist reader for SafePeer style blocklists"""

     def parse(self, line):
-        return line.strip().split(':')[-1].split('-')
+        return line.strip().split(':', 1)[-1].split('-')


 class PeerGuardianReader(SafePeerReader):

[will_the_canuck]

Hello all.

Although my problem has similarities, it is slightly different, in that the original poster was having issues with the Blocklist plugin failing to import a local file and it seems that the Blocklist plugin is just failing to load ANY file, automatically, even though there may still be cached information on the drive from a previous download of the blocklist information.

So to begin with, I am working in the environment of Linux Mint 20.3 w/XFCE. I did install Deluge 2.0.3 from the Synaptic Package Manager and this issue was happening. I then did upgrade via the PPA and am now running Deluge 2.1.1 with libtorrent version 1.2.16.0. The plugins I have set up here are the Blocklist plugin and the ltConfig plugin. I have set up the URL for the blocklist to be: https://github.com/Naunter/BT_BlockList ... cklists.gz. I do have it set up to check for new lists every 7 days and I have the checkbox marked to import the blocklist on startup.

Now, the problem is that when I start up Deluge, the blocklist is neither downloaded nor loaded automatically from cache files. Yes, I can click on a button within the preferences to download and then import the blocklist, but if it is failing to do it at start up automatically, then it is broken or that is a bug. It can be the equivalent of riding in an automobile without a seatbelt. I'll add that when I used version 1.3.15, I believe it was, on another computer running Lubuntu 18.04, the blocklist would load up automatically on startup and would also pause any active torrents so that they would not run without having the blocklist loaded. So, when or how did this feature break or get broken? And how does one fix this problem?

Thank you and please advise.

will_the_canuck

[mhertz]

I took a look/check, and it still pauses running torrents until list loaded etc, and isn't neither a deluge2 issue. The issue here is what's previously described in this thread by dumlavetra i.e. plugin crashes during downloading/saving list because don't support ipv6 addresses or ports on ipv4 addresses. The blocklist in question has in later times begun introducing said additions of ipv6 addresses and ipv4's with ports, and also on list's github is listed a known issue of earlier transmission versions will ignore the ipv6 unsupported addresses etc.

Anyway, the libtorrent call for adding ipv6 addresses to ipfilter seems straight-forward, same as ipv4 merely infact, but this plugin is pretty advanced with many places checking for proper ipv4 format and done in way over my head honestly, or too cumbersome at-least, so I will not try add ipv6 support myself, but luckily for us, dumlavetra atleast posted a workaround code-fix, to simply ignore the ipv6 addresses and ipv4 with additional ports defined - as he said it's merely e.g. 86 addresses skipped for him, and in same line for me in testing it, so I rebuilt plugin with said nice code-fix, curtezy by dumlavetra, thanks!, and I additionally changed some few other things so the skipped addresses are listed as debug-lines in log now, instead of as actual error's both in terminal and log as before, plus I made the list be downloaded automatically if not available at startup(when having load at startup enabled), so not need go into preferences and manually initiate the download(was issue of when having no list, but still a last_update entry in config-file, like e.g. happen when having run the broken plugin before, or restoring from backup, etc). Thanks again dumlavetra!

Blocklist-1.5.egg (Takes precedence over default included)

[will_the_canuck]

Well @mhertz, first off, I want to say thank you for putting out the 1.5 version of blocklist as it fixes what was missing from version 1.4, in comparison to my experience when using deluge 1.3.15 and the blocklist for that version. Ok, yes. The Naunter blocklist used would be different from the john.bitsurge.net blocklist I once used but as the Naunter one is a current one and the john.bitsurge.net blocklist is one that is no longer around, the code to handle the blocklist has to evolve with the times, so to speak. Even though the john.bitsurge.net blocklist can no longer be downloaded, when I use my computer that is running the old version of deluge, it will still load up the saved blocklist from there and use it. This is something I was expecting with the new version, but alas, it wasn't to be, until you fixed it. I guess I should also put out a thank you to @dumlavetra for figuring out the problem in the first place and coming up with a solution.

So hopefully, if you haven't done it already, I would hope or assume that you would incorporate the new version of Blocklist, 1.5, with the existing available copy of deluge for download, or else there can be many other people like me wondering what is wrong when they select a blocklist, that could be like the Naunter blocklist, and they are having issues like I had. So again, thank you for the good work that you did.

[mhertz]

Appreciate your kind post buddy, thank you! And much welcome of-course

I though am merely a regular forum member like rest of you, so don't control such things I'm affraid. Well I could make a PR and get the devs to consider including officially, but honestly hate doing it, as noob and make mistakes/misunderstand and don't wanna look stupid lol, e.g I believe only half of my submitted ones got accepted(I closed several myself too though, in hindsight, but still), and all was simple, so yeah, that's why usually refrain. Also as this isn't a true fix, but workaround, and not even mine(thanks dumlavetra again).

Anyway, @all, it just needs placed into own profile-folder's plugins dir, and will hence even survive deluge updates, so one time thing, until official 1.5 comes through from upstream.

Thanks for kind post again

[de.luge]

This thread (comes up tops in page rank and) describes a bug that persists to at least release 2.0.3; libtorrent 2.0.8.0.

To replicate, I first updated to latest LMDE, then gzipped existing blocklist file used successfully in two transmission variants (plain text, bluetack, peerGuardian, etc. format). Can also use any legitimate blocklist file downloaded from the web; result is the same.

Next, install deluge (apt install deluge), enable blocklist plugin, paste the blocklist file path in the URL field on the Blocklist Preferences page (either the straight ($HOME/wherever...) or file:///... formats are detected) and click either the Check... or Force... buttons to load the file.

If valid, whether the text file was gzipped or not, a new <filename>.cached file is created, which leads the user to believe that deluge has applied the blocklist. However, the first line of the Info block continues to show:

File Size: 0 B

If the blocklist file is indeed being used, then just a little confirmation would be much appreciated; an exact count of bytes, lines, or IPs is not necessary. But if the blocklist only appears to be used, but is not applied as an IP filter, then notice of that would be most appreciated.

So either way, just need a little feedback.

Much appreciation to all who make this project possible.