IP blocklist support patch

[shirish]

Please look at this post http://dev.deluge-torrent.org/ticket/370 which says the importance of having the blocklist importer.

Also where is this blocklist.cache situated? Also it would need loads to improvement. While users like me would be ok in importing manually but that too only if clear instructions are provided. These are the things which I found when I loaded the plugin.

there is blocklist.conf file in .config/deluge

Code: Select all

(dp0
S'url'
p1
S''
p2
sS'listtype'
p3
S'text'
p4
sS'load_on_start'
p5
I00
s.

While I did not fetch the url (the defs) , this is what is generated on the CLI :-

Code: Select all

Loading blocklist plugin ...
Fetching 
Traceback (most recent call last):
  File "/usr/share/deluge/plugins/BlocklistImport/ui.py", line 69, in ok
    self.plugin.setconfig(url, los, ltype)
  File "/usr/share/deluge/plugins/BlocklistImport/__init__.py", line 106, in setconfig
    self.loadlist(fetch=True)
  File "/usr/share/deluge/plugins/BlocklistImport/__init__.py", line 76, in loadlist
    reporthook=self._download_update)
  File "/usr/lib/python2.5/urllib.py", line 89, in urlretrieve
    return _urlopener.retrieve(url, filename, reporthook, data)
  File "/usr/lib/python2.5/urllib.py", line 222, in retrieve
    fp = self.open(url, data)
  File "/usr/lib/python2.5/urllib.py", line 190, in open
    return getattr(self, name)(url)
  File "/usr/lib/python2.5/urllib.py", line 459, in open_file
    return self.open_local_file(url)
  File "/usr/lib/python2.5/urllib.py", line 473, in open_local_file
    raise IOError(e.errno, e.strerror, e.filename)
IOError: [Errno 2] No such file or directory: ''
Cancelling

The best thing would be to tell :-

1. The formats which it supports, both compression as well as data formats (e.g. .gz & .dat)
2. Where this blocklist.cache
3. A way either as a % either in CLI or the GUI with some kind of ETA so that the blocklists are actually done.
4. Various sites from where the blocklist can be used. Hiding resources does not help the user.


AFA your concern is concerned, while its true, if the defs. are installed, then one can easily shutdown & restart deluge.
A good way to know would be something in the extreme right-hand corner with telling date of the blocklist either downloaded
or installed.

Just my 2 paise

[tarka]

Please look at this post http://dev.deluge-torrent.org/ticket/370 which says the importance of having the blocklist importer.

That doesn't appear to have anything to do with blocklists?

Also where is this blocklist.cache situated?

In the config directory, but you shouldn't really need to access it directly.

Also it would need loads to improvement. While users like me would be ok in importing manually but that too only if clear instructions are provided. These are the things which I found when I loaded the plugin.

there is blocklist.conf file in .config/deluge

That's the configuration file. It's a pickled python object, you probably don't want to touch that unless you know what you're doing.

While I did not fetch the url (the defs) , this is what is generated on the CLI :-

Sorry, are you saying you pointed it at a file that didn't exist?

1. The formats which it supports, both compression as well as data formats (e.g. .gz & .dat)

This is in the description and when importing lists you get a choice of format.

2. Where this blocklist.cache

You shouldn't need to know this, it's an implementation issue and may change.

3. A way either as a % either in CLI or the GUI with some kind of ETA so that the blocklists are actually done.

Currently there's a download feedback bar and an import activity bar. I haven't implemented a percentage progress bar for the import phase as this would require scanning the whole file before starting the import, which would make the import take longer. But I'll will have a look at it later and see exactly how long a scan takes.

4. Various sites from where the blocklist can be used. Hiding resources does not help the user.

Sites and mirrors change. I think the project wiki would be a better place to store this info. Feel free to start a page of links.

[shirish]

While I did not fetch the url (the defs) , this is what is generated on the CLI :-

Sorry, are you saying you pointed it at a file that didn't exist?

I did not point to anything. What I simply did was select the blocklist importer & did not give any url
and cancelled the downloading . Even then the blocklist importer tried to import stuff.

What I was saying about importing is how its with utorrent under windows. One can directly put
the ipfilter.dat in C:\Docs and Settings\Administrator <or any profile>\utorrent\
place the file there & in the next instance of utorrent up, it takes the new ipfilter by default.
While GNU/Linux is different is there some way in which a similar implementation could be done?

Also the blocklist.cache which is supposed to be in the config directory, is this done only after an initial import happens or is it something which is there the moment the blocklist importer plugin is turned on by default.

I have not downloaded the defs. file but this is how it looks in my config directory if its the same config directory . If I should be looking somewhere else feel free to guide me.

Code: Select all

shirish@ubuntu:~/.config$ ls -la
total 52
drwx------  9 shirish shirish  4096 2007-06-23 06:42 .
drwxr-xr-x 68 shirish shirish 12288 2007-06-24 07:26 ..
drwx------  3 shirish shirish  4096 2007-06-23 23:29 deluge
drwx------  2 shirish shirish  4096 2007-06-24 08:27 gtk-2.0
drwxr-xr-x  2 shirish shirish  4096 2007-06-12 14:10 mirage
drwx------  2 shirish shirish  4096 2007-06-09 09:57 mousepad
drwx------  2 shirish shirish  4096 2007-06-24 08:27 Thunar
-rw-------  1 shirish shirish   630 2007-06-23 06:42 user-dirs.dirs
-rw-r--r--  1 shirish shirish     5 2007-06-23 06:42 user-dirs.locale
drwxr-xr-x  8 shirish shirish  4096 2007-06-22 09:40 xfce4
drwx------  2 shirish shirish  4096 2007-06-24 02:36 xfce4-session

Of course this time I am in xfce as I keep moving between XFCE & GNOME

Another thing which I perhaps did not make perfectly clear is that it would be a good idea to use the status bar (currently quite empty except for Connections, download & upload info. you could use the right-side corner to have date & maybe format of the last blocklist import.

Something like

Code: Select all

blocklist importer plugin: 0.3 2007-06-23 or BIP: 0.3 blocklist 2007-06-23  

or something similar. The date there would be the last modified date and the user would know at all times when he last imported/updated the blocklist.

Edit:- Made a small intro. for the blocklist plugin at http://dev.deluge-torrent.org/wiki/BlocklistPlugin . Did not write any implementation things as I am myself unclear as to :-

1. time taken for import
2. which lists are best compatible with the plugin as of now,

Testing also the new import for emule blocklists. r733

[shirish]

Ok I made clean, built deluge from scratch, built release r733 (which uses emule .gz) so downloaded a filter file to see. This is what is being shown on the CLI

Code: Select all

Showing window
Loading blocklist plugin ...
Fetching /home/shirish/pipfilter.dat.gz
importing with gzmule
GZMuleReader loading /home/shirish/.config/deluge/blocklist.cache
MuleReader loading
TextBase loading

The importing tool is still working for about an hr. Is the plugin still doing it or has done its work is not known? After some time it stopped showing anything, does it mean the filter database is installed? I turned off the plugin & turned it on again, the importing dialog box is at it again, if the import has been done successfully at the last time, it should tell me that a blocklist is present there (some kind of checking required here perhaps? ) and then ask if I want to update the blocklist .

Right now it shows like this :-

Code: Select all

Showing window
Loading blocklist plugin ...
Fetching /home/shirish/pipfilter.dat.gz
importing with gzmule
GZMuleReader loading /home/shirish/.config/deluge/blocklist.cache
MuleReader loading
TextBase loading
Loading blocklist plugin ...
importing with gzmule
GZMuleReader loading /home/shirish/.config/deluge/blocklist.cache
MuleReader loading
TextBase loading

Atleast a good way should be to have some kind of statement either in the GUI or the CLI saying import completed. Dunno if it is there or not.

Another thing, the progress bar keeps moving back & forth much like apport (ubuntu reporting tool) does , What is required is for sure, the progress percentage bar otherwise one does not know if & when it does it work.


Put tickets #373 as well as #374 for the same.

[tarka]

I did not point to anything. What I simply did was select the blocklist importer & did not give any url
and cancelled the downloading . Even then the blocklist importer tried to import stuff.

Well, obviously that's not going to work. However one of the things I have to do is improve the error handling.

What I was saying about importing is how its with utorrent under windows. One can directly put
the ipfilter.dat in C:\Docs and Settings\Administrator <or any profile>\utorrent\
place the file there & in the next instance of utorrent up, it takes the new ipfilter by default.
While GNU/Linux is different is there some way in which a similar implementation could be done?

You can basically do this already. The URL can be a local file, so all you would have to do is point it at the file you downloaded.

Also the blocklist.cache which is supposed to be in the config directory, is this done only after an initial import happens or is it something which is there the moment the blocklist importer plugin is turned on by default.

All the importer does is download the blocklist and save it as blocklist.cache. The format of this file is dependant of the type of blocklist you selected.

I have not downloaded the defs. file but this is how it looks in my config directory if its the same config directory . If I should be looking somewhere else feel free to guide me.

You'll notice that there is a 'deluge' directory. The configuration and cache go in there. However nothing in that directory is human-editable (without knowing a fair amount about python pickling format and the blocklist format.

Another thing which I perhaps did not make perfectly clear is that it would be a good idea to use the status bar (currently quite empty except for Connections, download & upload info. you could use the right-side corner to have date & maybe format of the last blocklist import.

Yep, that's something I'd like to add, however I'm currently concentrating on adding formats and cleaning up what's already there.

Edit:- Made a small intro. for the blocklist plugin at http://dev.deluge-torrent.org/wiki/BlocklistPlugin .

[/quote]
Cool, thanks.

[tarka]

Ok I made clean, built deluge from scratch, built release r733 (which uses emule .gz) so downloaded a filter file to see. This is what is being shown on the CLI

I'm not sure what's happening here, but I'm working on a patch that has better error handling, which should help debug this.

[shirish]

Ok I made clean, built deluge from scratch, built release r733 (which uses emule .gz) so downloaded a filter file to see. This is what is being shown on the CLI

I'm not sure what's happening here, but I'm working on a patch that has better error handling, which should help debug this.

Looking forward to it while hopefully you can do also something about the the importing thing happening each time. It is going to be a nuisance rather than an asset for users if its going to take 20-30 mins. each time (showing importing) if it did the thing the first time around. But unless we have the patch which gives better error handling & completion statements on the CLI, nothing can be said.

Edit: For now I have turned off the plugin

[tarka]

Looking forward to it while hopefully you can do also something about the the importing thing happening each time. It is going to be a nuisance rather than an asset for users if its going to take 20-30 mins. each time (showing importing) if it did the thing the first time around. But unless we have the patch which gives better error handling & completion statements on the CLI, nothing can be said.

I'm afraid that's probably always going to be the case. libtorrent does not store the lists in-between runs, therefore the list must be imported each time. These blocklists have in the region of 180,000 IP ranges in them; without support in libtorrent for storing them in some sort of external DB it's just going to take a long time.

[shirish]

aha, ok. perhaps we can make a case for libtorrent then? There is http://code.rasterbar.com/libtorrent/timeline
where a ticket could be made but I do not know anything about libtorrent so I am not the person to do this.
Although have heard quite a bit about sqlite if this is something that could be used perhaps?

http://technology.guardian.co.uk/weekly ... 39,00.html

One thing, while svn to 735 this is what came up :-

Code: Select all

svn up
A    plugins/BlocklistImport/nipfilter.dat.gz
A    plugins/BlocklistImport/pg.txt
U    plugins/BlocklistImport/ui.py
U    plugins/BlocklistImport/__init__.py
A    plugins/BlocklistImport/test.py
A    plugins/BlocklistImport/splist.txt
Updated to revision 735.

Now I use the pipfilter.dat.gz instead of the nipfilter.dat.gz . In utorrent what I do manually is rename the pipfilter.dat to ipfilter.dat & it works. I do not know if one can use the pipfilter.dat anymore or just have to nipfilter.dat

[shirish]

I do not know what happened but this is the error which has come up with the latest svn 735. The error is persistent. I even removed the deluge config file (just in case) also but still the same error happens :-

Code: Select all

shirish@ubuntu:~$ deluge
no existing Deluge session
Starting new Deluge session...
deluge_core; using libtorrent 0.13.0.0. Compiled with NDEBUG value: 1
Applying preferences
Starting DHT...
No DHT file to resume
/usr/lib/python2.5/site-packages/deluge/core.py:757: DeprecationWarning: integer argument expected, got float
  PREF_FUNCTIONS[pref](self.get_pref(pref))
Scanning plugin dir /usr/share/deluge/plugins
Loading module ExamplePlugin
Initialising plugin ExamplePlugin
Loading module TorrentCreator
Initialising plugin TorrentCreator
Loading module TorrentSearch
Initialising plugin TorrentSearch
Loading module BlocklistImport
Traceback (most recent call last):
  File "/usr/bin/deluge", line 93, in <module>
    start_deluge()
  File "/usr/bin/deluge", line 67, in start_deluge
    interface = deluge.interface.DelugeGTK()
  File "/usr/lib/python2.5/site-packages/deluge/interface.py", line 65, in __init__
    self.plugins.scan_for_plugins()
  File "/usr/lib/python2.5/site-packages/deluge/plugins.py", line 60, in scan_for_plugins
    mod = __import__(modname, globals(), locals(), [''])
  File "/usr/share/deluge/plugins/BlocklistImport/__init__.py", line 75
    filename, headers = urllib.urlretrieve(self.config.get('url'),
           ^
IndentationError: expected an indented block