[Plugin] YaRSS2 - RSS plugin with GTKUI support

Suggest, post, or discuss plugins for Deluge
applejuicekiss
New User
New User
Posts: 4
Joined: Sat May 12, 2012 2:12 pm

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by applejuicekiss »

thanks for this. exactly what i needed.
working fine for me, but here is some arbitrary feedback:

deluged 1.3.1 from ubuntu repos (and whatever the default updated libtorrent)
ubuntu 11.04 server
yarss2 v1.0.2.2

it seems to take an arbitrarily long time to add torrents from subscriptions, and so far 2 out of 3 torrents that should have been added have been successfully downloaded. i added 3 ezrss subscriptions which were all already available on the site, and for testing purposes i reduced the update interval to 30 (i know, i'm contributing to those 503 errors... but i'll increase the value, i promise!).

anyway, after getting everything set up the way i wanted i restarted the server computer since it had been several months. about an hour and a half later the first torrent started from a subscription. about 4 hours after that the second one started. i'll take a closer look at the search terms for the third when i have a chance.

one thing i notice is that the "Last Update" field in the "RSS Feeds" tab does not show a value. also in the "RSS Feeds" tab, the "Subscriptions" field does not update properly and still displays "5(0)" which i take to mean 0 out of 5 subscriptions have been filled (i have now added a total of 5 subscriptions). perhaps this is because none are currently active? just a point of interest.

i have one question. i make use of the global deluge setting "move completed folder", so that every completed torrent is moved to a different dirrectory. deluge then has an annoyance where it won't check that second dirrectory if adding a duplicate torrent and will start downloading again from scratch rather than resuming seeding. not a big deal, but i wonder if yarss2 will repeatedly download the same torrent since the downloaded file is no longer in the download directory? perhaps the "last update timestamp" that was mentioned previously takes care of that? anyway, when i have a chance i will play with the per-subscripton "move completed" option, which i think could be more usefull for organizing tv shows.

anyhow, great work. for me, no other features are required.
bro
Top Bloke
Top Bloke
Posts: 364
Joined: Sun Aug 28, 2011 6:46 pm
Location: Norway

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by bro »

applejuicekiss wrote:thanks for this. exactly what i needed.
working fine for me, but here is some arbitrary feedback:

deluged 1.3.1 from ubuntu repos (and whatever the default updated libtorrent)
ubuntu 11.04 server
yarss2 v1.0.2.2

it seems to take an arbitrarily long time to add torrents from subscriptions, and so far 2 out of 3 torrents that should have been added have been successfully downloaded. i added 3 ezrss subscriptions which were all already available on the site, and for testing purposes i reduced the update interval to 30 (i know, i'm contributing to those 503 errors... but i'll increase the value, i promise!).

anyway, after getting everything set up the way i wanted i restarted the server computer since it had been several months. about an hour and a half later the first torrent started from a subscription. about 4 hours after that the second one started. i'll take a closer look at the search terms for the third when i have a chance.
Umm...that is strange. Could it be that the ezrss servers gave a 503 error on the first tries, so that it succeded first after a few tries (i.e. an hour and a half, and four hours)?
If they were both on the same RSS feed, they should both have been added when it first succeeded. YaRSS2 will only download the RSS one time when there are multiple subscriptions on the same RSS feed, so if the server responded properly all matching subscriptions should work.
one thing i notice is that the "Last Update" field in the "RSS Feeds" tab does not show a value. also in the "RSS Feeds" tab, the "Subscriptions" field does not update properly and still displays "5(0)" which i take to mean 0 out of 5 subscriptions have been filled (i have now added a total of 5 subscriptions). perhaps this is because none are currently active? just a point of interest.
Yes, the "Last Update" field not being updated is a bug, or more like a feature that was never implemented properly. Will be fixed in the next version.
The "Subscriptions" field is not explained properly so I understand why nobody would understand the value. The first integer is the number of active (enabled) subscriptions, and the integer in paranthesis is the number of inactive subscriptions.
i have one question. i make use of the global deluge setting "move completed folder", so that every completed torrent is moved to a different dirrectory. deluge then has an annoyance where it won't check that second dirrectory if adding a duplicate torrent and will start downloading again from scratch rather than resuming seeding. not a big deal, but i wonder if yarss2 will repeatedly download the same torrent since the downloaded file is no longer in the download directory? perhaps the "last update timestamp" that was mentioned previously takes care of that? anyway, when i have a chance i will play with the per-subscripton "move completed" option, which i think could be more usefull for organizing tv shows.
You guessed correctly. Each torrent on every RSS feed has a timestamp, so whenever a subscription matches, it will set the "last update" field. Later matches must have timestamp that is newer, so if you delete a torrent from deluge, YaRSS2 will not redownload the torrent. The only way to make YaRSS2 redownload is to right click the subscription and click the "Run this subscription" popup menu.
Maybe adding a menu item for reseting the timestamp would be useful?
anyhow, great work. for me, no other features are required.
Cool. Thanks for the feedback.
When reporting issues, please include any relevant information such as OS (and version), python version (for Windows users this depends on which Deluge installer was used), Deluge version and plugin version.
bro
Top Bloke
Top Bloke
Posts: 364
Joined: Sun Aug 28, 2011 6:46 pm
Location: Norway

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by bro »

crazycaveman wrote:When attempting to grab torrents that include get data in the URL, it fails to pass the URL correctly. For instance, the address

Code: Select all

http://broadcasthe.net/torrents.php?action=download&authkey=removed&torrent_pass=removed&id=152717
is attempted to be downloaded through YaRSS2 with the URL

Code: Select all

http://broadcasthe.net/torrents.php?action=download&ampauthkey=removed&amptorrent_pass=removed&ampid=152717
. Notice the '&amp' when it should just be '&'. I have also included the log file:

Code: Select all

[INFO    ] 13:15:23 rssfeed_handling:203 YARSS: Update handler executed on RSS Feed Episode Torrents :: BroadcasTheNet (broadcasthe.net) upate interval 60.
[INFO    ] 13:15:23 rssfeed_handling:218 YARSS: Fetching subscription 'Once Upon a Time'.
[INFO    ] 13:15:25 rssfeed_handling:63 YARSS: Fetching RSS Feed: 'Episode Torrents :: BroadcasTheNet' with Cookie: '{}'.
[INFO    ] 13:15:25 rssfeed_handling:239 YARSS: Adding torrent:http://broadcasthe.net/torrents.php?action=download&ampauthkey=removed&amptorrent_pass=removed&ampid=152836
[INFO    ] 13:15:25 rssfeed_handling:239 YARSS: Adding torrent:http://broadcasthe.net/torrents.php?action=download&ampauthkey=removed&amptorrent_pass=removed&ampid=152717
[ERROR   ] 13:15:26 torrent_handling:69 Unable to decode torrent file!: invalid bencoding
[INFO    ] 13:15:26 torrent_handling:99 Failed to add torrent Once Upon a Time - S01E20 [ 2012 ] [ AVI | XViD | HDTV | SD | Scene | FastTorrent ] [ Uploader: Anonymous ]  [ Once.Upon.a.Time.S01E20.The.Stranger.HDTV.XviD-2HD ].
[ERROR   ] 13:15:26 torrent_handling:69 Unable to decode torrent file!: invalid bencoding
[INFO    ] 13:15:26 torrent_handling:99 Failed to add torrent Once Upon a Time - S01E20 [ 2012 ] [ MP4 | x264 | HDTV | SD | Scene | FastTorrent ] [ Uploader: BladedThesis ]  [ Once.Upon.a.Time.S01E20.HDTV.x264-LOL ].
[edit] This is on XP. I'll give it a try on Ubuntu and post the results when I can
[edit2] Works correctly in Ubuntu 12.04 (using Win 7 client). I built the plugin from source, since I was having trouble getting the options to show up with the pre-built egg.
I believe the reason the url isn't properly parsed is because the url contains &amp and not &. I belive &amp in the url suggests there is a bug somwhere.
Can you check if the torrent url equals the url printed in the log? (look at the source of the RSS feed)
If e.g. the original url contains & (with a semicolon), then it's the feedparser library that parses it incorrectly.
When reporting issues, please include any relevant information such as OS (and version), python version (for Windows users this depends on which Deluge installer was used), Deluge version and plugin version.
crazycaveman
Member
Member
Posts: 28
Joined: Mon Apr 30, 2012 3:08 am

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by crazycaveman »

bro wrote:
crazycaveman wrote:snip
I believe the reason the url isn't properly parsed is because the url contains &amp and not &. I belive &amp in the url suggests there is a bug somwhere.
Can you check if the torrent url equals the url printed in the log? (look at the source of the RSS feed)
If e.g. the original url contains & (with a semicolon), then it's the feedparser library that parses it incorrectly.
Here is the URL of the torrent I wanted to snatch:

Code: Select all

http://broadcasthe.net/torrents.php?action=download&authkey=removed&torrent_pass=removed&id=157612
This only has '&', not '&', so it seems it's being added for Windows, but not Linux...

Running deluge on my Windows XP machine (YaRSS 1.0.2.2 2.6 egg) doesn't grab the torrent and produces the following log:

Code: Select all

[INFO    ] 12:45:19 rssfeed_handling:203 YARSS: Update handler executed on RSS Feed Episode Torrents :: BroadcasTheNet (broadcasthe.net) upate interval 15.
[INFO    ] 12:45:19 rssfeed_handling:218 YARSS: Fetching subscription 'Testing'.
[INFO    ] 12:45:23 rssfeed_handling:63 YARSS: Fetching RSS Feed: 'Episode Torrents :: BroadcasTheNet' with Cookie: '{}'.
[INFO    ] 12:45:23 rssfeed_handling:239 YARSS: Adding torrent:http://broadcasthe.net/torrents.php?action=download&ampauthkey=removed&amptorrent_pass=removed&ampid=157612
[ERROR   ] 12:45:24 torrent_handling:69 Unable to decode torrent file!: invalid bencoding
[DEBUG   ] 12:45:24 configmanager:111 Getting config 'core.conf'
[DEBUG   ] 12:45:24 torrentmanager:326 You must specify a valid torrent_info, torrent state or magnet.
[INFO    ] 12:45:24 torrent_handling:99 Failed to add torrent UEFA Under 17 Championship - Semifinal - Netherlands vs. Georgia [ 2012 ] [ MKV | x264 | HDTV | 720p | Scene | FastTorrent ] [ Uploader: Anonymous ]  [ UEFA.Under17.Championship.2012.Semifinal.Netherlands.vs.Georgia.720p.HDTV.x264-BALLS ].
Running the deluge daemon on my Ubuntu 12.04 (YaRSS 1.0.2.2 2.7 egg) using deluge on my XP box as a client grabs the torrent and produces:

Code: Select all

[INFO    ] 12:47:02 rssfeed_handling:203 YARSS: Update handler executed on RSS Feed Episode Torrents :: BroadcasTheNet (broadcasthe.net) upate interval 15.
[INFO    ] 12:47:02 rssfeed_handling:218 YARSS: Fetching subscription 'Test'.
[INFO    ] 12:47:08 rssfeed_handling:63 YARSS: Fetching RSS Feed: 'Episode Torrents :: BroadcasTheNet' with Cookie: '{}'.
[INFO    ] 12:47:08 rssfeed_handling:239 YARSS: Adding torrent:http://broadcasthe.net/torrents.php?action=download&authkey=removed&torrent_pass=removed&id=157612
[DEBUG   ] 12:47:09 configmanager:111 Getting config 'core.conf'
[DEBUG   ] 12:47:09 torrentmanager:329 torrentmanager.add
[DEBUG   ] 12:47:09 configmanager:111 Getting config 'core.conf'
[DEBUG   ] 12:47:09 torrentmanager:430 options: {'max_download_speed': -1, 'move_completed_path': u'/home/downloads/', 'max_upload_slots': -1, 'file_priorities': [], 'max_connections': -1, 'download_location': u'/home/downloads/', 'compact_allocation': False, 'max_upload_speed': -1, 'prioritize_first_last_pieces': True, 'auto_managed': True, 'stop_at_ratio': False, 'move_completed': False, 'add_paused': True, 'mapped_files': {}, 'remove_at_ratio': False, 'stop_ratio': 3.0}
[DEBUG   ] 12:47:09 torrentmanager:464 handle id: f602c3a6c51ac13fe119eab725a30a7419da2486
[DEBUG   ] 12:47:09 torrent:105 Creating torrent object f602c3a6c51ac13fe119eab725a30a7419da2486
[DEBUG   ] 12:47:09 configmanager:111 Getting config 'core.conf'
[DEBUG   ] 12:47:09 configmanager:111 Getting config 'core.conf'
[DEBUG   ] 12:47:09 torrent:296 file_priorities len != num_files
[DEBUG   ] 12:47:09 torrent:381 set_state_based_on_ltstate: Downloading
[DEBUG   ] 12:47:09 torrent:382 session.is_paused: False
[DEBUG   ] 12:47:09 torrent:212 Torrent object created.
[DEBUG   ] 12:47:09 alertmanager:123 state_changed_alert: UEFA.Under17.Championship.2012.Semifinal.Netherlands.vs.Georgia.720p.HDTV.x264-BALLS.mkv: state changed to: downloading
[DEBUG   ] 12:47:09 alertmanager:123 torrent_checked_alert: UEFA.Under17.Championship.2012.Semifinal.Netherlands.vs.Georgia.720p.HDTV.x264-BALLS.mkv checked
[DEBUG   ] 12:47:09 torrentmanager:687 Saving torrent state file.
[DEBUG   ] 12:47:09 rpcserver:466 intevents: {0: ['TorrentRemovedEvent', 'PluginEnabledEvent', 'TorrentAddedEvent', 'PluginDisabledEvent', 'TorrentStateChangedEvent', 'ConfigValueChangedEvent'], 1: ['NewVersionAvailableEvent', 'PluginEnabledEvent', 'TorrentResumedEvent', 'TorrentAddedEvent', 'TorrentRemovedEvent', 'PluginDisabledEvent', 'SessionResumedEvent', 'SessionPausedEvent', 'TorrentQueueChangedEvent', 'TorrentFolderRenamedEvent', 'TorrentStateChangedEvent', 'TorrentFinishedEvent', 'ConfigValueChangedEvent', 'TorrentFileRenamedEvent', 'AutoaddOptionsChangedEvent', 'ExecuteCommandAddedEvent', 'ExecuteCommandRemovedEvent', 'SchedulerEvent']}
[DEBUG   ] 12:47:09 rpcserver:470 Emit Event: TorrentAddedEvent ['f602c3a6c51ac13fe119eab725a30a7419da2486']
[DEBUG   ] 12:47:09 rpcserver:470 Emit Event: TorrentAddedEvent ['f602c3a6c51ac13fe119eab725a30a7419da2486']
[DEBUG   ] 12:47:09 core:138 post_torrent_add
[INFO    ] 12:47:09 torrentmanager:513 Torrent UEFA.Under17.Championship.2012.Semifinal.Netherlands.vs.Georgia.720p.HDTV.x264-BALLS.mkv added by user: crazycaveman
[INFO    ] 12:47:09 torrent_handling:102 Succesfully added torrent UEFA Under 17 Championship - Semifinal - Netherlands vs. Georgia [ 2012 ] [ MKV | x264 | HDTV | 720p | Scene | FastTorrent ] [ Uploader: Anonymous ]  [ UEFA.Under17.Championship.2012.Semifinal.Netherlands.vs.Georgia.720p.HDTV.x264-BALLS ].
I set up a clean config on my XP box to test this in and used the same feed URL on both instances. Let me know if you need any more info or help!
bro
Top Bloke
Top Bloke
Posts: 364
Joined: Sun Aug 28, 2011 6:46 pm
Location: Norway

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by bro »

Good news! I found the bug in the feedparser library. It's yet again caused by missing files in the limited Python included on Windows.
So with the XML Parser missing it tries to handle it on it's own, and fails on those urls.

I've reported the bug so hopefully this will be fixed soon.

I'll put out a new version of YaRSS2 with this fixed any minute!

Edit: Ok, v1.0.3 i out. Now it's actually starting to be stable, with no known bugs.
When reporting issues, please include any relevant information such as OS (and version), python version (for Windows users this depends on which Deluge installer was used), Deluge version and plugin version.
crazycaveman
Member
Member
Posts: 28
Joined: Mon Apr 30, 2012 3:08 am

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by crazycaveman »

bro wrote:Good news! I found the bug in the feedparser library. It's yet again caused by missing files in the limited Python included on Windows.
So with the XML Parser missing it tries to handle it on it's own, and fails on those urls.

I've reported the bug so hopefully this will be fixed soon.

I'll put out a new version of YaRSS2 with this fixed any minute!

Edit: Ok, v1.0.3 i out. Now it's actually starting to be stable, with no known bugs.
Awesome! Glad to see you figured it out. On an unrelated note, how far along are you to adding label support? I was able to get it somewhat working, but was having trouble incorporating the changes in to the GUI.
bro
Top Bloke
Top Bloke
Posts: 364
Joined: Sun Aug 28, 2011 6:46 pm
Location: Norway

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by bro »

crazycaveman wrote: Awesome! Glad to see you figured it out. On an unrelated note, how far along are you to adding label support? I was able to get it somewhat working, but was having trouble incorporating the changes in to the GUI.
I haven't looked into that yet. I've been working on adding feature to "Obey "TTL" for RSS feeds, and also fetching and searching the RSS feeds in a separate thread to avoid deluge hanging, but haven't finished yet. After that I guess labels would be the next feature to implement.
When reporting issues, please include any relevant information such as OS (and version), python version (for Windows users this depends on which Deluge installer was used), Deluge version and plugin version.
spazio88

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by spazio88 »

It seems not to work with pb magnet links:

Code: Select all

[INFO    ] 19:15:03 common:118 YaRSS2.rssfeed_handling:207: Update handler executed on RSS Feed 'PB (rss.thepiratebay.se)' upate interval 15.
[INFO    ] 19:15:53 common:118 YaRSS2.rssfeed_handling:64: Fetching RSS Feed: 'PB' with Cookie: '{}'.
[INFO    ] 19:16:11 common:118 YaRSS2.rssfeed_handling:312: Running Subscription 'Prova'
[INFO    ] 19:16:11 common:118 YaRSS2.rssfeed_handling:207: Update handler executed on RSS Feed 'PB (rss.thepiratebay.se)' upate interval 15.
[INFO    ] 19:16:11 common:118 YaRSS2.rssfeed_handling:229: Fetching subscription 'Prova'.
[INFO    ] 19:16:11 common:118 YaRSS2.rssfeed_handling:64: Fetching RSS Feed: 'PB' with Cookie: '{}'.
[INFO    ] 19:16:13 common:118 YaRSS2.rssfeed_handling:251: Adding torrent: 'magnet:%3Fxt%3Durn%3Abtih%3A58A75D2F4CB0339636C799ED57763E1F5A8FE98E%26dn%3DMLB.2012.AL.G1%2F4.%2828%2Bmay%29.DET.Tigers.vs.BOS.Red.Sox.720p'
[ERROR   ] 19:16:13 common:124 YaRSS2.torrent_handling:67: Failed to download torrent: <urlopen error unknown url type: magnet>
[WARNING ] 19:16:13 common:121 YaRSS2.torrent_handling:104: Failed to add torrent MLB.2012.AL.G1/4.(28 may).DET.Tigers.vs.BOS.Red.Sox.720p.
I'm using Deluge 1.3.5 on Ubuntu 12.04

this is the link to the test rss feed:

Code: Select all

http://rss.thepiratebay.se/0
bro
Top Bloke
Top Bloke
Posts: 364
Joined: Sun Aug 28, 2011 6:46 pm
Location: Norway

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by bro »

spazio88 wrote:It seems not to work with pb magnet links:

Code: Select all

[INFO    ] 19:15:03 common:118 YaRSS2.rssfeed_handling:207: Update handler executed on RSS Feed 'PB (rss.thepiratebay.se)' upate interval 15.
[INFO    ] 19:15:53 common:118 YaRSS2.rssfeed_handling:64: Fetching RSS Feed: 'PB' with Cookie: '{}'.
[INFO    ] 19:16:11 common:118 YaRSS2.rssfeed_handling:312: Running Subscription 'Prova'
[INFO    ] 19:16:11 common:118 YaRSS2.rssfeed_handling:207: Update handler executed on RSS Feed 'PB (rss.thepiratebay.se)' upate interval 15.
[INFO    ] 19:16:11 common:118 YaRSS2.rssfeed_handling:229: Fetching subscription 'Prova'.
[INFO    ] 19:16:11 common:118 YaRSS2.rssfeed_handling:64: Fetching RSS Feed: 'PB' with Cookie: '{}'.
[INFO    ] 19:16:13 common:118 YaRSS2.rssfeed_handling:251: Adding torrent: 'magnet:%3Fxt%3Durn%3Abtih%3A58A75D2F4CB0339636C799ED57763E1F5A8FE98E%26dn%3DMLB.2012.AL.G1%2F4.%2828%2Bmay%29.DET.Tigers.vs.BOS.Red.Sox.720p'
[ERROR   ] 19:16:13 common:124 YaRSS2.torrent_handling:67: Failed to download torrent: <urlopen error unknown url type: magnet>
[WARNING ] 19:16:13 common:121 YaRSS2.torrent_handling:104: Failed to add torrent MLB.2012.AL.G1/4.(28 may).DET.Tigers.vs.BOS.Red.Sox.720p.
I'm using Deluge 1.3.5 on Ubuntu 12.04

this is the link to the test rss feed:

Code: Select all

http://rss.thepiratebay.se/0
I'll take a look at it.
When reporting issues, please include any relevant information such as OS (and version), python version (for Windows users this depends on which Deluge installer was used), Deluge version and plugin version.
deluge001
Member
Member
Posts: 37
Joined: Fri Sep 12, 2008 8:36 am

Re: [Plugin] YaRSS2 - RSS plugin with GTKUI support

Post by deluge001 »

Please add an option to specify download location for subscription. Move after completion isn't the same, you know. Imagine a batch torrent which updates with new episodes for example.
Post Reply