Periodic Buffer overflow - Query_volume_information

General support for problems installing or using Deluge
Post Reply
Rudolpher
New User
New User
Posts: 5
Joined: Tue Nov 07, 2017 10:41 pm

Periodic Buffer overflow - Query_volume_information

Post by Rudolpher »

Hello!

I've probably messed up a Libtorrent-setting(ItConfig) to cause this or at least amplify it, but cannot for the life of me figure out what.
Every 6 minutes (Cache expiry time) when the throughput is high(70-100MiB/s), speed drops to zero for half a minute then recovers. Ran Process Monitor to see what goes on when that happens, and during that time it seemingly reads/checks every file of every torrent that is added(≈100, around 6TB) even if the torrent is paused. This is OK for the SSD's, but my poor rust-drive RAID10(T:) takes a little while to serve these requests.
Why is it happening, and what can I do to mitigate or fix this?

Deluge 1.3.15 - 1.0.11.0
Windows 7 64bit

Thanks for your time.

edit: Solved after restart of daemon. Suspect it'll start up again if I resume torrents that are on the RAID. A very similar thing happens if there is any significant throughput(a few MiB/s) on any torrent on the RAID. Disk IO for the RAID gets overloaded for 10+ seconds, then recovers for a short while and repeats. Haven't had a chance to take a closer look at what actually goes on when that happens, but will test further tomorrow.
Any suggestions on what causes this so I can avoid it in the future? Plenty of free system resource(s).

edit2: Problem back without changing a thing. Some time during the night, the periodic check-every-file started back up.
shamael
Compulsive Poster
Compulsive Poster
Posts: 667
Joined: Sat Oct 08, 2016 9:28 am

Re: Periodic Buffer overflow - Query_volume_information

Post by shamael »

Hi

Can you expose the ltconfig settings you changed? if a performance was selected which one?

Let's take the example of performance profile and the cache setting: if you have not the necessary amount of memory some swapping may occur (where is your swap?). A cache size too high may lead to massive I/O.
Rudolpher
New User
New User
Posts: 5
Joined: Tue Nov 07, 2017 10:41 pm

Re: Periodic Buffer overflow - Query_volume_information

Post by Rudolpher »

shamael wrote:Hi

Can you expose the ltconfig settings you changed? if a performance was selected which one?

Let's take the example of performance profile and the cache setting: if you have not the necessary amount of memory some swapping may occur (where is your swap?). A cache size too high may lead to massive I/O.
Swap is not being used, and it is on C:(SSD). Peak memory usage is around 20GB(Guessing Deluged accounts for around 18GB of that), with 12GB free(32GB total).


"settings": {
"connection_speed": 50,
"enable_outgoing_tcp": true,
"cache_buffer_chunk_size": 128,
"guided_read_cache": true,
"active_seeds": 250,
"max_failcount": 3,
"dht_upload_rate_limit": 20000,
"allowed_fast_set_size": 0,
"ignore_limits_on_local_network": true,
"enable_incoming_tcp": true,
"local_service_announce_interval": 300,
"enable_incoming_utp": true,
"unchoke_slots_limit": 500,
"tracker_completion_timeout": 10,
"max_rejects": 10,
"allow_multiple_connections_per_ip": true,
"max_paused_peerlist_size": 1000,
"recv_socket_buffer_size": 1048576,
"optimize_hashing_for_speed": true,
"active_limit": 300,
"num_want": 200,
"file_pool_size": 600,
"cache_expiry": 600,
"write_cache_line_size": 128,
"max_out_request_queue": 4000,
"send_buffer_watermark": 3145728,
"send_buffer_watermark_factor": 150,
"request_timeout": 10,
"send_buffer_low_watermark": 1048576,
"read_cache_line_size": 64,
"max_peerlist_size": 3000,
"mixed_mode_algorithm": 0,
"use_read_cache": true,
"max_suggest_pieces": 50,
"read_job_every": 100,
"max_queued_disk_bytes": 7340032,
"tracker_receive_timeout": 5,
"min_reconnect_time": 20,
"connections_limit": 2000,
"disk_io_write_mode": 1,
"active_downloads": 50,
"suggest_mode": 1,
"max_pex_peers": 100,
"tracker_backoff": 150,
"max_allowed_in_request_queue": 2000,
"auto_upload_slots": false,
"enable_outgoing_utp": true,
"peer_timeout": 30,
"inactivity_timeout": 60,
"listen_queue_size": 50,
"disk_io_read_mode": 1,
"half_open_limit": 100,
"max_http_recv_buffer_size": 6291456,
"cache_size": 50000,
"send_socket_buffer_size": 1048576,
"low_prio_disk": false

It is a mix of "High performance seed"-profile and running through one by one and referencing the libtorrent manual.
shamael
Compulsive Poster
Compulsive Poster
Posts: 667
Joined: Sat Oct 08, 2016 9:28 am

Re: Periodic Buffer overflow - Query_volume_information

Post by shamael »

So your cache is 800MB for 600s, some kind of long expiry time but I'm not sure it can hurt.

Can you monitor the T: disk queue length and latency? Check the disk tab of the Resource Monitor (in task manager) or better a perfmon. It can tell us if the R10 is a bottleneck (what about the read/write cache setting of the Raid?) but anyway I don't' think the buffer overflow is involved. You often see such think in procmon when the size of the initial buffer is too low but then it's adapted, I cannot tell you if it's the case here with trust.
Rudolpher
New User
New User
Posts: 5
Joined: Tue Nov 07, 2017 10:41 pm

Re: Periodic Buffer overflow - Query_volume_information

Post by Rudolpher »

Cache has been all over the place in the last few days, everything from 800MB up to 8GB to see if anything helps. Available memory is not a problem.

That is what made me check out what was going on, through Procmon.
When speed dropped to zero the first few times I checked out Resource monitor, and it showed max queue length for T: while the SSD's spiked for a very short time, then idle. As soon as queue length for T: dropped to 0, transfers resumed(on all drives/torrents).
It is not solely related to torrent transfer through T:, like I mentioned this is triggered by high-bandwidth transfer on any drive so long as T: is slightly used through another torrent. Example -
R:(SSD) has a torrent running at 90MiB/s. Alone this is no problem at all.
T:(RAID) has a torrent running at 1MiB/s at the same time. Suddenly Deluge has a problem and will (seemingly) on every cache flush go through every single added torrent, paused or not, and request information about it. This overflows the IO capabilities of T: (and SSD's but that only lasts for a few seconds), while T: holds up everything for 30s+ trying to serve these requests. The screenshot in OP was captured as the IO spike was happening, and no other program was notably accessing T:.
Rudolpher
New User
New User
Posts: 5
Joined: Tue Nov 07, 2017 10:41 pm

Re: Periodic Buffer overflow - Query_volume_information

Post by Rudolpher »

OK I give up on Deluge until next major release is out. There are multiple bugs at play here, and at this point I'm fairly confident it isn't the settings causing the weird issue mentioned above. Perhaps it is interaction with Intel RST, but would expect more reports if that was the case. To be fair it does not happen with everything defaulted, but suspect that has more to do with Deluge being incapable of high IO or bandwidth without tweaking.

Both client and daemon has serious memory leak issues. I can't let it run for more than 6 hours at a time, any heavy bandwidth usage cuts that time in half and with client running(This is a known issue) that time is further cut in half. It doesn't matter if cache is set to 800MiB or 9GiB, after several hours it has eaten up 15-20GB+. At that point it also seriously slows down transfers and new connections, until after restart(which can take up to a few minutes depending on how long daemon has been running).

Is there a max limit I've been unable to find documented? Any cache setting above 12GiB causes torrents to error(mismatching file size). This never happens if settings is below it, even up to the point where it starts eating into page file because there is no more free memory.

Deluge suffers from the same as quite a few other clients, there is seemingly no system to requested or sent(!) pieces. It causes a shotgun of IO to mechanical drives, and initial seeding on fresh torrents is a pain unless you get enough free cache for the entirety of it. Only choice is setting all to "super-seeding", or none.
Rudolpher
New User
New User
Posts: 5
Joined: Tue Nov 07, 2017 10:41 pm

Re: Periodic Buffer overflow - Query_volume_information

Post by Rudolpher »

As a late addition revisiting this after getting some extra help. I can provide capture logs of everything Deluge does as it happens. I've realized after acquiring better network monitoring tools that this happens with any drive, its just that it isn't apparent at first if it runs on SSD as Deluge averages transfer-speed over the last X second(s).
Image

This is how it looks if torrents placed on mechanical drives are paused, then daemon restarted, and the only active torrents residing on SSDs(RAID0). It still queries files on the mechanical drives, but guessing less information is requested. For comparison, if torrents are active on the mechanical drives that dip drops to 0 and stays there for 30+ seconds as the drives buckle under the amount of request. Doubt it is memory-related, at least it follows the same 3 minute(and some seconds) pattern no matter what cache expiry is set to. No other program or service queries the drives at that time.
Post Reply