Feature Request: Disk Cache

[cbsim]

I did a check at http://en.wikipedia.org/wiki/BitTorrent_client, and it has the cache put to yes, but I couldn't find this feature anywhere in the client as in version 0.5.6.2. So I guess this feature is not implemented yet. Therefore I hope this feature could be included in the future release as most BitTorrent clients now days support this already as a standard to prolong the hard disk life span, Thanks.

[Ux64]

I wonder why some program should have its own cache. Because disk cache is shared resource it's more efficient. And if there is not something seriously wrong, there is no point of having second or third level caches.

Operating System takes care of caching buffering writes and storing already read data and ahead reading so data needed in near term future is already in memory when it's requested by application (in case of linear reading).

I wonder why this is so new to everyone. I used to use HyperDisk disk cache about 15 years a go. And it was the best disk cache ever.

[performance]

Cache would be great. I download at about 800kb/s and the disk is going crazy.

[markybob]

i agree that disk cache should be an OS thing. sounds like some of you need to use hdparm to tweak your settings.

[performance]

Im using windows. I dont see any sort of disk cache from Windows. Right now I'm downloading at 750kb/s and the program is writing at the same amount to HD.

[markybob]

Im using windows. I dont see any sort of disk cache from Windows. Right now I'm downloading at 750kb/s and the program is writing at the same amount to HD.

ah, that explains a lot. none of deluge's developers actually use windows, so it's to be expected that it would perform better on linux. i'll explore this a bit, but no promises.

[performance]

Ok thanks

[Ux64]

Small question to developers. I guess they're doing what is needed anyway.

I expect them having files open all the time. Because windows cache flushes write cache if file is closed.

Reason might be that windows write cache per file (I'm not sure about this) is being flushed if file write is not sequential. Which happens all the time with torrents.

With Ubuntu (7.10 64bit) it seems that system is dumping downloaded data do disk (on my system) about ever two seconds in one big write burst. And in time between there aren't any disk writes even I'm downloading several torrents with total download speed of +800KiB/s.

I was quite happy about that, because I'm used seeing that Windows file (not disk) cache functionality.

I don't know if it really affects anything, but I have also writeback mode enabled. In theory it should help with performance in situations exactly like this. I also have hard drive which supports NCQ (http://en.wikipedia.org/wiki/Native_Command_Queuing) so writing "randomly" to disk isn't as slow as it would be without those (NCQ, writeback) features.

Confirmed, disk activity is very low. Meaning it isn't using much resources. 800KiB/s compared to hard drives 88MiB/s transferspeed is just few percents. I would need to test this with Gibit/s network connection to get really something happening.

Also using compact / full allocation might have some effect. I don't know about Windows environment but you should try it.

[markybob]

Small question to developers. I guess they're doing what is needed anyway.

I expect them having files open all the time. Because windows cache flushes write cache if file is closed.

Reason might be that windows write cache per file (I'm not sure about this) is being flushed if file write is not sequential. Which happens all the time with torrents.

With Ubuntu (7.10 64bit) it seems that system is dumping downloaded data do disk (on my system) about ever two seconds in one big write burst. And in time between there aren't any disk writes even I'm downloading several torrents with total download speed of +800KiB/s.

I was quite happy about that, because I'm used seeing that Windows file (not disk) cache functionality.

I don't know if it really affects anything, but I have also writeback mode enabled. In theory it should help with performance in situations exactly like this. I also have hard drive which supports NCQ (http://en.wikipedia.org/wiki/Native_Command_Queuing) so writing "randomly" to disk isn't as slow as it would be without those (NCQ, writeback) features.

Confirmed, disk activity is very low. Meaning it isn't using much resources. 800KiB/s compared to hard drives 88MiB/s transferspeed is just few percents. I would need to test this with Gibit/s network connection to get really something happening.

Also using compact / full allocation might have some effect. I don't know about Windows environment but you should try it.

compact always does a better job of using disk cache compared to full allocation. but you're absolutely right in your observation that things are fine in linux. basically, we need to switch to memory mapped files to help windows...which will probably happen. no timeline as of yet, though.

[performance]

If and when you guys implement a cache for windows, would it be possible to make it write and clear the entire cache every time? Like say the cache is 5mb, as soon as its filled, deluge writes 5mb onto disk and resets cache to zero.

I'm not a programmer so I dont know the pros and cons of that but with uT and Az, cache size made almost no difference over a certain size because It would write down only a few things in the cache.

Cache is meant to reduce # of disk writes but neither of the 2 did a good job at that, it was more of a delay of disk writes than anything else.