I would like to request a way to do full pre-allocation - either the fancy way using the fallocate() posix call, or just simply writing out a bazillion zeros instead of the seek trick for creating sparse files.
I've got about 4TB of torrented material and it is massively fragmented due to the sparse allocation method. It's on a raid and when something happens that requires a forced check performance is all over the map. Some of the files will do 200+ MB/s in the checking while others can't get above 10 MB/s. Incidentally, the raid has been measured at roughly 450MB/s reading a contiguous file, so you can see how 10MB/s is just killing me. How I wish all my torrents were contiguous on disk.
Thanks for reading my request.
Sparse Allocation Considered Harmful...
Re: Sparse Allocation Considered Harmful...
I'll second that. Sparse allocation should only be used for files with huge gaps, for files which eventually will be completely allocated (e.g. torrents) sparse allocation offers no advantages at all, leads to excessive fragmentation and may cause free disk space problems.
Re: Sparse Allocation Considered Harmful...
This is handled by libtorrent: http://www.rasterbar.com/products/libtorrent/
You should request support for non-sparse full allocation over there.
You should request support for non-sparse full allocation over there.
Re: Sparse Allocation Considered Harmful...
libtorrent supports full allocation, we just don't expose it in deluge. It's one of those things I'll get to eventually.johnnyg wrote:This is handled by libtorrent: http://www.rasterbar.com/products/libtorrent/
You should request support for non-sparse full allocation over there.
Re: Sparse Allocation Considered Harmful...
This is one of the reasons I will still continue to use rtorrent (a patched version to do full allocation) for downloading and use deluge for seeding. Keeping my torrent files without fragmentation is pretty much a requirement for me.
Re: Sparse Allocation Considered Harmful...
Hello,
We need this feature! This hurts the user's view of this software greatly.
silekonn
We need this feature! This hurts the user's view of this software greatly.
silekonn
Re: Sparse Allocation Considered Harmful...
I'd suggest creating a ticket and setting the milestone for 1.4; it's the best way to ensure something gets done.
Re: Sparse Allocation Considered Harmful...
If you have libtorrent 0.15+, you can just
edit the file and change
to
That will cause Full allocation mode to pre-allocate all files and as low fragmentation as possible (I'm yet to see a file in more than one fragment, as the filesystem is asked for whole filesize at once). On XFS, which I use, it's done instantly (no zerofill), however be aware that it preallocates the whole torrent, even if you select to only download a few files.
Just a workaround until it's implemented properly. Maybe could try to find time for doing this later.
And by the way, if I understand it correctly, using Compact allocation will cause less fragmentation than the default behaviour of Full allocation, because the files only grow in size (forward), but sparse files are filled randomly.
Code: Select all
locate torrentmanager.py
Code: Select all
storage_mode = lt.storage_mode_t(0)
Code: Select all
storage_mode = lt.storage_mode_t(1)
Just a workaround until it's implemented properly. Maybe could try to find time for doing this later.
And by the way, if I understand it correctly, using Compact allocation will cause less fragmentation than the default behaviour of Full allocation, because the files only grow in size (forward), but sparse files are filled randomly.
Re: Sparse Allocation Considered Harmful...
Thanks for the info! Now maybe we can get someone to add an option to the G.U.I.?Endeer wrote:Code: Select all
locate torrentmanager.py
That will cause Full allocation mode to pre-allocate all files and as low fragmentation as possible (I'm yet to see a file in more than one fragment, as the filesystem is asked for whole filesize at once). On XFS, which I use, it's done instantly (no zerofill), however be aware that it preallocates the whole torrent, even if you select to only download a few files.Code: Select all
storage_mode = lt.storage_mode_t(1)
Just a workaround until it's implemented properly. Maybe could try to find time for doing this later.
And by the way, if I understand it correctly, using Compact allocation will cause less fragmentation than the default behaviour of Full allocation, because the files only grow in size (forward), but sparse files are filled randomly.
Endeer, your terms are reversed in your last sentence (and it's not clear what you mean with "forward"... or preallocating the entire torrent [why would it allocate the disk for files that are not created?]). Full allocation writes the entire file to the disk immediately and alters bits, which only fragments on a drive lacking large enough space for the files' size. Compact allocation writes each piece of data in the first unused location, which is much greater fragmentation (and non-fragmented/inactive drives will be impacted as soon as a second torrent begins). The purpose of full allocation is a straight read-through instead of waiting on disk access every few pieces (which are in the high thousands for large torrents). It is a huge problem when you have a large number of torrents downloading together, and could be a game-changer for choice of software.
Silekonn
P.S. Hey O.P. What kind of R.A.I.D. requires a file system check? My cheap on-board Intel 4-disk 10 array does bit comparison every now and then but a file check would take an age (or maybe I don't understand it correctly..?). Also, you may want to try MyDefrag (or something else that will compact free space, etc.)
Re: Sparse Allocation Considered Harmful...
silekonn: My last sentence was written as I meant it. If you don't do the patch I wrote here, Compact will very probably cause less fragmentation than Full. Compact mode is meant for file systems which cannot do sparse files, so the downloaded pieces are written in download-order one after another to the file (so it only grows in size, forward, without holes, the file has a chance of being allocated sequentially) and after the file is finished, it rearranges them in the correct order ("alters the bits").
Whereas Full mode (without the patch) allocates sparse file of target size (=allocates nothing) and then fills it in in download (=random) order. So when Deluge wants to actually write a downloaded piece, then and only then it will get allocated. They maybe even get allocated one after another on the hard drive, but they are in download-order. And they stay that way, so when you want to read the file in correct order (e.g. copy), you will have to seek and seek and seek - I usually got 1000-2000 fragments per 1GB file. The read spead was downright terrible.
Only after the proposed patch Full allocation will actually _allocate_ the whole file, and then as you say "alter the bits".
So if you cannot / want not do this patch, and want to download all files, try using Compact allocation. It should improve things somewhat.
Whereas Full mode (without the patch) allocates sparse file of target size (=allocates nothing) and then fills it in in download (=random) order. So when Deluge wants to actually write a downloaded piece, then and only then it will get allocated. They maybe even get allocated one after another on the hard drive, but they are in download-order. And they stay that way, so when you want to read the file in correct order (e.g. copy), you will have to seek and seek and seek - I usually got 1000-2000 fragments per 1GB file. The read spead was downright terrible.
Only after the proposed patch Full allocation will actually _allocate_ the whole file, and then as you say "alter the bits".
So if you cannot / want not do this patch, and want to download all files, try using Compact allocation. It should improve things somewhat.