Sparse Allocation Considered Harmful...

[Suzuka]

I would like to request a way to do full pre-allocation - either the fancy way using the fallocate() posix call, or just simply writing out a bazillion zeros instead of the seek trick for creating sparse files.

I've got about 4TB of torrented material and it is massively fragmented due to the sparse allocation method. It's on a raid and when something happens that requires a forced check performance is all over the map. Some of the files will do 200+ MB/s in the checking while others can't get above 10 MB/s. Incidentally, the raid has been measured at roughly 450MB/s reading a contiguous file, so you can see how 10MB/s is just killing me. How I wish all my torrents were contiguous on disk.

Thanks for reading my request.

[lvm]

I'll second that. Sparse allocation should only be used for files with huge gaps, for files which eventually will be completely allocated (e.g. torrents) sparse allocation offers no advantages at all, leads to excessive fragmentation and may cause free disk space problems.

[johnnyg]

This is handled by libtorrent: http://www.rasterbar.com/products/libtorrent/
You should request support for non-sparse full allocation over there.

[andar]

This is handled by libtorrent: http://www.rasterbar.com/products/libtorrent/
You should request support for non-sparse full allocation over there.

libtorrent supports full allocation, we just don't expose it in deluge. It's one of those things I'll get to eventually.

[方向音痴]

This is one of the reasons I will still continue to use rtorrent (a patched version to do full allocation) for downloading and use deluge for seeding. Keeping my torrent files without fragmentation is pretty much a requirement for me.

[silekonn]

Hello,

We need this feature! This hurts the user's view of this software greatly.

silekonn

[johnnyg]

I'd suggest creating a ticket and setting the milestone for 1.4; it's the best way to ensure something gets done.

[Endeer]

If you have libtorrent 0.15+, you can just

Code: Select all

locate torrentmanager.py

edit the file and change

Code: Select all

storage_mode = lt.storage_mode_t(0)

to

Code: Select all

storage_mode = lt.storage_mode_t(1)

That will cause Full allocation mode to pre-allocate all files and as low fragmentation as possible (I'm yet to see a file in more than one fragment, as the filesystem is asked for whole filesize at once). On XFS, which I use, it's done instantly (no zerofill), however be aware that it preallocates the whole torrent, even if you select to only download a few files.
Just a workaround until it's implemented properly. Maybe could try to find time for doing this later.

And by the way, if I understand it correctly, using Compact allocation will cause less fragmentation than the default behaviour of Full allocation, because the files only grow in size (forward), but sparse files are filled randomly.

[silekonn]

Code: Select all

locate torrentmanager.py

Code: Select all

storage_mode = lt.storage_mode_t(1)

That will cause Full allocation mode to pre-allocate all files and as low fragmentation as possible (I'm yet to see a file in more than one fragment, as the filesystem is asked for whole filesize at once). On XFS, which I use, it's done instantly (no zerofill), however be aware that it preallocates the whole torrent, even if you select to only download a few files.
Just a workaround until it's implemented properly. Maybe could try to find time for doing this later.

And by the way, if I understand it correctly, using Compact allocation will cause less fragmentation than the default behaviour of Full allocation, because the files only grow in size (forward), but sparse files are filled randomly.

Thanks for the info! Now maybe we can get someone to add an option to the G.U.I.?

Endeer, your terms are reversed in your last sentence (and it's not clear what you mean with "forward"... or preallocating the entire torrent [why would it allocate the disk for files that are not created?]). Full allocation writes the entire file to the disk immediately and alters bits, which only fragments on a drive lacking large enough space for the files' size. Compact allocation writes each piece of data in the first unused location, which is much greater fragmentation (and non-fragmented/inactive drives will be impacted as soon as a second torrent begins). The purpose of full allocation is a straight read-through instead of waiting on disk access every few pieces (which are in the high thousands for large torrents). It is a huge problem when you have a large number of torrents downloading together, and could be a game-changer for choice of software.

Silekonn

P.S. Hey O.P. What kind of R.A.I.D. requires a file system check? My cheap on-board Intel 4-disk 10 array does bit comparison every now and then but a file check would take an age (or maybe I don't understand it correctly..?). Also, you may want to try MyDefrag (or something else that will compact free space, etc.)

[Endeer]

silekonn: My last sentence was written as I meant it. If you don't do the patch I wrote here, Compact will very probably cause less fragmentation than Full. Compact mode is meant for file systems which cannot do sparse files, so the downloaded pieces are written in download-order one after another to the file (so it only grows in size, forward, without holes, the file has a chance of being allocated sequentially) and after the file is finished, it rearranges them in the correct order ("alters the bits").

Whereas Full mode (without the patch) allocates sparse file of target size (=allocates nothing) and then fills it in in download (=random) order. So when Deluge wants to actually write a downloaded piece, then and only then it will get allocated. They maybe even get allocated one after another on the hard drive, but they are in download-order. And they stay that way, so when you want to read the file in correct order (e.g. copy), you will have to seek and seek and seek - I usually got 1000-2000 fragments per 1GB file. The read spead was downright terrible.

Only after the proposed patch Full allocation will actually _allocate_ the whole file, and then as you say "alter the bits".
So if you cannot / want not do this patch, and want to download all files, try using Compact allocation. It should improve things somewhat.