[NTFS] Full Allocation, Sparse Files and 20.000 Fragments

[stb]

Hi all,

I'm running Deluge 1.3.2 on Windows 7 (x32) SP1; it's configured for "Full Allocation"
and does it as expected; the data is stored on a 500 GB dedicated partition, which have
never been more than 25% full; deluged runs no more than 6-8 concurrent downloads.

The problem is a HUGE fragmentation: 1000/2000 fragments per GB on average !
You could easily find a flac track in one hundred pieces, and an 8GB movie sliced
in 20.000 pieces !

I tried different cache dimensions, from 512 to 32K and different cache timeout,
from 30 to 600... no difference. Should i increase them further ?

I understand all that sparse files stuff, but that level of fragmentation cannot
be "normal" and it's not safe for sure: i can hear the hdd's heads run back and
forth at Warp 9 !

[johnnyg]

Unfortunately this level of fragmentation is "normal"; it's one of the few disadvantages of sparse files.

This has come up before: http://forum.deluge-torrent.org/viewtop ... =8&t=27725

From that thread you can see that libtorrent does actually support full allocation without using sparse files, however the option isn't exposed in deluge (feel free to create a ticket for it though).

[stb]

Unfortunately this level of fragmentation is "normal"; it's one of the few disadvantages of sparse files.

Thank you very much for you reply.

Unfortunately I can't stand such HDD activity level on a server: it would kill it.

Could you please tell me what language is Deluge developed in (C, i guess ?)
and if the "do not use those fu%#@ng sparse files" [ ] capability is implemented
in a specific module ?

[Cas]

Deluge is written in Python and storage mode for added torrent is set using the settings in torrentmanager.py

Those lines set the storage mode as specified in the libtorrent docs and python bindings

So you can enable Full Allocation without sparse files by changing

Code: Select all

storage_mode = lt.storage_mode_t(1) 

to

Code: Select all

storage_mode = lt.storage_mode_t(0)

[stb]

Thanks, you've been very kind.

If the mod is that simple, I'll take a look inside those libs:
I've never used Python before, but it doesn't seem to be
a challenge

Could you answer to a last question ? Just to feed my curiosity...

You developers could make that mod being blindfolded; does the
reason for not doing it involves the annoyance to relase a modded
version of libtorrent ? (I'm a developer too, and I'd hate to do that)

[stb]

hum... it seems I was wrong, maybe it's too late for coding...
torrentmanager.py belongs to your wonderful project, not to
libtorrent, so there's no need for a modded library.

please, ignore my last question

[Locutus266]

I just wanted to say that I have done some testing on this issue and found that storage_mode=0 produces far less fragments.

Test scenario:
Measuring of fragmentation was done under Windows 7 64-bit with latest updates
on non-system hard disk drive (no SSD), size of drive is 1 tb,
fully defragmented before testing, 130 gig free

Downloaded one big file (approx 1,4 gig) to hard disk using
two different versions of deluge. Version one was hacked by me (stable 1.3.6) to
use storage_mode=0 and version 2 was stable 1.3.6

fsutil tool is a Windows tool to query sparse flag of files.
contig tool is from sysinternals suite and measures level of fragmentation

ticket url: http://dev.deluge-torrent.org/ticket/1923

Code: Select all

storage_mode=0 (storage_mode_allocate) on hacked version of deluge-1.3.6
========================================================================

hard-coded "storage_mode=0" into torrentmanager.py

FSUTIL:
-------
>fsutil sparse queryflag FILENAME_OF_BIG_FILE
This file is NOT set as sparse


CONTIG:
------
>contig
Contig v1.7 - Makes files contiguous
Copyright (C) 1998-2012 Mark Russinovich
Sysinternals - http://www.sysinternals.com

Processing
BIGFILENAME is in 4 fragments

Summary:
     Number of files processed   : 1
     Average fragmentation       : 4 frags/file




storage_mode=1 (storage_mode_sparse) ("Full" according to gtk-gui) on 1.3.6 stable
========================================================================

FSUTIL:
-------
>fsutil sparse queryflag FILENAME_OF_BIG_FILE
This file is set as sparse

CONTIG:
------
Contig v1.7 - Makes files contiguous
Copyright (C) 1998-2012 Mark Russinovich
Sysinternals - http://www.sysinternals.com

Processing
BIGFILENAME is in 4649 fragments

Summary:
     Number of files processed   : 1
     Average fragmentation       : 7381 frags/file

[Xand]

Guys, can somebody, please, build latest stable release for Win32 with 0 option or this https://github.com/gazpachoking/deluge/ ... reallocate and upload somewhere?

Fragmentation is killing me. I even tried moving all files to the end of the disk with MyDefrag, but 1gb file was shattered exactly in occupied area.

Thank you.

[Lejums]

Is there an easy way to enable full allocation without sparse files on Windows?
The heavy fragmentation and corresponding creation of nonresident attribute lists are annoying.
Compact allocation is not an alternative as you can't download individual files.

[robnitro]

Hi,
I'm having the same issue... even small torrents below 100 MB with full allocation come up with a lot of fragments.
Ex: 10 MB file has 154 fragments using 1.3.7. Same with 1.3.6, 1.3.5 and 1.3.6py2.7