ext3 - Full vs Compact Allocation?

[c-shadow]

Did anyone noticed some improvements using Full aloocation?
I'm running Deluge under ubuntu and have 2x300GB partitions under ext3 and have noticed that downloaded files are still somehow getting fragmented, especially large files. Filefrag is still giving lage number of extents I watched one mp3 file in a torrent, size 6.5 MB...
Immediately after deluge started:

Code: Select all

11 That's What Counts.mp3: 2 extents found, perfection would be 1 extent

And after downloading the whole file:

Code: Select all

11 That's What Counts.mp3: 205 extents found, perfection would be 1 extent

For example, azureus makes the same file in only one extent, but it writes zeroes in file on allocation...
With deluge i have also files like this (1GB file):

Code: Select all

74437 extents found, perfection would be 10 extents

It seems that it would be a good idea to repeat the test with the bigger file...
Is the full allocation working on ext3 or not?

[hasardeur]

i don't know if that is relevant but fragmentation itself is no problem with ext3 and the storage of files is not done by deluge. (caution: the last part is superficial knowledge)

fragmentation is an issue on windows filesystems like any fat and that other even older crap back in the dos days, because the gouverning method is 'best fit'

i remember reading that ext2/3 uses a priciple called 'worst fit'/'first fit'. its a tradeoff. one rather fragments quite early instead of doing it later, which would mean splitting every file into 1000s of blocks.
if one wants to keep fragmentation low, wheather thats useful or not, free space is the way to go. rule of thumb for ext 15 - 25% (at all times!).

as far as i know there isn't a built in way to defragment ext but check this out: http://vleu.net/shake/
sometimes i use this prog to show how severe fragmentation is on my hdd (160gb). it is bad. very very bad. my drive has been filled to the last bit several times and has not been reformatted since 2004. the free space varies between 1 - 5% and running a gentoo box means compiling a lot - which results in 1000s of files being in flux.
but apart from the usual mostly self infliceted breakdowns everything works fine and fast. i wouldn't worry too much about it.

i just ran 'shake -pvv <torrent dir>' and every 350mb file has about 18000 fragments and the 700mb files 38000 fragments. it does not seem to matter. (1.8GHz - 786mb SDRAM - IDE HDD - 2001 model)
everything is fine.

i got a litte lost here during writing - what are off-days for...

[c-shadow]

Fragmentation is the result of Full alocation creating a sparse files:
http://en.wikipedia.org/wiki/Sparse_file
I just added a torrent with two files, 20GB total and after a couple of seconds i got a 4+2 GB files on disk which is impossible with my ADSL connection
Perhaps developers can correct me, my conclusion is that if deluge tries to write a piece of a 20 GB file which is at the position of 10GB it will immediately create (extend) that sparse file to 10Gb and so on. This way you can have a 10 GB file that has only two pieces of torrent in it. Later pieces do not fill the blanks continuously and thus the partition gets fragmented A LOT. Normal copying of files would never fragment an ext3 partition like that.
Currently i don't have enough time to test the difference between full and compact allocation, but in it's current state Full seems useless to me.
I already mentioned azureus, it allocates the space and fills it with zeroes right away (it takes time to do this), thus minimizing fragmentation.
Deluge just allocates the space needed, but filesystem later fills it in no particular order.
Please feel free to correct me if I'm wrong.
You can try it by yourself, just download a GB or two torrent with deluge and azureus and compare the number of fragments running filefrag on files.
Used Azureus before and since I switched to deluge found out that are my files are much more fragmented.
Sometimes it takes ages to copy a couple of GB from one disk to another because of this.

[hasardeur]

maybe the problem is that ext3 does not support extents.
that is a good question for a dev: does deluge really simply instruct the "filesystem" to reserve space or does it request continous space?

given a 1024mb file with 2048 x 512kb, would that mean that jfs for example could simply map 1024mb and be done with it? and in turn as ext3 can't do that, would it only reserve the space and the pieces would be fit anywhere without the requirement to be continous?

maping a continous region is one thing, but what would enforce the order of the 2048 pieces within the extent? would that be jfs (in this case) or deluge? (deluge, right?!?)

anyway here is a short benchmark with a 714mb file:

/[...]/serenity-cd1.avi: 49540 extents found, perfection would be 6 extents
it took 2:19s to copy to an identical hdd

/[...]/serenity-cd2.avi: 2597 extents found, perfection would be 6 extents
it took 2:13s to copy

4.3% speed increase... though it might be more on sata hdds or given a less fragmented file...

[loki]

I think I remember reading somewhere on here that ntfs should be used... I could be completely wrong though. Is that why, to reduce the fragmentation?

Also, I thought full allocation was supposed to minimize the fragmentation, or is that just when it works properly? It doesn't seem as though it was working properly on ext3 if thats the case...

I've noticed the following before with deluge, I use it on windows most often with the ntfs and full allocation. First off I'm not sure how well I can explain this accurately. I notice when it starts downloading a file in the 2-3gb range and theres multiple files in the torrent. It starts downloading a specific file in the torrent and it creates the corresponding file, but not all of the files at once in the torrent. Wouldn't that be the proper way to start the torrent with full allocation, to create all the files at start? Well, hopefully what I explained makes sense to somebody other than me... it could explain why there's some fragmentation happening I don't know about thousands...

[c-shadow]

maybe the problem is that ext3 does not support extents.
that is a good question for a dev: does deluge really simply instruct the "filesystem" to reserve space or does it request continous space?

I'm not a dev but it seems it does not request continuous space. To do that deluge should actuallly write something into that preallocated space.

given a 1024mb file with 2048 x 512kb, would that mean that jfs for example could simply map 1024mb and be done with it? and in turn as ext3 can't do that, would it only reserve the space and the pieces would be fit anywhere without the requirement to be continous?

maping a continous region is one thing, but what would enforce the order of the 2048 pieces within the extent? would that be jfs (in this case) or deluge? (deluge, right?!?)

I think it would be the filesystem.

anyway here is a short benchmark with a 714mb file:

/[...]/serenity-cd1.avi: 49540 extents found, perfection would be 6 extents
it took 2:19s to copy to an identical hdd

/[...]/serenity-cd2.avi: 2597 extents found, perfection would be 6 extents
it took 2:13s to copy

4.3% speed increase... though it might be more on sata hdds or given a less fragmented file...

I would try copying files to /dev/null to avoid overhead with writing.

Code: Select all

ls -l
-rw-r--r-- 1 alen alen 7041306152 2007-12-17 09:09 Transformers.2007.HDDVD.720p.x264-iLL.mkv
sudo filefrag Transformers.2007.HDDVD.720p.x264-iLL.mkv
Transformers.2007.HDDVD.720p.x264-iLL.mkv: 258549 extents found, perfection would be 57 extents

time cp Transformers.2007.HDDVD.720p.x264-iLL.mkv /dev/null

real    13m12.083s
user    0m1.032s
sys     0m19.989s

The file is 6.6 GB, a nd it took 13 min to read it. I's about 9 MB/s!
This is at least 3x slower than it could be read...

Look at the speed below during copy:

Code: Select all

iostat -m 1

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.46    0.00    3.48   45.27    0.00   42.79

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
hda               0.00         0.00         0.00          0          0
sda               0.00         0.00         0.00          0          0
sdb             442.00         9.32         0.00          9          0
sdc               0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.00    0.00    5.50   44.50    0.00   43.00

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
hda               0.00         0.00         0.00          0          0
sda               0.00         0.00         0.00          0          0
sdb             329.00         5.92         0.00          5          0
sdc               0.00         0.00         0.00          0          0

[hasardeur]

ok THAT is crappy.

and something should be done about it... it seems like the choice between compact and full allocation is merely a choice between instantanious space reservation and and an incremental one.

[Ux64]

It seems like the choice between compact and full allocation is merely a choice between instantaneous space reservation and an incremental one.

Ext3 - Is Defrag required?
I totally agree about that with you.

If I remember it right, ext3 doesn't support pre allocation at all.

And I also found out that many *nix users are uptight and doesn't see the facts. ext3 fragments heavily in some cases and there is no way to get rid of that fragmentation without defrag. (Or backup, clear, restore) And even after that process, disk might be already fragmented!

I wrote a lot about this topic to here. And got sick'n'tired because users "I have read that ext3 doesn't fragment at all". stuff.

I also asked why current system doesn't support NCQ. And reply was something, that it isn't needed. WTF? If there is some fragmentation it should help a lot. And also when ever there are several processes accessing disk, it should also help. But ahem, what ever negative I say about Linux, it seems to be issue of faith. Not facts. ;(

Read this thread http://ubuntuforums.org/showthread.php? ... ght=defrag and especially fu buf posts which is my old handle.

I have been seriously considering installing Windows instead of Linux, because there are obvious flaws which are denied to the death. Linux akbar!

But not to leave so negative impression I have to say that disk caching is way better than Windows disk cache.

And you'll really notice it when using Deluge or doing other heavy disk activity stuff with huge number of small files handled regularly.

Full vs Compact allocation

I also said in some thread that compact allocation causes MORE disk access than full allocation. I don't know why they say in some documentation that compact allocation would be better. Because it'll require several read/writes for same data. I'll have to search for that writing again.

Yep, I found it.

Some thoughts about full / compact allocation and disk cache.

http://forum.deluge-torrent.org/viewtop ... tion#p3657

I like arguing. When ever you got proper basis and reasoning for discussion. I hate claims without facts. So, I would love to hear your comments. - Thanks!

[hasardeur]

it seems to be issue of faith. Not facts. ;(

I'll give you that one.

Yesterday night I put ext3 to the test and ran defrag (shake) on my files and the effect is clear. There is a performance gain and it's obvious without having to benchmark it. Things go much smoother and faster _overall_ .
To be honest - I did not really see that one coming.

This time, the case rests me...

[Ux64]

Afaik, it doesn't make any difference if you use compact or full allocation. Full allocation could cause less disk i/o.