Preallocate space

[wOxxOm]

On Windows platforms newer than WinXP besides SetFilePointer+SetEndOfFile preallocation method which fills required disk space with zeros which might take a lot of time in case of a very large file, there is another literally instant function: SetFileValidData (see MSDN), it just claims disk space leaving intact whatever data there was.

Despite common belief that fragmentation isn't an issue, it's quite the opposite - 1+GB files downloaded in such applications as FileZilla Client which doesn't preallocate disk space, are extremely slow to copy and work with when mounted and impose absolutely unnecessary huge load on HDD due to a lot of disk seeks.

Apparently only a few win32 ftp clients can preallocate space and none of them uses SetFileValidData. Out of about 20 most popular ftp clients I've tested, only Flashget and NetTransport had the feature. Actually I've searched all files on my PC and out of hundreds of small and large win32 apps that constantly deal with creation of files, large and small, only VirtualDub and uTorrent use SetFileValidData, also the latter allows user to select whether he prefers instant preallocation, complete with zeroing or none of the two.

There was a similar ticket (#1486) 7 years ago and now it's already 2012 but FileZilla Client still doesn't preallocate space.

[Christian Borchert]

Disappointed that this still has not yet been resolved.

[Underground78]

I just had a look to Filezilla's code and it would seems it uses 64KB buffers for its IO thread. I find that value really low and I think increasing it would already help reducing fragmentation.

[Underground78]

I still need to double check that everything works correctly but I think I got preallocation working with seeking and SetEndOfFile.

There is no need for SetFileValidData since FileZilla only does sequential writing and modern file systems like NTFS are reasonably clever. SetEndOfFile won't write zeros up to the current position in the file if the next writes happen at the end of the already written data.

[wOxxOm]

Indeed, FileZilla currently downloads only in a single thread, and SetFileValidData is only crucial for multi-threaded downloading when writes occur at arbitrary file offsets.

[Underground78]

I've attached the patch and uploaded a ​test build in case somebody wants to play with it.

Note that on my test case it only reduced the number of fragments by 25% (from about 8000 to about 6000 for a file around 3GB). However the fragments are much more grouped together with file allocation instead of being scattered on the disk.

[wOxxOm]

I've downloaded a 6GB file to a drive with zillion of files but with a lot of free space - 400GB - and your build stored it in one fragment. Then I've deleted it and downloaded with vanilla FileZilla - 3375 fragments as usual.

Note that on my test case it only reduced the number of fragments by 25%

Depends on free space fragmentation because NTFS cannot magically defragment it on-the-fly of course.

[Underground78]

I had plenty of non fragmented free space too but that's good to know!

[Underground78]

Patch: Preallocate files on Windows (v1b)

[Underground78]

I did more testings and I know why I still had fragmentation, the disk I used for the test has NTFS compression on.

[Underground78]

I did new tests on a disk without NTFS compression this time. I downloaded 3 files between 3 and 5 GB and all three were nicely happened at the end of the disk without any fragmentation (not even creating free space fragmentation).

Basically it seems to work fine! :)

[Underground78]

I've been using the patch for 2 weeks now without issue.

Would it be possible to have it reviewed so that there is a change it can be upstreamed?

[Tim Kosse]

The patch looks good so far, a couple of comments though:

1. Please update it so that it applies against SVN trunk

2. The Non-Windows implementation is missing

3. Depending on the filesystem used, preallocating files can take a very long time. I think there should be an option controlling preallocation

[Underground78]

1. Yeah I noticed there were quite a lot of changes in the engine. I will try to have a look and update the patch soon.
2. I am not really sure to the non-Windows way to preallocate files but I will give it a try. I will probably need someone to do the testing though.
3. Fair enough, but I don't know where to put it honestly.

[Tim Kosse]

2. I am not really sure to the non-Windows way to preallocate files but I will give it a try. I will probably need someone to do the testing though.

The non-Windows equivalent to SetEndOfFile is ftruncate. With ftruncate, be sure to handle EINTR errors as is done with read/write.

3. Fair enough, but I don't know where to put it honestly.

The internals go to src/include/optionsbase.h and src/interface/Options.cpp

The GUI part would best fit on the transfer settings page: src/interface/optionspage_transfer.cpp and src/interfase/resources/xrc/settings.xrc

[Underground78]

The non-Windows equivalent to SetEndOfFile is ftruncate.

Hmm so it is usually done with seek+truncate like on Windows? I thought the function for that was (posix_)fallocate.

[Underground78]

It's quite weird, it seems aboutdialog.cpp makes my compiler crash...

I will attach the current version of the patch but be aware that I wasn't able to test it yet. I've added a Truncate wrapper to your CFile class and used the same seek+truncate approach for both Windows and Linux.

[Underground78]

I've tried to update MSYS just in case to no avail. I have rebuilt all dependencies without issues but g++ seems to crash each time it tries to compile aboutdialog.cpp.

[Tim Kosse]

There's a workaround for this compiler bug, run FileZilla's configure with --disable-precomp

[Underground78]

Thanks, I was able to compile FileZilla with the patch v2.

Everything seems to work fine as far as I can tell, on Windows at least. I've checked that stopped transfers are correctly resumed (I've compared the md5 hash of the files to be sure).

I will try to work on adding an option.

[Underground78]

Hmm, where do you want me to add the options in the "Transfers" setting page exactly? It's rather crowded.

The only space left is in the "Speed limits" group box but it doesn't really fit here. I don't think there is enough space left to add another group box. I guess I can move the group boxes up and add the new option at the bottom of the page...

[Underground78]

I have attached an updated patch which includes the new option and a screenshot of the settings dialog.

[Tim Kosse]

Thanks, I've applied the patch.

I've done a few small changes on top of the patch:

• Changed keyboard mnemonic, p was already taken

• Don't enable preallocation by default. In case of crashes the file would not be truncated and I'm worried that moving the engine to its own thread might be prone to crashes. Threading issues can be quite subtle.

• Use SetCheckFromOption

[Underground78]

Great! :)

• Use SetCheckFromOption

I think (from memory so I might be wrong) that you missed another case since I copy-pasted that code from a few lines before or after my change.

[Christoph Rackwitz]

are you sure this code is active? in 3.26.2, even with checked option, files get fragmented. do I need to disable some other option for this to work?

[brainard]

Hi
Same for me, (option checked) a 700 MB file finished with more than 400 fragments
I tested many times
Last version 3.26.2 Windows 7

[Tim Kosse]

The purpose of preallocation is to reserve the required disk space prior to starting the transfer. It cannot prevent fragmentation.

A side-effect of preallocation can be reduced fragmentation due to the way some filesystems work, but that's purely incidental and cannot be relied upon.

[wOxxOm]

A side-effect of preallocation can be reduced fragmentation due to the way some filesystems work, but that's purely incidental and cannot be relied upon.

Yes, it's impossible to guarantee the file won't be fragmented especially on a drive where an OS like Windows is installed, which is what the users above have encountered.

Not sure which OS is the most popular where FileZilla client is used, but in Windows + NTFS (the most popular client OS and file system) preallocation significantly reduces fragmentation, depending on the level of fragmentation in the unused disk space. For example, if you have a low traffic disk (a typical storage device for archived data) with a lot of free space, the preallocated file won't be fragmented at all or will have just a few fragments in apps that preallocate properly like the above-mentioned uTorrent.

[Christoph Rackwitz]

AFAIK SetFileValidData needs privileges that a normal user doesn't have, so it'd require running filezilla as administrator initially, then drop all but this one privilege. since FileZilla never asked me for elevated privileges, I assume either SetFileValidData is not used, or it fails and the error result is ignored.

on my win7, SetEndOfFile seems to work in O(1) too, meaning it marks the clusters as allocated, the file is the expected size "on disk", and the contents appear initialized with zero (probably not physically). how about an option to use SetEndOfFile, if it works for some users?

afaik, SetEndOfFile only physically initializes parts of the file if you write past them. if you write sequentially from the start, it doesn't cost anything.