Opened 7 years ago

Closed 5 years ago

Last modified 2 years ago

#8094 closed Patch (fixed)

Preallocate space

Reported by: wOxxOm Owned by: Underground78
Priority: normal Component: FileZilla Client
Keywords: Cc: underground78@…
Component version: Operating system type:
Operating system version:

Description (last modified by Tim Kosse)

On Windows platforms newer than WinXP besides SetFilePointer+SetEndOfFile preallocation method which fills required disk space with zeros which might take a lot of time in case of a very large file, there is another literally instant function: SetFileValidData (see MSDN), it just claims disk space leaving intact whatever data there was.

Despite common belief that fragmentation isn't an issue, it's quite the opposite - 1+GB files downloaded in such applications as FileZilla Client which doesn't preallocate disk space, are extremely slow to copy and work with when mounted and impose absolutely unnecessary huge load on HDD due to a lot of disk seeks.

Apparently only a few win32 ftp clients can preallocate space and none of them uses SetFileValidData. Out of about 20 most popular ftp clients I've tested, only Flashget and NetTransport had the feature. Actually I've searched all files on my PC and out of hundreds of small and large win32 apps that constantly deal with creation of files, large and small, only VirtualDub and uTorrent use SetFileValidData, also the latter allows user to select whether he prefers instant preallocation, complete with zeroing or none of the two.

There was a similar ticket (#1486) 7 years ago and now it's already 2012 but FileZilla Client still doesn't preallocate space.

Attachments (4)

file_preallocation_for_windows.patch (2.6 KB) - added by Underground78 5 years ago.
Patch: Preallocate files on Windows (v1b)
file_preallocation_v2.patch (3.3 KB) - added by Underground78 5 years ago.
preallocate_option_preview.png (42.5 KB) - added by Underground78 5 years ago.
file_preallocation_v3.patch (6.5 KB) - added by Underground78 5 years ago.

Download all attachments as: .zip

Change History (32)

comment:1 Changed 5 years ago by Christian Borchert

Disappointed that this still has not yet been resolved.

comment:2 Changed 5 years ago by Underground78

Cc: underground78@… added

I just had a look to Filezilla's code and it would seems it uses 64KB buffers for its IO thread. I find that value really low and I think increasing it would already help reducing fragmentation.

comment:3 Changed 5 years ago by Underground78

Owner: set to Underground78
Status: newaccepted

I still need to double check that everything works correctly but I think I got preallocation working with seeking and SetEndOfFile.

There is no need for SetFileValidData since FileZilla only does sequential writing and modern file systems like NTFS are reasonably clever. SetEndOfFile won't write zeros up to the current position in the file if the next writes happen at the end of the already written data.

comment:4 Changed 5 years ago by wOxxOm

Summary: Preallocate space and avoid fragmentation of files with SetFileValidDataPreallocate space and avoid fragmentation of files

Indeed, FileZilla currently downloads only in a single thread, and SetFileValidData is only crucial for multi-threaded downloading when writes occur at arbitrary file offsets.

comment:5 Changed 5 years ago by Underground78

Type: Feature requestPatch

I've attached the patch and uploaded a test build in case somebody wants to play with it.

Note that on my test case it only reduced the number of fragments by 25% (from about 8000 to about 6000 for a file around 3GB). However the fragments are much more grouped together with file allocation instead of being scattered on the disk.

comment:6 in reply to:  5 Changed 5 years ago by wOxxOm

I've downloaded a 6GB file to a drive with zillion of files but with a lot of free space - 400GB - and your build stored it in one fragment. Then I've deleted it and downloaded with vanilla FileZilla - 3375 fragments as usual.

Note that on my test case it only reduced the number of fragments by 25%

Depends on free space fragmentation because NTFS cannot magically defragment it on-the-fly of course.

comment:7 Changed 5 years ago by Underground78

I had plenty of non fragmented free space too but that's good to know!

Changed 5 years ago by Underground78

Patch: Preallocate files on Windows (v1b)

comment:8 Changed 5 years ago by Underground78

I did more testings and I know why I still had fragmentation, the disk I used for the test has NTFS compression on.

comment:9 Changed 5 years ago by Underground78

I did new tests on a disk without NTFS compression this time. I downloaded 3 files between 3 and 5 GB and all three were nicely happened at the end of the disk without any fragmentation (not even creating free space fragmentation).

Basically it seems to work fine! :)

comment:10 Changed 5 years ago by Underground78

I've been using the patch for 2 weeks now without issue.

Would it be possible to have it reviewed so that there is a change it can be upstreamed?

comment:11 Changed 5 years ago by Tim Kosse

Status: acceptedmoreinfo_accepted

The patch looks good so far, a couple of comments though:

  1. Please update it so that it applies against SVN trunk
  1. The Non-Windows implementation is missing
  1. Depending on the filesystem used, preallocating files can take a very long time. I think there should be an option controlling preallocation

comment:12 Changed 5 years ago by Underground78

  1. Yeah I noticed there were quite a lot of changes in the engine. I will try to have a look and update the patch soon.
  2. I am not really sure to the non-Windows way to preallocate files but I will give it a try. I will probably need someone to do the testing though.
  3. Fair enough, but I don't know where to put it honestly.

comment:13 Changed 5 years ago by Tim Kosse

  1. I am not really sure to the non-Windows way to preallocate files but I will give it a try. I will probably need someone to do the testing though.

The non-Windows equivalent to SetEndOfFile is ftruncate. With ftruncate, be sure to handle EINTR errors as is done with read/write.

  1. Fair enough, but I don't know where to put it honestly.

The internals go to src/include/optionsbase.h and src/interface/Options.cpp

The GUI part would best fit on the transfer settings page: src/interface/optionspage_transfer.cpp and src/interfase/resources/xrc/settings.xrc

comment:14 Changed 5 years ago by Underground78

The non-Windows equivalent to SetEndOfFile is ftruncate.

Hmm so it is usually done with seek+truncate like on Windows? I thought the function for that was (posix_)fallocate.

comment:15 Changed 5 years ago by Underground78

It's quite weird, it seems aboutdialog.cpp makes my compiler crash...

I will attach the current version of the patch but be aware that I wasn't able to test it yet. I've added a Truncate wrapper to your CFile class and used the same seek+truncate approach for both Windows and Linux.

Changed 5 years ago by Underground78

Attachment: file_preallocation_v2.patch added

comment:16 Changed 5 years ago by Underground78

Status: moreinfo_acceptedaccepted

I've tried to update MSYS just in case to no avail. I have rebuilt all dependencies without issues but g++ seems to crash each time it tries to compile aboutdialog.cpp.

comment:17 Changed 5 years ago by Tim Kosse

There's a workaround for this compiler bug, run FileZilla's configure with --disable-precomp

comment:18 Changed 5 years ago by Underground78

Operating system type: Windows

Thanks, I was able to compile FileZilla with the patch v2.

Everything seems to work fine as far as I can tell, on Windows at least. I've checked that stopped transfers are correctly resumed (I've compared the md5 hash of the files to be sure).

I will try to work on adding an option.

comment:19 Changed 5 years ago by Underground78

Hmm, where do you want me to add the options in the "Transfers" setting page exactly? It's rather crowded.

The only space left is in the "Speed limits" group box but it doesn't really fit here. I don't think there is enough space left to add another group box. I guess I can move the group boxes up and add the new option at the bottom of the page...

Changed 5 years ago by Underground78

Changed 5 years ago by Underground78

Attachment: file_preallocation_v3.patch added

comment:20 Changed 5 years ago by Underground78

I have attached an updated patch which includes the new option and a screenshot of the settings dialog.

comment:21 Changed 5 years ago by Tim Kosse

Resolution: fixed
Status: acceptedclosed

Thanks, I've applied the patch.

I've done a few small changes on top of the patch:

  • Changed keyboard mnemonic, p was already taken
  • Don't enable preallocation by default. In case of crashes the file would not be truncated and I'm worried that moving the engine to its own thread might be prone to crashes. Threading issues can be quite subtle.
  • Use SetCheckFromOption


comment:22 Changed 5 years ago by Underground78

Great! :)

  • Use SetCheckFromOption

I think (from memory so I might be wrong) that you missed another case since I copy-pasted that code from a few lines before or after my change.

comment:23 Changed 2 years ago by Christoph Rackwitz

are you sure this code is active? in 3.26.2, even with checked option, files get fragmented. do I need to disable some other option for this to work?

comment:24 Changed 2 years ago by brainard

Hi
Same for me, (option checked) a 700 MB file finished with more than 400 fragments
on non-SSD disk
I tested many times (Tools defraggler64 to check framentation)
Last version 3.26.2 Windows 7

Last edited 2 years ago by brainard (previous) (diff)

comment:25 Changed 2 years ago by Tim Kosse

Description: modified (diff)

The purpose of preallocation is to reserve the required disk space prior to starting the transfer. It cannot prevent fragmentation.

A side-effect of preallocation can be reduced fragmentation due to the way some filesystems work, but that's purely incidental and cannot be relied upon.

comment:26 Changed 2 years ago by Tim Kosse

Summary: Preallocate space and avoid fragmentation of filesPreallocate space

comment:27 Changed 2 years ago by wOxxOm

A side-effect of preallocation can be reduced fragmentation due to the way some filesystems work, but that's purely incidental and cannot be relied upon.

Yes, it's impossible to guarantee the file won't be fragmented especially on a drive where an OS like Windows is installed, which is what the users above have encountered.

Not sure which OS is the most popular where FileZilla client is used, but in Windows + NTFS (the most popular client OS and file system) preallocation significantly reduces fragmentation, depending on the level of fragmentation in the unused disk space. For example, if you have a low traffic disk (a typical storage device for archived data) with a lot of free space, the preallocated file won't be fragmented at all or will have just a few fragments in apps that preallocate properly like the above-mentioned uTorrent.

comment:28 Changed 2 years ago by Christoph Rackwitz

AFAIK SetFileValidData needs privileges that a normal user doesn't have, so it'd require running filezilla as administrator initially, then drop all but this one privilege. since FileZilla never asked me for elevated privileges, I assume either SetFileValidData is not used, or it fails and the error result is ignored.

on my win7, SetEndOfFile seems to work in O(1) too, meaning it marks the clusters as allocated, the file is the expected size "on disk", and the contents appear initialized with zero (probably not physically). how about an option to use SetEndOfFile, if it works for some users?

afaik, SetEndOfFile only physically initializes parts of the file if you write past them. if you write sequentially from the start, it doesn't cost anything.

Note: See TracTickets for help on using tickets.