#8094 closed Patch (fixed)
Preallocate space
Reported by: | wOxxOm | Owned by: | Underground78 |
---|---|---|---|
Priority: | normal | Component: | FileZilla Client |
Keywords: | Cc: | underground78@… | |
Component version: | Operating system type: | ||
Operating system version: |
Description (last modified by )
On Windows platforms newer than WinXP besides SetFilePointer+SetEndOfFile preallocation method which fills required disk space with zeros which might take a lot of time in case of a very large file, there is another literally instant function: SetFileValidData (see MSDN), it just claims disk space leaving intact whatever data there was.
Despite common belief that fragmentation isn't an issue, it's quite the opposite - 1+GB files downloaded in such applications as FileZilla Client which doesn't preallocate disk space, are extremely slow to copy and work with when mounted and impose absolutely unnecessary huge load on HDD due to a lot of disk seeks.
Apparently only a few win32 ftp clients can preallocate space and none of them uses SetFileValidData. Out of about 20 most popular ftp clients I've tested, only Flashget and NetTransport had the feature. Actually I've searched all files on my PC and out of hundreds of small and large win32 apps that constantly deal with creation of files, large and small, only VirtualDub and uTorrent use SetFileValidData, also the latter allows user to select whether he prefers instant preallocation, complete with zeroing or none of the two.
There was a similar ticket (#1486) 7 years ago and now it's already 2012 but FileZilla Client still doesn't preallocate space.
Attachments (4)
Change History (32)
comment:1 by , 11 years ago
comment:2 by , 10 years ago
Cc: | added |
---|
I just had a look to Filezilla's code and it would seems it uses 64KB buffers for its IO thread. I find that value really low and I think increasing it would already help reducing fragmentation.
comment:3 by , 10 years ago
Owner: | set to |
---|---|
Status: | new → accepted |
I still need to double check that everything works correctly but I think I got preallocation working with seeking and SetEndOfFile
.
There is no need for SetFileValidData
since FileZilla only does sequential writing and modern file systems like NTFS are reasonably clever. SetEndOfFile
won't write zeros up to the current position in the file if the next writes happen at the end of the already written data.
comment:4 by , 10 years ago
Summary: | Preallocate space and avoid fragmentation of files with SetFileValidData → Preallocate space and avoid fragmentation of files |
---|
Indeed, FileZilla currently downloads only in a single thread, and SetFileValidData is only crucial for multi-threaded downloading when writes occur at arbitrary file offsets.
follow-up: 6 comment:5 by , 10 years ago
Type: | Feature request → Patch |
---|
I've attached the patch and uploaded a test build in case somebody wants to play with it.
Note that on my test case it only reduced the number of fragments by 25% (from about 8000 to about 6000 for a file around 3GB). However the fragments are much more grouped together with file allocation instead of being scattered on the disk.
comment:6 by , 10 years ago
I've downloaded a 6GB file to a drive with zillion of files but with a lot of free space - 400GB - and your build stored it in one fragment. Then I've deleted it and downloaded with vanilla FileZilla - 3375 fragments as usual.
Note that on my test case it only reduced the number of fragments by 25%
Depends on free space fragmentation because NTFS cannot magically defragment it on-the-fly of course.
by , 10 years ago
Attachment: | file_preallocation_for_windows.patch added |
---|
Patch: Preallocate files on Windows (v1b)
comment:8 by , 10 years ago
I did more testings and I know why I still had fragmentation, the disk I used for the test has NTFS compression on.
comment:9 by , 10 years ago
I did new tests on a disk without NTFS compression this time. I downloaded 3 files between 3 and 5 GB and all three were nicely happened at the end of the disk without any fragmentation (not even creating free space fragmentation).
Basically it seems to work fine! :)
comment:10 by , 10 years ago
I've been using the patch for 2 weeks now without issue.
Would it be possible to have it reviewed so that there is a change it can be upstreamed?
comment:11 by , 10 years ago
Status: | accepted → moreinfo_accepted |
---|
The patch looks good so far, a couple of comments though:
- Please update it so that it applies against SVN trunk
- The Non-Windows implementation is missing
- Depending on the filesystem used, preallocating files can take a very long time. I think there should be an option controlling preallocation
comment:12 by , 10 years ago
- Yeah I noticed there were quite a lot of changes in the engine. I will try to have a look and update the patch soon.
- I am not really sure to the non-Windows way to preallocate files but I will give it a try. I will probably need someone to do the testing though.
- Fair enough, but I don't know where to put it honestly.
comment:13 by , 10 years ago
- I am not really sure to the non-Windows way to preallocate files but I will give it a try. I will probably need someone to do the testing though.
The non-Windows equivalent to SetEndOfFile is ftruncate. With ftruncate, be sure to handle EINTR errors as is done with read/write.
- Fair enough, but I don't know where to put it honestly.
The internals go to src/include/optionsbase.h and src/interface/Options.cpp
The GUI part would best fit on the transfer settings page: src/interface/optionspage_transfer.cpp and src/interfase/resources/xrc/settings.xrc
comment:14 by , 10 years ago
The non-Windows equivalent to SetEndOfFile is ftruncate.
Hmm so it is usually done with seek+truncate like on Windows? I thought the function for that was (posix_)fallocate
.
comment:15 by , 10 years ago
It's quite weird, it seems aboutdialog.cpp
makes my compiler crash...
I will attach the current version of the patch but be aware that I wasn't able to test it yet. I've added a Truncate
wrapper to your CFile
class and used the same seek+truncate approach for both Windows and Linux.
by , 10 years ago
Attachment: | file_preallocation_v2.patch added |
---|
comment:16 by , 10 years ago
Status: | moreinfo_accepted → accepted |
---|
I've tried to update MSYS just in case to no avail. I have rebuilt all dependencies without issues but g++ seems to crash each time it tries to compile aboutdialog.cpp
.
comment:17 by , 10 years ago
There's a workaround for this compiler bug, run FileZilla's configure with --disable-precomp
comment:18 by , 10 years ago
Operating system type: | Windows |
---|
Thanks, I was able to compile FileZilla with the patch v2.
Everything seems to work fine as far as I can tell, on Windows at least. I've checked that stopped transfers are correctly resumed (I've compared the md5 hash of the files to be sure).
I will try to work on adding an option.
comment:19 by , 10 years ago
Hmm, where do you want me to add the options in the "Transfers" setting page exactly? It's rather crowded.
The only space left is in the "Speed limits" group box but it doesn't really fit here. I don't think there is enough space left to add another group box. I guess I can move the group boxes up and add the new option at the bottom of the page...
by , 10 years ago
Attachment: | preallocate_option_preview.png added |
---|
by , 10 years ago
Attachment: | file_preallocation_v3.patch added |
---|
comment:20 by , 10 years ago
I have attached an updated patch which includes the new option and a screenshot of the settings dialog.
comment:21 by , 10 years ago
Resolution: | → fixed |
---|---|
Status: | accepted → closed |
Thanks, I've applied the patch.
I've done a few small changes on top of the patch:
- Changed keyboard mnemonic, p was already taken
- Don't enable preallocation by default. In case of crashes the file would not be truncated and I'm worried that moving the engine to its own thread might be prone to crashes. Threading issues can be quite subtle.
- Use SetCheckFromOption
comment:22 by , 10 years ago
Great! :)
- Use SetCheckFromOption
I think (from memory so I might be wrong) that you missed another case since I copy-pasted that code from a few lines before or after my change.
comment:23 by , 7 years ago
are you sure this code is active? in 3.26.2, even with checked option, files get fragmented. do I need to disable some other option for this to work?
comment:24 by , 7 years ago
Hi
Same for me, (option checked) a 700 MB file finished with more than 400 fragments
on non-SSD disk
I tested many times (Tools defraggler64 to check framentation)
Last version 3.26.2 Windows 7
comment:25 by , 7 years ago
Description: | modified (diff) |
---|
The purpose of preallocation is to reserve the required disk space prior to starting the transfer. It cannot prevent fragmentation.
A side-effect of preallocation can be reduced fragmentation due to the way some filesystems work, but that's purely incidental and cannot be relied upon.
comment:26 by , 7 years ago
Summary: | Preallocate space and avoid fragmentation of files → Preallocate space |
---|
comment:27 by , 7 years ago
A side-effect of preallocation can be reduced fragmentation due to the way some filesystems work, but that's purely incidental and cannot be relied upon.
Yes, it's impossible to guarantee the file won't be fragmented especially on a drive where an OS like Windows is installed, which is what the users above have encountered.
Not sure which OS is the most popular where FileZilla client is used, but in Windows + NTFS (the most popular client OS and file system) preallocation significantly reduces fragmentation, depending on the level of fragmentation in the unused disk space. For example, if you have a low traffic disk (a typical storage device for archived data) with a lot of free space, the preallocated file won't be fragmented at all or will have just a few fragments in apps that preallocate properly like the above-mentioned uTorrent.
comment:28 by , 7 years ago
AFAIK SetFileValidData needs privileges that a normal user doesn't have, so it'd require running filezilla as administrator initially, then drop all but this one privilege. since FileZilla never asked me for elevated privileges, I assume either SetFileValidData is not used, or it fails and the error result is ignored.
on my win7, SetEndOfFile seems to work in O(1) too, meaning it marks the clusters as allocated, the file is the expected size "on disk", and the contents appear initialized with zero (probably not physically). how about an option to use SetEndOfFile, if it works for some users?
afaik, SetEndOfFile only physically initializes parts of the file if you write past them. if you write sequentially from the start, it doesn't cost anything.
Disappointed that this still has not yet been resolved.