Adaptive concurrent connections

[Alan Evans]

I have been using FileZilla to transfer files on a gigabit network and have made a few observations.

• With concurrent transfers = 1, transferring large files (8GB+) I can saturate a gigabit link
• With concurrent transfers > 1, transferring large files due to disk IO on the remote FTP server and local box I get varying transfer rates in the 550-700MBp/sec
• With concurrent transfers = 1, transferring many (~2000) small files (< 500Kb) it takes a long time to transfer the files
• With concurrent transfers > 1, transferring many small files the time it takes to transfer the files goes down in proportion to the number of concurrent connections

I see similar results when uploading.

So I propose adding a feature to FileZilla that would try to optimize the number of connections per server during a file transfer.

One way to go about this might be to sort the queue by file size from large to small and then transfer the large files with one concurrent connection and then the small files with many concurrent connections.

This would be a relatively simple way to go about it. Just put in a conditional cap, "When transfering fileslarger than : xxx, use no more than yyy concurrent connections" and then let the normal concurrent connection cap logic work.

The other way to make this work might be to keep track of the overall transfer rate between the client and server and then vary the connection count and decide which is best. Additionally if the connection count is capped at the server side you should get a too many connections message which would set your uppper bound.

Think of it like an eye exam. "Add one connection?" "Is this better?" "No" or if connections > 1 "Remove one connection?" "No"

Evaluation could be done with every new or closed connection up to a maximum of 10 or a min of 1.

Ok so my use case would seem rather specific I admit it. But a feature like this may also be helpful when transferring trees of files on smaller links.

Sometimes it might make the most sense to transfer one file at a time even though you could transfer 3. Or... Sometimes it might make sense to transfer 3 at a time rather than one. Again the whole link saturation thing vs time to start new files.