Maximizing Transfers

[Jim Robinson Jr.]

I frequently find myself transferring very large data sets across wide-open network pipes... and unable to fully utilize any component in the the path, including client, server or network. With 1G networks considered standard, and 10G becoming common, these systems have far more capacity than FileZilla - and virtually every other FTP client that I've found - can handle. The issue is two-fold.

1. Concurrent Transfers
2. Segmented Transfers

About 3 years ago someone opened a ticket (#2762) asking for segmented transfers and the response was that it wasted too much bandwidth. True. There is a fairly high overhead. However, when neither the client, server nor network is even breaking a sweat... why do we care if we waste more bandwidth but get better performance? I believe that capacity has outstripped the natural (and reasonable) limitations of the software.

Regarding feature 1, please consider either completely removing the concurrency restriction or making it significantly larger. I suspect that 100 is now a very reasonable MINIMUM number for a hard-coded threshold, but will rely on the opinions of folks much more knowledgeable than I am about the intricate working of FTP.

Regarding feature 2, the ability to break up large files into multiple segments could make a significant improvement on the overall performance of FTP. Even when FileZilla is configured to transfer up to 10 files at one time, if I am transferring a single 10GB file... I only get one thread. Breaking this up into 10 (or 20... or more) segments would indeed make a difference.

I would, however, also suggest a need to balance these two potentially competing features. Due to the overhead, smaller files would be slowed down by a segmented approach. A low end cap that forces files smaller than X to be single threaded could help. This too should be user-adjustable. Even better might be a multi-level approach. Thus, files smaller than (for example) 10MB could be set to "always single-segment", files between 10 and 100MB could be set to "max 2 segments", files between 100MB and 500MB could be set to "max 5 segments", and files larger than 500MB could be set to "use as many segments as available."

Between these two features, the Concurrency cap should be the sum of segments rather than files. Following this logic, if concurrency was set to 100, then 100 small files - at 1 segment each - would all be transferred simultaneously, or a single 10GB file could be broken into 100 segments... but not both.

I believe that these three changes, particularly implemented together, would be a huge win for FileZilla.

Thanks for listening,

Jim

[Jim Robinson Jr.]

Apologies. After submitting I discovered ticket #5062 that covers part of the same request.

[Tim Kosse]

What is your network latency between the client and the server?

FileZilla has been optimized to fully utilize 100Mbit at 333ms. As result, at 33ms it can saturate a 1Gbit link.

[Jim Robinson Jr.]

According to multiple PING tests, latency between my two primary systems ranges from "<1" to 4 ms. On a local segment, running GB Ethernet, with 2 fairly high-end Windows machines on either end (one running FileZilla server, the other FileZilla client, throughput maxes out at at about 18Mb with an average throughput of about 2mb. A good chunk of this is because I routinely transfer hundreds to thousands of very small files. Even with the overhead, the single biggest challenge is that FZ only queues 10 at a time. Both machines are bored, both tell me that they are maxing out at about 6% network utilization, and both could easily handle starting the queuing process for 20 times that number.

In addition, even when copying a single large file, I still hit a max of about 12mb. Copying this locally (e.g. not using FTP) easily triples that number, so I am certain it is not a machine / hardware latency issue. GbE should max out at a theoretical 125Mb... but then reality sets in. With SATA II drives I should be able to push something like 40Mb, but am not getting anywhere near that on file transfers. Local yes. Remote no.

This, and other troubleshooting, leads me to the conclusion that FTP is unnecessarily throttled.

That said, it is also possible that I'm missing something critical. I've been at this game for many years and am fairly fluent in communications, but there's always room for new knowledge. If I am misreading the facts, please point me in the right direction and I'll cheerfully learn something new.

While we need to be a good citizen when transferring across the public networks, on private networks it is very reasonable to push the envelope. I spend a lot of time on other peoples networks (as a consultant... not a hacker!) and I haven't seen anyone running <GbE data centers in a very long time. In fact, many of my customers are now standardizing on 10GbE... even to some IT desktops. With a target utilization for 100Mb FileZilla seems to have an artificial cap that hurts more than it helps.

Either way though, I have and will continue to cheerfully donate funds, tell everyone I can about this great product, and sincerely appreciate this team's efforts. You folks are doing a great job.

PS: An overall status line that shows total current throughout would be very helpful for this! :-)

[pboeyke1]

Absolutely correct, 21st century compute/storage and network infrastructure easily support hundreds, thousands of concurrent SFTP/FTP sessions (using CrushFTP running as much as 4000+ concurrent sessions)
State of art SSD drive technology supporting I/O speeds in excess of 300MB/Sec some high-end compute&storage arrays offering even faster throughputs.

Especially on high-speed, low latency LANs, inside core EDCs, enforcing 10 session limit, limits staff ability transferring files efficiently. Networks&compute resources are not optimally utilized, with storage/network throughput often seen as low <10% of available (peak) capacity.

Personally like FileZilla, great tool, great community, good features and love to see this limitation removed!
Keep up the good work.