Support multi-threaded multipart downloads

[mrcapture]

FileZilla is really great software, but the feature I am
missing most is the ability to actually download ONE
file using multiple parallel connections. Many FTP
servers (including the one of the company I work for)
limit the bandwidth per connection, very often this limit
is in the 20kB/sec range. I was using LeechFTP before
that supports multi-threaded multi-part download (using
multiple parallel connections) but development has been
discontinued and it does not support any security (SSL).
Whenever you have a high-speed internet connection
and need to download a large file (e.g. CD image) I was
able to cut the download time by over 75% using this
feature.

Would be great if you could consider this for a future
release.

Thanks,

Stephan

[codesquid]

Won't be implemented. The overhead from using multiple
connections reduced overall available bandwidth.
If someone uses multiple connections, all other users have
to pay for this by slower transfer speed

[slacka]

I agree that a without limits, this feature has the ability for abuse. However, a reasonable number like 5 or 10 connections would be a good compromise.

I first used this feature in leech FTP in 1998 and since then nearly all of modern FTP clients have added support for this feature (cuteFTP, ws-ftp, leechftp) in addition to all download accelerators for web browsers like downthemall use it. This has not led to the any major problems, so I see no reason why filezilla cannot keep up with the times.

[codesquid]

See my previous reply.

[slacka]

CodeSquid ,

I have read your previous reply. However, your argument for not supporting this feature “If someone uses multiple connections, all other users have to pay for this by slower transfer speed” is only valid in certain low bandwidth situations. Nearly all proprietary FTP clients support this multi-part downloads. So in these low bandwidth situations, the FTP server administrator can and should easily prevent multi-part download by not support resume or limiting maximum simultaneous connections.

The aggregate bandwidth used by a multipart transfer is identical to a single stream transfer. A user transfering a file in 2 parts, gets the file twice as fast, but also frees up bandwith for other users in ½ the time.

Using Multi-part download is no more demanding of bandwidth than using filezilla’s feature “multiple connections to transfer files”. If mult-part download is really so bad, then shouldn't this feature be removed from filezilla?

[codesquid]

Strictly speaking I'd have to remove the option for multiple simultaneous transfers. (The public outcry would be gigantic though)

At least no bandwidth is wasted with that option. Compare it to multi-threaded downloads. Let's ignore the fact that you'd double the control connection overhead. Even then the largest cause of overhead remains: The FTP protocol has no range command. You can only give an offset at which to start the transfer, but cannot tell the server to only send you that many bytes. The server always prepares to send you the full file starting from the resume offset.
Now assume a file of 1MiB and you want to only get the first half. Resume offset 0. After receiving the 512KiB you've got all the data you want, yet the server is still sending. So you have to forcefully close the connection. Now at that point, the server has already sent you a lot more data you just didn't receive yet. Depending on the network configuration it can be many megabytes which are still in various buffers or on route to the destination. Furthermore, the server or operating system likely has already read-ahead a significant portion of the file, wasting RAM and causing the HDD to wear down.

[jminn]

From a network administrator's perspective, codesquid is technically correct. Implementing this feature would waste a little bandwidth. But practically every other FTP client out there already supports this feature, so I think it should be up to us admin to support this feature or not. Besides for massive files, the bandwidth wasted would be a very small percent of the total bandwidth used.

It's great that FileZilla supports multiple simultaneous transfers. The problem is that these don't help at all if you are transferring only one single large file. For that you need full multi-part transfers. The existing framework in filezilla for pipelined transfers could be used to help implement multi-part transfers. When the top file in the transfer queue is large, you simply split it into several partial transfers, and then use pipelined transfers on those parts as if they were different complete files, filling up the transfer slots as usual. You still have to handle the pre-allocation of the file, and filling in the data into the right place for each part.

Codesquid does raise a good point about the FTP protocol lacking a range command. Couldn’t FileZilla address this deficiency in the FTP protocol with an extension? Initially only FileZilla clients connecting to FileZilla servers would support it, but if it was useful and implemented well, it could be adopted as an official extension to the FTP protocol, like RFC 2228. What do you think?

[rymo]

"...the largest cause of overhead remains: The FTP protocol has no range command...you have to forcefully close the connection"

...not so with SFTP! Regardless, we're only interested in this feature on really big downloads. If you're transferring a single 1GByte file with multiple FTP connections and end up with some overlap at the end, that's pretty minuscule overhead. If you are really that worried about wearing out server HDDs (seriously?!) then only enable multiple connections above a certain filesize threshold.

Parallel downloads need not and should not an on-by-default feature, but it's a vital option for enhancing poor connections just as custom PASV setting tweaks are necessary in poor server/network setups.

We're on a wireless connection here and for various reasons we can't get more than 2Mbps throughput on a single download connection -- yet run 10 or 20 parallel connections on the same server (a server we own, by the way) using axel under linux or Cygwin and we're able to max out our 10Mbps allocation on that single download. Why shouldn't we expect to accomplish the same in our favorite FTP client, FileZilla? Don't dumb down the client for fear of dumb servers.

[cookiecaper]

This really should be implemented. I use FileZilla as a client almost exclusively on private servers which I own and operate, and I want to use multiple connections per file. Why should I not be able to on my own server?

There are many measures server admins can take to protect themselves from abuse by excessive parallel connections. And as already stated, most other FTP clients support this feature, so FileZilla will not cause a massive influx of new unblockable abuse.

Don't deprive your users, many of whom have very legitimate uses for this, in favor of a smaller number of incompetent admins.