WRT110 router - FTP upload problem - Timeouts on large files

[ftpeter]

WRT110 router - FileZilla Client, FTP upload problem

- Timeouts on large files - ==

When I upload files via FTP and if the transfer of a file takes more then 3 minutes,

then my FTP programs doesn't jump to the next file to be uploaded when the previous one has just been finished.

The problem occurs only if I connect to the internet behind my wrt110 router.

I have tried FileZilla, FTPRush and FlashFXP
FTP softwares to upload via FTP.

At FileZilla sending keep-alives doesn't help,

BUT FTP upload properly works at FlashFXP if I send noop during transfer:

Sending anti-idle keep alives also does not work at FlashFXP from behind wrt110 router.

​http://www.abload.de/img/wrt110passthrough1eq0a.jpg

​http://www.abload.de/img/wrt110passthrough3crrp.jpg

At FileZilla I can right-click->Refresh the FTP window during transfer,
but I can't do that at FlashFXP.

This might be connected with the problem I have written above, I don't know.

Please if you can, put similar solution into FileZilla as it is in FlashFXP.

Thanks.

This is exactly what I am experiencing --> On FileZilla's site I can read this:

http://wiki.filezilla-project.org/Network_Configuration#Timeouts_on_large_files


'''Timeouts on large files'''

If you can transfer small files without any issues,

but transfers of larger files end with a timeout,

a broken router and/or firewall exists

between the client and the server and is causing a problem.

As mentioned above,  

FTP uses two TCP connections:

a control connection to submit commands and receive replies,

and a data connection for actual file transfers.

It is the nature of FTP that during a transfer the control connection stays completely idle.


The TCP specifications do not set a limit on the amount of time a connection can stay idle.

Unless explicitly closed, a connection is assumed to remain alive indefinitely.

However, many routers and firewalls automatically close idle connections after a certain period of time.  

Worse, they often don't notify the user, but just silently drop the connection.

 

For FTP, this means that during a long transfer

the control connection can get dropped because it is detected as idle, but neither client nor server are notified.

So when all data has been transferred, the server assumes the control connection is alive

and it sends the transfer confirmation reply.

 

Likewise, the client thinks the control connection is alive and it waits for the reply from the server.

But since the control connection got dropped without notification,

the reply never arrives and eventually the connection will timeout.


In an attempt to solve this problem,

the TCP specifications include a way to send keep-alive packets on otherwise idle TCP connections,

to tell all involved parties that the connection is still alive and needed.

 

However, the TCP specifications also make it very clear that these keep-alive packets should not be sent more often than once every two hours. Therefore, with added tolerance for network latency, connections can stay idle for up to 2 hours and 4 minutes.

 

However, many routers and firewalls drop connections that have been idle for less than 2 hours and 4 minutes. This violates the TCP specifications (RFC 5382 makes this especially clear). In other words, all routers and firewalls that are dropping idle connections too early cannot be used for long FTP transfers. Unfortunately manufacturers of consumer-grade router and firewall vendors do not care about specifications ... all they care about is getting your money (and only deliver barely working lowest quality junk).

To solve this problem, you need to uninstall affected firewalls and replace faulty routers with better-quality ones.

[Sergio Abreu]

I am facing the same problem. I think it is because the router has a property called MTU size, which is set to 1500. Check the router tutorial about it: "MTU Size - The normal MTU (Maximum Transmission Unit) value for most Ethernet networks is 1500 Bytes. For some ISPs you need reduce the MTU. But this is rarely required, and should not be done unless you are sure it is necessary for your ISP connection."
So, if the file is < 1500 bytes, it is transfered immediately, with no errors.
If you try to transfer files with more than 1500 bytes, it only works if you are NOT under the router.
The strange thing is that if you sent the same file using a web browser (under the router) the upload happens, it seems that internet browsers have any mechanism to keep the connection alive ? I wrote a tool to edit and send textfiles using http using server side scripts to save the new file.

I found out that some websites like twitter has the same problem that Filezilla has: it only works well if you are not under a router.

One idea that may be used to solve the problem - I don't know about the ease or difficulty to implement that - is to divide the file in small pieces of, lets say 1400 bytes BEFORE sending via ftp. The server would have a way to identify and join the corret files parts.
Let's work to improve this... remember: why can a browser upload large texts (much ore than 1500 characteres) and Filezilla gets lost if a file has more than 1500 bytes and you are under a MTU Limited router ???
There must be a way to improve it. Lets try to do it, Filezilla.

[Sergio Abreu]

I found that Filezilla gives us a way to minimize this problem, by the timeout in the Connection configuration. This value comes with - correct me if I am wrong - 120 seconds by default.
If you increase this value, it seems less probable that the connection will be lost. Now my upload can take more time, it seems that helps.

[Tautvydas Andrikys]

Fix for unix based servers

[Tautvydas Andrikys]

Attached possible fix it just decreases keep-alive packet frequency it should help in this case

[Sergio Abreu]

More detail about the problem is:
Sometimes when I try to send a file bigger than the network packet size - defined in MTU, there is a very annoying thing that happen:
1- The target file is reseted (size gets 0bytes)
2- Filezilla shows me the overwrite dialog, I say yes.
3- Filezilla tries to upload again, the progressbar sometimes says it transfered 100% but the process enters a loop, with the file being always with 0 bytes.
Files with size next to 1400 bytes never get crashed... I wonder if Filezilla could break the file internally and then joing the pieces once it gets the server...

[Sergio Abreu]

Another thing to think is:
WHY if I send the file via HTTP (I wrote kind a upload tool) it goes without problem? Why only ftp transfers gets in trouble when I am under a router and sending files bigger than 2000 bytes?

[Sergio Abreu]

I THINK I HAVE JUST FOUND THE SOLUTION !

Ajusting the MTU size in order NOT to fragment network packets, the ftp uploads seem to work fine now !!!

The trap is, looking at the router MTU max is 1500 bytes people tend to think that "the most the better...", in fact it is NOT true. MTU size bigger than the ISP configuration causes packet FRAGMENTATION. You must find your best ISP value for MTU, here I am using 1440 now. To see if 1500 causes packet fragmentation, use ping with these flags and a variable packet size, like this:

cmd:\> ping google.com -f -l 1500

and adjust it to 1490, 1480... 1440 until you see that is not fragmenting the packets.

I hope this is the final solution to everyone who follows this topic.
I am really upset how I could be facing this problems for years until discovering it now ... I am really glad now.

[REMY]

cannot even connect to filezilla - first tim in many months ??

[REMY]

impossible to get connection with the server since this pm dec. 23rd 2013 ???
in spite of correct ID & password :-(

please HELP to solve this ASAP

id = paroisse.stlhomer61
host = ftpperso.free.fr
my email contact: jean.remy@…

thks

[Alexander Schuch]

Replying to ftpperso.free.fr:

impossible to get connection with the server since this pm dec. 23rd 2013 ???
in spite of correct ID & password :-(

Please do not hijack this bug report. Your issue is completely unrelated. Please talk to your system or network administrator or to your support contact for further help. There is nothing we can do about this.

[Alexander Schuch]

The problem is fixed? The reason is a broken router or wrongly configured MTU? FileZilla is fine?

[Sergio Abreu]

For me it was due MTU configuration, so I believe Filezilla is fine.

[tanyado]

I had this problem for a long time.
My friend recommended me Long Path Tool for fixing this.
I am skeptic with this king of tool but I was wrong.
This tool can do anything.