Failed to retrieve directory listing after connecting in TLS 1.3 protocol

[AllaG]

The remote server on Linux RH.

[AllaG]

FileZilla Client log

[Tim Kosse]

The FTP server software for some reason cancels the TLS handhshake.

Please contact your server administrator for assistance to find out why the server cancels the handshake.

[AllaG]

Remote server is Apache Mina ftpserver-core-1.1.1

[AllaG]

I tested FileZilla client connection to the same Apache FTP over SSL server:
1) server configured to use TLS 1.2 protocol
2) server configured to use TLS 1.3 protocol

The directory listing fails in TLS 1.3.

Please see attached 2 FileZilla client log files in debug: filezilla_tls12.log, filezilla_tls13.log

Server compiled an run on Java 11 and java.security configuration is default one.

*FileZilla using GnuTLS library. May be you can consider to foreword the issue to GnuTLS.

Thanks.

Alla

[AllaG]

The remote server doesn't cancel the TLS handshake.

[VBaker]

We have bumped into the same thing. The latest version of FileZilla Client will not connect to CrushFTP.

• - - -

Command: PWD

Response: 257 "/" PWD command successful.

Command: TYPE I

Response: 200 Command ok : Binary type selected.

Command: PASV

Response: 227 Entering Passive Mode (x,x ... x,x)

Command: MLSD

Response: 150 Opening data connection for file list.

Error: Received TLS alert from the server: User canceled (90)

Error: Could not read from transfer socket: ECONNABORTED - Connection aborted

Response: 226 Directory transfer complete. (generate:128ms)(send:332ms)

Error: Failed to retrieve directory listing

• - - -

Older versions will connect just fine. We believe that TLS1.3 is somehow killing the connection during a directory read. Very odd, and no apparent way of working around the situation.

[VBaker]

Just a little extra data. FileZilla seems to work fine with 3.34.0, but newer installs do not.

P: FTP - File Transfer Protocol
Port: 1090
Require Implicit FTP over TLS
Normal

[Tim Kosse]

Contact your server vendor for assistance to have them figure out why the server sends a user canceled alert.

[spinkb]

Re-openning this as your reply saying to contact the server vendor got my attention. FZ 3.51.0, build 2020-10-20, running on macOS 10.13.6.

I am a server vendor for example, and I cannot make FZ client work with our server. For reference though, with absolutely no change at all, the following other clients *DO* work with TLS v1.3...

HTTPS with Chrome, Edge, FireFox.
FTPES with Cyberduck, WinSCP, Transmit.

Only FileZilla client complains...and it looks to be a false complaint at that. I tried a dir listing, and the entire dir listing shows in the FZ log, but apparently it doesn't like how the socket is closed...? Session is never disconnected, and the entire dir listing is there...it just didn't display it for unknown reasons.

Every Java VM I have used CrushFTP Server on with TLS v1.3 works with all other TLSv1.3 clients *except* FileZilla. We didn't make a single code change in CrushFTP to support TLSv1.3...nor should we need to as the underlying JVM since Java 13+ has supported TLSv1.3 in a stable manner.

If the only change I make is setting TLSv1.2 as the TLS version for my ServerSocket, then FZ is fine. But if I swap it back to TLSv1.3, the FZ fails. Considering all other clients I tried worked, and the entire dir listing is in the log, I don't believe there is any bug in multiple different server vendors code bases for TLSv1.3. This looks like a FZ client bug. The connection for the data socket is closed identically with TLS v1.2 or TLSv1.3. One works, one doesn't.

Can you revisit this in more detail and test it out?

20:13:05 Status: Logged in
20:13:05 Status: Retrieving directory listing...
20:13:05 Command: PWD
20:13:05 Response: 257 "/" PWD command successful.
20:13:05 Command: TYPE I
20:13:05 Response: 200 Command ok : Binary type selected.
20:13:05 Command: PASV
20:13:05 Response: 227 Entering Passive Mode (127,0,0,1,196,46)
20:13:05 Command: MLSD
20:13:05 Response: 150 Opening data connection for file list.
20:13:06 Listing: Type=dir;Size=0;Modify=19700101000000;Perm=e,l;UNIX.owner=user;UNIX.group=group; ..
20:13:06 Listing: Type=dir;Size=0;Modify=19700101000000;Perm=e,l;UNIX.owner=user;UNIX.group=group; .
20:13:06 Listing: Type=dir;Size=2;Modify=20120924131136;Perm=r,e,l,m,c,d,f,p;UNIX.owner=user;UNIX.group=group; #testdir
20:13:06 Error: Received TLS alert from the server: User canceled (90)
20:13:06 Error: Could not read from transfer socket: ECONNABORTED - Connection aborted
20:13:06 Response: 226 Directory transfer complete. (generate:542ms)(send:593ms)
20:13:06 Error: Failed to retrieve directory listing
20:13:36 Status: Sending keep-alive command
20:13:36 Command: NOOP
20:13:36 Response: 200 Command OK. (NOOP)
20:14:06 Status: Sending keep-alive command
20:14:06 Command: NOOP
20:14:06 Response: 200 Command OK. (NOOP)

[Tim Kosse]

Please have a look at the Java source code, in particular look at these lines:

​https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/sun/security/ssl/SSLSocketImpl.java#L555
​https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/sun/security/ssl/SSLSocketImpl.java#L598

Your FTP server is not indicating that it's done sending data by explicitly shutting down the output, as result Java is implicitly initiating a weird duplex close that abuses the user_canceled alert where no user is involved.

The solution is a one-line fix: Explicitly call SSLSocket::shutdownOutput() prior to (possibly indirectly) calling SSLSocket::close()

[Tim Kosse]

The connection for the data socket is closed identically with TLS v1.2 or TLSv1.3. One works, one doesn't.

Actually no, there has been a semantic change in the protocol specification how connection closure works.

To quote ​RFC 8556:

Each party MUST send a "close_notify" alert before closing its write
side of the connection, unless it has already sent some error alert.
This does not have any effect on its read side of the connection.
Note that this is a change from versions of TLS prior to TLS 1.3 in
which implementations were required to react to a "close_notify" by
discarding pending writes and sending an immediate "close_notify"
alert of their own. That previous requirement could cause truncation
in the read side. Both parties need not wait to receive a
"close_notify" alert before closing their read side of the
connection, though doing so would introduce the possibility of
truncation.

My best guess is that Java sends user_canceled in an ill-fated attempt to emulate the previous behavior.

[spinkb]

Debugged it down to the line of code that makes FZ report an error. When the output stream is closed (not the socket), FZ reports an error. Its not something a java app can control as far as I can tell. Yes, the TLS debug literally indicates "user_canceled" in the debug log...but that apparently is how OpenJDK does it, and not something I can control. I attached the debug output of this in the TXT file of this ticket. debug_on_close_outputstream.txt

So you are saying OpenJDK has implemented things incorrectly?

[Tim Kosse]

When the output stream is closed (not the socket), FZ reports an error.

Try calling shutdownOutput on the SSLSocket prior to closing the output stream.

So you are saying OpenJDK has implemented things incorrectly?

Yes, their approach is wrong, the right thing to do is to document the semantic change if TLS1.3 is used, instead of attempting to emulate the old behavior through misuse of the user_canceled alert.

In particular, closing the OutputStream should not perform a bi-directional close with TLS 1.3 as each direction can be closed independently in TLS 1.3

In case of FTP uploads for example, the server could immediately after the TLS handshake shut down the write side of the TLS channel, it should not affect the read side with TLS 1.3 at all.

Same with downloads, the client could immediately close its write site after the handshake. The server would receive a close_notify and would then continue sending out the file normally.

[spinkb]

I can work around this, but I'm still not certain your interpretation is correct...?

If it were, then why does every other client accept the OpenJDK behavior? Wouldn't Chrome be complaining too? And FireFox and so on? Cyberduck is Java based, so I could see why it might be different, but the HTTPS clients would surely be upset too...

And considering the canceled is immediately followed by a close, you know it was "done" at that point... Its no reason to abort the dir listing and not show it. The data transfer is complete at that point, cancelled, and closed...

I can work around the issue with this, but it feels wrong...

data_osw.flush();
if (data_sock instanceof SSLSocket) ((SSLSocket) data_sock).shutdownOutput();
data_osw.close();

[Tim Kosse]

If it were, then why does every other client accept the OpenJDK behavior?

Different TLS libraries or less strict implementations. FileZilla takes security serious and complains about many issues other clients don't care about.

Also, this argument could be turned around: Why is it that only OpenJDK sends these user_cancelled alerts, with no other TLS implementation doing it?

Wouldn't Chrome be complaining too? And FireFox and so on?

Web browsers do not speak FTP over TLS.

For HTTPS, make sure to try responses with identity transfer encoding and a no content-length header in the reply, thus requiring closing the connection to indicate end of response body. Only in this scenario is it actually important to know that the connection has been closed securely to detect truncation attacks. If either a Content-Length is given or chunked transfer encoding is used, complete transmission of the response body can be determined without needing to know whether the connection has been properly closed.

And considering the canceled is immediately followed by a close, you know it was "done" at that point... Its no reason to abort the dir listing and not show it. The data transfer is complete at that point, cancelled, and closed...

The semantics of the cancel alert are not well defined for this use case. It could just as well mean that the data has not been fully transferred yet and that the user has canceled the transfer. Afterwards the connection is just closed normally.

Lacking defined semantics outside the handshake, it is more than reasonable to assume that the transfer has not actually been performed completely given the name of the alert.

OpenJDK is assuming particular semantics on user_canceled that aren't covered by the specifications.

I can work around the issue with this, but it feels wrong...

Feel free to patch OpenJDK instead to remove sending of the user_cancelled alerts.

[AllaG]

Hello,

We are still experiencing the disconnecting issue during directory listing with TLS 1.3 protocol, after upgrading to Java 11.0.11 with bug fix for TLS 1.3 Half-Close Policy.

The OpenJDK bug that seams to be related to this issue:
​https://bugs.openjdk.java.net/browse/JDK-8208526

​https://bugs.openjdk.java.net/browse/JDK-8208538

​https://bugs.openjdk.java.net/browse/JDK-8241453

See TLS 1.3 Half-Close Policy in
​https://www.oracle.com/java/technologies/javase/11-relnote-issues.html

Please see attached file "FZ TLSv1.3 connection to server on Java 11.0.11.txt​"
includes java ssl debug log of server on Windows.

I tried using this solution by setting -Djdk.tls.acknowledgeCloseNotify=true
and it didn't work.

Kindly check if this OpenJDK bug fix could be related to the ticket's issue.
How we can proceed from here to make it work?

Regards,

AllaG

[spinkb]

Can you try with a newer Java version? My experience has been TLSv1.3 is buggy in Java 11...it starts working as of Java 13+. So update your JVM version.

This is at least true for CrushFTP as its Java based. We couldn't even function with Chrome until Java 13 was released. That flag may work fine in Java 13/14/15/16 etc...but may not work correctly in Java 11.

[AllaG]

Yes, I tried also to work on Oracle jdk-14.0.2 with setting
-Djdk.tls.acknowledgeCloseNotify=true
I got the same error in FileZilla:
Command: MLSD
Response: 150 File status okay; about to open data connection.
Error: Received TLS alert from the server: User canceled (90)
Error: Could not read from transfer socket: ECONNABORTED - Connection aborted
Response: 226 Closing data connection.
Error: Failed to retrieve directory listing

[spinkb]

Whoever is the server owner then has to change something to be compatible with FZ's interpretation of how TLS1.3 works.

We did that with CrushFTP to make it compatible with FZ. But I don't know what vendor thats server is.

This flag, after reading about it more, appears to control different behavior not relevant for this.

[Tim Kosse]

FileZilla does not interpret how TLS 1.3 works. It is Java's fault for assuming particular semantics on user_canceled that aren't covered by the specifications.

[AllaG]

From our investigation the issue depends on two factors.
First:
GnuTLS returns
| -12 | GNUTLS_E_FATAL_ALERT_RECEIVED | A TLS fatal alert has been received. |

on receiving "user_canceled" alert sent from OpenJDK server,

that according to documentation in
​https://www.rfc-editor.org/rfc/rfc8446#section-6.1
should be treated as warning.

I have open a bug to GnuTLS to change the error code returned to:
| -16 | GNUTLS_E_WARNING_ALERT_RECEIVED | A TLS warning alert has been received. |

Second:
FileZilla client
In tlssocket.cpp
CTlsSocket::Read
int res = gnutls_record_recv(m_session, buffer, len)
​https://github.com/basvodde/filezilla/blob/master/src/engine/tlssocket.cpp#L546

aborts connection and expects only for status
| 0 | GNUTLS_E_SUCCESS | Success. |
to close it right.

So, FileZilla should add additional status to treat warning alert, as suggested to GnuTLS (-16).

Please cooperate this fix with GnuTLS.

Regards,
AllaG

[Tim Kosse]

Thank you for the heads-up. If the behavior in GnuTLS changes to treat this alert as a warning instead, future versions of FileZilla will have to explicitly handle user_canceled as an error, to preserve the meaning of the alert as defined in the RFC: post-handshake it cancels an operation.

Nothing will change for you, the transfer will still fail. Just the error message will no longer speak of a fatal alert.

[Maria Brag]

Replying to Tim Kosse:

Thank you for the heads-up. If the behavior in GnuTLS changes to treat this alert as a warning instead, future versions of FileZilla will have to explicitly handle user_canceled as an error, to preserve the meaning of the alert as defined in the RFC: post-handshake it cancels an operation.

Nothing will change for you, the transfer will still fail. Just the error message will no longer speak of a fatal alert.

You are misunderstanding the meaning of user_canceled alert as it is described in TLS 1.3 RFC ​https://www.rfc-editor.org/rfc/rfc8446.

user_canceled: This alert notifies the recipient that the sender is canceling the handshake for some reason unrelated to a protocol failure. If a user cancels an operation after the handshake is complete, just closing the connection by sending a "close_notify" is more appropriate. This alert SHOULD be followed by a "close_notify". This alert generally has AlertLevel=warning.

If user_canceled alert is sent after the handshake is complete the recipient should close the connection as it would do for close_notify alert. In case of Java SUN TLS 1.3 implementation, provider is sending user_canceled alert and close_notify alert together.
So though using user_canceled alert is kind of a workaround, it is still preserving semantics of TLS 1.3 RFC.

We are expecting that Filezilla FTP client will follow this semantics too.

[Tim Kosse]

The RFC is quite clear, post-handshake it talks about the user cancelling an operation. Clearly and by definition a cancelled operation is not a successful operation.
The advice given by the RFC to instead just send a close_notify is actually not more appropriate, as doing so would make it impossible to distinguish between successful and cancelled operations.

The correct approach is to get rid of user_canceled as part of the normal connection closure in Java. Don't attempt to emulate TLS 1.2 connection closure behavior in TLS 1.3 through user_canceled.

If you worry about TLS 1.3 breaking existing Java applications that rely on the 1.2 connection closure behavior, you can always disable TLS 1.3 by default and make it opt-in by application developers so that they have a chance to test and adapt their software.

[Tim Kosse]

This seems to be related: ​https://gitlab.com/gnutls/gnutls/-/issues/1451

[aduzsardi]

so the conclusion is that every other client is wrong, only yours is correct... awesome

[Tim Kosse]

There are lots of other FTP clients and servers that are correct. It's just the third-party software you are using that isn't. Have you yet contacted your third-party software vendor for assistance?

Closing this ticket, as the fault is with the third-party software.

[aduzsardi]

Replying to Tim Kosse:

Have you yet contacted your third-party software vendor for assistance?

I did , thanks for the suggestion , waiting for a solution from the apache-mina project