ftp uploading: file corruption

[nRookie]

Hi All,

I recently encountered a very strange file corruption problem.

The scenario is that I upload a file from my PC(macOS) through FTP protocol with Filezilla(v3.55.1) to an FTP server(windows).

Note:

I have set the FTP mode to passive and FTP transfer mode to binary.

if the file size is bigger than 1M, it would be corrupted, but there are no errors shown at Filezilla. (File transfer successful, transferred 6014038 bytes in 12 seconds)
I check the MD5 of the file at the server-side. Which has diverted from my client-side.
I capture the packet of the communication between server and client with Wireshark, does not find any explicit error.
I also hex dump the content of the original file at the client-side and the file at the server-side and compared them, the former 16385 bytes are different, but the remaining bytes are the same. (approximately 360000 bytes).

More info:

if the file size is smaller than 100KB. It could be uploaded successfully.
if the upload speed is limited under 500KB/s(enable speed limit in FileZilla), the larger file could be uploaded successfully.
the file could be uploaded by other FTP clients to the same FTP server without corruption, so the FTP server is working fine.

I tried other FTP client, which can also upload the files to server without corruption.

I although check with my two colleagues' PC (both MacOS Bigsur), they can upload with FileZilla(same version) successfully without corruption.

at first, I doubt that there would be some problem with my network condition, but the md5 of the file I uploaded is the same every time even if it is corrupted.

Any help or tips would be very appreciated.

Best regards,
nRookie

[Tim Kosse]

Did you do the Wireshark capture on the client or the server machine? If client-side, it is captured before it goes through the network adapter out on the wire, thus you wouldn't detect an issue with the networking hardware.

In any case, if the file is correct in the packet capture, it's not an issue with FileZilla.

[nRookie]

Replying to Tim Kosse:

Did you do the Wireshark capture on the client or the server machine? If client-side, it is captured before it goes through the network adapter out on the wire, thus you wouldn't detect an issue with the networking hardware.

In any case, if the file is correct in the packet capture, it's not an issue with FileZilla.

I captured the packet on the client machine.

But I did try with another FTP client, which can upload the file without any corruption.

[Tim Kosse]

But I did try with another FTP client, which can upload the file without any corruption.

Different clients behave differently. From the used syscalls and their parameters, the subset of FTP commands used and the timing of everything, no two clients exactly the same. Sometimes an underlying fault in the system is only triggered in very specific circumstances.

Case-in-point: Setting a speed limit, thus changing the timing, appears to make the error go away.

Are you using any firewalls or AV products?

Would it be possible to attach the file in both of its states so that I can look at the exact nature of the corruption?

[nRookie]

this is the original file.

[nRookie]

this is the corrupted file.

[nRookie]

Replying to Tim Kosse:

But I did try with another FTP client, which can upload the file without any corruption.

Different clients behave differently. From the used syscalls and their parameters, the subset of FTP commands used and the timing of everything, no two clients exactly the same. Sometimes an underlying fault in the system is only triggered in very specific circumstances.

Yeah, that makes a good point, we suspected that there might be something wrong with the process of the application reading the contents of the file into memory before sending it through the network with FTP.

Case-in-point: Setting a speed limit, thus changing the timing, appears to make the error go away.

Are you using any firewalls or AV products?

I disabled the default firewall of Mac.
McAfee is installed on both my and my colleagues PC, and they should apply the same policy setting which is set by Our IT department. (and as I mentioned earlier they can upload with Filezilla without corruption). So I do not think this is the case.

Would it be possible to attach the file in both of its states so that I can look at the exact nature of the corruption?

I attach the original file and a corrupted file.

btw, Thanks a lot for your kindly reply.

[Tim Kosse]

Interesting. All the data is there, just not in the correct order.

Remove the first 64KiB of the file and then re-insert it after the next 3*64KiB data of the file to transform the working one into the corrupt one. Makes no sense, there is nothing in FileZilla that could result in such reordering of data.

What kind of hardware do you have, is it an Intel or an ARM-based Mac? What about your colleagues?

[nRookie]

Replying to Tim Kosse:

Interesting. All the data is there, just not in the correct order.

Yeah sure it is.

Remove the first 64KiB of the file and then re-insert it after the next 3*64KiB data of the file to transform the working one into the corrupt one. Makes no sense, there is nothing in FileZilla that could result in such reordering of data.

What kind of hardware do you have, is it an Intel or an ARM-based Mac? What about your colleagues?

Both of my colleagues and I are having ARM-based Mac.

[nRookie]

I also have asked for other colleagues to help me check with their Macs.
there are 5 Macs including mine.

1. McAfee installed, Big Sur 11.4, corrupted.
2. McAfee installed, Big Sur 11.52, no corruption.
3. McAfee installed, Big Sur 11.3.1, corrupted.
4. McAfee uninstalled Big Sur 11.3.1, no corruption.
5. McAfee uninstalled Big Sur 11.2.2, no corruption.

After I upgrade my system version to 11.6, I can upload the file without corruption.

Besides, a Windows PC, not sure if the firewall is installed or not, is corrupted.
I will update as soon as I get more information about Windows PC.

Any ideas?

[nRookie]

Hi Tim Kosse:

Thanks for your kindly support.

since the problem can be fixed by upgrading macOS system version.

for macOS version smaller than 11.4 with McAfee installed, I think we both do not have time to go into details.

thus, I will close this ticket.

Best regards,
nRookie

[nRookie]

Hi Tim Kosse:

Thanks for your kindly support.

since the problem can be fixed by upgrading macOS system version.

for macOS version smaller than 11.4 with McAfee installed, I think we both do not have time to go into details.

thus, I will close this ticket.

Best regards,
nRookie

[Tim Kosse]

Firewalls and AV products, the bane of my existence.