At 16,385 files, Filezilla goes into an infinite loop

[bill B]

I have done 3 times.
I am using FileZilla 3.5.1 on my PC, MS-Windows, Ver 5.1(Build 2600.xpsp_sp3_....:Service Pack 3)
I want to transfer 62,424 files, with names like PHO_0xxxxxxx-5_1003457.HL7;1 to an USB drive (E:) on my PC. I am connected to a VAX OpenVMS V8.2.
To gather the names (LIST in FTP) takes about an hour.
I then do a CTRL-A on the names in the "remote directory" screen, and put it into the "Home" screen. I am also watching My "Task Master" Perforamce tab, to watch the CPU Usage screen. After about 45 minutes, CPU Usage becomes 100% usage and no more files are being copied. I "stop" Filezilla and check the number of files on my USB "Memery stick" and all 3 times the same number files, 16,385. 16,385 = 214 + 1.
These files are anywhere 1KB big to 120KB big. And they are 100% text files, or HL7 files.

If possible, let me know.
Thanks

[Josh Davis]

Should you really be putting HL7 files on a USB drive? Hopefully it's a TrueCrypt volume, or similar. Check with your HIPAA compliance officer to be sure. It may be better to NFS mount on your private LAN and rsync the directory.

Anyway, to your point, a selection list with that many items is a struggle for Windows, and has been an issue for a long time. Even so, 3276x was the old limit, so that should be fine.

I'm not a dev, but looking at the code, it seems like the command queue is a linked list, not an index. As such, 65k files should be slow to parse, and consume substantial CPU, but shouldn't fail.

The next thing to check is your USB drive itself. If you're using FAT32, then the limit is 65,534 items per directory. However, your HL7 files are over 8.3. Long file names consume between 1 and 20 directory entries each, depending on the length. I think if you format your USB drive using NTFS, you should find that you can receive more HL7 files.

If this is not the case, then update here again, and perhaps someone can dig into the code further and perform a test-case using synthetic data.

Because this is a limited scope problem, and can be worked around, I'm dropping the priority from "critical" to "high". I think High is probably not appropriate either, so please update with your results.

[Alexander Schuch]

No reply for more than 28 days.

Please provide logs. - ​http://trac.filezilla-project.org/wiki/TicketSubmissionGuide#Logs

As you transfer lots of files and the log might be very large, please start with providing only the end of the log, for example the last 250 lines after FileZilla starts using 100% CPU.