Stable way to fetch sources for packaging

[Robin Candau]

Hi,

I'm the Arch Linux package maintainer for the filezilla & libfilezilla packages [1][2].

Recently the website's download link for sources became unstable. The related ticket [3] was closed explaining that this was intentional, in order to prevent eventual infrastructure abuses.

While I totally understand the desire to avoid such potential abuses, this causes problems for downstream distributions packaging & redistributing filezilla / libfilezilla.

Indeed, due to its (now) unstable nature, the website download link cannot be used to fetch sources in our packaging workflow anymore (both to fetch valid sources and for reproducibility matters).
I studied switching to SVN sources but it seems tags are not created there anymore [4][5], so we can't really use that either (unless we pinpoint specific revisions but this isn't really desirable)...

As such, distributions have started mirroring filezilla / libfilezilla tarballs on their side (see Open-Suse [6], Alpine [7] & Gentoo [8] for instance).
Of course, this technically works, but this represents significative extra burdens in our packaging workflow and automation.

Can a solution be discussed? If a stable download link is not desirable on your side (regarding eventual infrastructure abuses), would you eventually consider creating tags again on the SVN repo so distributions can use that as a source for their packages?

Since the download page for filezilla [9] "highly recommends to use the package management system of distributions", I think it's fair to ask for a way to facilitate packaging for such distributions.

I remain available to discuss eventual solutions or if any additional information is needed!

[1] ​https://archlinux.org/packages/extra/x86_64/filezilla/
[2] ​https://archlinux.org/packages/extra/x86_64/libfilezilla/
[3] ​https://trac.filezilla-project.org/ticket/13159#no4
[4] ​https://svn.filezilla-project.org/filezilla/FileZilla3/tags/?sortby=date
[5] ​https://svn.filezilla-project.org/filezilla/libfilezilla/tags/?sortby=date#dirlist
[6] ​https://build.opensuse.org/projects/openSUSE:Factory/packages/filezilla/files/filezilla.spec#L=29
[7] ​https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/community/filezilla/APKBUILD#L24
[8] ​https://gitweb.gentoo.org/repo/gentoo.git/tree/net-ftp/filezilla/filezilla-3.68.1.ebuild#n15
[9] https://filezilla-project.org/download.php?type=client

[Tim Kosse]

the website download link cannot be used to fetch sources in our packaging workflow anymore

Please clarify: Is this about the workflow of a) creating a new package, or version of it, or b) working with already created packages?

In the second case, distributions can simply places the sources right next to the built packages.

As such, distributions have started mirroring filezilla / libfilezilla tarballs on their side

Don't distributions in general already have to preserve a copy of the source tarballs for every single package?

On the technical side, not keeping a copy of the sources prevents rebuilding a package if upstream is unavailable. Thus the continuous availability of each individual download site for the hundreds if not thousands of critical packages turns each of it into a single point of failure. To illustrate with a hypothetical example: Imagine an unavoidable ABI change in glibc [1] forcing a rebuild of just about everything. In this situation, how would you deal with for example x.org [2] no longer being available? Drop every package depending on X11?

Then there is also the legal aspects with software distributed under the GPL and similar licenses: How can one distribute packages of GPL software, without being able to make the sources available, should upstream become unavailable?

[1] time_t on 32bit systems comes to mind.
[2] Or upstream OpenSSL, GTK, Qt, libpng, zlib, .... Endless possibilities

[Robin Candau]

I feel like I wasn't clear enough about the actual issue, sorry about that. Let me try to re-phrase it:

We do indeed preserve a copy of source tarballs for our packages (at least for GPL and similar licenses, due to legal aspect). But before being preserved, those source tarballs have to be downloaded / fetched, obviously.

For numerous distributions, this is handled as part of their packaging workflow. Indeed, their build scripts fetches sources directly from the upstream repo / tarballs (and then preserve it next to the built package when releasing the said package to their package repositories).

However, the intentionally added unstable part in the filezilla's download url (​https://dl3.cdn.filezilla-project.org/client/FileZilla_3.68.1_x86_64-linux-gnu.tar.xz ?h=pdb4gbqtw08yYaxFABrHYg&x=1732651086) now makes it impossible for distributions to fetch sources from it in their build scripts (as the latter part is unstable and unpredictable).
As such, fetching the filezilla source now became an extra manual step that has to be performed beforehand, outside from usual distributions' packaging workflow (where this part is handled automatically).
For instance, as you can see from the links I provided in my ticket, OpenSuse, Alpine & Gentoo (among others) had to stop fetching the source tarball from Filezilla's website in their build script (due to the above) to rely on a local copy of the said source that they now have to manually download and upload on their side beforehand.

To sum up, the issue is not the source mirroring / preserving, but the initial downloading of said sources that became an extra manual tasks for distributions' package maintainers (due to the download link now being unstable, making it unusuable in build scripts).

I hope this make things clearer, sorry if my initial wording was confusing.

[Robin Candau]

I will probably start to manually download filezilla sources and upload them somewhere on our side for our filezilla packages to fetch from, like other distributions already did.

I still hope that we can discuss that further and that we are hopefully able to find a solution / compromise that would suit everyone though.
Again, as Filezilla highly recommends users to use distributions' package management system, it feels fair to facilitate distributions' packaging by not requiring such additional manual step for packagers.

Thanks for your consideration.
I remain available :)

[Andreas Rönnquist]

As the maintainer of filezilla in Debian I completely agree with Robin here - We have a mechanism in Debian that can automatically scan repositories for new upstream versions, and can simplify the work for a maintainer when packaging a new upstream version. As Robin mentioned, this has been working just fine, but it of course cannot be used any longer.

I too would very much appreciate if a solution to this would be found.