Supporting variants in the Ports framework

Supporting variants in the Ports framework Brendan Molloy brendan+freebsd@bbqsrc.net Poudriere PoC with variants Ports makefile PoC with examples

I recently became involved with FreeBSD (as in, the last 2-3 weeks), and found myself quickly involved with Ports development. What quickly struck me was the difficulty in providing a Python package that was depended upon by multiple versions of Python. As it turns out, poudriere can currently only generate one package per port, meaning that a Python version-neutral (compatible with 2.x and 3.x) port cannot simultaneously be packaged for each variant at the same time.

I discussed the issue with koobs@, who suggested I look into implementing a 'variants protocol' within the Ports framework and necessary changes to poudriere in order to allow a port to generate more than one package.

Support for variants is strongly needed in Ports and provides significant benefits.

It would allow Python and other languages to provide packages for dependencies for multiple language versions from the same port
It alleviates the need for so-called 'slave ports', as a single port could now have multiple generated packages from a single port
It would have a very small impact on the greater Ports ecosystem: adding only two new variables, VARIANT and VARIANTS.
It would provide a more consistent approach between different packaging teams for handling variations.

For a simple example, editors/vim-lite could be folded into the editors/vim port, while still generating a vim and vim-lite package. For Python, VARIANTS can be derived from the already used USES flags and generate compatible packages. py27-foobar and py34-foobar could now be consistently generated by poudriere without issue.

Fortunately, this is not a wishful thinking piece. I dug in my heels and have implemented a proof-of-concept implementation of variants in the Ports framework, including necessary modifications to poudriere in order to support it. It was mildly upsetting to find that poudriere is mostly written in Bourne shell scripts, but press on I did nonetheless.

I started with the prototype made by bapt@ as the base, and built from there. The poudriere PoC limits changes as much as possible to merely adding support for the new variants flags, while also at the request of koobs@ making the logging output more package-centric as opposed to port-centric as a result of these changes.

This is a work in progress, and I would love to hea your feedback. I've enjoyed my first two weeks working on FreeBSD, and I hope to stay here for quite some time.

Any constructive feedback on the implementation would be very welcome! Hopefully the code will be of sufficient quality to be considered for formal review in the coming months New tools to enhance the porting experience Brendan Molloy brendan+freebsd@bbqsrc.net pytoport: Generate FreeBSD Ports from Python modules on PyPI bandar: Create development overlays for the Ports tree skog: Generate visual dependency trees for FreeBSD ports spdx-lookup: SPDX license list query tool

When I starting working on ports for FreeBSD in the last couple of weeks, I found that my workflow wasn't as efficient as it could be with the available tools, so I made a few that could be useful to the development community at large. All of these have been added to the Ports tree, or otherwise will soon have one, so you can play with them today!

pytoport is a command line application that generates a skeleton port for a given PyPI package name. It attempts to generate the correct dependencies, it makes a good attempt at guessing the license using spdx-lookup, and generates a pkg-descr. This made generating the fifteen or so ports I was working on a complete breeze.

While doing this however, I noticed that some ports were bringing in dependencies that I didn't expect, and I needed some way to visualise this. skog builds a dependency tree from the depends lists output by the Ports framework, and displays it on the command line (with extra shiny output if you are using UTF-8). No more pesky example and documentation dependencies being dragged in when you clearly toggled that OPTION as far off as it would go.

While doing all of this, I found it cumbersome to be copying ports back and forward between my small development tree living in git and the larger upstream SVN tree I was using in poudriere. I built a tool called bandar that takes advantage of the FUSE version of unionfs to easily overlay my dev tree on the upstream tree, run linting, poudriere and generate archives with ease.

I'm very impressed with how easy it was to build more tooling for FreeBSD. I hope some of these tools will be of some use to you, and as always, I'd love to hear your feedback!

Improve skog to support searching a tree for a certain port Get the bandar port completed Continue to improve pytoport, adding trove support and better depedency handling Patches welcome for all of the above!