Automating packaging and uploading of binary wheels

There has been plenty of frustration surrounding Python libraries which contain C extensions. Users are frustrated because pre-built wheels are not available for their platform, requiring them to install a compatible compiler. This has been particularly painful for Windows users. On the developer side, projects that have gone through the trouble of providing binary wheels often have a rather complicated process for building and uploading them, requiring several manual steps to get it done. On the other hand, projects that choose not to build binary wheels sometimes have droves of users nagging them to build those wheels (yes, I've been guilty of that). All in all, nobody's happy.

What makes this situation particularly challenging is that there is no single CI service available for F/OSS projects which could build wheels for all three major platforms (Linux, macOS and Windows). Travis supports Linux and macOS but not Windows. Then there's AppVeyor which only supports Windows. So in order to make everyone happy, you would need to combine the powers of both continuous integration services.

An additional challenge has been the lack of coordination tools for ensuring that a new release is uploaded to PyPI only if all the build jobs succeed. Naive configurations (yes, I've been guilty of that too) build the wheels and upload them to PyPI independently. This can lead to situations where a build job legitimately fails in a way that should've been a release blocker, but the jobs that succeeded have already uploaded their artifacts to PyPI. Now you have a botched release in your hands.

What if I told you that there is a way to set up your project so that all you have to do is add a git tag and push to Github, and the wheels and the source distribution would automatically get built and uploaded to PyPI if (and only if) all goes well?

Yes, folks. It can be done. You just need an adventurous mind. Take the red pill and find out how deep the rabbit hole goes.

How it works

As I hinted earlier, the recipe I'm about to present combines three important factors:

1. Use of Travis's "Build Stages" feature
2. Use of AppVeyor via its ReST API
3. Use of an external storage service (Amazon S3 is used here, but it could be something else)

The gist is this: The Travis build first runs the tests against all supported Python versions. After the tests have finished successfully, Travis starts building wheels for Linux and macOS. Meanwhile, an additional job is started which sends a request to AppVeyor's ReST API which tells it to start a build against the current git changeset. It will then poll the status of the build on regular intervals until it finishes one way or another. If it fails, the Travis build is failed. If it succeeds, the build artifacts are downloaded to the container running the Travis build.

When all the wheels have been built, their respective build jobs will upload them to the shared storage. Then the final job is started which pulls all the artifacts from this storage and uploads them to PyPI.

Setting it up

Tl;dr: Go see the example project and adapt its configuration to your needs.

You will need to have the following set up before proceeding:

• A project on Github
• A PyPI account
• An AppVeyor account
• An AWS account (if S3 is used)

The use of Amazon's S3 could be replaced with any other storage service. However, the free tier on AWS should get you enough disk space to satisfy the needs of most projects. You do need to have a valid credit card, however.

You will need at least these two configuration files present in the project's root directory:

• .travis.yml: build configuration for Travis (this is the most important part)
• appveyor.yml: build configuration for AppVeyor

You can copy the linked files to your own project as a base. Just remember to replace the environment variables in .travis.yml (or remove them altogether, as explained below).

Travis setup

If your project does not yet have Travis integration enabled, you need to do the following:

1. Go to your project's settings on Github
2. Click on "Integrations and services"
3. Click on "Add service"
4. Choose "Travis CI" and enter your Github password when prompted to do so
5. Go to your Travis profile settings on their site
6. Click on "Sync account" (at the top right) to refresh the list of projects
7. Find your project on the list after the sync is complete and turn the switch on
8. Click on the cogwheel next to your project's name to enter the settings page

The following Travis project settings are recommended:

Next, you will need to define the following environment variables:

• APPVEYOR_SLUG (your project name on AppVeyor)
• APPVEYOR_ACCOUNT (your account name on AppVeyor)
• TWINE_USERNAME (your PyPI user name)
• TWINE_PASSWORD (your PyPI password)
• AWS_ACCESS_KEY_ID (the access key ID from AWS, for shared storage)
• AWS_SECRET_ACCESS_KEY (the secret key from AWS, for shared storage)

There are two ways you can provide your build jobs environment variables:

1. Add them to your .travis.yml file, encrypting any confidential ones like passwords
2. Add them on your Travis project settings page

To encrypt an environment variable, you will need to have Travis's command line client installed. Then, you can do something like this:

echo -n TWINE_PASSWORD=foobarbaz | travis encrypt

This will output an encrypted secret which you can paste into .travis.yml. Note the importance of the -n switch, as without that a newline character would be added to the end which would cause the wrong text to be encrypted.

AppVeyor setup

Assuming you have your AppVeyor account set up, you need to add your project to it. First, go to the Projects section in the top level menu and click "New Project". Then select Github and pick the project from that list.

Next, you need to disable the web hook AppVeyor just added to your Github project. This is necessary because the AppVeyor build should only be triggered by the wheel build stage on Travis. On Github, go to Settings -> Webhooks and edit the AppVeyor hook. Uncheck the "Active" check box and press "Update webhook", as shown below:

That's it for the AppVeyor configuration, unless your project has some special requirements.

Afterthoughts

I hope this will lower the barrier for projects to start producing binary wheels.

It should be noted that some projects prefer running their test suites against the wheels separately on each platform, but this is left as an exercise for the reader to implement.

Hopefully Travis will some day sort out their lack of Windows support.

The example configuration could be simplified somewhat once pip starts supporting pyproject.toml (included in the sample project). That should enable the removal of the "pip install wheel Cython" lines all over the configuration.