need libffi-dev to build manylinux wheels

[tgarc]

thanks for working on this project! The windows build works for me right out of the box.

However, the manylinux builds don't work for my project as it requires the libffi-dev package. (simply: yum install libffi-dev). It's nice that this 'just works' but is there any room for customization in cases like this? BTW, I believe this is an issue with the osx build as well.

[joerick]

Thanks! Have you tried the CIBW_BEFORE_BUILD_LINUX environment variable? You could set that to yum install libffi-dev and it should do what you need. CIBW_BEFORE_BUILD_MACOS also exists. They are both just shell commands that run before the wheel is built.

[tgarc]

Oh right, I had initially dismissed that since it tries to run the command for *each* python build environment. I only need it to run for each container environment. In this case though it seems perfectly harmless to run yum install multiple times. However, when I tried this, the travis-ci log seems to indicate that everything is running smoothly, but then exits with a non-zero code! There's no indication of any error. Would you mind looking at this log to see if I missed something? https://travis-ci.org/tgarc/pastream/jobs/234053700

…

On Fri, May 19, 2017 at 5:08 AM, Joe Rickerby ***@***.***> wrote: Thanks! Have you tried the CIBW_BEFORE_BUILD_LINUX environment variable? You could set that to yum install libffi-dev and it should do what you need. CIBW_BEFORE_BUILD_MACOS also exists. They are both just shell commands that run before the wheel is built. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAaVHPpBKGyRyMFyVhLfeDoYil2kUZfuks5r7WoIgaJpZM4NgEvt> .

[tgarc]

Also, it looks like maybe your calling CIBW_BEFORE_BUILD using python.subprocess - which means that running multiple commands (e.g. cmd1 && cmd2) is not possible.

[tgarc]

However, when I tried this, the travis-ci log seems to indicate that everything is running smoothly, but then exits with a non-zero code! There's no indication of any error. Would you mind looking at this log to see if I missed something?

NVM, I figured out that this was because I didn't pass '-y' with the yum install command.

[joerick]

Great! Glad you got it working. Yes, I think it should be running those commands in a shell rather than directly. I'll raise another issue for that.

[tgarc]

Related question: is it possible to inject environment variables into a particular platform? I guess this request is only relevant for linux, since for other platforms you can just set up the environment before running cibuildwheel. The issue is that for certain builds, the libffi shared object can't be found. Python 2.6 succeeds but then python 2.7 fails. The confounding thing is that using the script from https://github.com/pypa/python-manylinux-demo works without a hitch once I added the yum install libffi-devel line to the top of it. It seems like your code for the linux build is nearly identical to that demo so I can't figure out why my build is failing with cibuildwheel :\

[joerick]

So you need some specific environment variables within the build environment? Could I ask what those are and what commands they'll affect in your case? For example, is it pip or yum that requires them?

[tgarc]

tl;dr I need to set PKG_CONFIG_PATH for the linux environment

Sure. First off, my package uses CFFI to wrap some C code, which seems to be what's causing all the build headaches. CFFI depends on libffi being installed and its pkg-config needs to be findable when doing pip install cffi.

Now, in macos it seems that it's required to set PKG_CONFIG_PATH before calling pip install. In Linux though, at least using the manylinux-demo I mentioned above, all I needed was to yum install libffi; I didn't have to set PKG_CONFIG_PATH.

For some reason, the linux cibuildwheel environment seems a little different because it is not able to find libffi. Hence, I wanted to see if manually setting PKG_CONFIG_PATH would fix this. However, because the linux environment is a docker image, I have no way to set up environment variables with cibuildwheel.

[joerick]

Okay, got it... No two wheels are the same! I am wondering why the pip install cffi won't install from wheel since there are quite a few wheels available for that package. But maybe a CIBW_ENVIRONMENT option is warranted also.

[tgarc]

There is some details on the CFFI webpage that I mentioned but it actually works out of the box with Windows (I think they packaged libffi into cffi for windows) so in that case I believe you can directly install it from the wheel. But not for linux or macos.

[tgarc]

I found the issue! Well, sort of. I found the crucial difference between your linux build script and the manylinux-demo one: before the pip wheel command (line54) there is a pip install command:

"$PYBIN/pip" install .

Adding this line to your build script fixed my no package 'libffi' found problems.

Given that this edit shouldn't have any negative side effects, and should fix issues for any linux users that depend on cffi, want me to go ahead and create a pull request?

Edit

I may have spoken slightly too soon: I still have issues building on travisci. However, running cibuildwheel on my own linux box now builds without issue.

[joerick]

Oh, interesting. Yes that sounds like it could be a good idea - I suppose it allows dependencies to install before trying to build the wheel. I'd have thought that pip wheel would do that also, but if it's fixing things for you, I can't see any downsides. Yes, please send a PR if you have time!

[joerick]

This was resolved in #9.