Fails to build on pypy 7.0 (stdlib 3.5) as pgenheaders.h doesn't exist in pypy

[Russell-Jones-OxPhys]

I'm not sure if this should be filed against typed_ast or pypy, as I can't find many references to pgenheaders.h and no documentation for it. Is this perhaps a situation similar to https://bitbucket.org/pypy/pypy/issues/2687/pypy3-is-missing-pystrhexh-breaking-pysha3 where it's a header file for an internal API?

(pypy3ve) user@host:/home/user/pypy3ve$ pypy3-c --version
Python 3.5.3 (1f86f25937b6ae6c8b25236c35228fac587678bf, Feb 12 2019, 13:00:04)
[PyPy 7.1.0-alpha0 with GCC 7.3.0]
(pypy3ve) user@host:/home/user/pypy3ve$ pypy3-c -m pip install --upgrade typed_ast
Collecting typed_ast
  Using cached https://files.pythonhosted.org/packages/fc/c6/61d6410fc70fda073bd1810f9b7f7022f00146b108f278a0c00041bfe5b0/typed-ast-1.3.1.tar.gz
Building wheels for collected packages: typed-ast
  Building wheel for typed-ast (setup.py) ... error
  Complete output from command /home/user/pypy3ve/bin/pypy3-c -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-usaqh1q6/typed-ast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-3s26ar_9 --python-tag pp371:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.5
  creating build/lib.linux-x86_64-3.5/typed_ast
  copying typed_ast/conversions.py -> build/lib.linux-x86_64-3.5/typed_ast
  copying typed_ast/ast27.py -> build/lib.linux-x86_64-3.5/typed_ast
  copying typed_ast/__init__.py -> build/lib.linux-x86_64-3.5/typed_ast
  copying typed_ast/ast3.py -> build/lib.linux-x86_64-3.5/typed_ast
  running build_ext
  building '_ast27' extension
  creating build/temp.linux-x86_64-3.5
  creating build/temp.linux-x86_64-3.5/ast27
  creating build/temp.linux-x86_64-3.5/ast27/Parser
  creating build/temp.linux-x86_64-3.5/ast27/Python
  creating build/temp.linux-x86_64-3.5/ast27/Custom
  gcc -pthread -DNDEBUG -O2 -fPIC -Iast27/Include -I/home/user/pypy3ve/include -I/home/user/pypy3/include -c ast27/Parser/acceler.c -o build/temp.linux-x86_64-3.5/ast27/Parser/acceler.o
  ast27/Parser/acceler.c:13:10: fatal error: pgenheaders.h: No such file or directory
   #include "pgenheaders.h"
            ^~~~~~~~~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for typed-ast
  Running setup.py clean for typed-ast
Failed to build typed-ast
Installing collected packages: typed-ast
  Running setup.py install for typed-ast ... error
    Complete output from command /home/user/pypy3ve/bin/pypy3-c -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-usaqh1q6/typed-ast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-drrzuuwb/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/pypy3ve/include/site/python3.5/typed-ast:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.5
    creating build/lib.linux-x86_64-3.5/typed_ast
    copying typed_ast/conversions.py -> build/lib.linux-x86_64-3.5/typed_ast
    copying typed_ast/ast27.py -> build/lib.linux-x86_64-3.5/typed_ast
    copying typed_ast/__init__.py -> build/lib.linux-x86_64-3.5/typed_ast
    copying typed_ast/ast3.py -> build/lib.linux-x86_64-3.5/typed_ast
    running build_ext
    building '_ast27' extension
    creating build/temp.linux-x86_64-3.5
    creating build/temp.linux-x86_64-3.5/ast27
    creating build/temp.linux-x86_64-3.5/ast27/Parser
    creating build/temp.linux-x86_64-3.5/ast27/Python
    creating build/temp.linux-x86_64-3.5/ast27/Custom
    gcc -pthread -DNDEBUG -O2 -fPIC -Iast27/Include -I/home/user/pypy3ve/include -I/home/user/pypy3/include -c ast27/Parser/acceler.c -o build/temp.linux-x86_64-3.5/ast27/Parser/acceler.o
    ast27/Parser/acceler.c:13:10: fatal error: pgenheaders.h: No such file or directory
     #include "pgenheaders.h"
              ^~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "/home/user/pypy3ve/bin/pypy3-c -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-usaqh1q6/typed-ast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-drrzuuwb/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/pypy3ve/include/site/python3.5/typed-ast" failed with error code 1 in /tmp/pip-install-usaqh1q6/typed-ast/

[msullivan]

It is an internal python header, though I think it wouldn't be hard for us to provide our own version of it.

[emmatyping]

I just checked and it seems we would need to remove includes of pydebug.h and codecs.h from tokenizer.c, which is pretty easy (replacing what they include is easy as well). But then the build fails with many undefined macros and functions from other includes. I'm not sure supporting PyPy would be that straightforward without a lot of work to fill in missing C extension APIs on their end. Here is the log FWIW: https://gist.github.com/ethanhs/f3483b44ba0d3b3fd420a272b8928d54

[scbedd]

I could be wrong, but I think is now occurring with the latest official python bits (this is just a standard clone -> config -> make)

Reading the What's New With Python 3.8, I noticed the following:

The header files and functions related to pgen have been removed after its replacement by a pure Python implementation. (Contributed by Pablo Galindo in bpo-36623.)

BPO-36623

When installing typed-ast leveraging my config-ed python 3.8 alpha, I see the following:

Building wheel for typed-ast (setup.py) ... error
  Complete output from command /mnt/c/projects/alpha-python/cpython/debug/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-x_j6kn2r/typed-ast/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-mnkzi3ca --python-tag cp38: 
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/typed_ast    
  copying typed_ast/__init__.py -> build/lib.linux-x86_64-3.8/typed_ast
  copying typed_ast/ast27.py -> build/lib.linux-x86_64-3.8/typed_ast
  copying typed_ast/ast3.py -> build/lib.linux-x86_64-3.8/typed_ast
  copying typed_ast/conversions.py -> build/lib.linux-x86_64-3.8/typed_ast
  running build_ext                                
  building '_ast27' extension                      
  creating build/temp.linux-x86_64-3.8             
  creating build/temp.linux-x86_64-3.8/ast27       
  creating build/temp.linux-x86_64-3.8/ast27/Parser
  creating build/temp.linux-x86_64-3.8/ast27/Python
  creating build/temp.linux-x86_64-3.8/ast27/Custom
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iast27/Include -I/mnt/c/projects/alpha-python/cpython/Include -I/mnt/c/projects/alpha-python/cpython/debug -c ast27/Parser/acceler.c -o build/temp.linux-x86_64-3.8/ast27/Parser/acceler.o
    ast27/Parser/acceler.c:13:10: fatal error: pgenheaders.h: No such file or directory
     #include "pgenheaders.h"
              ^~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1

Yes this is an output from WSL, but this does occur on standard Ubuntu LTS etc as well.

[gvanrossum]

Hmm. This will require some serious changes to typed-ast. While the 3.8 stdlib ast module can return type comments, it can’t parse Python 2, so we still need to support building ast27 with 3.8 dev headers.

[msullivan]

Wasn't that serious of a change fortunately.

We've released 1.3.5 which should work under 3.8-dev as it no longer requires pgenheaders.h.

I'm going to close this issue since the pgenheaders.h problem is fixed.

@Russell-Jones-OxPhys if it still fails to build under pypy in a way that seems like it might be on us, please file another issue!

[emmatyping]

Out of curiosity I worked a bit on trying to get this to build on pypy. It seems that there are several internal C API things we would need to copy over for PyPy support. I stopped for now, but so far I have found the following non-public C API items we use:

• PyArena
• _PyIdentifier (and associated _PyObject_CallMethodId and _PyObject_GetAttrId)
• pyctype.h contents

And public C APIs we use which aren't found:

• Various items in warnings.h such as PyErr_WarnExplicit which PyPy should implement.
• Py_UniversalNewlineFgets which PyPy also should implement.

There is likely more. For PyArena, we could probably just copy our own version into say, Include/pypy or something, and define PYPY through the setup script.

There are likely more things missing, but this is as far as I got when experimenting with this.

EDIT: so I guess this issue should be re-opened? I don't have the ability to do so as I am not a collaborator however.

[gvanrossum]

Yeah, it was never meant to run against PyPy. Once PyPy catches up with Python 3.8 and we stop caring about 2.7 it should be possible to run mypy under PyPy -- when run under Python 3.8 or later it uses the stdlib ast module (which has grown APIs to request type comments and to support earlier grammar versions (to some extent).

[stuaxo]

OK, it looks like pypy-nightly has started work on 3.8, I'm not sure how far along compatibility is yet
https://buildbot.pypy.org/nightly/py3.8/