Skip to content

OverflowError: can't convert negative value to uint32_t #279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
eheien opened this issue Apr 10, 2018 · 3 comments
Closed

OverflowError: can't convert negative value to uint32_t #279

eheien opened this issue Apr 10, 2018 · 3 comments
Assignees
Labels

Comments

@eheien
Copy link

eheien commented Apr 10, 2018

  • asyncpg version: 0.15.0
  • PostgreSQL version: PostgreSQL 9.5.10 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16), 64-bit (AWS RDS version)
    PostgreSQL 9.5.2 on x86_64-pc-linux-gnu, compiled by gcc (Alpine 6.3.0) 6.3.0, 64-bit (local)
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    : Yes, AWS RDS. Local install of PostgreSQL does not have the problem.
  • Python version: 3.6.4
  • Platform: Red Hat 4.8.5-16
  • Do you use pgbouncer?: no
  • Did you install asyncpg with pip?: yes
  • If you built asyncpg locally, which version of Cython did you use?: n/a
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : did not try
db_asyncpg.py:97: in connection_init_func
                await connection.set_type_codec('geometry', schema='public',
                                                encoder=geom_encoder_ogr, decoder=geom_decoder_ogr,
                                                format='binary')
/usr/lib64/python3.6/site-packages/asyncpg/connection.py:914: in set_type_codec
    encoder, decoder, format)
asyncpg/protocol/settings.pyx:35: in asyncpg.protocol.protocol.ConnectionSettings.add_python_codec
    ???
asyncpg/protocol/settings.pyx:56: in asyncpg.protocol.protocol.ConnectionSettings.add_python_codec
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   OverflowError: can't convert negative value to uint32_t

asyncpg/protocol/codecs/base.pyx:541: OverflowError

We constantly create and tear down test DBs, and I believe the problem is that OIDs constantly increase. With the type introspection query on namespace public and type geometry I get the following in the local DB:
7124428 0 b
and the following in the DB that has the failure:
2573727724 0 b

You can see the second OID will overflow to negative if cast as an int. I'm not sure what the solution is, but I'm pretty sure this is the cause of the failure.

@elprans
Copy link
Member

elprans commented Apr 17, 2018

Good catch. This should be easy to fix. That said, once you hit the 4 billion OIDs mark, the OID generator will wrap around, and you'll start at 1 again, which may cause other issues.

@elprans elprans self-assigned this Apr 17, 2018
@elprans elprans added the bug label Apr 17, 2018
elprans added a commit that referenced this issue May 31, 2018
Currently asyncpg (incorrectly) assumes OIDs to be signed 32-bit
integers, whereas in reality they are unsigned.  As a result, things
would crash once the OID sequence reaches 2**31.

Fix this by decoding OID values as unsigned longs.

Fixes: #279
elprans added a commit that referenced this issue May 31, 2018
Currently asyncpg (incorrectly) assumes OIDs to be signed 32-bit
integers, whereas in reality they are unsigned.  As a result, things
would crash once the OID sequence reaches 2**31.

Fix this by decoding OID values as unsigned longs.

Fixes: #279
elprans added a commit that referenced this issue May 31, 2018
Currently asyncpg (incorrectly) assumes OIDs to be signed 32-bit
integers, whereas in reality they are unsigned.  As a result, things
would crash once the OID sequence reaches 2**31.

Fix this by decoding OID values as unsigned longs.

Fixes: #279
elprans added a commit that referenced this issue May 31, 2018
Currently asyncpg (incorrectly) assumes OIDs to be signed 32-bit
integers, whereas in reality they are unsigned.  As a result, things
would crash once the OID sequence reaches 2**31.

Fix this by decoding OID values as unsigned longs.

Fixes: #279
@eheien
Copy link
Author

eheien commented Jun 1, 2018

@elprans Thank you very much for the fix! Do you know roughly when this will be put in a release?

@elprans
Copy link
Member

elprans commented Jun 1, 2018

I'm hoping to do a release over the weekend or early next week.

elprans added a commit that referenced this issue Jun 13, 2018
This was partially fixed in #300 (as a fix for #279), however that fix
missed the fact that OID arrays were still handled using the signed
integer codec.

Fixes: #316
elprans added a commit that referenced this issue Jun 13, 2018
This was partially fixed in #300 (as a fix for #279), however that fix
missed the fact that OID arrays were still handled using the signed
integer codec.

Fixes: #316
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants