Default max_buffer_size causes silent failures when streaming

[hauntsaninja]

Thanks for a useful library!

Consider the following repro:

import os
import msgpack
import tempfile

original_values = [
    (b"x" * 1024) * (1024 * 40 * i)
    for i in range(1, 5)
]

with tempfile.TemporaryDirectory() as tmpdir:
    fn = os.path.join(tmpdir, "test.msgpack")
    with open(fn, "wb") as f:
        for v in original_values:
            msgpack.pack(v, f)
    
    with open(fn, "rb") as f:
        unpack = msgpack.Unpacker(f, max_buffer_size=0)
        values = list(unpack)
    assert values == original_values

    with open(fn, "rb") as f:
        unpack = msgpack.Unpacker(f)
        values = list(unpack)
    # failed to unpack some of the stream, but we wouldn't know if we didn't have original_values to check against
    assert values == original_values

[hauntsaninja]

This was changed in #319 to avoid DoS. That's fine, but ideally the above code would fail loudly, rather than silently.

Maybe I'm using the library wrong, but iterating over a msgpack.Unpacker is recommended in the README without any mention of this: https://github.com/msgpack/msgpack-python#streaming-unpacking

[methane]

Thank you for reporting. I agree that Unpacker should raise an error when a message is larger than max_buffer_size.

[hauntsaninja]

Thanks! I opened a PR with a potential fix, although not sure it's correct #504 :-)