ValueError exceptions in hooks are not properly handled

[fgblomqvist]

This was discussed in #32.

I got an exception in my before_all hook that just lead to this output:

info: Configuration './dredd.yml' found, ignoring other arguments.
info: Starting backend server process with command: app run
info: Waiting 3 seconds for backend server process to start
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 317-228-483
info: Beginning Dredd testing...
info: Found Hookfiles: 0=/my/dir/hooks.py
info: Spawning 'python' hooks handler process.
info: Hooks handler stdout: Starting Dredd Python hooks handler

warn: Error connecting to the hooks handler process. Is the handler running? Retrying.
info: Successfully connected to hooks handler. Waiting 0.1s to start testing.
warn: Hook handling timed out.
warn: Hook handling timed out.
warn: Hook handling timed out.
fail: GET (200) / duration: 5004ms
warn: Hook handling timed out.

I figured out the line that was causing it and caught the exception myself:

    try:
        // code that throw exception
    except Exception as e:
        print(e)

and got this:

info: Configuration './dredd.yml' found, ignoring other arguments.
info: Starting backend server process with command: app run
info: Waiting 3 seconds for backend server process to start
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 317-228-483
info: Beginning Dredd testing...
info: Found Hookfiles: 0=/my/dir/hooks.py
info: Spawning 'python' hooks handler process.
info: Hooks handler stdout: Starting Dredd Python hooks handler

warn: Error connecting to the hooks handler process. Is the handler running? Retrying.
info: Successfully connected to hooks handler. Waiting 0.1s to start testing.
info: Hooks handler stdout: Can't reset a sequence on decendant factory <class 'tests.fixtures.factories.student.StudentFactory'>; reset sequence on <class 'tests.fixtures.factories.user.UserFactory'> or use `force=True`.

127.0.0.1 - - [20/Mar/2018 11:41:46] "GET / HTTP/1.1" 200 -
pass: GET (200) / duration: 123ms

Naturally, since I caught the exception, dredd continues to run.
But yeah, I would expect dredd_hooks to catch it for me (and stop running)?

[w-vi]

Thanks for pointing this out, would you have an example code which caused it? My test didn't fail so I am not sure what exactly is happening.

[fgblomqvist]

Just figured out more precisely what is wrong:
https://github.com/apiaryio/dredd-hooks-python/blob/master/dredd_hooks/dredd.py#L112-113

If a ValueError is raised inside a hook (which was the case above) no traceback is printed (and the exception is not passed on). Not sure why ValueErrors are treated differently, doesn't seem right. I was able to replicate it myself by simply doing this:

@hooks.before_all
def before_all(trans):
    raise ValueError()

It resulted in the same faulty behavior as above. If I instead raise an Exception, dredd_hooks prints some stuff that gets sent to dredd and is then showed to me.