asyncio code cannot run simultaneously in multiple sub-interpreters.

[romanvm]

Bug report

asyncio code cannot run simultaneously in multiple sub-interpreters residing in their own threads. If you try to start an event loop by calling asyncio.run() only the first sub-interpreter will succeed and others will fail with the following error:

File "/usr/lib/python3.10/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop

A minimal example to reproduce the issue:

#include <iostream>
#include <thread>

#include "Python.h"


const char* pyCode = R"""(
import asyncio
import threading

async def coro():
    await asyncio.sleep(3)
    print(f'Sleep in thread {threading.get_ident()} done')

asyncio.run(coro())
)""";


void runPython() {
  PyThreadState* ts = PyThreadState_New(PyInterpreterState_Main());
  PyEval_RestoreThread(ts);

  PyThreadState* interpTs = Py_NewInterpreter();

  PyRun_SimpleString(pyCode);
  PyObject* error = PyErr_Occurred();
  if (error) {
   PyErr_PrintEx(0);
  }

  Py_EndInterpreter(interpTs);

  PyThreadState_Swap(ts);
  PyEval_ReleaseThread(ts);
}


int main() {
  Py_Initialize();
  Py_BEGIN_ALLOW_THREADS

  std::thread thread1(runPython);
  std::thread thread2(runPython);
  thread1.join();
  thread2.join();

  Py_END_ALLOW_THREADS
  Py_Finalize();

  std::cout << "All done" << std::endl;

  return 0;
}

If you try to do the same without sub-interpeters everything works fine:

#include <iostream>
#include <thread>

#include "Python.h"


const char* pyCode = R"""(
import asyncio
import threading

async def coro():
    await asyncio.sleep(3)
    print(f'Sleep in thread {threading.get_ident()} done')

asyncio.run(coro())
)""";


void runPython() {
  PyGILState_STATE gstate = PyGILState_Ensure();

  PyRun_SimpleString(pyCode);
  PyObject* error = PyErr_Occurred();
  if (error) {
   PyErr_PrintEx(0);
  }
  
  PyGILState_Release(gstate);
}


int main() {
  Py_Initialize();
  Py_BEGIN_ALLOW_THREADS

  std::thread thread1(runPython);
  std::thread thread2(runPython);
  thread1.join();
  thread2.join();

  Py_END_ALLOW_THREADS
  Py_Finalize();

  std::cout << "All done" << std::endl;

  return 0;
}

Background: in Kodi project we use sub-interpreters to run Python add-ons in isolated (-ish) environments. But this issue prevents from using asyncio in multiple addons at the same time.

A clear and concise description of what the bug is.
Include a minimal, reproducible example (https://stackoverflow.com/help/minimal-reproducible-example), if possible.

Your environment

• CPython versions tested on: Python 3.10
• Operating system and architecture: Linux Ubuntu x86-64.

[AlexWaygood]

I believe this may be a duplicate of #91375 (@kumaraditya303, can you confirm? :)

[kumaraditya303]

Yeah, it is a duplicate of #91375. The workaround I posted in that issue should be applicable here.

[AlexWaygood]

Duplicate of #91375