Attempt to free memory that is still in use by an ongoing MPI communication

[artpol84]

We observed this error only once in our MTT. We are running v2.x with SLURM/pmix there. It is possible that it is somehow related to this configuration, though I doubt that.
Here is the error message:

export OMPI_MCA_btl_openib_if_include=mlx4_0:1
OMPI_MCA_btl_openib_if_include=mlx4_0:1
OMPI_MCA_mpi_add_procs_cutoff=0+ OMPI_MCA_pmix_base_async_modex=1
OMPI_MCA_pmix_base_collect_data=0
/tmp/mtt_116453_slurm/bin/srun -N 8 -n 64 --mpi=pmix_v1 -p pmellanox <mtt-base>/installs/T8JL/tests/mpich_tests/mpich-mellanox.git/test/mpi/coll/allgather2
[boo13:10605] Attempt to free memory that is still in use by an ongoing MPI communication (buffer 0xa89000, size 9302016).  MPI job will now abort.
srun: error: boo13: task 9: Exited with exit code 1
srun: Terminating job step 1592.0

[artpol84]

Update: this has triggered today again.
First time I noticed that was on Thursday, 03/30/2017.

[artpol84]

I always observe this with allgather2 test (but it's not 100% reproducible), so in conjunction with the time it first appeared and the commit history, I can say that this may be related to #3159

[artpol84]

@vspetrov

[hppritcha]

These MTT runs were done with vanilla Open MPI, no UCX, MXM, etc.

[gpaulsen]

From web-ex 04/11 - Not 100% reproducible, so lets keep this issue open a bit longer.

[artpol84]

still reproducible:
recent mtt report

[jsquyres]

Artem -- don't forget that you can click on the "Absolute date range" link on the right in MTT and get a short link: https://mtt.open-mpi.org/index.php?do_redir=2412

[artpol84]

Thanks, I didn't know that: https://mtt.open-mpi.org/index.php?do_redir=2412

[jsquyres]

@artpol84 Where are we on this issue?

[artpol84]

@jsquyres I don't see this error anymore in our MTT.

[jsquyres]

Sweet! I'll close.

[lcebaman]

I am getting exactly this error with OpenMPI 3.0.0 and using RMA (multithreaded) in my code. This is reproducible and it does not happen with MPICH. Any ideas?

[jsquyres]

@icebaman Can you open a new issue with a small reproducer code?

[chchang6]

Hi,
We have a user who is running into this at the moment on our system, with OpenMPI 1.10.7 built against GCC 7.3.0. (Version 1 is because we are running into startup/hanging issues with newer versions at larger scales). Was this thread ever continued?

[jsquyres]

I'm afraid not. Have you tried Open MPI v4.x?

[chchang6]

Hi Jeff, no but I also got that advice on another ticket for a different issue, so will try that shortly. Thanks!

[jsquyres]

Yeah, sorry, 1.10.x is ancient and not really supported any more.

[chchang6]

No worries; the only reason folks are running 1.10.X is that 2 and 3 are having scalability problems in our environment. Suspect it has something to do with startup mechanism, Slurm config, and our network, but hope 4 works out of the box.