Fix TCP BTL THREAD_MULTIPLE deadlock

[jsquyres]

Discussed at Feb 2016 Dallas meeting. @bosilca says he knows what the problem is and will fix it.

[jsquyres]

@hjelmn @bosilca Did this get fixed yet, perchance?

[jsquyres]

@bosilca Says that he has large modifications to the TCP BTL (it's a giant change). Not clear yet on how we want to handle this...

[hppritcha]

@bosilca is there a chance we can have a narrowly focused fix to the existing TCP BTL?

[jsquyres]

@bosilca If we can't get a focused fix, how about this:

• Have the old TCP BTL code selected on non-THREAD_MULTIPLE
• Have your new TCP BTL code selected on THREAD_MULTIPLE

This would allow us to have somewhat newer/less-tested code in the THREAD_MULTIPLE path (and therefore make solid advances in the THREAD_MULTIPLE category, modulo bug fixes), but have solid/reliable/time-tested code in the non-THREAD_MULTIPLE case (i.e., the majority of the world).

As the new code gets more testing and we get higher confidence in it, we can remove this division / remove the old code, and just have the new code.

What do you think?

[jsquyres]

@bosilca Any comment?

[bosilca]

#1491

[jsquyres]

Fixed via #1491.