Add a framework and components to expose shared memory single copy to all components #9154
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hjelmn
force-pushed
the
add_a_component_to_expose_shared_memory_single_copy_to_all_components
branch
2 times, most recently
from
add6656 to
2124629
Compare
bwbarrett
previously requested changes
Jul 24, 2021
There was a problem hiding this comment.
Overall, looks like a massive improvement. A couple of nits included.
One larger question... In the unlikely scenario that two processes come to different smsc selection answers, it looks like we're going to handle that poorly. Maybe I'm reading the code wrong, but only one of the get_endpoint() calls seems to check the other side is actually using the same smsc. Am I reading that wrong?
|
We discussed this in the Web-ex Today. Consensus was that it'd be good to get this into v5.0 as there are other optimizations and code-cleanup that Nathan's working on that this will enable. |
hjelmn
force-pushed
the
add_a_component_to_expose_shared_memory_single_copy_to_all_components
branch
2 times, most recently
from
4430b6e to
6943e24
Compare
…pport This commit introduces opal/mca/smsc which exposes support for shared-memory single copy mechanisms. The target is to support Linux CMA, XPMEM, and KNEM. This component will initially support btl/sm but will also be used to provide single copy for other components (coll/sm, osc/rdma). Signed-off-by: Nathan Hjelm <hjelmn@google.com>
This commit adds a component to the shared-memory single-copy framework to support Linux Cross Memory Attach (CMA). This component supports copy_to and copy_from without registration. Signed-off-by: Nathan Hjelm <hjelmn@google.com>
This commit adds a knem component to the shared-memory single-copy framework. This modules supports copy_to and copy_from with registration required on the remote peer. Signed-off-by: Nathan Hjelm <hjelmn@google.com>
This commit adds a new shared-memory single-copy component supporting Cray/SGI XPMEM. This component supports copy_to, copy_from, and memory mapping. Signed-off-by: Nathan Hjelm <hjelmn@google.com>
This commit removes the dedicated shared-memory single-copy support from btl/sm. This support is now provided by the shared-memory single-copy (smsc) framework. The btl_sm_single_copy_mechanism MCA variable has been removed. Use either the component priority parameters (ex. scsm_xpmem_priority) or component selection (--mca smsc) to select the single-copy mechanism. Signed-off-by: Nathan Hjelm <hjelmn@google.com>
jsquyres
added a commit
to jsquyres/ompi
that referenced
this pull request
Aug 1, 2021
since this will likely be merged for v5.0.0 Signed-off-by: Jeff Squyres <jsquyres@cisco.com>
jsquyres
requested changes
Aug 1, 2021
There was a problem hiding this comment.
I get this warning when compiling:
2032 make[2]: Entering directory `/home/jsquyres/git/ompi/opal/mca/smsc'
2033 CC base/smsc_base_frame.lo
2034 In file included from ../../../opal/mca/smsc/base/base.h:14,
2035 from base/smsc_base_frame.c:27:
2036 ../../../opal/mca/smsc/smsc.h:216:5: warning: "MCA_opal_smsc_DIRECT_CALL" is not defined, evaluates to 0 [-Wundef]
2037 #if MCA_opal_smsc_DIRECT_CALL
2038 ^~~~~~~~~~~~~~~~~~~~~~~~~
(repeats for all the BTL SM + SMSC component source files, too)
Feels like that is significant...
jsquyres
dismissed
bwbarrett’s stale review
Brian's away on leave. @jsquyres will take over the review of this PR.
hjelmn
force-pushed
the
add_a_component_to_expose_shared_memory_single_copy_to_all_components
branch
from
6943e24 to
95eb83f
Compare
|
@jsquyres Fixed. File missing from one of the commits. Please re-test. |
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR moves the shared-memory single-copy code out of btl/sm and puts it in its own framework. This will allow other components to make use of this functionality without relying on btl/sm.