Device op cuda #46

devreal · 2023-11-06T00:20:53Z

Trial PR for feedback from @bosilca. Probably needs some more cleanup but feedback on the design is appreciated. Commits will be squashed later.

Currently only implements offloading for allreduce algorithms. It's missing rooted reduce.

related to spack/spack#40725 Signed-off-by: Howard Pritchard <[email protected]>

github-actions · 2023-11-06T00:23:24Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-11-06T00:24:01Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-11-06T00:40:04Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

…0_dlopen spack:fix for dlopen missing symbol problem

github-actions · 2023-11-07T19:35:34Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-11-07T22:48:30Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-11-07T23:06:51Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

ef8b526: ROCM: add missing FUNC_FUNC_FN macro

check_signed_off: does not contain a valid Signed-off-by line

6fd216f: accelerator/rocm: regular memory behaves like unif...

check_signed_off: does not contain a valid Signed-off-by line

955849b: Device op: pass device to lower-level op to avoid ...

check_signed_off: does not contain a valid Signed-off-by line

3afec6b: Draft of ompi_op_select_device

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

Signed-off-by: Joseph Schuchart <[email protected]>

If the target process is unable to execute an RDMA operation it instructs the origin to change the communication protocol. When this happen theorigin must be informed to cancel all pending RDMA operations, and release the rdma_frag. Signed-off-by: George Bosilca <[email protected]>

…or allreduce recursive doubling Signed-off-by: Joseph Schuchart <[email protected]>

Signed-off-by: Joseph Schuchart <[email protected]>

The accelerator component may report the availability of a single accelerator whose ID is not zero. Signed-off-by: Joseph Schuchart <[email protected]>

Signed-off-by: Joseph Schuchart <[email protected]>

…_SUPPORT These macros are defined to either 1 or 0 Signed-off-by: Joseph Schuchart <[email protected]>

Signed-off-by: Joseph Schuchart <[email protected]>

…evices Signed-off-by: Joseph Schuchart <[email protected]>

Signed-off-by: Joseph Schuchart <[email protected]>

We know where source and target buffers are located, so pass the right transfer direction to the accelerator memcpy call. Signed-off-by: Joseph Schuchart <[email protected]>

Signed-off-by: Joseph Schuchart <[email protected]>

spack:fix for dlopen missing symbol problem

50731f0

related to spack/spack#40725 Signed-off-by: Howard Pritchard <[email protected]>

devreal requested a review from bosilca November 6, 2023 00:39

Merge pull request open-mpi#12050 from hppritcha/fix_for_slack_ompi50…

517dcba

…0_dlopen spack:fix for dlopen missing symbol problem

Joseph Schuchart and others added 21 commits November 7, 2023 18:09

Initial draft of CUDA device support for ops

35ff1da

Signed-off-by: Joseph Schuchart <[email protected]>

First working version of CUDA op support

b7e6f89

Signed-off-by: Joseph Schuchart <[email protected]>

Update copyright header

164388a

Signed-off-by: Joseph Schuchart <[email protected]>

Fix minor bugs to get osu_allreduce working

d8110ac

Signed-off-by: Joseph Schuchart <[email protected]>

cuMemAllocAsync is supported since CUDA 11.2.0

f609127

Signed-off-by: Joseph Schuchart <[email protected]>

coll/base/allreduce: Condition device allocation on op/dtype support

8ae3dac

Signed-off-by: Joseph Schuchart <[email protected]>

Make sure the device op callbacks are zero-initialized

655948f

Signed-off-by: Joseph Schuchart <[email protected]>

Be more graceful when creating a context and stream

7cdc828

Signed-off-by: Joseph Schuchart <[email protected]>

fix wrong call to memset

bdb16a1

Signed-off-by: Joseph Schuchart <[email protected]>

Add detector for cudart

5934f43

Signed-off-by: Joseph Schuchart <[email protected]>

Add CUDA stream-based allocator and memory pools

c2c3d0e

Signed-off-by: Joseph Schuchart <[email protected]>

Don't memset the CUDA op component, we need the version

5df449c

Signed-off-by: Joseph Schuchart <[email protected]>

Set the memory pool release threshold

812d068

Signed-off-by: Joseph Schuchart <[email protected]>

Implement device-compatible allocator to cache coll temporaries

a688c84

Signed-off-by: Joseph Schuchart <[email protected]>

Fix devicebucket allocator for larger sizes

bbd362d

Signed-off-by: Joseph Schuchart <[email protected]>

Stream-based reduction and ddt copy and 3buff cuda kernels, adopted f…

f2f0f2d

…or allreduce recursive doubling Signed-off-by: Joseph Schuchart <[email protected]>

Remove extra copies from allreduce redscat and ring

8f5b503

Signed-off-by: Joseph Schuchart <[email protected]>

Allow ops and memcpy on managed memory from the host

1c68d17

Signed-off-by: Joseph Schuchart <[email protected]>

reduce_local: add support for device memory

70dde0f

Signed-off-by: Joseph Schuchart <[email protected]>

Draft of ompi_op_select_device

e603bcc

Signed-off-by: Joseph Schuchart <[email protected]>

devreal added 28 commits November 7, 2023 18:48

Device allocators: correctly handle non-zero ID single accelerator

cfe8a5a

The accelerator component may report the availability of a single accelerator whose ID is not zero. Signed-off-by: Joseph Schuchart <[email protected]>

CUDA op: consistently name unsigned_long functions as ulong

3bc7676

Signed-off-by: Joseph Schuchart <[email protected]>

ROCM op: remove debug output

9c1da7e

Signed-off-by: Joseph Schuchart <[email protected]>

Reduce_local test: correctly test for OPAL_CUDA_SUPPORT and OPAL_ROCM…

a20f671

…_SUPPORT These macros are defined to either 1 or 0 Signed-off-by: Joseph Schuchart <[email protected]>

More unsigned_long -> ulong fixes in CUDA and ROCm op

97338db

Signed-off-by: Joseph Schuchart <[email protected]>

Fix type in ulong conversion

541b8a0

Signed-off-by: Joseph Schuchart <[email protected]>

Reduce_local: access only host-side memory in error message

8cb2feb

Signed-off-by: Joseph Schuchart <[email protected]>

Make sure CUDA accelerator is initialized before querying number of d…

2996ba0

…evices Signed-off-by: Joseph Schuchart <[email protected]>

Accelerator: provide peak bandwidth estimate

246003f

Signed-off-by: Joseph Schuchart <[email protected]>

accelerator/rocm: regular memory behaves like unified memory

6601484

Signed-off-by: Joseph Schuchart <[email protected]>

ROCM: add missing FUNC_FUNC_FN macro

d0fe9a2

Signed-off-by: Joseph Schuchart <[email protected]>

opal_datatype_accelerator_memcpy: determine device copy type

63b64a0

We know where source and target buffers are located, so pass the right transfer direction to the accelerator memcpy call. Signed-off-by: Joseph Schuchart <[email protected]>

accelerator rocm: fix global memcpy stream variable

5a29e13

Signed-off-by: Joseph Schuchart <[email protected]>

Thread base: fix missing include file

5c7c7a1

Signed-off-by: Joseph Schuchart <[email protected]>

Accelerator: Remove debug output

76f00c4

Signed-off-by: Joseph Schuchart <[email protected]>

Allreduce: don't copy inputs if data can be accessed from the host

56bcfee

Signed-off-by: Joseph Schuchart <[email protected]>

Be more careful when releasing temporary receive buffers

a1f089e

Signed-off-by: Joseph Schuchart <[email protected]>

Remove debug output and dead code

33616e6

Signed-off-by: Joseph Schuchart <[email protected]>

Bump max devicebucket allocator max size to 1GB

9da8b54

Signed-off-by: Joseph Schuchart <[email protected]>

accelerator/cuda: fix error message

93ded5e

Signed-off-by: Joseph Schuchart <[email protected]>

CUDA: Select compute capability 52 by default

182e6fa

Signed-off-by: Joseph Schuchart <[email protected]>

Sqash const correctness warnings

e5eb45f

Signed-off-by: Joseph Schuchart <[email protected]>

Squash warnings about mismatched function pointer types

14a5372

Signed-off-by: Joseph Schuchart <[email protected]>

Squash printfs

1f63809

Signed-off-by: Joseph Schuchart <[email protected]>

Replace fprintf with show_help

3d9f33a

Signed-off-by: Joseph Schuchart <[email protected]>

Squash compiler warnings

c878c4f

Signed-off-by: Joseph Schuchart <[email protected]>

Clean up cuda and rocm op codes

1c6667d

Signed-off-by: Joseph Schuchart <[email protected]>

Minor tweak to CUDA op configury

7bb4b95

Signed-off-by: Joseph Schuchart <[email protected]>

devreal force-pushed the device_op_cuda branch from 11e6d2a to 7bb4b95 Compare November 7, 2023 23:48

Fix rebase errors

d1382c3

Signed-off-by: Joseph Schuchart <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Device op cuda #46

Device op cuda #46

Uh oh!

devreal commented Nov 6, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Nov 6, 2023

Uh oh!

github-actions bot commented Nov 6, 2023

Uh oh!

github-actions bot commented Nov 6, 2023

Uh oh!

github-actions bot commented Nov 7, 2023

Uh oh!

github-actions bot commented Nov 7, 2023

Uh oh!

github-actions bot commented Nov 7, 2023

Uh oh!

Uh oh!

Device op cuda #46

Are you sure you want to change the base?

Device op cuda #46

Uh oh!

Conversation

devreal commented Nov 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 6, 2023

Uh oh!

github-actions bot commented Nov 6, 2023

Uh oh!

github-actions bot commented Nov 6, 2023

Uh oh!

github-actions bot commented Nov 7, 2023

Uh oh!

github-actions bot commented Nov 7, 2023

Uh oh!

github-actions bot commented Nov 7, 2023

Uh oh!

Uh oh!

devreal commented Nov 6, 2023 •

edited

Loading