default rmm alloc fraction to the max to avoid unnecessary fragmentation #2846
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Rong Ou <rong.ou@gmail.com>
rongou
added
shuffle
jlowe
reviewed
Jun 30, 2021
| @@ -330,7 +330,7 @@ object RapidsConf { | |||
| s"configured via $RMM_ALLOC_MAX_FRACTION_KEY.") | |||
| .doubleConf | |||
| .checkValue(v => v >= 0 && v <= 1, "The fraction value must be in [0, 1].") | |||
| .createWithDefault(0.9) | |||
| .createWithDefault(1) | |||
There was a problem hiding this comment.
Changing this default means a new warning now appears on startup with the default configs. For example:
21/06/30 14:38:36 WARN GpuDeviceManager: Initial RMM allocation (15568.75 MB) is larger than the adjusted maximum allocation (15136.5 MB), lowering initial allocation to the adjusted maximum allocation.
It would be nice to not emit warnings with the default configs on most setups. We should consider whether this should be a warning or just an info, or maybe change the conditions under which it is a warning.
There was a problem hiding this comment.
I would like to see us to take a deep look at how we are calculating the RMM pool size etc.
We have several configs associated with this.
spark.rapids.memory.gpu.allocFraction- 0 to 1 for the percentage of free memory allocated for the RMM pool. Percentage of free memory, but docs say available memory.spark.rapids.memory.gpu.maxAllocFraction- 0 to 1 maximum amount of total GPU memory that can be allocated for the RMM pool. This is percentage of total memory except we also remove from it a reserved but reserved is from total, so if I have 1 GiB used already reserved is not doing anything.spark.rapids.memory.gpu.minAllocFraction- 0 to 1 minimum amount of total GPU memory that can be allocated for the RMM pool before we throw an error saying it is too small. this is a percentage of the total memory.spark.rapids.memory.gpu.reserve- number of bytes that have to be reserved for cuda/cudf to load kernels etc so we don't crash because RMM has everything allocated.
The code that does the warnings is a bit confusing too.
- If initial allocation < min allocation throw an exception
- if max allocation < min allocation throw an exception
- if reserve amount >= max allocation throw an exception (not really sure what this is doing at all. If max and min are small but reserve is big that should be OK so long as everything fits.
- adjust max allocation by subtracting reserve amount from it.
- if adjusted max allocation <= initial allocation output a warning.
So to me it feels like we should instead.
- get the reserved amount
- get the amount of available memory on the GPU by taking free and subtracting reserved from it.
- min allocation is calculated from the total memory on the GPU
- if available is <= min allocation throw an exception this is not going to work.
- max allocation is a percentage of the available memory, not just the free memory
- initial allocation is a percentage of the available memory.
- if initial allocation > max allocation output a warning and adjust initial allocation down to max allocation.
We will need to update the docs in the config section accordingly to better explain what is happening.
There was a problem hiding this comment.
OK, changed how initial allocation is calculated to take into account the reserve amount. To me this seems to make sense since the most you can allocate is free-reserve.
|
Please retarget to the 21.10 branch when it is available. |
rongou
requested review from
GaryShen2008,
NvTimLiu and
tgravescs
as code owners
Signed-off-by: Rong Ou <rong.ou@gmail.com>
jlowe
approved these changes
Aug 2, 2021
|
build |
razajafri
pushed a commit
to razajafri/spark-rapids
that referenced
this pull request
Aug 23, 2021
…ion (NVIDIA#2846) * default rmm alloc fraction to the max to avoid unnecessary fragmentation Signed-off-by: Rong Ou <rong.ou@gmail.com> * initial allocation takes into account of the reserve Signed-off-by: Rong Ou <rong.ou@gmail.com> Signed-off-by: Raza Jafri <rjafri@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With the current default of
0.9forspark.rapids.memory.gpu.allocFraction, we first allocate 90% of GPU memory, and then when we run out of memory additional GPU memories are allocated. Because of how CUDA Unified Virtual Addressing (UVA) is set up, subsequently allocated addresses are discontinuous, causing unnecessary fragmentation.Helps with #2504 but doesn't completely fixes it. Having a single continuous GPU address space also helps with debugging and visualization.
Signed-off-by: Rong Ou rong.ou@gmail.com