Start Guide for RAPIDS on AWS EMR 6.2

[mgzhao]

No description provided.

[jlowe]

In addition to the specific comments I have some general ones:

• It would be nice to have all the files specific to AWS EMR be placed in a directory that is specific to EMR (e.g.: docs/get-started/aws-emr). Otherwise the docs/get-started directory is going to get even more cluttered as more platforms are added.
• The docs/FAQ.md should be updated to add a link to the new getting-started-aws-emr.md document.

[jlowe]

Elsewhere in the docs/ directory we're using img as the directory name to hold images. It would be nice to keep this consistent. Also RAPIDS should be all caps in the filenames. Interesting it's the only part of the image filenames that is not capitalized. 😄

[krajendrannv]

@mgzhao Tom pointed out that the config.json didn't have the right settings. I suggest that we provide a sample config.json here. For example; in this cluster you chose g4dn.2xlarge, which has 8 vCPU cores. The right config would be use all the cores, we need to spark.executor.cores=8, spark.task.resource.gpu.amount=0.125.
Here is the sample config for this cluster:
{
"Classification":"spark-defaults",
"Properties":{
"spark.plugins":"com.nvidia.spark.SQLPlugin",
"spark.sql.sources.useV1SourceList":"",
"spark.executor.extraJavaOptions":"-Dai.rapids.cudf.prefer-pinned=true",
"spark.executor.resource.gpu.discoveryScript":"/usr/lib/spark/scripts/gpu/getGpusResources.sh",
"spark.submit.pyFiles":"/usr/lib/spark/jars/xgboost4j-spark_3.0-1.0.0-0.2.0.jar",
"spark.executor.extraLibraryPath":"/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/compat/lib:/usr/local/cuda/lib:/usr/local/cuda/lib64:/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native",
"spark.rapids.sql.concurrentGpuTasks":"4",
"spark.executor.resource.gpu.amount":"1",
"spark.executor.cores":"8",
"spark.task.cpus ":"1",
"spark.task.resource.gpu.amount":"0.125",
"spark.rapids.memory.pinnedPool.size":"2G",
"spark.executor.memoryOverhead":"2G",
"spark.locality.wait":"0s",
"spark.sql.shuffle.partitions":"200",
"spark.sql.files.maxPartitionBytes":"256m",
"spark.sql.adaptive.enabled":"false"
}

[sameerz]

I updated the FAQ.md and created subdirectories in demos and img for different CSPs. I also updated the file names. I believe Kong had already updated the config settings in the json.

[tgravescs]

some of hte configs in EMR guide are not right or should be adjusted for cluster setup.

[jlowe]

Minor extra suggestion, but I'm going to go ahead and approve this now since I'll be on vacation for a bit. I don't want to totally block this for the 0.3 release. However it would be good to get a review from @sameerz or @krajendrannv before committing this.

[sameerz]

I updated the content, and modified a few related files (FAQ.md, the databricks and gcp dataproc getting started guides) as the img locations were also cleaned up. Please review.

[jlowe]

Looks good but there are some conflicts that need to be resolved.

[sameerz]

I think I resolved all the comments. I am not sure how to resolve the conflicting files - I followed the command line instructions, but no changes appear to have been applied or resolved.

[jlowe]

I followed the command line instructions, but no changes appear to have been applied or resolved.

Did you have the latest branch-0.3 changes in your local repository when you performed the merge from branch-0.3? That would explain the lack of changes.

[sameerz]

build