Skip to content
This repository has been archived by the owner on Feb 3, 2021. It is now read-only.

Feature: Sparklyr #243

Merged
merged 7 commits into from
Dec 8, 2017
Merged

Feature: Sparklyr #243

merged 7 commits into from
Dec 8, 2017

Conversation

brnleehng
Copy link
Contributor

No description provided.

jafreck
jafreck reviewed Nov 30, 2017
View reviewed changes
Copy link
Member

@jafreck jafreck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice looks good!

custom-scripts/rstudio_server.sh

rstudio-server start

fi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: add newline

docker-image/r/spark1.6.3/Dockerfile
&& apt-get autoclean -y \
&& rm -rf /var/lib/apt/lists/*

CMD ["/bin/bash"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: add newline

docker-image/r/spark2.1.0/Dockerfile
&& apt-get autoclean -y \
&& rm -rf /var/lib/apt/lists/*

CMD ["/bin/bash"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newline

docker-image/r/spark2.2.0/Dockerfile
&& apt-get autoclean -y \
&& rm -rf /var/lib/apt/lists/*

CMD ["/bin/bash"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newline

jiata
jiata reviewed Dec 1, 2017
View reviewed changes
custom-scripts/rstudio_server.sh

if [ "$IS_MASTER" = "1" ]; then

## Download and install Rstudio Server
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how long does this take? Can we package this into the docker image, and basically only do the setup here (instead of install too?).
Not sure if increase in time difference is worth the effort though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we allowed to redistribute RStudioServer like that? Downloading it on the fly seems OK, but packaging it up seems a little bit more risky. Any idea what the terms of agreement are? If we don't know, then it's best to leave this as-is.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Downloading and installing Rstudio Server takes roughly 15 secs.. The installation package itself is only 60mb.

Also we only need rstudio server installed on the master. Not sure about terms of agreements are.. But we are only using Rstudio server open source. So I think it would be fine

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rstudio server is AGPLv3. So unmodified redistribution, I believe, is fine.

This is the relevant section of the license:
To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work.

@brnleehng brnleehng force-pushed the feature/sparklyr branch 2 times, most recently from eb3c638 to fbad579 Compare December 5, 2017 18:55
@brnleehng brnleehng merged commit 0efadef into master Dec 8, 2017
@brnleehng brnleehng deleted the feature/sparklyr branch December 8, 2017 19:17
jafreck pushed a commit that referenced this pull request Jan 9, 2018
@emlyn @jafreck
Update VNET feature branch (#284)
4b7cb1f 
* remove redundant setting in non-master code section and use non-os drive to mount HDFS (#242)

* Feature: Azure Files (#241)

* initial take on installing azure files

* fix cluster.yaml parsing of files shares

* remove test code

* add docs for Azure Files

* Feature: Rename SDK (#231)

* initial refactor

* rename cli_fe to cli

* add docs for sdk client

* typo

* remove conflict

* fix zip node scripts bug, add sdk_example program

* start models docs

* add ClusterConfiguration docs, fix merge bug

* Application docs update

* added Application and SparkConfiguration docs

* whitespace

* rename cli.py and spark/cli

* add docstring for load_spark_client

* Bug: fix bad reference to FileShare (#245)

* Feature: Spark GPU (#206)

* conditionally install and use nvidia-docker

* status statements, and -y flag for install

* add example, remove unnecessary ppa

* rename custom script, remove print statement, update example

* add Dockerfile

* fix path in Dockerfile

* update Docker images to use service account

* updated docs, changed default docker repo for gpu skus

* make timing statements more verbose

* remove unnecessary script

* added gpu docs

* fix up docs and numba example

* Feature: update docker image doc (#251)

* update docker-image readme with new images

* update docs

* Update 60-gpu.md (#253)

* Update 60-gpu.md

make sure is available in region

* Update 60-gpu.md

* Feature: Sparklyr (#243)

* Added rstudio server script

* Added rstudio server port to aztk sdk

* Added R dockerfiles

* Added new line on dockerfiles

* Pointing dockerfiles to new aztk-base

* allow any user or application in the server to write to the history server log directory

* Retry asking for password when it doesn't match or is empty (#252)

* Retry asking for password when it doesn't match or is empty

* Limit to 3 retries and let user know of add-user command on failure

* Throw error on failure

* Bug: fix wrong path for global secrets (#265)

* fix wrong path for global secrets

* load spark_conf files correctly

* docker-image docs fix

* docker-image docs fix

* move load_aztk_spark_config function to config.py

* Feature: Default Spark filesystem master HA (#271)

* add default filesystem master ha

* move settings to spark-defaults.conf

* whitespace

* Docs: update (#263)

* Update README.md

streamline and update main readme.md

* Update README.md

* Update README.md

* Update 13-configuration.md

* Update 12-docker-image.md

* Update 12-docker-image.md

* Update README.md

* Create README.md

* Update README.md

* Update 10-clusters.md

* Feature: add feedback for cluster create wait (#273)

* add feedback for cluster create wait

* whitespace

* alphasort imports

* Bug: fix loading local spark config (#282)

* Fix secrets.yaml format and add service principal for storage

* Feature: update to v0.5.0 (#283)

* Pass credentials through to node scripts

* Bug: History server parse file not exist (#288)

* jupyter azfiles bug + gpu sample (#291)

* gpu sample + jupyter mnt point

* rename jupyter gpu sample

* Check for both service principal and shared key auth

* More checks

* Bug: fix logic for worker custom scripts (#295)

* Bug: suppress warning on add-user (#302)

* Bug: fix alignment in get print cluster (#312)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@jiata jiata jiata left review comments

@jafreck jafreck jafreck approved these changes

@paselem paselem Awaiting requested review from paselem

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@brnleehng @paselem @jiata @jafreck