workspace_setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Workspace setup\n",
    "\n",
    "<table align=\"left\">\n",
    "\n",
    "  <td>\n",
    "    <a href=\"https://github.com/verily-src/workbench-examples/blob/main/workspace_setup.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/verily-src/workbench-examples/main/workspace_setup.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      Open in a Verily Workbench notebook instance\n",
    "    </a>\n",
    "  </td>                                                                                               \n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook sets up some resources expected to exist for Verily Workbench tutorials. Add more setup to this notebook to meet your needs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d975e698c9a4"
   },
   "source": [
    "### Objective\n",
    "\n",
    "Perform common workspace setup tasks including:\n",
    "\n",
    "1. Creating the Cloud Storage buckets used in Verily Workbench tutorials.\n",
    "1. Creating the BigQuery dataset used in Verily Workbench tutorials.\n",
    "\n",
    "#### How to run this notebook\n",
    "\n",
    "Run this notebook cell by cell to set up your workspace. All setup steps are optional but highly recommended so that your workspace is compatible with the Verily Workbench tutorials.\n",
    "\n",
    "#### Costs\n",
    "\n",
    "This notebook takes less than a minute to run, which will typically cost less than $0.01 of compute time on your cloud environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup and Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not os.getenv('GOOGLE_CLOUD_PROJECT'):\n",
    "    raise Exception('Expected environment variables are not available. Please let workbench-support@verily.com know.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the Workbench CLI to create some default workspace resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These default workspace resources are  assumed to exist by Verily Workbench training materials. Specifically, the \"self-cleaning\" Google Cloud Storage bucket and BigQuery dataset are useful for tutorials. Those tutorials can create GCS files and BigQuery tables that you don't need to remember to clean up after the tutorial has been completed because they are created in the \"self-cleaning\" storage resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, run the following cell to confirm that you are using the workspace that you intend to work in. (You can also run `wb workspace list` to see your list of workspaces)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!wb status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at your workspace resources prior to creating these default resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!wb resource list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Cloud Storage buckets\n",
    "\n",
    "Create two Cloud Storage buckets in your workspace with the following workspace reference names:\n",
    "\n",
    "- `ws_files`: The Verily Workbench utility to share notebooks with other Workbench users will write files to this durable default bucket.\n",
    "- `ws_files_autodelete_after_two_weeks`: The code in Verily Workbench tutorials will write output files to the \"autodelete\" bucket by default. Any file in this bucket will be automatically deleted two weeks after it is written. This alleviates the need for you to remember to clean up temporary and example files manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!wb resource resolve --name ws_files || wb resource create gcs-bucket \\\n",
    "    --name=ws_files \\\n",
    "    --bucket-name=${GOOGLE_CLOUD_PROJECT}-ws-files \\\n",
    "    --cloning=COPY_NOTHING \\\n",
    "    --description=\"Bucket for reports and provenance records.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!wb resource resolve --name ws_files_autodelete_after_two_weeks || wb resource create gcs-bucket \\\n",
    "    --name=ws_files_autodelete_after_two_weeks \\\n",
    "    --bucket-name=${GOOGLE_CLOUD_PROJECT}-autodelete-after-two-weeks \\\n",
    "    --cloning=COPY_NOTHING \\\n",
    "    --auto-delete=14 \\\n",
    "    --description=\"Bucket for temporary storage of file data. Send test outputs here for automatic cleanup after two weeks.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a BigQuery dataset\n",
    "\n",
    "Create a BigQuery dataset in your workspace with reference name `tabular_data_autodelete_after_two_weeks`.\n",
    "The code in Verily Workbench tutorials will write BigQuery tables to the \"autodelete\" dataset by default.\n",
    "Any table in this dataset will be automatically deleted two weeks after it is written.\n",
    "This alleviates the need for you to remember to clean up temporary and example tables manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!wb resource resolve --name tabular_data_autodelete_after_two_weeks || wb resource create bq-dataset \\\n",
    "    --name=tabular_data_autodelete_after_two_weeks \\\n",
    "    --dataset-id=tabular_data_autodelete_after_two_weeks \\\n",
    "    --cloning=COPY_NOTHING \\\n",
    "    --default-table-lifetime=1209600 \\\n",
    "    --description=\"BigQuery dataset for temporary storage of tabular data. Send test outputs here for automatic cleanup after two weeks.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at your workspace resources after creating these default resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!wb resource list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Provenance\n",
    "\n",
    "Generate information about this notebook environment and the packages installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conda and pip installed packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!conda env export"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JupyterLab extensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!jupyter labextension list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of cores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!grep ^processor /proc/cpuinfo | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!grep \"^MemTotal:\" /proc/meminfo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Copyright 2024 Verily Life Sciences LLC\n",
    "\n",
    "Use of this source code is governed by a BSD-style   \n",
    "license that can be found in the LICENSE file or at   \n",
    "https://developers.google.com/open-source/licenses/bsd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": ".m115",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/:m115"
  },
  "kernelspec": {
   "display_name": "Python 3 (Local)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}