Nvidia contraner with jupyterlab_tensorboard

[dmitrysarov]

Maybe this Issue not really relevant to the repository but I do not know where else I can ask it
I am using release PyTorch Release 20.12 from nvidia https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_20-12.html#rel_20-12 with jupyter_tensorboard
But can not run tensorboard inside jupyter. Got following

Any ideas why? And how to fix?

[monsieurborges]

Hey @dmitrysarov ,

I think I know you from somewhere else (laughs). Maybe from pytorch_scatter?
I had the same problem as you using nvcr.io/nvidia/pytorch:20.12-py3.

The problem appears from tensorboard == 2.3 as reported by @NanoCode012 in #25.

Below I leave an example of how you can solve this:

# INSTALL JUPYTERLAB
# Keep compatibility with jupyterlab_tensorboard
# jupyterlab==2.2.9
# tensorboard==2.2.1
# https://github.com/chaoleili/jupyterlab_tensorboard/issues/25
pip install --no-cache-dir --upgrade \
    pip \
    jupyterlab==2.2.9 \
    jupyterlab_widgets \
    ipywidgets

# Remove NVIDIA Tensorboard
pip uninstall --no-cache-dir --yes \
    tensorboard nvidia-tensorboard nvidia-tensorboard-plugin-dlprof

pip install --no-cache-dir --upgrade \
    tensorboard==2.2.1 \
    tensorboard_plugin_profile \
    jupyter-tensorboard==0.2.0

jupyter notebook --generate-config

jupyter labextension uninstall jupyterlab_tensorboard --no-build
jupyter labextension install jupyterlab_tensorboard --no-build

jupyter lab clean
jupyter lab build

jupyter labextension list

[dmitrysarov]

@mgomesborges once again thanks for answering =)
It strange because pip list shows me tensorboard==1.15.0 inside container
I add your solution to the bottom of the Dockerfile

  RUN pip install --no-cache-dir --upgrade \
      pip \
      jupyterlab==2.2.9 \
      jupyterlab_widgets \
      ipywidgets
  
  # Remove NVIDIA Tensorboard
  RUN pip uninstall --no-cache-dir --yes \
      tensorboard nvidia-tensorboard nvidia-tensorboard-plugin-dlprof && \
	  pip install --no-cache-dir --upgrade \
      tensorboard==2.2.1 \
      tensorboard_plugin_profile \
      jupyter-tensorboard==0.2.0 && \
	  jupyter notebook --generate-config -y && \
	  conda install nodejs && \
	  jupyter labextension uninstall jupyterlab_tensorboard --no-build && \
	  jupyter labextension install jupyterlab_tensorboard --no-build && \
	  jupyter lab clean && \
	  jupyter lab build

Now, it probably another issue, but I get the following ( like TB can not see anything in the folder, but there are valid log files)

[monsieurborges]

I don't think @chaoleili will be happy that we use this space that way, but here we go:

1. About the error message, you need to provide information about the commands you are using, so we will know what is wrong.

2. If you want to improve your Dockerfile, here are some tips:

# Dockerfile TEMPLATE
FROM nvcr.io/nvidia/pytorch:20.12-py3

# Node Version Manager
ENV NVM_DIR=/usr/local/nvm

# CREATE A NEW DOCKER IMAGE
ENV DEBIAN_FRONTEND noninteractive
RUN echo "Welcome to the World baby!"\
    # INSTALL NODE
    #----------------------------------------------------------
    && source ${NVM_DIR}/nvm.sh \
    # INSTALL JUPYTERLAB
    #----------------------------------------------------------
    # INSTALL JUPYTERLAB
    # Keep compatibility with jupyterlab_tensorboard
    # jupyterlab==2.2.9
    # tensorboard==2.2.1
    # https://github.com/chaoleili/jupyterlab_tensorboard/issues/25
    && pip install --no-cache-dir --upgrade \
        pip \
        jupyterlab==2.2.9 \
        jupyterlab_widgets \
        ipywidgets \
    # Remove NVIDIA Tensorboard
    && pip uninstall --no-cache-dir --yes \
        tensorboard nvidia-tensorboard nvidia-tensorboard-plugin-dlprof \
    # Install Tensorboard
    && pip install --no-cache-dir --upgrade \
        tensorboard==2.2.1 \
        tensorboard_plugin_profile \
        jupyter-tensorboard==0.2.0 \
    && jupyter notebook --generate-config \
    # Install JupyterLab extension
    && jupyter labextension uninstall jupyterlab_tensorboard --no-build \
    && jupyter labextension install jupyterlab_tensorboard --no-build \
    && jupyter lab clean \
    && jupyter lab build \
    && jupyter labextension list

[dmitrysarov]

I literally grab your Dockerfile and launch, TL DR no success
Console output

[monsieurborges]

@dmitrysarov it seems to me that you have a problem with your internet network or maybe proxy.

If you know your proxy's address try something like this:

# Dockerfile TEMPLATE
FROM nvcr.io/nvidia/pytorch:20.12-py3

# PROXY
ENV http_proxy="HTTP://PROXY_ADDRESS_HERE"
ENV https_proxy="HTTP://PROXY_ADDRESS_HERE"
ENV ftp_proxy="HTTP://PROXY_ADDRESS_HERE"

. . .

[monsieurborges]

@dmitrysarov could you give us a feedback, so we can close this issue!

[dmitrysarov]

No success. I tried your Dockerfile on several machines same error message in the console

So the reason is not proxy.
It looks like a mystery to me.
Can you confirm that you check it and it has worked?

[monsieurborges]

Hey @dmitrysarov, I am sorry to hear that.

YES, my script works fine for me either with or without proxy!

As you said, you tested it on different machines and the problem is repeated. Make sure it is something related to your internet connection or some access block.

@chaoleili could you close this request?

[cliffwoolley]

Hi all, FYI the original issue here is fixed up in our 21.02 version of NVIDIA containers due to be released very soon -- the fixes needed were in jupyter-tensorboard (see lspvic/jupyter_tensorboard#63 ), not jupyterlab-tensorboard per se.