Support mounting host directories into pods

[vishh]

Having the ability to mount local directories to pods will improve the developer experience a lot.

• Builds can be done in a local kube cluster
• Testing changes to an app will be much faster, without requiring creation of a docker container and a registry.

Given that most hypervisors can already support mapping host directories, adding this feature should be straightforward.

[dlorenc]

One possibility here might be to support an NFS-based persistent volume claim from the guest to the host: http://kubernetes.io/docs/user-guide/persistent-volumes/

[rata]

On Mon, May 16, 2016 at 05:31:18PM -0700, dlorenc wrote:

One possibility here might be to support an NFS-based persistent volume claim from the guest to the host: http://kubernetes.io/docs/user-guide/persistent-volumes/

NFS persistent volume claim? What? Why?

Why not HostPath, to use a local directory, and that's all regarding the volume?

[dlorenc]

Of course, HostPath will work right now, but it won't mount things into the VM from the outside host.

For that to work we might need to do dynamic mounting/provisioning from the laptop into the VM.

[rata]

On Mon, May 16, 2016 at 05:42:00PM -0700, dlorenc wrote:

Of course, HostPath will work right now, but it won't mount things into the VM from the outside host.

Oh, I see what you mean now. But I'm not sure, as @vishh said in the original
report: "most hypervisors can already support mapping host directories". So,
maybe a host dir can be mounted in the VM and then in the container.

NFS is another option, but I'm not sure how that would work. A host dir in the
VM, that dir exported as NFS, and mounted by k8s as an NFS volume? Or what were
you thinking?

[rata]

Also, if docker is used, it can mount from host directly on mac/windows with docker "native" beta. One option is to use that.

Another is to do the same/similar trick that docker is using (probably the one @vishh said and the one I said in my previous post)

[vishh]

If we can have kubelet take in a configurable root directory for hostPath volumes, then we can have the kubelet map in directories from inside the VM that mirror the host root.

[Ehekatl]

this feature is super important for us, is there any way to let me add a shared folder before this come out ?

[dlorenc]

Hey, on a Mac using Virtualbox, the /Users/ directory is automatically mapped into the VM (also under /Users). You could then use a HostPath volume of "/Users/$USER/$DIR". Does that help?

[Ehekatl]

@dlorenc thx, I add mount -t command at boot2docker init script now, but hostpath volume will facing some permission issue (e.g. postgres container), seems nfs is the only workaround

[solsson]

The permission issues that @Ehekatl mentions are quite a blocker. With a PersistentVolume mounted to a pod with a https://hub.docker.com/_/postgres/ container I get:

root@postgres-0:/# ls -la /var/lib/postgresql/data/
drwxr-xr-x 2 root     root       40 Jul 18 15:05 .
drwxr-xr-x 3 postgres postgres 4096 Jul 18 15:09 ..

When postgres tries to create data there it fails with mkdir: cannot create directory ‘/var/lib/postgresql/data/pg_xlog’: Permission denied. No chmod or chown on the host system seems to make any difference. There's plenty of official and non-official Docker images that run a non-privileged user. Postgres prepends its start command with gosu postgres.

root@postgres-0:/# id     
uid=0(root) gid=0(root) groups=0(root)
root@postgres-0:/# gosu postgres id
uid=999(postgres) gid=999(postgres) groups=999(postgres),109(ssl-cert)

I've tested with both virtualbox and xhyve minikube, on Mac.

[rata]

@solsson: maybe it's an option to add permission to the volume API in kubernetes. I'm doing that for others volumes: kubernetes/kubernetes#28733 and kubernetes/kubernetes#28936

[solsson]

I noticed that kubernetes/kubernetes#28733 is merged now. Interesting.

Before Docker for Mac, with Docker Toolbox and boot2docker, I had this kind if issue often with plain docker containers. Hence I'm not so sure the issue is with Kubernetes. Maybe it's inherent with docker + non-privileged user + virtual machine mount.

If I use /tmp/somevolume as hostPath I do get the same issue at postgres start. But the difference is that I can chmod go+ws /tmp/somevolume to solve it (in virtualbox minikube).

This leads me to believe that the problem is with minikube. After all hostPath works fine on for example GKE. Maybe it's with how minikube mounts the shared folder?

[solsson]

This boot2docker issue describes the problem pretty accurately.

I think I found a workaround for virtualbox. After minikube start open the vm in virtualbox (or ssh) and run

umount /Users
mount -t vboxsf -o rw,nodev,relatime,suid,dmode=777,fmode=777 Users /Users

The first three options are from the default mount. Probably only one of the added options are needed.

With postgres howerver the next issue is:

LOG:  could not link file "pg_xlog/xlogtemp.20" to "pg_xlog/000000010000000000000001": Operation not permitted
FATAL:  could not open file "pg_xlog/000000010000000000000001": No such file or directory

which I think confims my suspicion that docker and vm shared folders is not a good fit.

[solsson]

In conclusion, an added mount option (see above) might improve minikube. But until native docker is supported, including Docker for Mac, I think that to stay sane one should realize that some services demand a lot from the file system and hostPath should really be the linux host's folder.

With /var/kubedata/pgdata-postgres-0 postgres runs fine. I noticed that /var has more liberal permissions than /tmp that I tested above.

So to access this data directly I'd have to share it from the VM to my physical host.

[rata]

On Mon, Jul 18, 2016 at 10:49:17AM -0700, solsson wrote:

This boot2docker issue describes the issue pretty accurately.

I think I found a workaround for virtualbox. After minikube start open the vm in virtualbox (or ssh) and run

umount /Users
mount -t vboxsf -o rw,nodev,relatime,suid,dmode=777,fmode=777 Users /Users

The first three options are from the default mount. Probably only one of the added options are needed.

With postgres howerver the next issue is:

LOG:  could not link file "pg_xlog/xlogtemp.20" to "pg_xlog/000000010000000000000001": Operation not permitted
FATAL:  could not open file "pg_xlog/000000010000000000000001": No such file or directory

which I think confims my suspicion that docker and vm shared folders is not a good fit.

Why it is not a good fit? Why do those errors happen?

[boj]

These are the steps I look to get hostPath working on OSX with NFS. Using xhyve, but I imagine this works for other VMs.

# install bits
brew install docker kubectl docker-machine-driver-xhyve
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

# setup bits
sudo echo "/Users -network 192.168.64.0 -mask 255.255.255.0 -alldirs -maproot=root:wheel" > /etc/exports
sudo nfsd restart
minikube start --vm-driver=xhyve
minikube ssh -- sudo /usr/local/etc/init.d/nfs-client start
minikube ssh -- sudo mkdir /Users
minikube ssh -- sudo mount 192.168.64.1:/Users /Users -o rw,async,noatime,rsize=32768,wsize=32768,proto=tcp
eval $(minikube docker-env)

[solsson]

Why it is not a good fit? Why do those errors happen?

Because vboxsf mounts are quite limited as file system. I did not investigate the postgres issue further.

For vbox minikube based on your NFS instructions (with minikube ip = 192.168.99.106) I did:

echo "/Users -network 192.168.99.0 -mask 255.255.255.0 -alldirs -maproot=root:wheel" | sudo tee -a /etc/exports
sudo nfsd restart
minikube start
minikube ssh -- sudo umount /Users
minikube ssh -- sudo /usr/local/etc/init.d/nfs-client start
minikube ssh -- sudo mount 192.168.99.1:/Users /Users -o rw,async,noatime,rsize=32768,wsize=32768,proto=tcp

Much better than shared folder, as uid + gid + mod is preserved.

[dlorenc]

The existing vboxfs system isn't great. We should figure out whether to use NFS or 9p, and make this work for all drivers.

[dlorenc]

We should document how to setup NFS in the meantime.

[gjcarneiro]

Latest minikube 0.15 iso image doesn't support vboxsf anymore, it seems. I also tried NFS to no avail.

Anyway, IMHO NFS is /not/ better than vboxsf. I usually have some python code running in gunicorn with the --reload option, which uses inotify. NFS doesn't support inotify, so it will be probably have to revert to polling, which is slower and more resource consuming.

Even NFS I couldn't get to work:

$ cat /etc/lfs-release 
NAME=Buildroot
VERSION=2016.08
ID=buildroot
VERSION_ID=2016.08
PRETTY_NAME="Buildroot 2016.08"
$ sudo mount -t nfs 192.168.99.1:/home /mnt/hosthome
mount: wrong fs type, bad option, bad superblock on 192.168.99.1:/home,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

How can you release a minikube version without any working option for sharing folders between guest and host?... :(